Persistent Reservations

The Linux SCSI Target Wiki

SCSI-3 and SCSI-4 Persistent Reservations (PRs, target_core_pr.c) are a feature that is used for a number of cluster storage configurations for I/O fencing and failover/retakeover. PRs are a SCSI protocol feature, so they have been implemented in the LIO engine, and hence can be used for all SCSI transport fabrics, such as FCoE, Fibre Channel, iSCSI, Serial Attached SCSI (SAS), etc. Using PRs with Asymmetric Logical Unit Assignment (ALUA) is also fully supported in LinuxIO.

Contents 1 Standards compliance 1.1 SPC-4 1.2 PROUT Service Actions 1.3 PROUT Reservation Types 1.4 PRIN Service Actions 2 Test and Verification 2.1 RHEL/CentOS ≥5.3 2.2 Microsoft Server 2008/R2 Failover Cluster 2.3 Other 3 ConfigFS object tree 3.1 Target 3.2 SPC-3 Reservations 3.3 T10 WWN numbers 4 Initiator 4.1 sg_persist 4.2 sg3_utils 4.3 fence_scsi_test 5 See also

Standards compliance

The SPC-3/SPC-4 features implemented (in target_core_pr.c) are described below. For legacy environments, compatible reservation handling for SPC-2 needs to be enabled.

SPC-4

LinuxIO implements full Persistent Reservation functionality, which entails two SPC-4 defined features:

• APTPL=1 (Activate Persist across Target Power Loss) is using key=value /var/target/pr/aptpl_$T10_UNIT_SERIAL using sg_persist (announcement).
• SP_I_PT=1 (Specify Initiator Ports Capable) bit is supported and tested using sg_persist (announcement).

PROUT Service Actions

All PROUT Service Actions have been implemented: REGISTER, RESERVE, RELEASE, CLEAR, REGISTER_AND_IGNORE, PREEMPT, PREEMPT_AND_ABORT and REGISTER_AND_MOVE

PROUT Reservation Types

All PROUT Reservation Types have been implemented: Write Exclusive, Exclusive Access, Write Exclusive Registrants Only, Exclusive Access Registrants Only Write Exclusive All Registrants, Exclusive Access All Registrants

PRIN Service Actions

All PRIN Service Actions have been implemented: READ_KEYS, READ_RESERVATION, REPORT_CAPABILITIES, READ_FULL_STATUS

Test and Verification

LinuxIO IBLOCK, FILEIO and RAMDISK subsystem plugins are passing SPC-3 level validation for iSCSI target PRs. The following cluster clients that use Linux-iSCSI target mode PR have been tested.

RHEL/CentOS ≥5.3

Test and validation for RHEL/CentOS was performed with GFS (e.g., with multiple writers). LinuxIO capable storage objects from IBLOCK, FILEIO, and RAMDISK pass the fence_scsi_test.

RHEL/CentOS uses SCSI I/O Fencing via the sg_persist CLI operation in sg3_util for Write Exclusive, Registrants Only and PREEMPT_AND_ABORT takeover using ext3 mounts.

SPC-3 PR emulation is disabled on top of the pass-through SCSI subsystem plugin (this is WIP for existing SCSI devices).

Microsoft Server 2008/R2 Failover Cluster

As of April 2009, LinuxIO is passing validation using the MS Verification test suite. Test and validation for Microsoft Server 2008 Failover Cluster was performed using the MS domain validation suite.

Microsoft Server 2008 Failover Cluster uses Write Exclusive, Registrants Only and PROUT PREEMPT for production failure cases.

Extensive work by the LinuxIO developer and user community has made this possible. If you are interested in further testing and want to deploy production services on our code please let us know.

Other

If you have an iSCSI Initiator that implements SPC-3 PR logic and would like to test it against LIO 3, please don't hesitate to contact us on our development mailing list.

ConfigFS object tree

Target

Target/configFS layout. This is the output of an IBLOCK exported Linux/LVM DM block_device (lvm_test0) via tree:

tree /sys/kernel/config/target/core
|-- iblock_0
|   |-- hba_info
|   `-- lvm_test0
|       |-- alua_lu_gp
|       |-- attrib
|       |   |-- block_size
|       |   |-- emulate_tas
|       |   |-- emulate_ua_intlck_ctrl
|       |   |-- hw_block_size
|       |   |-- hw_max_sectors
|       |   |-- hw_queue_depth
|       |   |-- max_sectors
|       |   |-- queue_depth
|       |   `-- task_timeout
|       |-- control
|       |-- enable
|       |-- fd
|       |-- info
|       |-- pr
|       |   |-- res_holder
|       |   |-- res_pr_all_tgt_pts
|       |   |-- res_pr_generation
|       |   |-- res_pr_holder_tg_port
|       |   |-- res_pr_registered_i_pts
|       |   |-- res_pr_type
|       |   `-- res_type
|       `-- wwn
|           |-- vpd_assoc_logical_unit
|           |-- vpd_assoc_scsi_target_device
|           |-- vpd_assoc_target_port
|           |-- vpd_protocol_identifier
|           `-- vpd_unit_serial

SPC-3 Reservations

The iblock_0/lvm_test0/pr/ configFS attributes are used to see the current reservation/registration status for different SCSI I_T nexuses:

cat /sys/kernel/config/target/core/iblock_0/lvm_test0/pr/res_*
SPC-3 Reservation: iSCSI Initiator: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 Reservation: All {{Target}} ports registration 0x00000002
SPC-3 Reservation: iSCSI target node Endpoint: iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd
SPC-3 Reservation: Relative Port Identifer Tag: 2 iSCSI Portal Group Tag: 1 iSCSI Logical Unit: 0
SPC-3 PR Registrations:
   iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000001234abcd PRgen: 0x00000000
   iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000004567ffff PRgen: 0x00000001
SPC-3 Reservation Type: Write Exclusive Access, Registrants Only

T10 WWN numbers

The iblock_0/lvm_test0/wwn configFS attributes are used to see world wide unique LUN information, be it real from the SCSI device, or emulated using a admin present value into iblock_0/lvm_test0/wwn/evp_unit_serial.

cat /sys/kernel/config/target/core/iblock_0/lvm_test0/wwn/*   
T10 VPD Identifier Association: addressed logical unit
T10 VPD Identifier Type: NAA
T10 VPD Binary Device Identifier: 36001405a97e4ce21c0711de829b000c2
T10 VPD Identifier Association: addressed logical unit
T10 VPD Identifier Type: T10 Vendor ID based
T10 VPD ASCII Device Identifier: LIO-ORG
T10 VPD Unit Serial Number: a97e4ce21c0711de829b000c2943d57b

This is required in order to use the NAA VPD device identifers.

Initiator

sg_persist

Here's a reference output on the initiator side (via PRIN REPORT_CAPABILITES):

initiator# sg_persist --in --report-capabilities -v /dev/sde
    inquiry cdb: 12 00 00 00 24 00 
  LIO-ORG  IBLOCK  3.0
  Peripheral device type: disk
    Persistent Reservation In cmd: 5e 02 00 00 00 00 00 20 00 00 
Report capabilities response:
  Compatible Reservation Handling(CRH): 0
  Specify Initiator Ports Capable(SIP_C): 0
  All Target Ports Capable(ATP_C): 1
  Persist Through Power Loss Capable(PTPL_C): 0
  Type Mask Valid(TMV): 1
  Allow commands: 1
  Persist Through Power Loss Active(PTPL_A): 0
    Support indicated in Type mask:
      Write Exclusive, all registrants: 1
      Exclusive Access, registrants only: 1
      Write Exclusive, registrants only: 1
      Exclusive Access: 1
      Write Exclusive: 1
      Exclusive Access, all registrants: 1

sg3_utils

Linux support for client side PRs using sg3_utils and RHEL cluster scripts. The fence_scsi_test was originally written for RHEL clusters, and is available on a number of other Linux distributions including SuSE, Debian, Ubuntu, etc.

initiator# lsscsi
[0:0:0:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sdb
[0:0:1:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sdc
[0:0:2:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sdd
[1:0:0:0]    disk    ATA      VMware Virtual I 0000  /dev/sda
[5:0:0:0]    disk    LIO-ORG  IBLOCK           3.0   /dev/sde
[6:0:0:0]    disk    LIO-ORG  IBLOCK           3.0   /dev/sdf
[6:0:0:1]    disk    VMware,  VMware Virtual S 1.0   /dev/sdg
[6:0:0:2]    disk    LIO-ORG  RAMDISK-MCP      3.0   /dev/sdh
[6:0:0:3]    disk    LIO-ORG  FILEIO           3.0   /dev/sdi

initiator# lsscsi --transport
[0:0:0:0]    disk                                    /dev/sdb
[0:0:1:0]    disk                                    /dev/sdc
[0:0:2:0]    disk                                    /dev/sdd
[1:0:0:0]    disk                                    /dev/sda
[5:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,t,0x1  /dev/sde
[6:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdf
[6:0:0:1]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdg
[6:0:0:2]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdh
[6:0:0:3]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdi

fence_scsi_test

The output of the client side fence_scsi_test looks like using Open-iSCSI Initiators connected to LIO. Note that the SPC-3 capable devices pass the simple PROUT REGISTER/UNREGISTER operations.

initiator# fence_scsi_test -s
Testing all SCSI block devices...
Attempted to register with devices:
-------------------------------------
        /dev/sda Failure
        /dev/sdb Failure
        /dev/sdc Failure
        /dev/sdd Failure
        /dev/sde Success
        /dev/sdf Success
        /dev/sdg Failure
        /dev/sdh Success
        /dev/sdi Success
-------------------------------------
Number of devices tested: 9
Number of devices passed: 4
Number of devices failed: 5

SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003
PSCSI Status Byte exception at task: d4771100 CDB: 0x5f Result: 0x08000002
HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense
PSCSI Status Byte exception at task: d4771200 CDB: 0x5f Result: 0x08000002
HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001

See also

• LIO
• SCSI: Asymmetric Logical Unit Assignment (ALUA), Multiple Connections per Session (MC/S), Error Recovery Level (ERL), vStorage APIs for Array Integration (VAAI)
• Fabric modules: FCoE, Fibre Channel, iSCSI, iSER, SRP and vHost
• Management: targetcli