Persistent Reservations

The Linux SCSI Target Wiki

(Difference between revisions)
Jump to: navigation, search
m (Unprotected "Persistent Reservations")
m
 
(23 intermediate revisions not shown)
Line 1: Line 1:
-
SCSI-3 and SCSI-4 '''Persistent Reservations''' ('''PRs''') are a feature that is used for a number of cluster storage configurations for I/O fencing and failover/retakeover.
+
SCSI-3 and SCSI-4 '''Persistent Reservations''' ('''PRs''', [http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=drivers/target/target_core_pr.c;hb=HEAD target_core_pr.c]) are a feature that is used for a number of cluster storage configurations for I/O fencing and failover/retakeover. PRs are a SCSI protocol feature, so they have been implemented in the {{Target}} engine, and hence can be used for all SCSI transport fabrics, such as [[FCoE]], [[Fibre Channel]], [[iSCSI]], [[iSER]], [[SRP]], [http://en.wikipedia.org/wiki/SAS Serial Attached SCSI] (SAS), etc. {{T}} also fully supports using PRs with [[Asymmetric Logical Unit Assignment]] (ALUA).
-
 
+
-
Persistent Reservations are a SCSI protocol feature, so they have been implemented in the generic [[Target]] engine, and hence can be used for all SCSI transport fabrics, such as [[iSCSI]], [[Fibre Channel]] (FC), [[Fibre Channel over Ethernet|FCoE]], [http://en.wikipedia.org/wiki/SAS Serial Attached SCSI] (SAS), etc.
+
-
 
+
-
Using PRs with [[ALUA]] is also fully supported with LIO v3.1.
+
== Standards compliance ==
== Standards compliance ==
-
The [[SPC-3]]/[[SPC-4]] features implemented are described below. For legacy environments, compatible reservation handling for [[SPC-2]] needs to be enabled.
+
The [[SPC-3]]/[[SPC-4]] features implemented (in [http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=blob;f=drivers/target/target_core_pr.c;hb=HEAD target_core_pr.c]) are described below. For legacy environments, compatible reservation handling for [[SPC-2]] needs to be enabled.
=== SPC-4 ===
=== SPC-4 ===
-
Implementing full Persistent Reservation functionality entails two [[SPC-4]] defined features:
+
{{T}} implements full Persistent Reservation functionality, which entails two [[SPC-4]] defined features:
* <code>APTPL=1</code> (''Activate Persist across Target Power Loss'') is using <code>key=value</code> <code>/var/target/pr/aptpl_$T10_UNIT_SERIAL</code> using <code>sg_persist</code> ([http://groups.google.com/group/linux-iscsi-target-dev/browse_thread/thread/9f4ed61379033ec8 announcement]).
* <code>APTPL=1</code> (''Activate Persist across Target Power Loss'') is using <code>key=value</code> <code>/var/target/pr/aptpl_$T10_UNIT_SERIAL</code> using <code>sg_persist</code> ([http://groups.google.com/group/linux-iscsi-target-dev/browse_thread/thread/9f4ed61379033ec8 announcement]).
Line 27: Line 23:
All PRIN Service Actions have been implemented: <code>READ_KEYS</code>, <code>READ_RESERVATION</code>, <code>REPORT_CAPABILITIES</code>, <code>READ_FULL_STATUS</code>
All PRIN Service Actions have been implemented: <code>READ_KEYS</code>, <code>READ_RESERVATION</code>, <code>REPORT_CAPABILITIES</code>, <code>READ_FULL_STATUS</code>
-
 
-
== Implementation ==
 
-
 
-
[[SPC-3]] and [[SPC-4]] PRs are supported in a SCSI fabric module independent method in [[Target]] with the following patch sets:
 
-
 
-
* 3/14/2010 ([http://groups.google.com/group/linux-iscsi-target-dev/browse_thread/thread/814e428cc257d2fc patch]): SPC-3 compliance for multi-fabric and inter-fabric operation.
 
-
* 8/10/2010 ([http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=commitdiff;h=27a87ce47401f5bc8123f0fed9de877d1b991914 patch], [http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=commitdiff;h=33818533e835b3dbaddb45acfc90c7406fffbcee patch]): full SPC-4 compliance.
 
== Test and Verification ==
== Test and Verification ==
-
[[Image:LIO-MSFCvalidation.jpg|right|thumb|300px|LIO v3.0 on v2.6.29 connected to Windows Cluster 2008 using iSCSI.]]
+
[[Image:LIO_MSFC-Validation.jpg|right|thumb|300px|{{T}} on Linux 2.6.29 connected to Windows Cluster 2008 using iSCSI.]]
-
The LIO&nbsp;3 [[IBLOCK]], [[FILEIO]] and [[RAMDISK]] subsystem plugins are passing [[SPC-3]] level validation for iSCSI target PRs. The following cluster clients that use Linux-iSCSI target mode PR have been tested.  
+
{{T}} [[IBLOCK]], [[FILEIO]] and [[RAMDISK]] subsystem plugins are passing [[SPC-3]] level validation for iSCSI target PRs. The following cluster clients that use Linux-iSCSI target mode PR have been tested.  
=== RHEL/CentOS &ge;5.3 ===
=== RHEL/CentOS &ge;5.3 ===
-
Test and validation for RHEL/CentOS was performed with [[GFS]] (e.g., with multiple writers). All LIO&nbsp;3 [[Target]]+[[iSCSI]] capable storage objects from [[IBLOCK]], [[FILEIO]], and [[RAMDISK]] pass the fence_scsi_test.
+
Test and validation for RHEL/CentOS was performed with GFS (e.g., with multiple writers). {{T}} capable storage objects from [[IBLOCK]], [[FILEIO]], and [[RAMDISK]] pass the fence_scsi_test.
RHEL/CentOS uses SCSI I/O Fencing via the sg_persist CLI operation in sg3_util for ''Write Exclusive'', ''Registrants Only'' and <code>PREEMPT_AND_ABORT</code> takeover using ext3 mounts.
RHEL/CentOS uses SCSI I/O Fencing via the sg_persist CLI operation in sg3_util for ''Write Exclusive'', ''Registrants Only'' and <code>PREEMPT_AND_ABORT</code> takeover using ext3 mounts.
-
[[SPC-3]] PR emulation is disabled on top of the passthrough SCSI subsystem plugin (this is WIP for existing SCSI devices).
+
[[SPC-3]] PR emulation is disabled on top of the pass-through SCSI subsystem plugin (this is WIP for existing SCSI devices).
=== Microsoft Server 2008/R2 Failover Cluster ===
=== Microsoft Server 2008/R2 Failover Cluster ===
-
As of April 2009, LIO [[Target]]+[[iSCSI]] is passing validation using the MS Verification test suite. Test and validation for Microsoft Server 2008 Failover Cluster was performed using the MSFT domain validation suite.
+
As of April 2009, {{T}} is passing validation using the MS Verification test suite. Test and validation for Microsoft Server 2008 Failover Cluster was performed using the MS domain validation suite.
Microsoft Server 2008 Failover Cluster uses ''Write Exclusive'', ''Registrants  Only'' and <code>PROUT PREEMPT</code> for production failure cases.
Microsoft Server 2008 Failover Cluster uses ''Write Exclusive'', ''Registrants  Only'' and <code>PROUT PREEMPT</code> for production failure cases.
-
Extensive work by the LIO developer and user community has made this possible. If you are interested in further testing and want to deploy production services on our code please let us know.
+
Extensive work by the {{T}} developer and user community has made this possible. If you are interested in further testing and want to deploy production services on our code please let us know.
=== Other ===
=== Other ===
Line 68: Line 57:
<pre>
<pre>
-
  tree /sys/kernel/config/target/core
+
tree /sys/kernel/config/target/core
-
  |-- iblock_0
+
|-- iblock_0
-
  |  |-- hba_info
+
|  |-- hba_info
-
  |  `-- lvm_test0
+
|  `-- lvm_test0
-
  |      |-- alua_lu_gp
+
|      |-- alua_lu_gp
-
  |      |-- attrib
+
|      |-- attrib
-
  |      |  |-- block_size
+
|      |  |-- block_size
-
  |      |  |-- emulate_tas
+
|      |  |-- emulate_tas
-
  |      |  |-- emulate_ua_intlck_ctrl
+
|      |  |-- emulate_ua_intlck_ctrl
-
  |      |  |-- hw_block_size
+
|      |  |-- hw_block_size
-
  |      |  |-- hw_max_sectors
+
|      |  |-- hw_max_sectors
-
  |      |  |-- hw_queue_depth
+
|      |  |-- hw_queue_depth
-
  |      |  |-- max_sectors
+
|      |  |-- max_sectors
-
  |      |  |-- queue_depth
+
|      |  |-- queue_depth
-
  |      |  `-- task_timeout
+
|      |  `-- task_timeout
-
  |      |-- control
+
|      |-- control
-
  |      |-- enable
+
|      |-- enable
-
  |      |-- fd
+
|      |-- fd
-
  |      |-- info
+
|      |-- info
-
  |      |-- pr
+
|      |-- pr
-
  |      |  |-- res_holder
+
|      |  |-- res_holder
-
  |      |  |-- res_pr_all_tgt_pts
+
|      |  |-- res_pr_all_tgt_pts
-
  |      |  |-- res_pr_generation
+
|      |  |-- res_pr_generation
-
  |      |  |-- res_pr_holder_tg_port
+
|      |  |-- res_pr_holder_tg_port
-
  |      |  |-- res_pr_registered_i_pts
+
|      |  |-- res_pr_registered_i_pts
-
  |      |  |-- res_pr_type
+
|      |  |-- res_pr_type
-
  |      |  `-- res_type
+
|      |  `-- res_type
-
  |      `-- wwn
+
|      `-- wwn
-
  |          |-- vpd_assoc_logical_unit
+
|          |-- vpd_assoc_logical_unit
-
  |          |-- vpd_assoc_scsi_target_device
+
|          |-- vpd_assoc_scsi_target_device
-
  |          |-- vpd_assoc_target_port
+
|          |-- vpd_assoc_target_port
-
  |          |-- vpd_protocol_identifier
+
|          |-- vpd_protocol_identifier
-
  |          `-- vpd_unit_serial
+
|          `-- vpd_unit_serial
</pre>
</pre>
=== SPC-3 Reservations ===
=== SPC-3 Reservations ===
-
The <code>iblock_0/lvm_test0/pr/</code> configFS attributes are used to see the current reservation/registration status for different SCSI (including LIO 3) I_T nexuses:
+
The <code>iblock_0/lvm_test0/pr/</code> configFS attributes are used to see the current reservation/registration status for different SCSI I_T nexuses:
<pre>
<pre>
-
  cat /sys/kernel/config/target/core/iblock_0/lvm_test0/pr/res_*
+
cat /sys/kernel/config/target/core/iblock_0/lvm_test0/pr/res_*
-
  SPC-3 Reservation: iSCSI Initiator: iqn.1993-08.org.debian:01:2dadf92d0ef
+
SPC-3 Reservation: iSCSI Initiator: iqn.1993-08.org.debian:01:2dadf92d0ef
-
  SPC-3 Reservation: All Target Ports registration 0x00000002
+
SPC-3 Reservation: All {{T}} ports registration 0x00000002
-
  SPC-3 Reservation: iSCSI Target Node Endpoint: iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd
+
SPC-3 Reservation: iSCSI target node Endpoint: iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd
-
  SPC-3 Reservation: Relative Port Identifer Tag: 2 iSCSI Portal Group Tag: 1 iSCSI Logical Unit: 0
+
SPC-3 Reservation: Relative Port Identifer Tag: 2 iSCSI Portal Group Tag: 1 iSCSI Logical Unit: 0
-
  SPC-3 PR Registrations:
+
SPC-3 PR Registrations:
-
    iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000001234abcd PRgen: 0x00000000
+
  iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000001234abcd PRgen: 0x00000000
-
    iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000004567ffff PRgen: 0x00000001
+
  iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000004567ffff PRgen: 0x00000001
-
  SPC-3 Reservation Type: Write Exclusive Access, Registrants Only
+
SPC-3 Reservation Type: Write Exclusive Access, Registrants Only
</pre>
</pre>
-
=== T10 WWU numbers ===
+
=== T10 WWN numbers ===
The <code>iblock_0/lvm_test0/wwn</code> configFS attributes are used to see world wide unique LUN information, be it real from the SCSI device, or emulated using a admin present value into <code>iblock_0/lvm_test0/wwn/evp_unit_serial</code>.  
The <code>iblock_0/lvm_test0/wwn</code> configFS attributes are used to see world wide unique LUN information, be it real from the SCSI device, or emulated using a admin present value into <code>iblock_0/lvm_test0/wwn/evp_unit_serial</code>.  
<pre>
<pre>
-
  cat /sys/kernel/config/target/core/iblock_0/lvm_test0/wwn/*   
+
cat /sys/kernel/config/target/core/iblock_0/lvm_test0/wwn/*   
-
  T10 VPD Identifier Association: addressed logical unit
+
T10 VPD Identifier Association: addressed logical unit
-
  T10 VPD Identifier Type: NAA
+
T10 VPD Identifier Type: NAA
-
  T10 VPD Binary Device Identifier: 36001405a97e4ce21c0711de829b000c2
+
T10 VPD Binary Device Identifier: 36001405a97e4ce21c0711de829b000c2
-
  T10 VPD Identifier Association: addressed logical unit
+
T10 VPD Identifier Association: addressed logical unit
-
  T10 VPD Identifier Type: T10 Vendor ID based
+
T10 VPD Identifier Type: T10 Vendor ID based
-
  T10 VPD ASCII Device Identifier: LIO-ORG
+
T10 VPD ASCII Device Identifier: LIO-ORG
-
  T10 VPD Unit Serial Number: a97e4ce21c0711de829b000c2943d57b
+
T10 VPD Unit Serial Number: a97e4ce21c0711de829b000c2943d57b
</pre>
</pre>
Line 143: Line 132:
<pre>
<pre>
-
  initiator# sg_persist --in --report-capabilities -v /dev/sde
+
initiator# sg_persist --in --report-capabilities -v /dev/sde
-
      inquiry cdb: 12 00 00 00 24 00  
+
    inquiry cdb: 12 00 00 00 24 00  
-
    LIO-ORG  IBLOCK  3.0
+
  LIO-ORG  IBLOCK  3.0
-
    Peripheral device type: disk
+
  Peripheral device type: disk
-
      Persistent Reservation In cmd: 5e 02 00 00 00 00 00 20 00 00  
+
    Persistent Reservation In cmd: 5e 02 00 00 00 00 00 20 00 00  
-
  Report capabilities response:
+
Report capabilities response:
-
    Compatible Reservation Handling(CRH): 0
+
  Compatible Reservation Handling(CRH): 0
-
    Specify Initiator Ports Capable(SIP_C): 0
+
  Specify Initiator Ports Capable(SIP_C): 0
-
    All Target Ports Capable(ATP_C): 1
+
  All Target Ports Capable(ATP_C): 1
-
    Persist Through Power Loss Capable(PTPL_C): 0
+
  Persist Through Power Loss Capable(PTPL_C): 0
-
    Type Mask Valid(TMV): 1
+
  Type Mask Valid(TMV): 1
-
    Allow commands: 1
+
  Allow commands: 1
-
    Persist Through Power Loss Active(PTPL_A): 0
+
  Persist Through Power Loss Active(PTPL_A): 0
-
      Support indicated in Type mask:
+
    Support indicated in Type mask:
-
        Write Exclusive, all registrants: 1
+
      Write Exclusive, all registrants: 1
-
        Exclusive Access, registrants only: 1
+
      Exclusive Access, registrants only: 1
-
        Write Exclusive, registrants only: 1
+
      Write Exclusive, registrants only: 1
-
        Exclusive Access: 1
+
      Exclusive Access: 1
-
        Write Exclusive: 1
+
      Write Exclusive: 1
-
        Exclusive Access, all registrants: 1
+
      Exclusive Access, all registrants: 1
</pre>
</pre>
Line 170: Line 159:
<pre>
<pre>
-
  initiator# lsscsi
+
initiator# lsscsi
-
  [0:0:0:0]    disk    VMware,  VMware Virtual S 1.0  /dev/sdb
+
[0:0:0:0]    disk    VMware,  VMware Virtual S 1.0  /dev/sdb
-
  [0:0:1:0]    disk    VMware,  VMware Virtual S 1.0  /dev/sdc
+
[0:0:1:0]    disk    VMware,  VMware Virtual S 1.0  /dev/sdc
-
  [0:0:2:0]    disk    VMware,  VMware Virtual S 1.0  /dev/sdd
+
[0:0:2:0]    disk    VMware,  VMware Virtual S 1.0  /dev/sdd
-
  [1:0:0:0]    disk    ATA      VMware Virtual I 0000  /dev/sda
+
[1:0:0:0]    disk    ATA      VMware Virtual I 0000  /dev/sda
-
  [5:0:0:0]    disk    LIO-ORG  IBLOCK          3.0  /dev/sde
+
[5:0:0:0]    disk    LIO-ORG  IBLOCK          3.0  /dev/sde
-
  [6:0:0:0]    disk    LIO-ORG  IBLOCK          3.0  /dev/sdf
+
[6:0:0:0]    disk    LIO-ORG  IBLOCK          3.0  /dev/sdf
-
  [6:0:0:1]    disk    VMware,  VMware Virtual S 1.0  /dev/sdg
+
[6:0:0:1]    disk    VMware,  VMware Virtual S 1.0  /dev/sdg
-
  [6:0:0:2]    disk    LIO-ORG  RAMDISK-MCP      3.0  /dev/sdh
+
[6:0:0:2]    disk    LIO-ORG  RAMDISK-MCP      3.0  /dev/sdh
-
  [6:0:0:3]    disk    LIO-ORG  FILEIO          3.0  /dev/sdi
+
[6:0:0:3]    disk    LIO-ORG  FILEIO          3.0  /dev/sdi
-
  initiator# lsscsi --transport
+
 
-
  [0:0:0:0]    disk                                    /dev/sdb
+
initiator# lsscsi --transport
-
  [0:0:1:0]    disk                                    /dev/sdc
+
[0:0:0:0]    disk                                    /dev/sdb
-
  [0:0:2:0]    disk                                    /dev/sdd
+
[0:0:1:0]    disk                                    /dev/sdc
-
  [1:0:0:0]    disk                                    /dev/sda
+
[0:0:2:0]    disk                                    /dev/sdd
-
  [5:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,t,0x1  /dev/sde
+
[1:0:0:0]    disk                                    /dev/sda
-
  [6:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdf
+
[5:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,t,0x1  /dev/sde
-
  [6:0:0:1]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdg
+
[6:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdf
-
  [6:0:0:2]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdh
+
[6:0:0:1]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdg
-
  [6:0:0:3]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdi
+
[6:0:0:2]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdh
 +
[6:0:0:3]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdi
</pre>
</pre>
=== fence_scsi_test ===
=== fence_scsi_test ===
-
The output of the client side <code>fence_scsi_test</code> looks like using [[Open-iSCSI]] Initiators connected to LIO 3.  Note that the SPC-3 capable devices pass the simple PROUT REGISTER/UNREGISTER operations.
+
The output of the client side <code>fence_scsi_test</code> looks like using [[Open-iSCSI]] Initiators connected to LIO.  Note that the SPC-3 capable devices pass the simple PROUT REGISTER/UNREGISTER operations.
<pre>
<pre>
-
  initiator# fence_scsi_test -s
+
initiator# fence_scsi_test -s
-
  Testing all SCSI block devices...
+
Testing all SCSI block devices...
-
  Attempted to register with devices:
+
Attempted to register with devices:
-
  -------------------------------------
+
-------------------------------------
-
          /dev/sda Failure
+
        /dev/sda Failure
-
          /dev/sdb Failure
+
        /dev/sdb Failure
-
          /dev/sdc Failure
+
        /dev/sdc Failure
-
          /dev/sdd Failure
+
        /dev/sdd Failure
-
          /dev/sde Success
+
        /dev/sde Success
-
          /dev/sdf Success
+
        /dev/sdf Success
-
          /dev/sdg Failure
+
        /dev/sdg Failure
-
          /dev/sdh Success
+
        /dev/sdh Success
-
          /dev/sdi Success
+
        /dev/sdi Success
-
  -------------------------------------
+
-------------------------------------
-
  Number of devices tested: 9
+
Number of devices tested: 9
-
  Number of devices passed: 4
+
Number of devices passed: 4
-
  Number of devices failed: 5
+
Number of devices failed: 5
</pre>
</pre>
<pre>
<pre>
-
  SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
+
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
-
  SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
+
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
-
  SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002
+
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002
-
  SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
+
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
-
  SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
+
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
-
  SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002
+
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002
-
  SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
+
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
-
  SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
+
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
-
  SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003
+
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003
-
  SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
+
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
-
  SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
+
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
-
  SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003
+
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003
-
  PSCSI Status Byte exception at task: d4771100 CDB: 0x5f Result: 0x08000002
+
PSCSI Status Byte exception at task: d4771100 CDB: 0x5f Result: 0x08000002
-
  HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense
+
HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense
-
  PSCSI Status Byte exception at task: d4771200 CDB: 0x5f Result: 0x08000002
+
PSCSI Status Byte exception at task: d4771200 CDB: 0x5f Result: 0x08000002
-
  HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense
+
HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense
-
  SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
+
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
-
  SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s)
+
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s)
-
  SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
+
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
-
  SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
+
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
-
  SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s)
+
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s)
-
  SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
+
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
-
  SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
+
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
-
  SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s)
+
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s)
-
  SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
+
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
-
  SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
+
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
-
  SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s)
+
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s)
-
  SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
+
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
</pre>
</pre>
== See also ==
== See also ==
-
* [[Target]]
+
* {{Target}}
-
* [[SCSI]]: [[ALUA]], [[Multiple Connections per Session|MC/S]], [[Error Recovery Level|ERL]]
+
* [[SCSI]]: [[Asymmetric Logical Unit Assignment]] (ALUA), [[Multiple Connections per Session]] (MC/S), [[Error Recovery Level]] (ERL), [[vStorage APIs for Array Integration]] (VAAI)
-
* Fabric modules: [[iSCSI]], [[Fibre Channel over Ethernet|FCoE]], [[Fibre Channel]] and [[InfiniBand]]
+
* Fabric modules: [[FCoE]], [[Fibre Channel]], [[iSCSI]], [[iSER]], [[SRP]] and [[vHost]]
-
* Management: [[RTS Director]], [[RTSadmin]], [[lio-utils]]
+
* Management: [[targetcli]]
-
{{AdSense See also}}
+
[[Category:Target]]
[[Category:Target]]

Latest revision as of 18:23, 29 September 2013

SCSI-3 and SCSI-4 Persistent Reservations (PRs, target_core_pr.c) are a feature that is used for a number of cluster storage configurations for I/O fencing and failover/retakeover. PRs are a SCSI protocol feature, so they have been implemented in the LinuxIO engine, and hence can be used for all SCSI transport fabrics, such as FCoE, Fibre Channel, iSCSI, iSER, SRP, Serial Attached SCSI (SAS), etc. LIO also fully supports using PRs with Asymmetric Logical Unit Assignment (ALUA).

Contents

Standards compliance

The SPC-3/SPC-4 features implemented (in target_core_pr.c) are described below. For legacy environments, compatible reservation handling for SPC-2 needs to be enabled.

SPC-4

LIO implements full Persistent Reservation functionality, which entails two SPC-4 defined features:

PROUT Service Actions

All PROUT Service Actions have been implemented: REGISTER, RESERVE, RELEASE, CLEAR, REGISTER_AND_IGNORE, PREEMPT, PREEMPT_AND_ABORT and REGISTER_AND_MOVE

PROUT Reservation Types

All PROUT Reservation Types have been implemented: Write Exclusive, Exclusive Access, Write Exclusive Registrants Only, Exclusive Access Registrants Only Write Exclusive All Registrants, Exclusive Access All Registrants

PRIN Service Actions

All PRIN Service Actions have been implemented: READ_KEYS, READ_RESERVATION, REPORT_CAPABILITIES, READ_FULL_STATUS

Test and Verification

LIO on Linux 2.6.29 connected to Windows Cluster 2008 using iSCSI.

LIO IBLOCK, FILEIO and RAMDISK subsystem plugins are passing SPC-3 level validation for iSCSI target PRs. The following cluster clients that use Linux-iSCSI target mode PR have been tested.

RHEL/CentOS ≥5.3

Test and validation for RHEL/CentOS was performed with GFS (e.g., with multiple writers). LIO capable storage objects from IBLOCK, FILEIO, and RAMDISK pass the fence_scsi_test.

RHEL/CentOS uses SCSI I/O Fencing via the sg_persist CLI operation in sg3_util for Write Exclusive, Registrants Only and PREEMPT_AND_ABORT takeover using ext3 mounts.

SPC-3 PR emulation is disabled on top of the pass-through SCSI subsystem plugin (this is WIP for existing SCSI devices).

Microsoft Server 2008/R2 Failover Cluster

As of April 2009, LIO is passing validation using the MS Verification test suite. Test and validation for Microsoft Server 2008 Failover Cluster was performed using the MS domain validation suite.

Microsoft Server 2008 Failover Cluster uses Write Exclusive, Registrants Only and PROUT PREEMPT for production failure cases.

Extensive work by the LIO developer and user community has made this possible. If you are interested in further testing and want to deploy production services on our code please let us know.

Other

If you have an iSCSI Initiator that implements SPC-3 PR logic and would like to test it against LIO 3, please don't hesitate to contact us on our development mailing list.

ConfigFS object tree

Target

Target/configFS layout. This is the output of an IBLOCK exported Linux/LVM DM block_device (lvm_test0) via tree:

tree /sys/kernel/config/target/core
|-- iblock_0
|   |-- hba_info
|   `-- lvm_test0
|       |-- alua_lu_gp
|       |-- attrib
|       |   |-- block_size
|       |   |-- emulate_tas
|       |   |-- emulate_ua_intlck_ctrl
|       |   |-- hw_block_size
|       |   |-- hw_max_sectors
|       |   |-- hw_queue_depth
|       |   |-- max_sectors
|       |   |-- queue_depth
|       |   `-- task_timeout
|       |-- control
|       |-- enable
|       |-- fd
|       |-- info
|       |-- pr
|       |   |-- res_holder
|       |   |-- res_pr_all_tgt_pts
|       |   |-- res_pr_generation
|       |   |-- res_pr_holder_tg_port
|       |   |-- res_pr_registered_i_pts
|       |   |-- res_pr_type
|       |   `-- res_type
|       `-- wwn
|           |-- vpd_assoc_logical_unit
|           |-- vpd_assoc_scsi_target_device
|           |-- vpd_assoc_target_port
|           |-- vpd_protocol_identifier
|           `-- vpd_unit_serial

SPC-3 Reservations

The iblock_0/lvm_test0/pr/ configFS attributes are used to see the current reservation/registration status for different SCSI I_T nexuses:

cat /sys/kernel/config/target/core/iblock_0/lvm_test0/pr/res_*
SPC-3 Reservation: iSCSI Initiator: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 Reservation: All {{T}} ports registration 0x00000002
SPC-3 Reservation: iSCSI target node Endpoint: iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd
SPC-3 Reservation: Relative Port Identifer Tag: 2 iSCSI Portal Group Tag: 1 iSCSI Logical Unit: 0
SPC-3 PR Registrations:
   iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000001234abcd PRgen: 0x00000000
   iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000004567ffff PRgen: 0x00000001
SPC-3 Reservation Type: Write Exclusive Access, Registrants Only

T10 WWN numbers

The iblock_0/lvm_test0/wwn configFS attributes are used to see world wide unique LUN information, be it real from the SCSI device, or emulated using a admin present value into iblock_0/lvm_test0/wwn/evp_unit_serial.

cat /sys/kernel/config/target/core/iblock_0/lvm_test0/wwn/*   
T10 VPD Identifier Association: addressed logical unit
T10 VPD Identifier Type: NAA
T10 VPD Binary Device Identifier: 36001405a97e4ce21c0711de829b000c2
T10 VPD Identifier Association: addressed logical unit
T10 VPD Identifier Type: T10 Vendor ID based
T10 VPD ASCII Device Identifier: LIO-ORG
T10 VPD Unit Serial Number: a97e4ce21c0711de829b000c2943d57b

This is required in order to use the NAA VPD device identifers.

Initiator

sg_persist

Here's a reference output on the initiator side (via PRIN REPORT_CAPABILITES):

initiator# sg_persist --in --report-capabilities -v /dev/sde
    inquiry cdb: 12 00 00 00 24 00 
  LIO-ORG  IBLOCK  3.0
  Peripheral device type: disk
    Persistent Reservation In cmd: 5e 02 00 00 00 00 00 20 00 00 
Report capabilities response:
  Compatible Reservation Handling(CRH): 0
  Specify Initiator Ports Capable(SIP_C): 0
  All Target Ports Capable(ATP_C): 1
  Persist Through Power Loss Capable(PTPL_C): 0
  Type Mask Valid(TMV): 1
  Allow commands: 1
  Persist Through Power Loss Active(PTPL_A): 0
    Support indicated in Type mask:
      Write Exclusive, all registrants: 1
      Exclusive Access, registrants only: 1
      Write Exclusive, registrants only: 1
      Exclusive Access: 1
      Write Exclusive: 1
      Exclusive Access, all registrants: 1

sg3_utils

Linux support for client side PRs using sg3_utils and RHEL cluster scripts. The fence_scsi_test was originally written for RHEL clusters, and is available on a number of other Linux distributions including SuSE, Debian, Ubuntu, etc.

initiator# lsscsi
[0:0:0:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sdb
[0:0:1:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sdc
[0:0:2:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sdd
[1:0:0:0]    disk    ATA      VMware Virtual I 0000  /dev/sda
[5:0:0:0]    disk    LIO-ORG  IBLOCK           3.0   /dev/sde
[6:0:0:0]    disk    LIO-ORG  IBLOCK           3.0   /dev/sdf
[6:0:0:1]    disk    VMware,  VMware Virtual S 1.0   /dev/sdg
[6:0:0:2]    disk    LIO-ORG  RAMDISK-MCP      3.0   /dev/sdh
[6:0:0:3]    disk    LIO-ORG  FILEIO           3.0   /dev/sdi

initiator# lsscsi --transport
[0:0:0:0]    disk                                    /dev/sdb
[0:0:1:0]    disk                                    /dev/sdc
[0:0:2:0]    disk                                    /dev/sdd
[1:0:0:0]    disk                                    /dev/sda
[5:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,t,0x1  /dev/sde
[6:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdf
[6:0:0:1]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdg
[6:0:0:2]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdh
[6:0:0:3]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdi

fence_scsi_test

The output of the client side fence_scsi_test looks like using Open-iSCSI Initiators connected to LIO. Note that the SPC-3 capable devices pass the simple PROUT REGISTER/UNREGISTER operations.

initiator# fence_scsi_test -s
Testing all SCSI block devices...
Attempted to register with devices:
-------------------------------------
        /dev/sda Failure
        /dev/sdb Failure
        /dev/sdc Failure
        /dev/sdd Failure
        /dev/sde Success
        /dev/sdf Success
        /dev/sdg Failure
        /dev/sdh Success
        /dev/sdi Success
-------------------------------------
Number of devices tested: 9
Number of devices passed: 4
Number of devices failed: 5
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003
PSCSI Status Byte exception at task: d4771100 CDB: 0x5f Result: 0x08000002
HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense
PSCSI Status Byte exception at task: d4771200 CDB: 0x5f Result: 0x08000002
HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s)
SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001

See also

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Google AdSense