Persistent Reservations

The Linux SCSI Target Wiki

(Difference between revisions)
Jump to: navigation, search
m (SPC-4)
m (SPC-4)
Line 15: Line 15:
All [[SPC-4]] defined features, which specifically entails two of them:
All [[SPC-4]] defined features, which specifically entails two of them:
-
* <code>APTPL=1</code> (''Activate Persist across Target Power Loss'') is using <code>key=value</code> <code>/var/target/pr/aptpl_$T10_UNIT_SERIAL</code> using <code>sg_persist</code> ([http://groups.google.com/group/linux-iscsi-target-dev/browse_thread/thread/9f4ed61379033ec8 more information]).
+
* <code>APTPL=1</code> (''Activate Persist across Target Power Loss'') is using <code>key=value</code> <code>/var/target/pr/aptpl_$T10_UNIT_SERIAL</code> using <code>sg_persist</code> ([http://groups.google.com/group/linux-iscsi-target-dev/browse_thread/thread/9f4ed61379033ec8 announcement]).
-
* <code>SP_I_PT=1</code> (''Specify Initiator Ports Capable'') bit is supported and tested using <code>sg_persist</code> ([http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=commitdiff;h=de4490f8b9f10a7048519527439b147c8d7b704f;hp=f38d010f717b37ae765f0544ca411616773605fc more information]).
+
* <code>SP_I_PT=1</code> (''Specify Initiator Ports Capable'') bit is supported and tested using <code>sg_persist</code> ([http://git.kernel.org/?p=linux/kernel/git/nab/lio-core-2.6.git;a=commitdiff;h=de4490f8b9f10a7048519527439b147c8d7b704f;hp=f38d010f717b37ae765f0544ca411616773605fc announcement]).
=== PROUT Service Actions ===
=== PROUT Service Actions ===

Revision as of 03:34, 27 November 2010

Persistent Reservations (PRs) are an SPC-3 and SPC-4 SCSI feature that is used for a number of cluster storage configurations for I/O fencing and failover/retakeover.

Contents

Overview

SCSI-3 and SCSI-4 Persistent Reservations (PRs) are used for a number of cluster storage configurations for I/O fencing and failover/retakeover. As they are a SCSI protocol features, they have been implemented in the generic Target engine, and can be used for all underlying fabrics, sich as iSCSI, FCoE, Fibre Channel, Serial Attached SCSI (SAS), etc.

Using PRs with ALUA is also fully supported with LIO v3.1.

Standards compliance

The SPC3/SPC-4 features implemented are described below. For legacy environments, compatible reservation handling for SPC-2 needs to be enabled.

SPC-4

All SPC-4 defined features, which specifically entails two of them:

PROUT Service Actions

All PROUT Service Actions have been implemented: REGISTER, RESERVE, RELEASE, CLEAR, REGISTER_AND_IGNORE, PREEMPT, PREEMPT_AND_ABORT and REGISTER_AND_MOVE

PROUT Reservation Types

All PROUT Reservation Types have been implemented: Write Exclusive, Exclusive Access, Write Exclusive Registrants Only, Exclusive Access Registrants Only Write Exclusive All Registrants, Exclusive Access All Registrants

PRIN Service Actions

All PRIN Service Actions have been implemented: READ_KEYS, READ_RESERVATION, REPORT_CAPABILITIES, READ_FULL_STATUS

Implementation

SPC-3 and SPC-4 PRs are supported in a SCSI fabric module independent method in Target.

The patch sets were as follows:

Test and Verification

File:LIO-MSFCvalidation.jpg
LIO v3.0 on v2.6.29 connected to Windows Cluster 2008 using iSCSI.

The LIO v3.0 Target/IBLOCK, Target/FILEIO and Target/RAMDISK subsystem plugins are passing SPC-3 level validation for iSCSI target PRs. The following cluster clients that use Linux-iSCSI target mode PR have been tested.

RHEL/CentOS ≥5.3

Test and validation for RHEL/CentOS was performed with GFS (e.g., with multiple writers). All LIO Target+iSCSI v3.0 capable storage objects from Target/IBLOCK, [[Target/FILEIO], and Target/RAMDISK pass the fence_scsi_test.

RHEL/CentOS uses SCSI I/O Fencing via the sg_persist CLI operation in sg3_util for Write Exclusive, Registrants Only and PREEMPT_AND_ABORT takeover using ext3 mounts.

SPC-3 PR emulation is disabled on top of the passthrough SCSI subsystem plugin (this is WIP for existing SCSI devices).

Microsoft Server 2008/R2 Failover Cluster

As of April 2009, LIO Target+iSCSI is passing validation using the MS Verification test suite. Test and validation for Microsoft Server 2008 Failover Cluster was performed using the MSFT domain validation suite.

Microsoft Server 2008 Failover Cluster uses Write Exclusive, Registrants Only and PROUT PREEMPT for production failure cases.

Extensive work by the LIO developer and user community has made this possible. If you are interested in further testing and want to deploy production services on our code please let us know.

Other

If you have an iSCSI Initiator that implements SPC-3 PR logic and would like to test it against LIO v3.0, please don't hesitate to contact us on our development mailing list.

ConfigFS object tree

Target

Target/configFS layout. This is the output of an Target/IBLOCK exported Linux/LVM DM block_device (lvm_test0) via tree:

   tree /sys/kernel/config/target/core
   |-- iblock_0
   |   |-- hba_info
   |   `-- lvm_test0
   |       |-- alua_lu_gp
   |       |-- attrib
   |       |   |-- block_size
   |       |   |-- emulate_tas
   |       |   |-- emulate_ua_intlck_ctrl
   |       |   |-- hw_block_size
   |       |   |-- hw_max_sectors
   |       |   |-- hw_queue_depth
   |       |   |-- max_sectors
   |       |   |-- queue_depth
   |       |   `-- task_timeout
   |       |-- control
   |       |-- enable
   |       |-- fd
   |       |-- info
   |       |-- pr
   |       |   |-- res_holder
   |       |   |-- res_pr_all_tgt_pts
   |       |   |-- res_pr_generation
   |       |   |-- res_pr_holder_tg_port
   |       |   |-- res_pr_registered_i_pts
   |       |   |-- res_pr_type
   |       |   `-- res_type
   |       `-- wwn
   |           |-- vpd_assoc_logical_unit
   |           |-- vpd_assoc_scsi_target_device
   |           |-- vpd_assoc_target_port
   |           |-- vpd_protocol_identifier
   |           `-- vpd_unit_serial

SPC-3 Reservations

The iblock_0/lvm_test0/pr/ configFS attributes are used to see the current reservation/registration status for different SCSI (including LIO v3.0) I_T nexuses:

   cat /sys/kernel/config/target/core/iblock_0/lvm_test0/pr/res_*
   SPC-3 Reservation: iSCSI Initiator: iqn.1993-08.org.debian:01:2dadf92d0ef
   SPC-3 Reservation: All Target Ports registration 0x00000002
   SPC-3 Reservation: iSCSI Target Node Endpoint: iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd
   SPC-3 Reservation: Relative Port Identifer Tag: 2 iSCSI Portal Group Tag: 1 iSCSI Logical Unit: 0
   SPC-3 PR Registrations:
      iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000001234abcd PRgen: 0x00000000
      iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000004567ffff PRgen: 0x00000001
   SPC-3 Reservation Type: Write Exclusive Access, Registrants Only

T10 WWU numbers

The iblock_0/lvm_test0/wwn configFS attributes are used to see world wide unique LUN information, be it real from the SCSI device, or emulated using a admin present value into iblock_0/lvm_test0/wwn/evp_unit_serial.

   cat /sys/kernel/config/target/core/iblock_0/lvm_test0/wwn/*   
   T10 VPD Identifier Association: addressed logical unit
   T10 VPD Identifier Type: NAA
   T10 VPD Binary Device Identifier: 36001405a97e4ce21c0711de829b000c2
   T10 VPD Identifier Association: addressed logical unit
   T10 VPD Identifier Type: T10 Vendor ID based
   T10 VPD ASCII Device Identifier: LIO-ORG
   T10 VPD Unit Serial Number: a97e4ce21c0711de829b000c2943d57b

This is required in order to use the NAA VPD device identifers.

Initiator

sg_persist

Here's a reference output on the initiator side (via PRIN REPORT_CAPABILITES):

   initiator# sg_persist --in --report-capabilities -v /dev/sde
       inquiry cdb: 12 00 00 00 24 00 
     LIO-ORG  IBLOCK  3.0
     Peripheral device type: disk
       Persistent Reservation In cmd: 5e 02 00 00 00 00 00 20 00 00 
   Report capabilities response:
     Compatible Reservation Handling(CRH): 0
     Specify Initiator Ports Capable(SIP_C): 0
     All Target Ports Capable(ATP_C): 1
     Persist Through Power Loss Capable(PTPL_C): 0
     Type Mask Valid(TMV): 1
     Allow commands: 1
     Persist Through Power Loss Active(PTPL_A): 0
       Support indicated in Type mask:
         Write Exclusive, all registrants: 1
         Exclusive Access, registrants only: 1
         Write Exclusive, registrants only: 1
         Exclusive Access: 1
         Write Exclusive: 1
         Exclusive Access, all registrants: 1

sg3_utils

Linux support for client side PRs using sg3_utils and RHEL cluster scripts. The fence_scsi_test was originally written for RHEL clusters, and is available on a number of other Linux distributions including SuSE, Debian, Ubuntu, etc.

   initiator# lsscsi
   [0:0:0:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sdb
   [0:0:1:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sdc
   [0:0:2:0]    disk    VMware,  VMware Virtual S 1.0   /dev/sdd
   [1:0:0:0]    disk    ATA      VMware Virtual I 0000  /dev/sda
   [5:0:0:0]    disk    LIO-ORG  IBLOCK           3.0   /dev/sde
   [6:0:0:0]    disk    LIO-ORG  IBLOCK           3.0   /dev/sdf
   [6:0:0:1]    disk    VMware,  VMware Virtual S 1.0   /dev/sdg
   [6:0:0:2]    disk    LIO-ORG  RAMDISK-MCP      3.0   /dev/sdh
   [6:0:0:3]    disk    LIO-ORG  FILEIO           3.0   /dev/sdi
   initiator# lsscsi --transport
   [0:0:0:0]    disk                                    /dev/sdb
   [0:0:1:0]    disk                                    /dev/sdc
   [0:0:2:0]    disk                                    /dev/sdd
   [1:0:0:0]    disk                                    /dev/sda
   [5:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,t,0x1  /dev/sde
   [6:0:0:0]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdf
   [6:0:0:1]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdg
   [6:0:0:2]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdh
   [6:0:0:3]    disk    iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1  /dev/sdi

fence_scsi_test

The output of the client side fence_scsi_test looks like using Open-iSCSI Initiators connected to LIO v3.0. Note that the SPC-3 capable devices pass the simple PROUT REGISTER/UNREGISTER operations.

   initiator# fence_scsi_test -s
   Testing all SCSI block devices...
   Attempted to register with devices:
   -------------------------------------
   	   /dev/sda Failure
           /dev/sdb Failure
           /dev/sdc Failure
           /dev/sdd Failure
           /dev/sde Success
           /dev/sdf Success
           /dev/sdg Failure
           /dev/sdh Success
           /dev/sdi Success
   -------------------------------------
   Number of devices tested: 9
   Number of devices passed: 4
   Number of devices failed: 5
   SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
   SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
   SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002
   SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
   SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
   SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002
   SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
   SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
   SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003
   SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
   SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s)
   SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003
   PSCSI Status Byte exception at task: d4771100 CDB: 0x5f Result: 0x08000002
   HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense
   PSCSI Status Byte exception at task: d4771200 CDB: 0x5f Result: 0x08000002
   HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense
   SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
   SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s)
   SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
   SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
   SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s)
   SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
   SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
   SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s)
   SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
   SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef
   SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s)
   SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001

See also

Personal tools
Namespaces
Variants
Actions
Navigation
Toolbox
Google AdSense