Persistent Reservations
The Linux SCSI Target Wiki
SCSI-3 and SCSI-4 Persistent Reservations (PRs, target_core_pr.c) are a feature that is used for a number of cluster storage configurations for I/O fencing and failover/retakeover. PRs are a SCSI protocol feature, so they have been implemented in the LIO engine, and hence can be used for all SCSI transport fabrics, such as FCoE, Fibre Channel, iSCSI, Serial Attached SCSI (SAS), etc. Using PRs with Asymmetric Logical Unit Assignment (ALUA) is also fully supported in LinuxIO.
Contents |
Standards compliance
The SPC-3/SPC-4 features implemented (in target_core_pr.c) are described below. For legacy environments, compatible reservation handling for SPC-2 needs to be enabled.
SPC-4
LinuxIO implements full Persistent Reservation functionality, which entails two SPC-4 defined features:
-
APTPL=1
(Activate Persist across Target Power Loss) is usingkey=value
/var/target/pr/aptpl_$T10_UNIT_SERIAL
usingsg_persist
(announcement). -
SP_I_PT=1
(Specify Initiator Ports Capable) bit is supported and tested usingsg_persist
(announcement).
PROUT Service Actions
All PROUT Service Actions have been implemented: REGISTER
, RESERVE
, RELEASE
, CLEAR
, REGISTER_AND_IGNORE
, PREEMPT
, PREEMPT_AND_ABORT
and REGISTER_AND_MOVE
PROUT Reservation Types
All PROUT Reservation Types have been implemented: Write Exclusive, Exclusive Access, Write Exclusive Registrants Only, Exclusive Access Registrants Only Write Exclusive All Registrants, Exclusive Access All Registrants
PRIN Service Actions
All PRIN Service Actions have been implemented: READ_KEYS
, READ_RESERVATION
, REPORT_CAPABILITIES
, READ_FULL_STATUS
Test and Verification

LinuxIO IBLOCK, FILEIO and RAMDISK subsystem plugins are passing SPC-3 level validation for iSCSI target PRs. The following cluster clients that use Linux-iSCSI target mode PR have been tested.
RHEL/CentOS ≥5.3
Test and validation for RHEL/CentOS was performed with GFS (e.g., with multiple writers). LinuxIO capable storage objects from IBLOCK, FILEIO, and RAMDISK pass the fence_scsi_test.
RHEL/CentOS uses SCSI I/O Fencing via the sg_persist CLI operation in sg3_util for Write Exclusive, Registrants Only and PREEMPT_AND_ABORT
takeover using ext3 mounts.
SPC-3 PR emulation is disabled on top of the pass-through SCSI subsystem plugin (this is WIP for existing SCSI devices).
Microsoft Server 2008/R2 Failover Cluster
As of April 2009, LinuxIO is passing validation using the MS Verification test suite. Test and validation for Microsoft Server 2008 Failover Cluster was performed using the MS domain validation suite.
Microsoft Server 2008 Failover Cluster uses Write Exclusive, Registrants Only and PROUT PREEMPT
for production failure cases.
Extensive work by the LinuxIO developer and user community has made this possible. If you are interested in further testing and want to deploy production services on our code please let us know.
Other
If you have an iSCSI Initiator that implements SPC-3 PR logic and would like to test it against LIO 3, please don't hesitate to contact us on our development mailing list.
ConfigFS object tree
Target
Target/configFS layout. This is the output of an IBLOCK exported Linux/LVM DM block_device (lvm_test0) via tree:
tree /sys/kernel/config/target/core |-- iblock_0 | |-- hba_info | `-- lvm_test0 | |-- alua_lu_gp | |-- attrib | | |-- block_size | | |-- emulate_tas | | |-- emulate_ua_intlck_ctrl | | |-- hw_block_size | | |-- hw_max_sectors | | |-- hw_queue_depth | | |-- max_sectors | | |-- queue_depth | | `-- task_timeout | |-- control | |-- enable | |-- fd | |-- info | |-- pr | | |-- res_holder | | |-- res_pr_all_tgt_pts | | |-- res_pr_generation | | |-- res_pr_holder_tg_port | | |-- res_pr_registered_i_pts | | |-- res_pr_type | | `-- res_type | `-- wwn | |-- vpd_assoc_logical_unit | |-- vpd_assoc_scsi_target_device | |-- vpd_assoc_target_port | |-- vpd_protocol_identifier | `-- vpd_unit_serial
SPC-3 Reservations
The iblock_0/lvm_test0/pr/
configFS attributes are used to see the current reservation/registration status for different SCSI I_T nexuses:
cat /sys/kernel/config/target/core/iblock_0/lvm_test0/pr/res_* SPC-3 Reservation: iSCSI Initiator: iqn.1993-08.org.debian:01:2dadf92d0ef SPC-3 Reservation: All {{Target}} ports registration 0x00000002 SPC-3 Reservation: iSCSI target node Endpoint: iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd SPC-3 Reservation: Relative Port Identifer Tag: 2 iSCSI Portal Group Tag: 1 iSCSI Logical Unit: 0 SPC-3 PR Registrations: iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000001234abcd PRgen: 0x00000000 iSCSI Node: iqn.1993-08.org.debian:01:2dadf92d0ef Key: 0x000000004567ffff PRgen: 0x00000001 SPC-3 Reservation Type: Write Exclusive Access, Registrants Only
T10 WWN numbers
The iblock_0/lvm_test0/wwn
configFS attributes are used to see world wide unique LUN information, be it real from the SCSI device, or emulated using a admin present value into iblock_0/lvm_test0/wwn/evp_unit_serial
.
cat /sys/kernel/config/target/core/iblock_0/lvm_test0/wwn/* T10 VPD Identifier Association: addressed logical unit T10 VPD Identifier Type: NAA T10 VPD Binary Device Identifier: 36001405a97e4ce21c0711de829b000c2 T10 VPD Identifier Association: addressed logical unit T10 VPD Identifier Type: T10 Vendor ID based T10 VPD ASCII Device Identifier: LIO-ORG T10 VPD Unit Serial Number: a97e4ce21c0711de829b000c2943d57b
This is required in order to use the NAA VPD device identifers.
Initiator
sg_persist
Here's a reference output on the initiator side (via PRIN REPORT_CAPABILITES):
initiator# sg_persist --in --report-capabilities -v /dev/sde inquiry cdb: 12 00 00 00 24 00 LIO-ORG IBLOCK 3.0 Peripheral device type: disk Persistent Reservation In cmd: 5e 02 00 00 00 00 00 20 00 00 Report capabilities response: Compatible Reservation Handling(CRH): 0 Specify Initiator Ports Capable(SIP_C): 0 All Target Ports Capable(ATP_C): 1 Persist Through Power Loss Capable(PTPL_C): 0 Type Mask Valid(TMV): 1 Allow commands: 1 Persist Through Power Loss Active(PTPL_A): 0 Support indicated in Type mask: Write Exclusive, all registrants: 1 Exclusive Access, registrants only: 1 Write Exclusive, registrants only: 1 Exclusive Access: 1 Write Exclusive: 1 Exclusive Access, all registrants: 1
sg3_utils
Linux support for client side PRs using sg3_utils
and RHEL cluster scripts. The fence_scsi_test
was originally written for RHEL clusters, and is available on a number of other Linux distributions including SuSE, Debian, Ubuntu, etc.
initiator# lsscsi [0:0:0:0] disk VMware, VMware Virtual S 1.0 /dev/sdb [0:0:1:0] disk VMware, VMware Virtual S 1.0 /dev/sdc [0:0:2:0] disk VMware, VMware Virtual S 1.0 /dev/sdd [1:0:0:0] disk ATA VMware Virtual I 0000 /dev/sda [5:0:0:0] disk LIO-ORG IBLOCK 3.0 /dev/sde [6:0:0:0] disk LIO-ORG IBLOCK 3.0 /dev/sdf [6:0:0:1] disk VMware, VMware Virtual S 1.0 /dev/sdg [6:0:0:2] disk LIO-ORG RAMDISK-MCP 3.0 /dev/sdh [6:0:0:3] disk LIO-ORG FILEIO 3.0 /dev/sdi initiator# lsscsi --transport [0:0:0:0] disk /dev/sdb [0:0:1:0] disk /dev/sdc [0:0:2:0] disk /dev/sdd [1:0:0:0] disk /dev/sda [5:0:0:0] disk iqn.2003-01.org.linux-iscsi.target.i686:sn.cff3eedbd2fd,t,0x1 /dev/sde [6:0:0:0] disk iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1 /dev/sdf [6:0:0:1] disk iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1 /dev/sdg [6:0:0:2] disk iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1 /dev/sdh [6:0:0:3] disk iqn.2003-01.org.linux-iscsi.target.i686:sn.e475ed6fcdd0,t,0x1 /dev/sdi
fence_scsi_test
The output of the client side fence_scsi_test
looks like using Open-iSCSI Initiators connected to LIO. Note that the SPC-3 capable devices pass the simple PROUT REGISTER/UNREGISTER operations.
initiator# fence_scsi_test -s Testing all SCSI block devices... Attempted to register with devices: ------------------------------------- /dev/sda Failure /dev/sdb Failure /dev/sdc Failure /dev/sdd Failure /dev/sde Success /dev/sdf Success /dev/sdg Failure /dev/sdh Success /dev/sdi Success ------------------------------------- Number of devices tested: 9 Number of devices passed: 4 Number of devices failed: 5
SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s) SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002 SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s) SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000002 SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s) SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003 SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef SPC-3 PR [iSCSI] for SINGLE TCM Subsystem iblock Object Target Port(s) SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000003 PSCSI Status Byte exception at task: d4771100 CDB: 0x5f Result: 0x08000002 HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense PSCSI Status Byte exception at task: d4771200 CDB: 0x5f Result: 0x08000002 HBA_[1]_PLUG[pscsi]: Set SAM STATUS: 0x02 and sense SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s) SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001 SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef SPC-3 PR [iSCSI] for SINGLE TCM Subsystem rd_mcp Object Target Port(s) SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001 SPC-3 PR [iSCSI] Service Action: REGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s) SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001 SPC-3 PR [iSCSI] Service Action: UNREGISTER Initiator Node: iqn.1993-08.org.debian:01:2dadf92d0ef SPC-3 PR [iSCSI] for SINGLE TCM Subsystem fileio Object Target Port(s) SPC-3 PR [iSCSI] SA Res Key: 0x000000007f000101 PRgeneration: 0x00000001
See also
- LIO
- SCSI: Asymmetric Logical Unit Assignment (ALUA), Multiple Connections per Session (MC/S), Error Recovery Level (ERL), vStorage APIs for Array Integration (VAAI)
- Fabric modules: FCoE, Fibre Channel, iSCSI, iSER, SRP and vHost
- Management: targetcli