The smp_utils package

The smp_utils package

Introduction

The smp_utils package contains utilities for the Serial Attached SCSI (SAS) Management Protocol (SMP). Most utilities correspond to a single SMP function, sending out a request, checking for errors and if all is well processing the response. The response is either decoded, printed out in ASCII hexadecimal or sent as binary to stdout.

At the lower levels SAS is a point to point interconnect (like SATA). SAS has devices called expanders which have a similar role to switches in the Fibre Channel protocol and Ethernet. The primary role of SMP is to monitor and control SAS expanders. Most SAS Host Bus Adapters (HBAs) contain a SMP initiator through which SMP requests are sent and responses received. SAS expanders contain SMP targets that respond to SMP requests. SAS disks do not normally contains SMP targets and SATA disks do not contain SMP targets. SAS has two other protocols: SSP for transporting SCSI command sets to SAS devices (e.g. SAS disks and tape drives) , and STP for tunnelling the ATA command set to SATA disks. and S-ATAPI devices.

SAS now has four generations. The first three generations are approved standards: the original SAS (SAS ANSI INCITS 376-2003), SAS 1.1 (INCITS 417-2006) and SAS-2 (SAS-2 ANSI INCITS 457-2010). SAS-2's draft just prior to standardization: sas2r16.pdf (section 10.4.3 for SMP functions) is a useful reference. Technical work is ongoing for SAS 2.1 in which the firmware/software protocols including SMP have been placed in a separate document called the SAS Protocol Layer (SPL). At the time of writing the most recent SPL draft is spl-r06g.pdf (see section 9.4.3 for SMP functions). To avoid confusion, the four generations of SAS will be referred to here as 1, 1.1, 2 and 2.1 . Drafts, including those just prior to standardization, can be found at the www.t10.org site. The SMP functions for reading and writing to GPIO registers are defined in the Small Form Factor document SFF-8485 found at www.sffcommittee.com .

The smp_utils package targets the linux kernel 2.4 and 2.6 series. Preliminary work has been done in version 0.95 to support FreeBSD and it should be reasonably easy to port this package to other operating systems.

Contents of smp_utils

Here is a list in alphabetical order of the utilities in the smp_utils package:

Table 1. Utilities
Utility	SMP function	SAS version	Notes
smp_conf_general	CONFIGURE GENERAL	2	added in smp_utils version 0.92
smp_conf_route_info	CONFIGURE ROUTE INFORMATION	1
smp_discover	DISCOVER	1	SAS HBA drivers use this command to discover the topology
smp_discover_list	DISCOVER LIST		added in smp_utils version 0.91 (recently added to SAS-2)
smp_phy_control	PHY CONTROL	1	amongst other things, can send link and hard resets to attached phy
smp_phy_test	PHY TEST FUNCTION	1.1	added in smp_utils version 0.91
smp_read_gpio	READ GPIO REGISTER	1	defined in SFF-8485. May also be able to read the sideband signals on a HBA's wide internal cables.
smp_rep_exp_route_tbl	REPORT EXPANDER ROUTE TABLE LIST	2	added in smp_utils version 0.93
smp_rep_general	REPORT GENERAL	1
smp_rep_manufacturer	REPORT MANUFACTURER INFORMATION	1	Response contains device identification string somewhat like the response to a SCSI INQUIRY command.
smp_rep_phy_err_log	REPORT PHY ERROR LOG	1
smp_rep_phy_sata	REPORT PHY SATA	1	for expander phys with a STP to SATA bridge
smp_rep_route_info	REPORT ROUTE INFORMATION	1
smp_write_gpio	WRITE GPIO REGISTER	1	defined in SFF-8485. May also be able to write to the sideband signals on a HBA's wide internal cables.

As can be seen, the utility name is closely related to the SMP function name. The "SAS version" column indicates the first SAS version in which the SMP function appeared. For more information see one of the SAS drafts (i.e. SAS 1, SAS 1.1 or SAS 2), in the "Application layer" section within which the SMP functions are described.

Each utility has its own man page. Information common to all utilities can be found in the smp_utils man page.

Interfaces

As SMP is a relatively new protocol there are not yet well established pass through interfaces like there are for the SCSI protocol. SMP is a request response protocol with each request and response limited currently to 1032 bytes in length (including a 4 byte CRC at the end of each request and response).

This package currently supports three interfaces:

mpt: based on the MPT Fusion /dev/mptctl or /dev/mpt2ctl device
sgv4: (also sg) uses the SCSI generic version 4 interface and the SAS transport layer (added in version 0.93 and requires linux kernel 2.6.24 or later)
tpl: (also ai) based on a read write attribute in sysfs called smp_portal. Luben Tuikov's aic94xx driver implements this.

The mpt interface uses a single device node per system and thus needs a "ioc_num" to identify which HBA on the system it is going to send the SMP request. The "ioc_num" defaults to 0. If another value is needed then a syntax like this "smp_rep_general /dev/mptctl,3" can be used. The ioc_num can be found in the /proc/scsi/mptsas/<host_no> file in the linux lk 2.4 series and additionally in /sys/class/scsi_host/host<n>/unique_id file in the lk 2.6 series. The user could also check the log in the section where the MPT SAS HBA is registered for "ioc" strings and look at the following number. The SAS address of the target SMP device (typically a SAS expander) also needs to be given. The /dev/mptctl device node is associated with the mptctl module. Hence a "modprobe mptctl" command may be required. The /dev/mptctl device node is a character device with major 10 and minor 220 while /dev/mpt2ctl device node is a character device with major 10 and minor 221 .

The sgv4 interface uses the SG_IO ioctl, the bsg driver and the SCSI generic version 4 (hence "sgv4" which can be shortened to "sg") structure. From linux kernel 2.6.24 the SAS transport layer within the SCSI subsystem accepts SMP requests and forwards them on to those SAS low level (HBA) drivers that accept them. At this time the MPT SAS Fusion driver and the aic94xx driver accept SMP requests issued via this interface. The SMP target device is identified in one of two possible ways: via a sysfs directory name (e.g. /sys/class/bsg/expander-4:0 ) or via a device node created dynamically by the bsg driver (e.g. /dev/expander-4:0 ). The SMP target device SAS address is implicit (and this design limits the usefulness of "sgv4" for user space SAS topology discovery).

The tpl interface uses the sysfs smp_portal attribute which is part of the topological representation of a SAS domain (only found in the lk 2.6 series). SAS devices that identify themselves (during the discover process) as an SMP target receive a smp_portal attribute. Hence the SMP target SAS address is implicit in a smp_portal attribute. In the linux 2.6 series a sysfs search like this "find /sys -name smp_portal" will help in finding the interface file. An alternate aic94xx driver uses this interface.

Each utility attempts to work out at run time which interface it has. While there are three interfaces this is a reasonably simple process as their characteristics are quite different. To explicitly set the interface type the '--interface=<intf>[,force]' option can be given where <intf> is one of "mpt", "tpl" or "sgv4" (or alternatives "ai" or "sg"). Checks are also performed that the given device name isn't a normal file or worse a block device. A command like "smp_discover /dev/sda" makes no sense but if allowed to go ahead, would write the SMP request over the disk's boot sector! Normally these utilities will reject such an invocation, before doing any damage. If the user feels that they know better then checks can be overridden with something like this: "smp_discover --interface=mpt,force /dev/xyz" .

If the <smp_device> is not given, then each utility will check if the SMP_UTILS_DEVICE environment variable exists. If so the contents of that environment variable are used.

If the SMP target SAS address is not given (i.e. "--sa=<sas_addr>" is not given) and it is required, then each utility will check if the SMP_UTILS_SAS_ADDR environment variable exists. If so the contents of that environment variable are used.

Notes

The SFF-8485 document ( see www.sffcommittee.com ) defines the READ GPIO REGISTER and WRITE GPIO REGISTER functions for manipulating registers associated with the SGPIO interface. The sideband signals found in the wide internal cables defined by SAS (SAS 4i and mini SAS 4i) may carry SGPIO signals. Some SAS HBAs have wide internal connectors and have virtual SMP target ports (visible to the HBA driver but not the rest of the attached SAS domain) so that the driver or user space programs can access or control those sideband signals. To find out if a virtual SMP target port is present on a HBA try something like this: "smp_rep_manufacturer --sa=<sas_addr_of_hba> /dev/mptctl". Some enclosures have monitoring logic that present a SES-2 target type device on a SGPIO interface.

Examples

The command line interface can get cluttered if the SAS address of the expander and the OS device node through which the SMP functions travel, are given on each invocation. Two environment variables have been provided to simplify command line invocations:

# export SMP_UTILS_SAS_ADDR=0x500605b0000033ef
# export SMP_UTILS_DEVICE=/dev/mptctl,0

For MPT fusion HBAs, the mptctl module may need to be loaded in Linux:

# modprobe mptctl

If there are problems check for the "ioc_num" as outlined in the interfaces section above. For example "ioc3" implies the above should change to "export SMP_UTILS_DEVICE=/dev/mptctl,3". In the linux kernel 2.6 series this sequence may help finding an expander SAS address:

# find /sys -name "sas_device:expander*"
# cat "<any_found_path>/sas_address"

The following utility sends a SMP REPORT GENERAL request to a SMP target (e.g. an expander in an external enclosure) and prints the response. Full explicit arguments are given. The response indicates that we have a 12 port expander:

# smp_rep_general --sa=0x500605b0000033ef /dev/mptctl,0
Report general response:
expander change count: 8
expander route indexes: 144
number of phys: 12
configuring: 0
externally configurable route table: 1

Both the '--sa=' option and the SMP device name can be dropped if we have the above environment variables defined:

# smp_rep_general
Report general response:
expander change count: 8
expander route indexes: 144
number of phys: 12
configuring: 0
externally configurable route table: 1

Here is a DISCOVER on phy 6 of the expander showing that a SATA disk is attached:

# smp_discover -b -p 6
Discover response (brief):
phy identifier: 6
attached device type: end device
negotiated logical link rate: phy enabled; 1.5 Gbps
attached initiator: ssp=0 stp=0 smp=0 sata_host=0
attached target: ssp=0 stp=0 smp=0 sata_device=1
SAS address: 0x500605b0000033ef
attached SAS address: 0x500605b0000033e6
attached phy identifier: 0
routing attribute: table

In version 0.91 a '--multiple' option was added to smp_discover to scan the available phys on an expander. Adding '--brief' reduces the amount of output:

# smp_discover --multiple
Device <500605b0000033ef>, expander:
phy   0:S:attached:[0000000000000000:00]
phy   1:S:attached:[0000000000000000:00]
phy   2:S:attached:[5000c500005208ee:01 t(SSP)] 3 Gbps
phy   3:S:attached:[0000000000000000:00]
phy   4:T:attached:[50000d10002dc000:03 i(SSP+STP+SMP)] 3 Gbps
phy   5:T:attached:[500605b00006f260:03 i(SSP+STP+SMP)] 3 Gbps
phy   6:T:attached:[500605b0000033e6:00 t(SATA)] 1.5 Gbps
phy   7:T:attached:[0000000000000000:00]
phy   8:T:attached:[0000000000000000:00]
phy   9:T:attached:[0000000000000000:00]
phy 10:T:attached:[0000000000000000:00]
phy 11:T:attached:[5000c50001b0213a:01 t(SSP)] 3 Gbps

# smp_discover --multiple --brief
Device <500605b0000033ef>, expander (only connected phys shown):
phy   2:S:attached:[5000c500005208ee:01 t(SSP)] 3 Gbps
phy   4:T:attached:[50000d10002dc000:03 i(SSP+STP+SMP)] 3 Gbps
phy   5:T:attached:[500605b00006f260:03 i(SSP+STP+SMP)] 3 Gbps
phy   6:T:attached:[500605b0000033e6:00 t(SATA)] 1.5 Gbps
phy 11:T:attached:[5000c50001b0213a:01 t(SSP)] 3 Gbps

These shows two SAS initiator ports (on phy ids 4 and 5), two SAS target ports (on phy ids 2 and 11) and a SATA device (on phy id 6). In this case that it two separate SAS HBAs, two SAS disks and one SATA disk respectively. More STP to SATA bridge information can be fetched for phy 6 of the expander:

# smp_rep_phy_sata -p 6
Report phy SATA response:
phy identifier: 6
affiliations supported: 1
affiliation valid: 1
STP SAS address: 0x500605b0000033e6
register device to host FIS:
    34 00 50 01 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
affiliated STP initiator SAS address: 0x500605b00006f260

The first two GPIO configuration registers on the HBA can be read as follows. Notice that the SAS address of the expander (in the environment variable) needs to be overridden with the SAS address of the HBA itself (because that is where the virtual SMP target port is):

# smp_read_gpio --sa=0x500605b00006f260 --count=2
Read GPIO register response:
GPIO_CFG[0]:
    version: 0
    GPIO enable: 1
    cfg register count: 2
    gp register count: 0
    supported drive count: 0
GPIO_CFG[1]:
    blink generator rate B: 15
    blink generator rate A: 8
    force activity off: 4
    maximum activity on: 8
    stretch activity off: 0
    stretch activity on: 1

Now for something potentially useful: doing a hard reset (formerly called a device reset) on an expander attached SAS disk. To do this, the attached phy on the expander needs to be instructed to do a hard reset. That can be done with the PHY CONTROL SMP function. The SAS address of a target port on the SAS disk needs to be found. That requires a SCSI command (INQUIRY for the device identification VPD page) to be sent to the disk to be reset. In this example "sg_vpd" from the sg3_utils package is used for that. Then the expander phy connected to the disk is checked with a smp_discover, mainly for confirmation. Prior to sending the hard reset, the disk state is checked with a test unit ready command (sg_turs also from the sg3_utils package). The hard reset is then sent to the expander phy (with smp_phy_control) and then a request sense is sent to the disk. If the hard reset succeeded, the disk should report that a SCSI bus reset has occurred (old SCSI parallel interface messages will take some time to be displaced).

# sg_vpd --page=di_port --quiet /dev/sdc
0x5000c500005208ee,1

# smp_discover --phy=2 --brief
Discover response (brief):
phy identifier: 2
attached device type: end device
negotiated logical link rate: phy enabled; 3 Gbps
attached initiator: ssp=0 stp=0 smp=0 sata_host=0
attached target: ssp=1 stp=0 smp=0 sata_device=0
SAS address: 0x500605b0000033ef
attached SAS address: 0x5000c500005208ee
attached phy identifier: 1
routing attribute: subtractive

# sg_turs /dev/sdc
<no response means that /dev/sdc is ready>

# smp_phy_control --phy=2 --op=hr

# sg_requests /dev/sdc
Decode parameter data as sense data:
Fixed format, current; Sense key: Unit Attention
Additional sense: SCSI bus reset occurred
Field replaceable unit code: 2

Notice that utilities from sg3_utils used when communicating with the SAS disk start with "sg_". Utilities communicating with the expander come from the smp_utils package and they start with "smp_".

How do we know phy id 2 is the appropriate phy to send the hard reset to? We could send a 'smp_discover -mb' to the expander and scan for a match. It is also possible to ask the SAS disk what it is connected to and importantly what the attached phy identifier is. This can be done by fetching the protocol (SAS) specific log page from the disk. Notice the attached phy number on relative target port id 2 (i.e. the secondary port on the disk).

# sg_logs -T /dev/sdc
    SEAGATE   ST336754SS        0003
SAS Protocol Specific page
relative target port id = 1
number of phys = 1
phy identifier = 0
    attached device type: no device attached
    negotiated logical link rate: phy enabled; 3 Gbps
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000c500005208ed
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
relative target port id = 2
number of phys = 1
phy identifier = 1
    attached device type: expander device
    negotiated logical link rate: phy enabled; 3 Gbps
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=1
    SAS address = 0x5000c500005208ee
    attached SAS address = 0x500605b0000033ef
    attached phy identifier = 2
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0

The SMP protocol can also play a role in controlling some special features when a SATA disk is connected to an expander phy.

Download

**Table 2. smp_utils tarballs, rpm + deb packages**
version	tarball	source rpm	i386 rpm binary	debian package
0.90	smp_utils-0.90.tgz	smp_utils-0.90-1.src.rpm	smp_utils-0.90-1.i386.rpm	smp-utils_0.90-0.1_i386.deb
0.91	smp_utils-0.91.tgz	smp_utils-0.91-1.src.rpm	smp_utils-0.91-1.i386.rpm	smp-utils_0.91-0.1_i386.deb
0.92	smp_utils-0.92.tgz	smp_utils-0.92-1.src.rpm	smp_utils-0.92-1.i386.rpm	smp-utils_0.92-0.1_i386.deb
0.93	smp_utils-0.93.tgz	smp_utils-0.93-1.src.rpm	smp_utils-0.93-1.i386.rpm	smp-utils_0.93-0.1_i386.deb
0.94	smp_utils-0.94.tgz	smp_utils-0.94-1.src.rpm	smp_utils-0.94-1.i386.rpm	smp-utils_0.94-0.1_i386.deb
0.95	smp_utils-0.95.tgz	smp_utils-0.95-1.src.rpm	smp_utils-0.95-1.i386.rpm	smp-utils_0.95-0.1_i386.deb

The tarball contains an "rpm" spec file which is used to build source rpm and the i386 binary rpm. There is a debian sub-directory with files to build a "deb" package. Here is the ChangeLog . There are also ".tar.gz" and ".tar.bz2" versions of the tarball at the same location.

Return to main page.

Last updated: 19th November 2010