The smp_utils package

The smp_utils package

Introduction

The smp_utils package contains utilities for the Serial Attached SCSI (SAS) Management Protocol (SMP). Most utilities correspond to a single SMP function, sending out a request, checking for errors and if all is well processing the response. The response is either decoded, printed out in ASCII hexadecimal or sent as binary to stdout.

At the lower levels SAS is a point to point interconnect (like SATA). SAS has devices called expanders which have a similar role as switches do in the Fibre Channel protocol and Ethernet. The primary role of SMP is to monitor and control SAS expanders. Most SAS Host Bus Adapters (HBAs) contain a SMP initiator through which SMP requests are sent and responses received. SAS expanders contain SMP targets that respond to SMP requests. SAS disks do not normally contains SMP targets and SATA disks do not contain SMP targets. SAS has two other protocols: SSP for transporting SCSI command sets to SAS devices (e.g. SAS disks and tape drives) , and STP for tunnelling the ATA command set to SATA disks. and S-ATAPI devices.

SAS is a rapidly expanding new technology which already has three generations. The first two generations are approved standards: the original SAS (SAS ANSI INCITS 376-2003) and SAS 1.1 (INCITS 417-2006). New technical work is going into SAS-2 and at the time of writing the most recent draft is sas2r07.pdf (see section 10.4.3 for SMP functions). Drafts, including those just prior to standardization can be found at the www.t10.org site. Two SMP functions for reading and writing to GPIO registers are defined in the Small Form Factor document SFF-8485 found at www.sffcommittee.com .

The smp_utils package targets the linux kernel 2.4 and 2.6 series. Preliminary work has been done in version 0.92 to support FreeBSD and it should be reasonably easy to port this package to other operating systems.

Contents of smp_utils

Here is a list in alphabetical order of the utilities in the smp_utils package:

Table 1. Utilities
Utility	SMP function	Notes
smp_conf_general	CONFIGURE GENERAL	added in smp_utils version 0.92
smp_conf_route_info	CONFIGURE ROUTE INFORMATION
smp_discover	DISCOVER	SAS HBA drivers use this command to discover the topology
smp_discover_list	DISCOVER LIST	added in smp_utils version 0.91 (recently added to SAS-2)
smp_phy_control	PHY CONTROL	amongst other things, can send link and hard resets to attached phy
smp_phy_test	PHY TEST FUNCTION	added in smp_utils version 0.91
smp_read_gpio	READ GPIO REGISTER	defined in SFF-8485. May also be able to read the sideband signals on a HBA's wide internal cables.
smp_rep_general	REPORT GENERAL
smp_rep_manufacturer	REPORT MANUFACTURER INFORMATION	Response contains device identification string somewhat like the response to a SCSI INQUIRY command.
smp_rep_phy_err_log	REPORT PHY ERROR LOG
smp_rep_phy_sata	REPORT PHY SATA	for expander phys with a STP to SATA bridge
smp_rep_route_info	REPORT ROUTE INFORMATION
smp_write_gpio	WRITE GPIO REGISTER	defined in SFF-8485. May also be able to write to the sideband signals on a HBA's wide internal cables.

Each utility has its own man page. Information common to all utilities can be found in the smp_utils man page.

Interfaces

As SMP is a relatively new protocol there are not yet well established pass through interfaces like there are for the SCSI protocol. SMP is a request response protocol with each request and response limited currently to 1032 bytes in length (including a 4 byte CRC at the end of each request and response).

This package currently supports two interfaces: one based on the MPT Fusion /dev/mptctl device and the other based on a read write attribute in sysfs called smp_portal.

The mptctl interface uses a single device node per system and thus needs a "ioc_num" to identify which HBA on the system it is going to send the SMP request. The "ioc_num" defaults to 0. If another value is needed then a syntax like this "smp_rep_general /dev/mptctl,3" can be used. The ioc_num can be found in the /proc/scsi/mptsas/<host_no> file in the linux lk 2.4 series and additionally in /sys/class/scsi_host/host<n>/unique_id file in the lk 2.6 series. The user could also check the log in the section where the MPT SAS HBA is registered for "ioc" strings and look at the following number. The SAS address of the target SMP device (typically a SAS expander) also needs to be given.

The smp_portal interface is part of the topological representation of a SAS domain in the sysfs file system (only found in the lk 2.6 series). SAS devices that identify themselves (during the discover process) as an SMP target receive a smp_portal attribute. Hence the SMP target SAS address is implicit in a smp_portal attribute. In the linux 2.6 series a sysfs search like this "find /sys -name smp_portal" will help in finding the interface file.

Each utility attempts to work out at run time which interface it has. While there are only two interfaces this is a reasonably simple process as their characteristics are quite different. To explicitly set the interface type the '--interface=<intf>[,force]' option can be given where <intf> is one of "mpt", "tpl" or "ai" (the latter two for the smp_portal interface). Checks are also performed that the given device name isn't a normal file or worse a block device. A command like "smp_discover /dev/sda" makes no sense but if allowed to go ahead, would write the SMP request over the disk's boot sector! Normally these utilities will reject such an invocation, before doing any damage. If the user feels that they know better then checks can be overridden with something like this: "smp_discover --interface=mpt,force /dev/xyz" .

If the <smp_device> is not given, then each utility will check if the SMP_UTILS_DEVICE environment variable exists. If so the contents of that environment variable are used.

If the SMP target SAS address is not given (i.e. "--sa=<sas_addr>" is not given) and it is required, then each utility will check if the SMP_UTILS_SAS_ADDR environment variable exists. If so the contents of that environment variable are used.

Notes

The SFF-8485 document ( see www.sffcommittee.com ) defines the READ GPIO REGISTER and WRITE GPIO REGISTER functions for manipulating registers associated with the SGPIO interface. The sideband signals found in the wide internal cables defined by SAS (SAS 4i and mini SAS 4i) may carry SGPIO signals. Some SAS HBAs have wide internal connectors and have virtual SMP target ports (visible to the HBA driver but not the rest of the attached SAS domain) so that the driver or user space programs can access or control those sideband signals. To find out if a virtual SMP target port is present on a HBA try something like this: "smp_rep_manufacturer --sa=<sas_addr_of_hba> /dev/mptctl". Notice that is not possible with the smp_portal interface because the HBA does not flag itself (to a SMP DISCOVER) as an STP target.

Examples

The command line interface can get cluttered if the SAS address of the expander and the OS device node through which the SMP functions travel, are given on each invocation. Two environment variables have been provided to simplify command line invocations:

# export SMP_UTILS_SAS_ADDR=0x500605b0000033ef
# export SMP_UTILS_DEVICE=/dev/mptctl,0

For MPT fusion HBAs, the mptctl module may need to be loaded in Linux:

# modprobe mptctl

If there are problems check for the "ioc_num" as outlined in the interfaces section above. For example "ioc3" implies the above should change to "export SMP_UTILS_DEVICE=/dev/mptctl,3". In the linux kernel 2.6 series this sequence may help finding an expander SAS address:

# find /sys -name "sas_device:expander*"
# cat "<any_found_path>/sas_address"

The following utility sends a SMP REPORT GENERAL request to a SMP target (e.g. an expander in an external enclosure) and prints the response. Full explicit arguments are given. The response indicates that we have a 12 port expander:

# smp_rep_general --sa=0x500605b0000033ef /dev/mptctl,0
Report general response:
expander change count: 8
expander route indexes: 144
number of phys: 12
table to table supported: 0
zone address resolved supported: 0
configures others: 0
configuring: 0
externally configurable route table: 1

Both the '--sa=' option and the SMP device name can be dropped if we have the above environment variables defined:

# smp_rep_general
Report general response:
expander change count: 8
expander route indexes: 144
number of phys: 12
table to table supported: 0
zone address resolved supported: 0
configures others: 0
configuring: 0
externally configurable route table: 1

Here is a DISCOVER on phy 6 of the expander showing that a SATA disk is attached:

# smp_discover -b -p 6
Discover response (brief):
phy identifier: 6
attached device type: end device
attached reason: unknown
negotiated logical link rate: phy enabled; 1.5 Gbps
attached initiator: ssp=0 stp=0 smp=0 sata_host=0
attached target: ssp=0 stp=0 smp=0 sata_device=1
SAS address: 0x500605b0000033ef
attached SAS address: 0x500605b0000033e6
attached phy identifier: 0
routing attribute: table

In version 0.91 a '--multiple' option was added to smp_discover to scan the available phys on an expander. Adding '--brief' reduces the amount of output:

# smp_discover --multiple
Device <500605b0000033ef>, expander:
phy   0:S:attached:[0000000000000000:00]
phy   1:S:attached:[0000000000000000:00]
phy   2:S:attached:[5000c500005208ee:01 t(SSP)] 3 Gbps
phy   3:S:attached:[0000000000000000:00]
phy   4:T:attached:[50000d10002dc000:03 i(SSP+STP+SMP)] 3 Gbps
phy   5:T:attached:[500605b00006f260:03 i(SSP+STP+SMP)] 3 Gbps
phy   6:T:attached:[500605b0000033e6:00 t(SATA)] 1.5 Gbps
phy   7:T:attached:[0000000000000000:00]
phy   8:T:attached:[0000000000000000:00]
phy   9:T:attached:[0000000000000000:00]
phy 10:T:attached:[0000000000000000:00]
phy 11:T:attached:[5000c50001b0213a:01 t(SSP)] 3 Gbps

# smp_discover --multiple --brief
Device <500605b0000033ef>, expander (only connected phys shown):
phy   2:S:attached:[5000c500005208ee:01 t(SSP)] 3 Gbps
phy   4:T:attached:[50000d10002dc000:03 i(SSP+STP+SMP)] 3 Gbps
phy   5:T:attached:[500605b00006f260:03 i(SSP+STP+SMP)] 3 Gbps
phy   6:T:attached:[500605b0000033e6:00 t(SATA)] 1.5 Gbps
phy 11:T:attached:[5000c50001b0213a:01 t(SSP)] 3 Gbps

These shows two SAS initiator ports (on phy ids 4 and 5), two SAS target ports (on phy ids 2 and 11) and a SATA device (on phy id 6). In this case that it two separate SAS HBAs, two SAS disks and one SATA disk respectively. More STP to SATA bridge information can be fetched for phy 6 of the expander:

# smp_rep_phy_sata -p 6
Report phy SATA response:
phy identifier: 6
STP I_T nexus loss occurred: 0
affiliations supported: 1
affiliation valid: 1
STP SAS address: 0x500605b0000033e6
register device to host FIS:
    34 00 50 01 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
affiliated STP initiator SAS address: 0x500605b00006f260

The first two GPIO configuration registers on the HBA can be read as follows. Notice that the SAS address of the expander (in the environment variable) needs to be overridden with the SAS address of the HBA itself (because that is where the virtual SMP target port is):

# smp_read_gpio --sa=0x500605b00006f260 --count=2
Read GPIO register response:
GPIO_CFG[0]:
    version: 0
    GPIO enable: 1
    cfg register count: 2
    gp register count: 0
    supported drive count: 0
GPIO_CFG[1]:
    blink generator rate B: 15
    blink generator rate A: 8
    force activity off: 4
    maximum activity on: 8
    stretch activity off: 0
    stretch activity on: 1

Now for something potentially useful: doing a hard reset (formerly called a device reset) on an expander attached SAS disk. To do this, the attached phy on the expander needs to be instructed to do a hard reset. That can be done with the PHY CONTROL SMP function. The SAS address of a target port on the SAS disk needs to be found. That requires a SCSI command (INQUIRY for the device identification VPD page) to be sent to the disk to be reset. In this example "sg_vpd" from the sg3_utils package is used for that. Then the expander phy connected to the disk is checked with a smp_discover, mainly for confirmation. Prior to sending the hard reset, the disk state is checked with a test unit ready command (sg_turs also from the sg3_utils package). The hard reset is then sent to the expander phy (with smp_phy_control) and then a request sense is sent to the disk. If the hard reset succeeded, the disk should report that a SCSI bus reset has occurred (old SCSI parallel interface messages will take some time to be displaced).

# sg_vpd --page=di_port --quiet /dev/sdc
0x5000c500005208ee

# smp_discover --phy=2 --brief
Discover response (brief):
phy identifier: 2
attached device type: end device
attached reason: unknown
negotiated logical link rate: phy enabled; 3 Gbps
attached initiator: ssp=0 stp=0 smp=0 sata_host=0
attached target: ssp=1 stp=0 smp=0 sata_device=0
SAS address: 0x500605b0000033ef
attached SAS address: 0x5000c500005208ee
attached phy identifier: 1
routing attribute: subtractive

# sg_turs /dev/sdc
<no response means that /dev/sdc is ready>

# smp_phy_control --phy=2 --op=hr

# sg_requests /dev/sdc
Decode parameter data as sense data:
Fixed format, current; Sense key: Unit Attention
Additional sense: SCSI bus reset occurred
Field replaceable unit code: 2

Notice that utilities from sg3_utils used when communicating with the SAS disk start with "sg_". Utilities communicating with the expander come from the smp_utils package and they start with "smp_".

How do we know phy id 2 is the appropriate phy to send the hard reset to? We could send a 'smp_discover -mb' to the expander and scan for a match. It is also possible to ask the SAS disk what it is connected to and importantly what the attached phy identifier is. This can be done by fetching the protocol (SAS) specific log page from the disk. Notice the attached phy number on relative target port id 2 (i.e. the secondary port on the disk).

# sg_logs -T /dev/sdc
    SEAGATE   ST336754SS        0003
SAS Protocol Specific page
relative target port id = 1
number of phys = 1
phy identifier = 0
    attached device type: no device attached
    negotiated logical link rate: phy enabled; 3 Gbps
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000c500005208ed
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
relative target port id = 2
number of phys = 1
phy identifier = 1
    attached device type: expander device
    negotiated logical link rate: phy enabled; 3 Gbps
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=1
    SAS address = 0x5000c500005208ee
    attached SAS address = 0x500605b0000033ef
    attached phy identifier = 2
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0

The SMP protocol can also play a role in controlling some special features when a SATA disk is connected to an expander phy.

Download

**Table 2. smp_utils tarballs, rpm + deb packages**
version	tarball	source rpm **	i386 rpm binary **	debian package
0.90	smp_utils-0.90.tgz	smp_utils-0.90-1.src.rpm	smp_utils-0.90-1.i386.rpm	smp-utils_0.90-0.1_i386.deb
0.91	smp_utils-0.91.tgz	smp_utils-0.91-1.src.rpm	smp_utils-0.91-1.i386.rpm	smp-utils_0.91-0.1_i386.deb
0.92	smp_utils-0.92.tgz	smp_utils-0.92-1.src.rpm	smp_utils-0.92-1.i386.rpm	smp-utils_0.92-0.1_i386.deb

The tarball contains an "rpm" spec file which is used to build source rpm and the i386 binary rpm. There is a debian sub-directory with files to build a "deb" package. Here is the ChangeLog .

** Some browsers (e.g. firefox) may interpret a file with an extension of "rpm" as an audio file. In which case press the right button over the link and select "Save link as ..." to download the file.

Return to main page.

Last updated: 8th December 2006