The smp_utils package

The smp_utils package

Introduction

The smp_utils package contains utilities for the Serial Attached SCSI (SAS) Management Protocol (SMP). Most utilities correspond to a single SMP function, sending out a request, checking for errors and if all is well processing the response. The response is either decoded, printed out in ASCII hexadecimal or sent as binary to stdout.

At the lower levels SAS is a point to point interconnect (like SATA). SAS has devices called expanders which have a similar role as switches do in the Fibre Channel protocol and ethernet. The primary role of SMP is to monitor and control SAS expanders. Most SAS Host Bus Adapters (HBAs) contain a SMP initiator through which SMP requests are sent and responses received. SAS expanders contain SMP targets that respond to SMP requests. SAS disks do not normally contains SMP targets and SATA disks do not contain SMP targets. SAS has two other protocols: SSP for transporting SCSI command sets to SAS devices (e.g. SAS disks and tape drives) , and STP for tunnelling the ATA command set to SATA disks. and S-ATAPI devices.

SAS is a rapidly expanding new technology which already has three generations. The first two generations are approved standards: the original SAS (SAS ANSI INCITS 376-2003) and SAS 1.1 (INCITS 417-2006). New technical work is going into SAS-2 and at the time of writing the most recent draft is sas2r05a.pdf (see section 10.4.3 for SMP functions). Drafts, including those just prior to standardization can be found at the www.t10.org site. Two SMP functions for reading and writing to GPIO registers are defined in the Small Form Factor document SFF-8485 found at www.sffcommittee.com .

The smp_utils package targets the linux kernel 2.4 and 2.6 series. It should be reasonably easy to port this package to other operating systems.

Contents of smp_utils

Here is a listing in alphabetical order of the utilities in the smp_utils package:

Table 1. Utilities
Utility	SMP function	Notes
smp_conf_route_info	CONFIGURE ROUTE INFORMATION
smp_discover	DISCOVER	SAS HBA drivers use this command to discover the topology
smp_discover_list	DISCOVER LIST	added in smp_utils version 0.91 (recently added to SAS-2)
smp_phy_control	PHY CONTROL	amongst other things, can send link and hard resets to attached phy
smp_phy_test	PHY TEST FUNCTION	added in smp_utils version 0.91
smp_read_gpio	READ GPIO REGISTER	defined in SFF-8485. May also be able to read the sideband signals on a HBA's wide internal cables.
smp_rep_general	REPORT GENERAL
smp_rep_manufacturer	REPORT MANUFACTURER INFORMATION
smp_rep_phy_err_log	REPORT PHY ERROR LOG
smp_rep_phy_sata	REPORT PHY SATA	for expander phys with a STP to SATA bridge
smp_rep_route_info	REPORT ROUTE INFORMATION
smp_write_gpio	WRITE GPIO REGISTER	defined in SFF-8485. May also be able to write to the sideband signals on a HBA's wide internal cables.

Each utility has its own man page. Information common to all utilities can be found in the smp_utils man page.

Interfaces

As SMP is a relatively new protocol there are not yet well established pass through interfaces like there are for the SCSI protocol. SMP is a request response protocol with each request and response limited currently to 1032 bytes in length (including a 4 byte CRC at the end of each request and response).

This package currently supports two interfaces: one based on the MPT Fusion /dev/mptctl device and the other based on a read write attribute in sysfs called smp_portal. The two interfaces are structurally different: the mptctl is a single device (per system) that needs a "ioc_num" to identify which HBA adapter on the system it is going to send the SMP request from. The SAS address of the target SMP device (typically a SAS expander) also needs to be given. The "ioc_num" defaults to 0. If another value is needed then a syntax like this "smp_rep_general /dev/mptctl,3" can be used.

The smp_portal interface is part of the topological representation of a SAS domain in the sysfs file system (only found in the lk 2.6 series). SAS devices that identify themselves (during the discover process) as an SMP target receive a smp_portal attribute. Hence the SMP target SAS address is implicit in a smp_portal attribute.

Each utility attempts to work out at run time which interface it has. While there are only two interfaces this is a reasonably simple process as their characteristics are quite different. To explicitly set the interface type the '--interface=<intf>[,force]' option can be given where <intf> is one of "mpt", "tpl" or "ai" (the latter two for the smp_portal interface). Checks are also performed that the given device name isn't a normal file or worse a block device. A command like "smp_discover /dev/sda" makes no sense but if allowed to go ahead, would write the SMP request over the disk's boot sector! Normally these utilities will reject such an invocation, before doing any damage. If the user feels that they know better then checks can be overridden with something like this: "smp_discover --interface=mpt,force /dev/xyz" .

If the <smp_device> is not given, then each utility will check if the SMP_UTILS_DEVICE environment variable exists. If so its contents are used.

If the SMP target SAS address is not given (i.e. "--sa=<sas_addr>" is not given) and it is required, then each utility will check if the SMP_UTILS_SAS_ADDR environment variable exists. If so its contents are used.

Notes

The SFF-8485 document ( see www.sffcommittee.com ) defines the READ GPIO REGISTER and WRITE GPIO REGISTER functions for manipulating registers associated with the SGPIO interface. The sideband signals found in the wide internal cables defined by SAS (SAS 4i and mini SAS 4i) may carry SGPIO signals. Some SAS HBAs have wide internal connectors and have virtual SMP target ports (visible to the HBA driver but not the rest of the attached SAS domain) so that the driver or user space programs can control those sideband signals. To find out if a virtual SMP target port is present on a HBA try something like this: "smp_rep_manufacturer --sa=<sas_addr_of_hba> /dev/mptctl". Notice that is not possible with the smp_portal interface because the HBA does not flag itself (to a SMP DISCOVER) as an STP target.

Examples

Set up environment variables to simplify command line invocations:

# export SMP_UTILS_SAS_ADDR=0x500605b000000af0
# export SMP_UTILS_DEVICE=/dev/mptctl,0

For MPT fusion HBAs, the mptctl module may need to be loaded:

# modprobe mptctl

The following indicates that we have a 12 port expander:

# smp_rep_general
Report general response:
expander change count: 8
expander route indexes: 144
number of phys: 12
configuring: 0
configurable route table: 1

Here is a DISCOVER on phy 7 of the expander showing that a SATA disk is attached:

# smp_discover -b -p 7
Discover response (brief):
phy identifier: 7
attached device type: end device
negotiated physical link rate: phy enabled; 1.5 Gbps
attached initiator: ssp=0 stp=0 smp=0 sata_host=0
attached target: ssp=0 stp=0 smp=0 sata_device=1
SAS address: 0x500605b000000af0
attached SAS address: 0x500605b000000af7
attached phy identifier: 0
routing attribute: table

In version 0.91 a '--multiple' option was added to smp_discover to scan the available phys on an expander. Adding '--brief' reduces the amount of output:

# smp_discover --multiple
Device <500605b000000af0>, expander:
phy   0:S:attached:[0000000000000000:00]
phy   1:S:attached:[0000000000000000:00]
phy   2:S:attached:[5000c500005208ee:01 t(SSP)] 3 Gbps
phy   3:S:attached:[5000c500005208ed:00 t(SSP)] 3 Gbps
phy   4:T:attached:[500605b00006f260:02 i(SSP+STP+SMP)] 3 Gbps
phy   5:T:attached:[50000d10002dc000:01 i(SSP+STP+SMP)] 3 Gbps
phy   6:T:attached:[0000000000000000:00]
phy   7:T:attached:[0000000000000000:00]
phy   8:T:attached:[5000c50000520a29:00 t(SSP)] 3 Gbps
phy   9:T:attached:[5000c50000520a2a:01 t(SSP)] 3 Gbps
phy 10:T:attached:[500605b000000afa:00 t(SATA)] 1.5 Gbps
phy 11:T:attached:[500605b000000afb:00 t(SATA)] 1.5 Gbps

# smp_discover --multiple --brief
Device <500605b000000af0>, expander (only connected phys shown):
phy   2:S:attached:[5000c500005208ee:01 t(SSP)] 3 Gbps
phy   3:S:attached:[5000c500005208ed:00 t(SSP)] 3 Gbps
phy   4:T:attached:[500605b00006f260:02 i(SSP+STP+SMP)] 3 Gbps
phy   5:T:attached:[50000d10002dc000:01 i(SSP+STP+SMP)] 3 Gbps
phy   8:T:attached:[5000c50000520a29:00 t(SSP)] 3 Gbps
phy   9:T:attached:[5000c50000520a2a:01 t(SSP)] 3 Gbps
phy 10:T:attached:[500605b000000afa:00 t(SATA)] 1.5 Gbps
phy 11:T:attached:[500605b000000afb:00 t(SATA)] 1.5 Gbps

More STP to SATA bridge information can be fetched for phy 10 of the expander:

# smp_rep_phy_sata -p 10
Report phy error sata response:
phy identifier: 10
STP I_T nexus loss occurred: 0
affiliations supported: 0
affiliation valid: 0
STP SAS address: 0x500605b000000afa
register device to host FIS:
    34 00 50 01 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
affiliated STP initiator SAS address: 0x500605b00006f260

The first two GPIO configuration registers on the HBA can be read as follows. Notice that the SAS address of the expander (in the environment variable) needs to be overridden with the SAS address of the HBA itself (because that is where the virtual SMP target port is):

# smp_read_gpio --sa=0x500605b00006f260 --count=2
Read GPIO register response:
GPIO_CFG[0]:
    version: 0
    GPIO enable: 1
    cfg register count: 2
    gp register count: 0
    supported drive count: 0
GPIO_CFG[1]:
    blink generator rate B: 15
    blink generator rate A: 8
    force activity off: 4
    maximum activity on: 8
    stretch activity off: 0
    stretch activity on: 1

Now for something potentially useful: doing a hard reset (formerly called a device reset) on an expander attached SAS disk. To do this, the attached phy on the expander needs to be instructed to do a hard reset. That can be done with the PHY CONTROL SMP function. First the SAS address of a target port on the SAS disk needs to be found, "sg_vpd" is used for that. Then the expander phy connected to the disk is checked with a smp_discover, mainly for confirmation. Prior to sending the hard reset, the disk state is checked with a test unit ready command (sg_turs). The hard reset is then sent to the expander phy (smp_phy_control) and then a test unit ready is sent to the disk again. If the hard reset succeeded, the disk should report that a SCSI bus reset has occurred (old SCSI parallel interface messages will take some time to be displaced).

# sg_vpd --page=di_port --quiet /dev/sdc
0x5000c50000520a2a

# smp_discover --phy=9 --brief
Discover response (brief):
phy identifier: 9
attached device type: end device
negotiated physical link rate: phy enabled; 3 Gbps
attached initiator: ssp=0 stp=0 smp=0 sata_host=0
attached target: ssp=1 stp=0 smp=0 sata_device=0
SAS address: 0x500605b000000af0
attached SAS address: 0x5000c50000520a2a
attached phy identifier: 1
routing attribute: table

# sg_turs /dev/sdc
Completed 1 Test Unit Ready commands with 0 errors

# smp_phy_control --phy=9 --op=hr

# sg_turs /dev/sdc
test unit ready: Fixed format, current; Sense key: Unit Attention
Additional sense: SCSI bus reset occurred
Field replaceable unit code: 2
Completed 1 Test Unit Ready commands with 1 errors

Notice that utilities from sg3_utils are used when communicating with the SAS disk. Those utilities start with "sg_". When communicating with the expander, utilities from this package are needed, and they start with "smp_".

How do we know phy id 9 is the appropriate phy to send the hard reset to? We could send a smp_discover to each phy on the expander. It is also possible to ask the SAS disk what it is connected to and importantly what the attached phy identifier is. This can be done by fetching the protocol (SAS) specific log page from the disk. Notice the attached phy number on relative target port 2 (i.e. the secondary port on the disk).

[root@sas ~]# sg_logs -T /dev/sdc
    SEAGATE   ST336754SS        0003
SAS Protocol Specific page
relative target port id = 1
number of phys = 1
phy identifier = 0
    attached device type: no device attached
    negotiated physical link rate: phy enabled; 3 Gbps
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=0
    SAS address = 0x5000c50000520a29
    attached SAS address = 0x0
    attached phy identifier = 0
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0
relative target port id = 2
number of phys = 1
phy identifier = 1
    attached device type: edge expander device
    negotiated physical link rate: phy enabled; 3 Gbps
    attached initiator port: ssp=0 stp=0 smp=0
    attached target port: ssp=0 stp=0 smp=1
    SAS address = 0x5000c50000520a2a
    attached SAS address = 0x500605b000000af0
    attached phy identifier = 9
    Invalid DWORD count = 0
    Running disparity error count = 0
    Loss of DWORD synchronization = 0
    Phy reset problem = 0

The SMP protocol can also play a role in controlling some special features when a SATA disk is connected to an expander phy.

Download

**Table 2. smp_utils tarballs, rpm + deb packages**
version	tarball	source rpm **	i386 rpm binary **	debian package
0.90	smp_utils-0.90.tgz	smp_utils-0.90-1.src.rpm	smp_utils-0.90-1.i386.rpm	smp-utils_0.90-0.1_i386.deb
0.91	smp_utils-0.91.tgz	smp_utils-0.91-1.src.rpm	smp_utils-0.91-1.i386.rpm	smp-utils_0.91-0.1_i386.deb

The tarball contains an "rpm" spec file which is used to build source rpm and the i386 binary rpm. There is a debian subdirectory with files to build a "deb" package. Here is the ChangeLog .

** Some browsers (e.g. firefox) may interpret a file with an extension of "rpm" as an audio file. In which case press the right button over the link and select "Save link as ..." to download the file.

Return to main page.

Last updated: 22nd August 2006