The  smp_utils package

1. The  smp_utils package

   1. Introduction

   2. Contents of smp_utils

   3. Interfaces

      1. Linux

      2. Solaris

      3. FreeBSD

   4. Notes

   5. Enclosure in and out ports

   6. Examples

      1. Finding expanders

      2. Discover

      3. General information

      4. SATA disk affiliations

      5. Zoning

      6. Expanders and enclosures

   7. Download

Introduction

The smp_utils package contains utilities for the Serial Attached SCSI (SAS) Serial Management Protocol (SMP).  Most utilities correspond to a single SMP function, sending out a request, checking for errors and if all is well processing the response. The response is either decoded, printed out in ASCII hexadecimal or sent as binary to stdout.

At the lower levels SAS is a point to point interconnect (like SATA). SAS has devices called expanders which have a similar role to switches in the Fibre Channel protocol and the Ethernet. The primary role of SMP is to monitor and control SAS expanders. Most SAS Host Bus Adapters (HBAs) contain a SMP initiator through which SMP requests are sent and responses received. SAS expanders contain SMP targets that respond to SMP requests. SAS disks do not normally contains SMP targets and SATA disks do not contain SMP targets. SAS has two other protocols: SSP for transporting SCSI command sets to SAS devices (e.g. SAS disks and tape drives) , and STP for tunnelling the ATA command set to a SAS expander.

SAS now has multiple generations. The first six generations are approved standards: the original SAS (SAS ANSI INCITS 376-2003), SAS 1.1 (INCITS 417-2006), SAS-2 (ANSI INCITS 457-2010), SAS-2.1 (ANSI INCITS 478-2011), SAS-3 (ANSI INCITS 519-2014) and SAS-4 (ANSI INCITS 534-2019) . In SAS-2.1 the higher level layers including the SMP protocol where placed in a separate standard known as the SAS Protocol Layer (SPL) and it now has four standards: SPL (ANSI INCITS 476-2011), SPL-2 (ANSI INCITS 505-2013), SPL-3 (ANSI INCITS 492-2015) and SPL-4 (ANSI INCITS 538-2018). The most recent SPL-5 draft is spl5r09.pdf . To lessen confusion, the six generations of SAS will be referred to here as 1, 1.1, 2, 2.1, 3.0 and 4.0. Drafts, including those just prior to standardization, can be found at the www.t10.org site. The SMP functions for reading and writing to GPIO registers are defined in the Small Form Factor document SFF-8485 found at www.sffcommittee.com .

The smp_utils package targets the Linux kernel 2.6, 3, 4 and 5 series. This page describes smp_utils version 0.99 . Ports to Solaris and FreeBSD (requiring FreeBSD 9.0 or later) were added in smp_utils version 0.97.

Due to a large number of changes between smp_utils version 0.96 and older versions, an older version of this page can be found at smp_utils095.html . Its examples were written in 2008 and may be useful for older Linux kernels (e.g. prior to lk 2.6.30).

Contents of smp_utils

Here is a list in alphabetical order of the utilities in the smp_utils package:

Table 1. Utilities

As can be seen, the utility name is closely related to the SMP function name. The "SAS version" column indicates the first SAS version in which the SMP function appeared. For more information see one of the SAS drafts (i.e. SAS 1, SAS 1.1, SAS 2, SPL [1], SPL 2, SPL 3, SPL 4 and SPL5), in the "Application layer" section within which the SMP functions are described.

Each utility has its own man page. Information common to all utilities can be found in the smp_utils man page.

Interfaces

Linux

SMP pass through interfaces are not yet as well established as those for the SCSI protocol. SMP is a request response protocol with each request and response limited currently to 1028 bytes in length (including a 4 byte CRC at the end of each request and response).

This package currently supports three interfaces:

• aac: supports Adaptec/PMC RAID adapters (added in version 0.98)

• mpt: based on the MPT Fusion /dev/mptctl, /dev/mpt2ctl or /dev/mpt3ctl  device

• sgv4: (also sg) uses the SCSI generic version 4 interface and the SAS transport layer (added in version 0.93 and requires Linux kernel 2.6.24 or later)

The aac interface uses a "decorated" device name of the form /dev/aac[N[,E_ID]] where "N" is the RAID controller identifier number (and is obtained from the unique_id as discussed below for the mpt interface) and "E_ID" is the expander identifier for cases where there is more than one expander. The mpt interface uses a single device node per system. Since there may be multiple HBAs in a system, an "ioc_num" is used to identify which HBA is going to send the SMP request.  The "ioc_num" defaults to 0. If another value is needed then a syntax like this "smp_rep_general /dev/mptctl,3" can be used. The ioc_num can be found in the /proc/scsi/mptsas/<host_no> file in the Linux lk 2.4 series and additionally in /sys/class/scsi_host/host<n>/unique_id file in the lk 2.6 and 3 series. The user could also check the log in the section where the MPT SAS HBA is registered for "ioc" strings and look at the following number. The SAS address of the target SMP device (typically a SAS expander) also needs to be given. The /dev/mptctl device node is associated with the mptctl module. Hence a "modprobe mptctl" command may be required. The /dev/mptctl device node is a character device with major 10 and minor 220 while /dev/mpt2ctl device node is a character device with major 10 and minor 221 .

The sgv4 interface uses the SG_IO ioctl, the bsg driver and the SCSI generic version 4 (hence "sgv4" which can be shortened to "sg") structure. From Linux kernel 2.6.24 the SAS transport layer within the SCSI subsystem accepts SMP requests and forwards them on to those SAS low level (HBA) drivers that accept them. At this time SAS HBAs based on libsas should support this interface (e.g. those that use mptsas, mpt2sas and mpt3sas). Note that most SAS RAID controllers do not use libsas. The SMP target device is identified in one of two possible ways: via a sysfs directory name (e.g. /sys/class/bsg/expander-4:0 ) or via a device node created dynamically by the bsg driver (e.g. /dev/bsg/expander-4:0 ). The SAS address of the expander can be found in the /sys/class/sas_device directory (e.g.  /sys/class/sas_device/expander-4:0/sas_address ).

Each utility attempts to work out at run time which interface it has. While there are two interfaces this is a reasonably simple process as their characteristics are quite different. To explicitly set the interface type the '--interface=<intf>[,force]' option can be given where <intf> is one of "mpt" or "sgv4" (or "sg"). Checks are also performed that the given device name isn't a normal file or worse a block device. A command like "smp_discover /dev/sda" makes no sense but if allowed to go ahead, would write the SMP request over the disk's boot sector! Normally these utilities will reject such an invocation, before doing any damage. If the user feels that they know better then checks can be overridden with something like this: "smp_discover --interface=mpt,force /dev/xyz" .

If the <smp_device> is not given, then each utility will check if the SMP_UTILS_DEVICE environment variable exists. If so the contents of that environment variable are used. If the SMP target SAS address is not given (i.e. "--sa=<sas_addr>" is not given) and it is required, then each utility will check if the SMP_UTILS_SAS_ADDR environment variable exists. If so the contents of that environment variable are used.

Solaris

The Solaris port uses the USMP pass-through mechanism which was introduced to Solaris in 2008. Expanders have device names like /dev/smp/expd3 .

FreeBSD

Logic to support SMP was added to FreeBSD in late 2010 and is available in FreeBSD 9 and later. The CAM subsystem was extended but it has a problem because it only supports SCSI devices and an expander is not a SCSI device. SAS-2 expanders are often packaged (on the same silicon) with a SCSI Enclosure Services (SES) device. So FreeBSD uses this "twin" SES device to communicate with an expander. So expanders have device names like /dev/ses2 .

Notes

The SFF-8485 document (see www.sffcommittee.com ) defines the READ GPIO REGISTER and WRITE GPIO REGISTER functions for manipulating registers associated with the SGPIO interface. The sideband signals found in the wide internal cables defined by SAS (SAS 4i and mini SAS 4i) may carry SGPIO signals. Some SAS HBAs have wide internal connectors and have virtual SMP target ports (visible to the HBA driver but not the rest of the attached SAS domain) so that the driver or user space programs can access or control those sideband signals. To find out if a virtual SMP target port is present on a HBA try something like this: "smp_rep_manufacturer /dev/bsg/sas_host6". Some enclosures have monitoring logic that present a SES-2 target type device on a SGPIO interface.

In SAS-1.1 edge expanders and fanout expanders where defined. In SAS-2 there are just "expanders" with that name replacing the identifier previously used for "edge expanders". In SAS-2 the attached device type identifier previously used for fanout expanders is now marked obsolete. SAS-2 expanders are a lot more capable than SAS-1.1 expanders. For example SAS-2 expanders can configure themselves and other (e.g. SAS-1.1) expanders. SAS-1.1 expanders are still useful, typically as leaf expanders with a SAS-2 expander "above" them; and if possible all hosts (initiators) should be connected to the SAS-2 expander. Also SAS-2 expanders run at 6 Gbps and some can also handle 6 Gbps SATA disks and SSDs.

Enclosure in and out ports

Each expander phy has a routing attribute. There are three routing attributes: direct, subtractive and table. Direct routing is the simplest and is all that is required if a single expander interconnects SAS end devices (e.g. SAS initiators (HBAs) and SAS targets (including SAS disks, tapes and SES devices)). Such an interconnect is termed in SAS as a "Service Delivery Subsystem" (SDS) while other transports might term this as a "fabric".

An expander port is typically a set of expander phys (all with the same routing attribute) on the same connector. Using mini SAS 4i and 4x cables, SFF-8087 refers to an internal cable which SFF-8088 refers to an external cable. Both SFF-8087 and SFF-8088 carry four phy circuits, additionally SFF-8087 cables may carry sideband signals (also know as SGPIO) which convey SES information to and from a disk enclosure. For SAS-3, the HD mini connectors are SFF-8643 (internal; optionally with sideband signals) and SFF-8644 (external).

Things become more complex when there are two or more expanders interconnected in the same SDS. In SAS-1 an expander port with the table routing attribute is termed as an enclosure out port. In SAS-2 and later if the expander supports table-to-table attachments then an expander port with the table routing attribute is termed as an enclosure universal port. In all versions of SAS an expander port with the subtractive routing attribute is termed as an enclosure in port.

The enclosure in, out and universal port indications need to be taken into account when interconnecting expanders. The ideal situation is when one of the above cables interconnects two expanders whose ports have the universal indication. An enclosure universal port may be connected to an enclosure in or out port with some routing restrictions. And an enclosure in port may be connected to an enclosure out port. Connecting an enclosure in port to an enclosure in port won't work. Also connecting an enclosure out port to an enclosure out port won't work. Note that the "won't work" cases do not cause any electrical problem (i.e. there is no risk of permanent damage).

Enclosure universal ports were introduced in SAS-2 and some SAS-2 expander products use them exclusively. That reduces this "in-out" level of complexity in large Service Delivery Subsystems. Some SAS-3 expanders still have a few non-universal ports (e.g. in ports using subtractive routing) perhaps for backward compatibility.

Examples

The utilities in the smp_utils package send requests to what the standard terms as a SMP target. A SMP target is not a SCSI device so SCSI commands like INQUIRY cannot be sent to a SMP target. Hence utilities such as those in the sg3_utils package cannot be used on a SMP target.

 A SAS expander will contain one SMP target with the SAS address of the expander. Some SAS host bus adapters (HBAs) have a (hidden) SMP target. Typically SAS end devices such as disk drives (both SAS and SATA) and tape drives do not contain a SMP target.

A disk enclosure may contain zero or more expanders and SCSI Enclosure Services (SES) processors. As seen below SAS-2 expander chips often include a SES device (with a different SAS address) connected to a virtual expander phy. There is more on this subject in the  Expanders and enclosures section below.

This example has two machines called "a" and "b" connected to a SAS-2 expander. Two disks (one SAS the other SATA) are connected to that expander.

Some other examples can be found in the previous version of this page at smp_utils095.html . Amongst other things those older examples show that a hard reset on an expander phy will cause an attached SAS disk to report a Unit Attention ("SCSI bus reset occurred").

Finding expanders

The first problem is to find a Linux device node corresponding to an expander. In Linux kernel 2.6.38 used for this example both LSI MPT and Adaptec SAS Host Bus Adapters (HBAs) work with the sgv4 interface provided by the bsg driver. If a expander is connected to machine "a" then the following command should list it:

root@a: # ls /dev/bsg/ex*
/dev/bsg/expander-6:0

It is often useful to find the SAS address of that expander:

root@a: # cat /sys/class/sas_device/expander-6:0/sas_address
0x5001517e85c3efff

Also the SAS address of HBA (a.k.a. initiator port identifier) will be helpful:

root@a: # lsscsi -Ht
...
[5]    usb-storage   usb: 1-2:1.0
[6]    mptsas        sas:0x500605b00006f260

Now we list the disks that machine "a" can see on that mptsas HBA:

root@a: # lsscsi -s 6
[6:0:0:0]    enclosu Intel    RES2SV240        0600  -               -
[6:0:1:0]    disk    SEAGATE  ST32000444SS     0006  /dev/sdb   2.00TB
[6:0:2:0]    disk    ATA      ST31000528AS     CC38  /dev/sdc   1.00TB

Following is the corresponding information for machine "b" plus an extra call to lsscsi asking for the transport address (i.e. SAS address) of each device on controller 6:

root@b: # ls /dev/bsg/ex*
/dev/bsg/expander-6:1
root@b: # cat /sys/class/sas_device/expander-6:1/sas_address
0x5001517e85c3efff
root@b: # lsscsi -Ht
...
[6]    mpt2sas       sas:0x500605b001d0d3e0
root@b: # lsscsi -s 6
[6:0:1:0]    disk    ATA      ST31000528AS     CC38  /dev/sdb   1.00TB
[6:0:2:0]    disk    SEAGATE  ST32000444SS     0006  /dev/sdc   2.00TB
[6:0:3:0]    enclosu Intel    RES2SV240        0600  -
root@b: # lsscsi -st 6
[6:0:1:0]    disk    sas:0x5001517e85c3efe2          /dev/sdb   1.00TB
[6:0:2:0]    disk    sas:0x5000c500215725bd          /dev/sdc   2.00TB
[6:0:3:0]    enclosu sas:0x5001517e85c3effd          -               -

This confirms that the two machines are connected to the same expander. Both machines can see the 2 TB SAS disk, the 1 TB SATA disk and the enclosure services device.

The following method is used to locate the "expander_id" needed to talk to the aac device driver interface: One needs to have PMC's command line interface copied into the system. Run the command "./arcconf expanderlist <controller#>" Below is a sample:

./arcconf expanderlist 1
Controllers found: 1

Expander List for Controller 1:

Expander# 1
Expander ID                    : 0
Enclosure ID                   : 0
Reported Channel,Device        : 2,0
Vendor ID                      : LSI CORP
Product ID                     : SAS2X36
Product Revision Level         : 0717
SAS Address                    : 500304800115C47F

Command completed successfully.

The "Expander ID" field is the expander_id which is used in the smp request command.

Discover(y)

To get an overview of what is connected to each phy on an expander, the DISCOVER or DISCOVER LIST SMP functions can be used. The corresponding utilities in the smp_utils are called smp_discover and smp_discover_list. The naming of all the utilities in the smp_utils package follows a similar pattern, but some words are shortened (e.g. "rep" instead of "report") so the utility names don't become too long.

Without options the smp_discover utility gives a one line summary for each active phy on the expander. First we check from machine "a":

root@a: # smp_discover /dev/bsg/expander-6:0
  phy   0:T:attached:[500605b00006f264:04  i(SSP+STP+SMP)]  3 Gbps
  phy   1:T:attached:[500605b00006f264:05  i(SSP+STP+SMP)]  3 Gbps
  phy   2:T:attached:[500605b00006f264:06  i(SSP+STP+SMP)]  3 Gbps
  phy   3:T:attached:[500605b00006f264:07  i(SSP+STP+SMP)]  3 Gbps
  phy   5:T:attached:[5001517e85c3efe5:00  t(SATA)]  3 Gbps
  phy   7:T:attached:[5000c500215725bd:00  t(SSP)]  6 Gbps
  phy  20:T:attached:[500605b001d0d3e0:03  i(SSP+STP+SMP)]  6 Gbps
  phy  21:T:attached:[500605b001d0d3e0:02  i(SSP+STP+SMP)]  6 Gbps
  phy  22:T:attached:[500605b001d0d3e0:01  i(SSP+STP+SMP)]  6 Gbps
  phy  23:T:attached:[500605b001d0d3e0:00  i(SSP+STP+SMP)]  6 Gbps
  phy  24:D:attached:[5001517e85c3effd:00  V i(SSP+SMP) t(SSP)]  6 Gbps

Now lets do to the equivalent check from machine "b":

root@b: # smp_discover_list /dev/bsg/expander-6:1
  phy   0:T:attached:[500605b00006f264:04  i(SSP+STP+SMP)]  3 Gbps
  phy   1:T:attached:[500605b00006f264:05  i(SSP+STP+SMP)]  3 Gbps
  phy   2:T:attached:[500605b00006f264:06  i(SSP+STP+SMP)]  3 Gbps
  phy   3:T:attached:[500605b00006f264:07  i(SSP+STP+SMP)]  3 Gbps
  phy   5:T:attached:[5001517e85c3efe5:00  t(SATA)]  3 Gbps
  phy   7:T:attached:[5000c500215725bd:00  t(SSP)]  6 Gbps
  phy  20:T:attached:[500605b001d0d3e0:03  i(SSP+STP+SMP)]  6 Gbps
  phy  21:T:attached:[500605b001d0d3e0:02  i(SSP+STP+SMP)]  6 Gbps
  phy  22:T:attached:[500605b001d0d3e0:01  i(SSP+STP+SMP)]  6 Gbps
  phy  23:T:attached:[500605b001d0d3e0:00  i(SSP+STP+SMP)]  6 Gbps
  phy  24:D:attached:[5001517e85c3effd:00  V i(SSP+SMP) t(SSP)]  6 Gbps

Unsurprisingly both machines see the same report when they ask the common expander what its active connections are. Summarizing those reports: expander phys 0, 1, 2 and 3 are connected to machine "a", expander phys 20, 21, 22 and 23 are connected to machine "b", expander phy 5 is connected to the SATA disk and expander phy 7 is connected to the SAS disk.

Expander phy 24 is flagged as "virtual" which means that it is an internal connection within the expander chip. Attached to expander phy 24 is a relatively uncommon SCSI device, one that is both an initiator and a target. The target part contains the enclosure services device "RES2SV240" listed in the lsscsi output in the previous section. The initiator part (i.e. "i(SSP+SMP)") is SAS-2 expander magic and is used when configuring other (e.g. SAS-1.1) expanders and itself.

SATA disks have a single phy connection while SAS disks can one or two phy connections. In this example both disks have a single phy connection to the expander. Connections between SAS HBAs and expanders are typically with cables that carry 4 phys (e.g. SFF-8087 based cables (a.k.a. mini 4i) and SFF-8088 based external cables (a.k.a. mini 4e)). So machine "a"'s  connection to the expander is spread across 4 phys (phy identifiers 0, 1, 2 and 3 at the expander end and 4, 5, 6 and 7 at the HBA end). The term "wide link" is used to describe this situation.

In smp_discover version 1.63 and smp_discover_list version 1.50, both soon to appear in smp_utils version (release) 1.00, an extra feature has been added to the --cap option when these utilities are in the one line per phy mode. Each attached phy capability field, if available, is checked to see if its speed capability exceeds that of the expander phy. If an excess is found a term like "[att: 22.5 G capable]' is appended to such lines. In the example below a SAS-3 expander is being probed and four of its phys are connected to an initiator (HBA) which is capable of SAS-4 speeds (i.e. 22.5 Gbps). Some SAS-2 expanders don't support phy capability fields (they are always zero). The "short format" descriptor (24 bytes long) returned by the DISCOVER LIST SMP function does not include any phy capability fields , so when smp_discover_list --cap is given, the long format descriptor is chosen unless there is an explicit request for the short format descriptor (e.g. --descriptor=1 ).

# smp_discover -c /dev/bsg/expander-3\:0 
 phy 12:U:attached:[50000d1e0111d680:06 i(SSP+STP+SMP)] 12 Gbps [att: 22.5 G capable] 
 phy 13:U:attached:[50000d1e0111d680:07 i(SSP+STP+SMP)] 12 Gbps [att: 22.5 G capable] 
 phy 14:U:attached:[50000d1e0111d680:05 i(SSP+STP+SMP)] 12 Gbps [att: 22.5 G capable] 
 phy 15:U:attached:[50000d1e0111d680:04 i(SSP+STP+SMP)] 12 Gbps [att: 22.5 G capable] 
 phy 16:U:attached:[500605b006d3b510:03 i(SSP+STP+SMP)] 12 Gbps 
 phy 17:U:attached:[500605b006d3b510:02 i(SSP+STP+SMP)] 12 Gbps 
 phy 18:U:attached:[500605b006d3b510:01 i(SSP+STP+SMP)] 12 Gbps 
 phy 19:U:attached:[500605b006d3b510:00 i(SSP+STP+SMP)] 12 Gbps 
 phy 32:U:attached:[5000cca02b38d0b9:00 t(SSP)] 12 Gbps 
 phy 34:U:attached:[5000c5003011cb29:00 t(SSP)] 12 Gbps 
 phy 36:D:attached:[5001b4d516ecc03d:00 V i(SMP) t(SSP)] 12 Gbps 


By default a smp_discover invocation without options will assume the "--summary" option that outputs one line of information for each active expander phy. More information can be found about a specific phy on an expander including its current state and what it is connected to with a smp_discover invocation with the "--phy=ID" option:


root@a: # smp_discover --phy=7 --brief /dev/bsg/expander-6:0
Discover response (brief):
 expander change count: 61
 phy identifier: 7
 attached device type: end device
 attached reason: power on
 negotiated logical link rate: phy enabled; 6 Gbps
 attached initiator: ssp=0 stp=0 smp=0 sata_host=0
 attached target: ssp=1 stp=0 smp=0 sata_device=0
 SAS address: 0x5001517e85c3efff
 attached SAS address: 0x5000c500215725bd
 attached phy identifier: 0
 routing attribute: table


Without the "--brief" option over 70 lines of information would have been produced. SAS-2 introduced the enhanced DISCOVER LIST SMP function which has some extra features. These include a header with information common to all phys plus the ability to report on multiple expander phys in one response (up to 40). The smp_discover_list utility without options will give the same output as smp_discover. The smp_discover_list utility shows more information (than smp_discover) when the "--phy=ID" option is given:


root@a: # smp_discover_list -p 7 /dev/bsg/expander-6:0
Discover list response header:
 starting phy id: 7
 number of discover list descriptors: 1
 expander change count: 61
 filter: 0
 descriptor type: 0
 discover list descriptor length: 116
 zoning supported: 1
 zoning enabled: 0
 self configuring: 0
 zone configuring: 0
 configuring: 0
 externally configurable route table: 0
 last self-configuration status descriptor index: 0
 last phy event list descriptor index: 0
descriptor 1:
 phy identifier: 7
 attached device type: end device
 attached reason: power on
 negotiated logical link rate: phy enabled; 6 Gbps
 attached initiator: ssp=0 stp=0 smp=0 sata_host=0
 attached sata port selector: 0
 attached target: ssp=1 stp=0 smp=0 sata_device=0
 SAS address: 0x5001517e85c3efff
 attached SAS address: 0x5000c500215725bd
 ...

An example of smp_discover_list request on aac device driver interface is shown below:

[root@yuexiu ~]# smp_discover_list /dev/aac0,0
phy4:T:attached:[50000d1106d15280:03 i(SSP+STP+SMP)] 6 Gbps
phy5:T:attached:[50000d1106d15280:02 i(SSP+STP+SMP)] 6 Gbps
phy6:T:attached:[50000d1106d15280:00 i(SSP+STP+SMP)] 6 Gbps
phy7:T:attached:[50000d1106d15280:01 i(SSP+STP+SMP)] 6 Gbps
phy12:T:attached:[500304800115c44c:00 t(SATA)] 3 Gbps
phy13:T:attached:[500304800115c44d:00 t(SATA)] 3 Gbps
phy14:T:attached:[500304800115c44e:00 t(SATA)] 3 Gbps
phy15:T:attached:[500304800115c44f:00 t(SATA)] 3 Gbps
phy16:T:attached:[500304800115c450:00 t(SATA)] 3 Gbps
phy17:T:attached:[500304800115c451:00 t(SATA)] 3 Gbps
phy18:T:attached:[500304800115c452:00 t(SATA)] 3 Gbps
phy19:T:attached:[500304800115c453:00 t(SATA)] 3 Gbps
phy20:T:attached:[5000c5001cc6678d:00 t(SSP)] 6 Gbps
phy21:T:attached:[500304800115c455:00 t(SATA)] 3 Gbps
phy22:T:attached:[5000c50023b12521:00 t(SSP)] 6 Gbps
phy23:T:attached:[5000c50023b1a98d:00 t(SSP)] 6 Gbps
phy24:T:attached:[5000c50023b11c19:00 t(SSP)] 6 Gbps
phy25:T:attached:[5000c5001cdf399d:00 t(SSP)] 6 Gbps
phy26:T:attached:[5000c5001cdf5a99:00 t(SSP)] 6 Gbps
phy27:T:attached:[5000c5001cde7195:00 t(SSP)] 6 Gbps
phy28:T:attached:[500304800115c45c:00 t(SATA)] 3 Gbps
phy29:T:attached:[5000c5001cdeacf9:00 t(SSP)] 6 Gbps
phy30:T:attached:[5000c5001cdea36d:00 t(SSP)] 6 Gbps
phy31:T:attached:[5000c5001cdd3705:00 t(SSP)] 6 Gbps
phy32:T:attached:[500304800115c460:00 t(SATA)] 3 Gbps
phy33:T:attached:[500304800115c461:00 t(SATA)] 3 Gbps
phy34:T:attached:[500304800115c462:00 t(SATA)] 3 Gbp
phy35:T:attached:[500304800115c463:00 t(SATA)] 3 Gbps
phy36:D:attached:[500304800115c47d:00 V i(SSP+SMP) t(SSP)] 6 Gbps

Like all the utilities in the smp_utils package, smp_discover and smp_discover_list have usage messages (shown when the "--help" option is given) and man pages.

General information

It may be useful to find out who manufactured the expander and what version it is:

root@a: # smp_rep_manufacturer /dev/bsg/expander-6:0
Report manufacturer response:
  Expander change count: 61
  SAS-1.1 format: 1
  vendor identification: Intel  
  product identification: RES2SV240      
  product revision level: 0600
  component vendor identification: LSI    
  component id: 544
  component revision level: 5

In this case the board is manufactured by Intel while the (expander) component is made by LSI. The REPORT GENERAL SMP function contains useful information including how many phys the expander has:

root@a: # smp_rep_general /dev/bsg/expander-6\:0
Report general response:
  expander change count: 65
  expander route indexes: 0
  long response: 1
  number of phys: 26
  table to table supported: 1
  zone configuring: 0
  self configuring: 0
  STP continue AWT: 0
  open reject retry supported: 1
  configures others: 1
  configuring: 0
  externally configurable route table: 0
  enclosure logical identifier (hex): 5001517e85c3efff
  ...
  number of zone groups: 1 (0->128, 1->256)
  ...

This shows the truncated response of the SAS-2 expander. Prior to SAS-2 only the information up to and including the "table to table supported:" line was provided.

An example of smp_report_general request on aac device driver interface is shown below:

[root@yuexiu ~]# smp_rep_general /dev/aac0,0
 Report general response:
 expander change count: 351
 expander route indexes: 0
 long response: 1
 number of phys: 38
 table to table supported: 1
 zone configuring: 0
 self configuring: 0
 STP continue AWT: 0
 open reject retry supported: 1
 configures others: 1
 configuring: 0
 externally configurable route table: 0
 initiates SSP close: 0
 enclosure logical identifier (hex): 500304800115c47f
 ...
 number of zone groups: 1 (0->128, 1->256)
 ...

The above example has the PMC controller connected to a LSI expander.

SAS-2 expanders are self-configuring and will configure other expanders (e.g. SAS-1.1 expanders) that they are connected two. Things can still go wrong, so if some disks have "disappeared" from the fabric, it may be useful to check the REPORT SELF-CONFIGURATION STATUS function:

root@a: # smp_rep_self_conf_stat --brief /dev/bsg/expander-6\:0
Report self-configuration status response:
  Expander change count: 86
  total number of self-configuration status descriptors: 0
  number of self-configuration status descriptors: 0

The fact that there are no self-configuration status descriptors indicates that no errors were detected since power up.

SAS-2.1 adds phy power management. However the expander used in these examples predates that standard. One simple way to save power is to disable expander phys not being used (e.g. not attached to anything). The next example disables phy 10:

root@a: # smp_phy_control --phy=10 --op=di /dev/bsg/expander-6:0

which saves about 150 mW of power. To re-enable a phy use link reset (hard reset or power cycle the expander):

root@a: # smp_phy_control --phy=10 --op=lr /dev/bsg/expander-6:0

Not much power saved but with enough unused phys disabled, it may amount to something.

SATA disk affiliations

The DISCOVER output from machines a and b (shown in the Discover section above) indicates both machines have access to the SATA disk as SAS address 0x5001517e85c3efe5 . The SATA model assumes only one host can access a SATA disk at a time. SAS expanders use a technique called "affiliations" to enforce this restriction. The REPORT PHY SATA functions shows which initiator (host) has the affiliation (i.e. is able to access that SATA disk):

root@a: # smp_rep_phy_sata --phy=5 /dev/bsg/expander-6:0
Report phy SATA response:
  expander change count: 161
  phy identifier: 5
  STP I_T nexus loss occurred: 0
  affiliations supported: 1
  affiliation valid: 1
  STP SAS address: 0x5001517e85c3efe5
  register device to host FIS:
    34 00 50 01 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
  affiliated STP initiator SAS address: 0x500605b001d0d3e0
  STP I_T nexus loss SAS address: 0x0
  affiliation context: 0
  current affiliation contexts: 1
  maximum affiliation contexts: 1

This shows that machine b (0x500605b001d0d3e0) has the affiliation. Machine b can do a "clear affiliation" operation with the PHY CONTROL function:

root@b: # smp_phy_control --phy=5 --op=ca /dev/bsg/expander-6:0

after which the SATA disk is available for machine a to "grab". Machine a could grab that SATA disk by doing an IO operation on that disk, or perhaps more safely with an ATA IDENTIFY DEVICE command (e.g. with the sg_sat_identify utility from sg3_utils). After that smp_rep_phy_sata could be called again:

root@a: # smp_rep_phy_sata --phy=5 /dev/bsg/expander-6:0
Report phy SATA response:
  expander change count: 167
  phy identifier: 5
  ...
  affiliated STP initiator SAS address: 0x500605b00006f264
  ...

So now machine a (0x500605b00006f264) has the SATA affiliation.

Zoning

An overview of SAS-2 zoning is given in this article . [Note that article makes this false statement: "All devices within a zone group can interact with each other by default." That is true only for zone group 1, otherwise, by default, ZP[n,n]=0 which means they can not interact by default.] While zoning could be used with a single initiator (host) to hide one or more disks, it is more likely that zoning will be used when there are two or more initiators (hosts).  The two machine, two disk fabric used in these examples, is the basis for two scripts found in the examples directory of the smp_utils version 0.96 source tarball. The zoning_ex.sh script sets up zoning while the dis_zoning_ex.sh script disables zoning. See the README file in the examples directory. Note the pconf_2i2t.txt file will most likely need to be changed to match the expander phy connections of the system under test.

There is an Annex in the SAS-2 draft (Annex H in SPL-2 rev 1) titled: "Zone permission configuration descriptor examples" which is worth reading. In the examples directory the t10annex_zoning_ex.sh script follows the SAS-2 annex example.

Probably the most difficult thing to visualize with zoning is the permission table. The Zone permission configuration descriptors don't help in this regard because they follow SCSI conventions in being byte oriented and big endian. So the smp_rep_zone_perm_tbl has a  "-bits=COLS" option to show the table as a bit array with its origin (i.e. ZP[0,0]) in the top left:

root@a: # smp_rep_zone_perm_tbl --bits=24 /dev/bsg/expander-6\:0
# Report zone permission table response:
#  Expander change count: 105
#  zone locked: 0
#  report type: 0 [current]
#  number of zone groups: 1 (256)
#  number of zone permission descriptors: 31


Output unsuitable for smp_conf_zone_perm_tbl utility

    012345678901234567890123

0   010000000000000000000000
1   111111111111111111111111
2   010000001000000000000000
3   010000001100000000000000
4   010000000000000000000000
5   010000000000000000000000
6   010000000000000000000000
7   010000000000000000000000
8   011100001000000010000000
9   010100000100000001000000
10  010000000010000000100000
11  010000000001000000010000
12  010000000000100000001000
13  010000000000010000000100
14  010000000000001000000010
15  010000000000000100000001
16  010000001000000000000000
17  010000000100000000000000
18  010000000010000000000000
19  010000000001000000000000
20  010000000000100000000000
21  010000000000010000000000
22  010000000000001000000000
23  010000000000000100000000

The last digit of the column numbers is displayed above the array while the row numbers are shown on the left. The source zone groups are the rows while the destination zone groups are the columns. The requested number of columns (24 in this case) will be displayed. The number of rows may be less than that since only one SMP REPORT ZONE TABLE request is sent (unless the "--multiple" option is given).

Expanders and enclosures

SAS expanders are often found in disk enclosures which are often monitored by SCSI Enclosure Services (SES) devices. SES devices are also known as SES processors or in SCSI terminology SES logical units. SES devices are often integrated in the same silicon as SAS-2 expanders. This is the case with these examples, as can be seen in the lsscsi output in the Finding expanders section above. This line output by the lsscsi command:

[6:0:3:0]    enclosu Intel    RES2SV240        0600  -

shows an integrated SES device. Its SAS address (0x5001517e85c3effd) is very close to the expander's SAS address (0x5001517e85c3efff). The "-g" option needs to be added to lsscsi to see if there is a generic device name since a SES device is not a Linux block device. Here is that output again from "lsscsi -g" (which may need the sg module loaded with "modprobe sg"):

[6:0:3:0]    enclosu Intel    RES2SV240        0600  -   /dev/sg3

It shows that SES device can be accessed via /dev/sg3 on machine b . It can also be accessed via the bsg driver with /dev/bsg/6:0:3:0 . The sg3_utils package contains the sg_ses utility. That sg_ses utility has recently been updated so try to use one from sg3_utils version 1.32 or later.

root@b: # sg_ses -V
version: 1.64 20111220

root@b: # sg_ses /dev/sg3
  Intel     RES2SV240         0600
Supported diagnostic pages:
  Supported diagnostic pages [sdp] [0x0]
  Configuration (SES) [cf] [0x1]
  Enclosure status/control (SES) [ec,es] [0x2]
  String In/Out (SES) [str] [0x4]
  Threshold In/Out (SES) [th] [0x5]
  Element descriptor (SES) [ed] [0x7]
  Additional element status (SES-2) [aes] [0xa]
  Supported SES diagnostic pages (SES-2) [ssp] [0xd]
  Download microcode (SES-2) [dm] [0xe]
  Subenclosure nickname (SES-2) [snic] [0xf]

The element descriptor page is a good place to start:

root@b: # sg_ses -p ed /dev/sg3
  Intel     RES2SV240         0600
  Primary enclosure logical identifier (hex): 5001517e85c3efff
Element descriptor In diagnostic page:
  generation code: 0x0
  element descriptor by type list
    Element type: Array device slot, subenclosure id: 0 [ti=0]
      Overall descriptor: ArrayDevicesInSubEnclsr0
      Element 0 descriptor: ArrayDevice00
....
      Element 9 descriptor: ArrayDevice09
      Element 10 descriptor: ArrayDevice0A
....
      Element 16 descriptor: ArrayDevice10
      Element 17 descriptor: ArrayDevice11
....
      Element 23 descriptor: ArrayDevice17

The output is abridged but shows the naming scheme: the "array device slot" subenclosure (there is only one) is named "ArrayDevicesInSubEnclsr0" while there are 24 "ArrayDevice"s counted in hex starting at 0 (all shown as two hex digits). Due to the close integration between the expander and the SES device, phy 7 on the expander corresponds to slot 7 on the SES device. And that is the 2 TB SAS disk known to Linux as /dev/sdb on machine a and /dev/sdc on machine b:

root@b: # sg_ses -D ArrayDevice07 --join /dev/sg3
  Intel     RES2SV240         0600
  Primary enclosure logical identifier (hex): 5001517e85c3efff
ArrayDevice07 [0,7]  Element type: Array device slot
  Enclosure status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    OK=0, Reserved device=0, Hot spare=0, Cons check=0
    In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
    App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
    Ready to insert=0, RMV=0, Ident=0, Report=0
    App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
    Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
  Additional element status:
    Transport protocol: SAS
    number of phys: 1, not all phys: 0, device slot number: 7
    phy index: 0
      device type: end device
      initiator port for:
      target port for: SSP
      attached SAS address: 0x5001517e85c3efff
      SAS address: 0x5000c500215725bd
      phy identifier: 0x0

Now to find that disk in an array (which has the SGPIO sideband signals connected, for example via a SFF 8087 cable) the following can be done to blink the locate LED (also known as "ident") on the disk carrier:

root@b: # sg_ses -D ArrayDevice07 --set=locate /dev/sg3

The locate LED can be turned off with this command:

root@b: # sg_ses -D ArrayDevice07 --clear=locate /dev/sg3

Note that what is called a "slot" above (actually a 'device slot number' (dsn)) was previously called a "bay" [t10 made the terminology change]. A slot may contain multiple phys but in this case there was only one phy per slot. And at the end of SAS-2 and in SAS-3 the dsn has become an important link between SMP DISCOVER and SES. smp_discover  has a "--dsn" option (introduced in 0.99) which shows, if available (and not 255), "dsn=<dsn>" at the end of each summary line. The <dsn> value can be used to index (search) the enclosure with:

# sg_ses --dev-slot-num=<dsn> /dev/sg3

If the <dsn> is found then above will output the array element status and most likely the corresponding additional element status (with SAS information).  Alternatively the locate (ident) LED on the corresponding enclosure disk holder could be made to flash with the addition of "--set=locate".

Download

Table 2. smp_utils tarballs, rpm + deb packages

The tarball contains an "rpm" spec file which is used to build source rpm and the i386 binary rpm. There is a Debian sub-directory with files to build a "deb" package. Here is the ChangeLog . A COVERAGE file shows a mapping from each SMP function name to the corresponding smp_utils utility name. There are also ".tar.gz", ".tar.bz2"  and ".tar.xz" versions of the tarball at the same location.

There is a github mirror at smp_utils maintained by the author. The author (showing his age) still prefers to use a subversion repository on his own equipment (a Raspberry Pi) for development.

Notice that a library package has been introduced in version 0.97 .

Return to main page.

Last updated: 4th November 2021