The smp_utils package contains utilities for the Serial
Attached SCSI (SAS) Serial Management Protocol (SMP). Most
utilities correspond to a single SMP function, sending out a request,
checking for errors and if all is well processing the response. The
response is either decoded, printed out in ASCII hexadecimal or sent
as binary to stdout.
At the lower levels SAS is a point to
point interconnect (like SATA). SAS has devices called expanders
which have a similar role to switches in the Fibre Channel protocol
and the Ethernet. The primary role of SMP is to monitor and control
SAS expanders. Most SAS Host Bus Adapters (HBAs) contain a SMP
initiator through which SMP requests are sent and responses received.
SAS expanders contain SMP targets that respond to SMP requests. SAS
disks do not normally contains SMP targets and SATA disks do not
contain SMP targets. SAS has two other protocols: SSP for
transporting SCSI command sets to SAS devices (e.g. SAS disks and
tape drives) , and STP for tunnelling the ATA command set to a SAS
expander.
SAS now has multiple generations. The first six
generations are approved standards: the original SAS (SAS ANSI INCITS
376-2003), SAS 1.1 (INCITS 417-2006), SAS-2 (ANSI INCITS 457-2010),
SAS-2.1 (ANSI INCITS 478-2011), SAS-3 (ANSI INCITS 519-2014) and
SAS-4 (ANSI INCITS 534-2019) . In SAS-2.1 the higher level layers
including the SMP protocol where placed in a separate standard known
as the SAS Protocol Layer (SPL) and it now has four standards: SPL
(ANSI INCITS 476-2011), SPL-2 (ANSI INCITS 505-2013), SPL-3 (ANSI
INCITS 492-2015) and SPL-4 (ANSI INCITS 538-2018). The most recent
SPL-5 draft is spl5r09.pdf . To lessen confusion, the six generations
of SAS will be referred to here as 1, 1.1, 2, 2.1, 3.0 and 4.0.
Drafts, including those just prior to standardization, can be found
at the www.t10.org site. The SMP
functions for reading and writing to GPIO registers are defined in
the Small Form Factor document SFF-8485 found at www.sffcommittee.com
.
The smp_utils package targets the Linux kernel
2.6, 3, 4 and 5 series. This page describes smp_utils version 0.99
. Ports to Solaris and FreeBSD (requiring FreeBSD 9.0 or later)
were added in smp_utils version 0.97.
Due to a large
number of changes between smp_utils version 0.96 and older versions,
an older version of this page can be found at smp_utils095.html
. Its examples were written in 2008 and may be useful for older Linux
kernels (e.g. prior to lk 2.6.30).
Here is a list in alphabetical order of the utilities in the smp_utils package:
Table 1. Utilities
Utility |
SMP function |
SAS |
smp_utils |
Notes |
smp_conf_general |
CONFIGURE GENERAL |
2 |
0.92 |
|
smp_conf_phy_event |
CONFIGURE PHY |
2 |
0.97 |
|
smp_conf_route_info |
CONFIGURE ROUTE INFORMATION |
1 |
|
SAS-2 expanders typically don't support this function |
smp_conf_zone_man_pass |
CONFIGURE ZONE MANAGER PASSWORD |
2 |
0.96 |
|
smp_conf_zone_perm_tbl |
CONFIGURE ZONE PERMISSION TABLE |
2 |
0.96 |
all expanders within a ZPSDS should have the same zone permission table |
smp_conf_zone_phy_info |
CONFIGURE ZONE PHY INFORMATION |
2 |
0.96 |
map expander phys to zone groups |
smp_discover |
DISCOVER |
1 |
|
SAS HBA drivers use this command to discover the topology |
smp_discover_list |
DISCOVER LIST |
2 |
0.91 |
more information returned in each response compared to the DISCOVER function |
smp_ena_dis_zoning |
ENABLE DISABLE ZONING |
2 |
0.96 |
|
smp_phy_control |
PHY CONTROL |
1 |
|
amongst other things, can send link and hard resets to attached phy |
smp_phy_test |
PHY TEST FUNCTION |
1.1 |
0.91 |
|
smp_read_gpio |
READ GPIO REGISTER |
1 |
|
defined in SFF-8485. May also be able to read the sideband signals on a HBA's wide internal cables. Can also invoke the "ENHANCED" variant function. |
smp_rep_broadcast |
REPORT BROADCAST |
2 |
0.96 |
|
smp_rep_exp_route_tbl |
REPORT EXPANDER ROUTE TABLE LIST |
2 |
0.93 |
|
smp_rep_general |
REPORT GENERAL |
1 |
|
|
smp_rep_manufacturer |
REPORT MANUFACTURER INFORMATION |
1 |
|
Response contains device identification string somewhat like the response to a SCSI INQUIRY command. |
smp_rep_phy_err_log |
REPORT PHY ERROR LOG |
1 |
|
|
smp_rep_phy_event |
REPORT PHY EVENT |
2 |
0.96 |
|
smp_rep_phy_event_list |
REPORT PHY EVENT LIST |
2 |
0.96 |
|
smp_rep_phy_sata |
REPORT PHY SATA |
1 |
|
for expander phys connected to a SATA device |
smp_rep_self_conf_stat |
REPORT SELF CONFIGURATION STATUS |
2 |
0.96 |
|
smp_rep_route_info |
REPORT ROUTE INFORMATION |
1 |
|
SAS-2 expanders typically don't support this function |
smp_rep_zone_man_pass |
REPORT ZONE MANAGER PASSWORD |
2 |
0.96 |
|
smp_rep_zone_perm_tbl |
REPORT ZONE PERMISSION TABLE |
2 |
0.96 |
can be written to file which can then be used as input to smp_conf_zone_perm_tbl |
smp_write_gpio |
WRITE GPIO REGISTER |
1 |
|
defined in SFF-8485. May also be able to write to the sideband signals on a HBA's wide internal cables. Can also invoke the "ENHANCED" variant function. |
smp_zone_activate |
ZONE ACTIVATE |
2 |
0.96 |
|
smp_zone_lock |
ZONE LOCK |
2 |
0.96 |
required prior to most other zone related functions; may need password |
smp_zone_unlock |
ZONE UNLOCK |
2 |
0.96 |
completes a sequence of zone related functions |
smp_zoned_broadcast |
ZONED BROADCAST |
2 |
0.96 |
|
As can be seen, the utility name is closely related to the
SMP function name. The "SAS version" column indicates the
first SAS version in which the SMP function appeared. For more
information see one of the SAS drafts (i.e. SAS 1, SAS 1.1, SAS 2,
SPL [1], SPL 2, SPL 3, SPL 4 and SPL5), in the "Application
layer" section within which the SMP functions are
described.
Each utility has its own man page. Information
common to all utilities can be found in the smp_utils man page.
SMP pass through interfaces are not yet as well established as
those for the SCSI protocol. SMP is a request response protocol with
each request and response limited currently to 1028 bytes in length
(including a 4 byte CRC at the end of each request and
response).
This package currently supports three
interfaces:
aac: supports Adaptec/PMC RAID adapters (added in version 0.98)
mpt: based on the MPT Fusion /dev/mptctl, /dev/mpt2ctl or /dev/mpt3ctl device
sgv4: (also sg) uses the SCSI generic version 4 interface and the SAS transport layer (added in version 0.93 and requires Linux kernel 2.6.24 or later)
The aac interface uses a "decorated" device name of the
form /dev/aac[N[,E_ID]] where "N" is the RAID controller
identifier number (and is obtained from the unique_id
as discussed below for the mpt interface) and "E_ID" is the
expander identifier for cases where there is more than one expander.
The mpt interface uses a single device node per system. Since there
may be multiple HBAs in a system, an "ioc_num" is used to
identify which HBA is going to send the SMP request. The
"ioc_num" defaults to 0. If another value is needed then a
syntax like this "smp_rep_general
/dev/mptctl,3" can be used. The ioc_num can be found in
the /proc/scsi/mptsas/<host_no>
file in the Linux lk 2.4 series and additionally in
/sys/class/scsi_host/host<n>/unique_id
file in the lk 2.6 and 3 series. The user could also check the log in
the section where the MPT SAS HBA is registered for "ioc"
strings and look at the following number. The SAS address of the
target SMP device (typically a SAS expander) also needs to be given.
The /dev/mptctl device node is
associated with the mptctl module. Hence a "modprobe
mptctl" command may be required. The /dev/mptctl
device node is a character device with major 10 and minor 220 while
/dev/mpt2ctl device node is a character
device with major 10 and minor 221 .
The sgv4 interface
uses the SG_IO ioctl, the bsg driver and the SCSI generic version 4
(hence "sgv4" which can be shortened to "sg")
structure. From Linux kernel 2.6.24 the SAS transport layer within
the SCSI subsystem accepts SMP requests and forwards them on to those
SAS low level (HBA) drivers that accept them. At this time SAS HBAs
based on libsas should support this interface (e.g. those that use
mptsas, mpt2sas and mpt3sas). Note that most SAS RAID controllers do
not use libsas. The SMP target device is identified in one of two
possible ways: via a sysfs directory name (e.g.
/sys/class/bsg/expander-4:0 ) or via a
device node created dynamically by the bsg driver (e.g.
/dev/bsg/expander-4:0 ). The SAS
address of the expander can be found in the /sys/class/sas_device
directory (e.g. /sys/class/sas_device/expander-4:0/sas_address
).
Each utility attempts to work out at run time which
interface it has. While there are two interfaces this is a reasonably
simple process as their characteristics are quite different. To
explicitly set the interface type the '--interface=<intf>[,force]'
option can be given where <intf> is one of "mpt" or
"sgv4" (or "sg"). Checks are also performed that
the given device name isn't a normal file or worse a block device. A
command like "smp_discover /dev/sda"
makes no sense but if allowed to go ahead, would write the SMP
request over the disk's boot sector! Normally these utilities will
reject such an invocation, before doing any damage. If the user feels
that they know better then checks can be overridden with something
like this: "smp_discover
--interface=mpt,force /dev/xyz" .
If the
<smp_device> is not given, then each utility will check if the
SMP_UTILS_DEVICE environment variable exists. If so the contents of
that environment variable are used. If the SMP target SAS address is
not given (i.e. "--sa=<sas_addr>" is not given) and
it is required, then each utility will check if the
SMP_UTILS_SAS_ADDR environment variable exists. If so the contents of
that environment variable are used.
The Solaris port uses the USMP pass-through mechanism which was introduced to Solaris in 2008. Expanders have device names like /dev/smp/expd3 .
Logic to support SMP was added to FreeBSD in late 2010 and is available in FreeBSD 9 and later. The CAM subsystem was extended but it has a problem because it only supports SCSI devices and an expander is not a SCSI device. SAS-2 expanders are often packaged (on the same silicon) with a SCSI Enclosure Services (SES) device. So FreeBSD uses this "twin" SES device to communicate with an expander. So expanders have device names like /dev/ses2 .
The SFF-8485 document (see www.sffcommittee.com
) defines the READ GPIO REGISTER and WRITE GPIO REGISTER functions
for manipulating registers associated with the SGPIO interface. The
sideband signals found in the wide internal cables defined by SAS
(SAS 4i and mini SAS 4i) may carry SGPIO signals. Some SAS HBAs have
wide internal connectors and have virtual SMP target ports (visible
to the HBA driver but not the rest of the attached SAS domain) so
that the driver or user space programs can access or control those
sideband signals. To find out if a virtual SMP target port is present
on a HBA try something like this: "smp_rep_manufacturer
/dev/bsg/sas_host6". Some enclosures have monitoring
logic that present a SES-2 target type device on a SGPIO
interface.
In SAS-1.1 edge expanders and fanout expanders
where defined. In SAS-2 there are just "expanders" with
that name replacing the identifier previously used for "edge
expanders". In SAS-2 the attached device type identifier
previously used for fanout expanders is now marked obsolete. SAS-2
expanders are a lot more capable than SAS-1.1 expanders. For example
SAS-2 expanders can configure themselves and other (e.g. SAS-1.1)
expanders. SAS-1.1 expanders are still useful, typically as leaf
expanders with a SAS-2 expander "above" them; and if
possible all hosts (initiators) should be connected to the SAS-2
expander. Also SAS-2 expanders run at 6 Gbps and some can also handle
6 Gbps SATA disks and SSDs.
Each expander phy has a routing attribute. There are three routing
attributes: direct, subtractive and table. Direct routing is the
simplest and is all that is required if a single expander
interconnects SAS end devices (e.g. SAS initiators (HBAs) and SAS
targets (including SAS disks, tapes and SES devices)). Such an
interconnect is termed in SAS as a "Service Delivery Subsystem"
(SDS) while other transports might term this as a "fabric".
An
expander port is typically a set of expander phys (all with the same
routing attribute) on the same connector. Using mini SAS 4i and 4x
cables, SFF-8087 refers to an internal cable which SFF-8088 refers to
an external cable. Both SFF-8087 and SFF-8088 carry four phy
circuits, additionally SFF-8087 cables may carry sideband signals
(also know as SGPIO) which convey SES information to and from a disk
enclosure. For SAS-3, the HD mini connectors are SFF-8643 (internal;
optionally with sideband signals) and SFF-8644 (external).
Things
become more complex when there are two or more expanders
interconnected in the same SDS. In SAS-1 an expander port with the
table routing attribute is termed as an enclosure out port. In
SAS-2 and later if the expander supports table-to-table attachments
then an expander port with the table routing attribute is termed as
an enclosure universal port. In all versions of SAS an
expander port with the subtractive routing attribute is termed as an
enclosure in port.
The enclosure in, out
and universal port indications need to be taken into account
when interconnecting expanders. The ideal situation is when one of
the above cables interconnects two expanders whose ports have the
universal indication. An enclosure universal port may
be connected to an enclosure in or out port with some
routing restrictions. And an enclosure in port may be
connected to an enclosure out port. Connecting an enclosure in
port to an enclosure in port won't work. Also connecting an
enclosure out port to an enclosure out port won't work.
Note that the "won't work" cases do not cause any
electrical problem (i.e. there is no risk of permanent
damage).
Enclosure universal ports were introduced
in SAS-2 and some SAS-2 expander products use them exclusively. That
reduces this "in-out" level of complexity in large Service
Delivery Subsystems. Some SAS-3 expanders still have a few
non-universal ports (e.g. in ports using subtractive routing)
perhaps for backward compatibility.
The utilities in the smp_utils package send requests to what the
standard terms as a SMP target. A SMP target is not a SCSI device so
SCSI commands like INQUIRY cannot be sent to a SMP target. Hence
utilities such as those in the sg3_utils package cannot be used on a
SMP target.
A SAS expander will contain one SMP
target with the SAS address of the expander. Some SAS host bus
adapters (HBAs) have a (hidden) SMP target. Typically SAS end devices
such as disk drives (both SAS and SATA) and tape drives do not
contain a SMP target.
A disk enclosure may contain zero or
more expanders and SCSI Enclosure Services (SES) processors. As seen
below SAS-2 expander chips often include a SES device (with a
different SAS address) connected to a virtual expander phy. There is
more on this subject in the Expanders
and enclosures section below.
This example has two
machines called "a" and "b" connected to a SAS-2
expander. Two disks (one SAS the other SATA) are connected to that
expander.
Some other examples can be found in the previous
version of this page at smp_utils095.html
. Amongst other things those older examples show that a hard reset on
an expander phy will cause an attached SAS disk to report a Unit
Attention ("SCSI bus reset occurred").
The first problem is to find a Linux device node corresponding to
an expander. In Linux kernel 2.6.38 used for this example both LSI
MPT and Adaptec SAS Host Bus Adapters (HBAs) work with the sgv4
interface provided by the bsg driver. If a expander is connected to
machine "a" then the following command should list
it:
root@a: # ls
/dev/bsg/ex*
/dev/bsg/expander-6:0
It
is often useful to find the SAS address of that expander:
root@a:
# cat /sys/class/sas_device/expander-6:0/sas_address
0x5001517e85c3efff
Also
the SAS address of HBA (a.k.a. initiator port identifier) will be
helpful:
root@a: # lsscsi
-Ht
...
[5]
usb-storage usb: 1-2:1.0
[6]
mptsas
sas:0x500605b00006f260
Now we list the disks that
machine "a" can see on that mptsas HBA:
root@a:
# lsscsi -s 6
[6:0:0:0]
enclosu Intel RES2SV240
0600 -
-
[6:0:1:0]
disk SEAGATE ST32000444SS
0006 /dev/sdb 2.00TB
[6:0:2:0]
disk ATA
ST31000528AS CC38 /dev/sdc
1.00TB
Following is the corresponding information
for machine "b" plus an extra call to lsscsi asking for the
transport address (i.e. SAS address) of each device on controller
6:
root@b: # ls
/dev/bsg/ex*
/dev/bsg/expander-6:1
root@b:
# cat /sys/class/sas_device/expander-6:1/sas_address
0x5001517e85c3efff
root@b:
# lsscsi -Ht
...
[6]
mpt2sas
sas:0x500605b001d0d3e0
root@b: #
lsscsi -s 6
[6:0:1:0]
disk ATA
ST31000528AS CC38 /dev/sdb
1.00TB
[6:0:2:0]
disk SEAGATE ST32000444SS
0006 /dev/sdc 2.00TB
[6:0:3:0]
enclosu Intel RES2SV240
0600 -
root@b: # lsscsi -st
6
[6:0:1:0]
disk sas:0x5001517e85c3efe2
/dev/sdb 1.00TB
[6:0:2:0]
disk sas:0x5000c500215725bd
/dev/sdc 2.00TB
[6:0:3:0]
enclosu sas:0x5001517e85c3effd
-
-
This confirms that the two machines are connected
to the same expander. Both machines can see the 2 TB SAS disk, the 1
TB SATA disk and the enclosure services device.
The
following method is used to locate the "expander_id" needed
to talk to the aac device driver interface: One needs to have PMC's
command line interface copied into the system. Run the command
"./arcconf expanderlist <controller#>"
Below is a sample:
./arcconf
expanderlist 1
Controllers found: 1
Expander List for
Controller 1:
Expander# 1
Expander
ID : 0
Enclosure
ID : 0
Reported
Channel,Device : 2,0
Vendor
ID : LSI
CORP
Product ID : SAS2X36
Product
Revision Level : 0717
SAS
Address : 500304800115C47F
Command
completed successfully.
The "Expander ID"
field is the expander_id which is used in the smp request command.
To get an overview of what is connected to each phy on an
expander, the DISCOVER or DISCOVER LIST SMP functions can be used.
The corresponding utilities in the smp_utils are called smp_discover
and smp_discover_list. The naming of all the utilities in the
smp_utils package follows a similar pattern, but some words are
shortened (e.g. "rep" instead of "report") so the
utility names don't become too long.
Without options the
smp_discover utility gives a one line summary for each active phy on
the expander. First we check from machine "a":
root@a:
# smp_discover /dev/bsg/expander-6:0
phy
0:T:attached:[500605b00006f264:04 i(SSP+STP+SMP)] 3
Gbps
phy
1:T:attached:[500605b00006f264:05 i(SSP+STP+SMP)] 3
Gbps
phy
2:T:attached:[500605b00006f264:06 i(SSP+STP+SMP)] 3
Gbps
phy
3:T:attached:[500605b00006f264:07 i(SSP+STP+SMP)] 3
Gbps
phy
5:T:attached:[5001517e85c3efe5:00 t(SATA)] 3 Gbps
phy
7:T:attached:[5000c500215725bd:00 t(SSP)] 6 Gbps
phy 20:T:attached:[500605b001d0d3e0:03
i(SSP+STP+SMP)] 6 Gbps
phy
21:T:attached:[500605b001d0d3e0:02 i(SSP+STP+SMP)] 6
Gbps
phy
22:T:attached:[500605b001d0d3e0:01 i(SSP+STP+SMP)] 6
Gbps
phy
23:T:attached:[500605b001d0d3e0:00 i(SSP+STP+SMP)] 6
Gbps
phy
24:D:attached:[5001517e85c3effd:00 V i(SSP+SMP) t(SSP)] 6
Gbps
Now lets do to the equivalent check from
machine "b":
root@b: #
smp_discover_list /dev/bsg/expander-6:1
phy
0:T:attached:[500605b00006f264:04 i(SSP+STP+SMP)] 3
Gbps
phy 1:T:attached:[500605b00006f264:05
i(SSP+STP+SMP)] 3 Gbps
phy
2:T:attached:[500605b00006f264:06 i(SSP+STP+SMP)] 3
Gbps
phy 3:T:attached:[500605b00006f264:07
i(SSP+STP+SMP)] 3 Gbps
phy
5:T:attached:[5001517e85c3efe5:00 t(SATA)] 3 Gbps
phy 7:T:attached:[5000c500215725bd:00 t(SSP)]
6 Gbps
phy 20:T:attached:[500605b001d0d3e0:03
i(SSP+STP+SMP)] 6 Gbps
phy
21:T:attached:[500605b001d0d3e0:02 i(SSP+STP+SMP)] 6
Gbps
phy 22:T:attached:[500605b001d0d3e0:01
i(SSP+STP+SMP)] 6 Gbps
phy
23:T:attached:[500605b001d0d3e0:00 i(SSP+STP+SMP)] 6
Gbps
phy 24:D:attached:[5001517e85c3effd:00 V
i(SSP+SMP) t(SSP)] 6 Gbps
Unsurprisingly both
machines see the same report when they ask the common expander what
its active connections are. Summarizing those reports: expander phys
0, 1, 2 and 3 are connected to machine "a", expander phys
20, 21, 22 and 23 are connected to machine "b", expander
phy 5 is connected to the SATA disk and expander phy 7 is connected
to the SAS disk.
Expander phy 24 is flagged as "virtual"
which means that it is an internal connection within the expander
chip. Attached to expander phy 24 is a relatively uncommon SCSI
device, one that is both an initiator and a target. The target part
contains the enclosure services device "RES2SV240"
listed in the lsscsi output in the previous section. The
initiator part (i.e. "i(SSP+SMP)") is SAS-2 expander magic
and is used when configuring other (e.g. SAS-1.1) expanders and
itself.
SATA disks have a single phy connection while SAS
disks can one or two phy connections. In this example both disks have
a single phy connection to the expander. Connections between SAS HBAs
and expanders are typically with cables that carry 4 phys (e.g.
SFF-8087 based cables (a.k.a. mini 4i) and SFF-8088 based external
cables (a.k.a. mini 4e)). So machine "a"'s connection
to the expander is spread across 4 phys (phy identifiers 0, 1, 2 and
3 at the expander end and 4, 5, 6 and 7 at the HBA end). The term
"wide link" is used to describe this situation.
By
default a smp_discover invocation without options will assume the
"--summary" option that outputs one line of information for
each active expander phy. More information can be found about a
specific phy on an expander including its current state and what it
is connected to with a smp_discover invocation with the "--phy=ID"
option:
root@a: # smp_discover
--phy=7 --brief /dev/bsg/expander-6:0
Discover
response (brief):
expander
change count: 61
phy
identifier: 7
attached
device type: end device
attached
reason: power on
negotiated
logical link rate: phy enabled; 6 Gbps
attached
initiator: ssp=0 stp=0 smp=0 sata_host=0
attached
target: ssp=1 stp=0 smp=0 sata_device=0
SAS
address: 0x5001517e85c3efff
attached
SAS address: 0x5000c500215725bd
attached
phy identifier: 0
routing
attribute: table
Without the "--brief"
option over 70 lines of information would have been produced. SAS-2
introduced the enhanced DISCOVER LIST SMP function which has some
extra features. These include a header with information common to all
phys plus the ability to report on multiple expander phys in one
response (up to 40). The smp_discover_list utility without options
will give the same output as smp_discover. The smp_discover_list
utility shows more information (than smp_discover) when the
"--phy=ID" option is given:
root@a:
# smp_discover_list -p 7 /dev/bsg/expander-6:0
Discover
list response header:
starting
phy id: 7
number of
discover list descriptors: 1
expander
change count: 61
filter:
0
descriptor type: 0
discover list descriptor length: 116
zoning supported: 1
zoning
enabled: 0
self
configuring: 0
zone
configuring: 0
configuring:
0
externally configurable
route table: 0
last
self-configuration status descriptor index: 0
last
phy event list descriptor index: 0
descriptor
1:
phy identifier: 7
attached device type: end device
attached reason: power on
negotiated logical link rate: phy enabled; 6
Gbps
attached initiator:
ssp=0 stp=0 smp=0 sata_host=0
attached
sata port selector: 0
attached
target: ssp=1 stp=0 smp=0 sata_device=0
SAS
address: 0x5001517e85c3efff
attached
SAS address: 0x5000c500215725bd
...
An
example of smp_discover_list request on aac device driver interface
is shown below:
[root@yuexiu ~]#
smp_discover_list
/dev/aac0,0
phy 4:T:attached:[50000d1106d15280:03
i(SSP+STP+SMP)] 6 Gbps
phy 5:T:attached:[50000d1106d15280:02
i(SSP+STP+SMP)] 6 Gbps
phy 6:T:attached:[50000d1106d15280:00
i(SSP+STP+SMP)] 6 Gbps
phy 7:T:attached:[50000d1106d15280:01
i(SSP+STP+SMP)] 6 Gbps
phy 12:T:attached:[500304800115c44c:00
t(SATA)] 3 Gbps
phy 13:T:attached:[500304800115c44d:00
t(SATA)] 3 Gbps
phy 14:T:attached:[500304800115c44e:00
t(SATA)] 3 Gbps
phy 15:T:attached:[500304800115c44f:00
t(SATA)] 3 Gbps
phy 16:T:attached:[500304800115c450:00
t(SATA)] 3 Gbps
phy 17:T:attached:[500304800115c451:00
t(SATA)] 3 Gbps
phy 18:T:attached:[500304800115c452:00
t(SATA)] 3 Gbps
phy 19:T:attached:[500304800115c453:00
t(SATA)] 3 Gbps
phy 20:T:attached:[5000c5001cc6678d:00
t(SSP)] 6 Gbps
phy 21:T:attached:[500304800115c455:00
t(SATA)] 3 Gbps
phy 22:T:attached:[5000c50023b12521:00
t(SSP)] 6 Gbps
phy 23:T:attached:[5000c50023b1a98d:00
t(SSP)] 6 Gbps
phy 24:T:attached:[5000c50023b11c19:00
t(SSP)] 6 Gbps
phy 25:T:attached:[5000c5001cdf399d:00
t(SSP)] 6 Gbps
phy 26:T:attached:[5000c5001cdf5a99:00
t(SSP)] 6 Gbps
phy 27:T:attached:[5000c5001cde7195:00
t(SSP)] 6 Gbps
phy 28:T:attached:[500304800115c45c:00
t(SATA)] 3 Gbps
phy 29:T:attached:[5000c5001cdeacf9:00
t(SSP)] 6 Gbps
phy 30:T:attached:[5000c5001cdea36d:00
t(SSP)] 6 Gbps
phy 31:T:attached:[5000c5001cdd3705:00
t(SSP)] 6 Gbps
phy 32:T:attached:[500304800115c460:00
t(SATA)] 3 Gbps
phy 33:T:attached:[500304800115c461:00
t(SATA)] 3 Gbps
phy 34:T:attached:[500304800115c462:00
t(SATA)] 3 Gbps
phy 35:T:attached:[500304800115c463:00
t(SATA)] 3 Gbps
phy 36:D:attached:[500304800115c47d:00
V i(SSP+SMP) t(SSP)] 6 Gbps
Like all the utilities
in the smp_utils package, smp_discover and smp_discover_list have
usage messages (shown when the "--help" option is given)
and man pages.
It may be useful to find out who manufactured the expander and
what version it is:
root@a: #
smp_rep_manufacturer /dev/bsg/expander-6:0
Report
manufacturer response:
Expander
change count: 61
SAS-1.1
format: 1
vendor
identification: Intel
product
identification: RES2SV240
product revision level: 0600
component vendor identification: LSI
component id: 544
component revision level: 5
In
this case the board is manufactured by Intel while the (expander)
component is made by LSI. The REPORT GENERAL SMP function contains
useful information including how many phys the expander has:
root@a:
# smp_rep_general /dev/bsg/expander-6\:0
Report
general response:
expander
change count: 65
expander
route indexes: 0
long
response: 1
number of phys:
26
table to table
supported: 1
zone
configuring: 0
self
configuring: 0
STP continue
AWT: 0
open reject retry
supported: 1
configures
others: 1
configuring: 0
externally configurable route table: 0
enclosure logical identifier (hex):
5001517e85c3efff
...
number of zone groups: 1 (0->128, 1->256)
...
This shows the truncated
response of the SAS-2 expander. Prior to SAS-2 only the information
up to and including the "table to table supported:" line
was provided.
An example of smp_report_general request on
aac device driver interface is shown below:
[root@yuexiu
~]# smp_rep_general /dev/aac0,0
Report general
response:
expander change count: 351
expander
route indexes: 0
long response: 1
number of
phys: 38
table to table supported: 1
zone
configuring: 0
self configuring: 0
STP continue
AWT: 0
open reject retry supported: 1
configures
others: 1
configuring: 0
externally
configurable route table: 0
initiates SSP close:
0
enclosure logical identifier (hex):
500304800115c47f
...
number of zone groups: 1
(0->128, 1->256)
...
The above
example has the PMC controller connected to a LSI expander.
SAS-2
expanders are self-configuring and will configure other expanders
(e.g. SAS-1.1 expanders) that they are connected two. Things can
still go wrong, so if some disks have "disappeared" from
the fabric, it may be useful to check the REPORT SELF-CONFIGURATION
STATUS function:
root@a: #
smp_rep_self_conf_stat --brief /dev/bsg/expander-6\:0
Report
self-configuration status response:
Expander
change count: 86
total
number of self-configuration status descriptors: 0
number of self-configuration status
descriptors: 0
The fact that there are no
self-configuration status descriptors indicates that no errors were
detected since power up.
SAS-2.1 adds phy power
management. However the expander used in these examples predates that
standard. One simple way to save power is to disable expander phys
not being used (e.g. not attached to anything). The next example
disables phy 10:
root@a: #
smp_phy_control --phy=10 --op=di /dev/bsg/expander-6:0
which
saves about 150 mW of power. To re-enable a phy use link reset (hard
reset or power cycle the expander):
root@a:
# smp_phy_control --phy=10 --op=lr /dev/bsg/expander-6:0
Not
much power saved but with enough unused phys disabled, it may amount
to something.
The DISCOVER output from machines a and b (shown in the Discover
section above) indicates both machines have access to the SATA disk
as SAS address 0x5001517e85c3efe5 . The
SATA model assumes only one host can access a SATA disk at a time.
SAS expanders use a technique called "affiliations" to
enforce this restriction. The REPORT PHY SATA functions shows which
initiator (host) has the affiliation (i.e. is able to access that
SATA disk):
root@a: #
smp_rep_phy_sata --phy=5 /dev/bsg/expander-6:0
Report
phy SATA response:
expander
change count: 161
phy
identifier: 5
STP I_T nexus
loss occurred: 0
affiliations
supported: 1
affiliation
valid: 1
STP SAS address:
0x5001517e85c3efe5
register
device to host FIS:
34
00 50 01 01 00 00 00 00 00 00 00 01 00 00 00 00 00 00 00
affiliated STP initiator SAS address:
0x500605b001d0d3e0
STP I_T
nexus loss SAS address: 0x0
affiliation
context: 0
current
affiliation contexts: 1
maximum
affiliation contexts: 1
This shows that machine b
(0x500605b001d0d3e0) has the affiliation. Machine b can do a "clear
affiliation" operation with the PHY CONTROL function:
root@b:
# smp_phy_control --phy=5 --op=ca /dev/bsg/expander-6:0
after
which the SATA disk is available for machine a to "grab".
Machine a could grab that SATA disk by doing an IO operation on that
disk, or perhaps more safely with an ATA IDENTIFY DEVICE command
(e.g. with the sg_sat_identify utility from sg3_utils). After that
smp_rep_phy_sata could be called again:
root@a:
# smp_rep_phy_sata --phy=5 /dev/bsg/expander-6:0
Report
phy SATA response:
expander
change count: 167
phy
identifier: 5
...
affiliated STP initiator SAS address:
0x500605b00006f264
...
So
now machine a (0x500605b00006f264) has the SATA affiliation.
An overview of SAS-2 zoning is given in this article
. [Note that article makes this false statement: "All devices
within a zone group can interact with each other by default."
That is true only for zone group 1, otherwise, by default, ZP[n,n]=0
which means they can not interact by default.] While zoning could be
used with a single initiator (host) to hide one or more disks, it is
more likely that zoning will be used when there are two or more
initiators (hosts). The two machine, two disk fabric used in
these examples, is the basis for two scripts found in the examples
directory of the smp_utils version 0.96 source tarball. The
zoning_ex.sh script sets up zoning while the dis_zoning_ex.sh script
disables zoning. See the README file in the examples directory. Note
the pconf_2i2t.txt file will most likely need to be changed to match
the expander phy connections of the system under test.
There
is an Annex in the SAS-2 draft (Annex H in SPL-2 rev 1) titled: "Zone
permission configuration descriptor examples" which is worth
reading. In the examples directory the t10annex_zoning_ex.sh script
follows the SAS-2 annex example.
Probably the most
difficult thing to visualize with zoning is the permission table. The
Zone permission configuration descriptors don't help in this regard
because they follow SCSI conventions in being byte oriented and big
endian. So the smp_rep_zone_perm_tbl has a "-bits=COLS"
option to show the table as a bit array with its origin (i.e.
ZP[0,0]) in the top left:
root@a: #
smp_rep_zone_perm_tbl --bits=24 /dev/bsg/expander-6\:0
#
Report zone permission table response:
#
Expander change count: 105
#
zone locked: 0
# report
type: 0 [current]
# number
of zone groups: 1 (256)
#
number of zone permission descriptors: 31
Output
unsuitable for smp_conf_zone_perm_tbl utility
012345678901234567890123
0
010000000000000000000000
1
111111111111111111111111
2
010000001000000000000000
3
010000001100000000000000
4
010000000000000000000000
5
010000000000000000000000
6
010000000000000000000000
7
010000000000000000000000
8
011100001000000010000000
9
010100000100000001000000
10
010000000010000000100000
11
010000000001000000010000
12
010000000000100000001000
13
010000000000010000000100
14
010000000000001000000010
15
010000000000000100000001
16
010000001000000000000000
17
010000000100000000000000
18
010000000010000000000000
19
010000000001000000000000
20
010000000000100000000000
21
010000000000010000000000
22
010000000000001000000000
23
010000000000000100000000
The last digit of the
column numbers is displayed above the array while the row numbers are
shown on the left. The source zone groups are the rows while the
destination zone groups are the columns. The requested number of
columns (24 in this case) will be displayed. The number of rows may
be less than that since only one SMP REPORT ZONE TABLE request is
sent (unless the "--multiple" option is given).
SAS expanders are often found in disk enclosures which are often
monitored by SCSI Enclosure Services (SES) devices. SES devices are
also known as SES processors or in SCSI terminology SES logical
units. SES devices are often integrated in the same silicon as SAS-2
expanders. This is the case with these examples, as can be seen in
the lsscsi output in the Finding expanders
section above. This line output by the lsscsi command:
[6:0:3:0]
enclosu Intel RES2SV240
0600 -
shows an integrated SES device. Its
SAS address (0x5001517e85c3effd) is
very close to the expander's SAS address (0x5001517e85c3efff).
The "-g" option needs to be added to lsscsi to see if there
is a generic device name since a SES device is not a Linux block
device. Here is that output again from "lsscsi -g" (which
may need the sg module loaded with "modprobe sg"):
[6:0:3:0]
enclosu Intel RES2SV240
0600 - /dev/sg3
It shows that SES
device can be accessed via /dev/sg3 on machine b . It can also be
accessed via the bsg driver with /dev/bsg/6:0:3:0 . The sg3_utils
package contains the sg_ses utility. That sg_ses utility has recently
been updated so try to use one from sg3_utils version 1.32 or
later.
root@b: # sg_ses -V
version:
1.64 20111220
root@b: # sg_ses /dev/sg3
Intel RES2SV240
0600
Supported diagnostic pages:
Supported
diagnostic pages [sdp] [0x0]
Configuration (SES) [cf]
[0x1]
Enclosure status/control (SES) [ec,es] [0x2]
String In/Out (SES) [str] [0x4]
Threshold In/Out (SES)
[th] [0x5]
Element descriptor (SES) [ed] [0x7]
Additional element status (SES-2) [aes] [0xa]
Supported
SES diagnostic pages (SES-2) [ssp] [0xd]
Download
microcode (SES-2) [dm] [0xe]
Subenclosure nickname
(SES-2) [snic] [0xf]
The element descriptor page is
a good place to start:
root@b: #
sg_ses -p ed /dev/sg3
Intel
RES2SV240 0600
Primary enclosure logical identifier (hex): 5001517e85c3efff
Element
descriptor In diagnostic page:
generation code: 0x0
element descriptor by type list
Element type:
Array device slot, subenclosure id: 0 [ti=0]
Overall descriptor: ArrayDevicesInSubEnclsr0
Element 0 descriptor: ArrayDevice00
....
Element 9 descriptor: ArrayDevice09
Element 10 descriptor: ArrayDevice0A
....
Element 16 descriptor: ArrayDevice10
Element 17 descriptor: ArrayDevice11
....
Element 23 descriptor: ArrayDevice17
The output is
abridged but shows the naming scheme: the "array device slot"
subenclosure (there is only one) is named "ArrayDevicesInSubEnclsr0"
while there are 24 "ArrayDevice"s counted in hex starting
at 0 (all shown as two hex digits). Due to the close integration
between the expander and the SES device, phy 7 on the expander
corresponds to slot 7 on the SES device. And that is the 2 TB SAS
disk known to Linux as /dev/sdb on machine a and /dev/sdc on machine
b:
root@b: # sg_ses -D
ArrayDevice07 --join /dev/sg3
Intel
RES2SV240 0600
Primary enclosure logical identifier (hex):
5001517e85c3efff
ArrayDevice07 [0,7] Element type: Array
device slot
Enclosure status:
Predicted failure=0, Disabled=0, Swap=0, status: OK
OK=0, Reserved device=0, Hot spare=0, Cons check=0
In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass
B=0
Ready to insert=0, RMV=0, Ident=0,
Report=0
App client bypass B=0, Fault
sensed=0, Fault reqstd=0, Device off=0
Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
Additional element status:
Transport
protocol: SAS
number of phys: 1, not all
phys: 0, device slot number: 7
phy index:
0
device type: end device
initiator port for:
target port
for: SSP
attached SAS address:
0x5001517e85c3efff
SAS address:
0x5000c500215725bd
phy
identifier: 0x0
Now to find that disk in an array
(which has the SGPIO sideband signals connected, for example via a
SFF 8087 cable) the following can be done to blink the locate LED
(also known as "ident") on the disk carrier:
root@b:
# sg_ses -D ArrayDevice07 --set=locate /dev/sg3
The
locate LED can be turned off with this command:
root@b:
# sg_ses -D ArrayDevice07 --clear=locate /dev/sg3
Note
that what is called a "slot" above (actually a 'device slot
number' (dsn)) was previously called a "bay" [t10 made the
terminology change]. A slot may contain multiple phys but in this
case there was only one phy per slot. And at the end of SAS-2 and in
SAS-3 the dsn has become an important link between SMP DISCOVER and
SES. smp_discover has a "--dsn"
option (introduced in 0.99) which shows, if available (and not 255),
"dsn=<dsn>" at the end of
each summary line. The <dsn> value can be used to index
(search) the enclosure with:
#
sg_ses --dev-slot-num=<dsn> /dev/sg3
If the
<dsn> is found then above will output the array element status
and most likely the corresponding additional element status (with SAS
information). Alternatively the locate (ident) LED on the
corresponding enclosure disk holder could be made to flash with the
addition of "--set=locate".
Table 2. smp_utils tarballs, rpm + deb packages
version |
tarball |
source rpm |
rpm binary |
Debian package |
0.90 |
||||
0.91 |
||||
0.92 |
||||
0.93 |
||||
0.94 |
||||
0.95 |
||||
0.96 |
||||
0.97 |
smp-utils_0.97-0.1_i386.deb |
|||
0.98 |
smp_utils-0.98-1.i386.rpm |
smp-utils_0.98-0.1_i386.deb |
||
0.99 |
smp_utils-0.99-1.x86_64.rpm |
smp-utils_0.99-0.1_amd64.deb |
The tarball contains an "rpm" spec file which is
used to build source rpm and the i386 binary rpm. There is a Debian
sub-directory with files to build a "deb" package. Here is
the ChangeLog . A COVERAGE
file shows a mapping from each SMP function name to the corresponding
smp_utils utility name. There are also ".tar.gz",
".tar.bz2" and ".tar.xz" versions of the
tarball at the same location.
There is a github mirror at smp_utils
maintained by the author. The author (showing his age) still prefers
to use a subversion repository on his own equipment (a Raspberry Pi)
for development.
Notice that a library package has been
introduced in version 0.97 .
Return to main page.
Last updated: 20th March 2020