The sg_ses utility

1. The sg_ses utility

   1. Introduction

   2. Overview

   3. Command line options

   4. Indexing

   5. Descriptor Names, Device Slot Numbers and SAS Addresses

   6. clear, get and set

   7. Record once, analyse often

   8. Raw page writes

   9. sg_ses and NVMe

   10. Conclusion

Introduction

The sg_ses utility enables a user "to manage and sense the state of the power supplies, cooling devices, displays, indicators, individual drives, and other non-SCSI elements installed in an enclosure". The SCSI Enclosure Services standards (most recent is SES-3  T10/BSR INCITS 518-2018) and the latest draft (ses4r05.pdf at www.t10.org) describe the format that the sg_ses utility expects to find in a SES device ("logical unit" or "process").

The sg_ses utility is found in the sg3_utils package. The utility was originally written for Linux (including Android) and has ports to FreeBSD, Solaris and Windows. This page outlines the features of the sg_ses utility version 2.48 found in the sg3_utils version 1.46 package.

The sg_ses_microcode utility was added in the sg3_utils release 1.40 . It is designed to download microcode (firmware) to an enclosure and report any errors returned by the enclosure.

Overview

Typically the first problem to solve is finding an Operating System handle for a SES device. SCSI disks are typically easier to find but rarely is a SCSI disk also a SES device (only if the EncServ bit is set in the disk's INQUIRY command response). First let us try the sg_ses utility on a SAS disk (in Linux):

$ sg_ses /dev/sdb
  SEAGATE   ST33000650SS      0002
    disk device (not an enclosure)
Supported diagnostic pages:
  Supported diagnostic pages [sdp] [0x0]
  Protocol specific (SAS transport) [] [0x3f]
  Translate address (SBC) [] [0x40]

Without any options sg_ses reports the names of the supported diagnostic pages. The SAS disk does support some pages but none that belong to the SES standard. On Linux it helps to list all SCSI devices with the lsscsi utility (including showing the generic device names with the '-g' option; 'modprobe sg' may be needed if hyphens appear in the last column):

$ lsscsi -g
[1:0:0:0]    disk    ATA      ST3320620AS      3.AA  /dev/sda   /dev/sg0
[6:0:0:0]    disk    SEAGATE  ST33000650SS     0002  /dev/sdb   /dev/sg1
[6:0:1:0]    disk    SEAGATE  ST32000444SS     0006  /dev/sdc   /dev/sg2
[6:0:2:0]    enclosu Intel    RES2SV240        0600  -          /dev/sg3

The last item is an enclosure and notice that it does not have a disk device node (i.e. there is no /dev/sdd). In Linux the bsg driver could also be used and its device node would be /dev/bsg/6:0:2:0 for the enclosure. Now let us try sg_ses on /dev/sg3 :

$ sg_ses /dev/sg3
  Intel     RES2SV240         0600
Supported diagnostic pages:
  Supported diagnostic pages [sdp] [0x0]
  Configuration (SES) [cf] [0x1]
  Enclosure status/control (SES) [ec,es] [0x2]
  String In/Out (SES) [str] [0x4]
  Threshold In/Out (SES) [th] [0x5]
  Element descriptor (SES) [ed] [0x7]
  Additional element status (SES-2) [aes] [0xa]
  Supported SES diagnostic pages (SES-2) [ssp] [0xd]
  Download microcode (SES-2) [dm] [0xe]
  Subenclosure nickname (SES-2) [snic] [0xf]

So now we see several diagnostic pages related to the SES (and SES-2) standard. Of these the most important are the Configuration page and the Enclosure Status/Control page. All the pages referred to below are actually diagnostic pages, so the term diagnostic will be dropped in many case for brevity.

Let us assume that a 6-disk SAS/SATA disk array is associated with this SES device. We might suspect the disk on the left of the array is "slot 0" but would like to check before removing the disk. One way to double check is to ask "slot 0" to identify itself by flashing its LED for 10 seconds. This can be done like this:

$ sg_ses --dev-slot-num=0 --set=ident /dev/sg3
$ sleep 10
$ sg_ses --dsn=0 --clear=ident /dev/sg3

In the latter invocation '--dsn=0' is a synonym for '--dev-slot-num=0'. Now check the disk on the right (slot 5), this time using shorter option variants ('-x 5' is equivalent to '--dev-slot-num=5'):

$ sg_ses -x 5 -S ident /dev/sg3
$ sleep 10
$ sg_ses -x 5 -C ident /dev/sg3

Typically there is a lot of information held by a SES device divided into pages. Several of those pages carry information with a somewhat similar structure to relational database tables. To see a lot of information try these commands (one after the other):

$ sg_ses --join /dev/sg3
$ sg_ses --join --filter /dev/sg3
$ sg_ses -j -ff /dev/sg3

The first command will list information on all disks, fans, power supplies etc, even for disks and fans not installed. The second command attempts to filter out some less relevant information. The third command tries harder to filter out some less relevant information.

Command line options

sg_ses is a command line utility whose syntax varies depending on its role. To fetch and then decode (or output in hex or binary) all pages, a single page or a field within a page:

   sg_ses [--all] [--descriptor=DES] [--dev-slot-num=SN] [--eiioe=A_F] [--filter] [--get=STR] [--hex] [--index=IIA | =TIA,II] [--inner-hex] [--join] [--maxlen=LEN] [--page=PG] [quiet] [--raw] [--readonly] [--sas-addr=SA] [--status] [--verbose] [--warn] DEVICE

To modify an SES page or a field within it:

  sg_ses [--byte1=B1] [--clear=STR] [--control] [--data=H,H...] [--descriptor=DES] [--dev-slot-num=SN] [--index=IIA | =TIA,II] [--mask] [--maxlen=LEN] [--nickname=SEN] [--nickid=SEID] [--page=PG] [--sas-addr=SA] [--set=STR] [--verbose] DEVICE

If neither --clear nor --set options are given then the --control option must be given to modify the state of the enclosure.

There is also the mode that fetches diagnostic page data from a file rather than a DEVICE:

  sg_ses --data=@FN --status [--raw --raw] <all options from the first form above>

And that leaves some housekeeping variants of which the most interesting is --enumerate which dumps tables held by this utility. These housekeeping variants are:

   sg_ses [--enumerate] [--help] [--list] [--version]

Below is a table of sg_ses options sorted alphabetically by their long option name:

-a

--all

-X FN

--inhex=FN

Table 1  sg_ses command line options

DEVICE should be a SCSI device (logical unit) whose INQUIRY response has its Peripheral Device Type field set to 13. Alternatively it could be another Peripheral Device Type with the EncServ bit (also in the INQUIRY response) set. This latter case is now rare.

The manpage for sg_ses (use 'man sg_ses' at the command prompt) has more details on the command line options. For downloading microcode to a SES device, try 'man sg_ses_microcode' for details on its command line options.

Indexing

A SES device often contains a daunting amount of information. Often we are interested in a single element, for example flashing the ident LED on one disk drive carrier prior to replacing it. The sg_ses utility uses indexing to pinpoint a single element. Actually the single element may be several closely related elements that can be viewed as one. Currently one of four indexing methods can be used:

1. --index= option, based on the structure outlined in the Configuration page

2. --descriptor=DES where DES matches the Descriptor entry in the Element Descriptor page

3. --dev-slot-num=SN (or --dsn=SN) where SN matches the device slot number entry in the Additional Element Status page

4. --sas-addr=SA where SA matches an SAS address entry in the Additional Element Status page

The first method is low level indexing and is described in this section. It can be used to index any element provided by the SES device. The final three methods are medium level indexing and are described in the next section. The final three methods are preferable (i.e. simpler) but require optional support from the SES device (e.g. --descriptor=DES needs the Element Descriptor page with sensible names (e.g. unique)).

There are two important pages to understand for low level indexing: the Configuration page and the Enclosure Status page. The Configuration page contains meta-information (information about information, similar to a database schema) while the Enclosure Status page contains 4-byte "elements". Amongst other things each disk slot in an enclosure has one 4-byte element in the Enclosure Status page; and that 4-byte restriction obviously limits the amount of information that can be held. So additional information is placed in other diagnostic pages that have a same (or similar) element order as the Enclosure Status page. Those extra pages are the Threshold page (also with 4-byte elements) and the Element Descriptor page (which has variable length fields). And there is the Additional Element Status page which has fewer elements but the elements it does have, correspond to elements in the Enclosure Status page.

The sg_ses utility uses a two level index or a descriptor name to identify a single element within the Enclosure Status page. The first level index is a Type Header index or an Element Type abbreviation. The second level is an overall element indication ("-1") or an individual element index. All indexes (apart from the overall element indication) are origin 0.

Below is the Type Descriptor Header format taken from a SES-3 draft. There is an array of these in the Configuration page.

 Figure 1  Taken from ses3r03.pdf at t10.org

It looks tempting to use the Element Type (an 8-bit number) as an index but the same element type can appear more than once. The example that the standard uses is having one "power supply" Element Type for 5 volt power supplies and another for 12 volt power supplies. So when the Element Type is used as an index it needs a sub-index to distinguish such cases. Each Type Descriptor Header has a corresponding "overall" element in the Enclosure Status page. In addition the Enclosure Status page has "Number of possible elements" of "individual" elements; note that may be zero. For indexing purposes the "Subenclosure identifier" field is ignored as is the "type descriptor text length" field.

As an example, let us assume a Configuration page has the following four Type Descriptor Headers (each number is placed in a byte):

• 23,3,0,0

• 2,2,0,0

• 2,1,0,0

• 134,0,0,0

The Type Header indexes are 0, 1, 2, 3 respectively; the Element Types are Array Device Slot, Power Supply, Power Supply and Vendor Specific LED display (element_type=134). The latter Element Type has been made up for this example. These four Type Descriptor Headers imply ten element instances in the Enclosure Status page: one overall instance for each Type Header (4) plus the sum of the "Number of possible elements" fields (3+2+1+0=6). The elements instances should appear in the order shown in the table below. The various ways that sg_ses can index these instances are shown in columns A, B, C, D, E and F:

Table 2  Indexing elements in the Enclosure Status page

Note that only two columns can index the full ten instances: A and E. The simpler of the two is column A using the Type Header index followed by the individual index and this is the scheme used internally by sg_ses. Column C may be more intuitive but gets a bit messy with the second Power Supply Element Type (i.e. the 12v one) and can't index the Vendor Specific Element Type (i.e. the last row).

Numbers shown above are in decimal. Their hexadecimal equivalents may also be given either prefixed with '0x' (or '0X') or with a trailing 'h' (or 'H'). One consideration is that strings like "ah" ('a' in hex (10 in decimal)) appearing in a position that can either be an abbreviation or a number, will be interpreted as an abbreviation. Adding a leading '0' will cause it (e.g. "0ah") to be interpreted as a number.

Also see index ranges described at the end of the clear, get and set section.

Descriptor Names, Device Slot Numbers and SAS Addresses

Optionally a SES device may support the Element Descriptor page. If so that page will have the same number of elements as the Enclosure Status page. So it is possible that the descriptor names found in the Element Descriptor page may be used as an alternative to the --index= option discussed in the previous section. The --descriptor=DES option was introduced for this purpose.

Apart from having a Element Descriptor page, the descriptor names need to be suitable. Some or all of the descriptor names might be empty which does not help indexing. Also the descriptor names might be so long (e.g. including part and serial numbers) as to make them unwieldy as indexes. A simple way to find out is to look at the descriptor names in the Element Descriptor page with:

$ sg_ses --page=ed /dev/sg3
Element descriptor In diagnostic page:
  generation code: 0x0
  element descriptor list (grouped by type):
    Element type: Array device slot, subenclosure id: 0 [ti=0]
      Overall descriptor: ArrayDevicesInSubEnclsr0
      Element 0 descriptor: ArrayDevice00
      Element 1 descriptor: ArrayDevice01
      Element 2 descriptor: ArrayDevice02
      ...

The above example shows descriptor names suitable for indexing. Given these descriptor names, one would expect these two commands to do the same thing, namely turn on the ident LED on the disk in slot 0:

$ sg_ses --index=arr,0 --set=ident /dev/sg3
$ sg_ses --descriptor=ArrayDevice00 --set=ident /dev/sg3

If an unrecognized descriptor name is given then response will be like this:

$ sg_ses --descriptor=xxx --set=ident /dev/sg3
descriptor name: xxx not found (check 'ed' page [0x7])

A SAS disk or tape can be indexed by its SAS address (also known as its target port identifier in SCSI jargon). SAS disks are typically dual ported and have two target ports whose SAS addresses are usually adjacent values. When a SATA disk is involved, it is usually connected via a SAS expander which typically assigns it an arbitrary "SAS" address. And that arbitrary "SAS" address is typically different from the SATA disk's WWN. One can see that in the following example where the SAS address (5001517e85c3efe1) is surprisingly close to that of the SAS expander (5001517e85c3efff).  If that SATA disk was placed in another slot, the address would probably change (slightly) in sympathy. For indexing based on SAS addresses to work, the Additional Element Status page needs to be present.

# sg_ses --sas-addr=5001517e85c3efe1 /dev/sg3
ArrayDevice01 [0,1]  Element type: Array device slot
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    OK=0, Reserved device=0, Hot spare=0, Cons check=0
    In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
    App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
    Ready to insert=0, RMV=0, Ident=0, Report=0
    App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
    Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
  Additional Element Status:
    Transport protocol: SAS
    number of phys: 1, not all phys: 0, device slot number: 1
    phy index: 0
      device type: no SAS device attached
      initiator port for:
      target port for: SATA_device
      attached SAS address: 0x5001517e85c3efff
      SAS address: 0x5001517e85c3efe1
      phy identifier: 0x0

The 'attached SAS address' shown above is that of the SAS expander. From the above we can see that the SATA disk is associated with device slot number 1. So the ident flag can be set as follows:

# sg_ses --dev-slot-num=1 --set=ident /dev/sg3

Device slot numbers describe a position in a disk enclosure. They may also be available in SAS expanders often associated with SES devices. Some SAS-2 and all SAS-3 expanders should report device slot numbers. The number is a 1 byte value from 0 to 255 where 255 indicates there is no associated slot. Device slot numbers, if available, will be in the output of the smp_discover utility in the smp_utils package.

clear, get and set

These three command line options allow a named field within an element to be fetched or changed. These options can only be used on the Enclosure Status, Element Descriptor, Threshold In (or Out) and the Additional Element Status pages. Since the Element Descriptor and Additional Element Status pages are read-only only the --get= option can be used with them.

The Enclosure Status and the Enclosure Control pages have the same page number, the difference is that the "Status" page is used for fetching fields (i.e. the current state) while the "Control" page is used for changing fields. Other pages such as the Threshold page use "In" and "Out" in the same way (i.e. "In" to fetch the current state, "Out" to change it).

sg_ses attempts to name some often used fields, for example the "ident" field (also known as "locate" by some vendors) found in the Array Device Slot element . In some cases this can be a bit difficult because the naming of a field may differ from one Element Type to another. The "ident" field is in byte 2, bit position 1 and is 1 bit long in the Array Device Slot status element:

Figure 2  Taken from ses3r03.pdf at t10.org

Notice in the corresponding "Control" element that field is called "Rqst Ident". Not a major problem, but the naming discrepancy can be larger for example the "Fault Sensed" field (byte 3, bit 6, 1 bit long) in the Status page has no corresponding field in the Control page. Starting with sg_ses version 1.99 a mask is applied to the status element so that status element bits corresponding to Reserved field in the control element will be zero. For example if "Fault sensed" is set then some SES devices will object to a --set=ident because byte 3, bit 6 is reserved in the control element. The prior sg_ses behaviour (i.e. no mask applied) can selected with the --mask option.

Figure 3  Taken from ses3r03.pdf at t10.org

When using the --clear, --get and --set options, the default page is the Enclosure Status/Control page (the Status page for --get, the Control page for --clear and --set). All of the following invocations attempt to set the "ident" field on the disk in slot 5 and all should cause its LED to flash:

$ sg_ses --index=arr,5 --set=ident /dev/sg3
$ sg_ses --index=arr,5 --set=ident=1 /dev/sg3
$ sg_ses --index=arr,5 --set=ident --page=ec /dev/sg3
$ sg_ses --index=5 --set=ident /dev/sg3
$ sg_ses --index=arr,5 --set=2:1 /dev/sg3
$ sg_ses --index=arr,5 --set=2:1=1 /dev/sg3
$ sg_ses --index=arr,5 --set=2:1:1=1 /dev/sg3

The latter three invocations use a numerical description of the field whose format is <start_byte>:<start_bit>[:<number_of_bits>] . The <number_of_bits> defaults to 1 when it is not given. To get a listing of the field names supported by sg_ses use the --enumerate option twice (or more simply: '-ee'). For example:

$ sg_ses -ee
--clear, --get, --set acronyms for enclosure status/control [0x2] page:
  ...
  ident  [Device slot] [2:1:1]
  ident  [Array device slot] [2:1:1]
  ident  [Power supply] [1:7:1]
  ident  [Cooling] [1:7:1]
  ...
  locate  [Device slot] [2:1:1]
  locate  [Array device slot] [2:1:1]
  locate  [Power supply] [1:7:1]
  locate  [Cooling] [1:7:1]
  ...

Assuming there is only one power supply and it has an "ident" LED then this invocation should cause it to flash:

$ sg_ses --index=ps,0 --set=ident /dev/sg3

Notice that the "ident" field is in a different location in the Power Supply and Cooling Element Types (i.e. byte 1, bit 7) compared to the Device Slot and Array Device Slot Element Types (i.e. byte 2, bit 1).

To be absolutely sure of what is written to a control element, the whole 4 bytes (32 bits) can be written explicitly:

$ sg_ses --dsn=3 --set=0:7:32=0x200 /dev/sg3

That will set the RQST IDENT bit in the Array device slot control element whose device slot number is 3.

From sg3_utils version 1.43 (subversion revision 711) multiple clear, get and set options can appear in one invocation (up to a maximum of 8). They are applied in the order they appear (i.e. left to right) so if they contradict one another, the rightmost is applied last.
At the same time, the numeric part of the --index= argument was expanded to accept a range. For example:

$ sg_ses --index=3-5 --set=ident /dev/sg3

will set the IDENT bit on the (array) indexes 3, 4 and 5.

Record once, analyse often

One or more diagnostic pages can be "captured" on a machine controlling or near an enclosure and stored in a file as either ASCII hexadecimal text or binary. Then later, potentially at another location, sg_ses can be run taking that file as input in order to analyse the state of the enclosure when the data was captured.

The '-HHH' (or '-HHHH' which adds dpage names as comments) option can be used together with redirection of stdout to capture the state of an enclosure (i.e. read all the SES diagnostic pages). For example:

$ sg_ses --page=all -HHHH /dev/sg3 > enc_stat.hex

To capture the equivalent enclosure state in binary:

$ sg_ses --page=all -rr /dev/sg3 > enc_stat.bin

Then, at some later time and place, that file can be used by sg_ses to do any operation (that does not attempt to change the state of the enclosure). for example:

$ sg_ses --data=@enc_stat.hex --status --join

The equivalent invocation for a binary file is:

$ sg_ses --data=@enc_stat.bin --status -rr  --join

Note that in "playback" or analyse mode, no device is needed. The --status option needs to be given to stress the the data file is being used for input and not output.

Raw page writes

There are six SES diagnostic pages that can be modified:  Enclosure Control, Threshold Out, String Out, Subenclosure String Out, Download Microcode Control and Subenclosure Nickname Control. Of those only the Enclosure Control and Threshold Out pages can be modified with the --set and --clear options as discussed in the previous section. So if we need to change one of the others then an alternative technique is required. However the use of this lower level technique is awkward involving three steps: read, modify then write. Following is an example of changing the String page using the --raw and --control options.

The current String In page can be shown with:

$ sg_ses --page=str /dev/bsg/6:0:2:0

Now the "read" step: the following command will send the contents of the String page (from byte 4 onwards) to stdout. The output will be in ASCII hex with pairs of hex digits representing a byte, 16 pairs per line, space separated. The redirection puts stdout in a file called "t":

$ sg_ses --page=str --raw /dev/bsg/6:0:2:0 > t

Then, perhaps with the aid of the SES-3 document (in SES-3 revision 3: section 6.1.6), use your favourite text editor to change t. The changes can be written to the device with:

$ sg_ses --page=str --control --data=- /dev/bsg/6:0:2:0 < t

If the above is successful, the String should have been changed. To check try:

$ sg_ses --page=str /dev/bsg/6:0:2:0

Note that the first four bytes of the fetched pages are not output when the --raw option is given. Likewise when the --control and --data options are given, sg_ses does not expect the first four bytes of the page to be provided. Byte 0 is the page number while bytes 2 and 3 contain the count of following bytes. Byte 1 is usually 0 but in some cases it may not be, hence the --byte1=B1 option.

In the Enclosure Control page the lower 4 bits of byte 1 contain the Info, Non-crit, Crit and Unrecov bit fields. To change these with sg_ses the --raw and --control technique can be used. Following is an example of setting the Info bit field (and zeroing the other byte 1 bits if they happened to be set):

$ sg_ses --page=es --raw /dev/bsg/6:0:2:0 > t
$ sg_ses --page=ec --byte1=0x8 --control --data=- /dev/bsg/6:0:2:0 < t

In the Subenclosure String Out, Subenclosure Nickname Control and Download Microcode Control pages, byte 1 is the Subenclosure identifier.

To set the Subenclosure nickname, the '--nickid=SEID' and '--nickname=SEN' options can be used. To download microcode, the sg_ses_microcode utility is recommended.

sg_ses and NVMe

The NVME-MI (Management Interface) specification has recently added the MI (Admin) commands: "SES Send" and "SES Receive". These NVMe commands are mirrors of the SCSI SEND DIAGNOSTIC and RECEIVE DIAGNOSTIC RESULTS commands. The purpose of those  new NVME-MI commands is to support the SES-3 standard almost "as-is" for NVMe enclosures. Support for this in sg_ses found in sg3_utils version 1.43 can be considered beta. sg_ses has been tested successfully on one NVMEoF enclosure.

Conclusion

The sg_ses utility was first written in 2004. It has gone through many iterations since then to allow more flexible access (e.g. indexing), track large additions to the SES standards and fix bugs. SAS-2 and SAS-3 expanders tend to have inbuilt SES devices so SES is becoming more common. Many of the improvements to sg_ses are thanks to reports and suggestions from users. Please continue to provide that feedback.

Return to main page.

Last updated: 8th April 2021