The sg_ses utility

  1. The sg_ses utility
    1. Introduction
    2. Overview
    3. Command line options
    4. Indexing
    5. Descriptor Names, Device Slot numbers and SAS Addresses
    6. clear, get and set
    7. Raw page writes
    8. Conclusion

Introduction

The sg_ses utility enables a user "to manage and sense the state of the power supplies, cooling devices, displays, indicators, individual drives, and other non-SCSI elements installed in an enclosure". The SCSI Enclosure Services standards (most recent is SES-2 ANSI INCITS 448-2008) and the latest draft (ses3r03.pdf at www.t10.org) describe the format that the sg_ses utility expects to find in a SES device ("logical unit" or "process").

The sg_ses utility is found in the sg3_utils package. The utility was originally written for Linux and has ports to FreeBSD, Solaris and Windows. This page outlines the features of the sg_ses utility version 1.91 found in the sg3_utils version 1.39 package.

Overview

Typically the first problem to solve is finding an Operating System handle for a SES device. SCSI disks are typically easier to find but rarely is a SCSI disk also a SES device (only if the EncServ bit is set in the disk's INQUIRY command response). First let us try the sg_ses utility on a SAS disk (in Linux):

$ sg_ses /dev/sdb
  SEAGATE   ST33000650SS      0002
    disk device (not an enclosure)
Supported diagnostic pages:
  Supported diagnostic pages [sdp] [0x0]
  Protocol specific (SAS transport) [] [0x3f]
  Translate address (SBC) [] [0x40]

Without any options sg_ses reports the names of the supported diagnostic pages. The SAS disk does support some pages but none that belong to the SES standard. On Linux it helps to list all SCSI devices (including showing the generic device names with the '-g' option; 'modprobe sg' may be needed if hyphens appear in the last column):

$ lsscsi -g
[1:0:0:0]    disk    ATA      ST3320620AS      3.AA  /dev/sda   /dev/sg0
[6:0:0:0]    disk    SEAGATE  ST33000650SS     0002  /dev/sdb   /dev/sg1
[6:0:1:0]    disk    SEAGATE  ST32000444SS     0006  /dev/sdc   /dev/sg2
[6:0:2:0]    enclosu Intel    RES2SV240        0600  -          /dev/sg3

The last item is an enclosure and notice that it does not have a disk device node (i.e. there is no /dev/sdd). In Linux the bsg driver could also be used and its device node would be /dev/bsg/6:0:2:0 for the enclosure. Now let us try sg_ses on /dev/sg3 :

$ sg_ses /dev/sg3
  Intel     RES2SV240         0600
Supported diagnostic pages:
  Supported diagnostic pages [sdp] [0x0]
  Configuration (SES) [cf] [0x1]
  Enclosure status/control (SES) [ec,es] [0x2]
  String In/Out (SES) [str] [0x4]
  Threshold In/Out (SES) [th] [0x5]
  Element descriptor (SES) [ed] [0x7]
  Additional element status (SES-2) [aes] [0xa]
  Supported SES diagnostic pages (SES-2) [ssp] [0xd]
  Download microcode (SES-2) [dm] [0xe]
  Subenclosure nickname (SES-2) [snic] [0xf]

So now we see several diagnostic pages related to the SES (and SES-2) standard. Of these the most important are the Configuration page and the Enclosure Status/Control page.

Let us assume that a 6-disk SAS/SATA disk array is associated with this SES device. We might suspect the disk on the left of the array is "slot 0" but would like to check before removing the disk. One way to double check is to ask "slot 0" to identify itself by flashing its LED for 5 seconds. This can be done like this:

$ sg_ses --dev-slot-num=0 --set=ident /dev/sg3
$ sleep 5
$ sg_ses --dev-slot-name=0 --clear=ident /dev/sg3
Now check the disk on the right (slot 5), this time using shorter option variants ('-x 5' is equivalent to '--dev-slot-num=5'):

$ sg_ses -x 5 -S ident /dev/sg3
$ sleep 5
$ sg_ses -x 5 -C ident /dev/sg3

Typically there is a lot of information held by a SES device. Several of those diagnostic pages carry information with a somewhat similar structure to relational database tables. To see a lot of information try these two commands (one after the other):

$ sg_ses --join /dev/sg3
$ sg_ses --join --filter /dev/sg3

The first command will list information on all disks, fans, power supplies etc, even for disks and fans not installed. The second command attempts to filter out some less relevant information.

Command line options

sg_ses is a command line utility whose normal syntax is:
  sg_ses [options] DEVICE

An exception is the --enumerate option which doesn't require a DEVICE argument and ignores it if one is given. Below is a table of sg_ses options sorted alphabetically by the long option name:
Table 1  sg_ses command line options
short option
 long option
 Brief description
-b B1
 --byte1=B1
 byte 1 (2nd byte) of control page set to B1. Only active with --control
-C STR
 --clear=STR
 clear value of field described by STR. Requires either --descriptor= or --index= option to choose an element.
-c
 --control
 send control information. That information provided by --data=
-d H,H... --data=H,H...
 H,H... is a string of ASCII hex bytes for control pages
-d -
 --data=-
 fetch that string of ASCII hex bytes from stdin
-D DN --descriptor=DN
 for indexing: find element whose Element Descriptor page's name entry matches DN. Implies --join
-x SN
 --dev-slot-num=
SN
 for indexing: find element whose Additional Element status page's device slot entry matches SN. Implies --join. Found in sg_ses version 1.79 and later (from sg3_utils version 1.38).
-E=A_F
 --eiioe=A_F
 prior to the EIIOE bit's introduction in ses3r06 several arrays acted as if this bit was set. If an argument of 'force' is given the Additional Element Status page's element index field is treated as if EIIOE=1. If an argument of 'auto' is given a heuristic is used to decide whether to set EIIOE or not. Found in sg_ses version 1.88 and later (from sg3_utils version 1.39).
-e
 --enumerate
 list recognized diagnostic page codes and Element Type names. When used twice lists acronyms for the --clear, --get and --set options. Ignores other options and DEVICE argument
-f
 --filter
 reduce the amount of output when displaying pages (e.g. Enclosure Status and the Additional Element Status).
-G STR
 --get=STR
 get value of field described by STR. Requires either --descriptor= or --index= option to choose an element.
-h
 --help
 print usage message then exit
-H
 --hex
 print the read diagnostic page in hexadecimal. Action varies depending on other options. For example with the --get=STR option prints the output value in hex.
-I IIA
 --index=IIA individual index ('-1' for overall) or Element Type abbreviation (e.g. 'arr')
-I TIA,II
 --index=TIA,II
 TIA is Type Header index or Element Type abbreviation; II is individual index ('-1' for overall)
-i
 --inner-hex decode page and print element(s) in hex
-j
 --join
 group ("join") Enclosure Status, Element Descriptor and Additional Element Status pages. Use twice to add Threshold In page
-l
 --list
 same action as --enumerate
-N SEID
 --nickid=SEID
 SEID is subenclosure identifier (default: 0)
-n SEN --nickname=SEN SEN is the new subenclosure nickname
-p PG
 --page=PG
 SES diagnostic page code PG is an abbreviation or a number in decimal (prefix with '0x' (or 'h' suffix) for hex)
-r
 --raw
 print status page in ASCII hex suitable for --data=- ; when used twice outputs page in binary to stdout
-A SA
 --sas-addr=SA
 for indexing: find element whose Additional Element status page's SAS address entry matches SA. Will only match device slot and array device slot elements. Implies --join . Found in sg_ses version 1.79 and later (from sg3_utils version 1.38).
-S STR
 --set=STR
 set value of field described by STR. Requires either --descriptor= or --index= option to choose an element.
-s
 --status
 fetch status information. The default action when no options given (including this one) is to display the Supported Diagnostic Pages page
-v
 --verbose
 increase verbosity of output. Use multiple times for more debug information
-V
 --version
 print version number and its date then exit

DEVICE should be a SCSI device (logical unit) whose INQUIRY response has its Peripheral Device Type field set to 13. Alternatively
it could be another Peripheral Device Type with the EncServ bit (also in the INQUIRY response) set.

The manpage for sg_ses (use 'man sg_ses' at the command prompt) has more details on the command line options.

Indexing

A SES device often contains a daunting amount of information. Often we are interested in a single element, for example flashing the ident LED on one disk drive carrier prior to replacing it. The sg_ses utility uses indexing to pinpoint a single element. Actually the single element may be several closely related elements that can be viewed as one. Currently one of four indexing methods can be used:
  1. either --index= option, based on the structure outlined in the Configuration page
  2. --descriptor=DN where DN matches the Descriptor entry in the Element Descriptor page
  3. --dev-slot-num=SN where SN matches the device slot number entry in the Additional Element Status page
  4. --sas-addr=SA where SA matches an SAS address entry in the Additional Element Status page
The first method is low level indexing and is described in the section. It can be used to index any element provided by the SES device. The final three methods are medium level indexing and are described in the next section. The final three methods are preferable (i.e. simpler) but require optional support from the SES device (e.g. --descriptor=DN needs the Element Descriptor page with sensible names (e.g. unique).

There are two important pages to understand for low level indexing: the Configuration page and the Enclosure Status page. The Configuration page contains meta-information (information about information, similar to a database schema) while the Enclosure Status page contains 4-byte "elements". Amongst other things each disk slot in an enclosure has one 4-byte element in the Enclosure Status page; and that 4-byte restriction obviously limits the amount of information that can be held. So additional information is placed in extra pages that have a same element order as the Enclosure Status page. Those extra pages are the Threshold page (also with 4-byte elements) and the Element Descriptor page (which has variable length fields). And there is the Additional Element Status page which has fewer elements but the elements it does have, correspond to elements in the Enclosure Status page.

The sg_ses utility uses a two level index or a descriptor name to identify a single element within the Enclosure Status page. The first level index is a Type Header index or an Element Type abbreviation. The second level is an overall element indication ("-1") or an individual element index. All indexes (apart from the overall element indication) are origin 0.

Below is the Type Descriptor Header format taken from a SES-3 draft. There is an array of these in the Configuration page.


 Figure 1  Taken from ses3r03.pdf at t10.org

It looks tempting to use the Element Type (an 8-bit number) as an index but the same element type can appear more than once. The example that the standard uses is having one "power supply" Element Type for 5 volt power supplies and another for 12 volt power supplies. So when the Element Type is used as an index it needs a sub-index to distinguish such cases. Each Type Descriptor Header has a corresponding "overall" element in the Enclosure Status page. In addition the Enclosure Status page has "Number of possible elements" of "individual" elements; note that may be zero. For indexing purposes the "Subenclosure identifier" field is ignored as is the "type descriptor text length" field.

As an example, let us assume a Configuration page has the following four Type Descriptor Headers (each number is placed in a byte):
The Type Header indexes are 0, 1, 2, 3 respectively; the Element Types are Array Device Slot, Power Supply, Power Supply and Vendor Specific LED display (element_type=134). The latter Element Type has been made up for this example. These four Type Descriptor Headers imply ten element instances in the Enclosure Status page: one overall instance for each Type Header (4) plus the sum of the "Number of possible elements" fields (3+2+1+0=6). The elements instances should appear in the order shown in the table below. The various ways that sg_ses can index these instances are shown in columns A, B, C, D, E and F:

Table 2  Indexing elements in the Enclosure Status page
Element instances

 		A
--index=
 B
--index=
 C
--index=
 D
--index=
 E
--index=
 F
--index=
Array Device Slot
overall element
 0,-1
 -1
 arr,-1
 arr
 _23,-1
 _23
Array Device Slot
individual element 0
 0,0
 0
 arr,0
_23,0
Array Device Slot
individual element 1
 0,1
 1
 arr,1
_23,1
Array Device Slot
individual element 2		 0,2
 2
 arr,2
_23,2
Power Supply (5v)
overall element
 1,-1
ps,-1
 ps
 _2,-1
 _2
Power Supply (5v)
individual element 0
 1,0
ps,0
_2,0
Power Supply (5v)
individual element 1
 1,1
ps,1
_2,1
Power Supply (12v)
overall element		 2,-1
ps1,-1
 ps1
 _2_1,-1
 _2_1
Power Supply (12v)
individual element 0		 2,0
ps1,0
_2_1,0
Vendor Specific LED display
overall element
 3,-1


_134,-1
 _134

Note that only two columns can index the full ten instances: A and E. The simpler of the two is column A using the Type Header index followed by the individual index and this is the scheme used internally by sg_ses. Column C may be more intuitive but gets a bit messy with the second Power Supply Element Type (i.e. the 12v one) and can't index the Vendor Specific Element Type (i.e. the last row).

Numbers shown above are in decimal. Their hexadecimal equivalents may also be given either prefixed with '0x' (or '0X') or with a trailing 'h' (or 'H'). One consideration is that strings like "ah" ('a' in hex (10 in decimal)) appearing in a position that can either be an abbreviation or a number, will be interpreted as an abbreviation. Adding a leading '0' will cause it (e.g. "0ah") to be interpreted as a number.

Descriptor Names, Device Slot numbers and SAS Addresses

Optionally a SES device may support the Element Descriptor page. If so that page will have the same number of elements as the Enclosure Status page. So it is possible that the descriptor names found in the Element Descriptor page. may be used as an alternative to the --index= option discussed in the previous section. The --descriptor=DN option was introduced for this purpose.

Apart from having a Element Descriptor page, the descriptor names need to be suitable. Some or all of the descriptor names might be empty which does not help indexing. Also the descriptor names might be so long (e.g. including part and serial numbers) as to make them unwieldy as indexes. A simple way to find out is to look at the descriptor names in the Element Descriptor page with:

$ sg_ses --page=ed /dev/sg3
Element descriptor In diagnostic page:
  generation code: 0x0
  element descriptor by type list
    Element type: Array device slot, subenclosure id: 0 [ti=0]
      Overall descriptor: ArrayDevicesInSubEnclsr0
      Element 0 descriptor: ArrayDevice00
      Element 1 descriptor: ArrayDevice01
      Element 2 descriptor: ArrayDevice02
      ...

The above example shows descriptor names suitable for indexing. Given these descriptor names, one would expect these two commands to do the same thing, namely turn on the ident LED on the disk in slot 0:

$ sg_ses --index=arr,0 --set=ident /dev/sg3
$ sg_ses --descriptor=ArrayDevice00 --set=ident /dev/sg3

If an unrecognized descriptor name is given then response will be like this:

$ sg_ses --descriptor=xxx --set=ident /dev/sg3
descriptor name: xxx not found (check 'ed' page [0x7])

Some care should be taken when indexing with a SAS address of a SATA disk since the SAS address is not the SATA disk's WWN. One can see that in the following example where the SAS address (5001517e85c3efe1) is surprisingly close to that of the SAS expander (5001517e85c3efff).  The SAS expander actually manufactured that address and if that SATA disk was placed in another slot, the address would probably change in sympathy. For SAS disks and tapes the SAS address seen in the Additional Element Status page will be one of two target port identifiers since these devices are typically dual ported.

# sg_ses --sas-addr=5001517e85c3efe1 /dev/sg3
ArrayDevice01 [0,1]  Element type: Array device slot
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    OK=0, Reserved device=0, Hot spare=0, Cons check=0
    In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0
    App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0
    Ready to insert=0, RMV=0, Ident=0, Report=0
    App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0
    Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0
  Additional Element Status:
    Transport protocol: SAS
    number of phys: 1, not all phys: 0, device slot number: 1
    phy index: 0
      device type: no SAS device attached
      initiator port for:
      target port for: SATA_device
      attached SAS address: 0x5001517e85c3efff
      SAS address: 0x5001517e85c3efe1
      phy identifier: 0x0

Noticing that SATA disk is associated with device slot 1, set the ident flag as follows:

# sg_ses --dev-slot-num=1 --set=ident /dev/sg3

clear, get and set

These three command line options allow a named field within an element to be fetched or changed. These options can only be used on the Enclosure Status, Element Descriptor, Threshold In (or Out) and the Additional Element Status pages. Since the Element Descriptor and Additional Element Status pages are read-only only the --get= option can be used with them.

The Enclosure Status and the Enclosure Control pages have the same page number, the difference is that the "Status" page is used for fetching fields (i.e. the current state) while the "Control" page is used for changing fields. Other pages such as the Threshold page use "In" and "Out" in the same way (i.e. "In" to fetch the current state, "Out" to change it).

sg_ses attempts to name some often used fields, for example the "ident" field (also known as "locate" by some vendors) found in the Array Device Slot element . In some cases this can be a bit difficult because the naming of a field may differ from one Element Type to another. The "ident" field is in byte 2, bit position 1 and is 1 bit long in the Array Device Slot status element:


Figure 2  Taken from ses3r03.pdf at t10.org

Notice in the corresponding "Control" element that field is called "Rqst Ident". Not a major problem, but the naming discrepancy can be larger for example the "Fault Sensed" field (byte 3, bit 6, 1 bit long) in the Status page has no corresponding field in the Control page.


Figure 3  Taken from ses3r03.pdf at t10.org

When using the --clear, --get and --set options, the default page is the Enclosure Status/Control page (the Status page for --get, the Control page for --clear and --set). All of the following invocations attempt to set the "ident" field on the disk in slot 5 and all should cause its LED to flash:

$ sg_ses --index=arr,5 --set=ident /dev/sg3
$ sg_ses --index=arr,5 --set=ident=1 /dev/sg3
$ sg_ses --index=arr,5 --set=ident --page=ec /dev/sg3
$ sg_ses --index=5 --set=ident /dev/sg3
$ sg_ses --index=arr,5 --set=2:1 /dev/sg3
$ sg_ses --index=arr,5 --set=2:1=1 /dev/sg3
$ sg_ses --index=arr,5 --set=2:1:1=1 /dev/sg3

The latter three invocations use a numerical description of the field whose format is <start_byte>:<start_bit>[:<number_of_bits>] . The <number_of_bits> defaults to 1 when it is not given. To get a listing of the field names supported by sg_ses use the --enumerate option twice (or more simply: '-ee'). For example:

$ sg_ses -ee
--clear, --get, --set acronyms for enclosure status/control [0x2] page:
  ...
  ident  [Device slot] [2:1:1]
  ident  [Array device slot] [2:1:1]
  ident  [Power supply] [1:7:1]
  ident  [Cooling] [1:7:1]
  ...
  locate  [Device slot] [2:1:1]
  locate  [Array device slot] [2:1:1]
  locate  [Power supply] [1:7:1]
  locate  [Cooling] [1:7:1]
  ...

Assuming there is only one power supply and it has an "ident" LED then this invocation should cause it to flash:

$ sg_ses --index=ps,0 --set=ident /dev/sg3

Notice that the "ident" field is in a different location in the Power Supply and Cooling Element Types (i.e. byte 1, bit 7).

Raw page writes

There are six SES diagnostic pages that can be modified:  Enclosure Control, Threshold Out, String Out, Subenclosure String Out, Download Microcode Control and Subenclosure Nickname Control. Of those only the Enclosure Control and Threshold Out pages can be modified with the --set and --clear options as discussed in the previous section. So if we need to change one of the others then an alternative technique is required. However the use of this lower level technique is awkward involving three steps: read, modify then write. Following is an example of changing the String page using the --raw and --control options.

The current String In page can be shown with:

$ sg_ses --page=str /dev/bsg/6:0:2:0

Now the "read" step: the following command will send the contents of the String page (from byte 4 onwards) to stdout. The output will be in ASCII hex with pairs of hex digits representing a byte, 16 pairs per line, space separated. The redirection puts stdout in a file called "t":

$ sg_ses --page=str --raw /dev/bsg/6:0:2:0 > t

Then, perhaps with the aid of the SES-3 document (in SES-3 revision 3: section 6.1.6), use your favourite text editor to change t. The changes can be written to the device with:

$ sg_ses --page=str --control --data=- /dev/bsg/6:0:2:0 < t

If the above is successful, the String should have been changed. To check try:

$ sg_ses --page=str /dev/bsg/6:0:2:0

Note that the first four bytes of the fetched pages are not output when the --raw option is given. Likewise when the --control and --data options are given, sg_ses does not expect the first four bytes of the page to be provided. Byte 0 is the page number while bytes 2 and 3 contain the count of following bytes. Byte 1 is usually 0 but in some cases it may not be, hence the --byte1=B1 option.

In the Enclosure Control page the lower 4 bits of byte 1 contain the Info, Non-crit, Crit and Unrecov bit fields. To change these with sg_ses the --raw and --control technique can be used. Following is an example of setting the Info bit field (and zeroing the other byte 1 bits if they happened to be set):

$ sg_ses --page=es --raw /dev/bsg/6:0:2:0 > t
$ sg_ses --page=ec --byte1=0x8 --control --data=- /dev/bsg/6:0:2:0 < t

In the Subenclosure String Out, Subenclosure Nickname Control and Download Microcode Control pages, byte 1 is the Subenclosure identifier.

Conclusion

The sg_ses utility was first written in 2004. It has gone through many iterations since then to fix bugs, allow more flexible access (e.g. indexing) and track large additions to the SES standards. SAS-2 expanders tend to have inbuilt SES devices so SES is becoming more common. Many of the improvements to sg_ses are thanks to reports and suggestions from users. Please continue to make those reports and suggestions.

Return to main page.

Last updated: 11th June 2014