The sg_ses utility

The sg_ses utility

Introduction

The sg_ses utility enables a user "to manage and sense the state of the power supplies, cooling devices, displays, indicators, individual drives, and other non-SCSI elements installed in an enclosure". The SCSI Enclosure Services standards (most recent is SES-2 ANSI INCITS 448-2008) and the latest draft (ses3r03.pdf at www.t10.org) describe the format that the sg_ses utility expects to find in a SES device ("logical unit" or "process").

The sg_ses utility is found in the sg3_utils package. The utility was originally written for Linux and has ports to FreeBSD, Solaris and Windows. This page outlines the features of the sg_ses utility version 1.64 found in the sg3_utils version 1.33 package.

Overview

Typically the first problem to solve is finding an Operating System handle for a SES device. SCSI disks are typically easier to find but rarely is a SCSI disk also a SES device (only if the EncServ bit is set in the disk's INQUIRY command response). First let us try the sg_ses utility on a SAS disk (in Linux):

$ sg_ses /dev/sdb

  SEAGATE   ST33000650SS     
0002

    disk device (not an enclosure)

Supported diagnostic pages:

  Supported diagnostic pages [sdp] [0x0]

  Protocol specific (SAS transport) [] [0x3f]

  Translate address (SBC) [] [0x40]

Without any options sg_ses reports the names of the supported diagnostic pages. The SAS disk does support some pages but none that belong to the SES standard. On Linux it helps to list all SCSI devices (including showing the generic device names with the '-g' option; 'modprobe sg' may be needed if hyphens appear in the last column):

$ lsscsi -g

[1:0:0:0]    disk   
ATA     
ST3320620AS      3.AA 
/dev/sda   /dev/sg0

[6:0:0:0]    disk    SEAGATE 
ST33000650SS     0002  /dev/sdb  
/dev/sg1

[6:0:1:0]    disk    SEAGATE 
ST32000444SS     0006  /dev/sdc  
/dev/sg2

[6:0:2:0]    enclosu Intel   
RES2SV240        0600 
-         
/dev/sg3

The last item is an enclosure and notice that it does not have a disk device node (i.e. there is no /dev/sdd). On Linux the bsg driver could also be used and its device node would be /dev/bsg/6:0:2:0 for the enclosure. Now let us try sg_ses on /dev/sg3 :

$ sg_ses /dev/sg3

  Intel    
RES2SV240         0600

Supported diagnostic pages:

  Supported diagnostic pages [sdp] [0x0]

  Configuration (SES) [cf] [0x1]

  Enclosure status/control (SES) [ec,es] [0x2]

  String In/Out (SES) [str] [0x4]

  Threshold In/Out (SES) [th] [0x5]

  Element descriptor (SES) [ed] [0x7]

  Additional element status (SES-2) [aes] [0xa]

  Supported SES diagnostic pages (SES-2) [ssp] [0xd]

  Download microcode (SES-2) [dm] [0xe]

  Subenclosure nickname (SES-2) [snic] [0xf]

So now we see several diagnostic pages related to the SES (and SES-2) standard. Of these the most important are the Configuration page and the Enclosure Status/Control page.

Let us assume that a 6-disk SAS/SATA disk array is associated with this SES device. We might suspect the disk on the left of the array is "slot 0" but would like to check before removing the disk. One way to double check is to ask "slot 0" to identify itself by flashing its LED for 5 seconds. This can be done like this:

$ sg_ses --index=0 --set=ident
/dev/sg3

$ sleep 5

$ sg_ses --index=0 --clear=ident
/dev/sg3

Now check the disk on the right (slot 5), this time using shorter option variants:

$ sg_ses -I 5 -S ident /dev/sg3

$ sleep 5

$ sg_ses -I 5 -C ident /dev/sg3

Typically there is a lot of information held by a SES device. Several of those diagnostic pages carry information with a somewhat similar structure to relational database tables. To see a lot of information try these two commands (one after the other):

$ sg_ses --join /dev/sg3

$ sg_ses --join --filter /dev/sg3

The first command will list information on all disks, fans, power supplies etc, even for disks and fans not installed. The second command attempts to filter out some of the unuseful information.

Command line options

sg_ses is a command line utility whose normal syntax is:

  sg_ses [options] DEVICE

An exception is the --enumerate option which doesn't require a DEVICE argument and ignores it if one is given. Below is a table of sg_ses options sorted alphabetically by the long option name:

Table 1 sg_ses command line options
short option	long option	Brief description
-b B1	--byte1=B1	byte 1 (2nd byte) of control page set to B1. Only active with --control
-C STR	--clear=STR	clear value of field described by STR. Requires either --descriptor= or --index= option to choose an element.
-c	--control	send control information. That information provided by --data=
-d H,H...	--data=H,H...	H,H... is a string of ASCII hex bytes for control pages
-d -	--data=-	fetch that string of ASCII hex bytes from stdin
-D DN	--descriptor=DN	the Element Descriptor page might contain DN. If so then this option is an alternative to --index= option
-e	--enumerate	list recognized diagnostic page codes and Element Type names. When used twice lists acronyms for the --clear, --get and --set options. Ignores other options and DEVICE argument
-f	--filter	reduce the amount of output when displaying pages (e.g. Enclosure Status and the Additional Element Status).
-G STR	--get=STR	get value of field described by STR. Requires either --descriptor= or --index= option to choose an element.
-h	--help	print usage message then exit
-H	--hex	print the read diagnostic page in hexadecimal. Action varies depending on other options. For example with the --get=STR option prints the output value in hex.
-i	--inner-hex	decode page and print element(s) in hex
-I IIA	--index=IIA	individual index ('-1' for overall) or Element Type abbreviation (e.g. 'arr')
-I TIA,II	--index=TIA,II	TIA is Type Header index or Element Type abbreviation; II is individual index ('-1' for overall)
-j	--join	group ("join") Enclosure Status, Element Descriptor and Additional Element Status pages. Use twice to add Threshold In page
-l	--list	same action as --enumerate
-p PG	--page=PG	SES diagnostic page code PG is an abbreviation or a number in decimal (prefix with '0x' (or 'h' suffix) for hex)
-r	--raw	print status page in ASCII hex suitable for --data=- ; when used twice outputs page in binary to stdout
-S STR	--set=STR	set value of field described by STR. Requires either --descriptor= or --index= option to choose an element.
-s	--status	fetch status information. The default action when no options given (including this one) is to display the Supported Diagnostic Pages page
-v	--verbose	increase verbosity of output. Use multiple times for more debug information
-V	--version	print version number and its date then exit

DEVICE should be a SCSI device (logical unit) whose INQUIRY response has its Peripheral Device Type field set to 13. Alternatively
it could be another Peripheral Device Type with the EncServ bit (also in the INQUIRY response) set.

The manpage for sg_ses (use 'man sg_ses' at the command prompt) has more details on the command line options.

Indexing

A SES device often contains a daunting amount of information. Often we are interested in a single element, for example flashing the ident LED on one disk drive carrier prior to replacing it. The sg_ses utility uses indexing to pinpoint a single element.

There are two important pages to understand for indexing: the Configuration page and the Enclosure Status page. The Configuration page contains meta-information (information about information, similar to a database schema) while the Enclosure Status page contains 4-byte "elements". Amongst other things each disk slot in an enclosure has one 4-byte element in the Enclosure Status page; and that 4-byte restriction obviously limits the amount of information that can be held. So additional information is placed in extra pages that have a same element order as the Enclosure Status page. Those extra pages are the Threshold page (also with 4-byte elements) and the Element Descriptor page (which has variable length fields). And there is the Additional Element Status page which has fewer elements but the elements it does have, correspond to elements in the Enclosure Status page.

The sg_ses utility uses a two level index or a descriptor name to identify a single element within the Enclosure Status page. The first level index is a Type Header index or an Element Type abbreviation. The second level is an overall element indication ("-1") or an individual element index. All indexes (apart from the overall element indication) are origin 0.

Below is the Type Descriptor Header format taken from a SES-3 draft. There is an array of these in the Configuration page.

Figure 1 Taken from ses3r03.pdf at t10.org

It looks tempting to use the Element Type (an 8-bit number) as an index but the same element type can appear more than once. The example that the standard uses is having one "power supply" Element Type for 5 volt power supplies and another for 12 volt power supplies. So when the Element Type is used as an index it needs a sub-index to distinguish such cases. Each Type Descriptor Header has a corresponding "overall" element in the Enclosure Status page. In addition the Enclosure Status page has "Number of possible elements" of "individual" elements; note that may be zero. For indexing purposes the "Subenclosure identifier" field is ignored as is the "type descriptor text length" field.

As an example, let us assume a Configuration page has the following four Type Descriptor Headers (each number is placed in a byte):

23,3,0,0
2,2,0,0
2,1,0,0
134,0,0,0

The Type Header indexes are 0, 1, 2, 3 respectively; the Element Types are Array Device Slot, Power Supply, Power Supply and Vendor Specific LED display (element_type=134). The latter Element Type has been made up for this example. These four Type Descriptor Headers imply ten element instances in the Enclosure Status page: one overall instance for each Type Header (4) plus the sum of the "Number of possible elements" fields (3+2+1+0=6). The elements instances should appear in the order shown in the table below. The various ways that sg_ses can index these instances are shown in columns A, B, C, D, E and F:

Table 2 Indexing elements in the Enclosure Status page
Element instances	A --index=	B --index=	C --index=	D --index=	E --index=	F --index=
Array Device Slot overall element	0,-1	-1	arr,-1	arr	_23,-1	_23
Array Device Slot individual element 0	0,0	0	arr,0		_23,0
Array Device Slot individual element 1	0,1	1	arr,1		_23,1
Array Device Slot individual element 2	0,2	2	arr,2		_23,2
Power Supply (5v) overall element	1,-1		ps,-1	ps	_2,-1	_2
Power Supply (5v) individual element 0	1,0		ps,0		_2,0
Power Supply (5v) individual element 1	1,1		ps,1		_2,1
Power Supply (12v) overall element	2,-1		ps1,-1	ps1	_2_1,-1	_2_1
Power Supply (12v) individual element 0	2,0		ps1,0		_2_1,0
Vendor Specific LED display overall element	3,-1				_134,-1	_134

Note that only two columns can index the full ten instances: A and E. The simpler of the two is column A using the Type Header index followed by the individual index and this is the scheme used internally by sg_ses. Column C may be more intuitive but gets a bit messy with the second Power Supply Element Type (i.e. the 12v one) and can't index the Vendor Specific Element Type (i.e. the last row).

Numbers shown above are in decimal. Their hexadecimal equivalents may also be given either prefixed with '0x' (or '0X') or with a trailing 'h' (or 'H'). One consideration is that strings like "ah" ('a' in hex (10 in decimal)) appearing in a position that can either be an abbreviation or a number, will be interpreted as an abbreviation. Adding a leading '0' will cause it (e.g. "0ah") to be interpreted as a number.

Descriptor Names

Optionally a SES device may support the Element Descriptor page. If so that page will have the same number of elements as the Enclosure Status page. So it is possible that the descriptor names found in the Element Descriptor page. may be used as an alternative to the --index= option discussed in the previous section. The --descriptor=DN option was introduced for this purpose.

Apart from having a Element Descriptor page, the descriptor names need to be suitable. Some or all of the descriptor names might be empty which does not help indexing. Also the descriptor names might be so long (e.g. including part and serial numbers) as to make them unwieldy as indexes. A simple way to find out is to look at the descriptor names in the Element Descriptor page with:

$ sg_ses --page=ed /dev/sg3

Element descriptor In diagnostic page:

  generation code: 0x0

  element descriptor by type list

    Element type: Array device slot,
subenclosure id: 0 [ti=0]

      Overall descriptor:
ArrayDevicesInSubEnclsr0

      Element 0 descriptor:
ArrayDevice00

      Element 1 descriptor:
ArrayDevice01

      Element 2 descriptor:
ArrayDevice02

      ...

The above example shows descriptor names suitable for indexing. Given these descriptor names, one would expect these two commands to do the same thing, namely turn on the ident LED on the disk in slot 0:

$ sg_ses --index=arr,0 --set=ident /dev/sg3

$ sg_ses --descriptor=ArrayDevice00 --set=ident /dev/sg3

If an unrecognized descriptor name is given then response will be like this:

$ sg_ses --descriptor=xxx --set=ident /dev/sg3

descriptor name: xxx not found (check 'ed' page [0x7])

clear, get and set

These three command line options allow a named field within an element to be fetched or changed. These options can only be used on the Enclosure Status, Element Descriptor, Threshold In (or Out) and the Additional Element Status pages. Since the Element Descriptor and Additional Element Status pages are read-only only the --get= option can be used with them.

The Enclosure Status and the Enclosure Control pages have the same page number, the difference is that the "Status" page is used for fetching fields (i.e. the current state) while the "Control" page is used for changing fields. Other pages such as the Threshold page use "In" and "Out" in the same way (i.e. "In" to fetch the current state, "Out" to change it).

sg_ses attempts to name some often used fields, for example the "ident" field (also known as "locate" by some vendors) found in the Array Device Slot element . In some cases this can be a bit difficult because the naming of a field may differ from one Element Type to another. The "ident" field is in byte 2, bit position 1 and is 1 bit long in the Array Device Slot status element:

Figure 2 Taken from ses3r03.pdf at t10.org

Notice in the corresponding "Control" element that field is called "Rqst Ident". Not a major problem, but the naming discrepancy can be larger for example the "Fault Sensed" field (byte 3, bit 6, 1 bit long) in the Status page has no corresponding field in the Control page.

Figure 3 Taken from ses3r03.pdf at t10.org

When using the --clear, --get and --set options, the default page is the Enclosure Status/Control page (the Status page for --get, the Control page for --clear and --set). All of the following invocations attempt to set the "ident" field on the disk in slot 5 and all should cause its LED to flash:

$ sg_ses --index=arr,5 --set=ident /dev/sg3

$ sg_ses --index=arr,5 --set=ident=1 /dev/sg3

$ sg_ses --index=arr,5 --set=ident --page=ec /dev/sg3

$ sg_ses --index=5 --set=ident /dev/sg3

$ sg_ses --index=arr,5 --set=2:1 /dev/sg3

$ sg_ses --index=arr,5 --set=2:1=1 /dev/sg3

$ sg_ses --index=arr,5 --set=2:1:1=1 /dev/sg3

The latter three invocations use a numerical description of the field whose format is <start_byte>:<start_bit>[:<number_of_bits>] . The <number_of_bits> defaults to 1 when it is not given. To get a listing of the field names supported by sg_ses use the --enumerate option twice (or more simply: '-ee'). For example:

$ sg_ses -ee

--clear, --get, --set acronyms for enclosure status/control [0x2] page:

  ...

  ident  [Device slot] [2:1:1]

  ident  [Array device slot] [2:1:1]

  ident  [Power supply] [1:7:1]

  ident  [Cooling] [1:7:1]

  ...

  locate  [Device slot] [2:1:1]

  locate  [Array device slot] [2:1:1]

  locate  [Power supply] [1:7:1]

locate  [Cooling] [1:7:1]

  ...

Assuming there is only one power supply and it has an "ident" LED then this invocation should cause it to flash:

$ sg_ses --index=ps,0 --set=ident /dev/sg3

Notice that the "ident" field is in a different location in the Power Supply and Cooling Element Types (i.e. byte 1, bit 7).

Raw page writes

There are six SES diagnostic pages that can be modified: Enclosure Control, Threshold Out, String Out, Subenclosure String Out, Download Microcode Control and Subenclosure Nickname Control. Of those only the Enclosure Control and Threshold Out pages can be modified with the --set and --clear options as discussed in the previous section. So if we need to change one of the others then an alternative technique is required. However the use of this lower level technique is awkward involving three steps: read, modify then write. Following is an example of changing the String page using the --raw and --control options.

The current String In page can be shown with:

$ sg_ses --page=str /dev/bsg/6:0:2:0

Now the "read" step: the following command will send the contents of the String page (from byte 4 onwards) to stdout. The output will be in ASCII hex with pairs of hex digits representing a byte, 16 pairs per line, space separated. The redirection puts stdout in a file called "t":

$ sg_ses --page=str --raw /dev/bsg/6:0:2:0 > t

Then, perhaps with the aid of the SES-3 document (in SES-3 revision 3: section 6.1.6), use your favourite text editor to change t. The changes can be written to the device with:

$ sg_ses --page=str --control --data=- /dev/bsg/6:0:2:0 < t

If the above is successful, the String should have been changed. To check try:

$ sg_ses --page=str /dev/bsg/6:0:2:0

Note that the first four bytes of the fetched pages are not output when the --raw option is given. Likewise when the --control and --data options are given, sg_ses does not expect the first four bytes of the page to be provided. Byte 0 is the page number while bytes 2 and 3 contain the count of following bytes. Byte 1 is usually 0 but in some cases it may not be, hence the --byte1=B1 option.

In the Enclosure Control page the lower 4 bits of byte 1 contain the Info, Non-crit, Crit and Unrecov bit fields. To change these with sg_ses the --raw and --control technique can be used. Following is an example of setting the Info bit field (and zeroing the other byte 1 bits if they happened to be set):

$ sg_ses --page=es --raw /dev/bsg/6:0:2:0 > t

$ sg_ses --page=ec --byte1=0x8

--control --data=-
/dev/bsg/6:0:2:0 < t

In the Subenclosure String Out, Subenclosure Nickname Control and Download Microcode Control pages, byte 1 is the Subenclosure identifier.

Conclusion

The sg_ses utility was first written in 2004. It has gone through many iterations since then to fix bugs, allow more flexible access (e.g. indexing) and track large additions to the SES standards. SAS-2 expanders tend to have inbuilt SES devices so SES is becoming more common. Many of the improvements to sg_ses are thanks to reports and suggestions from users. Please continue to make those reports and suggestions.

Return to main page.

Last updated: 19th January 2012