The sg_ses utility

The sg_ses utility

Introduction

The sg_ses utility enables a user "to manage and sense the state of the power supplies, cooling devices, displays, indicators, individual drives, and other non-SCSI elements installed in an enclosure". The SCSI Enclosure Services standards (most recent is SES-2 ANSI INCITS 448-2008) and the latest draft (ses3r06.pdf at www.t10.org) describe the format that the sg_ses utility expects to find in a SES device ("logical unit" or "process").

The sg_ses utility is found in the sg3_utils package. The utility was originally written for Linux and has ports to FreeBSD, Solaris and Windows. This page outlines the features of the sg_ses utility version 1.99 found in the sg3_utils version 1.40 package.

The sg_ses_microcode utility was added in the sg3_utils version 1.40 package. It is designed to download microcode (firmware) to an enclosure and report on that download.

Overview

Typically the first problem to solve is finding an Operating System handle for a SES device. SCSI disks are typically easier to find but rarely is a SCSI disk also a SES device (only if the EncServ bit is set in the disk's INQUIRY command response). First let us try the sg_ses utility on a SAS disk (in Linux):

$ sg_ses /dev/sdb

        SEAGATE  
      ST33000650SS      0002

          disk device (not an enclosure)

      Supported diagnostic pages:

        Supported diagnostic pages [sdp] [0x0]

        Protocol specific (SAS transport) [] [0x3f]

        Translate address (SBC) [] [0x40]

Without any options sg_ses reports the names of the supported diagnostic pages. The SAS disk does support some pages but none that belong to the SES standard. On Linux it helps to list all SCSI devices (including showing the generic device names with the '-g' option; 'modprobe sg' may be needed if hyphens appear in the last column):

$ lsscsi -g

      [1:0:0:0]    disk   
      ATA     
      ST3320620AS      3.AA 
      /dev/sda   /dev/sg0

      [6:0:0:0]    disk    SEAGATE 
      ST33000650SS     0002 
      /dev/sdb   /dev/sg1

      [6:0:1:0]    disk    SEAGATE 
      ST32000444SS     0006 
      /dev/sdc   /dev/sg2

      [6:0:2:0]    enclosu Intel   
      RES2SV240        0600 
      -          /dev/sg3

The last item is an enclosure and notice that it does not have a disk device node (i.e. there is no /dev/sdd). In Linux the bsg driver could also be used and its device node would be /dev/bsg/6:0:2:0 for the enclosure. Now let us try sg_ses on /dev/sg3 :

$ sg_ses /dev/sg3

        Intel    
      RES2SV240         0600

      Supported diagnostic pages:

        Supported diagnostic pages [sdp] [0x0]

        Configuration (SES) [cf] [0x1]

        Enclosure status/control (SES) [ec,es] [0x2]

        String In/Out (SES) [str] [0x4]

        Threshold In/Out (SES) [th] [0x5]

        Element descriptor (SES) [ed] [0x7]

        Additional element status (SES-2) [aes] [0xa]

        Supported SES diagnostic pages (SES-2) [ssp] [0xd]

        Download microcode (SES-2) [dm] [0xe]

        Subenclosure nickname (SES-2) [snic] [0xf]

So now we see several diagnostic pages related to the SES (and SES-2) standard. Of these the most important are the Configuration page and the Enclosure Status/Control page. All the pages referred to below are actually diagnostic pages, so the term diagnostic will be dropped in many case for brevity.

Let us assume that a 6-disk SAS/SATA disk array is associated with this SES device. We might suspect the disk on the left of the array is "slot 0" but would like to check before removing the disk. One way to double check is to ask "slot 0" to identify itself by flashing its LED for 10 seconds. This can be done like this:

$ sg_ses --dev-slot-num=0 --set=ident /dev/sg3

      $ sleep 10

      $ sg_ses --dsn=0 --clear=ident /dev/sg3

In the latter invocation '--dsn=0' is a synonym for '--dev-slot-num=0'. Now check the disk on the right (slot 5), this time using shorter option variants ('-x 5' is equivalent to '--dev-slot-num=5'):

$ sg_ses -x 5 -S ident /dev/sg3

      $ sleep 10

      $ sg_ses -x 5 -C ident /dev/sg3

Typically there is a lot of information held by a SES device divided into pages. Several of those pages carry information with a somewhat similar structure to relational database tables. To see a lot of information try these commands (one after the other):

$ sg_ses --join /dev/sg3

      $ sg_ses --join --filter /dev/sg3

$ sg_ses -j -ff /dev/sg3

The first command will list information on all disks, fans, power supplies etc, even for disks and fans not installed. The second command attempts to filter out some less relevant information. The third command tries harder to filter out some less relevant information.

Command line options

sg_ses is a command line utility whose normal syntax is:

  sg_ses [options] DEVICE

An exception is the --enumerate option which doesn't require a DEVICE argument and ignores it if one is given. Below is a table of sg_ses options sorted alphabetically by the long option name:

Table 1 sg_ses command line options
short option	long option	Brief description
-b B1	--byte1=B1	byte 1 (2nd byte) of control page set to B1. Only active with --control
-C STR	--clear=STR	clear value of field described by STR. Requires an indexing option to choose an element.
-c	--control	send control information. That information is provided by --data= option
-d H,H...	--data=H,H...	H,H... is a string of ASCII hex bytes for control pages
-d -	--data=-	fetch that string of ASCII hex bytes from stdin
-D DN	--descriptor=DN	for indexing: find element whose Element Descriptor page's name entry matches DN. Implies --join
-x SN	--dev-slot-num= SN --dsn=SN	for indexing: find element whose Additional Element status page's device slot number entry matches SN. Implies --join. Found in sg_ses version 1.79 and later (from sg3_utils version 1.38). The --dsn synonym was added in sg_ses version 1.94 .
-E=A_F	--eiioe=A_F	A_F is either 'auto' or 'force'. Prior to the EIIOE bit's introduction in ses3r06 several arrays acted as if this bit was prsent and set. If 'force' is given the Additional Element Status page's element index field is treated as if EIIOE=1. If 'auto' is given a heuristic is used to decide whether to set EIIOE or not. Found in sg_ses version 1.88 and later (from sg3_utils version 1.39).
-e	--enumerate	list recognized diagnostic page codes and Element Type names. When used twice lists acronyms for the --clear, --get and --set options. Ignores other options and DEVICE argument
-f	--filter	reduce the amount of output when displaying pages (e.g. Enclosure Status and the Additional Element Status). Use twice to further reduce output
-G STR	--get=STR	get value of field described by STR. Requires an indexing option to choose an element.
-h	--help	print usage message then exit
-H	--hex	print the read diagnostic page in hexadecimal. Action varies depending on other options. For example with the --get=STR option prints the output value in hex.
-I IIA	--index=IIA	individual index ('-1' for overall) or Element Type abbreviation (e.g. 'arr')
-I TIA,II	--index=TIA,II	TIA is Type Header index or Element Type abbreviation; II is individual index ('-1' for overall)
-i	--inner-hex	decode page and print element(s) in hex
-j	--join	group ("join") Enclosure Status, Element Descriptor and Additional Element Status pages. Use twice to add Threshold In page
-l	--list	same action as --enumerate
-m LEN	--maxlen=LEN	maximum response length in bytes. Default is 65532 which may be too large for old systems
-M	--mask	--clear and --set do a read (status element), mask, modify, write cycle attempting to preserve other settings in the element. The mask step zeros parts of the status element that are reserved (or different) in the control element. This option drops the mask step (which was the behaviour before sg_ses version 1.99)
-N SEID	--nickid=SEID	SEID is subenclosure identifier (default: 0)
-n SEN	--nickname=SEN	SEN is the new subenclosure nickname
-p PG	--page=PG	SES diagnostic page code PG is an abbreviation or a number in decimal (prefix with '0x' (or 'h' suffix) for hex)
-r	--raw	print status page in ASCII hex suitable for --data=- ; when used twice outputs page in binary to stdout
-A SA	--sas-addr=SA	for indexing: find element whose Additional Element status page's SAS address entry matches SA. Will only match device slot and array device slot elements. Implies --join . Found in sg_ses version 1.79 and later (from sg3_utils version 1.38).
-S STR	--set=STR	set value of field described by STR. Requires an indexing option to choose an element.
-s	--status	fetch status information. The default action when no options given is to display the Supported Diagnostic Pages page. If this is the only option, the action is the same.
-v	--verbose	increase verbosity of output. Use multiple times for more debug information
-V	--version	print version number and its date then exit
-w	--warn	warn about join problems and some other issues. Output is sent to stderr

DEVICE should be a SCSI device (logical unit) whose INQUIRY response has its Peripheral Device Type field set to 13. Alternatively
it could be another Peripheral Device Type with the EncServ bit (also in the INQUIRY response) set. This latter case is now rare.

The manpage for sg_ses (use 'man sg_ses' at the command prompt) has more details on the command line options. For downloading microcode to a SES device, try 'man sg_ses_microcode' for details on its command line options.

Indexing

A SES device often contains a daunting amount of information. Often we are interested in a single element, for example flashing the ident LED on one disk drive carrier prior to replacing it. The sg_ses utility uses indexing to pinpoint a single element. Actually the single element may be several closely related elements that can be viewed as one. Currently one of four indexing methods can be used:

--index= option, based on the structure outlined in the Configuration page
--descriptor=DN where DN matches the Descriptor entry in the Element Descriptor page
--dev-slot-num=SN (or --dsn=SN) where SN matches the device slot number entry in the Additional Element Status page
--sas-addr=SA where SA matches an SAS address entry in the Additional Element Status page

The first method is low level indexing and is described in this section. It can be used to index any element provided by the SES device. The final three methods are medium level indexing and are described in the next section. The final three methods are preferable (i.e. simpler) but require optional support from the SES device (e.g. --descriptor=DN needs the Element Descriptor page with sensible names (e.g. unique)).

There are two important pages to understand for low level indexing: the Configuration page and the Enclosure Status page. The Configuration page contains meta-information (information about information, similar to a database schema) while the Enclosure Status page contains 4-byte "elements". Amongst other things each disk slot in an enclosure has one 4-byte element in the Enclosure Status page; and that 4-byte restriction obviously limits the amount of information that can be held. So additional information is placed in other diagnostic pages that have a same (or similar) element order as the Enclosure Status page. Those extra pages are the Threshold page (also with 4-byte elements) and the Element Descriptor page (which has variable length fields). And there is the Additional Element Status page which has fewer elements but the elements it does have, correspond to elements in the Enclosure Status page.

The sg_ses utility uses a two level index or a descriptor name to identify a single element within the Enclosure Status page. The first level index is a Type Header index or an Element Type abbreviation. The second level is an overall element indication ("-1") or an individual element index. All indexes (apart from the overall element indication) are origin 0.

Below is the Type Descriptor Header format taken from a SES-3 draft. There is an array of these in the Configuration page.

Figure 1 Taken from ses3r03.pdf at t10.org

It looks tempting to use the Element Type (an 8-bit number) as an index but the same element type can appear more than once. The example that the standard uses is having one "power supply" Element Type for 5 volt power supplies and another for 12 volt power supplies. So when the Element Type is used as an index it needs a sub-index to distinguish such cases. Each Type Descriptor Header has a corresponding "overall" element in the Enclosure Status page. In addition the Enclosure Status page has "Number of possible elements" of "individual" elements; note that may be zero. For indexing purposes the "Subenclosure identifier" field is ignored as is the "type descriptor text length" field.

As an example, let us assume a Configuration page has the following four Type Descriptor Headers (each number is placed in a byte):

23,3,0,0
2,2,0,0
2,1,0,0
134,0,0,0

The Type Header indexes are 0, 1, 2, 3 respectively; the Element Types are Array Device Slot, Power Supply, Power Supply and Vendor Specific LED display (element_type=134). The latter Element Type has been made up for this example. These four Type Descriptor Headers imply ten element instances in the Enclosure Status page: one overall instance for each Type Header (4) plus the sum of the "Number of possible elements" fields (3+2+1+0=6). The elements instances should appear in the order shown in the table below. The various ways that sg_ses can index these instances are shown in columns A, B, C, D, E and F:

Table 2 Indexing elements in the Enclosure Status page
Element instances	A --index=	B --index=	C --index=	D --index=	E --index=	F --index=
Array Device Slot overall element	0,-1	-1	arr,-1	arr	_23,-1	_23
Array Device Slot individual element 0	0,0	0	arr,0		_23,0
Array Device Slot individual element 1	0,1	1	arr,1		_23,1
Array Device Slot individual element 2	0,2	2	arr,2		_23,2
Power Supply (5v) overall element	1,-1		ps,-1	ps	_2,-1	_2
Power Supply (5v) individual element 0	1,0		ps,0		_2,0
Power Supply (5v) individual element 1	1,1		ps,1		_2,1
Power Supply (12v) overall element	2,-1		ps1,-1	ps1	_2_1,-1	_2_1
Power Supply (12v) individual element 0	2,0		ps1,0		_2_1,0
Vendor Specific LED display overall element	3,-1				_134,-1	_134

Note that only two columns can index the full ten instances: A and E. The simpler of the two is column A using the Type Header index followed by the individual index and this is the scheme used internally by sg_ses. Column C may be more intuitive but gets a bit messy with the second Power Supply Element Type (i.e. the 12v one) and can't index the Vendor Specific Element Type (i.e. the last row).

Numbers shown above are in decimal. Their hexadecimal equivalents may also be given either prefixed with '0x' (or '0X') or with a trailing 'h' (or 'H'). One consideration is that strings like "ah" ('a' in hex (10 in decimal)) appearing in a position that can either be an abbreviation or a number, will be interpreted as an abbreviation. Adding a leading '0' will cause it (e.g. "0ah") to be interpreted as a number.

Descriptor Names, Device Slot Numbers and SAS Addresses

Optionally a SES device may support the Element Descriptor page. If so that page will have the same number of elements as the Enclosure Status page. So it is possible that the descriptor names found in the Element Descriptor page may be used as an alternative to the --index= option discussed in the previous section. The --descriptor=DN option was introduced for this purpose.

Apart from having a Element Descriptor page, the descriptor names need to be suitable. Some or all of the descriptor names might be empty which does not help indexing. Also the descriptor names might be so long (e.g. including part and serial numbers) as to make them unwieldy as indexes. A simple way to find out is to look at the descriptor names in the Element Descriptor page with:

$ sg_ses --page=ed /dev/sg3

      Element descriptor In diagnostic page:

        generation code: 0x0

        element descriptor list (grouped by type):

          Element type: Array device slot, subenclosure
      id: 0 [ti=0]

            Overall descriptor:
      ArrayDevicesInSubEnclsr0

            Element 0 descriptor: ArrayDevice00

            Element 1 descriptor: ArrayDevice01

            Element 2 descriptor: ArrayDevice02

            ...

The above example shows descriptor names suitable for indexing. Given these descriptor names, one would expect these two commands to do the same thing, namely turn on the ident LED on the disk in slot 0:

$ sg_ses --index=arr,0 --set=ident /dev/sg3

      $ sg_ses --descriptor=ArrayDevice00 --set=ident /dev/sg3

If an unrecognized descriptor name is given then response will be like this:

$ sg_ses --descriptor=xxx --set=ident /dev/sg3

      descriptor name: xxx not found (check 'ed' page [0x7])

A SAS disk or tape can be indexed by its SAS address (also known as its target port identifier in SCSI jargon). SAS disks are typically dual ported and have two target ports whose SAS addresses are usually adjacent values. When a SATA disk is involved, it is usually connected via a SAS expander which typically assigns it an arbitrary "SAS" address. And that arbitrary "SAS" address is typically different from the SATA disk's WWN. One can see that in the following example where the SAS address (5001517e85c3efe1)is surprisingly close to that of the SAS expander (5001517e85c3efff). If that SATA disk was placed in another slot, the address would probably change (slightly) in sympathy. For indexing based on SAS addresses to work, the Additional Element Status page needs to be present.

# sg_ses --sas-addr=5001517e85c3efe1 /dev/sg3ArrayDevice01 [0,1] Element type: Array device slot Enclosure Status: Predicted failure=0, Disabled=0, Swap=0, status: OK OK=0, Reserved device=0, Hot spare=0, Cons check=0 In crit array=0, In failed array=0, Rebuild/remap=0, R/R abort=0 App client bypass A=0, Do not remove=0, Enc bypass A=0, Enc bypass B=0 Ready to insert=0, RMV=0, Ident=0, Report=0 App client bypass B=0, Fault sensed=0, Fault reqstd=0, Device off=0 Bypassed A=0, Bypassed B=0, Dev bypassed A=0, Dev bypassed B=0 Additional Element Status: Transport protocol: SAS number of phys: 1, not all phys: 0, device slot number: 1 phy index: 0 device type: no SAS device attached initiator port for: target port for: SATA_device attached SAS address: 0x5001517e85c3efff SAS address: 0x5001517e85c3efe1 phy identifier: 0x0

The 'attached SAS address' shown above is that of the SAS expander. From the above we can see that the SATA disk is associated with device slot number 1. So the ident flag can be set as follows:

# sg_ses --dev-slot-num=1 --set=ident /dev/sg3

Device slot numbers describe a position in a disk enclosure. They may also be available in SAS expanders often associated with SES devices. Some SAS-2 and all SAS-3 expanders should report device slot numbers. The number is a 1 byte value from 0 to 255 where 255 indicates there is no associated slot. Device slot numbers, if available, will be in the output of the smp_discover utility in the smp_utils package.

clear, get and set

These three command line options allow a named field within an element to be fetched or changed. These options can only be used on the Enclosure Status, Element Descriptor, Threshold In (or Out) and the Additional Element Status pages. Since the Element Descriptor and Additional Element Status pages are read-only only the --get= option can be used with them.

The Enclosure Status and the Enclosure Control pages have the same page number, the difference is that the "Status" page is used for fetching fields (i.e. the current state) while the "Control" page is used for changing fields. Other pages such as the Threshold page use "In" and "Out" in the same way (i.e. "In" to fetch the current state, "Out" to change it).

sg_ses attempts to name some often used fields, for example the "ident" field (also known as "locate" by some vendors) found in the Array Device Slot element . In some cases this can be a bit difficult because the naming of a field may differ from one Element Type to another. The "ident" field is in byte 2, bit position 1 and is 1 bit long in the Array Device Slot status element:

Figure 2 Taken from ses3r03.pdf at t10.org

Notice in the corresponding "Control" element that field is called "Rqst Ident". Not a major problem, but the naming discrepancy can be larger for example the "Fault Sensed" field (byte 3, bit 6, 1 bit long) in the Status page has no corresponding field in the Control page. Starting with sg_ses version 1.99 a mask is applied to the status element so that status element bits corresponding to Reserved field in the control element will be zero. For example if "Fault sensed" is set then some SES devices will object to a --set=ident because byte 3, bit 6 is reserved in the control element. The prior sg_ses behaviour (i.e. no mask applied) can selected with the --mask option.

Figure 3 Taken from ses3r03.pdf at t10.org

When using the --clear, --get and --set options, the default page is the Enclosure Status/Control page (the Status page for --get, the Control page for --clear and --set). All of the following invocations attempt to set the "ident" field on the disk in slot 5 and all should cause its LED to flash:

$ sg_ses --index=arr,5 --set=ident /dev/sg3

      $ sg_ses --index=arr,5 --set=ident=1 /dev/sg3

      $ sg_ses --index=arr,5 --set=ident --page=ec /dev/sg3

$ sg_ses --index=5 --set=ident /dev/sg3

$ sg_ses --index=arr,5 --set=2:1 /dev/sg3

      $ sg_ses --index=arr,5 --set=2:1=1 /dev/sg3

      $ sg_ses --index=arr,5 --set=2:1:1=1 /dev/sg3

The latter three invocations use a numerical description of the field whose format is <start_byte>:<start_bit>[:<number_of_bits>] . The <number_of_bits> defaults to 1 when it is not given. To get a listing of the field names supported by sg_ses use the --enumerate option twice (or more simply: '-ee'). For example:

$ sg_ses -ee

      --clear, --get, --set acronyms for enclosure status/control [0x2]
      page:

        ...

        ident  [Device slot] [2:1:1]

        ident  [Array device slot] [2:1:1]

        ident  [Power supply] [1:7:1]

        ident  [Cooling] [1:7:1]

        ...

        locate  [Device slot] [2:1:1]

        locate  [Array device slot] [2:1:1]

        locate  [Power supply] [1:7:1]

locate  [Cooling] [1:7:1]

        ...

Assuming there is only one power supply and it has an "ident" LED then this invocation should cause it to flash:

$ sg_ses --index=ps,0 --set=ident /dev/sg3

Notice that the "ident" field is in a different location in the Power Supply and Cooling Element Types (i.e. byte 1, bit 7) compared to the Device Slot and Array Device Slot Element Types (i.e. byte 2, bit 1).

To be absolutely sure of what is written to a control element, the whole 4 bytes (32 bits) can be written explicitly:

$ sg_ses --dsn=3 --set=0:7:32=0x200 /dev/sg3

That will set the RQST IDENT bit in the Array device slot control element whose device slot number is 3.

Raw page writes

There are six SES diagnostic pages that can be modified: Enclosure Control, Threshold Out, String Out, Subenclosure String Out, Download Microcode Control and Subenclosure Nickname Control. Of those only the Enclosure Control and Threshold Out pages can be modified with the --set and --clear options as discussed in the previous section. So if we need to change one of the others then an alternative technique is required. However the use of this lower level technique is awkward involving three steps: read, modify then write. Following is an example of changing the String page using the --raw and --control options.

The current String In page can be shown with:

$ sg_ses --page=str /dev/bsg/6:0:2:0

Now the "read" step: the following command will send the contents of the String page (from byte 4 onwards) to stdout. The output will be in ASCII hex with pairs of hex digits representing a byte, 16 pairs per line, space separated. The redirection puts stdout in a file called "t":

$ sg_ses --page=str --raw /dev/bsg/6:0:2:0 > t

Then, perhaps with the aid of the SES-3 document (in SES-3 revision 3: section 6.1.6), use your favourite text editor to change t. The changes can be written to the device with:

$ sg_ses --page=str --control --data=- /dev/bsg/6:0:2:0 < t

If the above is successful, the String should have been changed. To check try:

$ sg_ses --page=str /dev/bsg/6:0:2:0

Note that the first four bytes of the fetched pages are not output when the --raw option is given. Likewise when the --control and --data options are given, sg_ses does not expect the first four bytes of the page to be provided. Byte 0 is the page number while bytes 2 and 3 contain the count of following bytes. Byte 1 is usually 0 but in some cases it may not be, hence the --byte1=B1 option.

In the Enclosure Control page the lower 4 bits of byte 1 contain the Info, Non-crit, Crit and Unrecov bit fields. To change these with sg_ses the --raw and --control technique can be used. Following is an example of setting the Info bit field (and zeroing the other byte 1 bits if they happened to be set):

$ sg_ses --page=es --raw /dev/bsg/6:0:2:0 > t

      $ sg_ses --page=ec --byte1=0x8

--control --data=-
      /dev/bsg/6:0:2:0 < t

In the Subenclosure String Out, Subenclosure Nickname Control and Download Microcode Control pages, byte 1 is the Subenclosure identifier.

To set the Subenclosure nickname, the '--nickid=SEID' and '--nickname=SEN' options can be used. To download microcode, the sg_ses_microcode utility is recommended.

Conclusion

The sg_ses utility was first written in 2004. It has gone through many iterations since then to allow more flexible access (e.g. indexing), track large additions to the SES standards and fix bugs. SAS-2 and SAS-3 expanders tend to have inbuilt SES devices so SES is becoming more common. Many of the improvements to sg_ses are thanks to reports and suggestions from users. Please continue to provide that feedback.

Return to main page.

Last updated: 13th November 2014