Smartmontools for SCSI devices

Smartmontools controls and monitors storage devices using the Self-Monitoring, Analysis and Reporting Technology (SMART) system. This toolset was originally built for the Linux operating system and has been ported to Darwin for Mac OS X (no SCSI support yet), FreeBSD, NetBSD, OpenBSD, OS/2 (no SCSI support), Solaris and Windows. This article describes how smartmontools interacts with SCSI devices. Passing reference is also made to devices that use the SCSI command set such as USB mass storage devices and IEEE1394 devices that use the "sbp2" protocol. In many situations SATA disks are accessed using a (partial) SCSI command set.

The primary web site for smartmontools is at smartmontools.sourceforge.net from which the latest versions (both source and binaries) can be obtained. Smartmontools grew out of the now dormant smartsuite project which is still available on its own sourceforge site. The smartmontools main page concentrates on ATA devices. This article supplies some SCSI specific information for those users of smartmontools that wish to monitor SCSI storage devices.

This document outlines the features found in smartmontools version 5.37 that are relevant to SCSI disks and tape drives. This document was last altered on 22nd October 2006.

Smartmontools is made up of two executable programs, a configuration file and online documentation (on Unix systems in the form of "man" pages). The two executable programs are:

SCSI disks and tape drives allow self tests of their media, often monitor the temperature of the device, maintain error counters and report when various failure prediction thresholds are exceeded. To view the information available try a command like: smartctl -a /dev/sda. If SMART reporting has not been turned on for this disk then use this command first: smartctl -s on /dev/sda. [For operating systems other than Linux replace /dev/sda with a SCSI disk device name.]

The smartd daemon program is a service typically started when a machine boots up. In can monitor multiple disks (both ATA and SCSI). In Unix systems its configuration file can be found /etc/smartd.conf. It sends alerts to the system logs and can be configured to email system administrators when pending failures are reported.

Smartmontools was originally written for Linux. Since then it has been ported to various other Unix based systems and Windows. Note that the device names are based on the transport that an operating system sees. These days it is not uncommon for an operating system to see a transport that only conveys SCSI commands connected, via some command translation bridge, to an ATA disk. Examples are USB external disk enclosures and SATA disks behind a SCSI to ATA Translation Layer (SATL) in a SAS or FC domain.

The names of SCSI disk and tape devices vary with the operating system. Here is a summary:

The above list is a simplification. In Linux there can be multiple drive letters followed by a partition number (1 to 15). Smartmontools will ignore the partition number if it is given and query the underlying device. In Linux the SCSI tape device name can be "nst" and a letter can be appended to the device name, both decorations are ignored by smartmontools as it accesses the underlying tape drive. Also in Linux, SCSI devices can be accessed via their generic name which is of the form /dev/sg[0-9].

Linux also has an optional Solaris like naming scheme for SCSI device (scsidev), devfs (mainly used in the lk 2.4 series) and udev (devfs's replacement in the lk 2.6 series). In short, device naming is a complex area and smartmontools does its best to find and identify (i.e. whether ATA or SCSI) a device depending on its name. In some cases smartmontools needs guidance from the user and this can be given by the '-d ata|scsi|sat|marvell|3ware,N' option in the smartctl utility and in smartd daemon's configuration file.

Windows has several schemes for naming devices. The "scsi[0-9][0-f]" scheme uses the aspi dll from Adaptec. That dll is not distributed with Windows. The other schemes use the "SCSI Pass Through" interface which is native to Windows in NT and later. In all cases for Windows, the leading /dev/ is optional.

What is a SCSI disk? A SCSI disk is a storage device that "talks" the SCSI command set. An ATA disk is a storage device that "talks" the ATA command set. That seems pretty clear. However the command set that a disk uses at its connector (and thus shown on its label) may not be the command set that the operating system needs to use due to command set translation between the OS and the disk.

The ATA command set is used over native ATA transports which are parallel ATA (PATA) up to 133 MB/sec and serial ATA (SATA) at link speeds of 1.5 Gbps (approximately 150 MB/sec) or 3 Gbps. In the past when ATA disks needed to use some other transport (e.g. USB and IEEE1394) the SCSI command set was sent over the foreign transport. So in this case the operating system sees a device "talking" the SCSI command set but the device is really an ATA disk. Many current disk external enclosures contains ATA disks yet seen from the operating systems view point are USB mass storage devices talking the SCSI command set.

The SCSI command set is used over various transports: the SCSI Parallel Interface (SPI), Fibre Channel (FCP), Serial Attached SCSI (SAS), IEEE1394 (SBP), USB (mass storage) and iSCSI. Many of these transports can convey multiple command sets (i.e. not just the SCSI command set). The SAS transport is interesting as it can convey both the SCSI and ATA command sets. There is also the case of a RAID made up of ATA disks which communicates to host operating system with the SCSI command set (e.g. 3ware RAID controller).

So what does all this mean for smartmontools? In most cases the answer is not good news. Devices such as USB external disk enclosures translate incoming (from the host) SCSI commands to their ATA equivalents and process responses as required. This translation is limited typically to a small number of SCSI commands (e.g. READ and WRITE) but not those commands needed by smartmontools. The author does not know of any SCSI_over_USB devices that support Smartmontools. The 3ware RAID (6000, 7000, 8000 and 9000 series Escalade) controllers are supported on several operating systems with special code. ^[1]

There is an emerging SCSI to ATA Translation (SAT) standard at www.t10.org that may lead to improvements in this area. Apart from defining some of the facilities smartmontools needs, it defines two ATA PASS THROUGH SCSI commands. These pass through commands could be used in much the same way that the 3ware RAID tunnels ATA commands.

The device type '-d sat' instructs the smartctl command and the smartd daemon, to form SMART commands for the ATA command set and then package those commands within the ATA PASS THROUGH SCSI commands. The SCSI commands are then sent to the "SCSI" device that the operating system has been given. In version 5.37 of smartmontools it is no longer necessary to specify '-d sat' in this situation. All that is needed is a SATL that complies with the emerging SAT standard. If the automatic detection of an ATA disk behind a SATL is tricked, '-d scsi' (or some other device type) can be used to override.

It has been reported that many external USB enclosures use a "Cypress" chipset. This contains an ATACB proprietary pass through (for ATA commands passed through SCSI commands) for which some publicly available information is available. Smartmontools has no ATACB specific code but may move in this direction in the future. Another approach is to hope USB and SBP2 external enclosures adopt the SAT standard in the near future. One interesting comment about ATACB is that it should not be used at the same time as other types of access to the disk (e.g. a mounted file system)! That implies that a disk should be taken offline before smartmontools is used on it. It also implies that the smartd background daemon should not be used.

SATA disks use a 1.5 or 3 Gbps serial transport which carries the ATA command set. The serial connection is point to point so each SATA disk needs its own cable and plug on the host adapter or motherboard. ^[2] Many aspects of SATA are like SCSI and some operating systems use existing SCSI infrastructure to handle SATA hosts (e.g. Linux's libata).

Serial Attached SCSI (SAS) can be viewed as a superset of SATA. It can directly connect thousands of SAS disks to one or more controllers spread across multiple machines in one SAS "domain". Such a domain can also contain SATA disks, connected to intermediate fanout devices called expanders (similar to switches in networking). Most SAS host adapters can also have SATA disks connected directly to the adapter (which technically is not a usage of SAS but that is of little concern to the end user).

So a SATA disk may be connected

Since all but the first item might have other disks connected which use the SCSI command set (e.g. SAS and FC disks) often the SATA disks have a SAT layer put in front of them so they look like SCSI disks. That SAT layer may be in:

For normal file system work, a SCSI to ATA Translation Layer (SATL) only needs to concern itself with around 6 commands. Unfortunately smartmontools uses other commands (both in the SCSI and ATA command sets). Probably the simplest way to handle SMART for SATA disks behind a SAT layer is to use the ATA PASS THROUGH SCSI commands.

smartmontools guesses the disk command set (i.e. ATA or SCSI) based on the device node it is given. For example in Linux, /dev/hda would be assumed to use the ATA command set while /dev/sda would be assumed to use the SCSI command set. ^[3] By using either the '-d ata' or the '-d scsi' option, the command set guess made by smartmontools can be overridden. The '-d sat' device type causes smartmontools to generate ATA commands which are then packaged within the ATA PASS THROUGH SCSI commands (defined by the SAT standard) and then sent to the device via a SCSI pass through mechanism. As noted in the previous section, version 5.37 of smartmontools now automatically detects a SATA disk behind a SAT layer and acts as if '-d sat' has been given.

SMART never attained the status of "standard" and its original documents have been withdrawn. Its catchy name lives on, especially on vendors' web sites and obviously in the name of this toolset. Luckily the good ideas in SMART have been incorporated into the ATA and SCSI standards albeit in slightly different forms.

Initially SMART began on SCSI disks as vendor specific extensions. Gradually the SMART functionality has moved into the standards (often by other names) and vendors are improving their standards' compliance. [In the vendors' defence some of the "standards" are drafts and are yet to be ratified.] Some SCSI disk vendors have product manuals (available on the net) that cover the parts of the SCSI command set that their disks support. Some of these manuals fill in details that are left deliberately vague in the the standards. ^[4]

SCSI standards (found at www.t10.org) only make one footnote reference to the term SMART. In its place the awkward term "Informational Exceptions" is used. For SCSI tapes the term "TapeAlert" is used.

The smartctl command line utility gets SMART information from the nominated device. In some cases SMART information held by the nominated device can be modified by the smartctl command. The command has many options that can be viewed by the long usage message output be either of these invocations: smartctl -h or smartctl --help. Those options that are only available to ATA disks (i.e. not available to SCSI disks or tape drives) are marked with "(ATA)". Unix style "man" page documentation is also available.

The following options are currently available for SCSI disks and tape drives unless otherwise noted:

After the options smartctl expects a device name. This device name is not required for the '--help' or '--version' options. If no options are given and a valid device name is given then the copyright notice is output and the program exits. If the device name is invalid then that is reported. Only one device name can be given.

Examples of various invocations of smartctl on a SCSI disk follow:

# smartctl -i /dev/sdc
smartctl version 5.37 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Device: SEAGATE  ST336754SS       Version: 0003
Serial number: xxxxxxxx
Device type: disk
Transport protocol: SAS
Local Time is: Fri Apr 28 15:55:34 2006 EDT
Device supports SMART and is Enabled
Temperature Warning Enabled

# smartctl -H /dev/sdd
smartctl version 5.37 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

SMART Health Status: O

# smartctl -A /dev/sdc
smartctl version 5.37 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Current Drive Temperature:     42 C
Drive Trip Temperature:        68 C
Elements in grown defect list: 0
Vendor (Seagate) cache information
  Blocks sent to initiator = 1666124337
  Blocks received from initiator = 1517744621
  Blocks read from cache and sent to initiator = 384030649
  Number of read and write commands whose size <= segment size = 21193148
  Number of read and write commands whose size > segment size = 1278317
Vendor (Seagate/Hitachi) factory information
  number of hours powered up = 277.08
  number of minutes until next internal SMART test = 108

Rather than wait for thresholds to be tripped, an administrator can request a self test. Alternatively a self test can be scheduled periodically (e.g. at 3 a.m. every night or perhaps weekly) with smartd. All SCSI disks and tape drives should support a default self test since it is mandatory. This can be invoked with the smartctl -t offline <device> command. Despite the term "offline" this is actually a foreground test of less than 2 minutes. On completion the default self test reports any errors detected in its response. The default self test makes no entry into the self test log. Most SCSI devices perform a default self test when they are being powered up.

The other self tests that are optionally supported by the device are listed here with the smartctl invocation in brackets:

Short self tests should take less than two minutes to complete. The extended self tests have been known to take more than one hour for disks that are over 100 GBytes in size. Care should be taken with foreground tests on disks with mounted file systems as the OS may not take kindly to an hour delay on a simple READ command. ^[5]

Background self tests can be aborted with the smartctl -X <device> command. The self test log will note that an abort was requested.

Self tests other than the default self test cause an entry to be placed in the self test results log page. The 20 most recent self tests are held. The self test results can be viewed with the smartctl -l selftest <device> command. All tests output the accumulated power on hours when the test was performed and the success or otherwise (e.g. the self test was aborted by the user's request) of the test. Unsuccessful self tests output a self test segment number (vendor specific), the logical block address of the first failure (if appropriate) and a sense_key,asc,ascq triple (see appendix). Following the self test result table is the expected duration of an uninterrupted extended self test (when that figure is provided by the device).

Here is an example of a self test log:

# smartctl -l selftest /dev/sdd
smartctl version 5.37 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/


SMART Self-test log
Num  Test              Status                 segment  LifeTime  LBA_first_err [SK ASC ASQ]
     Description                              number   (hours)
# 1  Background long   Completed                   -     100                 - [-   -    -]
# 2  Background long   Completed                   -      25                 - [-   -    -]
# 3  Background long   Completed                   -      24                 - [-   -    -]
# 4  Background short  Completed                   -       0                 - [-   -    -]

Long (extended) Self Test duration: 603 seconds [10.1 minutes]

The smartctl -l error <device> command displays the error counters maintained in the device's log pages. Here is an example of an error log:

# smartctl -l error /dev/sdd
smartctl version 5.37 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/


Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:       5805        0         0      5805       5805        121.451           0
write:         0        0         0         0          0        471.291           0

Non-medium error count:        0

The displayed error logs (if available) are displayed on separate lines:

Each of the write, read and verify error counter logs has various parameters codes. They are itemized below with the smartctl column name followed, in brackets, with SCSI standard's description and parameter code). A description taken from Seagate's SCSI manual (publication 77738479, Rev J) is then given.

The SCSI standard (SPC-3) cautions that the exact definitions of the error counters is not part of the standard (i.e. they are vendor specific). As noted the above list contains Seagate's explanation for its disk products (the last revision of that document was 1999). Seagate's disk product manuals imply that the disk firmware collects these counter values and periodically commit them to persistent storage (disk or non-volatile RAM). ^[6] They also imply that their firmware is monitoring these error counters and if they exceed some threshold (e.g. in a certain time interval) then the firmware will report a thresholds exceeded.

The error counter logs for some disks (e.g. some Seagate models) can look worrying:

# smartctl -l error /dev/sdc
smartctl version 5.37 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/


Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:    1111396        0         0   1111396    1113203        781.138          13
write:         0        0         0         0         92        822.450           4
verify:   341115        0         0    341115     341115         42.159           0

Non-medium error count:        1

The "fast" ECC corrected number is high. However the '-H' option reports the disk is in good health as does an extended (long) background self test. The uncorrected errors would be a problem had in not been for the fact that the author caused them on purpose (by writing a bad sector with the SCSI WRITE LONG command).

Recent SCSI disks can perform what are termed as "background scans". These are reads of the whole media with recoverable errors acted on and unrecoverable errors noted. If a sector (block) is found with a recoverable error (i.e. the error correction codes (ECC) detect a problem but contain enough redundant information to fix the problem) it may be fixed with a re-write "in place". Alternatively the disk may decide to re-assign the recovered data to another physical sector which is assigned the same logical block address (and the original faulted sector is unmapped and placed on the grown defect list (GLIST)). Since unrecoverable errors potentially involve user data being lost, no automatic recovery action is undertaken by the disk. However logical block addresses that contain either recovered data or unrecoverable errors are noted in the Background Scan Results log page. The smartctl --log=background command decodes and outputs that log page.

Background scans may be performed periodically (e.g. every 24 hours) or every time the disk is powered up (or both). These parameters can be controlled via the Background Control mode page. The sdparm utility can be used to access and modify this mode page.

Here is an example of the output from the Background Scan Results log page. The first descriptor in that log page shows the status followed by up to 2048 entries for background scan "events". In this case a background scan is still in progress and 3 scans have been completed in the past. The "events" shown are all recoverable errors that the disk dealt with by rewriting the block.

# smartctl -l background /dev/sda
smartctl version 5.37 [i686-pc-linux-gnu] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/

Background scan results log
  Status: scan is active
    Accumulated power on time, hours:minutes 618:01 [37081 minutes]
    Number of background scans performed: 3,  scan progress: 59.81%

   #  when        lba(hex)    [sk,asc,ascq]    reassign_status
   1  617:13  0000000001fbc5b2  [1,17,1]   Recovered via rewrite in-place
   2  617:13  00000000022756d2  [1,17,1]   Recovered via rewrite in-place
   3  617:14  000000000227727f  [1,17,1]   Recovered via rewrite in-place
   4  617:18  00000000023568e5  [1,17,1]   Recovered via rewrite in-place
   5  617:22  00000000024fab5f  [1,17,1]   Recovered via rewrite in-place
   6  617:23  00000000025aa29a  [1,17,1]   Recovered via rewrite in-place
   7  617:27  000000000275d0bc  [1,17,1]   Recovered via rewrite in-place

In this case the reassign_status shows that no user intervention is required. The other "don't worry (too much)" reassign_status is "Logical block successfully reassigned". Any other reassign_status will require user intervention to correct. There is a LOWIR ("log only when intervention required") bit in the Background Control mode page that the user can set (e.g. with the sdparm utility) to filter out "noisy" entries like those shown above.

The user can manually re-assign logical blocks with a utility like sg_reassign found in the sg3_utils package. The background scan output contains a "[sk,asc,ascq]" tuple of numbers. The one shown above translates to "recovered error, recovered data with retries". Unrecoverable errors would most likely have 3 ("medium eror") or 4 ("hardware error") as the first number. A decoding of the latter two numbers can be found in the "Numeric Order Codes" annex of SPC-4 (see www.t10.org) in the Additional Sense Codes section.

smartd is a daemon for monitoring disks (both ATA and SCSI). It is recommended that tape drives and medium changers are monitored in a more manual fashion with the smartctl command as discussed in the section called “TapeAlert”.

The configuration file for smartd is called /etc/smartd.conf and has a man page (as does the smartd command). The controlling daemon script is placed in the normal place for a distribution, typically /etc/rc.d/init.d/smartd.

smartd polls the devices it has recognized when it was started. By default it polls every 30 minutes. It reports any adverse finding and noteworthy occurrences (e.g. disk drive temperature changes) to a log file (/var/log/messages). smartd can be configured to take other actions, for example send email to a system administrator.

SCSI disks can be discovered by smartd via a scan of device nodes (for linux: /dev/sda through to /dev/sdz) by placing the word "DEVICESCAN" in /etc/smartd.conf file. Alternatively the "DEVICESCAN" word can be removed (or commented out) and SCSI devices named explicitly:

/dev/sda -d scsi
/dev/sdb -d scsi

The "-d scsi" argument overrides what smartd would guess as the device class (i.e. "ata", "scsi", "sat", "marvell", "3ware,N", "hpt,L/N[,M]" or "cciss,N").

TapeAlert (or "tape alerts") is closely related to the SMART infrastructure provided for SCSI disks. TapeAlert is specialized for tape and medium changer devices. An example of a TapeAlert is an indication that the tape drive heads need to be cleaned.

Pending TapeAlert errors can be read from the TapeAlert log page (using smartctl). This can be done even when SMART monitoring is disabled (e.g. after smartctl -s off <tape_device >). In fact, the best way to use the TapeAlert mechanism is to poll the flags (with smartctl) at relevant times when using the tape, for example:

The TapeAlert information is divided into three severity classes: Critical, Warning, and Information. The critical messages require urgent user intervention. Both critical and warning errors may lead to loss of data. Some of the errors are related to the medium and others to the tape drive itself. This is why the TapeAlert information should be checked when the tape is in use and not polled periodically (i.e. the smartd daemon with its periodic polling is not particularly useful for TapeAlert mechanism).

Different sets of flags are defined for tape drives and media changers. Most of the flags are optional and the set of flags supported depends on the device. TapeAlert is being included into the SCSI-3 standards. Many SCSI-2 drives support TapeAlert but the implementation may not fully conform to the SCSI-3 draft definition used by smartmontools.

It is important that only one application (or OS driver) is monitoring tape alerts since reading the TapeAlert log page deactivates all flags after they are read. ^[7] Currently the Linux SCSI tape drivers (st and osst) do not check the TapeAlert log page. In Linux, a medium changer device (i.e. the robot in a tape jukebox) is accessed via its SCSI generic (sg) device name.

Code and information on the TapeAlert mechanism have been provided by Kai Mäkisara <Kai.Makisara at kolumbus dot fi>.

Here is some output from the smartctl command. Mostly it is for the '--all' option.

It is unlikely that a hardware RAID controller will directly support smartmontools. A SCSI RAID controller is a virtual target device that essentially remaps the SCSI commands it receives to the physical disks on its internal buses. The physical disks in a "SCSI" RAID could be ATA or sATA disks, in this case a SCSI bus is used between the host computer and an external RAID controller since LVD SCSI buses (SPI-2,3 and 4) can run up to 25 metres (plus other protocol related issues).

Some SCSI RAIDs equipped internally with SCSI disks allow access to the physical disks via logical unit numbers (LUNs) greater than 0. The SCSI RAID controller itself takes a LUN equal to 0. In this case smartmontools could be applied to the LUNs greater than 0 that refer to physical disks.

Some SCSI RAIDs equipped internally ATA disks have a mechanism that allows ATA commands to be tunnelled to the ATA disks. The 3ware 6000 and 7000 series Escalade controllers are examples. In this case, special provision has been made in smartmontools (starting with release 5.1-16) to tunnel the ATA command required through to the physical disks. This is done by using the -d 3ware,N option/Directive. See the smartctl and smartd man pages for details.

The approach that smartmontools takes is to communicate directly with physical storage devices (e.g. a disk). Another approach is to collectively monitor and manage a group of disks and/or tape drives (be they a RAID, "Just a Bunch Of Disks" JBOD or a collection of disks and tape drives) in an enclosure. The SCSI Enclosure Services SES (reference: SES-2 at www.t10.org) is designed for this task. Both SCSI device and recent SATA disk enclosures are using SES. Amongst other things SES can monitor the state of individual devices within the enclosure, the temperature, power supplies and fans. A user can set thresholds, define alarm types and remotely administer the enclosure.