The Linux SCSI Generic (sg) HOWTO

This must be set to 'S' (capital ess). If not, the ENOSYS error message is placed in errno. The idea is to allow interface variants in the future that identify themselves with a different value. [The parallel port generic driver (pg) uses the letter 'P' to identify itself.] The type of interface_id is int.

The type of dxfer_direction is int. This is required to be one of the following:

The value SG_DXFER_NONE should be used when there is no data transfer associated with a command (e.g. TEST UNIT READY). The value SG_DXFER_TO_DEV should be used when data is being moved from user memory towards the device (e.g. WRITE). The value SG_DXFER_FROM_DEV should be used when data is being moved from the device towards user memory (e.g. READ).

The value SG_DXFER_TO_FROM_DEV is only relevant to indirect IO (otherwise it is treated like SG_DXFER_FROM_DEV). Data is moved from the user space to the kernel buffers. The command is then performed and most likely a READ-like command transfers data from the device into the kernel buffers. Finally the kernel buffers are copied back into the user space. This technique allows application writers to initialize the buffer and perhaps deduce the number of bytes actually read from the device (i.e. detect underrun). This is better done by using 'resid' if it is supported.

The value SG_DXFER_UNKNOWN is for those (rare) situations where the data direction is not known. It may be useful for backward compatibility of existing applications when the relevant direction information is not available in the sg interface layer. There is a (minor) performance "hit" associated with choosing this option (e.g. on the PCI bus). Some recent pseudo device drivers (e.g. USB mass storage) may have problems handling this value (especially on vendor-specific SCSI commands).

N.B. 'dxfer_direction' must have one of the five indicated values and cannot be uninitialized or zero.

If 'dxfer_len' is zero then all values are treated like SG_DXFER_NONE.

This is the length in bytes of the SCSI command that 'cmdp' points to. As a SCSI command is expected an EMSGSIZE error number is produced if the value is less than 6 or greater than 16. Further, if the SCSI mid level has a further limit then EMSGSIZE is produced in this case as well. ^[5] The type of cmd_len is unsigned char.

This is the maximum size that can be written back to the 'sbp' pointer when a sense_buffer is output which is usually in an error situation. The actual number written out is given by 'sb_len_wr'. In all cases 'sb_len_wr' <= 'mx_sb_len' . The type of mx_sb_len is unsigned char.

This is the number of scatter gather elements in an array pointed to by 'dxferp'. If the value is zero then scatter gather (in the user space) is _not_ being used and 'dxferp' points to the data transfer buffer. If the value is greater than zero then each element of the array is assumed to be of the form:

            typedef struct sg_iovec
            {
                void * iov_base; /* starting address */
                size_t iov_len;  /* length in bytes */
            } sg_iovec_t;

Note that this structure has been named and defined in such a way to parallel "struct iovec" used by the readv() and writev() system calls in Linux. See "man 2 readv".

Note that the scatter gather capability offered by 'iovec_count' is unrelated to the scatter gather capability (often associated with DMA) offered by most modern SCSI adapters. Furthermore iovec_count's variety of scatter gather (into the user space) is only available when normal (or "indirect") IO is being used. Hence when the SG_FLAG_DIRECT_IO or SG_FLAG_MMAP_IO are set in 'flags' then 'iovec_count' should be zero.

The type of iovec_count is unsigned short.

This is the number of bytes to be moved in the data transfer associated with the command. The direction of the transfer is indicated by 'dxfer_direction'. If 'dxfer_len' is zero then no data transfer takes place. ^[6]

If iovec_count is non-zero then 'dxfer_len' should be equal to the sum of iov_len lengths. If not, the minimum of the two is the transfer length. The type of dxfer_len is unsigned int.

If 'iovec_count' is zero then this value is a pointer to user memory of at least 'dxfer_len' bytes in length. If there is a data transfer associated with the command then the data will be transferred to or from this user memory. If 'iovec_count' is greater than zero then this value points to a scatter-gather array in user memory. Each element of this array should be an object of type sg_iovec_t. Note that data is sometimes written to user memory (e.g. from a failed SCSI READ) even when an error has occurred.

If mmap-ed IO is selected then the value in 'dxferp' is ignored and any data transfers will be to and from the address returned by the prior mmap() call.

The type of dxferp is void * .

This value points to the SCSI command to be executed. The command is assumed to be 'cmd_len' bytes long. If cmdp is NULL then the system call yields an EMSGSIZE error number. The user memory pointed to is only read (not written to). The type of cmdp is unsigned char * .

This value points to user memory of at least 'mx_sb_len' bytes length where the SCSI sense buffer will be output. Most successful commands do not output a sense buffer and this will be indicated by 'sb_len_wr' being zero. Note that there are error conditions that don't result in a sense buffer be generated. The sense buffer results from the "autosense" mechanism in the SCSI mid-level or lower level driver. This mechanism detects a CHECK_CONDITION status and either

The type of sbp is unsigned char * .

This value is used to timeout the given command. The units of this value are milliseconds. The time being measured is from when a command is sent until when sg is informed the request has been completed. A following read() can take as long as the user likes. Timeouts are best avoided, especially if SCSI bus resets will adversely effect other devices on that SCSI bus. When the timeout expires, the SCSI mid level attempts error recovery. Error recovery completes when the first action in the following list is successful. Note that a more extreme measure is being taken at each step.

If all these fail then the device may be set "offline" which means that it is no longer accessible (except by this driver when open()-ed O_NONBLOCK) until the machine is rebooted. Offline devices still appear in the cat /proc/scsi/scsi listing. The last column of the cat /proc/scsi/sg/devices listing shows the online/offline status of a device ("1" means online while "0" is offline). The exact status returned depends on which level of error recovery succeeded. Most likely the 'host_status' will be set to DID_ABORT or DID_RESET.

The two error statuses containing the word "TIME(_)OUT" are typically _not_ related to a command timing out. DID_TIME_OUT in the 'host_status' usually means an (unexpected) device selection timeout. DRIVER_TIMEOUT in the 'driver_status' byte means the SCSI adapter is unable to control the devices on its SCSI bus (and has given up).

The type of timeout is unsigned int (and it represents milliseconds).

These are single or multi-bit values that can be "or-ed" together:

The type of flags is unsigned int.

This value is not normally acted upon by the sg driver. It is provided so the user can identify the request. This is useful when command queuing is being used. The "abnormal" case is when SG_SET_FORCE_PACK_ID is set and a 'pack_id' other than -1 is given to read(). In this case the read() will wait to fetch a request that matches this 'pack_id'. If this mode is used be careful to set 'dxfer_direction' to a valid value (actually any of the SG_DXFER_* values will do) on input to the read(), together with the wanted pack_id. The type of pack_id is int.

This value is not acted upon by the sg driver. It is meant to allow the user to associate some object with this request (e.g. to maintain state information). The type of usr_ptr is void * .

This is the SCSI status byte as defined by the SCSI standard. Note that it can have vendor information set in bits 0, 6 and 7 (although this is uncommon). Further note that this 'status' data does _not_ match the definitions in <scsi/scsi.h> (e.g. CHECK_CONDITION). The following 'masked_status' does match those definitions. ^[8] The type of status is unsigned char .

Logically: masked_status == ((status & 0x3e) >> 1) . So 'masked_status' strips the vendor information bits off 'status' and then shifts it right one position. This makes it easier to do things like "if (CHECK_CONDITION == masked_status) ..." using the definitions in <scsi/scsi.h>. The defined values in this file are:

N.B. 1 bit offset from usual SCSI status values

Note that SCSI 3 defines some additional status codes. ^[9] The type of masked_status is unsigned char .

The messaging level in SCSI is under the command level and knowledge of what is happening at the messaging level is very rarely needed. Furthermore most modern chip-sets used in SCSI adapters completely hide this value. Nearly all adapters will return zero in 'msg_status' all the time. The type of msg_status is unsigned char .

This is the actual number of bytes written to the user memory pointed to by 'sbp'. 'sb_len_wr' is always <= 'mx_sb_len'. Linux 2.2 series kernels (and earlier) truncate this value to a maximum of 16 bytes. The actual number of bytes written will not exceed the length indicated by "Additional Sense Length" field (byte 7) of the Request Sense response. The type of sb_len_wr is unsigned char .

These codes potentially come from the firmware on a host adapter or from one of several hosts that an adapter driver controls. The 'host_status' field has the following values whose #defines mimic those which are only visible within the kernel (with the "SG_ERR_" removed from the front of each define). A copy of these defines can be found in sg_err.h (see Appendix A, Sg3_utils package):

The type of host_status is unsigned short .

One driver can potentially control several host adapters. For example Advansys provide one Linux adapter driver that controls all adapters made by that company - if 2 of more Advansys adapters are in 1 machine, then 1 driver controls both. When ('driver_status' & SG_ERR_DRIVER_SENSE) is true the 'sense_buffer' is also output. The 'driver_status' field has the following values whose #defines mimic those which are only visible within the kernel (with the "SG_ERR_" removed from the front of each define). A copy of these defines can be found in sg_err.h (see the utilities section):

The type of driver_status is unsigned short .

This is the residual count from the data transfer. It is 'dxfer_len' less the number of bytes actually transferred. In practice it only reports underruns (i.e. positive number) as data overruns should never happen. This value will be zero if there was no underrun or the SCSI adapter doesn't support this feature. ^[11] The type of resid is int .

This value will be the number of milliseconds from when a SCSI command was sent until sg is informed that it is complete. For i386 machines the granularity is 10ms while on alpha machines it is 1ms. This value is rounded toward zero. The type of duration is unsigned int .

This value is designed to convey useful information back to the user about the associated request. This information does not necessarily indicate an error. Several single bit and multi-bit fields are "or-ed" together to make this value.

A single bit component contained in SG_INFO_OK_MASK indicates whether some error or status field is non-zero. If either 'masked_status', 'host_status' or 'driver_status' are non-zero then SG_INFO_CHECK is set. The associated values are:

A multi bit component contained in SG_INFO_DIRECT_IO_MASK indicates what type of data transfer has just taken place. If indirect IO (or no data transfer) has taken place then SG_INFO_INDIRECT_IO is matched. Note that even if direct IO was requested in 'flags' the driver may choose to do indirect IO instead. If direct IO was requested and performed then SG_INFO_DIRECT_IO will be matched. Currently SG_INFO_MIXED_IO is never set. The associated values are:

The type of info is unsigned int .

open(const char * filename, int flags).  The filename should be a sg device file name as discussed in the Chapter 4, Interface. Flags can be a number of the following or-ed together:

Either O_RDONLY or O_RDWR must be set in flag. Either of the other 2 flags (but not both) can be or-ed in.

Note that multiple file descriptors may be open to the same SCSI device. [This is a way of side stepping the SG_MAX_QUEUE limit.] At the sg level separate state information is maintained. This means that even if multiple file descriptors are open to a single SCSI device their write() read() sequences are essentially independent.

Open() calls may be blocked due to exclusive locks (i.e. O_EXCL). An exclusive lock applies to a single SCSI device and only to sg's use of that device (i.e. it has no effect on access via sd, sr or st to that device). If the O_NONBLOCK flag is used then open() calls that would have otherwise blocked, yield EBUSY. Applications that scan sg devices trying to determine their identity (e.g. whether one is a scanner) should use the O_NONBLOCK flag otherwise they run the risk of blocking.

The driver will attempt to reserve SG_DEF_RESERVED_SIZE bytes (32KBytes in the current sg.h) on open(). The size of this reserved buffer can subsequently be modified with the SG_SET_RESERVED_SIZE ioctl(). In both cases these are requests subject to various dynamic constraints. The actual amount of memory obtained can be found by the SG_GET_RESERVED_SIZE ioctl(). The reserved buffer will be used if:

Returns a file descriptor if >= 0 , otherwise -1 implies an error.

write(int sg_fd, const void * buffer, size_t count).  The action of write() with a control block based on struct sg_header is discussed in the earlier document: www.torque.net/sg/p/scsi-generic.txt (i.e the sg version 2 documentation). This section describes the action of write() when it is given a control block based on struct sg_io_hdr.

The 'buffer' should point to an object of type sg_io_hdr_t and 'count' should be sizeof(sg_io_hdr_t) [it can be larger but the excess is ignored]. If the write() call succeeds then the 'count' is returned as the result.

Up to SG_MAX_QUEUE (16) write()s can be queued up before any finished requests are completed by read(). An attempt to queue more than that will result in an EDOM error. ^[12] The write() command should return more or less immediately. ^[13]

The version 2 sg driver defaulted the maximum queue length to 1 (and made available the SG_SET_COMMAND_Q ioctl() to switch it to SG_MAX_QUEUE). So for backward compatibility a file descriptor that only receives sg_header structures in its write() will have a default "max" queue length of 1. As soon as a sg_io_hdr_t structure is seen by a write() then the maximum queue length is switched to SG_MAX_QUEUE on that file descriptor.

The "const" on the 'buffer' pointer is respected by the sg driver. Data is read in from the sg_io_hdr object that is pointed to. Significantly this is when the 'sbp' and the 'dxferp' are recorded internally (i.e. not from the sg_io_hdr object given to the corresponding read() ).

read(int sg_fd, void * buffer, size_t count).  The action of read() with a control block based on struct sg_header is discussed in the earlier document: www.torque.net/sg/p/scsi-generic.txt (i.e. the sg version 2 documentation). This section describes the action of read() when it is given a control block based on struct sg_io_hdr.

The 'buffer' should point to an object of type sg_io_hdr_t and 'count' should be sizeof(sg_io_hdr_t) [it can be larger but the excess is ignored]. If the read() call succeeds then the 'count' is returned as the result.

By default, read() will return the oldest completed request that is queued up. A read() will not interfere with any request associated with the SG_IO ioctl() on this file descriptor except in a special case when a SG_IO ioctl() is interrupted by a signal.

If the SG_SET_FORCE_PACK_ID,1 ioctl() is active then read() will attempt to fetch the packet whose pack_id (given earlier to write()) matches the sg_io_hdr_t::pack_id given to this read(). If not available it will either wait or yield EAGAIN. As a special case, -1 in sg_io_hdr_t::pack_id given to read() will match the request whose response has been waiting for the longest time. Take care to also set 'dxfer_direction' to any valid value (e.g. SG_DXFER_NONE) when in this mode. The 'interface_id' member should also be set appropriately.

Apart from the SG_SET_FORCE_PACK_ID case (and then only for the 3 indicated fields), the sg_io_hdr_t object given to read() can be uninitialized. Note that the 'sbp' pointer value for optionally outputting a sense buffer was recorded from the earlier, corresponding write().

poll(struct pollfd *ufds, unsigned int nfds, int timeout).  This call can be used to check the state of a sg file descriptor. It will always respond immediately. Typical usages are to periodically poll the state of a sg file descriptor and to determine why a SIG_IO signal was received.

For file descriptors associated with sg devices:

POLLOUT indicates the sg will not block a new write() or SG_IO ioctl(). However it is still possible (but unlikely) that the mid level or an adapter may block (or yield EAGAIN).

close(int sg_fd).  Preferably a close() should be done after all issued write()s have had their corresponding read() calls completed. Unfortunately this is not always possible (e.g. the user may choose to send a kill signal to a running process). The sg driver implements "fast" close semantics and thus will return more or less immediately (i.e. not wait on any event). This is application friendly but requires the sg driver to arrange for an orderly cleanup of those packets that are still "in flight".

When close() leaves outstanding SCSI commands still awaiting responses, the sg driver maintains its internal structures for the now defunct file descriptor. These internal structures are maintained until all outstanding responses (some might be timeouts) are received. When the sg driver is loaded as a module and has any open file descriptors or "defunct" file descriptors then it cannot be unloaded. An attempt to call rmmod sg will report the driver is busy. Defunct file descriptors that remain for some time, perhaps awaiting a timeout, can be observed with the cat /proc/scsi/sg/debug command. In this case "closed=1" will be set on the defunct file descriptor [see the section called “/proc/scsi/sg/debug”]. Defunct file descriptors do not impede attempts by applications to open() new file descriptors on the same SCSI device.

The kernel arranges for only the last close() on a file descriptor to be seen by a driver (and to emphasize this, the corresponding sg driver call is named sg_release() rather than sg_close()). This is only significant when an application uses fork() or dup().

Returns 0 if successful, otherwise -1 implies an error.

mmap(void * start, size_t length, int prot, int flags, int sg_fd, off_t offset).  This system call returns a pointer to the beginning of the reserved buffer associated with the sg file descriptor 'sg_fd'. The 'start' argument is a hint to the kernel and is ignored by this driver; best set it to 0. The 'length' argument should be less than or equal to the size of the reserved buffer associated with 'sg_fd'. If it exceeds the reserved buffer size (after 'length' has been rounded up to a page size multiple) then MAP_FAILED is returned and ENOMEM is placed in errno. The 'prot' argument should either be PROT_READ or (PROT_READ | PROT_WRITE). The 'flags' argument should contain MAP_SHARED. In a sense, the user application is "sharing" data with the sg driver. The MAP_PRIVATE flag does not play well with compiler optimization flags such as '-O2'. The 'offset' argument must be set to 0 (or NULL).

The mmap() system call can be made multiple times on the same sg_fd. The munmap() system call is not required if close() is called on sg_fd. Mmap-ed IO is well-behaved when a process is fork()-ed (or the equivalent finer grained clone() system call is made). In the case of a fork(), 2 processes will be sharing the same memory mapped area together with the sg driver for a sg_fd and the last one to close the sg_fd (or exit) will cause the shared memory to be freed.

It is assumed that if the default reserved buffer size of 32 KB is not sufficient then a ioctl(SG_SET_RESERVED_SIZE) call is made prior to any calls to mmap(). If the required size is not a multiple of the kernel's page size (returned by getpagesize() system call) then the size passed to ioctl(SG_SET_RESERVED_SIZE) should be rounded up to the next page size multiple.

Mmap-ed IO is requested by setting (or or-ing in) the SG_FLAG_MMAP_IO constant into the flag member of the the sg_io_hdr structure prior to a call to write() or ioctl(SG_IO). The logic to do mmap-ed IO _assumes_ that an appropriate mmap() call has been made by the application. In other words it does not check. ^[14]

fcntl(int sg_fd, int cmd, long arg).  There are several uses for this system call in association with a sg file descriptor. The following pseudo code shows code that is useful for scanning the sg devices, taking care not to be caught in a wait for an O_EXCL lock by another process, and when the appropriate device is found, switching to normal blocked io. A working example of this logic is in the sg_scan utility program.

open("/dev/sg0", O_RDONLY | O_NONBLOCK)
/* check device, EBUSY means some other process has O_EXCL lock on it */
/* when the device you want is found then ... */
flags = fcntl(sg_fd, F_GETFL)
fcntl(sg_fd, F_SETFL, flags & (~ O_NONBLOCK))
/* since, with simple apps, it is easier to use normal blocked io */

The sg driver supports asynchronous notification. This is a non-blocking mode of operation in which, when the driver receives data back from a device so that a read() can be done, it sends a SIGPOLL (aka SIGIO) signal to the owning process. Here is a code snippet from the sg_poll test program.

sigemptyset(&sig_set)
sigaddset(&sig_set, SIGPOLL)
sigaction(SIGPOLL, &s_action, 0)
fcntl(sg_fd, F_SETOWN, getpid())
flags = fcntl(sg_fd, F_GETFL);
fcntl(sg_fd, F_SETFL, flags | O_ASYNC)

With the original interface almost any string could be accidentally given to write() and potentially (but rarely) something nasty could happen. If some error was detected then more than likely EIO was placed in errno.

Unfortunately this can still happen with write() since it can accept both the original struct sg_header or the newer sg_io_hdr_t described in this note. However since the SG_IO ioctl() will only accept the sg_io_hdr_t structure there is less chance of a random string being interpreted as a command. Since the sg_io_hdr_t interface does a lot more error checking, it attempts to give out more precise errno values to help the user pinpoint the problem. [Admittedly some of these errno values are picked in an arbitrary way from the large set of available values.]

In most cases when a system call on a sg file descriptor fails, the call in question will return -1. After an application detects that a system call has failed it should read the value in the "errno" variable (prior to do any more system calls). Applications should include the <errno.h> header.

Below is a table of errno values indicating which calls to sg will generate them and the meaning of the error. A write() call is indicated by "w", a read() call by "r" and an open() call by "o".

errno    which_calls    Meaning
-----    -----------    ----------------------------------------------
EACCES    <some ioctls> Root permission (more precisely CAP_SYS_ADMIN
                        or CAP_SYS_RAWIO) required. Also may occur during
                        an attempted write to /proc/scsi/sg files.
EAGAIN    r             The file descriptor is non-blocking and the request
                        has not been completed yet.
EAGAIN    w,SG_IO       SCSI sub-system has (temporarily) run out of 
                        command blocks.
EBADF     w             File descriptor was not open()ed O_RDWR.
EBUSY     o             Someone else has an O_EXCL lock on this device.
EBUSY     w             With mmap-ed IO, the reserved buffer already in use.
EBUSY     <some ioctls> Attempt to change something (e.g. reserved buffer
                        size) when the resource was in use.
EDOM      w,SG_IO       Too many requests queued against this file
                        descriptor. Limit is SG_MAX_QUEUE active requests.
                        If sg_header interface is being used then the
                        default queue depth is 1. Use SG_SET_COMMAND_Q
                        ioctl() to increase it.
EFAULT    w,r,SG_IO     Pointer to user space invalid.
          <most ioctls> 
EINVAL    w,r           Size given as 3rd argument not large enough for the
                        sg_io_hdr_t structure. Both direct and mmap-ed IO
			selected.
EIO       w             Size given as 3rd argument less than size of old
                        header structure (sg_header). Additionally a write()
                        with the old header will yield this error for most
                        detected malformed requests.
EIO       r             A read() with the older sg_header structure yields
			this value for some errors that it detects.
EINTR     o             While waiting for the O_EXCL lock to clear this call
                        was interrupted by a signal.
EINTR     r,SG_IO       While waiting for the request to finish this call
                        was interrupted by a signal.
EINTR     w             [Very unlikely] While waiting for an internal SCSI
                        resource this call was interrupted by a signal.
EMSGSIZE  w,SG_IO       SCSI command size ('cmd_len') was too small 
                        (i.e. < 6) or too large
ENODEV    o             Tried to open() a file with no associated device.
                        [Perhaps sg has not been built into the kernel or
                        is not available as a module?]
ENODEV    o,w,r,SG_IO   SCSI device has detached, awaiting cleanup.
                        User should close fd. Poll() will yield POLLHUP.
ENOENT    o             Given filename not found.
ENOMEM    o             [Very unlikely] Kernel was not even able to find
                        enough memory for this file descriptor's context.
ENOMEM    w,SG_IO       Kernel unable to find memory for internal buffers.
                        This is usually associated with indirect IO.
			For mmap-ed IO 'dxfer_len' greater than reserved
			buffer size.
			Lower level (adapter) driver does not support enough
			scatter gather elements for requested data transfer.
ENOSYS    w,SG_IO       'interface_id' of a sg_io_hdr_t object was _not_ 'S'.
ENXIO     o             "remove-single-device" may have removed this device.
ENXIO     o, w,r,SG_IO  Internal error (including SCSI sub-system busy doing
                        error processing - e.g. SCSI bus reset). When a
			SCSI device is offline, this is the response. This 
			can be bypassed by opening O_NONBLOCK.
EPERM     o             Can't use O_EXCL when open()ing with O_RDONLY
EPERM     w,SG_IO       File descriptor open()-ed O_RDONLY but O_RDWR
          <some ioctls> access mode needed for this operation.

SG_IO 0x2285.  The idea is deceptively simple: just hand a sg_io_hdr_t object to an ioctl() and it will return when the SCSI command is finished. It is logically equivalent to doing a write() followed by a blocking read(). The word "blocking" here implies the read() will wait until the SCSI command is complete.

The same file descriptor can be used both for SG_IO synchronous calls and the write() read() sequences at the same time. The sg driver makes sure that the response to a SG_IO call will never accidentally be fetched by a read(). Even though a single file descriptor can be shared in this manner, it is probably more sensible (and results in cleaner code) if separate file descriptors to the same SCSI device are used in this case.

It is possible that the wait for the command completion is interrupted by a signal. In this case the SG_IO call will yield an EINTR error. This is reasonably complex to handle and is discussed in the ioctl(SG_SET_KEEP_ORPHAN) description below. The following SCSI commands will be permitted by SG_IO when the sg file descriptor was opened O_RDONLY:

All commands to SCSI device type SCANNER are accepted. Other cases yield an EPERM error. Note that the write() read() interface must have the sg file descriptor open()-ed with O_RDWR as write permission is required by Linux to execute a write() system call.

The ability of the SG_IO ioctl() to issue certain SCSI commands has led to some relaxation on file descriptors open()ed "read-only" compared with the version 2 sg driver. The open() call will now attempt to allocate a reserved buffer for all newly opened file descriptors. The ioctl(SG_SET_RESERVED_SIZE) will now work on "read-only" file descriptors.

SG_GET_ACCESS_COUNT 0x2289.  This ioctl() yields the access count maintained by the mid level for this SCSI device. This number is incremented by each open() call done by the upper level SCSI drivers (i.e. sd, sr, st and sg) and decremented by those drivers' release(). [A driver's release() corresponds to the last close() on a file descriptor, or is supplied by the kernel when a process is aborted.] Each SCSI device has a separate access count.

SG_SET_COMMAND_Q 0x2271 [_GET_ 0x2270] .  The default it the original sg driver was not to allow commands to be queued on the same file descriptor (actually it was more restrictive, commands could not be queued on a SCSI device). The version 2 sg driver kept this action as its default (for backward compatibility) and offered these ioctl()s to change and monitor the command queuing state.

SG_SET_DEBUG 0x227e.  The third argument is assumed to point to an int. The default value is 0. If this call is made pointing to an int greater than 0 then any SCSI request that is issued that results in the SCSI status of CHECK_CONDITION (or COMMAND_TERMINATED) will cause a message to be sent to the log (and perhaps the console). The message is information derived from the sense buffer (i.e. the SCSI error message) and it is prefixed with "sg_cmd_done_bh".

The other actions of debug mode performed in version 2 of the sg driver have been removed as they are no longer needed. The internal state of the sg driver can now be found by viewing the output of cat /proc/scsi/sg/debug.

SG_EMULATED_HOST 0x2203.  Assumes 3rd argument points to an int and outputs a flag indicating whether the host (adapter) is connected to a "real" SCSI bus or is an emulated one (e.g. ide-scsi or usb storage device driver). A value of 1 means emulated while 0 is not. [To check: is IEEE1394 a "real" SCSI serial bus?]

SG_SET_KEEP_ORPHAN 0x2287 [_GET_ 0x2288].  These ioctl()s allow the setting and reading of the "keep_orphan" flag. This controls what happens to the request associated with a SG_IO ioctl() that is interrupted (i.e. errno is EINTR). The default action is to drop the response as soon as it is received. This corresponds to the "keep_orphan" flag being 0. When the "keep_orphan" flag is 1 then the response is transformed in such a way that it can be fetched by a read(). This is the only circumstance in which a request sent by a SG_IO ioctl() can have the associated response fetched by a read().

SG_SET_FORCE_LOW_DMA 0x2279.  Assumes 3rd argument points to an int containing 0 or 1. 0 (default) means sg decides whether to use memory above 16 Mbyte level (on i386) based on the host adapter being used by this SCSI device. Typically PCI SCSI adapters will indicate they can DMA to the whole 32 bit address space. If 1 is given then the host adapter is overridden and only memory below the 16MB level is used for DMA. A requirement for this should be extremely rare. If the "reserved" buffer allocated on open() is not in use then it will be de-allocated and re-allocated under the 16MB level (and the latter operation could fail yielding ENOMEM). Only the current file descriptor is affected.

SG_GET_LOW_DMA 0x227a.  Assumes 3rd argument points to an int and places 0 or 1 in it. 0 indicates the whole 32 bit address space is being used for DMA transfers on this file descriptor. 1 indicates the memory below the 16MB level (on i386) is being used (and this may be the case because the host adapters setting has been overridden by SG_SET_FORCE_LOW_DMA,1 .

SG_NEXT_CMD_LEN 0x2283.  This ioctl() is not required with sg_io_hdr structure since command length is set explicitly for every command. Assumes 3rd argument is pointing to an int. The value of the int (if > 0) will be used as the SCSI command length of the next SCSI command sent to a write() using the sg_header interface. After that write() the SCSI command length logic is reset to use automatic length detection (i.e. depending on SCSI command group and the 'twelve_byte' field). If the current SCSI command length maximum of 16 is exceeded then the affected write() will yield an EDOM error. Giving this ioctl() a value of 0 will set automatic length detection for the next write(). N.B. Only the following write() on this fd is affected by this ioctl().

SG_GET_NUM_WAITING 0x227d.  Assumes 3rd argument points to an int and places the number of packets waiting to be read in it. Only those requests that have been issued by a write() and are now available to be read() are counted. In other words any ioctl(SG_IO) operations underway on this file descriptor will not effect this count ^[15].

SG_SET_FORCE_PACK_ID 0x227b.  Assumes 3rd argument is pointing to an int. 0 (default) instructs read() to return the oldest (written) packet if multiple packets are waiting to be read. 1 instructs read() to view the sg_io_hdr::pack_id (or sg_header::pack_id) as input and return the oldest packet matching that pack_id or wait until it arrives. If the file descriptor is in O_NONBLOCK state, rather than wait this ioctl() will yield EAGAIN. As a special case the pack_id of -1 given to read() in the mode will match the oldest packet. Only the current file descriptor is affected by this command.

SG_GET_PACK_ID 0x227c.  Assumes 3rd argument points to an int and places the pack_id of the oldest (written) packet in it. If no packet is waiting to be read then yields -1.

SG_GET_REQUEST_TABLE 0x2286.  This ioctl outputs an array of information about the status of requests associated with the current file descriptor. Its 3rd argument should point to memory large enough to receive SG_MAX_QUEUE objects of the sg_req_info_t structure. This structure has the following members:

        req_state
            0 -> request not in use
            1 -> request has been sent, but is not finished (i.e. it is
                 between stages 1 and 2 in the "theory of operation")
            2 -> request is ready to be read() (i.e. it is between stages
                 2 and 3 in the "theory of operation")
        orphan
            0 -> normal request
            1 -> request sent by SG_IO ioctl() which has been interrupted
                 by a signal
        sg_io_owned
            0 -> request sent by a write()
            1 -> request sent by a SG_IO ioctl()
        problem
            0 -> no problem (or 1 == req_state)
            1 -> req_state is 2 and either masked_status, host_status or
                 driver_status is non-zero
        duration
            [if 1 == req_state] time since request was sent (in millisecs)
            [if 2 == req_state] duration of request (in millisecs). Clock
                 is stopped when stage 2 in "theory of operation" is
                 reached
        pack_id
        usr_ptr
            these are user provided values in the sg_io_hdr_t (or
            struct sg_header) that sent the request

SG_SET_RESERVED_SIZE 0x2275 [_GET_ 0x2272].  Both ioctl()s assume the 3rd argument is pointing to an int.

For ioctl(SG_SET_RESERVED_SIZE) the value will be used to request a new reserved buffer of that size. The previous reserved buffer is freed (if it is not in use; if it was in use then the ioctl() fails and EBUSY is placed in errno). A new reserved buffer is then allocated and its actual size can be found by calling the ioctl(SG_GET_RESERVED_SIZE). The reserved buffer is then used for DMA purposes by subsequent write() and ioctl(SG_IO) commands if it is not already in use and if the write() is not calling for a buffer size larger than that reserved. The reserved buffer may well be a series of kernel buffers if the adapter supports scatter-gather. Large buffers can be requested (e.g. 4 MB) but not necessarily granted. Once a mmap() call has been made on a sg file descriptor, subsequent calls to this ioctl() will fail with EBUSY placed in errno.

In the case of ioctl(SG_GET_RESERVED_SIZE) the size in bytes of the reserved buffer from open() or the most recent SG_SET_RESERVED_SIZE ioctl() call on this fd. The result can be 0 if memory is very tight. In this case it may not be wise to attempt something like burning a CD on this file descriptor.

SG_SCSI_RESET 0x2284.  Assumes 3rd argument points to an int. That int should be one of the following defined in the sg.h header:

The reset options are in ascending order of severity. Not all levels are supported by all linux lower level drivers. Most lower level (adapter) drivers support the SCSI bus reset. These boards often issue a SCSI bus reset during their initialization.

Unfortunately this ioctl() doesn't currently do much (but may in the future after other issues are resolved). Yields an EBUSY error if the SCSI bus or the associated device is being reset when this ioctl() is called, otherwise returns 0. N.B. In some recent distributions there is a patch to the SCSI mid level code that activates this ioctl. Check your distribution.

SG_GET_SCSI_ID 0x2276.  Assumes 3rd argument is pointing to an object of type Sg_scsi_id (see sg.h) and populates it. That structure contains ints for host_no, channel, scsi_id, lun, scsi_type, allowable commands per lun and queue_depth. Most of this information is available from other sources (e.g. SCSI_IOCTL_GET_IDLUN and SCSI_IOCTL_GET_BUS_NUMBER) but tends to be awkward to collect. Allowable commands per lun and queue_depth give an insight to the command queuing capabilities of the adapters and the device. The latter overrides the former (logically) and the former is only of interest if it is equal to queue_depth which probably indicates the device does not support queuing commands (e.g. most scanners).

typedef struct sg_scsi_id { /* used by SG_GET_SCSI_ID ioctl() */
    int host_no;        /* as in "scsi<n>" where 'n' is one of 0, 1, 2 etc */
    int channel;
    int scsi_id;        /* scsi id of target device */
    int lun;
    int scsi_type;      /* TYPE_... defined in scsi/scsi.h */
    short h_cmd_per_lun;/* host (adapter) maximum commands per lun */
    short d_queue_depth;/* device (or adapter) maximum queue length */
    int unused[2];      /* probably find a good use, set 0 for now */
} sg_scsi_id_t;

SG_GET_SG_TABLESIZE 0x227F.  Assumes 3rd argument points to an int and places the maximum number of scatter gather elements supported by the host adapter associated with the current SCSI device. 0 indicates that the adapter does support scatter gather.

SG_GET_TIMEOUT 0x2202.  Ignores its 3rd argument and _returns_ the timeout value (which will be >= 0 ). The unit of this timeout is "jiffies" which are currently 10 millisecond intervals on i386 (less on an alpha). Linux supplies a manifest constant HZ which is the number of "jiffies" in 1 second. This ioctl() is not relevant to the sg version 3 driver because timeouts are specified explicitly for each command in the sg_io_hdr structure.

SG_SET_TIMEOUT 0x2201.  Assumes 3rd argument points to an int containing the new timeout value for this file descriptor. The unit is a "jiffy". Packets that are already "in flight" will not be affected. The default value is set on open() and is SG_DEFAULT_TIMEOUT (defined in sg.h). This default is currently 1 minute and may not be long enough for formats. Negative values will yield an EIO error. This ioctl() is not relevant to the sg version 3 driver because timeouts are specified explicitly for each command in the sg_io_hdr structure. Only when the sg_header structure is used is the timeout inherited from this value (help on a per file descriptor basis).

SG_SET_TRANSFORM 0x2204.  Only is meaningful when SG_EMULATED host has yielded 1 (i.e. the low-level is the ide-scsi device driver); otherwise an EINVAL error occurs. The default state is to _not_ transform SCSI commands to the corresponding ATAPI commands but pass them straight through as is. [Only certain classes of SCSI commands need to be transformed to their ATAPI equivalents.] The third argument is interpreted as an integer. When it is non-zero then a flag is set inside the ide-scsi driver that transforms subsequent commands sent to this driver. When zero is passed as the 3rd argument to this ioctl then the flag within the ide-scsi driver is cleared and subsequent commands are not transformed. Beware, this state will affect all devices (and hence all related sg file descriptors) associated with this ide-scsi "bus".

SG_GET_TRANSFORM 0x2205.  Third argument is ignored. Only is meaningful when SG_EMULATED host has yielded 1 (i.e. the low-level is the ide-scsi device driver); otherwise an EINVAL error occurs. Returns 0 to indicate _not_ transforming SCSI to ATAPI commands (default). Returns 1 when it is transforming them.

Some seldom used ioctl()s introduced in the sg 2.x series drivers have been withdrawn. They are:

SCSI_IOCTL_GET_IDLUN 0x5382.  This ioctl takes a pointer to a "struct scsi_idlun" object as its third argument. The "struct scsi_idlun" is not visible to user applications. To use this, that structure needs to be replicated in the user's program. Something like:

typedef struct my_scsi_idlun {
    int four_in_one;    /* 4 separate bytes of info compacted into 1 int */
    int host_unique_id; /* distinguishes adapter cards from same supplier */
} My_scsi_idlun;

"four_in_one" is made up as follows:

(scsi_device_id | (lun << 8) | (channel << 16) | (host_no << 24))

These 4 components are assumed (or masked) to be 1 byte each. These are the four numbers that the SCSI subsystem uses to index devices, often written as "<host_no, channel, scsi_id, lun>". The 'host_unique_id' assigns a different number to each controller from the same manufacturer/low-level device driver. Most of the information provided by this command is more easily obtained from SG_GET_SCSI_ID.

The 'host_no' element is a change in lk 2.4 kernels. [In the lk 2.2 series and earlier, it was 'low_inode & 0xff' from the procfs entry corresponding to the host.] This change makes the use of the SCSI_IOCTL_GET_BUS_NUMBER ioctl() superfluous.

The advantage of this ioctl() is that it can be called on any SCSI file descriptor.

SCSI_IOCTL_GET_PCI 0x5387.  Yields the PCI slot name (pci_dev::slot_name) associated with the lower level (adapter) driver that controls the current device. Up to 8 characters are output to the location pointed to by 'arg'. If the current device is not controlled by a PCI device then errno is set to ENXIO. [This ioctl() was introduced in lk 2.4.4]

SCSI_IOCTL_PROBE_HOST 0x5385.  This command should be given a pointer to a 'char' array as its 3rd argument. That array should be at least sizeof(int) long and have the length of the array as an 'int' at the beginning of the array! An ASCII string of no greater than that length containing "information" (or the name) of SCSI host (i.e. adapter) associated with this file descriptor is then placed in the given byte array. N.B. A trailing '\0' may need to be put on the output string if it has been truncated by the input length. Returns 1 if host is present, 0 if it is not and a negative value if there is an error.

SCSI_IOCTL_SEND_COMMAND 0x1.  This ioctl() also offers a "pass through" SCSI command capability which is a subset of what is offered by the sg driver.

The structure that we are passed should look like:

   struct sdata {
    unsigned int inlen;     [i] Length of data written to device
    unsigned int outlen;    [i] Length of data read from device
    unsigned char cmd[x];   [i] SCSI command (6 <= x <= 16)
                            [o] Data read from device starts here
                            [o] On error, sense buffer starts here
    unsigned char wdata[y]; [i] Data written to device starts here
   };

Notes:

Direct IO uses the kiobuf mechanism [see the Linux Device Drivers book] to manipulate memory allocated within the user space so that a lower level (adapter) driver can DMA directly to or from that user space memory. Since the user can give a different data buffer to each SCSI command passed through the sg interface then the kiobuf mechanism needs to setup its structures (and undo that setup) for each SCSI command. ^[18] Direct IO is available as an option in sg 3.1.18 (before that the sg driver needed to be recompiled with an altered define). Direct IO support is designed in such a way that if it is requested and cannot be performed then the command will still be performed using indirect IO. If direct IO is requested and has been performed then the SG_INFO_DIRECT_IO bit will be set in the 'info' member of the sg_io_hdr_t control structure after the request has been completed. Direct IO is not supported on ISA SCSI adapters since they only can address a 24 bit address space.

One limit on direct IO is that sg_io_hdr_t::iovec_count==0. So the user cannot (currently) use application level scatter gather and direct IO on the same request.

For direct IO to be worthwhile, a reasonable amount of data should be requested for data transfer. For transfers less than 8 KByte it is probably not worth the trouble. On the other hand "locking down" a multiple 512 KB blocks of data for direct IO could adversely impact overall system performance. Remember that for the duration of a direct IO request, the data transfer buffer is mapped to a fixed memory location and locked in such a way that it won't be swapped out. This can "cramp the style" of the kernel if it is overdone.

Prior to sg 3.1.18 the direct IO code was commented out with the "SG_ALLOW_DIO" define. In sg 3.1.18 (available for lk 2.4.2 and later) the direct IO code is active but is defaulted off by a run time value. This value can be accessed via the "proc" file system at /proc/scsi/sg/allow_dio . Direct IO is enabled when a user with root permissions writes "1" to that file: echo 1 > /proc/scsi/sg/allow_dio . If SG_FLAG_DIRECT_IO is set in sg_io_hdr::flags but /proc/scsi/sg/allow_dio holds "0" then indirect IO will be performed (and this is indicated by ((sg_io_hdr::info & SG_INFO_DIRECT_IO_MASK) == SG_INFO_INDIRECT_IO) after the request is completed).

Memory-mapped IO takes a different approach from direct IO to removing the extra data copy performed by normal ("indirect") IO. With mmap-ed IO the application calls the mmap() system call to memory map sg's reserved buffer. The sg driver maintains one reserved buffer per file descriptor. The default size of the reserved buffer is 32 KB and it can be changed with the ioctl(SG_SET_RESERVED_SIZE). The mmap() system call only needs to be called once prior ^[19] to doing mmap-ed IO. For more details on the mmap() see the section called “mmap()”. An application indicates that it wants mmap-ed on a SCSI request by setting the SG_FLAG_MMAP_IO value in 'flags'.

Since there is only reserved buffer per sg file descriptor then only one mmap-ed IO command can be active at one time. In order to perform command queuing with mmap-ed IO, an application will need to open() multiple file descriptors to the same SCSI device. With mmap-ed IO the various status values and the sense buffer (if required) are conveyed back to an application in the same fashion as normal ("indirect") IO.

Mmap-ed has very low per command latency since the reserved buffer mapping only needs to be done once per file descriptor. Also the reserved buffer is set up by the sg driver to aid the efficient construction of the internal scatter gather list used by the lower level (adapter) driver for DMA purposes. This tends to be more efficient than the user memory that direct IO requires the sg driver to process into an internal scatter gather list. So on both these counts, mmap-ed IO has the edge over direct IO.

This appendix explains the output from the /proc/scsi/sg/debug which is typically viewed by the command cat /proc/scsi/sg/debug. Below is the (slightly abridged) output while this command: sgp_dd if=/dev/sg0 of=/dev/null bs=512 is executing on the system. That sgp_dd command is using command queuing to read a disk (and the data is written to /dev/null which forgets it).

$ cat /proc/scsi/sg/debug
dev_max(currently)=7 max_active_device=1 (origin 1)
 scsi_dma_free_sectors=416 sg_pool_secs_aval=320 def_reserved_size=32768
 >>> device=sg0 scsi0 chan=0 id=0 lun=0   em=0 sg_tablesize=255 excl=0
   FD(1): timeout=60000ms bufflen=65536 (res)sgat=2 low_dma=0
   cmd_q=1 f_packid=1 k_orphan=0 closed=0
     fin: id=3949312 blen=65536 dur=10ms sgat=2 op=0x28
     act: id=3949440 blen=65536 t_o/elap=60000/10ms sgat=2 op=0x28
     rb>> act: id=3949568 blen=65536 t_o/elap=60000/10ms sgat=2 op=0x28
     act: id=3949696 blen=65536 t_o/elap=60000/0ms sgat=2 op=0x28

Those items output above that are significant to user applications are described below.

Broadly speaking the above output shows everything is going fine. Four SCSI READ(10) commands (SCSI opcode 0x28) for different ids are underway. Three commands are active while one is finished with its status and data read() and the request structure is pending deletion. The "id" corresponds to the pack_id given in the sg_io_hdr structure (or the sg_header structure). In the case if sgp_dd the pack_id value is the block number being given to the SCSI READ (or WRITE). You will notice the 4 ids are 128 apart.

The ">>>" line shows the sg device name followed by the linux scsi adapter, channel, scsi id and lun numbers. The "em=" argument indicates whether the driver emulates a SCSI HBA. The ide-scsi driver would set "em=1". The "sg_tablesize" is the maximum number of scatter gather elements supported by the adapter driver. The "excl=0" indicates no sg open() on this device is currently using the O_EXCL flag.

The next two lines starting with "FD(1)" supply data about the first (and only in this case) open file descriptor on /dev/sg0 . The default timeout is 60 seconds however this is only significant if the sg_header interface is being used since the sg_io_hdr interface explicits sets the timeout on a per command basis. "bufflen=65536" is the reserved buffer size for this file descriptor. The "(res)sgat=2" indicates that this reserved buffer requires 2 scatter gather elements. The "low_dma" will be set to 1 for ISA HBAs indicating only the bottom 16 MB of RAM can be used for its kernel buffers. The "cmd_q=1" indicates command queuing is being allowed. The "f_packid=1" indicates the SG_SET_FORCE_PACK_ID mode is on. The "k_orphan" value is 1 in the rare cases when a SG_IO is interrupted while a SCSI command is "in flight". The "closed" value is 1 in the rare cases the file descriptor has been closed while a SCSI command is "in flight".

Each line indented with 5 spaces represents a SCSI command. The state of the command is either:

These states can be optionally prefixed by "rb>>" which means the reserved buffer is being used, "dio>>" which means this command is using direct IO, or "mmap>>" which means that mmap-ed IO is being used by this command. The "id" is the pack_id from this command's interface structure. The "blen" is the buffer length used by the data transfer associated with this command. For commands that a response has been received "dur" shows its duration in milliseconds. For commands still "in flight" an indication of "t_o/elap=60000/10ms" means this command has a timeout of 60000 milliseconds of which 10 milliseconds has already elapsed. The "sgat=2" argument indicates that this command's "blen" requires 2 scatter gather elements. The "op" value is the hexadecimal value of the SCSI command being executed.

If sg has lots of activity then the "debug" output may span many lines and in some cases appear to be corrupted. This occurs because procfs requests fixed buffer sizes of information and, if there is more data to output, returns later to get the remainder. The problem with this strategy is that sg's internal state may have changed. Rather than double buffering, the sg driver just continues from the same offset. While procfs is very useful, ioctl()s (such as SG_GET_REQUEST_TABLE) still have their place.