9.3.2 Understanding zpool status Output

A complete zpool status output looks similar to the following:

# zpool status tank
  pool: tank
 state: ONLINE
reason: Data corruption detected.
action: Remove corrupted data or restore from backup.
   see: http://www.sun.com/msg/ZFS-XXXX-09
config:
        NAME                  STATE     READ WRITE CKSUM 
        test                  ONLINE       0     0     0
          mirror              ONLINE       0     0     0
            c0t0d2            ONLINE       0     0     0
            c0t0d1            ONLINE       0     0     0

 scrub: ...
 errors: 4 errors detected.  Use '-v' for a complete list.

This output is divided into 4 basic sections:

9.3.2.1 Overall Status Information

This header section contains the following fields, some of which are only displayed for pools exhibiting problems:

9.3.2.2 Configuration Information

The config field describes the configuration layout of the devices comprising the pool, as well as their state and any errors seen from the devices. The state can be one of ONLINE, FAULTED, DEGRADED, or OFFLINE. If it is anything but ONLINE, it indicates that the fault tolerance of the pool has been compromised.

The second section of the configuration output displays error statistics. These errors are divided into three categories:

These errors can be used to determine if the damage is permanent. A small number of I/O errors may indicate a temporary outage, while a large number may indicate a permanent problem with the device. These errors do not necessarily correspond to data corruption as seen by applications. If the device is in a redundant configuration, the disk devices may show uncorrectable errors, while no errors appear at the mirror or RAID-Z device level. If this is the case, then ZFS successfully retrieved the good data, and attempted to heal the damaged data from existing replicas. For more information on interpreting these errors to determine device failure, see 9.6.1 Determining Type of Failure.

Finally, additional auxiliary information is displayed in the last column of output. This expands on the state field, aiding in diagnosis of failure modes. If a device is FAULED, this field indicates whether it is because the device is inaccessible or whether the data on the device is corrupted. If the device undergoing resilvering, this field displays the current progress. For more information on monitoring resilvering progress, see 9.6.3.4 Viewing Resilvering Status.

9.3.2.3 Scrubbing Status

The third section of output describes the current status of any explicit scrubs. This information is orthogonal to whether any errors are detected on the system, though it can be used to determine how accurate the data corruption error reporting is. If the last scrub ended recently, it is fairly certain that any known data corruption has been discovered.

For more information on data scrubbing and how to interpret this information, see 9.2 Checking Data Integrity.

9.3.2.4 Data Corruption Errors

The status also shows whether there are any known errors associated with the pool. These errors may have been found while scrubbing the disks, or during normal operation. ZFS keeps a persistent log of all data errors associated with the pool. This log is rotated whenever a complete scrub of the system finishes. These errors are always fatal -- their presence indicates that at least one application experienced an I/O error due to corrupt data within the pool. Device errors within a replicated pool do not result in data corruption, and are not recorded as part of this log. By default, only the number of errors found is displayed. A complete list of errors and their specifics can be found using the -v option.

For more information on interpreting data corruption errors, see 9.7.1 Identifying Type of Data Corruption.

9.3.3 System Messaging

In addition to persistently keeping track of errors with the pool, ZFS also displays syslog messages when events of interest occur. The following scenarios generate events to notify the administrator:

• Device state transition -- If a device becomes FAULTED, ZFS logs a message indicating that the fault tolerance of the pool may be compromised. A similar message is sent if the device is later brought online, restoring the pool to health.

• Data corruption -- If any data corruption is seen, ZFS logs a message describing when and where the corruption was seen. This message is only logged the first time it is seen; subsequent accesses do not generate a message.

It is important to note that device errors are not mentioned here. If ZFS detects a device error and automatically recovers from it, there is no notification for the administrator. This is because such errors do not constitute a failure in the pool redundancy or data integrity, and because they are typically the result of a driver problem accompanied by its own set of error messages.

9.4 Damaged Configuration

ZFS keeps a cache of active pools and their configuration on the root filesystem. If this file is corrupted or somehow becomes out of sync with what is stored on disk, the pool can no longer be opened. ZFS does everything possible to avoid this situation, though arbitrary corruption is always possible given the qualities of the underlying filesystem and storage. This typically results in a pool 'disappearing' from the system when it should otherwise be available, though it can also manifest itself as a 'partial' configuration which is missing an unknown number of top level virtual devices. In either case, the configuration can be recovered by exporting the pool (if its visible at all), and re-importing it.

For more information on importing and exporting pools, see 4.7 Storage Pool Migration.

9.5 Repairing a Missing Device

If a device cannot be opened, it displays as UNAVAILABLE in the zpool status output. This means that ZFS was unable to open the device when the pool was first accessed, or the device has since become unavailable. If the device causes a top level virtual device to be unavailable, then nothing in the pool can be accessed. Otherwise, the fault tolerance of the pool may be compromised. In either case, the device simply needs to be reattached to the system in order to restore normal operation.

9.5.1 Physically Reattaching the Device

Exactly how to reattach a missing device depends completely on the device in question. If the device is a network attached drive, connectivity should be restored. If the device is a USB or other removable media, it should be reattached to the system. If the device is a local disk, a controller may have died such that the device is no longer visible to the system. In this case, the controller should be replaced at which point the disks should again be available. Other pathologies exist and depend on the type of hardware and its configuration. If a drive fails such that it is no longer visible to the system (an unlikely event), it should be treated as a damaged device and follow the procedures outlined in 9.6 Repairing a Damaged Device.

9.5.2 Notifying ZFS of Device Availability

Once the device is reattached to the system, ZFS may or may not automatically detect its availability. If the pool was previously faulted, or the system is rebooted as part of the attach procedure, then it automatically rescans all devices when it tries to open the pool. If the pool was degraded and the device was replaced while the system was up, the administrator will need to notify ZFS that the device is now available and should be reopened. This can be done using the zpool online command:

# zpool online tank c0t1d0

For more information on onlining devices, see 4.5.2.2 Bringing a Device Online.

9.6 Repairing a Damaged Device

This section describes how to determine device failure types, clearing transient errors, and replacing a device.

9.6.1 Determining Type of Failure

The term 'damaged device' is rather vague, and can describe a number of possible situations:

• Bit rot -- Over time, random events (such as magnetic influences and cosmic rays) can cause bits stored on disk to flip in unpredictable events. These events are relatively rare, but common enough to cause potential data corruption in large or long-running systems. These errors are typically transient.

• Misdirected reads or writes -- Firmware bugs or hardware faults can cause reads or writes of entire blocks to reference the incorrect location on disk. These errors are typically transient, though a large number may indicate a faulty drive.

• Administrator error -- Administrators can unknowingly overwrite portions of the disk with bad data (such as dd(1)ing /dev/zero over portions of the disk) that cause permanent corruption on disk. These errors are always transient.

• Temporary outage -- A disk may become unavailable for a period time, causing I/Os to fail. This is typically associated with network attached devices, though local disks can experience temporary outages as well. These errors may or may not be transient.

• Bad or flaky hardware -- This is a catch-all for the various problems that bad hardware exhibits. This could be consistent I/O errors, faulty transports causing random corruption, or any number of failures. These errors are typically permanent.

• Offlined device -- If the device is offline, it is assumed that the administrator placed it in this state because it is presumed faulty. Whether or not this is true can be determined by the administrator who placed it in this state.

Determining exactly what is wrong can be a difficult and arduous process. The first step is to examine the error counts in zpool status output:

# zpool status pool -v

The errors are divided into I/O errors and checksum errors, which may indicate the possible failure type. Typical operation predicts a very small number of errors (just a few over long periods of time). If you are seeing large numbers of errors, then it probably indicates impending or complete device failure (although the pathology for administrator error can result in large error counts). The other source of information is the system log. If there are a large number of messages from the SCSI or fibre channel driver, then it probably indicates serious hardware problems. If there are no syslog messages whatsoever, then the damage is likely transient.

The goal is to answer the following question:

Is another error likely to be seen on this device?

Those errors that were one-of-a-kind are considered "transient", and do not indicate potential failure. Those errors which are persistent, or severe enough to indicate potential hardware failure, are considered "fatal". The act of determining the type of error is beyond the scope of any automated software currently available with ZFS, and so much be done manually by the administrator. Once the determination is made, the appropriate action can be taken: either clear the transient errors or replace the device due to fatal errors. These repair procedures are described in the next sections.