VCS I/O Fencing

When communication between cluster nodes fails, ensuring data integrity involves determining who remains in the cluster (membership) and blocking storage access from any system that is not an acknowledged member of the cluster (fencing). VCS provides a new capability called I/O fencing to meet this need.

VCS I/O Fencing Components

Coordinator Disks

VCS uses special-purpose disks, called coordinator disks, for I/O fencing during cluster membership change. These are three standard disks or LUNs, which together act as a global lock device during a cluster reconfiguration. VCS uses this lock mechanism to determine which nodes remain in a cluster and which node gets to fence off data drives from other nodes.

Click the thumbnail above to view full-sized image.

Coordinator disks cannot be used for any other purpose in the VCS configuration. You cannot store data on these disks or include the disks in a disk group used by user data. Any disks that support SCSI-III Persistent Reservation can serve as coordinator disks. VERITAS recommends the smallest possible LUNs for coordinator use.

Fencing Module

Each system in the cluster runs a kernel module called vxfen, or the fencing module. This module works to maintain tight control on cluster membership. It is responsible for the following actions:

• Registering with the coordinator disks during normal operation.
• Racing for control of the coordinator disks during membership changes.

SCSI-III Persistent Reservations

VCS I/O fencing uses SCSI-III Persistent Reservation (SCSI-III PR), which is an enhancement to the SCSI specification. SCSI-III PR resolves the issues of using SCSI reservations in a modern clustered SAN environment. It supports multiple nodes accessing a device and blocking access to other nodes. SCSI-III PR ensures persistent reservations across SCSI bus resets. It also supports multiple paths from a host to a disk.

SCSI-III PR uses a concept of registration and reservation. Systems wishing to participate register a key with a SCSI-III device. Multiple systems registering a key form a membership. Registered systems can then establish a reservation, which is typically set to Write Exclusive Registrants Only (WERO). This means that only registered systems can write.

SCSI-III PR technology makes blocking write access as simple as removing a registration from a device. If node A wishes to block node B, it removes node B's registration by issuing a "preempt and abort" command. Only registered members can "eject" the registration of other members. Once a node is ejected, it cannot eject other nodes. This makes the process of ejecting final and "atomic."

The SCSI-III PR specification simply describes the method to control access to disks with the registration and reservation mechanism. The method to determine who can register with a disk and when a registered member should eject another node is implementation-specific.

Data Disks

Data disks are standard disk devices used for data storage. These can be physical disks or RAID Logical Units (LUNs). These disks must support SCSI-III PR. Data disks are incorporated in standard disk groups managed using VERITAS Volume Manager.

The VCS DiskGroup agent is responsible for fencing failover disk groups and Cluster Volume Manager (CVM) handles any shared CVM disk groups.