I/O Fencing Startup

The startup sequence of I/O fencing is designed to prevent pre-existing network problems from affecting cluster membership. The startup sequence ensures that all members can access the coordinator disks and determine if a node should be allowed to join a cluster.

The startup sequence algorithm is as follows:

1. Determine which disks are to be used as coordinator disks.
   1. Read the file /etc/vxfendg to determine the name of the VERITAS Volume Manager disk group containing the coordinator disks. See the VCS Installation Guide for information on creating the coordinator disk group.
   2. Use Volume Manager tools to determine the disks in the disk group and the paths available to these disks.
   3. Populate the file /etc/vxfentab with this information.
2. Start the fencing driver.
   1. The fencing driver first reads the serial numbers of the coordinator disks from the file /etc/vxfentab and builds an in-memory list of these drives.
   2. The driver then determines if it is the first node to start fencing. If other members are up and operating on GAB port B, it asks for a configuration snapshot from a running member. This is done to verify members in the cluster see the same coordinator disks. Otherwise, the fencing driver enters an error state.
3. Determine if a network partition exists.
   1. Determine if any node has registered keys on the coordinator disks.
   2. If any keys are present, verify the corresponding member can be seen in the current GAB membership. If the member cannot be seen, the fencing driver assumes the node starting up has been fenced out of the cluster due to a network partition. The fencing driver prints a warning to the console and the system log about a pre-existing network partition and does not start.
   3. If the owners of the coordinator disks can be seen, or if no keys are seen on disk, the fencing driver proceeds.
4. Register keys with each coordinator disk in sequence.

I/O Fencing Scenario: Preexisting Network Partition

The fencing module prevents a node from starting up after network partition and the subsequent panic and reboot. Another scenario that could cause similar symptoms would be a two-node cluster with one node shut down for maintenance. During the outage, the private interconnect cables are disconnected.

Click the thumbnail above to view full-sized image.

The following steps describe this scenario:

1. Node 0 wins a coordinator race following a network failure.
2. Node 1 panics and reboots.
3. Node 0 has keys registered on the coordinator disks. When node 1 boots up, it sees the node 0 keys, but cannot see node 0 in the current GAB membership. It senses a potential preexisting split brain and causes the vxfen module to print an error message to the console. The vxfen module prevents fencing from starting, which, in turn, prevents VCS from coming online.

   To recover from this situation, shut down node 1, reconnect the private interconnect, and restart node 1.

I/O Fencing Scenario: Partition In Time

In this scenario, the node that has registered keys on disk fails and is not present in the GAB membership to allow other members to join.

Click the thumbnail above to view full-sized image.

1. In the first failure, node 1 fails, and is fenced out by ejecting the node from the coordinator disks.
2. Before node 1 can be restarted, node 0 fails.
3. When node 1 restarts, it sees the keys left behind by node 0, but cannot see node 0 in the GAB membership. The fencing driver prints an error message.
4. The operator runs the vxfenclearpre utility to clear the keys left by node 0 after physically verifying that node 0 is down. The operator then reboots node1, which comes up normally.