The Steward Process: Handling Split-brain in Two-Cluster Global Clusters

Failure of all heartbeats between any two clusters in a global cluster indicates one of the following:

• The remote cluster is faulted.
• All communication links between the two clusters are broken.

In global clusters with more than three clusters, VCS queries the connected clusters to confirm that the remote cluster is truly down. This mechanism is called inquiry.

In a two-cluster setup, VCS uses the Steward process to minimize chances of a wide-area split-brain. The process runs as a standalone binary on a system outside of the global cluster configuration.

Click the thumbnail above to view full-sized image.

When all communication links between any two clusters are lost, each cluster contacts the Steward with an inquiry message. The Steward sends an ICMP ping to the cluster in question and responds with a negative inquiry if the cluster is running or with positive inquiry if the cluster is down. The Steward can also be used configurations with more than two clusters. See Configuring the Steward Process (Optional) for more information.

A Steward is effective only if there are independent paths from each cluster to the host running the Steward. If there is only one path between the two clusters, you must prevent split-brain by confirming manually via telephone or some messaging system with administrators at the remote site if a failure has occurred. By default, VCS global clusters fail over an application across cluster boundaries with administrator confirmation. You can configure automatic failover by setting the ClusterFailOverPolicy attribute to Auto.

If you start a service group on a remote cluster while the service group is running on the primary cluster, data corruption does not occur because the clusters use replicated data. Instead, divergent data sets result, which must be merged manually once the split-brain is resolved. VCS does not automatically take a service group offline after an inter-cluster split-brain is reconnected.