<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Transitional//EN">
<HTML>
<HEAD>
  <!--
    *************************************************************************************
    * Copyright (c) 2005 VERITAS Software Corporation. All rights reserved. *

      For third-party license citations, please see the bottom of this file.
    *************************************************************************************
  -->
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="TEMPLATEBASE" content="vx_vxhelp_2005c">
<meta name="LASTUPDATED" content="06/17/05 09:52:27">
<meta name="PRODUCTNAME" content="VERITAS Software Documentation">
<meta name="FILENAME" content="08_fence_admin.fm">
     
<title>Troubleshooting Fenced Configurations</title>

</HEAD>

<BODY bgcolor="#FFFFFF" link="#000066" vlink="#0000ff" marginheight="0" topmargin="0">
<basefont=1>

<a name="152004"></a>

<div align="left">
<table border="0" cellspacing="0" cellpadding="0" width="100%">
 <tr>
   <td height="2" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#000000">
   </td>
 </tr>
 <tr>
   <td height="15" width="50%" border="0" cellspacing="0" cellpadding="0" bgcolor="#cccccc">
    <img src="images/vxsymbol.gif" height="20" width="31" border="0" cellspacing="0" cellpadding="0" align="left">
     
     
   </td>
   <td align="right" width="50%" bgcolor="#cccccc">

<!-- Header Breadcrumbs -->
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&lt;&nbsp;<a href="08_fence_admin_sfcfs47_145278.html" title="Previous - Verifying Fenced Configurations">Previous</a>&nbsp;&nbsp;|&nbsp;&nbsp;<a href="09_cvm_admin_sfcfs49_123508.html" title="Next - CVM Administration">Next</a>
&nbsp;&gt;&nbsp;&nbsp;</font>
   </td>
 </tr>

 <tr>
  <td bgcolor="#CCCCCC" colspan="2" height="15" align="right">
  <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Product:  
<a href="sfcfs_splashpage_docCD_sfcfs1_103036.vxhd" title="Storage Foundation Cluster File System Guides title page ">Storage Foundation Cluster File System Guides</a>
  &nbsp;&nbsp;
  </font>
  </td>
 </tr>
 <tr>
  <td bgcolor="#CCCCCC" colspan="2" height="15" align="right">
  <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Manual:  
<a href="sfcfs_iag_titlepage_docCD_sfcfs2_73700.vxhd" title="Cluster File System 4.1 Installation and Administration Guide title page ">Cluster File System 4.1 Installation and Administration Guide</a>
  &nbsp;&nbsp;
  </font>
  </td>
 </tr>


</table>
</div>

<div align="left">
 <table border="0" width="100%" bgcolor="#FFFFFF">
  <tr>
   <td width="100%" valign="top" bgcolor="#FFFFFF">

<!-- Start Main Topic Content -->
        <a name="startcontent"></a>
	
<h1><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="+1" color="#000000"><a name="152004"></a>Troubleshooting Fenced Configurations</font>
</h1>
 
  
  
  

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152005"></a>The following information provides describes network partitioning in a fenced environment. For a more comprehensive explanation of this topic, see the chapter on VCS communications in the <em>VERITAS Cluster Server User's Guide</em>.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="152006"></a>Example of a Preexisting Network Partition (Split-Brain)</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152007"></a>The scenario illustrated below shows a two-node cluster in which the severed cluster interconnect poses a potential split-brain condition. </font>
</p>

<a name="no_figure_caption0"> </a>  <!-- Start Image Code -->
  <!--
   CAUTION: If graphics output is truncated, change the paper size setting 
   of your WebWorks Rasterizer printer driver from "A4" to "Letter" and regenerate your project.
  -->

<p>
<a href="08_fence_admina_t.htm">
<img src="images/08_fence_admina.gif" width="109" height="68" border="0" ></a>
</p>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1"><em>Click the thumbnail above to view full-sized image. </font></em></p>
   <!-- End Image Code -->



<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152437"></a>Because the fencing module operates identically on each system, both nodes assume the other is failed, and carry out fencing operations to confirm. The VCS GAB module on each node determines the peer has failed due to loss of heartbeats and passes the membership change to the fencing module. </font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152439"></a>Each side &quot;races&quot; to gain control of the coordinator disks. Only a registered node can eject the registration of another node, so only one side successfully completes the command on each disk.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152441"></a>The side that successfully ejects the peer from a majority of the coordinator disks wins. The fencing module on the winning side then passes the membership change up to VCS and other higher-level packages registered with the fencing module, allowing VCS to invoke recovery actions. The losing side forces a kernel panic and reboots.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="152442"></a>Recovering from a Preexisting Network Partition (Split-Brain)</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152443"></a>The fencing module <font face="Courier New, Courier, sansserif, sans serif" size="2">vxfen</font> prevents a node from starting up after a network partition and subsequent panic and reboot of a node.</font>
</p>




<h3><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="2" color="#000000"><a name="152444"></a>Example Scenario I</font>
</h3>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152445"></a>Another scenario that could cause similar symptoms would be a two-node cluster with one node shut down for maintenance. During the outage, the private interconnect cables are disconnected.</font>
</p>

<a name="no_figure_caption1"> </a>  <!-- Start Image Code -->
  <!--
   CAUTION: If graphics output is truncated, change the paper size setting 
   of your WebWorks Rasterizer printer driver from "A4" to "Letter" and regenerate your project.
  -->

<p>
<a href="08_fence_admina2_t.htm">
<img src="images/08_fence_admina2.gif" width="109" height="41" border="0" ></a>
</p>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1"><em>Click the thumbnail above to view full-sized image. </font></em></p>
   <!-- End Image Code -->



<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152867"></a>In this scenario:</font>
</p>

<ul><p><img src="images/check.gif" alt="Checkmark">&nbsp;
<a name="152869"></a><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2">	Node 0 wins a coordinator race following to a network failure.</font>
</p><p><img src="images/check.gif" alt="Checkmark">&nbsp;
<a name="152870"></a><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2">	Node 1 panics and reboots.</font>
</p><p><img src="images/check.gif" alt="Checkmark">&nbsp;
<a name="152871"></a><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2">	Node 0 has keys registered on the coordinator disks. When node 1 boots up, it sees the Node 0 keys, but cannot see node 0 in the current GAB membership. It senses a potential preexisting split brain and causes the vxfen module to print an error message to the console. The vxfen module prevents fencing from starting, which, in turn, prevents VCS from coming online. </font>
</p><blockquote>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152872"></a><em>Suggested solution:</em> Shut down Node 1, reconnect the cables, and restart Node 1.</font>
</blockquote>
</ul>


<h3><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="2" color="#000000"><a name="152873"></a>Example Scenario II</font>
</h3>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152874"></a>Similar to scenario I, if private interconnect cables are disconnected in a two-node cluster, Node 1 is fenced out of the cluster, panics, and reboots. If before the private interconnect cables are fixed and Node 1 rejoins the cluster, Node 0 panics and reboots (or just reboots). No node can write to the data disks until the private networks are fixed. This is because GAB membership cannot be formed, therefore the cluster cannot be formed.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152875"></a><em>Suggested solution:</em> Shut down both nodes, reconnect the cables, restart the nodes.</font>
</p>




<h3><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="2" color="#000000"><a name="152876"></a>Example Scenario III</font>
</h3>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152877"></a>Similar to scenario II, if private interconnect cables are disconnected in a two-node cluster, Node 1 is fenced out of the cluster, panics, and reboots. If before the private interconnect cables are fixed and Node 1 rejoins the cluster, Node 0 panics due to hardware failure and cannot come back up, Node 1 cannot rejoin.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152878"></a><em>Suggested solution: </em>Shut down Node 1, reconnect the cables, restart the node. You must then clear the registration of Node 0 from the coordinator disks. </font>
</p>

<ol>
<li value="1">
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152879"></a>On Node 1, type:
</font>
<br>
<font face="Courier New, Courier, sansserif, sans serif" size="2"><a name="152880"></a>&nbsp;&nbsp;#&nbsp;<font face="Courier New, Courier, sansserif, sans serif" size="2"><b>/opt/VRTSvcs/vxfen/bin/vxfenclearpre</b></font>
</font>
<li value="2">
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="152881"></a>Restart the node.
</font>
</ol>

<!-- End Main Topic Content -->

   </td>
  </tr>
 </table>
</div>

<div>
<table border="0" cellspacing="0" cellpadding="0" width="100%">
 <tr>
  <td height="18" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#FFFFFF">
  </td>
 </tr>
  <td align="left"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&nbsp;&#94;&nbsp;<a href="08_fence_admin_sfcfs48_152004.html" title="Troubleshooting Fenced Configurations">Return to Top</a></font>

  </td>
  <td align="right"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&lt;&nbsp;<a href="08_fence_admin_sfcfs47_145278.html" title="Previous - Verifying Fenced Configurations">Previous</a>&nbsp;&nbsp;|&nbsp;&nbsp;<a href="09_cvm_admin_sfcfs49_123508.html" title="Next - CVM Administration">Next</a>
&nbsp;&gt;&nbsp;&nbsp;</font>
  </td>
 <tr>
   <td height="9" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#FFFFFF">
   </td>
 </tr>

<!-- Footer Breadcrumbs -->
<tr>
<td bgcolor="#FFFFFF" colspan="2" height="15" align="right">
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Product:  
<a href="sfcfs_splashpage_docCD_sfcfs1_103036.vxhd" title="Storage Foundation Cluster File System Guides title page ">Storage Foundation Cluster File System Guides</a>&nbsp;&nbsp;</font></td>
</tr>

<tr>
<td bgcolor="#FFFFFF" colspan="2" height="15" align="right">
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Manual:  
<a href="sfcfs_iag_titlepage_docCD_sfcfs2_73700.vxhd" title="Cluster File System 4.1 Installation and Administration Guide title page ">Cluster File System 4.1 Installation and Administration Guide</a>&nbsp;&nbsp;</font></td>
</tr>


 <tr>
  <td colspan="2" align="left"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" color="#000000" align="right" size="-2">
<em>VERITAS Software Corporation
</em></font><br>

<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">www.veritas.com</font><br>
  </td>
 </tr>
</table>
</div>

</BODY>
  <!--
    *************************************************************************************
    * Copyright (c) 2005 VERITAS Software Corporation. All rights reserved. *

    VERITAS, VERITAS SOFTWARE, the VERITAS Logo, and VERITAS Software Documentation 
    are trademarks or registered trademarks of VERITAS Software Corporation 
    or its affiliates in the U.S. and other countries. Other names may be 
    trademarks of their respective owners. 

    Apache Jakarta Lucene:
    Apache Software License
    Version 1.1

    Copyright (c) 2000 The Apache Software Foundation. All rights reserved.

    Redistribution and use in source and binary forms, with or without modification, 
    are permitted provided that the following conditions are met:

    1. Redistributions of source code must retain the above copyright notice, this 
    list of conditions and the following disclaimer.

    2. Redistributions in binary form must reproduce the above copyright notice, this 
    list of conditions and the following disclaimer in the documentation and/or other 
    materials provided with the distribution.

    3. The end-user documentation included with the redistribution, if any, must include 
    the following acknowledgment: "This product includes software developed by the Apache 
    Software Foundation (http://www.apache.org/)." Alternately, this acknowledgment may 
    appear in the software itself, if and wherever such third-party acknowledgments 
    normally appear.

    4. The names "Apache" and "Apache Software Foundation" must not be used to endorse or 
    promote products derived from this software without prior written permission. For 
    written permission, please contact apache@apache.org.

    5. Products derived from this software may not be called "Apache", nor may "Apache" 
    appear in their name, without prior written permission of the Apache Software Foundation.

    THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, 
    BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 
    PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE APACHE SOFTWARE FOUNDATION 
    OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, 
    OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 
    OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 
    ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 
    NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 
    ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

    This software consists of voluntary contributions made by many individuals on behalf 
    of the Apache Software Foundation. For more information on the Apache Software Foundation, 
    please see <http://www.apache.org/>. Portions of this software are based upon public domain 
    software originally written at the National Center for Supercomputing Applications, University 
    of Illinois, Urbana-Champaign.
    
    This product includes WebWindow Software developed by Horst Heistermann.
    *************************************************************************************
  -->
</HTML>
