<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Transitional//EN">
<HTML>
<HEAD>
  <!--
    *************************************************************************************
    * Copyright (c) 2005 VERITAS Software Corporation. All rights reserved. *

      For third-party license citations, please see the bottom of this file.
    *************************************************************************************
  -->
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="TEMPLATEBASE" content="vx_vxhelp_2005c">
<meta name="LASTUPDATED" content="06/15/05 07:49:40">
<meta name="PRODUCTNAME" content="VERITAS Software Documentation">
<meta name="FILENAME" content="01_oview_rep.fm">
     
<title>I/O Fencing</title>

</HEAD>

<BODY bgcolor="#FFFFFF" link="#000066" vlink="#0000ff" marginheight="0" topmargin="0">
<basefont=1>

<a name="269179"></a>

<div align="left">
<table border="0" cellspacing="0" cellpadding="0" width="100%">
 <tr>
   <td height="2" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#000000">
   </td>
 </tr>
 <tr>
   <td height="15" width="50%" border="0" cellspacing="0" cellpadding="0" bgcolor="#cccccc">
    <img src="images/vxsymbol.gif" height="20" width="31" border="0" cellspacing="0" cellpadding="0" align="left">
     
     
   </td>
   <td align="right" width="50%" bgcolor="#cccccc">

<!-- Header Breadcrumbs -->
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&lt;&nbsp;<a href="01_oview_rep_sf_ora_rac18_269164.html" title="Previous - Storage Foundation for Oracle RAC OSD Layer Support">Previous</a>&nbsp;&nbsp;|&nbsp;&nbsp;<a href="02_prep_install_rep_sf_ora_rac20_300351.html" title="Next - Preparing to Install Storage Foundation for Oracle RAC">Next</a>
&nbsp;&gt;&nbsp;&nbsp;</font>
   </td>
 </tr>

 <tr>
  <td bgcolor="#CCCCCC" colspan="2" height="15" align="right">
  <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Product:  
<a href="sf_ora_rac_splashpage_docCD_sf_ora_rac1_103036.vxhd" title="Storage Foundation for Oracle RAC Guides title page ">Storage Foundation for Oracle RAC Guides</a>
  &nbsp;&nbsp;
  </font>
  </td>
 </tr>
 <tr>
  <td bgcolor="#CCCCCC" colspan="2" height="15" align="right">
  <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Manual:  
<a href="sfrac_icg_titlepage_docCD_sf_ora_rac2_73700.vxhd" title="Storage Foundation 4.1 for Oracle RAC Installation and Configuration title page ">Storage Foundation 4.1 for Oracle RAC Installation and Configuration</a>
  &nbsp;&nbsp;
  </font>
  </td>
 </tr>


</table>
</div>

<div align="left">
 <table border="0" width="100%" bgcolor="#FFFFFF">
  <tr>
   <td width="100%" valign="top" bgcolor="#FFFFFF">

<!-- Start Main Topic Content -->
        <a name="startcontent"></a>
	
<h1><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="+1" color="#000000"><a name="269179"></a>I/O Fencing</font>
</h1>
 
  
  
  

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="269181"></a>I/O fencing is a feature within a kernel module of Storage Foundation <em>for Oracle RAC</em> designed to guarantee data integrity. This feature works even in the case of faulty cluster communications causing a split-brain condition.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="269183"></a>Understanding Split Brain and the Need for I/O Fencing</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="293514"></a>Split brain is an issue faced by all cluster solutions. To provide high availability, the cluster must be capable of taking corrective action when a node fails. In this situation, Storage Foundation <em>for Oracle RAC</em> configures CVM, CFS, and RAC to reflect the altered membership. </font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="293559"></a>Problems arise when the mechanism that detects the failure breaks down because symptoms appear identical to those of a failed node. For example, if a system in a two-node cluster fails, the system stops sending heartbeats over the private interconnects and the remaining node takes corrective action. However, the failure of private interconnects (instead of the actual nodes) would present identical symptoms and cause each node to determine its peer has departed. This situation typically results in data corruption because both nodes attempt to take control of data storage in an uncoordinated manner.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="293594"></a>In addition to a broken set of private networks, other scenarios can cause this situation. If a system is so busy that it appears to stop responding or &quot;hang,&quot; the other nodes could declare it as dead. This declaration may also occur for nodes using hardware that supports a &quot;break&quot; and &quot;resume&quot; function. When a node drops to PROM level with a break and subsequently resumes operations, the other nodes may declare the system dead even though the system later returns and begins write operations.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="269188"></a>Storage Foundation <em>for Oracle RAC</em> uses a technology called I/O fencing to remove the risk associated with split brain. I/O fencing blocks access to storage from specific nodes; even a node that is alive is unable to cause damage.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="269190"></a>SCSI-3 Persistent Reservations</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="269192"></a>Storage Foundation <em>for Oracle RAC</em> uses SCSI-3 Persistent Reservations (SCSI-3 PR). SCSI-3 PR is designed to resolve the issues of using SCSI reservations in a clustered SAN environment. SCSI-3 PR enables access by multiple nodes to a device and simultaneously blocks access for other nodes. SCSI-3 PR reservations are persistent across SCSI bus resets and SCSI-3 PR supports multiple paths from a host to a disk. In contrast, only one host can use SCSI-2 reservations with one path. If the need arises to block access because of data integrity concerns, only one host and one path remain active. The requirements for larger clusters, with multiple nodes reading and writing to storage in a controlled manner, make SCSI-2 reservations obsolete.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="269194"></a>SCSI-3 PR uses a concept of registration and reservation. Each system registers its own &quot;key&quot; with a SCSI-3 device. Multiple systems registering keys form a membership and establish a reservation, typically set to &quot;Write Exclusive Registrants Only&quot; (WERO). This setting enables only registered systems to perform write operations. For a given disk, only one reservation can exist amidst numerous registrations.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="269195"></a>With SCSI-3 PR technology, blocking write access is as simple as removing a registration from a device. Only registered members can &quot;eject&quot; the registration of another member. A member wishing to eject another member issues a &quot;preempt and abort&quot; command. Ejecting a node is final and &quot;atomic;&quot; an ejected node cannot eject another node.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="269196"></a>In Storage Foundation <em>for Oracle RAC</em>, a node registers the same key for all paths to the device. A single preempt and abort command ejects a node from all paths to the storage device.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="269201"></a>To summarize, the SCSI-3 PR specification describes the method to control access to disks with the registration and reservation mechanism. The method to determine who can register with a disk and when a registered member should eject another node is specific to the implementation. The following paragraphs describe I/O fencing concepts and implementation for Storage Foundation <em>for Oracle RAC</em>.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="269203"></a>I/O Fencing Components</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="269204"></a>I/O Fencing, or simply fencing, gives write access to members of the active cluster and blocks access to non-members. The physical components of I/O fencing in Storage Foundation <em>for Oracle RAC</em> are <em>coordinator</em> disks and <em>data</em> disks. Each component has a unique purpose and uses different physical disk devices.</font>
</p>




<h3><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="2" color="#000000"><a name="293900"></a>Data Disks</font>
</h3>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="293901"></a>Data disks are standard disk devices for data storage and are either physical disks or RAID Logical Units (LUNs). These disks must support SCSI-3 PR and are part of standard VxVM/CVM disk groups.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="293920"></a>CVM is responsible for fencing data disks on a disk group basis. VxVM enables I/O fencing and provides additional features. Disks added to a disk group are automatically fenced, as are new paths discovered to a device.</font>
</p>




<h3><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="2" color="#000000"><a name="269210"></a>Coordinator Disks</font>
</h3>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="293949"></a>Coordinator disks are three (or an odd number greater than three) standard disks or LUNs set aside for I/O fencing during cluster reconfiguration. These disks provide a lock mechanism to determine which nodes get to fence off data drives from other nodes. A node must eject a peer from the coordinator disks before it can fence the peer from the data drives. This concept of racing for control of the coordinator disks to gain the ability to fence data disks is key to understanding prevention of split brain through fencing.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="293971"></a>Coordinator disks do not serve any other purpose in the Storage Foundation <em>for Oracle RAC</em> configuration. Users cannot store data on these disks or include the disks in a disk group for user data. The coordinator disks can be any three disks that support SCSI-3 PR. VERITAS recommends using the smallest possible LUNs for coordinator disks. Since coordinator disks do not store any data, cluster nodes need only register with them and do not need to reserve them.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="269216"></a>I/O Fencing Operations</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="293992"></a>I/O fencing, provided by the kernel-based fencing module (VXFEN), performs identically on node failures and communications failures. When the fencing module on a node is informed of a change in cluster membership by the GAB module, it immediately begins the fencing operation. The node attempts to eject the key for departed node(s) from the coordinator disks using the preempt and abort command. When the node successfully ejects the departed nodes from the coordinator disks, it ejects the departed nodes from the data disks. In a split-brain scenario, both sides of the split would race for control of the coordinator disks. The side winning the majority of the coordinator disks wins the race and fences the loser. The loser then panics and reboots the system.</font>
</p>


<!-- End Main Topic Content -->

   </td>
  </tr>
 </table>
</div>

<div>
<table border="0" cellspacing="0" cellpadding="0" width="100%">
 <tr>
  <td height="18" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#FFFFFF">
  </td>
 </tr>
  <td align="left"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&nbsp;&#94;&nbsp;<a href="01_oview_rep_sf_ora_rac19_269179.html" title="I/O Fencing">Return to Top</a></font>

  </td>
  <td align="right"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&lt;&nbsp;<a href="01_oview_rep_sf_ora_rac18_269164.html" title="Previous - Storage Foundation for Oracle RAC OSD Layer Support">Previous</a>&nbsp;&nbsp;|&nbsp;&nbsp;<a href="02_prep_install_rep_sf_ora_rac20_300351.html" title="Next - Preparing to Install Storage Foundation for Oracle RAC">Next</a>
&nbsp;&gt;&nbsp;&nbsp;</font>
  </td>
 <tr>
   <td height="9" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#FFFFFF">
   </td>
 </tr>

<!-- Footer Breadcrumbs -->
<tr>
<td bgcolor="#FFFFFF" colspan="2" height="15" align="right">
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Product:  
<a href="sf_ora_rac_splashpage_docCD_sf_ora_rac1_103036.vxhd" title="Storage Foundation for Oracle RAC Guides title page ">Storage Foundation for Oracle RAC Guides</a>&nbsp;&nbsp;</font></td>
</tr>

<tr>
<td bgcolor="#FFFFFF" colspan="2" height="15" align="right">
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Manual:  
<a href="sfrac_icg_titlepage_docCD_sf_ora_rac2_73700.vxhd" title="Storage Foundation 4.1 for Oracle RAC Installation and Configuration title page ">Storage Foundation 4.1 for Oracle RAC Installation and Configuration</a>&nbsp;&nbsp;</font></td>
</tr>


 <tr>
  <td colspan="2" align="left"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" color="#000000" align="right" size="-2">
<em>VERITAS Software Corporation
</em></font><br>

<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">www.veritas.com</font><br>
  </td>
 </tr>
</table>
</div>

</BODY>
  <!--
    *************************************************************************************
    * Copyright (c) 2005 VERITAS Software Corporation. All rights reserved. *

    VERITAS, VERITAS SOFTWARE, the VERITAS Logo, and VERITAS Software Documentation 
    are trademarks or registered trademarks of VERITAS Software Corporation 
    or its affiliates in the U.S. and other countries. Other names may be 
    trademarks of their respective owners. 

    Apache Jakarta Lucene:
    Apache Software License
    Version 1.1

    Copyright (c) 2000 The Apache Software Foundation. All rights reserved.

    Redistribution and use in source and binary forms, with or without modification, 
    are permitted provided that the following conditions are met:

    1. Redistributions of source code must retain the above copyright notice, this 
    list of conditions and the following disclaimer.

    2. Redistributions in binary form must reproduce the above copyright notice, this 
    list of conditions and the following disclaimer in the documentation and/or other 
    materials provided with the distribution.

    3. The end-user documentation included with the redistribution, if any, must include 
    the following acknowledgment: "This product includes software developed by the Apache 
    Software Foundation (http://www.apache.org/)." Alternately, this acknowledgment may 
    appear in the software itself, if and wherever such third-party acknowledgments 
    normally appear.

    4. The names "Apache" and "Apache Software Foundation" must not be used to endorse or 
    promote products derived from this software without prior written permission. For 
    written permission, please contact apache@apache.org.

    5. Products derived from this software may not be called "Apache", nor may "Apache" 
    appear in their name, without prior written permission of the Apache Software Foundation.

    THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, 
    BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 
    PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE APACHE SOFTWARE FOUNDATION 
    OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, 
    OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 
    OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 
    ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 
    NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 
    ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

    This software consists of voluntary contributions made by many individuals on behalf 
    of the Apache Software Foundation. For more information on the Apache Software Foundation, 
    please see <http://www.apache.org/>. Portions of this software are based upon public domain 
    software originally written at the National Center for Supercomputing Applications, University 
    of Illinois, Urbana-Champaign.
    
    This product includes WebWindow Software developed by Horst Heistermann.
    *************************************************************************************
  -->
</HTML>
