<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Transitional//EN">
<HTML>
<HEAD>
  <!--
    *************************************************************************************
    * Copyright (c) 2005 VERITAS Software Corporation. All rights reserved. *

      For third-party license citations, please see the bottom of this file.
    *************************************************************************************
  -->
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="TEMPLATEBASE" content="vx_vxhelp_2005c">
<meta name="LASTUPDATED" content="06/17/05 09:52:25">
<meta name="PRODUCTNAME" content="VERITAS Software Documentation">
<meta name="FILENAME" content="06_cfs_archtcr.fm">
     
<title>About CFS</title>

</HEAD>

<BODY bgcolor="#FFFFFF" link="#000066" vlink="#0000ff" marginheight="0" topmargin="0">
<basefont=1>

<a name="156886"></a>

<div align="left">
<table border="0" cellspacing="0" cellpadding="0" width="100%">
 <tr>
   <td height="2" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#000000">
   </td>
 </tr>
 <tr>
   <td height="15" width="50%" border="0" cellspacing="0" cellpadding="0" bgcolor="#cccccc">
    <img src="images/vxsymbol.gif" height="20" width="31" border="0" cellspacing="0" cellpadding="0" align="left">
     
     
   </td>
   <td align="right" width="50%" bgcolor="#cccccc">

<!-- Header Breadcrumbs -->
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&lt;&nbsp;<a href="06_cfs_archtcr_sfcfs33_156899.html" title="Previous - The Role of Component Products">Previous</a>&nbsp;&nbsp;|&nbsp;&nbsp;<a href="06_cfs_archtcr_sfcfs35_155432.html" title="Next - About CVM">Next</a>
&nbsp;&gt;&nbsp;&nbsp;</font>
   </td>
 </tr>

 <tr>
  <td bgcolor="#CCCCCC" colspan="2" height="15" align="right">
  <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Product:  
<a href="sfcfs_splashpage_docCD_sfcfs1_103036.vxhd" title="Storage Foundation Cluster File System Guides title page ">Storage Foundation Cluster File System Guides</a>
  &nbsp;&nbsp;
  </font>
  </td>
 </tr>
 <tr>
  <td bgcolor="#CCCCCC" colspan="2" height="15" align="right">
  <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Manual:  
<a href="sfcfs_iag_titlepage_docCD_sfcfs2_73700.vxhd" title="Cluster File System 4.1 Installation and Administration Guide title page ">Cluster File System 4.1 Installation and Administration Guide</a>
  &nbsp;&nbsp;
  </font>
  </td>
 </tr>


</table>
</div>

<div align="left">
 <table border="0" width="100%" bgcolor="#FFFFFF">
  <tr>
   <td width="100%" valign="top" bgcolor="#FFFFFF">

<!-- Start Main Topic Content -->
        <a name="startcontent"></a>
	
<h1><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="+1" color="#000000"><a name="156886"></a> About CFS</font>
</h1>
         
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155351"></a>SFCFS uses a master/slave, or <em>primary</em>/<em>secondary</em>, architecture to manage file system metadata on shared disk storage. The first node that mounts is the primary node and the remaining nodes are secondaries. Secondaries send requests to the primary to perform metadata updates.The primary node updates the metadata and maintains the file system's metadata update intent log. Data can also be updated from any node directly to shared storage. Other file system operations, such as allocating or deleting files, can originate from any node in the cluster.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155352"></a>If the server on which the SFCFS primary is running fails, the remaining cluster nodes elect a new primary. See <a href="06_cfs_archtcr_sfcfs34_156886.html#156304">Distributing Load on a Cluster</a> for details on the election process. The new primary reads the file system intent log and completes any metadata updates that were in process at the time of the failure. Application I/O from other nodes may block during this process and cause a delay. When the file system is again consistent, application processing resumes.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155356"></a>Because nodes using a cluster file system in secondary mode do not update file system metadata directly, failure of a secondary node does not require metadata repair. SFCFS recovery from secondary node failure is therefore faster than from primary node failure.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="155357"></a>SFCFS and the Group Lock Manager</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155359"></a>SFCFS uses the VERITAS Group Lock Manager (GLM) to reproduce UNIX single-host file system semantics in clusters. UNIX file systems make writes appear atomic. This means when an application writes a stream of data to a file, a subsequent application reading from the same area of the file retrieves the new data, even if it has been cached by the file system and not yet written to disk. Applications cannot retrieve stale data or partial results from a previous write.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155360"></a>To reproduce single-host write semantics, system caches must be kept coherent, and each must instantly reflect updates to cached data, regardless of the node from which they originate.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="155361"></a>Asymmetric Mounts</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155362"></a>A VxFS file system mounted with the <font face="Courier New, Courier, sansserif, sans serif" size="2">mount</font> <font face="Courier New, Courier, sansserif, sans serif" size="2">&#8211;o</font> <font face="Courier New, Courier, sansserif, sans serif" size="2">cluster</font> option is a cluster, or <em>shared,</em> mount, as opposed to a non-shared or <em>local</em> mount. A file system mounted in shared mode must be on a VxVM shared volume in a cluster environment. A local mount cannot be remounted in shared mode and a shared mount cannot be remounted in local mode. File systems in a cluster can be mounted with different read/write options. These are called <em>asymmetric</em> mounts.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155364"></a>Asymmetric mounts allow shared file systems to be mounted with different read/write capabilities. One node in the cluster can mount read/write, while other nodes mount read-only.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155365"></a>You can specify the cluster read-write (<font face="Courier New, Courier, sansserif, sans serif" size="2">crw</font>) option when you first mount the file system, or the options can be altered when doing a remount (<font face="Courier New, Courier, sansserif, sans serif" size="2">mount</font> <font face="Courier New, Courier, sansserif, sans serif" size="2">&#8211;o</font> <font face="Courier New, Courier, sansserif, sans serif" size="2">remount</font>). The first column in the following table shows the mode in which the primary is mounted. The check marks indicate the mode secondary mounts can use. See the <font face="Courier New, Courier, sansserif, sans serif" size="2">mount_vxfs</font>(1M) man page for details on the cluster read-write (<font face="Courier New, Courier, sansserif, sans serif" size="2">crw</font>) mount option.
<Table width="100%" border="0" cellpadding="1" cellspacing="0" >
<tr valign="top"><td><b><a name="155368"></a></b>
</td><td colspan=2 rowspan=1><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155370"></a>&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;Secondary</font>
</p>

</td><td><b><a name="155374"></a></b>
</td><td><b><a name="155376"></a></b>
</td></tr><tr valign="top"><td><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155378"></a></font>
</p>
</td><td><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155380"></a></font>
</p>

</td><td><b><a name="155382"></a>ro</b>
</td><td><b><a name="155384"></a>rw</b>
</td><td><b><a name="155386"></a>ro, crw</b>
</td></tr><tr valign="top"><td><b><a name="155388"></a>Primary</b>
</td><td><b><a name="155390"></a>ro</b>
</td><td><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155392"></a></font>
</p>

</td><td><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155394"></a></font>
</p>

</td><td><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155396"></a></font>
</p>

</td></tr><tr valign="top"><td><b><a name="155398"></a></b>
</td><td><b><a name="155400"></a>rw</b>
</td><td><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155402"></a></font>
</p>

</td><td><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155404"></a></font>
</p>

</td><td><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155406"></a></font>
</p>

</td></tr><tr valign="top"><td><b><a name="155408"></a></b>
</td><td><b><a name="155410"></a>ro, crw</b>
</td><td><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155412"></a></font>
</p>

</td><td><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155414"></a></font>
</p>

</td><td><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155416"></a></font>
</p>

</td></tr>
</Table>


<Table width="100%" border="0" cellpadding="1" cellspacing="0">
<tr><td>
</Table></font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155424"></a>Mounting the primary with only the <font face="Courier New, Courier, sansserif, sans serif" size="2">&#8211;o</font> <font face="Courier New, Courier, sansserif, sans serif" size="2">cluster,ro</font> option prevents the secondaries from mounting in a different mode; that is, read/write. Note that <font face="Courier New, Courier, sansserif, sans serif" size="2">rw</font> implies read/write capability throughout the cluster. </font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="155425"></a>Parallel I/O</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155426"></a>Some distributed applications read and write to the same file concurrently from one or more nodes in the cluster; for example, any distributed application where one thread appends to a file and there are one or more threads reading from various regions in the file. Several high-performance compute (HPC) applications can also benefit from this feature, where concurrent I/O is performed on the same file. Applications do not require any changes to use parallel I/O feature.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155427"></a>Traditionally, the entire file is locked to perform I/O to a small region. To support parallel I/O, CFS locks ranges in a file that correspond to an I/O request. The granularity of the locked range is a page. Two I/O requests conflict if at least one is a write request, and the I/O range of the request overlaps the I/O range of the other.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155428"></a>The parallel I/O feature enables I/O to a file by multiple threads concurrently, as long as the requests do not conflict. Threads issuing concurrent I/O requests could be executing on the same node, or on a different node in the cluster.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155429"></a>An I/O request that requires allocation is not executed concurrently with other I/O requests. Note that when a writer is extending the file and readers are lagging behind, block allocation is not necessarily done for each extending write.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155430"></a>If the file size can be predetermined, the file can be preallocated to avoid block allocations during I/O. This improves the concurrency of applications performing parallel I/O to the file. Parallel I/O also avoids unnecessary page cache flushes and invalidations using range locking, without compromising the cache coherency across the cluster.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="155431"></a>For applications that update the same file from multiple nodes, the <font face="Courier New, Courier, sansserif, sans serif" size="2">-nomtime</font> mount option provides further concurrency. Modification and change times of the file are not synchronized across the cluster, which eliminates the overhead of increased I/O and locking. </font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="156287"></a>CFS Namespace</font>
</h2>
 
<blockquote>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156288"></a>The mount point name must remain the same for all nodes mounting the same cluster file system. This is required for the VCS mount agents (online, offline, and monitoring) to work correctly.</font>
</blockquote>



<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="156289"></a>SFCFS Backup Strategies</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156290"></a>The same backup strategies used for standard VxFS can be used with SFCFS because the APIs and commands for accessing the namespace are the same. <em>File System checkpoints</em> provide an on-disk, point-in-time copy of the file system. Because performance characteristics of a checkpointed file system are better in certain I/O patterns, they are recommended over file system snapshots (described below) for obtaining a frozen image of the cluster file system.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156291"></a><em>File System snapshots</em> are another method of a file system on-disk frozen image. The frozen image is non-persistent, in contrast to the checkpoint feature. A snapshot can be accessed as a read-only mounted file system to perform efficient online backups of the file system. Snapshots implement &quot;copy-on-write&quot; semantics that incrementally copy data blocks when they are overwritten on the snapped file system. Snapshots for cluster file systems extend the same copy-on-write mechanism for the I/O originating from any cluster node.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156292"></a>Mounting snapshot file system for backups increases the load on the system because of the resources used to perform copy-on-writes and to read data blocks from snapshot. In this situation, cluster snapshots can be used to do <em>off-host</em> backups. Off-host backups reduce the load of a backup application from the primary server. Overhead from remote snapshots is small when compared to overall snapshot overhead. Therefore, running a backup application by mounting a snapshot from a relatively less loaded node is beneficial to overall cluster performance.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156293"></a>There are several characteristics of a cluster snapshot, including:</font>
</p>

<p>
<ul type="disc"><li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156295"></a>A snapshot for a cluster mounted file system can be mounted on any node in a cluster. The file system can be a primary, secondary, or secondary-only. A stable image of the file system is provided for writes from any node.</font>
</li>
<li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156296"></a>Multiple snapshots of a cluster file system can be mounted on the same or different cluster node.</font>
</li>
<li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156297"></a>A snapshot is accessible only on the node mounting a snapshot. The snapshot device cannot be mounted on two nodes simultaneously.</font>
</li>
<li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156298"></a>The device for mounting a snapshot can be a local disk or a shared volume. A shared volume is used exclusively by a snapshot mount and is not usable from other nodes as long as the snapshot is active on that device. </font>
</li>
<li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156299"></a>On the node mounting a snapshot, the snapped file system cannot be unmounted while the snapshot is mounted.</font>
</li>
<li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156300"></a>A SFCFS snapshot ceases to exist if it is unmounted or the node mounting the snapshot fails. However, a snapshot is not affected if a node leaves or joins the cluster.</font>
</li>
<li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156301"></a>A snapshot of a read-only mounted file system cannot be taken. It is possible to mount snapshot of a cluster file system only if the snapped cluster file system is mounted with the <font face="Courier New, Courier, sansserif, sans serif" size="2">crw</font> option.</font>
</li>
</ul>
</p>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156302"></a>In addition to file-level frozen images, there are volume-level alternatives available for shared volumes using mirror split and rejoin. Features such as Fast Mirror Resync and Space Optimized snapshot are also available. See the <em>VERITAS Volume Manager System Administrator's Guide</em> for details.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="156403"></a>Synchronizing Time on Cluster File Systems</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156405"></a>SFCFS requires that the system clocks on all nodes are synchronized using some external component such as the Network Time Protocol (NTP) daemon. If the nodes are not in sync, timestamps for creation (<font face="Courier New, Courier, sansserif, sans serif" size="2">ctime</font>) and modification (<font face="Courier New, Courier, sansserif, sans serif" size="2">mtime</font>) may not be consistent with the sequence in which operations actually happened.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="156304"></a>Distributing Load on a Cluster</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156305"></a>Because the primary node on the cluster performs all metadata updates for a given file system, it makes sense to distribute this load by designating different nodes to serve as primary for each of the cluster file systems in use. Distributing workload in a cluster provides performance and failover advantages. Because each cluster mounted file system can have a different node as its primary, SFCFS enables easy load distribution. </font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156307"></a>For example, if you have eight file systems and four nodes, designating two file systems per node as the primary is beneficial. The first node that mounts a file system becomes the primary for that file system.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156308"></a>You can also use the <font face="Courier New, Courier, sansserif, sans serif" size="2">fsclustadm</font> to designate a SFCFS primary. The <font face="Courier New, Courier, sansserif, sans serif" size="2">fsclustadm</font> <font face="Courier New, Courier, sansserif, sans serif" size="2">setprimary</font> mount point can be used to change the primary. This change to the primary is not persistent across unmounts or reboots. The change is in effect as long as one or more nodes in the cluster have the file system mounted. The primary selection policy can also be defined by a VCS attribute associated with the SFCFS mount resource.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156309"></a>A node can be mounted using the <font face="Courier New, Courier, sansserif, sans serif" size="2">mount</font> <font face="Courier New, Courier, sansserif, sans serif" size="2">&#8211;seconly</font> option, the result of which the node does not assume the primary mode for the file system. If a primary node fails and the remaining nodes are have the file systems mounted as secondary-only, the file system is disabled. The <font face="Courier New, Courier, sansserif, sans serif" size="2">mount</font> <font face="Courier New, Courier, sansserif, sans serif" size="2">&#8211;seconly</font> option overrides the <font face="Courier New, Courier, sansserif, sans serif" size="2">fsclustadm</font> command. See the <font face="Courier New, Courier, sansserif, sans serif" size="2">mount_vxfs</font>(1M) man page for details on the <font face="Courier New, Courier, sansserif, sans serif" size="2">&#8211;seconly</font> option.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156310"></a>Note that recovery for the primary takes more time than the secondary. Distributing primaries across the cluster would provide a uniform recovery time in the case of a node failure.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="156312"></a>File System Tuneables</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156313"></a>Tuneable parameters are updated at the time of mount using the <font face="Courier New, Courier, sansserif, sans serif" size="2">tunefstab</font> file or <font face="Courier New, Courier, sansserif, sans serif" size="2">vxtunefs</font> command. The file system <font face="Courier New, Courier, sansserif, sans serif" size="2">tunefs</font> parameters are set to be identical on all nodes by propagating the parameters to each cluster node. When the file system is mounted on the node, the <font face="Courier New, Courier, sansserif, sans serif" size="2">tunefs</font> parameters of the primary node are used. The <font face="Courier New, Courier, sansserif, sans serif" size="2">tunefstab</font> file on the node is used if this is the first node to mount the file system. VERITAS recommends that this file be identical on each node.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="156230"></a>Split-Brain and Jeopardy Handling</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156231"></a>A <em>split-brain</em> occurs when the cluster membership view differs among the cluster nodes, increasing the chance of data corruption. Membership change also occurs when all private-link cluster interconnects fail simultaneously, or when a node is unable to respond to heartbeat messages.With I/O fencing, the potential for data corruption is eliminated. I/O fencing requires disks that support SCSI-3 PGR. </font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="156232"></a>Jeopardy State</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156233"></a>In the absence of I/O fencing, SFCFS installation requires two heartbeat links. When a node is down to a single heartbeat connection, SFCFS can no longer discriminate between loss of a system and loss of the final network connection. This state is defined as <em>jeopardy</em>.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156234"></a>SFCFS employs jeopardy to prevent data corruption following a split-brain. Note that in certain scenarios, the possibility of data corruption remains. For example:</font>
</p>

<p>
<ul type="disc"><li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156235"></a>All links go down simultaneously.</font>
</li>
<li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156236"></a>A node hangs and is unable to respond to heartbeat messages.</font>
</li>
</ul>
</p>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156237"></a>To eliminate the chance of data corruption in these scenarios, I/O fencing is required. With I/O fencing, the jeopardy state does not require special handling by the SFCFS stack.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="156239"></a>Jeopardy Handling</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156240"></a>For installations that do not support SCSI-3 PGR, potential split-brain conditions are safeguarded by <em>jeopardy handling</em>. If any cluster node fails following a jeopardy state notification, the cluster file system mounted on the failed nodes is disabled. If a node fails after the jeopardy state notification, all cluster nodes also leave the shared disk group membership.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="156241"></a>Recovering from Jeopardy</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156242"></a>The disabled file system can be restored by a force unmount and the resource can be brought online without rebooting, which also brings the shared disk group resource online. Note that if the jeopardy condition is not fixed, the nodes are susceptible to leaving the cluster again on subsequent node failure. For a detailed explanation of this topic, see the <em>VERITAS Cluster Server User's Guide</em>.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="157295"></a>Fencing</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="157296"></a>With the use of I/O enabled fencing, all remaining cases with the potential to corrupt data (for which jeopardy handling cannot protect) are addressed. For more information see <a href="08_fence_admin_sfcfs42_123508.html#123508">Fencing Administration</a>.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="157300"></a></font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="156479"></a>Single Network Link and Reliability</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156480"></a>Certain environments may prefer using a single private link or a pubic network for connecting nodes in a cluster, despite the loss of redundancy for dealing with network failures. The benefits of this approach include simpler hardware topology and lower costs; however, there is obviously a tradeoff with high availability.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156481"></a>For the above environments, SFCFS provides the option of a single private link, or using the public network as the private link if I/O fencing is present. Note that these nodes start in jeopardy state, as described in <a href="08_fence_admin_sfcfs43_145308.html#145308">I/O Fencing</a>. I/O fencing is used to handle split-brain scenarios. The option for single network is given during installation.</font>
</p>




<h3><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="2" color="#000000"><a name="156485"></a>Low Priority Link</font>
</h3>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156486"></a>LLT can be configured to use a low-priority network link as a backup to normal heartbeat channels. Low-priority links are typically configured on the customer's public or administrative network. This typically results in a completely different network infrastructure than the cluster private interconnect, and reduces the chance of a single point of failure bringing down all links. The low-priority link is not used for cluster membership traffic until it is the only remaining link. In normal operation, the low-priority link carries only heartbeat traffic for cluster membership and link state maintenance. The frequency of heartbeats drops 50 percent to reduce network overhead. When the low-priority link is the only remaining network link, LLT also switches over all cluster status traffic. Following repair of any configured private link, LLT returns cluster status traffic to the high-priority link.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156487"></a>LLT links can be added or removed while clients are connected. Shutting down GAB or the high-availability daemon, <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">HAD,</font> is not required.</font>
</p>

</ol>
<p><img src="images/stepintro.gif" height="16" width="16">
<font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="2" color="#000000"><a name="156488"></a>&nbsp;&nbsp;<b>To add a link</b></font>
</p>
<br>
<font face="Courier New, Courier, sansserif, sans serif" size="2"><a name="156489"></a>#&nbsp;<font face="Courier New, Courier, sansserif, sans serif" size="2"><b>lltconfig&nbsp;-d&nbsp;</b></font><font face="Courier New, Courier, sansserif, sans serif" size="2"><b>device</b></font><font face="Courier New, Courier, sansserif, sans serif" size="2"><b>&nbsp;-t&nbsp;</b></font><em>tag</em>
</font>
</ol>
<p><img src="images/stepintro.gif" height="16" width="16">
<font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="2" color="#000000"><a name="156490"></a>&nbsp;&nbsp;<b>To remove a link</b></font>
</p>
<br>
<font face="Courier New, Courier, sansserif, sans serif" size="2"><a name="156491"></a><font face="Courier New, Courier, sansserif, sans serif" size="2"><b>lltconfig&nbsp;-u&nbsp;</b></font><font face="Courier New, Courier, sansserif, sans serif" size="2"><b>tag</b></font>
</font>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156493"></a>Changes take effect immediately and are lost on the next reboot. For changes to span reboots you must also update <font face="Courier New, Courier, sansserif, sans serif" size="2">/etc/llttab</font>.</font>
</p>

<hr noshade size="2" width="100%" color="#000000">
<img src="images/note.gif" alt="Note " height="16" width="16">&nbsp;
<font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="2" color="#000000">
<a name="156494"></a><b>Note  &nbsp;&nbsp;</b>
</font>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2" color="#000000">LLT clients do not recognize the difference unless only one link is available and GAB declares jeopardy.
</font>
<hr noshade size="2" width="100%" color="#000000">



<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="156566"></a>I/O Error Handling Policy</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156567"></a>I/O errors can occur for several reasons, including failures of Fibre Channel link, host-bus adapters, and disks. SFCFS disables the file system on the node encountering I/O errors. The file system remains available from other nodes.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="156568"></a>After the hardware error is fixed (for example, the Fibre Channel link is reestablished), the file system can be force unmounted and the mount resource can be brought online from the disabled node to reinstate the file system.</font>
</p>


<!-- End Main Topic Content -->

   </td>
  </tr>
 </table>
</div>

<div>
<table border="0" cellspacing="0" cellpadding="0" width="100%">
 <tr>
  <td height="18" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#FFFFFF">
  </td>
 </tr>
  <td align="left"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&nbsp;&#94;&nbsp;<a href="06_cfs_archtcr_sfcfs34_156886.html" title="About CFS">Return to Top</a></font>

  </td>
  <td align="right"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&lt;&nbsp;<a href="06_cfs_archtcr_sfcfs33_156899.html" title="Previous - The Role of Component Products">Previous</a>&nbsp;&nbsp;|&nbsp;&nbsp;<a href="06_cfs_archtcr_sfcfs35_155432.html" title="Next - About CVM">Next</a>
&nbsp;&gt;&nbsp;&nbsp;</font>
  </td>
 <tr>
   <td height="9" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#FFFFFF">
   </td>
 </tr>

<!-- Footer Breadcrumbs -->
<tr>
<td bgcolor="#FFFFFF" colspan="2" height="15" align="right">
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Product:  
<a href="sfcfs_splashpage_docCD_sfcfs1_103036.vxhd" title="Storage Foundation Cluster File System Guides title page ">Storage Foundation Cluster File System Guides</a>&nbsp;&nbsp;</font></td>
</tr>

<tr>
<td bgcolor="#FFFFFF" colspan="2" height="15" align="right">
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Manual:  
<a href="sfcfs_iag_titlepage_docCD_sfcfs2_73700.vxhd" title="Cluster File System 4.1 Installation and Administration Guide title page ">Cluster File System 4.1 Installation and Administration Guide</a>&nbsp;&nbsp;</font></td>
</tr>


 <tr>
  <td colspan="2" align="left"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" color="#000000" align="right" size="-2">
<em>VERITAS Software Corporation
</em></font><br>

<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">www.veritas.com</font><br>
  </td>
 </tr>
</table>
</div>

</BODY>
  <!--
    *************************************************************************************
    * Copyright (c) 2005 VERITAS Software Corporation. All rights reserved. *

    VERITAS, VERITAS SOFTWARE, the VERITAS Logo, and VERITAS Software Documentation 
    are trademarks or registered trademarks of VERITAS Software Corporation 
    or its affiliates in the U.S. and other countries. Other names may be 
    trademarks of their respective owners. 

    Apache Jakarta Lucene:
    Apache Software License
    Version 1.1

    Copyright (c) 2000 The Apache Software Foundation. All rights reserved.

    Redistribution and use in source and binary forms, with or without modification, 
    are permitted provided that the following conditions are met:

    1. Redistributions of source code must retain the above copyright notice, this 
    list of conditions and the following disclaimer.

    2. Redistributions in binary form must reproduce the above copyright notice, this 
    list of conditions and the following disclaimer in the documentation and/or other 
    materials provided with the distribution.

    3. The end-user documentation included with the redistribution, if any, must include 
    the following acknowledgment: "This product includes software developed by the Apache 
    Software Foundation (http://www.apache.org/)." Alternately, this acknowledgment may 
    appear in the software itself, if and wherever such third-party acknowledgments 
    normally appear.

    4. The names "Apache" and "Apache Software Foundation" must not be used to endorse or 
    promote products derived from this software without prior written permission. For 
    written permission, please contact apache@apache.org.

    5. Products derived from this software may not be called "Apache", nor may "Apache" 
    appear in their name, without prior written permission of the Apache Software Foundation.

    THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, 
    BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 
    PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE APACHE SOFTWARE FOUNDATION 
    OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, 
    OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 
    OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 
    ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 
    NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 
    ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

    This software consists of voluntary contributions made by many individuals on behalf 
    of the Apache Software Foundation. For more information on the Apache Software Foundation, 
    please see <http://www.apache.org/>. Portions of this software are based upon public domain 
    software originally written at the National Center for Supercomputing Applications, University 
    of Illinois, Urbana-Champaign.
    
    This product includes WebWindow Software developed by Horst Heistermann.
    *************************************************************************************
  -->
</HTML>
