<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Transitional//EN">
<HTML>
<HEAD>
  <!--
    *************************************************************************************
    * Copyright (c) 2005 VERITAS Software Corporation. All rights reserved. *

      For third-party license citations, please see the bottom of this file.
    *************************************************************************************
  -->
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
<meta name="TEMPLATEBASE" content="vx_vxhelp_2005c">
<meta name="LASTUPDATED" content="06/13/05 10:42:24">
<meta name="PRODUCTNAME" content="VERITAS Software Documentation">
<meta name="FILENAME" content="03config.fm">
     
<title>Configuration Concepts</title>

</HEAD>

<BODY bgcolor="#FFFFFF" link="#000066" vlink="#0000ff" marginheight="0" topmargin="0">
<basefont=1>

<a name="121109"></a>

<div align="left">
<table border="0" cellspacing="0" cellpadding="0" width="100%">
 <tr>
   <td height="2" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#000000">
   </td>
 </tr>
 <tr>
   <td height="15" width="50%" border="0" cellspacing="0" cellpadding="0" bgcolor="#cccccc">
    <img src="images/vxsymbol.gif" height="20" width="31" border="0" cellspacing="0" cellpadding="0" align="left">
     
     
   </td>
   <td align="right" width="50%" bgcolor="#cccccc">

<!-- Header Breadcrumbs -->
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&lt;&nbsp;<a href="03config_vcs_agents105_127578.html" title="Previous - Cluster Heartbeats">Previous</a>&nbsp;&nbsp;|&nbsp;&nbsp;<a href="03config_vcs_agents107_132329.html" title="Next - Configuring the Agent Using the Wizard">Next</a>
&nbsp;&gt;&nbsp;&nbsp;</font>
   </td>
 </tr>

 <tr>
  <td bgcolor="#CCCCCC" colspan="2" height="15" align="right">
  <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Product:  
<a href="vcs_agents_splashpage_docCD_vcs_agents1_103036.vxhd" title="Cluster Server Enterprise Agents Guides title page ">Cluster Server Enterprise Agents Guides</a>
  &nbsp;&nbsp;
  </font>
  </td>
 </tr>
 <tr>
  <td bgcolor="#CCCCCC" colspan="2" height="15" align="right">
  <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Manual:  
<a href="vcshp_srdf_titlepage_docCD_vcs_agents88_73700.vxhd" title="Cluster Server 4.1 Enterprise Agent for EMC SRDF Installation and Configuration Guide title page ">Cluster Server 4.1 Enterprise Agent for EMC SRDF Installation and Configuration Guide</a>
  &nbsp;&nbsp;
  </font>
  </td>
 </tr>


</table>
</div>

<div align="left">
 <table border="0" width="100%" bgcolor="#FFFFFF">
  <tr>
   <td width="100%" valign="top" bgcolor="#FFFFFF">

<!-- Start Main Topic Content -->
        <a name="startcontent"></a>
	
<h1><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="+1" color="#000000"><a name="121109"></a>Configuration Concepts</font>
</h1>
 
  
  
  

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120913"></a>This section describes some failure scenarios and provides guidelines on how to configure the agent.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="121589"></a>Individual Component Failure</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="126550"></a>In a replicated data cluster, you can prevent unnecessary SRDF failover or failback by configuring hosts attached to an array as part of the same system zone. VCS attempts to fail over applications within the same system zone before failing them over across system zones.</font>
</p>

<a name="no_figure_caption0"> </a>  <!-- Start Image Code -->
  <!--
   CAUTION: If graphics output is truncated, change the paper size setting 
   of your WebWorks Rasterizer printer driver from "A4" to "Letter" and regenerate your project.
  -->

<p>
<a href="03configa5_t.htm">
<img src="images/03configa5.gif" width="110" height="43" border="0" ></a>
</p>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1"><em>Click the thumbnail above to view full-sized image. </font></em></p>
   <!-- End Image Code -->



<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="121591"></a>In this sample, hosta and hostb are in one system zone, and hostc and hostd are in another system zone. The SystemZones attribute enables you to create these zones. You can modify the SystemZones attribute using the following command:</font>
</p>

<br>
<font face="Courier New, Courier, sansserif, sans serif" size="2"><a name="120924"></a>#&nbsp;<font face="Courier New, Courier, sansserif, sans serif" size="2"><b>hagrp&nbsp;-modify&nbsp;</b></font><em>grpname</em><font face="Courier New, Courier, sansserif, sans serif" size="2"><b>&nbsp;SystemZones&nbsp;hosta&nbsp;0&nbsp;hostb&nbsp;0&nbsp;hostc&nbsp;1&nbsp;hostd&nbsp;1</b></font>
</font>
<blockquote>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="121950"></a>The variable <em>grpname</em> represents the service group in the cluster.</font>
</blockquote>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120925"></a>This command creates two system zones: zone 0 with hosta and hostb, zone 1 with hostc and hostd.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="129164"></a>Global clusters do not require system zones because failover occurs on a remote cluster if all local targets have been exhausted.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120927"></a>	While running on R2 devices, SRDF does not synchronize data back to the R1 automatically. You must update out-of-synch tracks manually. Monitor the number of out-of-synch tracks by viewing the ResourceInfo attribute of an online SRDF resource. If the value is too high, update tracks to the R1 using the update action, which is defined as a supported action in the SRDF resource type.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="120929"></a>All Host or All Application Failure</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120930"></a>If all hosts on the R1 side are disabled or if the application cannot start successfully on any R1 hosts, but both arrays are operational, the service group fails over. </font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="128933"></a>In replicated data cluster environments, the failover can be automatic, whereas in global cluster environments, failover requires user confirmation by default. In both environments, multiple device groups may fail over in parallel. VCS serializes <font face="Courier New, Courier, sansserif, sans serif" size="2">symrdf</font> commands to ensure that SRDF does not lock out a command while another command is running. </font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="129230"></a>Set the OnlineTimeout and RestartLimit attributes for the SRDF resource to make sure that its entry points do not time out, or that they are automatically restarted if they time out. </font>
</p>

</ol>
<p><img src="images/stepintro.gif" height="16" width="16">
<font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="2" color="#000000"><a name="129233"></a>&nbsp;&nbsp;<b>To set the OnlineTimeout attribute</b></font>
</p>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="129234"></a>Use the following formula to calculate an appropriate value for the OnlineTimeout attribute:</font>
</p>

<a name="no_figure_caption1"> </a>  <!-- Start Image Code -->
  <!--
   CAUTION: If graphics output is truncated, change the paper size setting 
   of your WebWorks Rasterizer printer driver from "A4" to "Letter" and regenerate your project.
  -->

<p>
<a href="03configa2_t.htm">
<img src="images/03configa2.gif" width="110" height="14" border="0" ></a>
</p>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1"><em>Click the thumbnail above to view full-sized image. </font></em></p>
   <!-- End Image Code -->



<p>
<ul type="disc"><li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120946"></a>ndevices represents the number of devices in a device group.</font>
</li>
<li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="129272"></a>dfailovertime represents the value of the DevFOTime attribute.</font>
</li>
<li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="129289"></a>ndevicegroups represents the total number of device groups managed by VCS that might fail over simultaneously. </font>
</li>
<li><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="129316"></a>The additional epsilon is for the command instantiation overhead.</font>
</li>
</ul>
</p>
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="129346"></a>If the resulting value seems excessive, divide it by two for every increment in the value of the RestartLimit. However, the OnlineTimeout must be at least the time taken for the largest device group to fail over, otherwise the group will never be able to complete its operation, regardless of the value of RestartLimit.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="129348"></a>Run the perl script <font face="Courier New, Courier, sansserif, sans serif" size="2">/opt/VRTSvcs/bin/SRDF/sigma</font> to get recommendations for VCS attribute values.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="123257"></a>Run the script on a node where VCS is running and has the SRDF agent configured. Note that the sigma calculator adds 10 seconds to the value for each device group to compensate for the overhead of launching an symrdf command. Specify another value to the sigma script if you feel the instantiation takes shorter or longer.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120948"></a>The script assumes that all devices in the Symmetrix array are managed by VCS. Other operations outside of VCS that hold the array lock might delay the online operation unexpectedly.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="120950"></a>Total Site Disaster</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120951"></a>In a total site failure, all hosts and the Symmetrix array are completely disabled.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120952"></a>In a replicated data cluster, VCS detects site failure and total host failure by the loss of all LLT heartbeats. </font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120953"></a>In a global cluster, VCS detects site failure by the loss of both the ICMP and Symm heartbeats. In order not to confuse a site failure with an all-host failure, the AYARetryLimit for the Symm heartbeat must be shorter than the ICMP retry limit, so that the failure of the Symmetrix array is detected first.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120954"></a>A total disaster renders the devices on the surviving array in the <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">PARTITIONED</font> state. If the AutoTakeover attribute is set to its default value of 1, the online entry point runs the <font face="Courier New, Courier, sansserif, sans serif" size="2">symrdf_rw</font> command. If the attribute is set to 0, no takeover occurs and the online entry point times out and faults.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120955"></a>The online entry point detects whether any synchronization was in progress when the source array was lost. Since synchronization renders the target SRDF devices inconsistent until the synchronization completes, write-enabling the devices would be futile since the data stored on them is unusable. In this case, the agent does not enable the devices and instead times out and faults. In such a scenario, you must restore consistent data from a BCV or tape backup.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="120956"></a>Replication Link Failure</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120957"></a>SRDF detects link failures, monitors changed tracks on devices, and resynchronizes R2 devices if the R1 was active at the time of the link failure. </font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120958"></a>If the two arrays are healthy and the link fails and is restored, and if a failover is initiated while one or more devices are in the <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">SYNCINPROG</font> state, the SRDF agent waits for the synchronization to complete before running the symrdf failover command. If the agent times out before the synchronization completes, the resource faults.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120959"></a>If a failover is initiated due to a disaster at the R1 site, and if a synchronization was in progress, the R2 devices are rendered inconsistent and unusable. In this case, even if the AutoTakeover attribute of the agent is set to 1, the agent does not enable read-write access to the devices and instead it faults. You must restore consistent data to these devices, either from BCV or from a tape backup, and then enable read-write access to the devices manually before they can be used.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120960"></a>If the AutoTakeover attribute is set to 0, the agent does not attempt a symrdf rw_enable, but it times out and faults. If you write-enable the devices manually, the agent can come online after it is cleared.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="120962"></a>Split-brain</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120963"></a>Split-brain occurs when all heartbeat links between the R1 and R2 hosts are cut and each side mistakenly thinks the other side is down. To minimize the effects of split-brain, it is best if the cluster heartbeat links pass through similar physical infrastructure as the replication links so that if one breaks, so does the other.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120965"></a>In a replicated data cluster, VCS attempts to start the application assuming a total disaster because the R1 hosts and array are unreachable. Once the heartbeats are restored, VCS stops the applications on one side and restarts the VCS engine (<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">HAD</font>) to eliminate concurrency violation of the same group being online at two places simultaneously. You must resynchronize the volumes manually using the symrdf merge or symrdf restore commands. </font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120966"></a>In a global cluster, you can confirm the failure before failing over the service groups. You can check with the site administrator to identify the cause of the failure. If you do mistakenly fail over, the situation is similar to the replicated data cluster case; however, when the heartbeat is restored, VCS does not stop <font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">HAD </font>at either site. VCS forces you to choose which group to take offline. You must resynchronize data manually.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="120967"></a>If it is physically impossible to place the heartbeats alongside the replication links, there is a possibility that the cluster heartbeats are disabled, but the replication link is not. A failover transitions the original R2 volumes to R1 volumes and vice-versa. In this case, the application faults because its underlying volumes become write-disabled. VCS tries to fail the application over to another host, causing the same consequence in the reverse direction. This phenomenon continues until the group comes online on the final node. This situation can be avoided by setting up your infrastructure such that the loss of heartbeat links also means the loss of replication links.</font>
</p>




<h2><font face="Arial, Helvetica, Geneva, sansserif, sans serif, Verdana" size="3" color="#000000"><a name="130650"></a>Dynamic Swap</font>
</h2>
 
<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="130652"></a>The agent supports the SRDF dynamic swap capability. If all devices are configured for dynamic swap, when a service group fails over between arrays, the agent performs a swap if both arrays are healthy. If one array is down, a unilateral read-write enable occurs. The agent fails over device groups that are not configured for dynamic swap using the <font face="Courier New, Courier, sansserif, sans serif" size="2">symrdf</font> failover command, which read-write enables the R2.</font>
</p>

<p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="130664"></a>The agent checks the following criteria before determining if a swap will occur:</font>
</p>

<ul><p><img src="images/check.gif" alt="Checkmark">&nbsp;
<a name="130667"></a><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2">All devices in the device group are configured as dynamic devices.</font>
</p><p><img src="images/check.gif" alt="Checkmark">&nbsp;
<a name="130822"></a><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2">Dynamic RDF is configured on the local Symmetrix array. </font>
</p><p><img src="images/check.gif" alt="Checkmark">&nbsp;
<a name="130845"></a><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2">The SYMCLI version is 5.4 or above.</font>
</p><p><img src="images/check.gif" alt="Checkmark">&nbsp;
<a name="130886"></a><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2">The microcode is level 5567 or above.</font>
</p></ul><p>
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="2"><a name="130673"></a>Dynamic swap does not affect the ability to perform fire drills.</font>
</p>


<!-- End Main Topic Content -->

   </td>
  </tr>
 </table>
</div>

<div>
<table border="0" cellspacing="0" cellpadding="0" width="100%">
 <tr>
  <td height="18" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#FFFFFF">
  </td>
 </tr>
  <td align="left"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&nbsp;&#94;&nbsp;<a href="03config_vcs_agents106_121109.html" title="Configuration Concepts">Return to Top</a></font>

  </td>
  <td align="right"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">&lt;&nbsp;<a href="03config_vcs_agents105_127578.html" title="Previous - Cluster Heartbeats">Previous</a>&nbsp;&nbsp;|&nbsp;&nbsp;<a href="03config_vcs_agents107_132329.html" title="Next - Configuring the Agent Using the Wizard">Next</a>
&nbsp;&gt;&nbsp;&nbsp;</font>
  </td>
 <tr>
   <td height="9" border="0" cellspacing="0" cellpadding="0" colspan="2" bgcolor="#FFFFFF">
   </td>
 </tr>

<!-- Footer Breadcrumbs -->
<tr>
<td bgcolor="#FFFFFF" colspan="2" height="15" align="right">
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Product:  
<a href="vcs_agents_splashpage_docCD_vcs_agents1_103036.vxhd" title="Cluster Server Enterprise Agents Guides title page ">Cluster Server Enterprise Agents Guides</a>&nbsp;&nbsp;</font></td>
</tr>

<tr>
<td bgcolor="#FFFFFF" colspan="2" height="15" align="right">
<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="1">Manual:  
<a href="vcshp_srdf_titlepage_docCD_vcs_agents88_73700.vxhd" title="Cluster Server 4.1 Enterprise Agent for EMC SRDF Installation and Configuration Guide title page ">Cluster Server 4.1 Enterprise Agent for EMC SRDF Installation and Configuration Guide</a>&nbsp;&nbsp;</font></td>
</tr>


 <tr>
  <td colspan="2" align="left"><font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" color="#000000" align="right" size="-2">
<em>VERITAS Software Corporation
</em></font><br>

<font face="Verdana, Arial, Helvetica, Geneva, sansserif, sans serif" size="-2">www.veritas.com</font><br>
  </td>
 </tr>
</table>
</div>

</BODY>
  <!--
    *************************************************************************************
    * Copyright (c) 2005 VERITAS Software Corporation. All rights reserved. *

    VERITAS, VERITAS SOFTWARE, the VERITAS Logo, and VERITAS Software Documentation 
    are trademarks or registered trademarks of VERITAS Software Corporation 
    or its affiliates in the U.S. and other countries. Other names may be 
    trademarks of their respective owners. 

    Apache Jakarta Lucene:
    Apache Software License
    Version 1.1

    Copyright (c) 2000 The Apache Software Foundation. All rights reserved.

    Redistribution and use in source and binary forms, with or without modification, 
    are permitted provided that the following conditions are met:

    1. Redistributions of source code must retain the above copyright notice, this 
    list of conditions and the following disclaimer.

    2. Redistributions in binary form must reproduce the above copyright notice, this 
    list of conditions and the following disclaimer in the documentation and/or other 
    materials provided with the distribution.

    3. The end-user documentation included with the redistribution, if any, must include 
    the following acknowledgment: "This product includes software developed by the Apache 
    Software Foundation (http://www.apache.org/)." Alternately, this acknowledgment may 
    appear in the software itself, if and wherever such third-party acknowledgments 
    normally appear.

    4. The names "Apache" and "Apache Software Foundation" must not be used to endorse or 
    promote products derived from this software without prior written permission. For 
    written permission, please contact apache@apache.org.

    5. Products derived from this software may not be called "Apache", nor may "Apache" 
    appear in their name, without prior written permission of the Apache Software Foundation.

    THIS SOFTWARE IS PROVIDED "AS IS" AND ANY EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, 
    BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A 
    PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE APACHE SOFTWARE FOUNDATION 
    OR ITS CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, 
    OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS 
    OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND 
    ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING 
    NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF 
    ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

    This software consists of voluntary contributions made by many individuals on behalf 
    of the Apache Software Foundation. For more information on the Apache Software Foundation, 
    please see <http://www.apache.org/>. Portions of this software are based upon public domain 
    software originally written at the National Center for Supercomputing Applications, University 
    of Illinois, Urbana-Champaign.
    
    This product includes WebWindow Software developed by Horst Heistermann.
    *************************************************************************************
  -->
</HTML>
