1 Oracle Clusterware Preinstallation Tasks

1.1.1 Completing a Default Linux Installation

Oracle recommends that you install your Linux operating system with the default software packages (RPMs). This installation includes most of the required packages and helps you limit manual checks of package dependencies. Oracle recommends that you do not customize the RPMs during installation.

After installation, add all of the Legacy Software Development packages. Review system requirements for your distribution to ensure that you have all required kernel packages installed, and complete all other configuration tasks required for your distribution, and for your system configuration.

1.1.2 About the Oracle Validated Configuration RPM

If your Linux distribution is Oracle Enterprise Linux, or Red Hat Enterprise Linux, and you are an Unbreakable Linux customer, then you can complete most preinstallation configuration tasks by using the Oracle Validated Configurations Setup RPM, available from the Unbreakable Linux Network (ULN).

When it is installed, the Oracle Validated Configuration RPM sets and verifies system parameters based on recommendations from the Oracle Validated Configurations program, and installs any additional packages needed for installing Oracle Clusterware and Oracle Database. It also updates sysctl.conf settings, system startup parameters, user limits, and driver parameters to values extensive testing shows will provide better performance.

To become an Oracle Unbreakable Linux customer, contact your sales representative, or purchase a license from the Unbreakable Linux store:

http://oraclestore.oracle.com/linux

To register your server on the Unbreakable Linux Network, or to find out more information, refer to the following URL:

https://linux.oracle.com

1.1.3 Installing the Oracle Validated Configuration RPM

Use the following procedure to subscribe to Oracle Unbreakable Linux channels, and to add the Oracle Software for Enterprise Linux channel that distributes the Oracle Validated Configurations Setup RPM:

1. Complete a default Oracle Enterprise Linux workstation installation, or a default Red Hat Enterprise Linux installation.

2. Register your server with Unbreakable Linux Network (ULN). By default, you are registered for the Enterprise Linux Latest channel for your operating system and hardware.

3. Log in to ULN at the following URL:

   https://linux.oracle.com

4. Click the Systems tab, and in the System Profiles list, select a registered server. The System Details window opens, and displays the subscriptions for the server.

5. From the Available Channels list, select the Oracle Software for Enterprise Linux channel that is appropriate for your installation of Linux (for example: "Oracle Software for Enterprise Linux 4 (x86_64)."

6. Click Subscribe.

7. From a terminal session, as root, enter the following command:

   # up2date --nox --show-channels
   

   You should see output indicating that you have subscribed to the Oracle Software for Enterprise Linux channel. For example:

   el4_i386_latest
   el4_i386_oracle
   

8. Open a terminal session as root, and install the Oracle Validated Configurations Setup RPM with up2date, using the following command:

   # up2date --install oracle-validated
   

9. Repeat steps 1 through 8 on all other servers in your cluster.

   Note:

   Check the Oracle Validated Configuration RPM log file to review system configuration changes:

   /etc/sysconfig/oracle-validated/results/orakernel.log
   

1.4.1 Understanding the Oracle Inventory Group

You must have a group whose members are given access to write to the Oracle Central Inventory (oraInventory). The Central Inventory contains the following:

• A registry of the Oracle home directories (Oracle Clusterware, Oracle Database, and Automatic Storage Management) on the system

• Installation logs and trace files from installations of Oracle software. These files are also copied to the respective Oracle homes for future reference.

Other metadata inventory information regarding Oracle installations are stored in the individual Oracle home inventory directories, and are separate from the Central Inventory.

1.4.2 Understanding the Oracle Inventory Directory

The first time you install Oracle software on a system, Oracle Universal Installer checks to see if you have created an Optimal Flexible Architecture (OFA) compliant path in the format u[01-09]/app, such as /u01/app, and that the user running the installation has permissions to write to that path. If this is true, then Oracle Universal Installer creates the Oracle Inventory directory in the path /u[01-09]/app/oraInventory. For example:

/u01/app/oraInventory

If you have set the environment variable $ORACLE_BASE for the user performing the Oracle Clusterware installation, then OUI creates the Oracle Inventory directory in the path $ORACLE_BASE/../oraInventory. For example, if $ORACLE_BASE is set to /opt/oracle/11, then the Oracle Inventory directory is created in the path /opt/oracle/oraInventory.

If you have created neither an OFA-compliant path nor set $ORACLE_BASE, then the Oracle Inventory directory is placed in the home directory of the user that is performing the installation. For example:

/home/oracle/oraInventory

As this placement can cause permission errors during subsequent installations with multiple Oracle software owners, Oracle recommends that you either create an OFA-compliant installation path, or set an $ORACLE_BASE environment path.

For new installations, Oracle recommends that you allow OUI to create the Central Inventory directory. By default, if you create an Oracle path in compliance with OFA structure, such as /u01/app, that is owned by an Oracle software owner, then the Central Inventory is created in the path u01/app/oraInventory using correct permissions to allow all Oracle installation owners to write to this directory.

1.4.3 Determining If the Oracle Inventory and Oracle Inventory Group Exists

When you install Oracle software on the system for the first time, OUI creates the oraInst.loc file. This file identifies the name of the Oracle Inventory group (typically, oinstall), and the path of the Oracle Central Inventory directory. An oraInst.loc file has contents similar to the following:

inventory_loc=central_inventory_location
inst_group=group

In the preceding example, central_inventory_location is the location of the Oracle Central Inventory, and group is the name of the group that has permissions to write to the central inventory.

If you have an existing Oracle Inventory, then ensure that you use the same Oracle Inventory for all Oracle software installations, and ensure that all Oracle software users you intend to use for installation have permissions to write to this directory.

To determine if you have an Oracle Inventory on your system:

On x86 systems, enter the following command:

# more /etc/oraInst.loc

If the oraInst.loc file exists, then the output from this command is similar to the following:

inventory_loc=/u01/app/oracle/oraInventory
inst_group=oinstall

In the previous output example:

• The inventory_loc group shows the location of the Oracle Inventory

• The inst_group parameter shows the name of the Oracle Inventory group (in this example, oinstall).

1.4.4 Creating the Oracle Inventory Group If an Oracle Inventory Does Not Exist

If the oraInst.loc file does not exist, then create the Oracle Inventory group by entering a command similar to the following:

# /usr/sbin/groupadd -g 501 oinstall

The preceding command creates the group oinstall, with the group ID number 501.

1.4.5 Creating the Oracle Clusterware User

You must create a software owner for Oracle Clusterware in the following circumstances:

• If an Oracle software owner user does not exist; for example, if this is the first installation of Oracle software on the system

• If an Oracle software owner user exists, but you want to use a different operating system user, such as crs, with different group membership, to give separate clusterware and database administrative privileges to those groups in a new Oracle Clusterware and Oracle Database installation.

  In Oracle documentation, a user created to own only Oracle Clusterware software installations is called the crs user. A user created to own either all Oracle installations, or only Oracle database installations, is called the oracle user.

# id oracle

uid=501(oracle) gid=501(oinstall) groups=502(dba),503(oper)

1.4.6 Example of Creating the Oracle Clusterware User and OraInventory Path

The following is an example of how to create the Oracle Clusterware software owner (in this case, crs), and a path compliant with OFA structure with correct permissions for the oraInventory directory. This example also shows how to create separate Oracle Database and Oracle ASM homes with correct ownership and permissions:

# mkdir -p  /u01/app/crs
# chown -R crs:oinstall /u01/app
# mkdir  /u01/app/oracle
# chown oracle:oinstall /u01/app/oracle
# chmod 775 /u01/app/
# mkdir  /u01/app/asm
# chown asm:oinstall /u01/app/asm

At the end of this procedure, you will have the following:

• /u01 owned by root.

• /u01/app owned by crs:oinstall with 775 permissions. This ownership and permissions enables OUI to create the oraInventory directory, in the path /u01/app/oraInventory.

• /u01/app/crs owned by crs:oinstall with 775 permissions. These permissions are required for installation, and are changed during the installation process.

• /u01/app/oracle owned by oracle:oinstall with 775 permissions.

• /u01/app/asm owned by asm:oinstall with 775 permissions.

1.6.1 Network Hardware Requirements

The following is a list of requirements for network configuration:

• Each node must have at least two network adapters or network interface cards (NICs): one for the public network interface, and one for the private network interface (the interconnect).

  If you want to use more than one NIC for the public network or for the private network, then Oracle recommends that you use NIC bonding.

• The public interface names associated with the network adapters for each network must be the same on all nodes, and the private interface names associated with the network adaptors should be the same on all nodes.

  For example: With a two-node cluster, you cannot configure network adapters on node1 with eth0 as the public interface, but on node2 have eth1 as the public interface. Public interface names must be the same, so you must configure eth0 as public on both nodes. You should configure the private interfaces on the same network adapters as well. If eth1 is the private interface for node1, then eth1 should be the private interface for node2.

• For the public network, each network adapter must support TCP/IP.

• For the private network, the interconnect must support the user datagram protocol (UDP) using high-speed network adapters and switches that support TCP/IP (Gigabit Ethernet or better required).

  Note:

  UDP is the default interconnect protocol for Oracle RAC, and TCP is the interconnect protocol for Oracle Clusterware. You must use a switch for the interconnect. Oracle recommends that you use a dedicated switch.

  Oracle does not support token-rings or crossover cables for the interconnect.

• For the private network, the endpoints of all designated interconnect interfaces must be completely reachable on the network. There should be no node that is not connected to every private network interface. You can test whether an interconnect interface is reachable using a ping command.

1.6.2 IP Address Requirements

Before starting the installation, you must have the following IP addresses available for each node:

• An IP address with an associated host name (or network name) registered in the DNS for the public interface. If you do not have an available DNS, then record the host name and IP address in the system hosts file, /etc/hosts.

• One virtual IP (VIP) address with an associated host name registered in a DNS. If you do not have an available DNS, then record the host name and VIP address in the system hosts file, /etc/hosts. Select an address for your VIP that meets the following requirements:

  • The IP address and host name are currently unused (it can be registered in a DNS, but should not be accessible by a ping command)

  • The VIP is on the same subnet as your public interface

• A private IP address with a host name for each private interface

  Oracle recommends that you use private network IP addresses for these interfaces (for example: 10.*.*.* or 192.168.*.*). You can use DNS servers, or the /etc/hosts file, or both to register the private IP address. Note that if you use DNS servers alone, and the public network becomes unreachable due to NIC or cable failure, then the private IP addresses can fail to resolve.

  For the private interconnects, because of Cache Fusion and other traffic between nodes, Oracle strongly recommends using a physically separate, private network. You should ensure that the private IP addresses are reachable only by the cluster member nodes.

  During installation, you are asked to identify the planned use for each network interface that OUI detects on your cluster node. You must identify each interface as a public or private interface, and you must use the same private interfaces for both Oracle Clusterware and Oracle RAC.

  You can bond separate interfaces to a common interface to provide redundancy, in case of a NIC failure, but Oracle recommends that you do not create separate interfaces for Oracle Clusterware and Oracle RAC. If you use more than one NIC for the private interconnect, then Oracle recommends that you use NIC bonding. Note that multiple private interfaces provide load balancing but not failover, unless bonded.

  For example, if you intend to use the interfaces eth2 and eth3 as interconnects, then before installation, you must configure eth2 and eth3 with the private interconnect addresses. If the private interconnect addresses are 10.10.1.1 for eth2 and 10.10.2.1 for eth3, then bond eth2 and eth3 to an interface, such as bond0, using a separate subnet such as 10.10.222.0. During installation, define the Oracle Clusterware private node names on 10.10.222.0, and then define 10.10.222.0 (and only that one) as a private interconnect. This ensures that Oracle Clusterware and Oracle RAC are using the same network.

  After installation, if you modify interconnects on Oracle RAC with the CLUSTER_INTERCONNECTS initialization parameter, then you must change it to a private IP address, on a subnet that is not used with a public IP address, nor marked as a public subnet by oifcfg. Oracle does not support changing the interconnect to an interface using a subnet that you have designated as a public subnet.

  See Also:

  Oracle Clusterware Administration and Deployment Guide for further information about setting up and using bonded multiple interfaces

  You should not use a firewall on the network with the private network IP addresses, as this can block interconnect traffic.

Before installation, check that the default gateway can be accessed by a ping command. To find the default gateway, use the route command, as described in your operating system's help utility. After installation, configure clients to use either the VIP address, or the host name associated with the VIP. If a node fails, then the node's virtual IP address fails over to another node.

For example, with a two node cluster where each node has one public and one private interface, you might have the configuration shown in the following table for your network interfaces, where the hosts file is /etc/hosts:

To enable VIP failover, the configuration shown in the preceding table defines the public and VIP addresses of both nodes on the same subnet, 143.46.43.

1.6.3 Node Time Requirements

Before starting the installation, ensure that each member node of the cluster is set as closely as possible to the same date and time. Oracle strongly recommends using the Network Time Protocol feature of most operating systems for this purpose, with all nodes using the same reference Network Time Protocol server.

1.6.4 Network Configuration Options

The precise configuration you choose for your network depends on the size and use of the cluster you want to configure, and the level of availability you require.

If certified Network-attached Storage (NAS) is used for Oracle RAC and this storage is connected through Ethernet-based networks, then you must have a third network interface for I/O. Failing to provide three separate interfaces in this case can cause performance and stability problems under load.

For high capacity clusters with a small number of multiprocessor servers, to ensure high availability, you may want to configure redundant network interfaces to prevent a NIC failure from reducing significantly the overall cluster capacity. If you are using network storage, and want to provide redundant network interfaces, then Oracle recommends that you provide six network interfaces: two for the public network interface, two for the private network interface, and two for the network storage.

1.6.5 Configuring the Network Requirements

To verify that each node meets the requirements, follow these steps:

1. If necessary, install the network adapters for the public and private networks and configure them with either public or private IP addresses.

2. If you are using a domain name server (DNS), then for each node, register the host names and IP addresses for the public network interfaces in the DNS.

3. Even if you are using a DNS, Oracle recommends that you add lines to the /etc/hosts file on each node, specifying the private IP addresses and associated private host names. Oracle also recommends that you add public and virtual IP addresses. Configure the /etc/host file so that it is similar to as shown in the following example, with private interface eth1, and private hosts nodeint1 and nodeint2:, where xxx represents parts of a valid IP address.

   #eth0 - PUBLIC
   xxx.xxx.100.45   node1.example.com    node1
   xxx.xxx.100.46   node2.example.com    node2
    
   #eth1 - PRIVATE
   10.0.0.1      nodeint1.example.com    nodeint1
   10.0.0.2      nodeint2.example.com    nodeint2
    
   #VIPs
   xxx.xxx.100.47   pmvip1.example.com    nodevip1
   xxx.xxx.100.48   pmvip2.example.com    nodevip2
   

4. To check network configuration, on each node, enter the following commands:

   # hostname
   # /sbin/ifconfig
   

   Ensure that each server is properly identified, and that the interface name and IP address for all network adapters that you want to specify as public or private network interfaces are properly configured. In addition, use the ping command to ensure that each node can obtain a response for the public and private IP addresses from each other node in the cluster.

   Note:

   When you install Oracle Clusterware and Oracle RAC, you will require the public, private and virtual IP addresses. Make a note of the addresses you configured in the /etc/hosts file or DNS.

5. To prevent public network failures with Oracle RAC databases using NAS devices or NFS mounts, enter the following command as root to enable the Name Service Cache Daemon (nscd):

   # /sbin/service  nscd start
   

1.7.1 Software Requirements List for x86 (32-bit) Linux Platforms

For installations only of Oracle Clusterware, ensure that you have the kernel versions and packages listed in Table 1-1 and Table 1-2.

If you intend to install Oracle Database or Oracle RAC in addition to Oracle Clusterware, then ensure that you have the kernel packages listed in Table 1-1 and Table 1-3, and check Table 1-4 to determine if you need to install additional packages for the features you plan to use.

binutils-2.15.92.0.2-18
elfutils-libelf-0.97-5
elfutils-libelf-devel-0.97.5
glibc-2.3.9.4-2.19
glibc-common-2.3.9.4-2.19
glibc-devel-2.3.9.4-2.19
gcc-3.4.5-2
gcc-c++-3.4.5-2
libaio-devel-0.3.105-2
libaio-0.3.105-2
libgcc-3.4.5
libstdc++-3.4.5-2
libstdc++-devel-3.4.5-2
make-3.80-5

binutils-2.17.50.0.6-2.el5
elfutils-libelf-0.125
elfutils-libelf-devel-0.125
glibc-2.5-12
glibc-common-2.5-12
glibc-devel-2.5-12
gcc-4.1.1-52
gcc-c++-4.1.1-52
libaio-0.3.106
libaio-devel-0.3.106 
libgcc-4.1.1-52
libstdc++-4.1.1 
libstdc++-devel-4.1.1
make-3.81-1.1

binutils-2.16.91.0.5
glibc-2.4-31.2
glibc-devel-2.4-31.2
gcc-4.1.0
libaio-0.3.104
libaio-devel-0.3.104
libelf-0.8.5
libgcc-4.1.0
libstdc++-4.1.0
libstdc++-devel-4.1.0
make-3.80

binutils-2.15.92.0.2-18
compat-libstdc++-33.2.3-47.3
elfutils-libelf-0.97-5
elfutils-libelf-devel-0.97.5
glibc-2.3.9.4-2.19
glibc-common-2.3.9.4-2.19
glibc-devel-2.3.9.4-2.19
gcc-3.4.5-2
gcc-c++-3.4.5-2
libaio-devel-0.3.105-2
libaio-0.3.105-2
libgcc-3.4.5
libstdc++-3.4.5-2
libstdc++-devel-3.4.5-2
make-3.80-5
sysstat-5.0.5
unixODBC-2.2.11
unixODBC-devel-2.2.11

binutils-2.17.50.0.6-2.el5
compat-libstdc++-33-3.2.3-61
elfutils-libelf-0.97-5
elfutils-libelf-devel-0.125
glibc-2.5-12
glibc-common-2.5-12
glibc-devel-2.5-12
gcc-4.1.1-52
gcc-c++-4.1.1-52
libaio-0.3.106
libaio-devel-0.3.106 
libgcc-4.1.1-52
libstdc++-4.1.1 
libstdc++-devel-3.4.3-22.1
make-3.81-1.1
sysstat-7.0.0
unixODBC-2.2.11
unixODBC-devel-2.2.11

binutils-2.16.91.0.5
compat-libstdc++-5.0.7
glibc-2.4-31.2
glibc-devel-2.4-31.2
gcc-4.1.0
ksh-93r-12.9
libaio-0.3.104
libaio-devel-0.3.104
libelf-0.8.5
libgcc-4.1.0
libstdc++-4.1.0
libstdc++-devel-4.1.0
make-3.80
sysstat-6.0.2
unixODBC-2.2.11
unixODBC-devel-2.2.11

http://www.unixodbc.org

http://oss.oracle.com/projects/ocfs2/

1.9.1 Installing the cvuqdisk Package for Linux

If you are using Red Hat or SUSE Linux, then you must download and install the operating system package cvuqdisk. Without cvuqdisk, CVU is unable to discover shared disks, and you receive the error message "Package cvuqdisk not installed" when you run CVU. Use the cvuqdisk rpm for your hardware (i386).

To install the cvuqdisk RPM, complete the following procedure:

CV_RAW_CHECK_ENABLED=FALSE 

1. Locate the cvuqdisk RPM package, which is in the directory rpm on the installation media. If you have already installed Oracle Clusterware, then it is located in the directory CRS_home/rpm.

2. Copy the cvuqdisk package to each node on the cluster. You should ensure that each node is running the same version of Linux.

3. Log in as root.

4. Using the following command, check to see if you have an existing version of the cvuqdisk package:

   # rpm -qi cvuqdisk
   

   If you have an existing version, then enter the following command to de-install the existing version:

   # rpm -e cvuqdisk
   

5. Set the environment variable CVUQDISK_GRP to point to the group that will own cvuqdisk, typically oinstall. For example:

   # CVUQDISK_GRP=oinstall; export CVUQDISK_GRP
   

6. In the directory where you have saved the cvuqdisk rpm, use the following command to install the cvuqdisk package:

   # rpm -iv cvuqdisk-1.0.1-1.rpm
   

1.10.1 Checking Existing SSH Configuration on the System

To determine if SSH is running, enter the following command:

$ pgrep sshd

If SSH is running, then the response to this command is one or more process ID numbers. In the home directory of the software owner that you want to use for the installation (crs, oracle), use the command ls -al to ensure that the .ssh directory is owned and writable only by the user.

You need either an RSA or a DSA key for the SSH protocol. RSA is used with the SSH 1.5 protocol, while DSA is the default for the SSH 2.0 protocol. With OpenSSH, you can use either RSA or DSA. The instructions that follow are for SSH1. If you have an SSH2 installation, and you cannot use SSH1, then refer to your SSH distribution documentation to configure SSH1 compatibility or to configure SSH2 with DSA.

1.10.2 Configuring SSH on Cluster Member Nodes

To configure SSH, you must first create RSA or DSA keys on each cluster node, and then copy all the keys generated on all cluster node members into an authorized keys file that is identical on each node. Note that the SSH files must be readable only by root and by the software installation user (oracle, crs, asm), as SSH ignores a private key file if it is accessible by others. When this is done, then start the SSH agent to load keys into memory. In the examples that follow, the RSA key is used.

You must configure SSH separately for each Oracle software installation owner that you intend to use for installation.

To configure SSH, complete the following:

1.10.3 Enabling SSH User Equivalency on Cluster Member Nodes

After you have copied the authorized_keys file that contains all keys to each node in the cluster, complete the following procedure, in the order listed. In this example, the Oracle Clusterware software owner is named crs:

1. On the system where you want to run OUI, log in as the crs user.

2. Use the following command syntax, where hostname1, hostname2, and so on, are the public hostnames (alias and fully qualified domain name) of nodes in the cluster to run SSH from the local node to each node, including from the local node to itself, and from each node to each other node:

   [crs@nodename]$ ssh hostname1 date
   [crs@nodename]$ ssh hostname2 date
       .
       .
       .
   

   For example:

   [crs@node1 crs]$ ssh node1 date
   The authenticity of host 'node1 (xxx.xxx.100.101)' can't be established.
   RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added 'node1,xxx.xxx.100.101' (RSA) to the list of
   known hosts.
   Enter passphrase for key '/home/crs/.ssh/id_rsa':
   Mon Dec 4 11:08:13 PST 2006
   [crs@node1 crs]$ ssh node1.example.com date
   The authenticity of host 'node1.example.com (xxx.xxx.100.101)' can't be
   established.
   RSA key fingerprint is 7z:60:60:zz:48:48:z1:a0:f7:4e.
   Are you sure you want to continue connecting (yes/no)? yes
   Warning: Permanently added 'node1.example.com,xxx.xxx.100.101' (RSA) to the
   list of known hosts.
   Enter passphrase for key '/home/crs/.ssh/id_rsa':
   Mon Dec 4 11:08:13 PST 2006
   [crs@node1 crs]$ ssh node2 date
   Enter passphrase for key '/home/crs/.ssh/id_rsa':
   Mon Dec 4 11:08:35 PST 2006
   .
   .
   .
   

   At the end of this process, the public hostname for each member node should be registered in the known_hosts file for all other cluster member nodes.

   If you are using a remote client to connect to the local node, and you see a message similar to "Warning: No xauth data; using fake authentication data for X11 forwarding," then this means that your authorized keys file is configured correctly, but your ssh configuration has X11 forwarding enabled. To correct this, proceed to "Setting Display and X11 Forwarding Configuration".

3. Repeat step 2 on each cluster node member.

4. On each node, enter the following commands to start the SSH agent, and to load the SSH keys into memory:

   $ exec /usr/bin/ssh-agent $SHELL
   $ /usr/bin/ssh-add
   

   At the prompt, enter the pass phrase for each key that you generated.

   For example:

   [crs@node1 .ssh]$ exec /usr/bin/ssh-agent $SHELL
   [crs@node1 .ssh]$ ssh-add
   Enter passphrase for /home/crs/.ssh/id_rsa
   Identity added: /home/crs/.ssh/id_rsa (/home/crs/.ssh/id_rsa)
   

   These commands start the ssh-agent on the node, and load the RSA keys into memory so that you are not prompted to use pass phrases when issuing SSH commands

If you have configured SSH correctly, then you can now use the ssh or scp commands without being prompted for a password or a pass phrase. For example:

[crs@node1 ~]$ ssh node2 date
Mon Feb 26 23:34:42 UTC 2007
[crs@node1 ~]$ ssh node1 date
Mon Feb 26 23:34:48 UTC 2007
[crs@node1 ~]$ ssh node2

If any node prompts for a password or pass phrase, then verify that the ~/.ssh/authorized_keys file on that node contains the correct public keys, and that you have created an Oracle software owner with identical group membership and IDs.

1.10.4 Setting Display and X11 Forwarding Configuration

• If you are on a remote terminal, and the local node has only one visual (which is typical), then use the following syntax to set the DISPLAY environment variable:

  Bourne, Korn, and Bash shells

  $ export DISPLAY=hostname:0
  

  C shell:

  $ setenv DISPLAY=hostname:0
  

  For example, if you are using the Bash shell, and if your hostname is node1, then enter the following command:

  $ export DISPLAY=node1:0
  

• To ensure that X11 forwarding will not cause the installation to fail, create a user-level SSH client configuration file for the Oracle software owner user, as follows:

  1. Using any text editor, edit or create the software installation owner's ~/.ssh/config file.

  2. Make sure that the ForwardX11 attribute is set to no. For example:

     Host *
           ForwardX11 no
     

1.10.5 Preventing Oracle Clusterware Installation Errors Caused by stty Commands

During an Oracle Clusterware installation, OUI uses SSH to run commands and copy files to the other nodes. During the installation, hidden files on the system (for example, .bashrc or .cshrc) will cause makefile and other installation errors if they contain stty commands.

To avoid this problem, you must modify these files in each Oracle installation owner user home directory to suppress all output on STDERR, as in the following examples:

• Bourne, Bash, or Korn shell:

  if [ -t 0 ]; then
     stty intr ^C
  fi
  

• C shell:

  test -t 0
  if ($status == 0) then
     stty intr ^C
  endif
  

  Note:

  When SSH is not available, the Installer uses the rsh and rcp commands instead of ssh and scp.

  If there are hidden files that contain stty commands that are loaded by the remote shell, then OUI indicates an error and stops the installation.

1.11.1 Environment Requirements for Oracle Clusterware Software Owner

You must make the following changes to configure the Oracle Clusterware software owner environment:

• Set the installation software owner user (crs, oracle, asm) default file mode creation mask (umask) to 022 in the shell startup file. Setting the mask to 022 ensures that the user performing the software installation creates files with 644 permissions.

• Set ulimit settings for file descriptors and processes for the installation software owner (crs, oracle)

• Set the software owner's environment variable DISPLAY environment variables in preparation for the Oracle Clusterware installation

1.11.2 Procedure for Configuring Oracle Software Owner Environments

To set the Oracle software owners' environments, follow these steps, for each software owner (crs, oracle, asm):

1. Start a new terminal session; for example, start an X terminal (xterm).

2. Enter the following command to ensure that X Window applications can display on this system:

   $ xhost + hostname
   

   The hostname is the name of the local host.

3. If you are not already logged in to the system where you want to install the software, then log in to that system as the software owner user.

4. If you are not logged in as the user, then switch to the software owner user you are configuring. For example, with the crs user:

   $ su - crs
   

5. To determine the default shell for the user, enter the following command:

   $ echo $SHELL
   

6. Open the user's shell startup file in any text editor:

   Note:

   On Red Hat Linux, .bash_profile is the user startup file for the Bash shell.

   • Bourne shell (sh), Bash shell (bash) or Korn shell (ksh):

     $ vi .bash_profile
     

   • C shell (csh or tcsh):

     % vi .login
     

7. Enter or edit the following line, specifying a value of 022 for the default file mode creation mask:

   umask 022
   

8. If the ORACLE_SID, ORACLE_HOME, or ORACLE_BASE environment variable is set in the file, then remove the appropriate lines from the file.

9. Save the file, and exit from the text editor.

10. To run the shell startup script, enter one of the following commands:

    • Bash shell on Red Hat Enterprise Linux:

      $ . ./.bash_profile
      

    • Bourne, Bash, or Korn shell:

      $ . ./.profile
      

    • C shell:

      % source ./.login
      

11. If you are not installing the software on the local system, then enter a command similar to the following to direct X applications to display on the local system:

    • Bourne, Bash, or Korn shell:

      $ DISPLAY=local_host:0.0 ; export DISPLAY
      

    • C shell:

      % setenv DISPLAY local_host:0.0
      

    In this example, local_host is the host name or IP address of the system that you want to use to display OUI (your workstation or PC).

12. If you determined that the /tmp directory has less than 400 MB of free disk space, then identify a file system with at least 400 MB of free space and set the TEMP and TMPDIR environment variables to specify a temporary directory on this file system:

    Note:

    You cannot use a shared file system as the location of the temporary file directory (typically /tmp) for Oracle RAC installation. If you place /tmp on a shared file system, then the installation fails.

    1. Use the df -h command to identify a suitable file system with sufficient free space.

    2. If necessary, enter commands similar to the following to create a temporary directory on the file system that you identified, and set the appropriate permissions on the directory:

       $ su - root
       # mkdir /mount_point/tmp
       # chmod 775 /mount_point/tmp
       # exit
       

    3. Enter commands similar to the following to set the TEMP and TMPDIR environment variables:

       • Bourne, Bash, or Korn shell:

         $ TEMP=/mount_point/tmp
         $ TMPDIR=/mount_point/tmp
         $ export TEMP TMPDIR
         

       • C shell:

         % setenv TEMP /mount_point/tmp
         % setenv TMPDIR /mount_point/tmp
         

1.11.3 Setting Shell Limits for the Oracle Software Installation Users

To improve the performance of the software on Linux systems, you must increase the following shell limits for the Oracle software owner users (crs, oracle, asm):

To increase the shell limits:

1. Add the following lines to the /etc/security/limits.conf file (the following example shows the oracle user as the software owner):

   oracle               soft    nproc   2047
   oracle               hard    nproc   16384
   oracle               soft    nofile  1024
   oracle               hard    nofile  65536
   

2. Add or edit the following line in the /etc/pam.d/login file, if it does not already exist:

   session    required     pam_limits.so
   

3. Depending on your shell environment, make the following changes to the default shell startup file (note that these examples show the user oracle):

   • For the Bourne, Bash, or Korn shell, add the following lines to the /etc/profile file (or the file on SUSE systems)/etc/profile.local

     if [ $USER = "oracle" ]; then
             if [ $SHELL = "/bin/ksh" ]; then
                   ulimit -u 16384
                   ulimit -n 65536
             else
                   ulimit -u 16384 -n 65536
             fi
             umask 022
     fi
     

   • For the C shell (csh or tcsh), add the following lines to the /etc/csh.login file (or the file on SUSE systems)/etc/csh.login.local:

     if ( $USER == "oracle" ) then
             limit maxproc 16384
             limit descriptors 65536
     endif
     

4. Repeat this procedure on all other nodes in the cluster, and for all Oracle software owners that you intend to use to install Oracle software (crs, asm, oracle).

1.13.1 Entering Cluster Verification Utility Commands

CVU is provided with two scripts: runcluvfy.sh, which is designed to be used before installation, and cluvfy, which is in the path CRS_home/bin. The script runcluvfy.sh contains temporary variable definitions which enable it to be run before installing Oracle Clusterware or Oracle Database. After you install Oracle Clusterware, use the command cluvfy to check prerequisites and perform other system readiness checks.

Before Oracle software is installed, to enter a CVU command, change directories and start runcluvfy.sh using the following syntax:

cd /mountpoint
./runcluvfy.sh options

In the preceding example, the variable mountpoint represents the mountpoint path for the installation media and the variable options represents the CVU command options that you select. For example:

$ cd /mnt/dvdrom
$ ./runcluvfy.sh comp nodereach -n node1,node2 -verbose

By default, when you enter a CVU command, CVU provides a summary of the test. During preinstallation, Oracle recommends that you obtain detailed output by using the -verbose argument with the CVU command. The -verbose argument produces detailed output of individual checks. Where applicable, it shows results for each node in a tabular layout.

1.13.2 Using CVU to Determine if Installation Prerequisites are Complete

You can use CVU to determine which system prerequisites for installation are already completed. Use this option if you are installing Oracle 11g release 1 (11.1) on a system with a pre-existing Oracle software installation. In using this option, note the following:

• You must complete the prerequisites for using CVU, notably configuring SSH between all nodes in the cluster, before you can complete a clusterwide status check.

• CVU can assist you by finding preinstallation steps that need to be completed, but it cannot perform preinstallation tasks

Use the following syntax to determine what preinstallation steps are completed, and what preinstallation steps must be performed

$ ./runcluvfy.sh stage -pre crsinst -n node_list 

In the preceding syntax example, replace the variable node_list with the names of the nodes in your cluster, separated by commas.

For example, for a cluster with mountpoint /mnt/dvdrom/, and with nodes node1, node2, and node3, enter the following command:

$ cd /mnt/dvdrom/
$ ./runcluvfy.sh stage -pre crsinst -n node1,node2,node3

Review the CVU report, and proceed to the sections of the preinstallation chapter to complete additional steps as needed.

1.13.3 Using the Cluster Verification Utility Help

The cluvfy commands have context-sensitive help that shows correct syntax usage based on the command line arguments that you enter.

If you enter an invalid CVU command, then CVU shows the correct usage for that command. For example, if you type runcluvfy.sh stage -pre dbinst, then CVU shows the correct syntax for the database preinstallation checks that CVU performs with the dbinst stage option. The following is a list of context help commands.

• cluvfy -help —CVU displays detailed CVU command information.

• cluvfy comp -list—CVU displays a list of components that can be checked, and brief descriptions of how each component is checked.

• cluvfy comp -help—CVU displays detailed syntax for each of the valid component checks.

• cluvfy stage -list—CVU displays a list of valid stages.

• cluvfy stage -help—CVU displays detailed syntax for each of the valid stage checks.

1.13.4 Using Cluster Verification Utility with Oracle Database 10g Release 1 or 2

You can use CVU on the Oracle Database 11g release 1 (11.1) media to check system requirements for Oracle Database 10g release 1 (10.1) and later installations. To use CVU to check 10. 2 installations, append the command flag -r 10gR2 to the standard CVU system check commands.

For example, to perform a verification check for a Cluster Ready Services 10. 2 installation, on a system where the media mountpoint is /mnt/dvdrom, and the cluster nodes are node1, node2, and node3, enter the following command:

$ cd /mnt/dvdrom
$ ./runcluvfy.sh stage -pre crsinst -n node1,node2,node3 -r 10gR2

1.13.5 Verbose Mode and "Unknown" Output

If you run CVU using the -verbose argument, and a CVU command responds with UNKNOWN for a particular node, then this is because the CVU cannot determine whether a check passed or failed. The following is a list of possible causes for an "Unknown" response:

• The node is down

• Executables required by CVU are missing in the /bin directory in the Oracle Clusterware home or Oracle home directory

• The user account starting CVU does not have privileges to run common operating system executables on the node

• The node is missing an operating system patch, or a required package

• The node has exceeded the maximum number of processes or maximum number of open files, or there is a problem with IPC segments, such as shared memory or semaphores