| Oracle® Database 2 Day + Real Application Clusters Guide 11g Release 1 (11.1) Part Number B28252-01 |
|
|
View PDF |
| Oracle® Database 2 Day + Real Application Clusters Guide 11g Release 1 (11.1) Part Number B28252-01 |
|
|
View PDF |
This chapter contains the information that your system administrator and network administrator need to help you, as the DBA, configure the two nodes in your cluster. This chapter assumes a basic understanding of the Red Hat Linux operating system. In some cases, you may need to refer to details in Oracle Real Application Clusters Installation Guide for Linux and UNIX. In addition, you must have root privileges to perform the tasks in this chapter.
This chapter includes the following sections:
Before you begin your installation, you should check to make sure that your system meets the requirements for Oracle Real Application Clusters (Oracle RAC). The requirements can be grouped into the following three categories:
Each node that you want to make part of your Oracle Clusterware, or Oracle Clusterware and Oracle RAC installation, must satisfy the minimum hardware requirements of the software. These hardware requirements can be categorized as follows:
Physical memory (at least 1 gigabyte (GB) of RAM)
Swap space (at least 2 GB of available swap space)
Temporary space (at least 400 megabytes (MB))
Processor type (CPU) that is certified with the version of the Oracle software being installed
Note:
When you install the Oracle Database software, Oracle Universal Installer (OUI) automatically performs hardware prerequisite checks and notifies you if they are not met.You will need at least 1.5 GB of available disk space for the Oracle Database home directory and 1.5 GB of available disk space for the Automatic Storage Management (ASM) home directory. You will also need 120 MB of disk available space for the Oracle Clusterware software installation. For best performance and protection, you should have multiple disks, each using a different disk controller.
An Oracle RAC database is a shared everything database. All datafiles, control files, redo log files, and the server parameter file (SPFILE) used by the Oracle RAC database must reside on shared storage that is accessible by all the Oracle RAC database instances. The Oracle RAC installation that is described in this guide uses ASM for the shared storage of the database files.
Oracle Clusterware achieves superior scalability and high availability by using the following components:
Voting disk–Manages cluster membership and arbitrates cluster ownership between the nodes in case of network failures. The voting disk is a file that resides on shared storage. For high availability, Oracle recommends that you have more than one voting disk, and that you have an odd number of voting disks. If you define a single voting disk, then use mirroring at the file system level for redundancy.
Oracle Cluster Registry (OCR)–Maintains cluster configuration information as well as configuration information about any cluster database within the cluster. The OCR contains information such as which database instances run on which nodes and which services run on which databases. The OCR also stores information about processes that Oracle Clusterware controls. The OCR resides on shared storage that is accessible by all the nodes in your cluster. Oracle Clusterware can multiplex, or maintain multiple copies of, the OCR and Oracle recommends that you use this feature to ensure high availability.
Note:
Both the voting disks and the OCR must reside on shared devices that you configure before you install Oracle Clusterware and Oracle RAC.These Oracle Clusterware components require the following additional disk space:
Two Oracle Clusterware Registry files, 280 MB each, or 560 MB total disk space
Three voting disk files, 280 MB each, or 840 MB total disk space
For voting disk file placement, ensure that each voting disk is configured so that it does not share any hardware device or disk, or other single point of failure. See "Configuring Block Devices for Oracle Clusterware Files" for more information about configuring Oracle Clusterware files.
See Also:
Your platform-specific Oracle Clusterware installation guide
An Oracle RAC cluster comprises two or more nodes that are linked by a private interconnect. The interconnect serves as the communication path between nodes in the cluster. Each cluster database instance uses the interconnect for messaging to synchronize the use of shared resources by each instance. Oracle RAC also uses the interconnect to transmit data blocks that are shared between the instances.
Oracle Clusterware requires that you connect the nodes in the cluster to a private network by way of a private interconnect. The private interconnect is a separate network that you configure between cluster nodes. The interconnect used by Oracle RAC is the same interconnect that Oracle Clusterware uses. This interconnect should be a private interconnect, meaning it is not accessible to nodes that are not members of the cluster.
When you configure the network for Oracle RAC and Oracle Clusterware, each node in the cluster must meet the following requirements:
Each node has at least two network interface cards, or network adapters. One adapter is for the public network and the other adapter is for the private network used by the interconnect. Install additional network adapters on a node if that node meets either of the following conditions:
Does not have at least two network adapters
Has two network interface cards but is using network attached storage (NAS). You should have a separate network adapter for NAS.
Note:
For the most current information about supported network protocols and hardware for Oracle RAC installations, refer to the Certify pages on OracleMetaLink, which is located athttps://metalink.oracle.com
You must have at least three IP addresses available for each node:
An IP address with an associated host name (or network name) for the public interface.
A private IP address with a host name for each private interface.
Note:
Oracle recommends that you use private network IP addresses for the private interfaces (for example: 10.*.*.* or 192.168.*.*).One virtual IP address with an associated network name. Select a virtual IP (VIP) address that meets the following requirements:
The VIP address and associated network name are currently unused.
The VIP is on the same subnet as your public interface.
Public interface names must be the same for all nodes. If the public interface on one node uses the network adapter eth0, then you must configure eth0 as the public interface on all nodes.You should configure the same private interface names for all nodes as well. If eth1 is the private interface name for the first node, then eth1 should be the private interface name for your second node.
For the private network, the end points of all designated interconnect interfaces must be completely reachable on the network. There should be no node that is inaccessible by other nodes in the cluster using the private network.
To determine what interfaces are configured on a node running Red Hat Linux, use the following command as the root user:
# /sbin/ifconfig
You may need to work with your system or network administrator to obtain IP addresses for each node.
Refer to Oracle Clusterware and Oracle Real Application Clusters Installation and Configuration Guide for your platform for information about exact requirements. These requirements can include any of the following:
The operating system version
The kernel version of the operating system
Installed packages, patches, or patch sets
Installed compilers and drivers
Web browser type and version
Additional application software requirements
If you are currently running an operating system version that is not supported by Oracle Database 11g Release 1 (11.1), then you must first upgrade your operating system before installing Oracle Real Application Clusters 11g.
To determine if the operating system requirements for Red Hat Linux have been met:
To determine which distribution and version of Linux is installed, run the following command at the operating system prompt as the root user:
# cat /etc/issue
To determine if the required errata level is installed, use the following procedure as the root user:
# uname -r 2.6.9-42.EL
Like most software, the Linux kernel is updated to fix bugs in the operating system. These kernel updates are referred to as erratum kernels or errata levels.
The output in the previous example shows that the kernel version is 2.6.9, and the errata level (EL) is 22. Review the required errata level for your distribution. If the errata level is below the required minimum errata level, then install the latest kernel update for your operating system. The kernel updates are available from your operating system vendor.
To ensure there are no operating system issues affecting installation, make sure you have installed all the operating system patch updates and packages that are listed in Oracle Clusterware and Oracle Real Application Clusters Installation Guide for your platform. If you are using Red Hat Linux, you can determine if the required packages, or programs that perform specific functions or calculations, are installed by using the following command as the root user:
# rpm -q package_name
The variable package_name is the name of the package you are verifying, such as setarch. If a package is not installed, then install it from your Linux distribution media or download the required package version from your Linux vendor's Web site.
See Also:
Oracle Clusterware and Oracle Real Application Clusters Installation and Configuration Guide for your platform
In this section, you will perform the following tasks:
See Also:
Depending on whether or not this is the first time Oracle software is being installed on this server, you may need to create operating system groups.
The following operating system groups are used when installing Oracle software:
The OSDBA group (typically, dba) for Oracle Database authentication
The Oracle Inventory group (typically, oinstall) for all installations
(Optional) A separate OSASM group (for example, asm) for Automatic Storage Management (ASM) authentication. If this option is not chosen, then dba is the default OSASM group.
The following operating system users are required for all installations:
An unprivileged user (for example, the nobody user on Linux systems)
A single Oracle Inventory group is required for all installations of Oracle software on the system. After the first installation of Oracle software, you must use the same Oracle Inventory group for all subsequent Oracle software installations on that system. However, you can choose to create different users to own the Oracle software and use different operating system groups for authenticating administrative access to each software installation. If an operating system user (for example, oracle) is a member of an operating system group that is used for authenticating access to Oracle software (for example, the dba group), then that user have administrative access to the associated software.
By using different operating system groups for authenticating administrative access to each Oracle Database installation, members of the different groups have SYSDBA privileges for only one database, rather than for all the databases on the system. Also, if you configure a separate operating system group for ASM authentication, then you can have users that have SYSASM access to the ASM instances and do not have SYSDBA access to the database instances.
Note:
If installing Oracle RAC on Microsoft Windows, Oracle Universal Installer automatically creates theORA_DBA group for authenticating SYSDBA access. It does not create an ORA_ASM group for authenticating SYSASM access. Also, if you install the Oracle RAC software while logged in to an account with administrative privileges, you do not need to create a separate user for the installation.To create the required operating system user and groups on Red Hat Linux:
To determine the groups that exist on your server, list the contents of the /etc/group file.
cat /etc/group
If this is the first time Oracle software has been installed on your server, and the Oracle Inventory group does not exist, then create the Oracle Inventory group by entering a command as the root user that is similar to the following:
/usr/sbin/groupadd oinstall
Create an OSDBA group by entering a command as the root user that is similar to the following:
/usr/sbin/groupadd dba
If the user that owns the Oracle software does not exist on your server, you must create the user. Select a user ID (UID) that is currently not in use on all the nodes in your cluster. The following command shows how to create the oracle user and the user's home directory (/home/oracle) with the default group as oinstall and the secondary group as dba, using a UID of 200:
useradd -u 200 –g oinstall -G dba -d /home/oracle -r oracle
To determine which users have already been created on your server, list the contents of the /etc/passwd file.
cat /etc/passwd
Set the password for the oracle account using the following command. Replace password with your own password.
passwd oracle Changing password for user oracle. New UNIX password: password retype new UNIX password: password passwd: all authentication tokens updated successfully.
Repeat Step 1 through Step 4 on each node in your cluster as needed.
Verify that the attributes of the user oracle are identical on both docrac1 and docrac2:
id oracle
The command output should be similar to the following:
uid=200(oracle) gid=500(oinstall) groups=500(oinstall),501(dba)
When installing Oracle RAC on UNIX and Linux platforms, the software is installed on one node, and OUI uses secure communication to copy the software binary files to the other cluster nodes. OUI uses the Secure Shell (SSH) for the communication. Various other components of Oracle RAC and Oracle Clusterware also use SSH for secure communication.
To configure SSH, you must first create Rivest-Shamir-Adleman (RSA) keys and Digital Signature Algorithm (DSA) keys on each cluster node. After you have created the private and public keys, you copy the keys from all cluster node members into an authorized keys file that is identical on each node. When this is done, you then start the SSH agent to load the keys into memory.
See Also:
Oracle Database Advanced Security Administrator's Guide for more information about data security using encryption keys
Create the RSA and DSA keys on each cluster node as the first step in configuring SSH.
To configure the RSA and DSA keys on Red Hat Linux:
Log out and then log back in to the operating system as the oracle user on docrac1.
Note:
Do not use thesu command to switch from the root user to the oracle user for these steps. You must completely exit your operating system session as the root user and start a new session as oracle for these steps to succeed.Determine if a .ssh directory exists in the oracle user's home directory. If not, create the .ssh directory and set the directory permission so that only the oracle user has access to the directory, as shown here:
$ ls -a $HOME $ mkdir ~/.ssh $ chmod 700 ~/.ssh
Create the RSA-type public and private encryption keys. Open a terminal window and run the following command:
/usr/bin/ssh-keygen -t rsa
At the prompts:
Accept the default location for the key file by pressing the Enter key.
When prompted for a pass phrase, enter and confirm a pass phrase that is different from the oracle user's password.
This command creates the public key in the /home/oracle/.ssh/id_rsa.pub file and the private key in the /home/oracle/.ssh/id_rsa file.
Caution:
To protect the security of your system, never distribute the private key to anyone.
Create the DSA type public and private keys on both docrac1 and docrac2. In the terminal window for each node, run the following command:
/usr/bin/ssh-keygen -t dsa
At the prompts:
Accept the default location for the key file by pressing the Enter key.
When prompted for a pass phrase, enter and confirm a pass phrase that is different from the oracle user's password.
This command creates the public key in the /home/oracle/.ssh/id_dsa.pub file and the private key in the /home/oracle/.ssh/id_dsa file.
Caution:
To protect the security of your system, never distribute the private key to anyone.
Repeat Step 1 through Step 4 on each node that you intend to add to the cluster.
See Also:
Oracle Database Advanced Security Administrator's Guide for more information about data security using encryption keys
After you have generated the keys, you copy the keys for each node to an authorized_keys file and copy this file to all nodes in the cluster.
To add the generated keys to an authorized keys file:
On the local node, change directories to the .ssh directory in the oracle user home directory.
cd ~/.ssh
Add the RSA and DSA keys to the authorized_keys file using the following commands, then list the contents of the .ssh directory:
$ cat id_rsa.pub >>authorized_keys $ cat id_dsa.pub >>authorized_keys $ ls
You should see the id_dsa.pub and id_rsa.pub keys that you generated, the id_dsa and id_rsa private key files, and the authorized_keys file.
Use Secure Copy (SCP) or Secure FTP (SFTP) to copy the authorized_keys file to the oracle user .ssh directory on a remote node. The following example uses SCP to copy the authorized_keys file to docrac2, and the oracle user path is /home/oracle:
[oracle@docrac1 .ssh]scp authorized_keys docrac2:/home/oracle/.ssh/ The authenticity of host 'docrac2(143.46.43.101)' can't be established.RSA key fingerprint is 7z:ez:e7:f6:f4:f2:d1:a6:f7:4e:zz:me:a7:48:ae:f6:7e. Are you sure you want to continue connecting (yes/no)? yes oracle@docrac2's password:
You are prompted to accept an RSA or DSA key. Enter yes.
When prompted, provide the password for the oracle user, which should be the same on all the nodes in the cluster. (Note: This is the user password, not the newly specified pass phrase.) The authorized_keys file is then copied to the remote node.
Using SSH, log in to the node where you copied the authorized_keys file, using the pass phrase you created. Then change to the .ssh directory, and using the cat command, add the RSA and DSA keys for the second node to authorized_keys file, as demonstrated here:
[oracle@docrac1 .ssh]$ ssh docrac2 Enter passphrase for key '/home/oracle/.ssh/id_rsa': [oracle@docrac2 oracle]S cd .ssh [oracle@docrac2 ssh]$ cat id_rsa.pub >> authorized_keys [oracle@docrac2 ssh]$ cat id_dsa.pub >> authorized_keys
If you have more than two nodes in your cluster, repeat Step 3 and Step 4 for each node you intend to add to your cluster. Copy the most recently updated authorized_keys file to the next node, then add the public keys for that node to the authorized_keys file.
When you have updated the authorized_keys file on all nodes, use SCP to copy the complete authorized_keys file from the last node to be updated to all the other cluster nodes, overwriting the existing version on the other nodes, for example:
[oracle@docrac2 .ssh]scp authorized_keys docrac1:/home/oracle/.ssh/ The authenticity of host 'docrac1(143.46.43.100)' can't be established. RSA key fingerprint is 7e:62:60:f6:f4:f2:d1:a6:f7:4e:zz:me:b9:48:dc:e3:9c. Are you sure you want to continue connecting (yes/no)? yes oracle@docrac2's password: Warning: Permanently added 'docrac1,143.46.43.100' (RSA) to the list of known hosts. oracle@docrac1's password: authorized_keys 100% 1656 19.9MB.s 00:00
At this point, if you use SSH to log in to or run a command on another node, you are prompted for the pass phrase that you specified when you created the RSA and DSA keys.
See Also:
Oracle Database Advanced Security Administrator's Guide for more information about data security using encryption keys
User equivalency exists in a cluster when the following occurs on all nodes in the cluster:
A given user has the same user name, user ID (UID), and password.
A given user belongs to the same groups.
A given group has the same group ID (GID).
On Linux systems, to enable Oracle Universal Installer to use the ssh and scp commands without being prompted for a pass phrase, you must configure SSH user equivalency.
To configure user SSH equivalency on Red Hat Linux:
On the system where you want to run Oracle Universal Installer, log in to the operating system as the oracle user.
Start the SSH agent and load the SSH keys into memory using the following commands:
$ exec /usr/bin/ssh-agent $SHELL $ /usr/bin/ssh-add
The ssh-add program prompts you to enter the pass phrase for each key that you generated when configuring SSH, for example:
[oracle@docrac1 .ssh]$ exec /usr/bin/ssh-agent $SHELL [oracle@docrac1 .ssh]$ /usr/bin/ssh-add Enter passphrase for /home/oracle/.ssh/id_rsa Identity added: /home/oracle/.ssh/id_rsa (/home/oracle/.ssh/id_rsa) Identity added: /home/oracle/.ssh/id_dsa (/home/oracle/.ssh/id_dsa)
These commands start the ssh-agent program on the node, and load the RSA and DSA keys into memory so that you are not prompted to use pass phrases when issuing SSH commands.If you have configured SSH correctly, then you can now use the ssh or scp commands without being prompted for a password or a pass phrase.
Note:
Do not close this terminal window until you have completed the Oracle Clusterware and Oracle RAC software installation. If you must close this terminal window before the installation is complete, repeat Step 2 before starting or continuing the software installation.Complete the SSH configuration by using the ssh command to retrieve the date on each node in the cluster.
For example, in a two-node cluster, with nodes named docrac1 and docrac2, you would enter the following commands:
$ ssh docrac1 date $ ssh docrac2 date
The first time you use SSH to connect to one node from another node, you see a message similar to the following:
The authenticity of host 'docrac1(143.46.43.100)' can't be established. RSA key fingerprint is 7z:ez:e7:f6:f4:f2:d1:a6:f7:4e:zz:me:a7:48:ae:f6:7e. Are you sure you want to continue connecting (yes/no)? yes
Enter yes at the prompt to continue. You should not see this message again when you connect to this node from the other node. If you see any other messages or text, apart from the date, then the installation can fail.
If any node prompts for a password or pass phrase, then verify that the ~/.ssh/authorized_keys file on that node contains the correct public keys. Make any changes required to ensure that only the date is displayed when you enter these commands. You should also ensure that any parts of login scripts that generate output or ask any questions are modified so that they act only when the shell is an interactive shell.
After completing Step 1 through Step 3, each public host name for each node in the cluster should be registered in the known_hosts file for all other members of the cluster.
On Red Hat Linux, you run Oracle Universal Installer (OUI) from the oracle account. Oracle Universal Installer obtains information from the environment variables configured for the oracle user. Prior to running OUI, you should modify the oracle user environment variables to configure the following:
Set the default file mode creation mask (umask) to 022 in the shell startup file on Linux and UNIX systems.
Set the ORACLE_BASE environment variable to the location in which you plan to install the Oracle Database software. Refer to "About Choosing an Oracle Base Directory" for more information about the ORACLE_BASE directory.
Also, if the /tmp directory has less than 400 MB of available disk space, but you have identified a different file system that has at least 400 MB of available space, you can set the TEMP and TMPDIR environment variables to specify the alternate temporary directory on this file system.
Prior to installing Oracle Clusterware, you can set the ORACLE_HOME variable to the location of the Oracle Clusterware home (also called the CRS home) directory. However, you must also specify the directory in which the software should be installed as part of the installation process. After Oracle Clusterware has been installed, the ORACLE_HOME environment variable will be modified to reflect the value of the Oracle Database home directory.
Note:
Remove anystty commands from such files before you start the installation. On Linux systems, if there are hidden files (such as logon or profile scripts) that contain stty commands, when these files are loaded by the remote shell during installation, OUI indicates an error and stops the installation.Oracle Clusterware requires that you connect the nodes in the cluster to a private network by way of a private interconnect. Each node in the cluster must also be accessible by way of the public network.
To configure the network and ensure that each node in the cluster is able to communicate with the other nodes in the cluster:
Determine your cluster name. The cluster name should satisfy the following conditions:
The cluster name is globally unique throughout your host domain.
The cluster name is at least 1 character long and less than 15 characters long.
The cluster name consists of the same character set used for host names: underscores (_), hyphens (-), and single-byte alphanumeric characters (a to z, A to Z, and 0 to 9).
If you use third-party vendor clusterware, then Oracle recommends that you use the vendor cluster name.
Determine the public node names, private node names, and virtual node names for each node in the cluster.
For the public node name, use the primary host name of each node. In other words, use the name displayed by the hostname command. This node name can be either the permanent or the virtual host name, for example: docrac1.
Determine a private node name or private IP address for each node. The private IP address is an address that is accessible only by the other nodes in this cluster. Oracle Database uses private IP addresses for internode, or instance-to-instance Cache Fusion communication. Oracle recommends that you provide a name in the format public_hostname-priv, for example: docrac1-priv.
Determine a virtual host name for each node. A virtual host name is a public node name that is used to reroute client requests sent to the node if the node is down. Oracle Database uses virtual IP addresses for client-to-database connections, so the VIP address must be publicly accessible. Oracle recommends that you provide a name in the format public_hostname-vip, for example: docrac1-vip.
Identify the interface names and associated IP addresses for all network adapters by running the following command on each node:
# /sbin/ifconfig
From the output, identify the interface name (such as eth0) and IP address for each network adapter that you want to specify as a public or private network interface.
Note:
When you install Oracle Clusterware and Oracle RAC, you will require this information.On each node in the cluster, assign a public IP address with an associated network name to one network adapter, and a private IP address with an associated network name to the other network adapter.
The public name for each node should be registered with your domain name system (DNS). If you do not have an available DNS, then record the network name and IP address in the system hosts file, /etc/hosts. Use the /etc/hosts file on each node to associate the private network name for that host with its private IP address.
You can test whether or not an interconnect interface is reachable using a ping command.
On each node in the cluster, configure a third IP address that will serve as a virtual IP address. Use an IP address that meets the following requirements:
The virtual IP address and the network name must not be currently in use.
The virtual IP address must be on the same subnet as your public IP address.
The virtual host name for each node should be registered with your DNS. If you do not have an available DNS, then record the virtual host name and IP address in the system hosts file, /etc/hosts.
When you complete the network configuration, the IP address and network interface configuration should be similar to what is shown in the following table (your node names and IP addresses might be different):
| Node | Node Name | Type | IP Address | Registered in |
|---|---|---|---|---|
| docrac1 | docrac1 | Public | 143.46.43.100 | DNS (if available, else the hosts file) |
| docrac1 | docrac1-vip | Virtual | 143.46.43.104 | DNS (if available, else the hosts file) |
| docrac1 | docrac1-priv | Private | 10.10.10.11 | Hosts file |
| docrac2 | docrac2 | Public | 143.46.43.101 | DNS (if available, else the hosts file) |
| docrac2 | docrac2-vip | Virtual | 143.46.43.105 | DNS (if available, else the hosts file) |
| docrac2 | docrac2-priv | Private | 10.10.10.12 | Hosts file |
After you have completed the installation process, configure clients to use either the virtual IP address or the network name associated with the virtual IP address.
See Also:
Your platform-specific Oracle Clusterware installation guide
After you have configured the network, perform verification tests to make sure it is configured properly. If there are problems with the network connection between nodes in the cluster, the Oracle Clusterware installation will fail.
To verify the network configuration on a two-node cluster that is running Red Hat Linux:
As the root user, verify the configuration of the public and private networks. Verify that the interfaces are configured on the same network on both docrac1 and docrac2.
In this example, eth0 is used for the public network and eth1 is used for the private network, which is used for Cache Fusion communications.
# /sbin/ifconfig
eth0 Link encap:Ethernet HWaddr 00:0E:0C:08:67:A9
inet addr: 143.46.43.100 Bcast:143.46.43.255 Mask:255.255.240.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:270332689 errors:0 dropped:0 overruns:0 frame:0
TX packets:112346591 errors:2 dropped:0 overruns:0 carrier:2
collisions:202 txqueuelen:1000
RX bytes:622032739 (593.2 MB) TX bytes:2846589958 (2714.7 MB)
Base address:0x2840 Memory:fe7e0000-fe800000
eth1 Link encap:Ethernet HWaddr 00:04:23:A6:CD:59
inet addr: 10.10.10.11 Bcast: 10.10.10.255 Mask:255.255.240.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:21567028 errors:0 dropped:0 overruns:0 frame:0
TX packets:15259945 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:4091201649 (3901.6 MB) TX bytes:377502797 (360.0 MB)
Base address:0x2800 Memory:fe880000-fe8a0000
lo Link encap:Local Loopback
inet addr:127.0.0.1 Mask:255.0.0.0
UP LOOPBACK RUNNING MTU:16436 Metric:1
RX packets:52012956 errors:0 dropped:0 overruns:0 frame:0
TX packets:52012956 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:905082901 (863.1 MB) TX bytes:905082901 (863.1 MB)
As the root user, verify that the /etc/hosts file on the node docrac1 contains the host IP addresses, virtual IP addresses, and private network IP addresses from both nodes in the cluster, as follows:
# Do not remove the following line, or various programs # that require network functionality will fail. 127.0.0.1 localhost.localdomain localhost 143.46.43.100 docrac1.mycompany.com docrac1 143.46.43.104 docrac1-vip.mycompany.com docrac1-vip 10.10.10.11 docrac1-priv 143.46.43.101 docrac2.mycompany.com docrac2 143.46.43.105 docrac2-vip.mycompany.com docrac2-vip 10.10.10.12 docrac2-priv
If the /etc/hosts file is missing any of the preceding information, then edit the file to add the necessary information.
After the /etc/hosts file is configured on docrac1, edit the /etc/hosts file on docrac2 so it contains the same information for the cluster IP addresses.
As the root user, verify the network configuration by using the ping command to test the connection from docrac1 from docrac2 and the reverse. As the root user, run the following commands on each node:
# ping -c 3 docrac1.mycompany.com # ping -c 3 docrac1 # ping -c 3 docrac1-priv # ping -c 3 docrac2.mycompany.com # ping -c 3 docrac2 # ping -c 3 docrac2-priv
You will not be able to discover the nodes using the ping command for the virtual IPs (docrac1-vip, docrac2-vip) until after Oracle Clusterware is installed and running. If the ping commands for the public or private addresses fail, resolve the issue before you proceed.
Ensure that you can access the default gateway with a ping command. To identify the default gateway, use the route command, as described in the Red Hat Linux Help utility.
When you install the Oracle software on your server, Oracle Universal Installer expects the operating system to have specific packages and software applications installed.
This section covers the following topics:
You must ensure that you have a certified combination of the operating system and the Oracle Database software by referring to OracleMetaLink certification, which is located at the following Web site
https://metalink.oracle.com
You can find this by clicking Certify & Availability and then selecting 1.View Certifications by Product.
Note:
Oracle Universal Installer verifies that your server and operating system meet the listed requirements. Check the requirements before you start Oracle Universal Installer, to ensure your server and operating system meet will meet the requirements.Before starting the installation, ensure that the date and time settings on both nodes are set as closely as possible to the same date and time. Oracle strongly recommends using the Network Time Protocol (NTP) feature of most operating systems for this purpose.
NTP is a protocol designed to synchronize the clocks of servers connected by a network. When using NTP, each server on the network runs client software to periodically make timing requests to one or more servers, referred to as reference NTP servers. The information returned by the timing request is used to adjust the server's clock.
All the nodes in your cluster should use the same reference NTP server.
See Also:
Your platform-specific Oracle Clusterware installation guide
OUI checks the current settings for various kernel parameters to ensure they meet the minimum requirements for deploying Oracle RAC. For production database systems, Oracle recommends that you tune the settings to optimize the performance of your particular system.
Note:
If you find parameter settings or shell limit values on your system that are greater than the values mentioned in this section, then do not modify the parameter setting.See Also:
Your platform-specific Oracle Clusterware installation guide
You may be required to perform special configuration steps that are specific to the operating system on which you are installing Oracle RAC, or for the components used with your cluster. The following list provides examples of operating-specific installation tasks:
Configure the use of Huge Pages on SUSE Linux Enterprise Server 9 or Red Hat Enterprise Linux 4.
Configure the hangcheck-timer module on Red Hat Linux 4.0 and SUSE 9 systems.
Set shell limits for the oracle user on Red Hat Linux systems to increase the number of files and processes available to Oracle Clusterware and Oracle RAC.
Start the Telnet service on Microsoft Windows.
Create X library symbolic links on HP-UX.
Configure network tuning parameters on AIX Based Systems.
See Also:
Your platform-specific Oracle Clusterware installation guide
This section describes the storage configuration tasks that you must complete before you start Oracle Universal Installer. It includes information about the following tasks:
See Also:
Your platform-specific Oracle Clusterware installation guide
Each node in a cluster requires external shared disks for storing the Oracle Clusterware (Oracle Cluster Registry and voting disk) files, and Oracle Database files. The supported types of shared storage depend upon the platform you are using, for example:
A supported cluster file system, such as Oracle Cluster File System 2 (OCFS2) for Microsoft Windows and Linux or General Parallel File System (GPFS) on IBM platforms
Network file system (NFS), which is not supported on AIX Based Systems, Linux on POWER, or on IBM zSeries Based Linux
Shared disk partitions consisting of block devices. Block devices are disk partitions that are mounted using the Linux file system. Oracle Clusterware and Oracle RAC write to these partitions directly.
Automatic Storage Management for Oracle Database files (strongly recommended)
Note:
Oracle Clusterware files cannot be stored in ASM.For all installations, you must choose the storage option that you want to use for Oracle Clusterware files and Oracle Database files.
Note:
For the most up-to-date information about supported storage options for Oracle RAC installations, refer to the Certify pages on OracleMetaLinkIf you decide to use OCFS to store the Oracle Clusterware files, you must use the proper version of OCFS for your operating system version. OCFS2 works with Red Hat Linux and kernel version 2.6
The examples in this guide, which are based on Red Hat Linux, use shared disk partitions to store the Oracle Clusterware files and ASM to store the Oracle database files. The Oracle Clusterware and Oracle RAC software will be installed on disks local to each node, not on a shared file system.
The following section describes how to configure the shared disk partitions for the Oracle Clusterware files on Red Hat Linux.
See Also:
Your platform-specific Oracle Clusterware installation guide if you are using a cluster file system or NFS
With Oracle Clusterware release 10.2 and later, you can use block devices instead of raw devices with Red Hat Enterprise Linux 4.0. Oracle Clusterware files are configured by default to use direct I/O (O_DIRECT), which enables direct writes to the block devices.
Before you install Oracle Clusterware, you will need to configure 5 shared disk partitions:
1 partition which is 280 MB in size for storing the Oracle Cluster Registry (OCR)
1 partition which is 280 MB in size for storing a duplicate OCR file on a different disk, referred to as the OCR mirror
3 partitions which are 280 MB in size, 1 for each voting disk location
Note:
When you create partitions usingfdisk by specifying a device size, such as +256M, the actual device created may be smaller than the size requested, based on the cylinder geometry of the disk. This is due to current fdisk restrictions.
Oracle configuration software checks to ensure that devices contain a minimum of 256MB of available disk space. Therefore, Oracle recommends using at least 280MB for the device size. You can check partition sizes by using the command syntax fdisk -s partition.
To configure block devices if you are using Red Hat Enterprise Linux 4.0:
Log in to the operating system as the root user.
To identify the device name for the disks that you want to use, enter the following command on the first node in your cluster, for example, docrac1:
# /sbin/fdisk -l
You can create the required disk partitions either on new block devices that you added or on previously partitioned devices that have unpartitioned available space. To identify devices that have unpartitioned available space, examine the start and end cylinder numbers of the existing partitions and determine whether or not the device contains unused cylinders.
Create two disk partitions, each 280 MB in size, for the OCR and its mirror, and three partitions, each 280 MB in size, for the Oracle Clusterware voting disks.
To create partitions on a block device, as the root user, enter a command similar to the following, where devicename is the name of a block device:
# /sbin/fdisk devicename
Use the following guidelines when creating partitions:
Use the p command to list the partition table of the device.
Use the n command to create a partition.
After you have created the required partitions on this device, use the w command to write the modified partition table to the device.
Refer to the fdisk entry in the Linux Help system for more information about creating partitions.
The following example uses fdisk to create a 280 MB partition on the block device, /dev/sda, on the first node. This partition, or slice, will be used for the OCR disk. You will create another 280 MB partition on a different disk and disk controller for the OCR mirror. Each file should be on a different disk and disk controller. The bold text in the following example represents user-entered commands.
# /sbin/fdisk /dev/sda The number of cylinders for this disk is set to 1024. Command (m for help): p Disk /dev/sdb: 1073 MB, 107341824 bytes 34 heads, 61 sectors/track, 1011 cylinders Units = cylinders of 2074 * 512 = 1061888 bytes Device boot Start End Blocks ID System Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 1 First cylinder (1-1011, default 1): Using default value 1 Last cylinder of +size or +sizeM or +sizeK (1-1011, default 1011): +280M Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. #
Enter the following command to create a 280 MB partition on the second block device, /dev/sdb. This partition will be used for the OCR mirror. Use the same prompts as shown in the previous example.
fdisk /dev/sdb
Use the fdisk command to create a 280 MB partition on the block device /dev/sda. This partition will be used for the voting disk file. Each voting disk file should be on a different disk and controller.
# /sbin/fdisk /dev/sda The number of cylinders for this disk is set to 1024. Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (8-1024, default 8): Using default value 8 Last cylinder of +size or +sizeM or +sizeK (8-1024, default 1024): +280M Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. #
Use the fdisk command to create a 280 MB partition on the block device /dev/sdb. This partition will be used for the voting disk file. Each voting disk file should be on a different disk and controller.
# /sbin/fdisk /dev/sdb The number of cylinders for this disk is set to 1024. Command (m for help): n Command action e extended p primary partition (1-4) p Partition number (1-4): 2 First cylinder (8-1024, default 8): Using default value 8 Last cylinder of +size or +sizeM or +sizeK (8-1024, default 1024): +280M Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. #
Use the fdisk command to create a 280 MB partition on the block device /dev/sdc. This partition will be used for the voting disk file.
# /sbin/fdisk /dev/sdc
When you run the command, use the same response as in Step 3, but specify a partition size of +280M.
On the node docrac2, as the root user, for each of the disks you used previously in Steps 3 through Step 7, you need to run the partprobe command. For example, if you configured disks /dev/sda, /dev/sdb, and /dev/sdc in the previous commands, then you would run the following commands:
# /sbin/partprobe /dev/sda # /sbin/partprobe /dev/sdb # /sbin/partprobe /dev/sdc
This forces the operating system on the other node in the cluster to refresh its kernel partition table for the shared storage device.
Change the ownership of the OCR partitions to the installation owner on all nodes in the cluster.
In the session where you run OUI, the OCR partitions must be owned by the installation owner (such as oracle) that performs the Oracle Clusterware installation. The installation owner must own the OCR partitions so that OUI can write to them. During installation, OUI changes ownership of the OCR partitions back to root.
When you restart a Red Hat Enterprise Linux 4.0 system, ownership and permissions on block devices revert by default to the root user. If you are using block devices with this operating system for your Oracle Clusterware files, then you need to override this default.
To create a permissions file if you are using Red Hat Enterprise Linux 4.0:
Log in to the operating system as the root user.
Change to the /etc/udev/permissions.d directory.
Use a text editor to create a file named 49-oracle.permissions to ensure correct ownership of the block devices when the operating system is restarted.
The following is an example of the contents of the /etc/udev/permissions.d/49-oracle.permissions file:
# OCR sda1:root:oinstall:0640 sdb1:root:oinstall:0640 # Voting Disks sda2:oracle:oinstall:0640 sdb2:oracle:oinstall:0640 sdc1:oracle:oinstall:0640 # ASM sdd:oracle:dba:0660 sde:oracle:dba:0660
Save the file.
(Optional) After creating the oracle.permissions file, the permissions on the shared devices are set automatically the next time the system is restarted. To set permissions to take effect immediately, without restarting the system, use the chown and chmod commands:
chown root:oinstall /dev/sda1 chmod 640 /dev/sda1 chown root:oinstall /dev/sdb1 chown 640 /dev/sdb1 chown oracle:oinstall /dev/sda2 chmod 640 /dev/sda2 chown oracle:oinstall /dev/sdb2 chown 640 /dev/sdb2 chown oracle:oinstall /dev/sdc1 chown 640 /dev/sdc1 chown oracle:dba /dev/sdd chown 660 /dev/sdd chown oracle:dba /dev/sde chown 660 /dev/sde
Repeat these steps on each node in the cluster.
Oracle Universal Installer (OUI) creates the Oracle base directory for you in the location you specify. The Oracle base directory (ORACLE_BASE) acts as a top-level directory for Oracle software installations. Optimal Flexible Architecture (OFA) guidelines recommend that you use a path similar to the following for the Oracle base directory:
/mount_point/app/oracle
In the preceding path example, the variable mount_point is the mount point directory for the file system where you intend to install the Oracle software.
The file system that you use for the Oracle base directory must have at least 1.5 GB of available disk space for installing the Oracle Database software. The path to the Oracle base directory must be the same on all nodes.
For Red Hat Linux systems, you can use the df -h command to determine the available disk space on each mounted file system. Choose a file system that has sufficient available space. For the sample installation described in this guide, the chosen mount point must have at least 3 GB of available space, for installing Oracle RAC and ASM in separate home directories. The examples in this guide use /opt/oracle/11gR1 for the Oracle base directory.
OUI installs Oracle Clusterware into a directory structure referred to as CRS_homeroot user must own the CRS home directory.
The file system that you use for the CRS home directory must have at least 120 MB of available disk space. The path to the CRS home directory must be the same on all nodes.
For Red Hat Linux, you can use the df -h command to determine the available disk space on each mounted file system. Choose a file system that has appropriate available space. For the examples in this guide, the directory /crs is used for the CRS home directory.
Note:
Ensure the CRS home directory is not a subdirectory of the Oracle base directory.