3 Configuring Oracle Clusterware Storage

3.1.1 Overview of Oracle Clusterware Storage Options

There are two ways of storing Oracle Clusterware files:

• A supported shared file system: Supported file systems include the following:

  • A supported cluster file system

    Note:

    For information about how to download and configure Oracle Cluster File System 2 (OCFS2), refer to the following URL

    http://oss.oracle.com/projects/ocfs/documentation/
    

    OCFS (version 1) is designed for the 2.4 kernel. You must use OCFS2 with this release.

    See Also:

    The Certify page on OracleMetalink for supported cluster file systems

  • Network File System (NFS): A file-level protocol that enables access and sharing of files

  See Also:

  The Certify page on OracleMetalink for supported Network Attached Storage (NAS) devices

• Block or Raw Devices: Oracle Clusterware files can be placed on either Block or RAW devices based on shared disk partitions. Oracle recommends using Block devices for easier usage.

3.1.2 General Storage Considerations for Oracle Clusterware

For all installations, you must choose the storage option that you want to use for Oracle Clusterware files and Oracle Database files, or for Oracle Clusterware with Oracle Real Application Clusters databases (Oracle RAC). You do not have to use the same storage option for each file type.

Oracle Clusterware files include voting disks, used to monitor cluster node status, and Oracle Cluster Registry (OCR) which contains configuration information about the cluster. The voting disks and OCR are shared files on a cluster or network file system environment. If you do not use a cluster file system, then you must place these files on shared block devices or shared raw devices. Oracle Universal Installer (OUI) automatically initializes the OCR during the Oracle Clusterware installation.

For voting disk file placement, Oracle recommends that each voting disk is configured so that it does not share any hardware device or disk, or other single point of failure. Any node that does not have available to it an absolute majority of voting disks configured (more than half) will be restarted.

The following table shows the storage options supported for storing Oracle Clusterware files. Oracle Clusterware files include the Oracle Cluster Registry (OCR), a mirrored OCR file (optional), the Oracle Clusterware voting disk, and additional voting disk files (optional).

http://oss.oracle.com/projects/ocfs2/

Use the following guidelines when choosing the storage options that you want to use for Oracle Clusterware:

• You can choose any combination of the supported storage options for each file type provided that you satisfy all requirements listed for the chosen storage options.

• You cannot use ASM to store Oracle Clusterware files, because these files must be accessible before any ASM instance starts.

• If you do not have a storage option that provides external file redundancy, then you must configure at least three voting disk areas to provide voting disk redundancy.

3.1.3 Quorum Disk Location Restriction with Existing 9.2 Clusterware Installations

When upgrading your Oracle9i release 9.2 Oracle RAC environment to Oracle Database 11g release 1 (11.1), you are prompted to specify one or more voting disks during the Oracle Clusterware installation. You must specify a new location for the voting disk in Oracle Database 11g release 1 (11.1). You cannot reuse the old Oracle9i release 9.2 quorum disk for this purpose.

3.1.4 After You Have Selected Disk Storage Options

When you have determined your disk storage options, you must perform the following tasks in the order listed:

1: Check for available shared storage with CVU

Refer to Checking for Available Shared Storage with CVU.

2: Configure shared storage for Oracle Clusterware files

• To use a file system (NFS, OCFS2, GPS, GPFS) for Oracle Clusterware files, refer to Configuring Storage for Oracle Clusterware Files on a Supported Shared File System.

• To use block devices for Oracle Clusterware files, refer to Configuring Disk Devices for Oracle Clusterware Files.

3.3.1 Requirements for Using a File System for Oracle Clusterware Files

To use a file system for Oracle Clusterware files, the file system must comply with the following requirements:

• To use a cluster file system, it must be a supported cluster file system, as listed in the section "Deciding to Use a Cluster File System for Oracle Clusterware Files".

• To use an NFS file system, it must be on a certified NAS device. Log in to OracleMetaLink at the following URL, and click the Certify tab to find a list of certified NAS devices.

  https://metalink.oracle.com/

• If you choose to place your Oracle Cluster Registry (OCR) files on a shared file system, then Oracle recommends that one of the following is true:

  • The disks used for the file system are on a highly available storage device, (for example, a RAID device that implements file redundancy).

  • At least two file systems are mounted, and use the features of Oracle Clusterware 11g release 1 (11.1) to provide redundancy for the OCR.

• The user account with which you perform the installation (oracle or crs) must have write permissions to create the files in the path that you specify.

Use Table 3-2 to determine the partition size for shared file systems.

In Table 3-2, the total required volume size is cumulative. For example, to store all Oracle Clusterware files on the shared file system with normal redundancy, you should have at least 1.3 GB of storage available over a minimum of three volumes (two separate volume locations for the OCR and OCR mirror, and one voting disk on each volume).

[root@node1]$ fdisk -s /dev/sdb1
281106

3.3.2 Deciding to Use a Cluster File System for Oracle Clusterware Files

For Linux x86 (32-bit) and x86 (64-bit) platforms, Oracle provides Oracle Cluster File System (OCFS2). Use Oracle Cluster File System 2 (OCFS2), rather than OCFS version 1 (OCFS), as OCFS2 is designed for Linux kernel 2.6. You can have a shared Oracle home on OCFS2.

If you have an existing Oracle installation, then use the following command to determine if OCFS2 is installed:

# rpm -qa | grep ocfs

To ensure that OCFS2 is loaded, enter the following command:

/etc/init.d/ocfs status

If you want to install Oracle Clusterware files on an OCFS2 file system, and the packages are not installed, then download them from the following Web site. Follow the instructions listed with the kit to install the packages and configure the file system:

OCFS2:

http://oss.oracle.com/projects/ocfs2/

https://metalink.oracle.com

3.3.3 Checking NFS Mount Buffer Size Parameters for Clusterware

If you are using NFS, then you must set the values for the NFS buffer size parameters rsize and wsize to at least 16384. Oracle recommends that you use the value 32768.

For example, if you decide to use rsize and wsize buffer settings with the value 32768, then update the /etc/fstab file on each node with an entry similar to the following:

nfs_server:/vol/DATA/oradata  /home/oracle/netapp     nfs\   
rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600

3.3.4 Creating Required Directories for Oracle Clusterware Files on Shared File Systems

Use the following instructions to create directories for Oracle Clusterware files. You can also configure shared file systems for the Oracle Database and recovery files.

To create directories for the Oracle Clusterware files on separate file systems from the Oracle base directory, follow these steps:

1. If necessary, configure the shared file systems that you want to use and mount them on each node.

   Note:

   The mount point that you use for the file system must be identical on each node. Ensure that the file systems are configured to mount automatically when a node restarts.

2. Use the df -h command to determine the free disk space on each mounted file system.

3. From the display, identify the file systems that you want to use. Choose a file system with a minimum of 512 MB of free disk space (one OCR and one voting disk, with external redundancy).

   If you are using the same file system for more than one type of file, then add the disk space requirements for each type to determine the total disk space requirement.

4. Note the names of the mount point directories for the file systems that you identified.

5. If the user performing installation (typically, crs or oracle) has permissions to create directories on the disks where you plan to install Oracle Clusterware and Oracle Database, then OUI creates the Oracle Clusterware file directory, and DBCA creates the Oracle Database file directory, and the Recovery file directory.

   If the user performing installation does not have write access, then you must create these directories manually using commands similar to the following to create the recommended subdirectories in each of the mount point directories and set the appropriate owner, group, and permissions on the Oracle Clusterware home (or CRS home). For example, where the user is oracle, and the CRS home is oracrs:

   # mkdir /mount_point/oracrs
   # chown oracle:oinstall /mount_point/oracrs
   # chmod 640 /mount_point/oracrs
   

   Note:

   After installation, directories in the installation path for the Oracle Cluster Registry (OCR) files should be owned by root, and not writable by any account other than root.

When you have completed creating a subdirectory in the mount point directory, and set the appropriate owner, group, and permissions, you have completed OCFS2 or NFS configuration for Oracle Clusterware.

3.4.1 Example of Creating a Udev Permissions File for Oracle Clusterware

The procedure to create a permissions file to grant oinstall group members write privileges to block devices is as follows:

1. Log in as root.

2. Change to the /etc/udev/permissions.d directory:

   # cd /etc/udev/permissions.d
   

3. Start a text editor, such as vi, and enter the partition information where you want to place the OCR and voting disk files, using the syntax device[partitions]:root:oinstall:0640. Note that Oracle recommends that you place the OCR and the voting disk files on separate physical disks. For example, to grant oinstall members access to SCSI disks to place OCR files on sda1 and sdb2, and to grant the Oracle Clusterware owner (in this example crs) permissions to place voting disks on sdb3, sdc1 and sda2, add the following information to the file:

   # OCR disks
   sda1:root:oinstall:0640
   sdb2:root:oinstall:0640
   # Voting disks
   sda2:crs:oinstall:0640
   sdb3:crs:oinstall:0640
   sdc1:crs:oinstall:0640
   

4. Save the file:

   On Red Hat and Oracle Enterprise Linux 4 systems, save the file as 49-oracle.permissions.

   On SUSE Linux Enterprise Server 10 systems, save the file as 51-oracle.permissions.

5. Using the following command, assign the permissions in the udev file to the devices:

   # /sbin/udevstart
   

3.4.2 Example of Configuring Block Device Storage for Oracle Clusterware

The procedure to create partitions for Oracle Clusterware files on block devices is as follows:

1. log in as root

2. Enter the fdisk command to format a specific storage disk (for example, /sbin/fdisk /dev/sdb)

3. Create a new partition, and make the partition 280 MB in size for both OCR and voting disk partitions.

4. Use the command syntax /sbin/partprobe diskpath on each node in the cluster to update the kernel partition table for the shared storage device on each node.

The following is an example of how to use fdisk to create one partition on a shared storage block disk device for an OCR file:

[crs@localnode /] $ su 
Password:
[root@localnode /] # /sbin/fdisk /dev/sdb
The number of cylinders for this disk is set to 1024.
Command (m for help): n
Command action
  e   extended
  P   primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-1024, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1024, default 1)
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-1024, default 1024):+280m

Command (m for help):w

The partition table has been altered!
Calling ioctl () to re-read partition table.
Synching disks.
[root@localnode /]#
exit
[crs@localnode /] $ ssh remotenode
Last login Wed Feb 21 20:23:01 from localnode
[crs@remotenode ~]$ su
Password:
[root@localnode /] # /sbin/partprobe /dev/sdb1