12.1.0.2 RAC Installation

========================

1. RPMs for 12c on RHEL 6.x

Check if the pre-requisite packages are installed using rpm –q  command as shown below.

rpm –q <package>

e.g: rpm –q binutils

The list of packages to be installed are below

 

binutils-2.20.51.0.2-5.11.el6 (x86_64)

compat-libcap1-1.10-1 (x86_64)

compat-libstdc++-33-3.2.3-69.el6 (x86_64)

compat-libstdc++-33-3.2.3-69.el6 (i686)

gcc-4.4.4-13.el6 (x86_64)

gcc-c++-4.4.4-13.el6 (x86_64)

glibc-2.12-1.7.el6 (i686)

glibc-2.12-1.7.el6 (x86_64)

glibc-devel-2.12-1.7.el6 (x86_64)

glibc-devel-2.12-1.7.el6 (i686)

ksh

libgcc-4.4.4-13.el6 (i686)

libgcc-4.4.4-13.el6 (x86_64)

libstdc++-4.4.4-13.el6 (x86_64)

libstdc++-4.4.4-13.el6 (i686)

libstdc++-devel-4.4.4-13.el6 (x86_64)

libstdc++-devel-4.4.4-13.el6 (i686)

libaio-0.3.107-10.el6 (x86_64)

libaio-0.3.107-10.el6 (i686)

libaio-devel-0.3.107-10.el6 (x86_64)

libaio-devel-0.3.107-10.el6 (i686)

libXext-1.1 (x86_64)

libXext-1.1 (i686)

libXtst-1.0.99.2 (x86_64)

libXtst-1.0.99.2 (i686)

libX11-1.3 (x86_64)

libX11-1.3 (i686)

libXau-1.0.5 (x86_64)

libXau-1.0.5 (i686)

libxcb-1.5 (x86_64)

libxcb-1.5 (i686)

libXi-1.3 (x86_64)

libXi-1.3 (i686)

make-3.81-19.el6

sysstat-9.0.4-11.el6 (x86_64)

unixODBC-2.2.14-11.el6 (x86_64) or later

unixODBC-2.2.14-11.el6.i686 or later

unixODBC-devel-2.2.14-11.el6 (x86_64) or later

unixODBC-devel-2.2.14-11.el6.i686 or later

 

use rpm –ivh <package name> or yum install <package> commands to install the RPMs.

2. Kernel Parameters and Security

• Configure Kernel Parameters

Update /etc/sysctl.conf file with below parameters

fs.file-max = 6815744

kernel.sem = 250 32000 100 128

kernel.shmmni = 4096

kernel.shmall = 1073741824

kernel.shmmax = 4398046511104

net.core.rmem_default = 262144

net.core.rmem_max = 4194304

net.core.wmem_default = 262144

net.core.wmem_max = 1048576

fs.aio-max-nr = 1048576

net.ipv4.ip_local_port_range = 9000 65500

 

run /sbin/sysctl –p for the parameters to take effect.

 

• Oracle User Limits

 

Add Limits for Oracle user in /etc/security/limits.conf file with below values.

 

oracle   soft   nofile    1024

oracle   hard   nofile    65536

oracle   soft   nproc    2047

oracle   hard   nproc    16384

oracle   soft   stack    10240

oracle   hard   stack    32768

 

• Create groups

 

Create dba, oinstall and other groups as needed using below commands.

 

groupadd -g 1000 dba

groupadd -g 1200 oinstall

#groupadd -g 1700 asmdba

#groupadd -g 1800 asmoper

#groupadd -g 1900 asmadmin

 

• Oracle User

Create Oracle and Grid users as needed.

Useradd –u 1100 –g  oinstall –G dba oracle

Useradd –u 900 –g oinstall –G dba grid

 

3. Storage Configuration

• Recommended Storage disks.

Can be changed as needed.

 

Virtual Disk	Device Name	Virtual Device	Size (GB)	Purpose
Hard Disk 1	/dev/sda	SCSI 0:0	100	OS,Grid and Oracle Binaries
Hard Disk 2	/dev/sdb	SCSI 1:0	10	OCR
Hard Disk 3	/dev/sdc	SCSI 1:1	10	OCR
Hard Disk 4	/dev/sdd	SCSI 1:2	50	FRA – Archivelogs
Hard Disk 5	/dev/sde	SCSI 1:3	300	DATA1 – Oracle datafiles
Hard Disk 6	/dev/sdf	SCSI 1:4	300	DATA2 – Oracle datafiles

 

• Add Hard Disks on all Nodes of the cluster

1. In vSphere Client, select a virtual machine.
2. Right-click the name of the virtual machine and select Edit Settings to display the Virtual Machine Properties pane.

iii. Click Add to display the Add Hardware wizard.

1. In the Add Hardware wizard select Hard Disk for the Device Type, and click Next.
2. In the Disk section of the Select a Disk page, select Create a new virtual disk. Set the disk size to 20GB, Click Next.
3. On the Compatibility Mode page, click Next.

vii. On the Advanced Options page, in the Virtual Device Node section, select SCSI (1:0). In the Mode section, select Independent. Click Next.

viii. On the Ready to Complete page, click Finish.

1. Click OK.

 

Note: Repeat the same steps from i to ix for all Hard Disks to be added on all the nodes.

Note2: When Adding Hard Disks on Node2, point to the same vmdk storage file as in Node1.

 

• Configure Multi-writer options to enable shared storage.

In Virtual Machine Properties Pane, Select the Hard Disk, Click on Options tab, and Under the Advanced settings, select General and click on configuration parameters to display the pane as in the screenshot below.

Click on Add Row and add the values for all the disks from Hard Disk 2 to 5.

Name	Value
scsi1:0.sharing	multi-writer
scsi1:1.sharing	multi-writer
scsi1:2.sharing	multi-writer
scsi1:3.sharing	multi-writer
scsi1:4.sharing	multi-writer

Note: Repeat the same on all Nodes of the cluster.

• Partition the disks using fdisk utility.

1. Enter the command:

fdisk /dev/sdb

1. Enter n to create a new partition.

iii. Enter p to create a primary partition.

1. Enter 1 to create the first partition.
2. Press Enter to take the default value of 1 for the first cylinder.
3. Press Enter to take the default value for the last cylinder.

vii. Enter w to write the new partition table to disk and exit fdisk .

E.g as seen in the below screenshot.

Note1: Repeat the same for all hard disk which are not partitioned.

Note2: Partitioning needs to be done on only one of the cluster Nodes.

• ASM Configuration
  • Below list of RPMs need to be installed for Kernel version

kmod-oracleasm-2.0.6.rh1-2.el6.x86_64.rpm

oracleasmlib-2.0.4-1.el6.x86_64.rpm

oracleasm-support-2.1.8-1.el6.x86_64.rpm

 

• Configure ASM Library.

Configure ASM as seen in below screenshot

Note: Repeat this on all nodes of the cluster.

• Create ASM Disks

Run Below commands to Create, Scan and List the Disks.

oracleasm createdisk DISK1 /dev/sdb1

oracleasm createdisk DISK2 /dev/sdc1

oracleasm createdisk DISK3 /dev/sdd1

oracleasm createdisk DISK4 /dev/sde1

oracleasm createdisk DISK5 /dev/sdf1

 

Note: Createdisk commands need to be run any one of the cluster nodes.

And run below commands on all nodes of the cluster to make sure they are visible on all nodes.

 

Oracleasm scandisks

Oracleasm listdisks

 

4. Network Configuration

 

• Configure the /etc/hosts on all the Nodes as below.

 

127.0.0.1       localhost.localdomain.com   localhost

# Public

192.168.0.101   RACDBP1.localdomain.com        RACDBP1

192.168.0.102   RACDBP 2.localdomain.com        RACDBP2

# Private

192.168.1.101   RACDBP1-priv.localdomain.com   RACDBP1-priv

192.168.1.102   RACDBP2-priv.localdomain.com   RACDBP2-priv

# Virtual

192.168.0.111   RACDBP1-vip.localdomain.com    RACDBP1-vip

192.168.0.112   RACDBP2-vip.localdomain.com    RACDBP2-vip

# SCAN

#192.168.0.121   RACDBP-scan.localdomain.com                RACDBP-scan

#192.168.0.122   RACDBP-scan.localdomain.com                RACDBP-scan

#192.168.0.123   RACDBP-scan.localdomain.com                RACDBP-scan

 

Note: If DNS is configured to resolve the SCAN name to all the assigned IPs of SCAN, /etc/hosts  file does not need to be updated with SCAN IPs hence commented out in the above list.

The SCAN IPs need to be updated in /etc/hosts only when DNS is not working.

 

• DNS Configuration

The SCAN name should be able resolve to all the assigned IPs.

In this case, SCAN Name is RACDBP-scan and it resolves into the IPs 192.168.0.121, 192.168.0.122 and 192.168.0.123

On all the Nodes configure /etc/resolv.conf with to reflect the organizations DNS configuration. Example is below.

#domain localdomain.comsearch localdomain.comnameserver 10.2.2.4

If both domain and search entries exists, comment out one of them as shown above.

 

Check if DNS is working.

nslookup RACDBP-scan

nslookup RACDBP-scan.localdomain.com

# nslookup smdmdbs-scan

Server:        10.2.2.4

Address:       10.2.2.4#53

 

Name:   RACDBP-scan.localdomain.com

Address: 192.168.0.121

Name:   RACDBP-scan.localdomain.com

Address: 192.168.0.122

Name:   RACDBP-scan.localdomain.com

Address: 192.168.0.123

 

5. Others configurations

 

• Disable SELINUX

Disable SELINUX by updating the parameter in /etc/selinux/config as below.

SELINUX=disabled

 

• Umask

Umask value for oracle and root user should be set to 0022.

 

• Stop IPtables

service iptables stop

chkconfig iptables off

 

• Nozeroconf

Add the parameter nozeroconf to /etc/sysconfig/network file as below.

NOZEROCONF=yes

 

• Disable NTP

Disable NTP using below commands.

# service ntpd stop

# chkconfig ntpd off

# mv /etc/ntp.conf /etc/ntp.conf.orig

# rm /var/run/ntpd.pid

 

• Directories

Create directories for Grid and Oracle Binaries.

mkdir -p  /u01/app/12.1.0/grid

mkdir -p /u01/app/oracle/product/12.1.0

chown -R oracle:oinstall /u01

chmod -R 775 /u01/

 

• Bash_profile

Update the bash_profile on each of the nodes accordingly as below.

# Oracle Settings

export TMP=/tmp

export TMPDIR=$TMP

export ORACLE_HOSTNAME=RACDBP1.localdomain.com

export ORACLE_UNQNAME=SMDMDBS

export ORACLE_BASE=/u01/app/oracle

export GRID_HOME=/u01/app/12.1.0/grid

export DB_HOME=$ORACLE_BASE/product/12.1.0

export ORACLE_HOME=$DB_HOME

export ORACLE_SID=SMDMDBS1

export ORACLE_TERM=xterm

export BASE_PATH=/usr/sbin:$PATH

export PATH=$ORACLE_HOME/bin:$BASE_PATH

export LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib

export CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib

 

 

• Softwares

Download and unzip the Softwares from Metalink or from the link below.

unzip linuxamd64_12102_grid_1of2.zip

unzip linuxamd64_12102_grid_2of2.zip

unzip linuxamd64_12102_database_1of2.zip

unzip linuxamd64_12102_database_2of2.zip

 

http://www.oracle.com/technetwork/database/enterprise-edition/downloads/index-092322.html

 

• Cluster Verification Utility

Run Cluster Verification Utility to make sure all the pre-requisites for Grid Installation have been met.

./runcluvfy.sh stage -pre crsinst -n RACDBP1,RACDBP2 –verbose

 

Check for the errors in the output of the above command and fix them before proceeding with installation.

 

 

6. Installation

 

• GRI Infrastructure Installation

Go to the location where GRID software is unzipped and invoke runInstaller as grid or oracle user.

Select the “Install and Configure Oracle Grid Infrastructure for a Cluster” option, then click the “Next” button.

Accept the “Configure a Standard cluster” option by clicking the “Next” button.

Select the “Typical Installation” option, then click the “Next” button.

On the “Specify Cluster Configuration” screen, enter the correct SCAN Name and click the “Add” button.

Enter the details of the second node in the cluster, then click the “OK” button.

Click the “SSH Connectivity…” button and enter the password for the “oracle” user. Click the “Setup” button to configure SSH connectivity, and the “Test” button to test it once it is complete. Once the test is complete, click the “Next” button.

If you are doing a 12.1.0.1 installation, you will have to click the “Identify network interfaces” button, but in 12.1.0.2 this is on the following screen.

Check the public and private networks are specified correctly. If the NAT interface is displayed, remember to mark it as “Do Not Use”. Click the “Next” button.

Enter “/u01/app/12.1.0.2/grid” as the software location and “Automatic Storage Manager” as the cluster registry storage type. Enter the ASM password, select “dba” as the group and click the “Next” button.

Set the redundancy to “External”, click the “Change Discovery Path” button and set the path to “/dev/asm*”. Return the main screen and select all 4 disks and click the “Next” button.

Accept the default inventory directory by clicking the “Next” button.

If you want the root scripts to run automatically, enter the relevant credentials. I prefer to run them manually. Click the “Next” button.

Wait while the prerequisite checks complete. If you have any issues use the “Fix & Check Again” button. Once possible fixes are complete, check the “Ignore All” checkbox and click the “Next” button. It is likely the “Physical Memory” and “Device Checks for ASM” tests will fail for this type of installation. This is OK.

If you are happy with the summary information, click the “Install” button.

Wait while the installation takes place.

When prompted, run the configuration scripts on each node.

Once the scripts have completed, return to the “Execute Configuration Scripts” screen and click the “OK” button.

Wait for the configuration assistants to complete.

Click the “Close” button to exit the installer.

 

• Oracle Cluster Software Installation

Go to the location where cluster database software is unzipped and invoke runInstaller.

Uncheck the security updates checkbox and click the “Next” button and “Yes” on the subsequent warning dialog.

Select the “Install database software only” option, then click the “Next” button.

Accept the “Oracle Real Application Clusters database installation” option by clicking the “Next” button.

Make sure both nodes are selected, then click the “Next” button.

Select the required languages, then click the “Next” button.

Select the “Enterprise Edition” option, then click the “Next” button.

Enter “/u01/app/oracle” as the Oracle base and “/u01/app/oracle/product/12.1.0” as the software location, then click the “Next” button.

Select the desired operating system groups, then click the “Next” button.

Wait for the prerequisite check to complete. If there are any problems either click the “Fix & Check Again” button, or check the “Ignore All” checkbox and click the “Next” button.

click the “Install” button.

Wait while the installation takes place.

When prompted, run the configuration script on each node. When the scripts have been run on each node, click the “OK” button.

Click the “Close” button to exit the installer.

 

This is THE END of installation. Hope you have found it useful.

 

Questions are welcome and feedback is appreciated.

jamalocpdba@gmail.com