Friday, August 9, 2013

Post 55 of series - Install Oracle RAC 12c on Vmware with Windows 7 64-bit as host OS and Linux CentOS or RHEL 6 as guest OS (Part 8, Fix shared memory problem before container database install)


The true society is the company of those who meditate on the Name of the Lord. Do not associate with those, O Nanak, who look out only for their own interests. || 2 || PAUREE: Approved is that time, when one meets the True Guru. Joining the Saadh Sangat, the Company of the Holy, he does not suffer pain again. When he attains the eternal place, he does not have to enter the womb again. He comes to see the One God everywhere. He focuses his meditation on the essence of spiritual wisdom, and withdraws his attention from other sights. All chants are chanted by one who chants them with his mouth. Realizing the Hukam of the Lord's Command, he becomes happy, and he is filled with peace and tranquility. Those who are assayed, and placed in the Lord?s treasury, are not declared counterfeit again.





Problem:

On my other VM I had the following problem after I installed the container databases on top of the database
we created in the last post. So lets fix this before installing container database:

WARNING: You are trying to use the MEMORY_TARGET feature. This feature requires the /dev/shm file system to be mounted for at
least 754974720 bytes. /dev/shm is either not mounted or is mounted with available space less than this size. Please fix this so                                                                                          
that MEMORY_TARGET can work as expected. Current available is 316739584 and used is 663584768 bytes. Ensure that the
mount point is /dev/shm for this directory.



As of now we have following, but we need to to have 4gb pace on mount point /dev/shm :

oracle : rac20.rac.meditate.com : @prabgun1 : /home/oracle
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_bbm-lv_root
                       84G   20G   60G  25% /
tmpfs                 747M  134M  613M  18% /dev/shm
/dev/sda1             485M   59M  401M  13% /boot
/dev/mapper/vg_bbm-lv_home
                      9.9G  151M  9.2G   2% /home

oracle : rac20.rac.meditate.com : @prabgun1 : /home/oracle
$

Fix:

Configuring Shared Memory
The shared memory can be configured using a swap partition or swap file. Best practice is to create an appropriately
sized swap partition. However, if the existing swap partition exists and is too small, you can add a swap file by following
the example below (as root user):

So, lets fix it by adding space to it.

oracle : rac20.rac.meditate.com : @prabgun1 : /home/oracle
$ df -h /dev/shm/
Filesystem            Size  Used Avail Use% Mounted on
tmpfs                 747M  134M  613M  18% /dev/shm

oracle : rac20.rac.meditate.com : @prabgun1 : /home/oracle
$


Create a 4G swap file, so as root execute the following command on both nodes, node rac20 and rac21
dd if=/dev/zero of=/swap_file_to_increase_shared_mem bs=1M count=4096


Here is the output:

oracle : rac20.rac.meditate.com : @prabgun1 : /home/oracle
$ su - root
Password:
[root@rac20 ~]# dd if=/dev/zero of=/swap_file_to_increase_shared_mem bs=1M count=4096
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 9.74795 s, 441 MB/s
[root@rac20 ~]#


and

oracle : rac21.rac.meditate.com : @crs : /home/oracle
$ su - root
Password:
[root@rac21 ~]#  dd if=/dev/zero of=/swap_file_to_increase_shared_mem bs=1M count=4096
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB) copied, 9.95022 s, 432 MB/s
[root@rac21 ~]#


Setup the
swap file with the command on both the nodes as root:
mkswap /swap_file_to_increase_shared_mem

Here is the output:
[root@rac20 ~]# mkswap /swap_file_to_increase_shared_mem
mkswap: /swap_file_to_increase_shared_mem: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 4194300 KiB
no label, UUID=fff56d7b-2ed3-4ed7-b909-86f965fd3538
[root@rac20 ~]#


and

[root@rac21 ~]# mkswap /swap_file_to_increase_shared_mem
mkswap: /swap_file_to_increase_shared_mem: warning: don't erase bootbits sectors
        on whole disk. Use -f to force.
Setting up swapspace version 1, size = 4194300 KiB
no label, UUID=57b6e171-1d59-4fff-b474-7433201281e8
[root@rac21 ~]#


To enable it at boot time, edit /etc/fstab to include:
/swap_file_to_increase_shared_mem   swap  swap  defaults    0 0

Though you can use the following to enable the swap file immediately but not automatically at boot time but I am using the
above option as we need to fix the machines permanently:
swapon /swap_file_to_increase_shared_mem


Edit /etc/fstab to set shmfs size at bootup:
shmfs       /dev/shm    tmpfs size=4g    0

If you want to set it on the fly the following but I am not using it and will use the command mentioned above:
mount -t tmpfs shmfs -o size=4g /dev/shm

So now on both the nodes /etc/fstab looks like the following:

[root@rac20 ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Mon Jul 15 00:03:49 2013
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_bbm-lv_root /                       ext4    defaults        1 1
UUID=a84ca6e5-0680-45b8-96af-150faecab250 /boot                   ext4    defaults        1 2
/dev/mapper/vg_bbm-lv_home /home                   ext4    defaults        1 2
/dev/mapper/vg_bbm-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/swap_file_to_increase_shared_mem   swap  swap  defaults    0 0
shmfs       /dev/shm    tmpfs size=4g    0

[root@rac20 ~]#


and

[root@rac21 ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Mon Jul 15 00:03:49 2013
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/vg_bbm-lv_root /                       ext4    defaults        1 1
UUID=a84ca6e5-0680-45b8-96af-150faecab250 /boot                   ext4    defaults        1 2
/dev/mapper/vg_bbm-lv_home /home                   ext4    defaults        1 2
/dev/mapper/vg_bbm-lv_swap swap                    swap    defaults        0 0
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
/swap_file_to_increase_shared_mem   swap  swap  defaults    0 0
shmfs       /dev/shm    tmpfs size=4g    0

[root@rac21 ~]#

Now reboot both the machines and you should see the following "df -h" output:

oracle : rac20.rac.meditate.com : @crs : /home/oracle
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_bbm-lv_root
                       84G   24G   56G  30% /
tmpfs                 4.0G     0  4.0G   0% /dev/shm
/dev/sda1             485M   59M  401M  13% /boot
/dev/mapper/vg_bbm-lv_home
                      9.9G  151M  9.2G   2% /home
shmfs                 4.0G     0  4.0G   0% /dev/shm

oracle : rac20.rac.meditate.com : @crs : /home/oracle
$


and

oracle : rac21.rac.meditate.com : @crs : /home/oracle
$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_bbm-lv_root
                       84G   19G   61G  24% /
tmpfs                 4.0G     0  4.0G   0% /dev/shm
/dev/sda1             485M   59M  401M  13% /boot
/dev/mapper/vg_bbm-lv_home
                      9.9G  151M  9.2G   2% /home
shmfs                 4.0G     0  4.0G   0% /dev/shm

oracle : rac21.rac.meditate.com : @crs : /home/oracle
$


The True Guru is the boat; the Word of the Shabad will carry them across. There is neither wind nor fire, neither water nor form there. The True Name of the True Lord is there; it carries them across the terrifying world-ocean. || 2 || The Gurmukhs reach the shore beyond, lovingly focusing on the True Lord. Their comings and goings are ended, and their light merges into the Light. Following the Guru?s Teachings, intuitive peace wells up within them, and they remain merged in the True Lord. || 3 || The snake may be locked in a basket, but it is still poisonous, and the anger within its mind remains. One obtains what is preordained; why does he blame others? If one, as Gurmukh, hears and believes in the Name, the charm against poison, his mind becomes content. || 4 || The crocodile is caught by the hook and line; caught in the trap of evil-mindedness, he regrets and repents, again and again. He does not understand birth and death; the inscription of one's past actions cannot be erased. || 5 || Injecting the poison of egotism, the world was created; with the Shabad enshrined within, the poison is eliminated. Old age cannot torment one who remains lovingly absorbed in the True Lord. He alone is called Jivan-Mikta, liberated while yet alive, from within whom egotism is eradicated.


Monday, July 22, 2013

Post 54 of series - Install Oracle RAC 12c on Vmware with Windows 7 64-bit as host OS and Linux CentOS or RHEL 6 as guest OS (Part 7, Create Oracle database 12c with dbca)


Uncounted millions, so many they are endless, search for Him. But only that one, who understands his own self, sees God near at hand.


[oracle@rac20 ~]$ cd /u01/app/12.1.0.1/grid/bin/
[oracle@rac20 bin]$ ./asmca


Make sure external is clicked and pick disk 3 as shown below. I will name the diskgroup as FLASH.


Click "OK" to create this diskgroup.
 


Click "OK"


Now add disk to data disk group. Right click on DATA disk group and click "Add Disks".


Pick disk 4 and click "OK".






Click "OK"


Click "Close".


Now start "dbca" as :

[oracle@rac20 bin]$ cd /u01/app/oracle/product/12.1.0.1/bin/
[oracle@rac20 bin]$ ./dbca


Click "Create Database" and click "Next".


Type "Global DatabaseName" as "prabgun"
Then make sure ASM is selected
Datafiles location as "+DATA" and
"Fast Recovery Area" as "+FLASH"
Select database character set as:  AL32UTF8
Put your admin password and click "Next". If "Fast Recovery Area" is not "+FLASH" then select it by clicking "BROWSE" button.




Click "Yes".



Check all the warnings and these can be ignored and click next.







Click "Finish" and see the install go through.





Click "Exit".


Click Close and the database is ready. 




Now login into the database and see its up and running:

[oracle@rac20 ~]$ export ORACLE_SID=prabgun1
[oracle@rac20 ~]$ cd /u01/app/oracle/product/12.1.0.1/bin
[oracle@rac20 bin]$ export ORACLE_HOME=/u01/app/oracle/product/12.1.0.1
[oracle@rac20 bin]$ ./sqlplus / as sysdba

SQL*Plus: Release 12.1.0.1.0 Production on Mon Jul 22 12:07:11 2013

Copyright (c) 1982, 2013, Oracle.  All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options

SQL> set line 300
SQL> select * from gv$instance;

   INST_ID INSTANCE_NUMBER INSTANCE_NAME    HOST_NAME                                                        VERSION           STARTUP_T STATUS       PAR    THREAD# ARCHIVE LOG_SWITCH_WAIT LOGINS      SHU DATABASE_STATUS   INSTANCE_ROLE      ACTIVE_ST BLO     CON_ID INSTANCE_MO EDITION
---------- --------------- ---------------- ---------------------------------------------------------------- ----------------- --------- ------------ --- ---------- ------- --------------- ---------- --- ----------------- ------------------ --------- --- ---------- ----------- -------
FAMILY
--------------------------------------------------------------------------------
         2               2 prabgun2         rac21.rac.meditate.com                                           12.1.0.1.0        21-JUL-13 OPEN         YES          2 STOPPED         ALLOWED     NO  ACTIVE            PRIMARY_INSTANCE   NORMAL    NO           0 REGULAR     EE


         1               1 prabgun1         rac20.rac.meditate.com                                           12.1.0.1.0        21-JUL-13 OPEN         YES          1 STOPPED         ALLOWED     NO  ACTIVE            PRIMARY_INSTANCE   NORMAL    NO           0 REGULAR     EE



SQL>



They alone join the Saadh Sangat, unto whom the Lord becomes Merciful. They chant and meditate on the Ambrosial Name of the Lord.


Friday, July 19, 2013

Post 53 of series - Install Oracle RAC 12c on Vmware with Windows 7 64-bit as host OS and Linux CentOS or RHEL 6 as guest OS (Part 6, Install Oracle 12c database software)


The infinite treasure of the Naam abides within the minds of the
devotees. Birth and death, attachment and suffering, are erased in the Saadh Sangat, the Company of the Holy.



Now let’s create DATA disk group for storing data.

Now login in different window as user “oracle” on server rac20 and make sure your “Xming” is up and running and execute following as oracle user:

cd /u01/app/12.1.0.1/grid/bin
./asmsa

[oracle@rac20 bin]$ pwd
/u01/app/12.1.0.1/grid/bin
[oracle@rac20 bin]$ ./asmca


Click "Create" and on following screen select disk 2 and name the diskgroup as "Data"and make sure its "external" and then click "OK"


Click OK on the below window as well.



Click "Exit"

Now lets install the database software:

Now as user “oracle” execute following:

cd /software/12c_database_and_grid_infrastructure/database/
./runInstaller


Click Next


Click "Yes"


Click "Next"


Click Yes


Select "Install Database Software Only"


Make sure "Oracle Real Application Clusters Database installation" is selected and click "Next"



Make sure both nodes are selected and now click "Next".





Click Next on following:


Make sure "Enterprise Edition" is checked and click next.


Update software location to "/u01/app/oracle/product/12.1.0.1" and click "Next"




In all boxes check "oinstall" and click Next.


Click Install.






As root execute the following script on rac20 and let it finish and then execute on rac21

Here is the output:

[root@rac20 ~]# /u01/app/oracle/product/12.1.0.1/root.sh
Performing root user operation for Oracle 12c

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/12.1.0.1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@rac20 ~]#


[root@rac21 ~]# /u01/app/oracle/product/12.1.0.1/root.sh
Performing root user operation for Oracle 12c

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/12.1.0.1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
[root@rac21 ~]#

Click OK.


Click Close.



We are done with the install of Oracle 12c Database software now.


Those who indulge in attachment, conflict and egotism shall surely weep and cry. Those who are separated from the Naam
shall never find any peace.

Post 52 of series - Install Oracle RAC 12c on Vmware with Windows 7 64-bit as host OS and Linux CentOS or RHEL 6 as guest OS (Part 5, Install Grid Infrastructure 12c)



Seeing Him, my mind is comforted and consoled, and my sins and mistakes are far removed. How could I ever forget Him, O
my mother? He is permeating and pervading everywhere. |


Index of all the posts of Gurpartap Singh's Blog

As user "oracle" execute the following:

[oracle@rac20 grid]$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 72456 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 5119 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed

Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-07-19_12-16-38AM. Please wait ...[oracle@rac20 grid]$


Clcik "Next" on the following screen.



Click "Next" again.


Click "Install and Configure Oracle Grid Infrastructure for a cluster.


Make sure "Choose a standard cluster" is clicked.


Make sure "Typical Installation" is checked.


Update scan name to rac20-scan and click "Add"


On the following screen for Public Hostname add "rac21.rac.meditate.com"
and 
for Virtual Hostname add "rac21-vip.rac.meditate.com"


On the following screen clcik "SSH Connectivity"


Supply your password for user oracle in "OS Password" and click "Setup"





On following screen click "OK"


On following screen click "Next"






Update software location to "/u01/app/12.1.0.1/grid and supply password for "SYSASM"  and click "Next"



On following screen click "Next"


On following screen update "Disk group name" to GRID, click radio button in front of "external" and click "candidate disks" and you should see the following.


Click "Change Discovery Path" and change on following screen to "/dev/asm*"



Now you should see the luns we created as follows:


Select disk 1 as follows and click Next.


On the following screen just click next.


Click "Next" again


Click "Install"








Execute script "/u01/app/oraInventory/orainstRoot.sh" as root on "rac20" and after it finishes on "rac21"

Here is the output:
[root@rac20 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac20 ~]#

[root@rac21 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac21 ~]#


Now as root execute following script on rac20 and let it finish before you execute it on rac21. Here is the output:

[root@rac20 ~]# /u01/app/12.1.0.1/grid/root.sh
Performing root user operation for Oracle 12c

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0.1/grid/crs/install/crsconfig_params
OLR initialization - successful
  root wallet
  root wallet cert
  root cert export
  peer wallet
  profile reader wallet
  pa wallet
  peer wallet keys
  pa wallet keys
  peer cert request
  pa cert request
  peer cert
  pa cert
  peer root cert TP
  profile reader root cert TP
  pa root cert TP
  peer pa cert TP
  pa peer cert TP
  profile reader pa cert TP
  profile reader peer cert TP
  peer user cert
  pa user cert
2013/07/19 00:41:57 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-2672: Attempting to start 'ora.evmd' on 'rac20'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac20'
CRS-2676: Start of 'ora.mdnsd' on 'rac20' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac20'
CRS-2676: Start of 'ora.gpnpd' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac20'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac20'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac20' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac20'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac20'
CRS-2676: Start of 'ora.diskmon' on 'rac20' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac20' succeeded

ASM created and started successfully.

Disk Group GRID created successfully.

CRS-2672: Attempting to start 'ora.storage' on 'rac20'
CRS-2676: Start of 'ora.storage' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac20'
CRS-2676: Start of 'ora.crsd' on 'rac20' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 0fd6a6ad76a14fe9bf206366b40d69d0.
Successfully replaced voting disk group with +GRID.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
##  STATE    File Universal Id                File Name Disk group
--  -----    -----------------                --------- ---------
 1. ONLINE   0fd6a6ad76a14fe9bf206366b40d69d0 (/dev/asm-grid-disk1) [GRID]
Located 1 voting disk(s).
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'rac20'
CRS-2673: Attempting to stop 'ora.crsd' on 'rac20'
CRS-2677: Stop of 'ora.crsd' on 'rac20' succeeded
CRS-2673: Attempting to stop 'ora.evmd' on 'rac20'
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac20'
CRS-2673: Attempting to stop 'ora.storage' on 'rac20'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac20'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac20'
CRS-2677: Stop of 'ora.storage' on 'rac20' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac20'
CRS-2677: Stop of 'ora.mdnsd' on 'rac20' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'rac20' succeeded
CRS-2677: Stop of 'ora.evmd' on 'rac20' succeeded
CRS-2677: Stop of 'ora.asm' on 'rac20' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'rac20'
CRS-2677: Stop of 'ora.ctssd' on 'rac20' succeeded
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'rac20' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac20'
CRS-2677: Stop of 'ora.cssd' on 'rac20' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac20'
CRS-2677: Stop of 'ora.gipcd' on 'rac20' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'rac20' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac20'
CRS-2672: Attempting to start 'ora.evmd' on 'rac20'
CRS-2676: Start of 'ora.mdnsd' on 'rac20' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac20'
CRS-2676: Start of 'ora.gpnpd' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac20'
CRS-2676: Start of 'ora.gipcd' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac20'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac20'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac20'
CRS-2676: Start of 'ora.diskmon' on 'rac20' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rac20'
CRS-2676: Start of 'ora.cssd' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac20'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac20'
CRS-2676: Start of 'ora.ctssd' on 'rac20' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac20'
CRS-2676: Start of 'ora.asm' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac20'
CRS-2676: Start of 'ora.storage' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac20'
CRS-2676: Start of 'ora.crsd' on 'rac20' succeeded
CRS-6023: Starting Oracle Cluster Ready Services-managed resources
CRS-6017: Processing resource auto-start for servers: rac20
CRS-6016: Resource auto-start has completed for server rac20
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/07/19 00:47:42 CLSRSC-343: Successfully started Oracle clusterware stack

CRS-2672: Attempting to start 'ora.asm' on 'rac20'
CRS-2676: Start of 'ora.asm' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.GRID.dg' on 'rac20'
CRS-2676: Start of 'ora.GRID.dg' on 'rac20' succeeded
Preparing packages for installation...
cvuqdisk-1.0.9-1
2013/07/19 00:48:53 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

[root@rac20 ~]#


Now execute it on rac21 as:

[root@rac21 ~]# /u01/app/12.1.0.1/grid/root.sh
Performing root user operation for Oracle 12c

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
   Copying dbhome to /usr/local/bin ...
   Copying oraenv to /usr/local/bin ...
   Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0.1/grid/crs/install/crsconfig_params
OLR initialization - successful
2013/07/19 00:50:39 CLSRSC-330: Adding Clusterware entries to file 'oracle-ohasd.conf'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac21'
CRS-2672: Attempting to start 'ora.evmd' on 'rac21'
CRS-2676: Start of 'ora.mdnsd' on 'rac21' succeeded
CRS-2676: Start of 'ora.evmd' on 'rac21' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac21'
CRS-2676: Start of 'ora.gpnpd' on 'rac21' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac21'
CRS-2676: Start of 'ora.gipcd' on 'rac21' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac21'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac21' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac21'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac21'
CRS-2676: Start of 'ora.diskmon' on 'rac21' succeeded
CRS-2789: Cannot stop resource 'ora.diskmon' as it is not running on server 'rac21'
CRS-2676: Start of 'ora.cssd' on 'rac21' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'rac21'
CRS-2672: Attempting to start 'ora.ctssd' on 'rac21'
CRS-2676: Start of 'ora.ctssd' on 'rac21' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'rac21' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac21'
CRS-2676: Start of 'ora.asm' on 'rac21' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'rac21'
CRS-2676: Start of 'ora.storage' on 'rac21' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac21'
CRS-2676: Start of 'ora.crsd' on 'rac21' succeeded
CRS-6017: Processing resource auto-start for servers: rac21
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'rac20'
CRS-2672: Attempting to start 'ora.ons' on 'rac21'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'rac20' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'rac20'
CRS-2677: Stop of 'ora.scan1.vip' on 'rac20' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'rac21'
CRS-2676: Start of 'ora.scan1.vip' on 'rac21' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'rac21'
CRS-2676: Start of 'ora.ons' on 'rac21' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'rac21' succeeded
CRS-6016: Resource auto-start has completed for server rac21
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
2013/07/19 00:54:25 CLSRSC-343: Successfully started Oracle clusterware stack

Preparing packages for installation...
cvuqdisk-1.0.9-1
2013/07/19 00:54:46 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

[root@rac21 ~]#

Then click "OK" and above screen and you see the following screens and let the install complete as follows. 





Click close on the following screen.



 We are done with the Grid Infrastructure install:

Run Installer window log after finish the install:

[oracle@rac20 grid]$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 72456 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 5119 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 16777216    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-07-19_12-16-38AM. Please wait ...[oracle@rac20 grid]$ Logfile Location : /tmp/OraInstall2013-07-19_12-16-38AM/sshsetup1_2013-07-19_12-16-38AM.log
ClusterLogger - log file location: /tmp/Logs/remoteInterfaces2013-07-19_12-16-38AM.log
Validating remote binaries..
Remote binaries check succeeded
You can find the log of this install session at:
 /u01/app/oraInventory/logs/installActions2013-07-19_12-16-38AM.log


[oracle@rac20 grid]$


Now as root check all oracle grid infrastructure processes are online with the following commands.

[root@rac20 ~]#  /u01/app/12.1.0.1/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GRID.dg
               ONLINE  ONLINE       rac20                    STABLE
               ONLINE  ONLINE       rac21                    STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac20                    STABLE
               ONLINE  ONLINE       rac21                    STABLE
ora.asm
               ONLINE  ONLINE       rac20                    STABLE
               ONLINE  ONLINE       rac21                    Started,STABLE
ora.net1.network
               ONLINE  ONLINE       rac20                    STABLE
               ONLINE  ONLINE       rac21                    STABLE
ora.ons
               ONLINE  ONLINE       rac20                    STABLE
               ONLINE  ONLINE       rac21                    STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac21                    STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac20                    STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac20                    STABLE
ora.cvu
      1        ONLINE  ONLINE       rac20                    STABLE
ora.oc4j
      1        OFFLINE OFFLINE                               STABLE
ora.rac20.vip
      1        ONLINE  ONLINE       rac20                    STABLE
ora.rac21.vip
      1        ONLINE  ONLINE       rac21                    STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac21                    STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rac20                    STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rac20                    STABLE
--------------------------------------------------------------------------------
[root@rac20 ~]#










[root@rac21 ~]# /u01/app/12.1.0.1/grid/bin/crsctl stat res -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GRID.dg
               ONLINE  ONLINE       rac20                    STABLE
               ONLINE  ONLINE       rac21                    STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       rac20                    STABLE
               ONLINE  ONLINE       rac21                    STABLE
ora.asm
               ONLINE  ONLINE       rac20                    STABLE
               ONLINE  ONLINE       rac21                    Started,STABLE
ora.net1.network
               ONLINE  ONLINE       rac20                    STABLE
               ONLINE  ONLINE       rac21                    STABLE
ora.ons
               ONLINE  ONLINE       rac20                    STABLE
               ONLINE  ONLINE       rac21                    STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       rac21                    STABLE
ora.LISTENER_SCAN2.lsnr
      1        ONLINE  ONLINE       rac20                    STABLE
ora.LISTENER_SCAN3.lsnr
      1        ONLINE  ONLINE       rac20                    STABLE
ora.cvu
      1        ONLINE  ONLINE       rac20                    STABLE
ora.oc4j
      1        OFFLINE OFFLINE                               STABLE
ora.rac20.vip
      1        ONLINE  ONLINE       rac20                    STABLE
ora.rac21.vip
      1        ONLINE  ONLINE       rac21                    STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       rac21                    STABLE
ora.scan2.vip
      1        ONLINE  ONLINE       rac20                    STABLE
ora.scan3.vip
      1        ONLINE  ONLINE       rac20                    STABLE
--------------------------------------------------------------------------------

[root@rac21 ~]#



Now shutdown the machines and take the backup before Installing RAC database software. We will install the RAC database Database software in the next post.  


In humility, I bowed down in surrender to Him, and He
naturally met me. I have received what was pre-ordained for me, O Nanak, with the help and assistance of the Saints.