Saturday, July 3, 2010

Install Oracle RAC 11gR2 on Vmware with Windows 7 64-bit as host OS and Linux as guest OS (Part 8, Add node and instance to existing 11gR2 cluster db)

He is Self-Existent


Index of all the posts of Gurpartap Singh's Blog


Now start all the machines starting with "san" and then "rac1" and then rac2". Now also start "rac3" that you had made in step 4. We will use that machine to extend our RAC cluster to a new node and add instance to database "simar" on this node. Incase you had not made rac3 machine, you can make it now by just following the instructions in Part4.

Login to node 1 as oracle and using ". oraenv" set environment to crs and run "crs_stat -t -v" and you should have the output :

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.RECV1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ ONLINE ONLINE rac1
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1
ora.simar.db ora....se.type 0/2 0/1 ONLINE ONLINE rac1

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$

Logon as oracle on rac3 (the node we are going to add) and execute following to see if we it can see luns.

If you get following, then it means you can see them.

[oracle@rac3 ~]$ ls -l /dev/sd*
brw-r----- 1 root disk 8, 0 Jul 1 12:08 /dev/sda
brw-r----- 1 root disk 8, 1 Jul 1 19:09 /dev/sda1
brw-r----- 1 root disk 8, 2 Jul 1 12:08 /dev/sda2
brw-r----- 1 oracle oinstall 8, 16 Jul 1 19:10 /dev/sdb
brw-r----- 1 oracle oinstall 8, 32 Jul 1 19:10 /dev/sdc
brw-r----- 1 oracle oinstall 8, 48 Jul 1 19:10 /dev/sdd
brw-r----- 1 oracle oinstall 8, 64 Jul 1 19:10 /dev/sde
brw-r----- 1 oracle oinstall 8, 80 Jul 1 19:10 /dev/sdf
brw-r----- 1 oracle oinstall 8, 96 Jul 1 19:10 /dev/sdg
brw-r----- 1 oracle oinstall 8, 112 Jul 1 19:10 /dev/sdh
brw-r----- 1 oracle oinstall 8, 128 Jul 1 19:10 /dev/sdi
brw-r----- 1 oracle oinstall 8, 144 Jul 1 19:10 /dev/sdj
brw-r----- 1 oracle oinstall 8, 160 Jul 1 19:10 /dev/sdk
brw-r----- 1 oracle oinstall 8, 176 Jul 1 19:10 /dev/sdl
brw-r----- 1 oracle oinstall 8, 192 Jul 1 19:10 /dev/sdm
[oracle@rac3 ~]$


Let's set user equivalency as:
Generate authorized key as oracle on node 3 as:

On node 3 execute:
mkdir ~/.ssh
chmod 700 ~/.ssh
/usr/bin/ssh-keygen -t rsa
and accept defaults to all prompts.

as:

[oracle@rac3 ~]$ mkdir ~/.ssh
[oracle@rac3 ~]$ chmod 700 ~/.ssh
[oracle@rac3 ~]$ /usr/bin/ssh-keygen -t rsa
Generating public/private rsa key pair.
Enter file in which to save the key (/home/oracle/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/oracle/.ssh/id_rsa.
Your public key has been saved in /home/oracle/.ssh/id_rsa.pub.
The key fingerprint is:
aa:36:84:81:35:00:93:a9:56:93:32:3a:bc:7f:78:51 oracle@rac3.rac.meditate.com
[oracle@rac3 ~]$

Then execute the following commands as oracle on node 3.
cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys
cat /.ssh/authorized_keys



Now copy the last 2 lines of file "~/.ssh/authorized_keys" on node 1 to the end of file "~/.ssh/authorized_keys" on node 3.

Now copy the first line of file "~/.ssh/authorized_keys" to the end of file "~/.ssh/authorized_keys
" on node 1 and 2.

Now ssh from each server to other 2 servers and you should be able to login without password.

Now lets run the cluter verify utility so see what is the result:


Now change the directory to /software/grid and run the following command:
oracle : rac1.rac.meditate.com : @crs : /software/grid
$ ./runcluvfy.sh stage -pre crsinst -n rac1,rac2,rac3 -verbose > runcluvfy_before_adding_node.log

oracle : rac1.rac.meditate.com : @crs : /software/grid
$


The contents of the log file are:
oracle : rac1.rac.meditate.com : @crs : /software/grid
$ cat runcluvfy_before_adding_node.log

Performing pre-checks for cluster services setup

Checking node reachability...

Check: Node reachability from node "rac1"
Destination Node Reachable?
------------------------------------ ------------------------
rac2 yes
rac1 yes
rac3 yes
Result: Node reachability check passed from node "rac1"


Checking user equivalence...

Check: User equivalence for user "oracle"
Node Name Comment
------------------------------------ ------------------------
rac2 passed
rac1 passed
rac3 passed
Result: User equivalence check passed for user "oracle"

Checking node connectivity...

Checking hosts config file...
Node Name Status Comment
------------ ------------------------ ------------------------
rac2 passed
rac1 passed
rac3 passed

Verification of the hosts config file successful


Interface information for node "rac2"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.177 192.168.1.0 0.0.0.0 192.168.1.1 00:0C:29:07:23:50 1500
eth1 10.10.1.177 10.10.1.0 0.0.0.0 192.168.1.1 00:0C:29:07:23:5A 1500
eth2 10.10.2.101 10.10.2.0 0.0.0.0 192.168.1.1 00:0C:29:07:23:64 1500
eth2 10.10.2.177 10.10.2.0 0.0.0.0 192.168.1.1 00:0C:29:07:23:64 1500
eth3 10.10.3.177 10.10.3.0 0.0.0.0 192.168.1.1 00:0C:29:07:23:6E 1500
eth4 10.10.4.177 10.10.4.0 0.0.0.0 192.168.1.1 00:0C:29:07:23:78 1500


Interface information for node "rac1"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.176 192.168.1.0 0.0.0.0 192.168.1.1 00:0C:29:B2:A9:C3 1500
eth1 10.10.1.176 10.10.1.0 0.0.0.0 192.168.1.1 00:0C:29:B2:A9:CD 1500
eth2 10.10.2.100 10.10.2.0 0.0.0.0 192.168.1.1 00:0C:29:B2:A9:D7 1500
eth2 10.10.2.166 10.10.2.0 0.0.0.0 192.168.1.1 00:0C:29:B2:A9:D7 1500
eth2 10.10.2.176 10.10.2.0 0.0.0.0 192.168.1.1 00:0C:29:B2:A9:D7 1500
eth3 10.10.3.176 10.10.3.0 0.0.0.0 192.168.1.1 00:0C:29:B2:A9:E1 1500
eth4 10.10.4.176 10.10.4.0 0.0.0.0 192.168.1.1 00:0C:29:B2:A9:EB 1500


Interface information for node "rac3"
Name IP Address Subnet Gateway Def. Gateway HW Address MTU
------ --------------- --------------- --------------- --------------- ----------------- ------
eth0 192.168.1.178 192.168.1.0 0.0.0.0 192.168.1.1 00:0C:29:41:E9:21 1500
eth1 10.10.1.178 10.10.1.0 0.0.0.0 192.168.1.1 00:0C:29:41:E9:2B 1500
eth2 10.10.2.102 10.10.2.0 0.0.0.0 192.168.1.1 00:0C:29:41:E9:35 1500
eth3 10.10.3.178 10.10.3.0 0.0.0.0 192.168.1.1 00:0C:29:41:E9:3F 1500
eth4 10.10.4.178 10.10.4.0 0.0.0.0 192.168.1.1 00:0C:29:41:E9:49 1500


Check: Node connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac2:eth0 rac1:eth0 yes
rac2:eth0 rac3:eth0 yes
rac1:eth0 rac3:eth0 yes
Result: Node connectivity passed for subnet "192.168.1.0" with node(s) rac2,rac1,rac3


Check: TCP connectivity of subnet "192.168.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac1:192.168.1.176 rac2:192.168.1.177 passed
rac1:192.168.1.176 rac3:192.168.1.178 passed
Result: TCP connectivity check passed for subnet "192.168.1.0"


Check: Node connectivity of subnet "10.10.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac2:eth1 rac1:eth1 yes
rac2:eth1 rac3:eth1 yes
rac1:eth1 rac3:eth1 yes
Result: Node connectivity passed for subnet "10.10.1.0" with node(s) rac2,rac1,rac3


Check: TCP connectivity of subnet "10.10.1.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac1:10.10.1.176 rac2:10.10.1.177 passed
rac1:10.10.1.176 rac3:10.10.1.178 passed
Result: TCP connectivity check passed for subnet "10.10.1.0"


Check: Node connectivity of subnet "10.10.2.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac2:eth2 rac2:eth2 yes
rac2:eth2 rac1:eth2 yes
rac2:eth2 rac1:eth2 yes
rac2:eth2 rac1:eth2 yes
rac2:eth2 rac3:eth2 yes
rac2:eth2 rac1:eth2 yes
rac2:eth2 rac1:eth2 yes
rac2:eth2 rac1:eth2 yes
rac2:eth2 rac3:eth2 yes
rac1:eth2 rac1:eth2 yes
rac1:eth2 rac1:eth2 yes
rac1:eth2 rac3:eth2 yes
rac1:eth2 rac1:eth2 yes
rac1:eth2 rac3:eth2 yes
rac1:eth2 rac3:eth2 yes
Result: Node connectivity passed for subnet "10.10.2.0" with node(s) rac2,rac1,rac3


Check: TCP connectivity of subnet "10.10.2.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac1:10.10.2.100 rac2:10.10.2.101 passed
rac1:10.10.2.100 rac2:10.10.2.177 passed
rac1:10.10.2.100 rac1:10.10.2.166 passed
rac1:10.10.2.100 rac1:10.10.2.176 passed
rac1:10.10.2.100 rac3:10.10.2.102 passed
Result: TCP connectivity check passed for subnet "10.10.2.0"


Check: Node connectivity of subnet "10.10.3.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac2:eth3 rac1:eth3 yes
rac2:eth3 rac3:eth3 yes
rac1:eth3 rac3:eth3 yes
Result: Node connectivity passed for subnet "10.10.3.0" with node(s) rac2,rac1,rac3


Check: TCP connectivity of subnet "10.10.3.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac1:10.10.3.176 rac2:10.10.3.177 passed
rac1:10.10.3.176 rac3:10.10.3.178 passed
Result: TCP connectivity check passed for subnet "10.10.3.0"


Check: Node connectivity of subnet "10.10.4.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac2:eth4 rac1:eth4 yes
rac2:eth4 rac3:eth4 yes
rac1:eth4 rac3:eth4 yes
Result: Node connectivity passed for subnet "10.10.4.0" with node(s) rac2,rac1,rac3


Check: TCP connectivity of subnet "10.10.4.0"
Source Destination Connected?
------------------------------ ------------------------------ ----------------
rac1:10.10.4.176 rac2:10.10.4.177 passed
rac1:10.10.4.176 rac3:10.10.4.178 passed
Result: TCP connectivity check passed for subnet "10.10.4.0"


Interfaces found on subnet "192.168.1.0" that are likely candidates for VIP are:
rac2 eth0:192.168.1.177
rac1 eth0:192.168.1.176
rac3 eth0:192.168.1.178

Interfaces found on subnet "10.10.1.0" that are likely candidates for a private interconnect are:
rac2 eth1:10.10.1.177
rac1 eth1:10.10.1.176
rac3 eth1:10.10.1.178

Interfaces found on subnet "10.10.2.0" that are likely candidates for a private interconnect are:
rac2 eth2:10.10.2.101 eth2:10.10.2.177
rac1 eth2:10.10.2.100 eth2:10.10.2.166 eth2:10.10.2.176
rac3 eth2:10.10.2.102

Interfaces found on subnet "10.10.3.0" that are likely candidates for a private interconnect are:
rac2 eth3:10.10.3.177
rac1 eth3:10.10.3.176
rac3 eth3:10.10.3.178

Interfaces found on subnet "10.10.4.0" that are likely candidates for a private interconnect are:
rac2 eth4:10.10.4.177
rac1 eth4:10.10.4.176
rac3 eth4:10.10.4.178

Result: Node connectivity check passed


Check: Total memory
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 1.47GB (1542316.0KB) 1.5GB (1572864.0KB) failed
rac1 1.47GB (1542316.0KB) 1.5GB (1572864.0KB) failed
rac3 1.47GB (1542316.0KB) 1.5GB (1572864.0KB) failed
Result: Total memory check failed

Check: Available memory
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 427.27MB (437524.0KB) 50MB (51200.0KB) passed
rac1 498.57MB (510540.0KB) 50MB (51200.0KB) passed
rac3 1.41GB (1481208.0KB) 50MB (51200.0KB) passed
Result: Available memory check passed

Check: Swap space
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
Result: Swap space check failed

Check: Free disk space for "rac2:/tmp"
Path Node Name Mount point Available Required Comment
---------------- ------------ ------------ ------------ ------------ ------------
/tmp rac2 / 72.42GB 1GB passed
Result: Free disk space check passed for "rac2:/tmp"

Check: Free disk space for "rac1:/tmp"
Path Node Name Mount point Available Required Comment
---------------- ------------ ------------ ------------ ------------ ------------
/tmp rac1 / 69.68GB 1GB passed
Result: Free disk space check passed for "rac1:/tmp"

Check: Free disk space for "rac3:/tmp"
Path Node Name Mount point Available Required Comment
---------------- ------------ ------------ ------------ ------------ ------------
/tmp rac3 / 80.81GB 1GB passed
Result: Free disk space check passed for "rac3:/tmp"

Check: User existence for "oracle"
Node Name Status Comment
------------ ------------------------ ------------------------
rac2 exists passed
rac1 exists passed
rac3 exists passed
Result: User existence check passed for "oracle"

Check: Group existence for "oinstall"
Node Name Status Comment
------------ ------------------------ ------------------------
rac2 exists passed
rac1 exists passed
rac3 exists passed
Result: Group existence check passed for "oinstall"

Check: Group existence for "dba"
Node Name Status Comment
------------ ------------------------ ------------------------
rac2 exists passed
rac1 exists passed
rac3 exists passed
Result: Group existence check passed for "dba"

Check: Membership of user "oracle" in group "oinstall" [as Primary]
Node Name User Exists Group Exists User in Group Primary Comment
---------------- ------------ ------------ ------------ ------------ ------------
rac2 yes yes yes yes passed
rac1 yes yes yes yes passed
rac3 yes yes yes yes passed
Result: Membership check for user "oracle" in group "oinstall" [as Primary] passed

Check: Membership of user "oracle" in group "dba"
Node Name User Exists Group Exists User in Group Comment
---------------- ------------ ------------ ------------ ----------------
rac2 yes yes yes passed
rac1 yes yes yes passed
rac3 yes yes yes passed
Result: Membership check for user "oracle" in group "dba" passed

Check: Run level
Node Name run level Required Comment
------------ ------------------------ ------------------------ ----------
rac2 3 3,5 passed
rac1 3 3,5 passed
rac3 3 3,5 passed
Result: Run level check passed

Check: Hard limits for "maximum open file descriptors"
Node Name Type Available Required Comment
---------------- ------------ ------------ ------------ ----------------
rac2 hard 131072 65536 passed
rac1 hard 131072 65536 passed
rac3 hard 131072 65536 passed
Result: Hard limits check passed for "maximum open file descriptors"

Check: Soft limits for "maximum open file descriptors"
Node Name Type Available Required Comment
---------------- ------------ ------------ ------------ ----------------
rac2 soft 131072 1024 passed
rac1 soft 131072 1024 passed
rac3 soft 131072 1024 passed
Result: Soft limits check passed for "maximum open file descriptors"

Check: Hard limits for "maximum user processes"
Node Name Type Available Required Comment
---------------- ------------ ------------ ------------ ----------------
rac2 hard 131072 16384 passed
rac1 hard 131072 16384 passed
rac3 hard 131072 16384 passed
Result: Hard limits check passed for "maximum user processes"

Check: Soft limits for "maximum user processes"
Node Name Type Available Required Comment
---------------- ------------ ------------ ------------ ----------------
rac2 soft 131072 2047 passed
rac1 soft 131072 2047 passed
rac3 soft 131072 2047 passed
Result: Soft limits check passed for "maximum user processes"

Check: System architecture
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 x86_64 x86_64 passed
rac1 x86_64 x86_64 passed
rac3 x86_64 x86_64 passed
Result: System architecture check passed

Check: Kernel version
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 2.6.18-164.el5 2.6.9 passed
rac1 2.6.18-164.el5 2.6.9 passed
rac3 2.6.18-164.el5 2.6.9 passed
Result: Kernel version check passed

Check: Kernel parameter for "semmsl"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 250 250 passed
rac1 250 250 passed
rac3 250 250 passed
Result: Kernel parameter check passed for "semmsl"

Check: Kernel parameter for "semmns"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 32000 32000 passed
rac1 32000 32000 passed
rac3 32000 32000 passed
Result: Kernel parameter check passed for "semmns"

Check: Kernel parameter for "semopm"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 100 100 passed
rac1 100 100 passed
rac3 100 100 passed
Result: Kernel parameter check passed for "semopm"

Check: Kernel parameter for "semmni"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 142 128 passed
rac1 142 128 passed
rac3 142 128 passed
Result: Kernel parameter check passed for "semmni"

Check: Kernel parameter for "shmmax"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 4398046511104 536870912 passed
rac1 4398046511104 536870912 passed
rac3 4398046511104 536870912 passed
Result: Kernel parameter check passed for "shmmax"

Check: Kernel parameter for "shmmni"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 4096 4096 passed
rac1 4096 4096 passed
rac3 4096 4096 passed
Result: Kernel parameter check passed for "shmmni"

Check: Kernel parameter for "shmall"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 1073741824 2097152 passed
rac1 1073741824 2097152 passed
rac3 1073741824 2097152 passed
Result: Kernel parameter check passed for "shmall"

Check: Kernel parameter for "file-max"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 6815744 6815744 passed
rac1 6815744 6815744 passed
rac3 6815744 6815744 passed
Result: Kernel parameter check passed for "file-max"

Check: Kernel parameter for "ip_local_port_range"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 between 9000 & 65500 between 9000 & 65500 passed
rac1 between 9000 & 65500 between 9000 & 65500 passed
rac3 between 9000 & 65500 between 9000 & 65500 passed
Result: Kernel parameter check passed for "ip_local_port_range"

Check: Kernel parameter for "rmem_default"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 262144 262144 passed
rac1 262144 262144 passed
rac3 262144 262144 passed
Result: Kernel parameter check passed for "rmem_default"

Check: Kernel parameter for "rmem_max"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 4194304 4194304 passed
rac1 4194304 4194304 passed
rac3 4194304 4194304 passed
Result: Kernel parameter check passed for "rmem_max"

Check: Kernel parameter for "wmem_default"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 262144 262144 passed
rac1 262144 262144 passed
rac3 262144 262144 passed
Result: Kernel parameter check passed for "wmem_default"

Check: Kernel parameter for "wmem_max"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 1048576 1048576 passed
rac1 1048576 1048576 passed
rac3 1048576 1048576 passed
Result: Kernel parameter check passed for "wmem_max"

Check: Kernel parameter for "aio-max-nr"
Node Name Configured Required Comment
------------ ------------------------ ------------------------ ----------
rac2 3145728 1048576 passed
rac1 3145728 1048576 passed
rac3 3145728 1048576 passed
Result: Kernel parameter check passed for "aio-max-nr"

Check: Package existence for "make-3.80"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 make-3.81-3.el5 make-3.80 passed
rac1 make-3.81-3.el5 make-3.80 passed
rac3 make-3.81-3.el5 make-3.80 passed
Result: Package existence check passed for "make-3.80"

Check: Package existence for "binutils-2.15.92.0.2"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 binutils-2.17.50.0.6-12.el5 binutils-2.15.92.0.2 passed
rac1 binutils-2.17.50.0.6-12.el5 binutils-2.15.92.0.2 passed
rac3 binutils-2.17.50.0.6-12.el5 binutils-2.15.92.0.2 passed
Result: Package existence check passed for "binutils-2.15.92.0.2"

Check: Package existence for "gcc-3.4.6"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 gcc-4.1.2-48.el5 gcc-3.4.6 passed
rac1 gcc-4.1.2-48.el5 gcc-3.4.6 passed
rac3 gcc-4.1.2-48.el5 gcc-3.4.6 passed
Result: Package existence check passed for "gcc-3.4.6"

Check: Package existence for "libaio-0.3.105 (i386)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 libaio-0.3.106-5 (i386) libaio-0.3.105 (i386) passed
rac1 libaio-0.3.106-5 (i386) libaio-0.3.105 (i386) passed
rac3 libaio-0.3.106-5 (i386) libaio-0.3.105 (i386) passed
Result: Package existence check passed for "libaio-0.3.105 (i386)"

Check: Package existence for "libaio-0.3.105 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 libaio-0.3.106-5 (x86_64) libaio-0.3.105 (x86_64) passed
rac1 libaio-0.3.106-5 (x86_64) libaio-0.3.105 (x86_64) passed
rac3 libaio-0.3.106-5 (x86_64) libaio-0.3.105 (x86_64) passed
Result: Package existence check passed for "libaio-0.3.105 (x86_64)"

Check: Package existence for "glibc-2.3.4-2.41 (i686)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 glibc-2.5-49.el5_5.2 (i686) glibc-2.3.4-2.41 (i686) passed
rac1 glibc-2.5-49.el5_5.2 (i686) glibc-2.3.4-2.41 (i686) passed
rac3 glibc-2.5-49.el5_5.2 (i686) glibc-2.3.4-2.41 (i686) passed
Result: Package existence check passed for "glibc-2.3.4-2.41 (i686)"

Check: Package existence for "glibc-2.3.4-2.41 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 glibc-2.5-49.el5_5.2 (x86_64) glibc-2.3.4-2.41 (x86_64) passed
rac1 glibc-2.5-49.el5_5.2 (x86_64) glibc-2.3.4-2.41 (x86_64) passed
rac3 glibc-2.5-49.el5_5.2 (x86_64) glibc-2.3.4-2.41 (x86_64) passed
Result: Package existence check passed for "glibc-2.3.4-2.41 (x86_64)"

Check: Package existence for "compat-libstdc++-33-3.2.3 (i386)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 compat-libstdc++-33-3.2.3-61 (i386) compat-libstdc++-33-3.2.3 (i386) passed
rac1 compat-libstdc++-33-3.2.3-61 (i386) compat-libstdc++-33-3.2.3 (i386) passed
rac3 compat-libstdc++-33-3.2.3-61 (i386) compat-libstdc++-33-3.2.3 (i386) passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3 (i386)"

Check: Package existence for "compat-libstdc++-33-3.2.3 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 compat-libstdc++-33-3.2.3-61 (x86_64) compat-libstdc++-33-3.2.3 (x86_64) passed
rac1 compat-libstdc++-33-3.2.3-61 (x86_64) compat-libstdc++-33-3.2.3 (x86_64) passed
rac3 compat-libstdc++-33-3.2.3-61 (x86_64) compat-libstdc++-33-3.2.3 (x86_64) passed
Result: Package existence check passed for "compat-libstdc++-33-3.2.3 (x86_64)"

Check: Package existence for "elfutils-libelf-0.97 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 elfutils-libelf-0.137-3.el5 (x86_64) elfutils-libelf-0.97 (x86_64) passed
rac1 elfutils-libelf-0.137-3.el5 (x86_64) elfutils-libelf-0.97 (x86_64) passed
rac3 elfutils-libelf-0.137-3.el5 (x86_64) elfutils-libelf-0.97 (x86_64) passed
Result: Package existence check passed for "elfutils-libelf-0.97 (x86_64)"

Check: Package existence for "elfutils-libelf-devel-0.97"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.97 passed
rac1 elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.97 passed
rac3 elfutils-libelf-devel-0.137-3.el5 elfutils-libelf-devel-0.97 passed
Result: Package existence check passed for "elfutils-libelf-devel-0.97"

Check: Package existence for "glibc-common-2.3.4"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 glibc-common-2.5-49.el5_5.2 glibc-common-2.3.4 passed
rac1 glibc-common-2.5-49.el5_5.2 glibc-common-2.3.4 passed
rac3 glibc-common-2.5-49.el5_5.2 glibc-common-2.3.4 passed
Result: Package existence check passed for "glibc-common-2.3.4"

Check: Package existence for "glibc-devel-2.3.4 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 glibc-devel-2.5-49.el5_5.2 (x86_64) glibc-devel-2.3.4 (x86_64) passed
rac1 glibc-devel-2.5-49.el5_5.2 (x86_64) glibc-devel-2.3.4 (x86_64) passed
rac3 glibc-devel-2.5-49.el5_5.2 (x86_64) glibc-devel-2.3.4 (x86_64) passed
Result: Package existence check passed for "glibc-devel-2.3.4 (x86_64)"

Check: Package existence for "glibc-headers-2.3.4"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 glibc-headers-2.5-49.el5_5.2 glibc-headers-2.3.4 passed
rac1 glibc-headers-2.5-49.el5_5.2 glibc-headers-2.3.4 passed
rac3 glibc-headers-2.5-49.el5_5.2 glibc-headers-2.3.4 passed
Result: Package existence check passed for "glibc-headers-2.3.4"

Check: Package existence for "gcc-c++-3.4.6"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 gcc-c++-4.1.2-48.el5 gcc-c++-3.4.6 passed
rac1 gcc-c++-4.1.2-48.el5 gcc-c++-3.4.6 passed
rac3 gcc-c++-4.1.2-48.el5 gcc-c++-3.4.6 passed
Result: Package existence check passed for "gcc-c++-3.4.6"

Check: Package existence for "libaio-devel-0.3.105 (i386)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 libaio-devel-0.3.106-5 (i386) libaio-devel-0.3.105 (i386) passed
rac1 libaio-devel-0.3.106-5 (i386) libaio-devel-0.3.105 (i386) passed
rac3 libaio-devel-0.3.106-5 (i386) libaio-devel-0.3.105 (i386) passed
Result: Package existence check passed for "libaio-devel-0.3.105 (i386)"

Check: Package existence for "libaio-devel-0.3.105 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 libaio-devel-0.3.106-5 (x86_64) libaio-devel-0.3.105 (x86_64) passed
rac1 libaio-devel-0.3.106-5 (x86_64) libaio-devel-0.3.105 (x86_64) passed
rac3 libaio-devel-0.3.106-5 (x86_64) libaio-devel-0.3.105 (x86_64) passed
Result: Package existence check passed for "libaio-devel-0.3.105 (x86_64)"

Check: Package existence for "libgcc-3.4.6 (i386)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 libgcc-4.1.2-48.el5 (i386) libgcc-3.4.6 (i386) passed
rac1 libgcc-4.1.2-48.el5 (i386) libgcc-3.4.6 (i386) passed
rac3 libgcc-4.1.2-48.el5 (i386) libgcc-3.4.6 (i386) passed
Result: Package existence check passed for "libgcc-3.4.6 (i386)"

Check: Package existence for "libgcc-3.4.6 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 libgcc-4.1.2-48.el5 (x86_64) libgcc-3.4.6 (x86_64) passed
rac1 libgcc-4.1.2-48.el5 (x86_64) libgcc-3.4.6 (x86_64) passed
rac3 libgcc-4.1.2-48.el5 (x86_64) libgcc-3.4.6 (x86_64) passed
Result: Package existence check passed for "libgcc-3.4.6 (x86_64)"

Check: Package existence for "libstdc++-3.4.6 (i386)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 libstdc++-4.1.2-48.el5 (i386) libstdc++-3.4.6 (i386) passed
rac1 libstdc++-4.1.2-48.el5 (i386) libstdc++-3.4.6 (i386) passed
rac3 libstdc++-4.1.2-48.el5 (i386) libstdc++-3.4.6 (i386) passed
Result: Package existence check passed for "libstdc++-3.4.6 (i386)"

Check: Package existence for "libstdc++-3.4.6 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 libstdc++-4.1.2-48.el5 (x86_64) libstdc++-3.4.6 (x86_64) passed
rac1 libstdc++-4.1.2-48.el5 (x86_64) libstdc++-3.4.6 (x86_64) passed
rac3 libstdc++-4.1.2-48.el5 (x86_64) libstdc++-3.4.6 (x86_64) passed
Result: Package existence check passed for "libstdc++-3.4.6 (x86_64)"

Check: Package existence for "libstdc++-devel-3.4.6 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 libstdc++-devel-4.1.2-48.el5 (x86_64) libstdc++-devel-3.4.6 (x86_64) passed
rac1 libstdc++-devel-4.1.2-48.el5 (x86_64) libstdc++-devel-3.4.6 (x86_64) passed
rac3 libstdc++-devel-4.1.2-48.el5 (x86_64) libstdc++-devel-3.4.6 (x86_64) passed
Result: Package existence check passed for "libstdc++-devel-3.4.6 (x86_64)"

Check: Package existence for "sysstat-5.0.5"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 sysstat-7.0.2-3.el5 sysstat-5.0.5 passed
rac1 sysstat-7.0.2-3.el5 sysstat-5.0.5 passed
rac3 sysstat-7.0.2-3.el5 sysstat-5.0.5 passed
Result: Package existence check passed for "sysstat-5.0.5"

Check: Package existence for "unixODBC-2.2.11 (i386)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 unixODBC-2.2.11-7.1 (i386) unixODBC-2.2.11 (i386) passed
rac1 unixODBC-2.2.11-7.1 (i386) unixODBC-2.2.11 (i386) passed
rac3 unixODBC-2.2.11-7.1 (i386) unixODBC-2.2.11 (i386) passed
Result: Package existence check passed for "unixODBC-2.2.11 (i386)"

Check: Package existence for "unixODBC-2.2.11 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 unixODBC-2.2.11-7.1 (x86_64) unixODBC-2.2.11 (x86_64) passed
rac1 unixODBC-2.2.11-7.1 (x86_64) unixODBC-2.2.11 (x86_64) passed
rac3 unixODBC-2.2.11-7.1 (x86_64) unixODBC-2.2.11 (x86_64) passed
Result: Package existence check passed for "unixODBC-2.2.11 (x86_64)"

Check: Package existence for "unixODBC-devel-2.2.11 (i386)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 unixODBC-devel-2.2.11-7.1 (i386) unixODBC-devel-2.2.11 (i386) passed
rac1 unixODBC-devel-2.2.11-7.1 (i386) unixODBC-devel-2.2.11 (i386) passed
rac3 unixODBC-devel-2.2.11-7.1 (i386) unixODBC-devel-2.2.11 (i386) passed
Result: Package existence check passed for "unixODBC-devel-2.2.11 (i386)"

Check: Package existence for "unixODBC-devel-2.2.11 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 unixODBC-devel-2.2.11-7.1 (x86_64) unixODBC-devel-2.2.11 (x86_64) passed
rac1 unixODBC-devel-2.2.11-7.1 (x86_64) unixODBC-devel-2.2.11 (x86_64) passed
rac3 unixODBC-devel-2.2.11-7.1 (x86_64) unixODBC-devel-2.2.11 (x86_64) passed
Result: Package existence check passed for "unixODBC-devel-2.2.11 (x86_64)"

Check: Package existence for "pdksh-5.2.14"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 pdksh-5.2.14-36.el5 pdksh-5.2.14 passed
rac1 pdksh-5.2.14-36.el5 pdksh-5.2.14 passed
rac3 pdksh-5.2.14-36.el5 pdksh-5.2.14 passed
Result: Package existence check passed for "pdksh-5.2.14"

Check: Package existence for "expat-1.95.7 (x86_64)"
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 expat-1.95.8-8.2.1 (x86_64) expat-1.95.7 (x86_64) passed
rac1 expat-1.95.8-8.2.1 (x86_64) expat-1.95.7 (x86_64) passed
rac3 expat-1.95.8-8.2.1 (x86_64) expat-1.95.7 (x86_64) passed
Result: Package existence check passed for "expat-1.95.7 (x86_64)"

Checking for multiple users with UID value 0
Result: Check for multiple users with UID value 0 passed

Check: Current group ID
Result: Current group ID check passed
Checking Core file name pattern consistency...
Core file name pattern consistency check passed.

Checking to make sure user "oracle" is not in "root" group
Node Name Status Comment
------------ ------------------------ ------------------------
rac2 does not exist passed
rac1 does not exist passed
rac3 does not exist passed
Result: User "oracle" is not part of "root" group. Check passed

Check default user file creation mask
Node Name Available Required Comment
------------ ------------------------ ------------------------ ----------
rac2 0022 0022 passed
rac1 0022 0022 passed
rac3 0022 0022 passed
Result: Default user file creation mask check passed

Starting Clock synchronization checks using Network Time Protocol(NTP)...

NTP Configuration file check started...
Network Time Protocol(NTP) configuration file not found on any of the nodes. Oracle Cluster Time Synchronization Service(CTSS) can be used instead of NTP for time synchronization on the cluster nodes

Result: Clock synchronization check using Network Time Protocol(NTP) passed


Pre-check for cluster services setup was unsuccessful on all the nodes.

oracle : rac1.rac.meditate.com : @crs : /software/grid
$



If the contents of your log file are also similar and failing only on swap then you are good to go the next step.

Now from node 1 as oracle:
. oraenv
crs
cd /u02/app/11.2.0.1/grid/oui/bin
and execute the following script, it will copy the grid infrastructure software to third node and will install it.

./addNode.sh -silent 'CLUSTER_NEW_NODES={vxnode3}' 'CLUSTER_NEW_VIRTUAL_HOSTNAMES={vxnode3_vip}'

as

oracle : rac1.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/oui/bin
$ ./addNode.sh -silent 'CLUSTER_NEW_NODES={rac3}' 'CLUSTER_NEW_VIRTUAL_HOSTNAMES={rac3-vip}'
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 3672 MB Passed
Oracle Universal Installer, Version 11.2.0.1.0 Production
Copyright (C) 1999, 2009, Oracle. All rights reserved.


Performing tests to see whether nodes rac2,rac3 are available
............................................................... 100% Done.

.
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u02/app/11.2.0.1/grid
New Nodes
Space Requirements
New Nodes
rac3
/: Required 3.77GB : Available 75.26GB
Installed Products
Product Names
Oracle Grid Infrastructure 11.2.0.1.0
Sun JDK 1.5.0.17.0
Installer SDK Component 11.2.0.1.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.1.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Enterprise Manager Common Core Files 10.2.0.4.2
Oracle DBCA Deconfiguration 11.2.0.1.0
Oracle RAC Deconfiguration 11.2.0.1.0
Oracle Quality of Service Management (Server) 11.2.0.1.0
Installation Plugin Files 11.2.0.1.0
Universal Storage Manager Files 11.2.0.1.0
Oracle Text Required Support Files 11.2.0.1.0
Automatic Storage Management Assistant 11.2.0.1.0
Oracle Database 11g Multimedia Files 11.2.0.1.0
Oracle Multimedia Java Advanced Imaging 11.2.0.1.0
Oracle Globalization Support 11.2.0.1.0
Oracle Multimedia Locator RDBMS Files 11.2.0.1.0
Oracle Core Required Support Files 11.2.0.1.0
Bali Share 1.1.18.0.0
Oracle Database Deconfiguration 11.2.0.1.0
Oracle Quality of Service Management (Client) 11.2.0.1.0
Expat libraries 2.0.1.0.1
Oracle Containers for Java 11.2.0.1.0
Perl Modules 5.10.0.0.1
Secure Socket Layer 11.2.0.1.0
Oracle JDBC/OCI Instant Client 11.2.0.1.0
Oracle Multimedia Client Option 11.2.0.1.0
LDAP Required Support Files 11.2.0.1.0
Character Set Migration Utility 11.2.0.1.0
Perl Interpreter 5.10.0.0.1
PL/SQL Embedded Gateway 11.2.0.1.0
OLAP SQL Scripts 11.2.0.1.0
Database SQL Scripts 11.2.0.1.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
SSL Required Support Files for InstantClient 11.2.0.1.0
SQL*Plus Files for Instant Client 11.2.0.1.0
Oracle Net Required Support Files 11.2.0.1.0
Oracle Database User Interface 2.2.13.0.0
RDBMS Required Support Files for Instant Client 11.2.0.1.0
Enterprise Manager Minimal Integration 11.2.0.1.0
XML Parser for Java 11.2.0.1.0
Oracle Security Developer Tools 11.2.0.1.0
Oracle Wallet Manager 11.2.0.1.0
Enterprise Manager plugin Common Files 11.2.0.1.0
Platform Required Support Files 11.2.0.1.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
RDBMS Required Support Files 11.2.0.1.0
Oracle Ice Browser 5.2.3.6.0
Oracle Help For Java 4.2.9.0.0
Enterprise Manager Common Files 10.2.0.4.2
Deinstallation Tool 11.2.0.1.0
Oracle Java Client 11.2.0.1.0
Cluster Verification Utility Files 11.2.0.1.0
Oracle Notification Service (eONS) 11.2.0.1.0
Oracle LDAP administration 11.2.0.1.0
Cluster Verification Utility Common Files 11.2.0.1.0
Oracle Clusterware RDBMS Files 11.2.0.1.0
Oracle Locale Builder 11.2.0.1.0
Oracle Globalization Support 11.2.0.1.0
Buildtools Common Files 11.2.0.1.0
Oracle RAC Required Support Files-HAS 11.2.0.1.0
SQL*Plus Required Support Files 11.2.0.1.0
XDK Required Support Files 11.2.0.1.0
Agent Required Support Files 10.2.0.4.2
Parser Generator Required Support Files 11.2.0.1.0
Precompiler Required Support Files 11.2.0.1.0
Installation Common Files 11.2.0.1.0
Required Support Files 11.2.0.1.0
Oracle JDBC/THIN Interfaces 11.2.0.1.0
Oracle Multimedia Locator 11.2.0.1.0
Oracle Multimedia 11.2.0.1.0
HAS Common Files 11.2.0.1.0
Assistant Common Files 11.2.0.1.0
PL/SQL 11.2.0.1.0
HAS Files for DB 11.2.0.1.0
Oracle Recovery Manager 11.2.0.1.0
Oracle Database Utilities 11.2.0.1.0
Oracle Notification Service 11.2.0.0.0
SQL*Plus 11.2.0.1.0
Oracle Netca Client 11.2.0.1.0
Oracle Net 11.2.0.1.0
Oracle JVM 11.2.0.1.0
Oracle Internet Directory Client 11.2.0.1.0
Oracle Net Listener 11.2.0.1.0
Cluster Ready Services Files 11.2.0.1.0
Oracle Database 11g 11.2.0.1.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Thursday, July 1, 2010 9:14:11 PM PDT)
. 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Thursday, July 1, 2010 9:14:17 PM PDT)
............................................................................................... 96% Done.
Home copied to new nodes

Saving inventory on nodes (Thursday, July 1, 2010 9:20:05 PM PDT)
. 100% Done.
Save inventory complete
WARNING:A new inventory has been created on one or more nodes in this session. However, it has not yet been registered as the central inventory of this system.
To register the new inventory please run the script at '/u01/app/oraInventory/orainstRoot.sh' with root privileges on nodes 'rac3'.
If you do not register the inventory, you may not be able to update or patch the products you installed.
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/app/oraInventory/orainstRoot.sh #On nodes rac3
/u02/app/11.2.0.1/grid/root.sh #On nodes rac3
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node

The Cluster Node Addition of /u02/app/11.2.0.1/grid was successful.
Please check '/tmp/silentInstall.log' for more details.

oracle : rac1.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/oui/bin
$


Now logon to "rac3" as root and run the following script. It should start all the rac services and ASM on "rac3". Remeber to to hit enter whereever the script needs your input (i.e. pick default answer) as following:

Output of root script is:
[root@rac3 ~]# /u01/app/oraInventory/orainstRoot.sh
Creating the Oracle inventory pointer file (/etc/oraInst.loc)
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac3 ~]# /u02/app/11.2.0.1/grid/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u02/app/11.2.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-07-01 21:32:50: Parsing the host name
2010-07-01 21:32:50: Checking for super user privileges
2010-07-01 21:32:50: User has super user privileges
Using configuration parameter file: /u02/app/11.2.0.1/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac3'
CRS-2676: Start of 'ora.mdnsd' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac3'
CRS-2676: Start of 'ora.gipcd' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac3'
CRS-2676: Start of 'ora.gpnpd' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac3'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac3'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac3'
CRS-2676: Start of 'ora.diskmon' on 'rac3' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac3'
CRS-2676: Start of 'ora.ctssd' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac3'
CRS-2676: Start of 'ora.drivers.acfs' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac3'
CRS-2676: Start of 'ora.asm' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac3'
CRS-2676: Start of 'ora.crsd' on 'rac3' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac3'
CRS-2676: Start of 'ora.evmd' on 'rac3' succeeded
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.

rac3 2010/07/01 21:37:24 /u02/app/11.2.0.1/grid/cdata/rac3/backup_20100701_213724.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 5023 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.

[root@rac3 ~]# ps -ef grep pmon
oracle 13904 1 0 21:36 ? 00:00:00 asm_pmon_+ASM3
root 14672 11929 0 21:38 pts/0 00:00:00 grep pmon
[root@rac3 ~]#


Onw copy and install database software on third node as:

Now . oraenv on node 1 to smiar1 and cd to /u01/app/oracle/product/11.2.0.1/dbhome_1/oui/bin
and run the following command to copy and install database software to 3rd node as:
./addNode.sh -silent 'CLUSTER_NEW_NODES={rac3}'


as :


oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/11.2.0.1/dbhome_1/oui/bin
$ ./addNode.sh -silent 'CLUSTER_NEW_NODES={rac3}'
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 3655 MB Passed
Oracle Universal Installer, Version 11.2.0.1.0 Production
Copyright (C) 1999, 2009, Oracle. All rights reserved.


Performing tests to see whether nodes rac2,rac3 are available
............................................................... 100% Done.

..
-----------------------------------------------------------------------------
Cluster Node Addition Summary
Global Settings
Source: /u01/app/oracle/product/11.2.0.1/dbhome_1
New Nodes
Space Requirements
New Nodes
rac3
/: Required 3.97GB : Available 71.68GB
Installed Products
Product Names
Oracle Database 11g 11.2.0.1.0
Sun JDK 1.5.0.17.0
Installer SDK Component 11.2.0.1.0
Oracle One-Off Patch Installer 11.2.0.0.2
Oracle Universal Installer 11.2.0.1.0
Oracle Configuration Manager Deconfiguration 10.3.1.0.0
Oracle DBCA Deconfiguration 11.2.0.1.0
Oracle RAC Deconfiguration 11.2.0.1.0
Oracle Database Deconfiguration 11.2.0.1.0
Oracle Configuration Manager 10.3.1.1.0
Oracle ODBC Driverfor Instant Client 11.2.0.1.0
LDAP Required Support Files 11.2.0.1.0
SSL Required Support Files for InstantClient 11.2.0.1.0
Bali Share 1.1.18.0.0
Oracle Extended Windowing Toolkit 3.4.47.0.0
Oracle JFC Extended Windowing Toolkit 4.2.36.0.0
Oracle Real Application Testing 11.2.0.1.0
Oracle Database Vault J2EE Application 11.2.0.1.0
Oracle Label Security 11.2.0.1.0
Oracle Data Mining RDBMS Files 11.2.0.1.0
Oracle OLAP RDBMS Files 11.2.0.1.0
Oracle OLAP API 11.2.0.1.0
Platform Required Support Files 11.2.0.1.0
Oracle Database Vault option 11.2.0.1.0
Oracle RAC Required Support Files-HAS 11.2.0.1.0
SQL*Plus Required Support Files 11.2.0.1.0
Oracle Display Fonts 9.0.2.0.0
Oracle Ice Browser 5.2.3.6.0
Oracle JDBC Server Support Package 11.2.0.1.0
Oracle SQL Developer 11.2.0.1.0
Oracle Application Express 11.2.0.1.0
XDK Required Support Files 11.2.0.1.0
RDBMS Required Support Files for Instant Client 11.2.0.1.0
SQLJ Runtime 11.2.0.1.0
Database Workspace Manager 11.2.0.1.0
RDBMS Required Support Files 11.2.0.1.0
Oracle Globalization Support 11.2.0.1.0
Exadata Storage Server 11.2.0.1.0
Provisioning Advisor Framework 10.2.0.4.2
Enterprise Manager Database Plugin -- Repository Support 11.2.0.1.0
Enterprise Manager Repository Core Files 10.2.0.4.2
Enterprise Manager Database Plugin -- Agent Support 11.2.0.1.0
Enterprise Manager Grid Control Core Files 10.2.0.4.2
Enterprise Manager Common Core Files 10.2.0.4.2
Enterprise Manager Agent Core Files 10.2.0.4.2
Agent Required Support Files 10.2.0.4.2
regexp 2.1.9.0.0
Parser Generator Required Support Files 11.2.0.1.0
Oracle 11g Warehouse Builder Required Files 11.2.0.1.0
Oracle Notification Service (eONS) 11.2.0.1.0
Oracle Text Required Support Files 11.2.0.1.0
Precompiler Required Support Files 11.2.0.1.0
Oracle Database 11g Multimedia Files 11.2.0.1.0
Oracle Multimedia Java Advanced Imaging 11.2.0.1.0
Oracle Multimedia Annotator 11.2.0.1.0
Oracle JDBC/OCI Instant Client 11.2.0.1.0
Oracle Multimedia Locator RDBMS Files 11.2.0.1.0
Oracle Core Required Support Files 11.2.0.1.0
Oracle Help For Java 4.2.9.0.0
Sample Schema Data 11.2.0.1.0
Oracle Starter Database 11.2.0.1.0
Oracle Message Gateway Common Files 11.2.0.1.0
Oracle XML Query 11.2.0.1.0
XML Parser for Oracle JVM 11.2.0.1.0
Expat libraries 2.0.1.0.1
Installation Plugin Files 11.2.0.1.0
Enterprise Manager Common Files 10.2.0.4.2
Perl Modules 5.10.0.0.1
Deinstallation Tool 11.2.0.1.0
Oracle Quality of Service Management (Client) 11.2.0.1.0
Perl Interpreter 5.10.0.0.1
JAccelerator (COMPANION) 11.2.0.1.0
Oracle Containers for Java 11.2.0.1.0
Oracle Code Editor 1.2.1.0.0I
Oracle Net Required Support Files 11.2.0.1.0
Secure Socket Layer 11.2.0.1.0
Oracle Universal Connection Pool 11.2.0.1.0
Oracle JDBC/THIN Interfaces 11.2.0.1.0
Oracle Multimedia Client Option 11.2.0.1.0
Oracle Java Client 11.2.0.1.0
Character Set Migration Utility 11.2.0.1.0
Oracle Locale Builder 11.2.0.1.0
PL/SQL Embedded Gateway 11.2.0.1.0
OLAP SQL Scripts 11.2.0.1.0
Database SQL Scripts 11.2.0.1.0
Oracle Globalization Support 11.2.0.1.0
Required Support Files 11.2.0.1.0
SQL*Plus Files for Instant Client 11.2.0.1.0
Oracle ODBC Driver 11.2.0.1.0
Oracle Database User Interface 2.2.13.0.0
Oracle Notification Service 11.2.0.0.0
Enterprise Manager Minimal Integration 11.2.0.1.0
XML Parser for Java 11.2.0.1.0
Oracle Security Developer Tools 11.2.0.1.0
Oracle Wallet Manager 11.2.0.1.0
Cluster Verification Utility Common Files 11.2.0.1.0
Oracle Clusterware RDBMS Files 11.2.0.1.0
Oracle UIX 2.2.24.5.0
Enterprise Manager plugin Common Files 11.2.0.1.0
HAS Common Files 11.2.0.1.0
Precompiler Common Files 11.2.0.1.0
Installation Common Files 11.2.0.1.0
Oracle Help for the Web 2.0.14.0.0
Oracle LDAP administration 11.2.0.1.0
Buildtools Common Files 11.2.0.1.0
Assistant Common Files 11.2.0.1.0
Oracle Recovery Manager 11.2.0.1.0
PL/SQL 11.2.0.1.0
Generic Connectivity Common Files 11.2.0.1.0
Oracle Database Gateway for ODBC 11.2.0.1.0
Oracle Programmer 11.2.0.1.0
Oracle Database Utilities 11.2.0.1.0
Enterprise Manager Agent 10.2.0.4.2
Oracle Netca Client 11.2.0.1.0
SQL*Plus 11.2.0.1.0
Oracle Call Interface (OCI) 11.2.0.1.0
Oracle Multimedia Locator 11.2.0.1.0
Oracle Multimedia 11.2.0.1.0
Oracle Net 11.2.0.1.0
Database Configuration and Upgrade Assistants 11.2.0.1.0
Oracle XML Development Kit 11.2.0.1.0
Oracle JVM 11.2.0.1.0
Oracle Advanced Security 11.2.0.1.0
Oracle Internet Directory Client 11.2.0.1.0
HAS Files for DB 11.2.0.1.0
Oracle Enterprise Manager Console DB 11.2.0.1.0
Oracle Net Listener 11.2.0.1.0
Oracle Text 11.2.0.1.0
Oracle Net Services 11.2.0.1.0
Oracle Database 11g 11.2.0.1.0
Oracle OLAP 11.2.0.1.0
Oracle Spatial 11.2.0.1.0
Oracle Partitioning 11.2.0.1.0
Enterprise Edition Options 11.2.0.1.0
-----------------------------------------------------------------------------


Instantiating scripts for add node (Thursday, July 1, 2010 10:09:11 PM PDT)
. 1% Done.
Instantiation of add node scripts complete

Copying to remote nodes (Thursday, July 1, 2010 10:09:20 PM PDT)
............................................................................................... 96% Done.
Home copied to new nodes

Saving inventory on nodes (Thursday, July 1, 2010 10:18:54 PM PDT)
. 100% Done.
Save inventory complete
WARNING:
The following configuration scripts need to be executed as the "root" user in each cluster node.
/u01/app/oracle/product/11.2.0.1/dbhome_1/root.sh #On nodes rac3
To execute the configuration scripts:
1. Open a terminal window
2. Log in as "root"
3. Run the scripts in each cluster node

The Cluster Node Addition of /u01/app/oracle/product/11.2.0.1/dbhome_1 was successful.
Please check '/tmp/silentInstall.log' for more details.

oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/11.2.0.1/dbhome_1/oui/bin
$

Now logon as root on "rac3" and run the script root.sh as follows:

and then on node 3 run root.sh as:
[root@rac3 ~]# /u01/app/oracle/product/11.2.0.1/dbhome_1/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.2.0.1/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@rac3 ~]#




Now we will add database instance on node 3 by running dbca from first node after doing .oraenv to simar1 as:
$ dbca

Click "Oracle Real Application Cluster Database"
and then click "Next"



Click "Instance Management" and then click next.



Click "Add an Instance" and then Click Next.



Clcik "Simar" and then enter
username as "sys" and its password.



Double check that both the instances of database "simar" as listed and then click Next.




Here I will select the default instance name as "simar3" and make sure "rac3" is selected.




Check the details here and it would add the new undo tablespace and so new undo datafile and also the redologs for the new instance and then clcik Finish.




Click Ok.




and you will see the following screens.







Clcik Yes.





SQL> select INST_ID,INSTANCE_NUMBER,INSTANCE_NAME,HOST_NAME,STATUS from gv$instance;

INST_ID INSTANCE_NUMBER INSTANCE_NAME HOST_NAME STATUS
---------- --------------- ---------------- ---------------------------------------------------------------- ------------
1 1 simar1 rac1.rac.meditate.com OPEN
3 3 simar3 rac3.rac.meditate.com OPEN
2 2 simar2 rac2.rac.meditate.com OPEN

SQL>

ALso

$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.RECV1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ ONLINE ONLINE rac1
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE rac3
ora....C3.lsnr application 0/5 0/0 ONLINE ONLINE rac3
ora.rac3.gsd application 0/5 0/0 ONLINE ONLINE rac3
ora.rac3.ons application 0/3 0/0 ONLINE ONLINE rac3
ora.rac3.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac3
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1
ora.simar.db ora....se.type 0/2 0/1 ONLINE ONLINE rac1

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$

Thats it.

By keeping silent (Thinking), the mind does not receive satisfaction even though
it thinks with constant absorption.


Part 1 - Vmware Setup

Part 2 - Setup Base Brick Machine

Part 3 - SAN Setup

Part 4 - RAC VM Setup

Part 5 - Install Grid Infrastructure

Part 6 - Run asmca and Install RDBMS Software

Part 7 - Create RAC database with DBCA

Part 8 - Add node and instance to existing 11gR2 Cluster db

Part 9 - Delete node and inst from exisiting 11gR2 cluster db

No comments:

Post a Comment