Friday, July 2, 2010

Install Oracle RAC 11gR2 on Vmware with Windows 7 64-bit as host OS and Linux as guest OS (Part 6, Run asmca and Install RDBMS software)

He has image of undying i.e. He has no effect of time


Index of all the posts of Gurpartap Singh's Blog

After you start all the machines clusterware will bring up everything as below.

Logon on to node 1 as oracle/oracle
. oraenv
crs

$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ ONLINE ONLINE rac1
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$


Now run ASMCA , X11 should have been fowarded through putty as we did earlier and Xming should be running.:

$ . oraenv
ORACLE_SID = [crs] ? +ASM1
The Oracle base for ORACLE_HOME=/u02/app/11.2.0.1/grid is /u01/app/oracle

oracle : rac1.rac.meditate.com : @+ASM1 : /home/oracle
$ asmca

It will open the following window. Right click on disk group "DISK1" and click add disks and select 3 disks and add them.




Then you will see teh following screen.



Then the following message. Click ok on this.





Now on the following screen click create to create the new diskgroup.



I namded it "RECV1" and click external and click show candidate and pick 2 disks here and click ok.




Then you will see the following screen.



and then at the following screen click ok.




Now on the following screen verify both the disk groups are mounted on both the nodes.

Then on the following screen click exit and then yes on next screen.






Now we are ready to install database software.

Now we are ready to install database software as:
cd /software

Now unzip the database software as:
unzip linux.x64_11gR2_database_1of2.zip

and then
unzip linux.x64_11gR2_database_2of2.zip

and after that we can remove the zipped files as:
rm linux.x64_11gR2_database_1of2.zip
and
rm linux.x64_11gR2_database_2of2.zip
oracle : rac1.rac.meditate.com : @+ASM1 : /software
$ cd database

oracle : rac1.rac.meditate.com : @+ASM1 : /software/database
$ ls -lart
total 44
-rwxr-xr-x 1 oracle oinstall 3226 Aug 15 2009 runInstaller
drwxr-xr-x 2 oracle oinstall 4096 Aug 15 2009 rpm
drwxr-xr-x 4 oracle oinstall 4096 Aug 15 2009 install
drwxr-xr-x 14 oracle oinstall 4096 Aug 15 2009 stage
drwxrwxr-x 2 oracle oinstall 4096 Aug 15 2009 sshsetup
drwxrwxr-x 2 oracle oinstall 4096 Aug 15 2009 response
drwxr-xr-x 12 oracle oinstall 4096 Aug 16 2009 doc
-rw-r--r-- 1 oracle oinstall 5402 Aug 17 2009 welcome.html
drwxr-xr-x 8 oracle oinstall 4096 Aug 20 2009 .
drwxr-xr-x 3 oracle oinstall 4096 Jul 1 16:06 ..

Now Xming should be running and X11 should have been forwarded on putty.
oracle : rac1.rac.meditate.com : @+ASM1 : /software/database
$ ./runInstaller

and you will see the following screens.



And then on the following screen.
Uncheck "I wish to receive security updates via My oracle Support"


Click yes


Click install database software only.




Click Real application cluster database installation and make sure both nodes rac1 and rac1 are checked and clicked SSH connectivity and enter oracle password just to make sure silent SSH between nodes is Ok and then click "TEST".



On the following screen click OK.




Select english and click OK.




Click enterprise edition and then click next.



Here change oracle home to:
/u01/app/oracle/product/11.2.0.1/dbhome_1
and click next.



Pick default i.e. dba and then click next.




Next it will do pre-checks



Here click next.



You will see the installation screens. Depending on your RAM and CPU the speed of install will vary. Have patience as it may take long time and you will see the following screens.






Now run the root.sh scritps as:
First run on node1 i.e. rac1 as following and accept defaults for all the questions as:

[root@rac1 ~]# /u01/app/oracle/product/11.2.0.1/dbhome_1/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.2.0.1/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@rac1 ~]#


and then on node 2 as:

[root@rac2 ~]# /u01/app/oracle/product/11.2.0.1/dbhome_1/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.2.0.1/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@rac2 ~]#



Then go back to root.sh screen and click ok and then on the following screen click close.




Now we are all set to create the database. Stop the machines and take the backup of all the machines and restart then with SAN machine being the first to be started and last to be switched off.

Remember:
On the journey where no one knows you, with the Name of the Lord, you shall be recognized.

Thursday, July 1, 2010

Install Oracle RAC 11gR2 on Vmware with Windows 7 64-bit as host OS and Linux as guest OS (Part 5, Install Grid Infrastructure)

He has no Hatred and no enmity


Index of all the posts of Gurpartap Singh's Blog



Finally we are ready to install Grid Infrastructure. Copy the software (one zipped file of grid and two zipped files for database) you downloaded from oracle to rac1 under /software directory using Winscp or anyother method and do this as user oracle. I did it using Winscp.














Now login to rac1 as oracle from putty and unzip the grid file as, following are the screenshots that show you how to open a putty window that has X11 forward enabled. I had explained it in the last part also but here I am explaining agin with screen shots.



Open the putty window and click on ssh under connection and click X11 and then check enable X11 forwarding and login. I am logging in as oracle as we will install Grid infrastructure now.





Now logon to machine rac1 from this putty window and unzip the grid software file as following:

cd /software

unzip linux.x64_11gR2_grid.zip


Once unzip completes you can remove the zipped file of grid to save space.
rm linux.x64_11gR2_grid.zip


At this point start Xming if you have not started by now and run installer as follows.

Now we will run installer as:
cd /software/grid
./runInstaller

and you will see the following screen:








Click next make sure "Install and configure Grid infrastructure for a cluster" is already checked.








I will do a default install so click "Advanced Installation" and next









Here select language and I will select English and next.









Here keep cluster name default and make scan name "scan.rac.meditate.com" and uncheck GNS as will not be using GNS and click next.







Here installer will pick the hostname and vip and will do its best guess but you need to make sure that the installer is picking right values. If not fix them, I had following:
rac1.rac.meditate.com and rac1-vip.rac.meditate.com so nothing to change. Now I will add second node here, I will not add third node as it will not work on 4GB ram also I want to show how to add node to existing 2 node rac later. So, I am saving machine rac3 for now. But if you want to install on 3 nodes add other 2 nodes here and it would work but remember you need at least 6GB ram for that.












So click add and on the following screen add the second node as:
rac2.rac.meditate.com and rac2-vip.rac.meditate.com
and click OK. and then click next.













Now select both the nodes and click SSH connectivity and enter the password of oracle user , in our case its (oracle) and click setup.

This is a nice new install feature that takes the guesswork of setting up ssh connectivity between the nodes. Note, that if you made a typo in the /etc/hosts file the installer might fail to set up the ssh connectivity. Though Oracle RAC is getting heavier on RAM but they are adding cool new features.












You will see the following and click Ok.



Then, Click ssh test and you will see the following and click ok.










When you click “Next” the installer will test the ssh connectivity, validate node readiness, user equivalence, etc. before going to the next step, interface setup.

Our public is 10.10.2.0 and private is 10.10.3.0 and set all others to don't use. We should use the public ip as scan and vip ip's.




Here click next.



You will not see any luns here but just change the discovery path to /dev/sd* and you be able to see all the luns that we had created earlier.














click external and I will have the disk group named as default and i.e. DATA1 and will pick 3 luns randomley, I picked just one lun at this point and will add luns later and click next.







I will use same password for simplicity.




Click yes.




Click don't use intelligent platform.



Click Next and choose all as oinstall.


click yes.





Next change the Software location for Grid Home to /u02/app/11.2.0.1/grid. When you click “Next” the installer doesn’t jump to the next panel right away - be patient.


Click next on next screen where we need to pick inventory location as we will pick default.
and then it will run prechecks and you will see the following screen.








If you see it only fails at swap and physical memory then don't worry.



Details of swap error.



Details of physical memory erro. Its some bug just forget about it for now and continue.




On the next screen clcik finish. Depending on the RAM and CPU of your computer the speed of install will vary, be patient and if you are curious then do tail on the installer log to see whats happening behind the sceens.












Now run these scripts. Logon to rac1 and ececute both the scripts one by one. First script is quick and the second script takes time. Be patient, second script also brings up ASM.
Once both the scripts are a success on rac1 then execute both the scripts on rac2. Remeber to accept all teh default values. Just hit enter key where it needs some input.




Here are my outputs:

Node 1:

[root@rac1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac1 ~]# /u02/app/11.2.0.1/grid/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u02/app/11.2.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-07-01 14:16:36: Parsing the host name
2010-07-01 14:16:36: Checking for super user privileges
2010-07-01 14:16:36: User has super user privileges
Using configuration parameter file: /u02/app/11.2.0.1/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded

ASM created and started successfully.

DiskGroup DATA1 created successfully.

clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
CRS-4256: Updating the profile
Successful addition of voting disk 44d575b60a4b4fb5bf3cf8b51d001778.
Successfully replaced voting disk group with +DATA1.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 44d575b60a4b4fb5bf3cf8b51d001778 (/dev/sdc) [DATA1]
Located 1 voting disk(s).
CRS-2673: Attempting to stop 'ora.crsd' on 'rac1'
CRS-2677: Stop of 'ora.crsd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'rac1'
CRS-2677: Stop of 'ora.asm' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.ctssd' on 'rac1'
CRS-2677: Stop of 'ora.ctssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'rac1'
CRS-2677: Stop of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'rac1'
CRS-2677: Stop of 'ora.cssd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'rac1'
CRS-2677: Stop of 'ora.gpnpd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'rac1'
CRS-2677: Stop of 'ora.gipcd' on 'rac1' succeeded
CRS-2673: Attempting to stop 'ora.mdnsd' on 'rac1'
CRS-2677: Stop of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac1'
CRS-2676: Start of 'ora.ctssd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac1'
CRS-2676: Start of 'ora.crsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac1'
CRS-2676: Start of 'ora.evmd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.DATA1.dg' on 'rac1'
CRS-2676: Start of 'ora.DATA1.dg' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac1'
CRS-2676: Start of 'ora.registry.acfs' on 'rac1' succeeded

rac1 2010/07/01 14:23:43 /u02/app/11.2.0.1/grid/cdata/rac1/backup_20100701_142343.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 4349 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[root@rac1 ~]#

On node 2:

[root@rac2 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.

Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
[root@rac2 ~]# /u02/app/11.2.0.1/grid/root.sh
Running Oracle 11g root.sh script...

The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u02/app/11.2.0.1/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...


Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
Now product-specific root actions will be performed.
2010-07-01 14:24:58: Parsing the host name
2010-07-01 14:24:58: Checking for super user privileges
2010-07-01 14:24:58: User has super user privileges
Using configuration parameter file: /u02/app/11.2.0.1/grid/crs/install/crsconfig_params
Creating trace directory
LOCAL ADD MODE
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Adding daemon to inittab
CRS-4123: Oracle High Availability Services has been started.
ohasd is starting
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac2'
CRS-2676: Start of 'ora.mdnsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'rac2'
CRS-2676: Start of 'ora.gipcd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac2'
CRS-2676: Start of 'ora.gpnpd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac2'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac2'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac2'
CRS-2676: Start of 'ora.diskmon' on 'rac2' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.ctssd' on 'rac2'
CRS-2676: Start of 'ora.ctssd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.drivers.acfs' on 'rac2'
CRS-2676: Start of 'ora.drivers.acfs' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'rac2'
CRS-2676: Start of 'ora.asm' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'rac2'
CRS-2676: Start of 'ora.crsd' on 'rac2' succeeded
CRS-2672: Attempting to start 'ora.evmd' on 'rac2'
CRS-2676: Start of 'ora.evmd' on 'rac2' succeeded

rac2 2010/07/01 14:28:34 /u02/app/11.2.0.1/grid/cdata/rac2/backup_20100701_142834.olr
Preparing packages for installation...
cvuqdisk-1.0.7-1
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Updating inventory properties for clusterware
Starting Oracle Universal Installer...

Checking swap space: must be greater than 500 MB. Actual 5023 MB Passed
The inventory pointer is located at /etc/oraInst.loc
The inventory is located at /u01/app/oraInventory
'UpdateNodeList' was successful.
[root@rac2 ~]#


Then comeback and click ok on root.sh screen and the post install will continue.













Click Ok on teh above screens as cluster is up and running, you can check that with the following commands:

[oracle@rac1 bin]$ ./olsnodes
rac1
rac2
[oracle@rac1 bin]$


[oracle@rac2 bin]$ ./olsnodes
rac1
rac2
[oracle@rac2 bin]$

and then close the installer after that on teh following screen.




At this point Grid infrastructure is all installed and we are ready to install the database software. But
here I would like to setup profile command to set the environment

Add the following line to the end of /etc/oratab on both the nodes:
crs:/u02/app/11.2.0.1/grid:N

Execute the following commands on both the nodes
mkdir -p /u01/app/oracle/dbm/ora_rac
mkdir -p ${HOME}/dbasetup
cd ${HOME}/dbasetup
touch dbaalias

Change directory to home directory by executing "cd" and
replace the contents of .bash_profile with the following:

#############################################################

# ============================== RAC Environment Settings ================================
# ----- Set the default ORA_CRS_HOME, ORACLE_BASE and ORACLE_HOME values -----
export ORACLE_BASE=/u01/app/oracle
export ORACLE_HOME= /u01/app/oracle/product/11.2.0.1/dbhome_1
export ORA_CRS_HOME=/u02/app/11.2.0.1/grid
export CRS_HOME=${ORA_CRS_HOME}
export NLS_LANG=AMERICAN_AMERICA.AL32UTF8

# Default path settings.
export export PATH=/usr/local/bin:/sbin:/usr/sbin:$PATH:${ORACLE_BASE}/dbm/ora_rac:.

# Reference commands only:
# export SRVM_TRACE=true ; # only when server control trace is required
# export PATH=${ORACLE_HOME}/Opatch:${PATH} ; # only during patches

# ============================ End of RAC Environment Settings ================================

# Default oracle environment settings.
export ORACLE_SID=crs
export ORAENV_ASK=NO
#--- comment out the following line before the first oracle install ---
#. /usr/local/bin/oraenv
export ORAENV_ASK=YES

# Default editor.
export EDITOR=vi

# Character settings for reflections.
if tty -s
then
stty intr ^C
stty kill ^U
stty erase ^H
set -o vi
fi

# History will list last 200 commands issued.
history=200

# Default file creation mask
umask 0022

# Set platform-specific variables
UNIXNAME=`uname -a awk '{ print $1 }' `
case $UNIXNAME in
AIX)
export TNS_ADMIN=/etc
export AIXTHREAD_SCOPE=S
ulimit -m 2045680
;;
Linux)
export TNS_ADMIN=/etc/
;;
SunOS)
export TNS_ADMIN=/var/opt/oracle
;;
esac

# Set the hostname.
HOST=`hostname`

# Set the 2 line prompt.
PS1='
'`whoami`' : ${HOST} : @${ORACLE_SID} : $PWD
\$ '

# Call the dba alias for all other aliases and such required for dba productivity.
. ${HOME}/dbasetup/dbaalias


From node 1:

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ OFFLINE OFFLINE
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type 0/5 0/0 OFFLINE OFFLINE
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 OFFLINE OFFLINE
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$

1.1 Enable GSD
The Global Services Daemon (GSD) was started automatically with previous versions of Oracle but after the install, you’ll find the “gsd” processes offline. These processes are stated by Oracle documentation as coordinating requests between the cluster manager and other processes. It’s actually possible that this is enabled after you create your first database. If not, enable it like so:
From node 1 execute following:
srvctl enable nodeapps
srvctl start nodeapps

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ srvctl enable nodeapps
PRKO-2415 : VIP is already enabled on node(s): rac1,rac2
PRKO-2416 : Network resource is already enabled.
PRKO-2417 : ONS is already enabled on node(s): rac1,rac2
PRKO-2418 : eONS is already enabled on node(s): rac1,rac2


oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ srvctl start nodeapps
PRKO-2421 : Network resource is already started on node(s): rac1,rac2
PRKO-2420 : VIP is already started on node(s): rac1,rac2
PRKO-2420 : VIP is already started on node(s): rac1,rac2
PRKO-2422 : ONS is already started on node(s): rac1,rac2
PRKO-2423 : eONS is already started on node(s): rac1,rac2


oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ ONLINE ONLINE rac1
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.oc4j ora.oc4j.type 0/5 0/0 OFFLINE OFFLINE
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$

For now I will remove oc4j, if you need it later its easy to add it.
From node 1 execute:
srvctl remove oc4j

There’s no output from this command, so nothing to show in a sample output.

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ora.DATA1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ ONLINE ONLINE rac1
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1


Now I will stop all the machines and take the backup of them and restart them all and then I will run asmca to add new diskgroup and more disks to DATA1 and then will install the database software.
Remember:
Cherish in your consciousness the One who created you as He alone shall go along with you