Thursday, June 30, 2011
Part 34 of series - Create standby database 11gR2 from 3 node RAC primary to 2 node RAC standby
He sleeps, intoxicated by corruption and Maya; he does not come to realize or understand. Seizing him by the hair, the Messenger of Death pulls him up; then, he comes to his senses. 1 Those who are attached to the poison of greed and sin grab at the wealth of others; they only bring pain on themselves. They are intoxicated by their pride in those things which shall be destroyed in an instant; those demons do not understand. 1 Pause The Vedas, the Shaastras and the holy men proclaim it, but the deaf do not hear it. When the game of life is over, and he has lost, and he breathes his last, then the fool regrets and repents in his mind. 2 He paid the fine, but it is in vain -- in the Court of the Lord, his account is not credited. Those deeds which would have covered him -- those deeds, he has not done. 3 The Guru has shown me the world to be thus; I sing the Kirtan of the Praises of the One Lord. Renouncing his pride in strength and cleverness, Nanak has come to the Lord's Sanctuary.
Index of all the posts of Gurpartap Singh's Blog
Creating a Physical Standby
The following names will be used when building our configuration:
Primary Site:
rac1 is hostname ,simar database name and is Primary Node 1 and instance name is simar1 and tns alias is simar1
rac2 is hostname ,simar database name and is Primary Node 2 and instance name is simar2 and tns alias is simar2
rac3 is hostname ,simar database name and is Primary Node 3 and instance name is simar3 and tns alias is simar3
Service simarsv1 is configured to run on node 1.
ASM diskgroups are: +DATA1 and +RECV1 having luns in san named "san".
Standby Site:
dgrac1 is hostname,simar database name and is Primary Node 1 and instance name is simarh1
dgrac2 is hostname,simar database name and is Primary Node 2 and instance name is simarh2
simarh is the default service that will be used to transfer the logs to the standby site from the primay site.
ASM diskgroups are: +DATADG1 and +RECVDG1 having luns in san named "san1".
Though you can let RMAN create the standby database and all the directory structures for you but, I don't like that and want to have
control on the process and will create the standby database myself. That will create all the directory structures required and then
will delelte all the datafiles, redolog files and controlfiles. We will be left with just spfile and the structure of the database
athat will be regeistered with the cluster as well.
So, you can create a new SAN by looking at post-3, this time name this san as "san1". Now create luns on it and then install grid
and database binaries on it. Then create a database simarh on this infrastructure by looking at posts 4 to 7. Just note to
keep the sys password same while creating the database OR you will have to modify the password file later.
Now that the database is ready. Take its backup if you want. Then stop the database and delete all the datafiles of this new database
simarh and then all the redolog files and all the controlfiles.
Now lets do some of the checks as:
1. Enable archiving on primary database if its not already enabled.
2. Create a password file (in our case its already there) as dbca created it for us and keep sys password same on both the sites and
set remote_login_passwordfile=exclusive on both the sites.
3. Enable force logging on primary(It's optional but I think it should be used).
alter database force logging;
Output on primary is as:
SQL> alter database force logging;
Database altered.
SQL>
4. Create standby redo logs on primay site so that after rman duplicate they exist on standby site as well.
Some of the protection modes like maximum protection and maximum availability, require standby redo logs to be present.
The size of the standby redo logs must match exactly with the size of online redo logs. We should have one more standby
redo log file group than the number of online redo log file groups on the primary database.
The following syntax is used to create the standby redo logs:
alter database add standby logfile thread 1 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 1 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 1 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 1 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 1 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 1 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 1 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 2 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 2 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 2 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 2 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 2 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 2 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 2 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 3 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 3 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 3 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 3 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 3 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 3 ('+RECV1') SIZE 50M;
alter database add standby logfile thread 3 ('+RECV1') SIZE 50M;
I will create 21 standby redologs on primary with 6 logs per thread.
5. Configure the primary database initialization parameters.
After considering the future role transitions we should configure the parameters to control log transport services and apply
services. If we do that the database will seamlessly work in any role without any parameter modifications. When the database
is running on primay controlfile , standby parameters are not read by it and will not effect the database operation when it will
be in primary role.
Place the following parameters in the primary database parameter file.
######## Primary Role Parameters ############
LOG_ARCHIVE_DEST_1=
'LOCATION=+RECV1
VALID_FOR=(ALL_LOGFILES,ALL_ROLES)
DB_UNIQUE_NAME=simar'
LOG_ARCHIVE_DEST_2=
'SERVICE=simarh
VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)
DB_UNIQUE_NAME=simarh'
LOG_ARCHIVE_DEST_STATE_1=ENABLE
LOG_ARCHIVE_DEST_STATE_2=DEFER
DB_UNIQUE_NAME=simar
SERVICE_NAMES=simar
LOG_ARCHIVE_CONFIG='DG_CONFIG=(simar,simarh)'
## Standby Role Parameters ##
DB_FILE_NAME_CONVERT=
('+DATADG1','+DATA')
LOG_FILE_NAME_CONVERT=
('+RECVDG1','+RECV')
STANDBY_FILE_MANAGEMENT=AUTO
FAL_SERVER=simarh ## Tnsentry_name
FAL_CLIENT=simar ## Tnsentry_name
Paramters DB_FILE_NAME_CONVERT and the LOG_FILE_NAME_CONVERT are required only if the paths of datafiles and logfiles are not identical between
the primary and the standby host.
For now we will keep LOG_ARCHIVE_DEST_2 as 'DEFER' but later once the standby database is created and mounted we will change it
to 'ENABLE'.
After all the modifications my pfile on standby looks like:
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/11.2.0.1/dbhome_1/dbs
$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Wed Jun 29 22:54:06 2011
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> create pfile='/u01/app/oracle/product/11.2.0.1/dbhome_1/dbs/temp_simar1.ora' from spfile='+DATA1/simar/spfilesimar.ora';
File created.
SQL> !cat /u01/app/oracle/product/11.2.0.1/dbhome_1/dbs/temp_simar1.ora
simar3.__db_cache_size=369098752
simar2.__db_cache_size=440401920
simar1.__db_cache_size=377487360
simar3.__java_pool_size=4194304
simar2.__java_pool_size=4194304
simar1.__java_pool_size=4194304
simar3.__large_pool_size=4194304
simar2.__large_pool_size=8388608
simar1.__large_pool_size=8388608
simar1.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
simar3.__pga_aggregate_target=226492416
simar2.__pga_aggregate_target=226492416
simar1.__pga_aggregate_target=226492416
simar3.__sga_target=905969664
simar2.__sga_target=905969664
simar1.__sga_target=905969664
simar3.__shared_io_pool_size=0
simar2.__shared_io_pool_size=0
simar1.__shared_io_pool_size=0
simar3.__shared_pool_size=515899392
simar2.__shared_pool_size=440401920
simar1.__shared_pool_size=503316480
simar3.__streams_pool_size=0
simar2.__streams_pool_size=0
simar1.__streams_pool_size=0
*.aq_tm_processes=2
*.audit_file_dest='/u01/app/oracle/admin/simar/adump'
*.audit_trail='db'
*.cluster_database=TRUE
*.compatible='11.2.0.0.0'
*.control_files='+RECV1/simar/controlfile/current.264.734044335'
*.db_block_size=8192
*.db_create_file_dest='+DATA1'
*.db_create_online_log_dest_1='+RECV1'
*.db_domain=''
*.db_name='simar'
*.DB_UNIQUE_NAME='simar'
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=simarXDB)'
simar2.instance_number=2
simar3.instance_number=3
simar1.instance_number=1
*.LOG_ARCHIVE_CONFIG='DG_CONFIG=(simar,simarh)'
*.log_archive_dest_1='LOCATION=+RECV1 VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=simar'
*.log_archive_dest_2='SERVICE=simarh LGWR ASYNC VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=simarh'
*.LOG_ARCHIVE_DEST_STATE_1='ENABLE'
*.log_archive_dest_state_2='DEFER'
*.log_archive_format='%t_%s_%r.arc'
*.open_cursors=300
*.pga_aggregate_target=225443840
*.processes=150
*.query_rewrite_enabled='TRUE'
*.recyclebin='OFF'
*.remote_listener='scan.rac.meditate.com:1521'
*.remote_login_passwordfile='exclusive'
simar1.service_names='SIMARSV1','SIMAR'
*.sga_max_size=905969664
*.sga_target=905969664
simar2.thread=2
simar3.thread=3
simar1.thread=1
simar1.undo_tablespace='UNDOTBS1'
simar3.undo_tablespace='UNDOTBS3'
simar2.undo_tablespace='UNDOTBS2'
SQL>
6. Create a standby controlfile on the primary database as:
backup current controlfile for standby ;
Output is as:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/11.2.0.1/dbhome_1/dbs
$ rman
Recovery Manager: Release 11.2.0.1.0 - Production on Wed Jun 29 22:56:32 2011
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
RMAN> connect target /
connected to target database: SIMAR (DBID=668825007)
RMAN> backup current controlfile for standby ;
Starting backup at 29-JUN-11
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=58 instance=simar1 device type=DISK
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
including standby control file in backup set
channel ORA_DISK_1: starting piece 1 at 29-JUN-11
channel ORA_DISK_1: finished piece 1 at 29-JUN-11
piece handle=/u01/app/oracle/product/11.2.0.1/dbhome_1/dbs/15mg4ppm_1_1 tag=TAG20110629T225654 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time: 00:00:04
Finished backup at 29-JUN-11
Starting Control File and SPFILE Autobackup at 29-JUN-11
piece handle=/home/oracle/backup/c-668825007-20110629-01 comment=NONE
Finished Control File and SPFILE Autobackup at 29-JUN-11
RMAN> exit
Recovery Manager complete.
oracle : rac1.rac.meditate.com : @simar1 : /u02/app/11.2.0.1/grid/network/admin
$
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
7. Modify the initialization parameter parameter file on the standby site as following:
Create a pfile from spfile on ASM on the standby site and add the following parameters to that pfile for use.
create pfile= '/database/10gDR/backup/initNashville.ora' from spfile;
Once you have a pfile created for your standby, you will need to adjust several parameters. Below are parameters that needed to
be modified in our configuration:
control_files='+DATADG','+RECVDG'
LOG_ARCHIVE_CONFIG='DG_CONFIG=(simarh,simar)'
LOG_ARCHIVE_DEST_1=
'LOCATION=+RECVDG
VALID_FOR=(ALL_LOGFILES,ALL_ROLES)
DB_UNIQUE_NAME=simarh'
LOG_ARCHIVE_DEST_2=
'SERVICE=simarsv1
VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE)
DB_UNIQUE_NAME=simar'
LOG_ARCHIVE_DEST_STATE_1=ENABLE
LOG_ARCHIVE_DEST_STATE_2=ENABLE
DB_UNIQUE_NAME=simarh
SERVICE_NAMES=simarh
## Standby Role Parameters ##
DB_FILE_NAME_CONVERT=
('+DATA','+DATADG1')
LOG_FILE_NAME_CONVERT=
('+RECV','+RECVDG1')
STANDBY_FILE_MANAGEMENT=AUTO
FAL_SERVER=simar ## Tnsentry_name
FAL_CLIENT=simarh ## Tnsentry_name
My spfile on standby site looks like:
oracle : dgrac01.rac.meditate.com : @simarh1 : /u01/app/oracle/product/11.2.0.1/dbhome_1/dbs
$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Wed Jun 29 23:16:21 2011
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> create pfile='/u01/app/oracle/product/11.2.0.1/dbhome_1/dbs/simarh_temp2.ora' from SPFILE='+DATADG1/simarh/spfilesimarh.ora';
File created.
SQL> !cat /u01/app/oracle/product/11.2.0.1/dbhome_1/dbs/simarh_temp2.ora
simarh.__db_cache_size=318767104
simarh2.__db_cache_size=247463936
simarh1.__db_cache_size=213909504
simarh2.__java_pool_size=4194304
simarh1.__java_pool_size=4194304
simarh.__large_pool_size=4194304
simarh2.__large_pool_size=4194304
simarh1.__large_pool_size=4194304
simarh.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
simarh1.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
simarh2.__pga_aggregate_target=159383552
simarh1.__pga_aggregate_target=159383552
simarh2.__sga_target=473956352
simarh1.__sga_target=473956352
simarh2.__shared_io_pool_size=0
simarh1.__shared_io_pool_size=0
simarh2.__shared_pool_size=209715200
simarh1.__shared_pool_size=243269632
simarh2.__streams_pool_size=0
simarh1.__streams_pool_size=0
*.audit_file_dest='/u01/app/oracle/admin/simarh/adump'
*.audit_trail='db'
*.cluster_database=TRUE
*.compatible='11.2.0.0.0'
*.control_files='+DATADG1/simarh/controlfile/current.263.755135005','+RECVDG1/simarh/controlfile/current.272.755135005'#Set by RMAN
*.db_block_size=8192
*.db_create_file_dest='+DATADG1'
*.db_domain='rac.meditate.com'
*.db_file_name_convert='+RECV','+RECVDG1'
*.db_name='simar'
*.db_recovery_file_dest='+RECVDG1'
*.db_recovery_file_dest_size=4070572032
*.db_unique_name='SIMARH'
*.diagnostic_dest='/u01/app/oracle'
*.dispatchers='(PROTOCOL=TCP) (SERVICE=simarhXDB)'
*.fal_client='SIMARH'
*.fal_server='SIMARSV1'
simarh1.instance_number=1
simarh2.instance_number=2
*.LOG_ARCHIVE_CONFIG='DG_CONFIG=(simarh,simar)'
*.LOG_ARCHIVE_DEST_1='LOCATION=+RECVDG1/ VALID_FOR=(ALL_LOGFILES,ALL_ROLES) DB_UNIQUE_NAME=simarh'
*.LOG_ARCHIVE_DEST_2='SERVICE=simarsv1 VALID_FOR=(ONLINE_LOGFILES,PRIMARY_ROLE) DB_UNIQUE_NAME=simar'
*.LOG_ARCHIVE_DEST_STATE_1='ENABLE'
*.LOG_ARCHIVE_DEST_STATE_2='DEFER'
*.log_archive_format='%t_%s_%r.arc'
*.log_file_name_convert='+RECV','+RECVDG'
*.open_cursors=300
*.pga_aggregate_target=157286400
*.processes=150
*.remote_listener='dgscan.rac.meditate.com:1521'
*.remote_login_passwordfile='exclusive'
*.service_names='simarh.rac.meditate.com,simarh'
*.sga_target=471859200
*.standby_file_management='AUTO'
simarh2.thread=2
simarh1.thread=1
simarh1.undo_tablespace='UNDOTBS1'
simarh2.undo_tablespace='UNDOTBS2'
SQL>
8. Configure Oracle Network components as:
To get the changes from primary site to the standby site oracle dataguard relies on oracle net transportation mechanism.
In order to support dataguard communication needs we need to configure listners.ora and tnsnames.ora on both the standby and
the primary site. Specifically we should configure the following:
a. A listener running on the primary hosts, that will be there i.e. our grid listener on all the three nodes on primay site with a static listener entry
b. A static listener entry on all standby hosts for default grid listener.
Output of step b is as:
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Output from standby node 1 is as:
oracle : dgrac01.rac.meditate.com : @crs : /home/oracle
$ echo $ORACLE_HOME
/u02/app/11.2.0.1/grid
oracle : dgrac01.rac.meditate.com : @crs : /home/oracle
$ cd $ORACLE_HOME/network/admin
oracle : dgrac01.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$ cat listener.ora
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
SID_LIST_LISTENER=
(SID_LIST=
(SID_DESC=(GLOBAL_DBNAME = simarh) (SID_NAME=simarh1) (ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1))
)
SID_LIST_LISTENER_SCAN1=
(SID_LIST=
(SID_DESC=(GLOBAL_DBNAME = simarh) (SID_NAME=simarh1) (ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1))
)
ADR_BASE_LISTENER_DGRAC01 = /u01/app/oracle
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent
oracle : dgrac01.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$
srvctl stop listener -n `hostname -s`
srvctl start listener -n `hostname -s`
srvctl stop scan_listener
srvctl start scan_listener
lsnrctl status
and output is as:
oracle : dgrac01.rac.meditate.com : @crs : /home/oracle/gurpartap
$ srvctl stop listener -n `hostname -s`
oracle : dgrac01.rac.meditate.com : @crs : /home/oracle/gurpartap
$ srvctl start listener -n `hostname -s`
oracle : dgrac01.rac.meditate.com : @crs : /home/oracle/gurpartap
oracle : dgrac01.rac.meditate.com : @crs : /home/oracle/gurpartap
$ srvctl stop scan_listener
oracle : dgrac01.rac.meditate.com : @crs : /home/oracle/gurpartap
$ srvctl start scan_listener
oracle : dgrac01.rac.meditate.com : @crs : /home/oracle/gurpartap
$
$ lsnrctl status
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 02-JUL-2011 20:03:30
Copyright (c) 1991, 2009, Oracle. All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date 01-JUL-2011 00:31:39
Uptime 1 days 19 hr. 31 min. 51 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /etc/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/dgrac01/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.216)(PORT=1521)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.2.190)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "SIMARH.rac.meditate.com" has 1 instance(s).
Instance "simarh1", status READY, has 1 handler(s) for this service...
Service "simarh" has 1 instance(s).
Instance "simarh1", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully
oracle : dgrac01.rac.meditate.com : @crs : /home/oracle/gurpartap
$
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Output from standby node 2 is as:
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$ echo $ORACLE_HOME
/u02/app/11.2.0.1/grid
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$ cd $ORACLE_HOME/network/admin
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$ cat listener.ora
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent
SID_LIST_LISTENER=
(SID_LIST=
(SID_DESC=(GLOBAL_DBNAME = simarh) (SID_NAME=simarh2) (ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1))
)
SID_LIST_LISTENER_SCAN1=
(SID_LIST=
(SID_DESC=(GLOBAL_DBNAME = simarh) (SID_NAME=simarh2) (ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1))
)
ADR_BASE_LISTENER_DGRAC02 = /u01/app/oracle
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$
Output at starting this listener is as:
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$ ps -ef grep pmon
oracle 12063 1 0 21:27 ? 00:00:00 asm_pmon_+ASM2
oracle 12885 1 0 21:32 ? 00:00:00 ora_pmon_BAL2
oracle 21439 1 0 22:29 ? 00:00:00 ora_pmon_simarh2
oracle 21591 8968 0 22:30 pts/0 00:00:00 grep pmon
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$
Now execute the following commands on node 2.
srvctl stop listener -n `hostname -s`
srvctl start listener -n `hostname -s`
srvctl stop scan_listener
srvctl start scan_listener
lsnrctl status
Output is as:
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$ srvctl stop listener -n `hostname -s`
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$ srvctl start listener -n `hostname -s`
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$ srvctl stop scan_listener
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$ srvctl start scan_listener
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$ lsnrctl status
LSNRCTL for Linux: Version 11.2.0.1.0 - Production on 01-JUL-2011 00:19:10
Copyright (c) 1991, 2009, Oracle. All rights reserved.
Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias LISTENER
Version TNSLSNR for Linux: Version 11.2.0.1.0 - Production
Start Date 01-JUL-2011 00:17:22
Uptime 0 days 0 hr. 1 min. 48 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /etc/listener.ora
Listener Log File /u01/app/oracle/diag/tnslsnr/dgrac02/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=dgrac02.rac.meditate.com)(PORT=1521)))
Services Summary...
Service "+ASM" has 1 instance(s).
Instance "+ASM2", status READY, has 1 handler(s) for this service...
Service "BAL" has 1 instance(s).
Instance "BAL2", status READY, has 1 handler(s) for this service...
Service "BALXDB" has 1 instance(s).
Instance "BAL2", status READY, has 1 handler(s) for this service...
Service "SIMARH.rac.meditate.com" has 1 instance(s).
Instance "simarh2", status READY, has 1 handler(s) for this service...
The command completed successfully
oracle : dgrac02.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid/network/admin
$
On Primay node 1 listener.ora looks like following:
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent
SID_LIST_LISTENER=
(SID_LIST=
(SID_DESC=(GLOBAL_DBNAME = simar) (SID_NAME=simar1) (ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1))
)
SID_LIST_LISTENER_SCAN1=
(SID_LIST=
(SID_DESC=(GLOBAL_DBNAME = simar) (SID_NAME=simar1) (ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1))
)
ADR_BASE_LISTENER_DG2 = /u01/app/oracle
Now execute the following commands on node 1.
srvctl stop listener -n `hostname -s`
srvctl start listener -n `hostname -s`
srvctl stop scan_listener
srvctl start scan_listener
lsnrctl status
On Primay node 2 listener.ora looks like following:
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
SID_LIST_LISTENER=
(SID_LIST=
(SID_DESC=(GLOBAL_DBNAME = simar) (SID_NAME=simar2) (ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1))
)
SID_LIST_LISTENER_SCAN1=
(SID_LIST=
(SID_DESC=(GLOBAL_DBNAME = simar) (SID_NAME=simar2) (ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1))
)
Now execute the following commands on node 2.
srvctl stop listener -n `hostname -s`
srvctl start listener -n `hostname -s`
srvctl stop scan_listener
srvctl start scan_listener
lsnrctl status
On Primay node 3 listener.ora looks like following:
LISTENER_SCAN1=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))) # line added by Agent
LISTENER=(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))) # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER=ON # line added by Agent
ENABLE_GLOBAL_DYNAMIC_ENDPOINT_LISTENER_SCAN1=ON # line added by Agent
SID_LIST_LISTENER=
(SID_LIST=
(SID_DESC=(GLOBAL_DBNAME = simar) (SID_NAME=simar2) (ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1))
)
SID_LIST_LISTENER_SCAN1=
(SID_LIST=
(SID_DESC=(GLOBAL_DBNAME = simar) (SID_NAME=simar2) (ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1))
)
ADR_BASE_LISTENER_DG2 = /u01/app/oracle
Now execute the following commands on node 3.
srvctl stop listener -n `hostname -s`
srvctl start listener -n `hostname -s`
srvctl stop scan_listener
srvctl start scan_listener
lsnrctl status
c. An Oracle Net alias on the primary and standby for both primay and standby databases as:
RAC1ASM =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = +ASM)
)
)
BAL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dgscan.rac.meditate.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = BAL)
)
)
EMREP =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = goldengate)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = emrep)
)
)
SIMAR =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = scan.rac.meditate.com)(PORT = 1522))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = simar)
(UR = A)
)
)
SIMARH =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = dgrac01.rac.meditate.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = simarh)
(UR = A)
)
)
SIMAR3 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = scan.rac.meditate.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = simar3)
)
)
SIMAR2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = scan.rac.meditate.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = simar2)
)
)
SIMAR1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = scan.rac.meditate.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = simarsv1)
)
)
SIMARSV1 =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = scan.rac.meditate.com)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = simarsv1)
(FAILOVER_MODE =
(TYPE = select)
(METHOD = basic)
)
)
)
9. Start the standby database in nomount state.
Output is as:
oracle : dgrac01.rac.meditate.com : @simarh1 : /u02/app/11.2.0.1/grid
$ srvctl start database -d simarh -o nomount
oracle : dgrac01.rac.meditate.com : @simarh1 : /u02/app/11.2.0.1/grid
$
10. Now we are ready to create a standby database.
Finally, we are ready to create the standby database. Do the following to create it:
Onstandby node 1, do the following:
Logon to RMAN prompt and execute the followimg:
connect target sys/gurpartap@simarsv1
connect auxiliary sys/gurpartap@simarh
DUPLICATE TARGET DATABASE FOR STANDBY from active database;
Output is as:
oracle : dgrac01.rac.meditate.com : @simarh1 : /u01/app/oracle/product/11.2.0.1/dbhome_1/network/admin
$ rman
Recovery Manager: Release 11.2.0.1.0 - Production on Mon Jun 27 23:01:37 2011
Copyright (c) 1982, 2009, Oracle and/or its affiliates. All rights reserved.
RMAN> connect target sys/gurpartap@simarsv1
connected to target database: SIMAR (DBID=668825007)
RMAN> connect auxiliary sys/gurpartap@simarh
connected to auxiliary database: SIMAR (not mounted)
RMAN> DUPLICATE TARGET DATABASE FOR STANDBY from active database;
Starting Duplicate Db at 29-JUN-11
using target database control file instead of recovery catalog
allocated channel: ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: SID=32 instance=simarh1 device type=DISK
contents of Memory Script:
{
backup as copy reuse
targetfile '/u01/app/oracle/product/11.2.0.1/dbhome_1/dbs/orapwsimar1' auxiliary format
'/u01/app/oracle/product/11.2.0.1/dbhome_1/dbs/orapwsimarh1' ;
}
executing Memory Script
Starting backup at 29-JUN-11
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=58 instance=simar1 device type=DISK
Finished backup at 29-JUN-11
contents of Memory Script:
{
sql clone "alter system set control_files =
''+DATADG1/simarh/controlfile/current.296.755135005'', ''+RECVDG1/simarh/controlfile/current.290.755135005'' comment=
''Set by RMAN'' scope=spfile";
backup as copy current controlfile for standby auxiliary format '+DATADG1/simarh/controlfile/current.263.755135005';
restore clone controlfile to '+RECVDG1/simarh/controlfile/current.272.755135005' from
'+DATADG1/simarh/controlfile/current.263.755135005';
sql clone "alter system set control_files =
''+DATADG1/simarh/controlfile/current.263.755135005'', ''+RECVDG1/simarh/controlfile/current.272.755135005'' comment=
''Set by RMAN'' scope=spfile";
shutdown clone immediate;
startup clone nomount;
}
executing Memory Script
sql statement: alter system set control_files = ''+DATADG1/simarh/controlfile/current.296.755135005'', ''+RECVDG1/simarh/controlfile/current.290.755135005'' comment= ''Set by RMAN'' scope=spfile
Starting backup at 29-JUN-11
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
copying standby control file
output file name=/u01/app/oracle/product/11.2.0.1/dbhome_1/dbs/snapcf_simar1.f tag=TAG20110629T234326 RECID=4 STAMP=755135028
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:45
Finished backup at 29-JUN-11
Starting restore at 29-JUN-11
using channel ORA_AUX_DISK_1
channel ORA_AUX_DISK_1: copied control file copy
Finished restore at 29-JUN-11
sql statement: alter system set control_files = ''+DATADG1/simarh/controlfile/current.263.755135005'', ''+RECVDG1/simarh/controlfile/current.272.755135005'' comment= ''Set by RMAN'' scope=spfile
Oracle instance shut down
connected to auxiliary database (not started)
Oracle instance started
Total System Global Area 471830528 bytes
Fixed Size 2214456 bytes
Variable Size 251659720 bytes
Database Buffers 213909504 bytes
Redo Buffers 4046848 bytes
contents of Memory Script:
{
sql clone 'alter database mount standby database';
}
executing Memory Script
sql statement: alter database mount standby database
contents of Memory Script:
{
set newname for tempfile 1 to
"+datadg1";
switch clone tempfile all;
set newname for datafile 1 to
"+datadg1";
set newname for datafile 2 to
"+datadg1";
set newname for datafile 3 to
"+datadg1";
set newname for datafile 4 to
"+datadg1";
set newname for datafile 5 to
"+datadg1";
set newname for datafile 6 to
"+datadg1";
set newname for datafile 7 to
"+datadg1";
set newname for datafile 8 to
"+datadg1";
backup as copy reuse
datafile 1 auxiliary format
"+datadg1" datafile
2 auxiliary format
"+datadg1" datafile
3 auxiliary format
"+datadg1" datafile
4 auxiliary format
"+datadg1" datafile
5 auxiliary format
"+datadg1" datafile
6 auxiliary format
"+datadg1" datafile
7 auxiliary format
"+datadg1" datafile
8 auxiliary format
"+datadg1" ;
sql 'alter system archive log current';
}
executing Memory Script
executing command: SET NEWNAME
renamed tempfile 1 to +datadg1 in control file
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
executing command: SET NEWNAME
Starting backup at 29-JUN-11
using channel ORA_DISK_1
channel ORA_DISK_1: starting datafile copy
input datafile file number=00002 name=+DATA1/simar/datafile/sysaux.273.743406845
output file name=+DATADG1/simarh/datafile/sysaux.267.755135281 tag=TAG20110629T234737
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:10:07
channel ORA_DISK_1: starting datafile copy
input datafile file number=00001 name=+DATA1/simar/datafile/system.267.743406749
output file name=+DATADG1/simarh/datafile/system.266.755135885 tag=TAG20110629T234737
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:02:05
channel ORA_DISK_1: starting datafile copy
input datafile file number=00003 name=+DATA1/simar/datafile/undotbs1.269.743406693
output file name=+DATADG1/simarh/datafile/undotbs1.262.755136011 tag=TAG20110629T234737
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:02:35
channel ORA_DISK_1: starting datafile copy
input datafile file number=00007 name=+DATA1/simar/datafile/undotbs3.262.743406695
output file name=+DATADG1/simarh/datafile/undotbs3.261.755136167 tag=TAG20110629T234737
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:25
channel ORA_DISK_1: starting datafile copy
input datafile file number=00006 name=+DATA1/simar/datafile/undotbs2.266.743406695
output file name=+DATADG1/simarh/datafile/undotbs2.279.755136253 tag=TAG20110629T234737
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:16
channel ORA_DISK_1: starting datafile copy
input datafile file number=00008 name=+DATA1/simar/datafile/goldengate.271.743406751
output file name=+DATADG1/simarh/datafile/goldengate.278.755136327 tag=TAG20110629T234737
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:01:05
channel ORA_DISK_1: starting datafile copy
input datafile file number=00005 name=+DATA1/simar/datafile/example.272.743406751
output file name=+DATADG1/simarh/datafile/example.277.755136393 tag=TAG20110629T234737
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35
channel ORA_DISK_1: starting datafile copy
input datafile file number=00004 name=+DATA1/simar/datafile/users.263.743406851
output file name=+DATADG1/simarh/datafile/users.276.755136429 tag=TAG20110629T234737
channel ORA_DISK_1: datafile copy complete, elapsed time: 00:00:35
Finished backup at 30-JUN-11
sql statement: alter system archive log current
contents of Memory Script:
{
switch clone datafile all;
}
executing Memory Script
datafile 1 switched to datafile copy
input datafile copy RECID=4 STAMP=755136449 file name=+DATADG1/simarh/datafile/system.266.755135885
datafile 2 switched to datafile copy
input datafile copy RECID=5 STAMP=755136450 file name=+DATADG1/simarh/datafile/sysaux.267.755135281
datafile 3 switched to datafile copy
input datafile copy RECID=6 STAMP=755136450 file name=+DATADG1/simarh/datafile/undotbs1.262.755136011
datafile 4 switched to datafile copy
input datafile copy RECID=7 STAMP=755136450 file name=+DATADG1/simarh/datafile/users.276.755136429
datafile 5 switched to datafile copy
input datafile copy RECID=8 STAMP=755136450 file name=+DATADG1/simarh/datafile/example.277.755136393
datafile 6 switched to datafile copy
input datafile copy RECID=9 STAMP=755136450 file name=+DATADG1/simarh/datafile/undotbs2.279.755136253
datafile 7 switched to datafile copy
input datafile copy RECID=10 STAMP=755136450 file name=+DATADG1/simarh/datafile/undotbs3.261.755136167
datafile 8 switched to datafile copy
input datafile copy RECID=11 STAMP=755136450 file name=+DATADG1/simarh/datafile/goldengate.278.755136327
Finished Duplicate Db at 30-JUN-11
RMAN>
Step 11. Begin shipping redo to the standby database.
I hope you remmeber that we had earlier deferred LOG_ARCHIVE_DEST_2 on the primary database until we had the
standby mounted. Now we cam enalbe the second destination to ship the redo logs to standby location.
Here is teh following command to so that:
alter system set log_archive_dest_state_2=enable scope=both;
Output is as:
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/11.2.0.1/dbhome_1/dbs
$ sqlplus / as sysdba
SQL*Plus: Release 11.2.0.1.0 Production on Thu Jun 30 00:12:05 2011
Copyright (c) 1982, 2009, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> alter system set log_archive_dest_state_2=enable scope=both;
System altered.
SQL>
Step 12. Next, perform a log switch on the primary site on all the instances and verify that the transmission of that
log was successful:
alter system switch logfile;
select status,error from v$archive_dest where dest_id=2;
Output is as follows, I executed the following commands and got the following output on all the nodes :
SQL> alter system switch logfile;
System altered.
SQL> select status,error from v$archive_dest where dest_id=2;
STATUS ERROR
--------- -----------------------------------------------------------------
VALID
SQL>
If the transmission was successful, the status of the destination should be valid.
If the status is invalid, investigate the error listed in the error column to correct any issues.
Step 13. Now start the recovery on the standby site on node 1 using the following command :
alter database recover managed standby database disconnect from session;
Output is as:
SQL> alter database recover managed standby database disconnect from session;
Database altered.
SQL>
Now check if the logs are being applied or not using following commands:
Primary on any node execute the following command:
select max(sequence#),THREAD# from v$archived_log group by THREAD#;
Output is:
SQL> select max(sequence#),THREAD# from v$archived_log group by THREAD#;
MAX(SEQUENCE#) THREAD#
-------------- ----------
1178 1
934 2
967 3
SQL>
Standby:
select max(sequence#),THREAD# from v$archived_log where applied = 'YES' group by THREAD#;
Output is:
SQL> select max(sequence#),THREAD# from v$archived_log where applied = 'YES' group by THREAD#;
MAX(SEQUENCE#) THREAD#
-------------- ----------
1175 1
929 2
964 3
SQL>
If the no. on standby are increasing it means the logs are being applied otherwise look at the alertlogs to
investigate the error and fix it.
Meditate on the Lord, the Lord of the Universe. Cherish the Beloved Lord, Har, Har, in your mind. The Guru says to install it in your consciousness. Turn away from others, and turn to Him. Thus you shall obtain your Beloved, O my companion. 1 Pause In the pool of the world is the mud of attachment. Stuck in it, the his feet cannot walk towards the Lord. The fool is stuck; he cannot do anything else. Only by entering the Lord’s Sanctuary, O my companion, will you be released. 1
Thus your consciousness shall be stable and steady and firm. Wilderness and household are the same. Deep within dwells the One Husband Lord; outwardly, there are many distractions. Practice Raja Yoga, the Yoga of meditation and success. Says
Nanak, this is the way to dwell with the people, and yet remain apart from them.
Thursday, March 3, 2011
Part 33 of series - Script to monitor Goldengate extract/replicat process status & get the email alert or the remedy ticket if any process is ABENDED
O Destroyer of the pains and the suffering of the poor, O Master of each and every heart, O Masterless One: I have come seeking Your Sanctuary. O God, please be with Nanak!
Where there is no mother, father, children, friends or siblings - O my
mind, there, only the Naam, the Name of the Lord, shall be with you as your help and support. Where the great and terrible
Messenger of Death shall try to crush you, there, only the Naam shall go along with you. Where the obstacles are so very
heavy, the Name of the Lord shall rescue you in an instant. By performing countless religious rituals, you shall not be saved.
The Name of the Lord washes off millions of sins. As Gurmukh, chant the Naam, O my mind.
Index of all the posts of Gurpartap Singh's Blog
Here is the script that can be used to monitor the goldengate extract/replicat process. It will email you to the specified
email address or you can configure remedy to cut the ticket by letting its agent read the logfile.
This script will send the email for every extract that has a status of "ABENDED". Then you should go in and look at the
problem that caused it and fix it and restart it. You can schedule this script in crontab for every 30 minutes or as
much frequency as you want.
If you are using it on a home network this script will send email only if you have configured email. On my home network
I have configured email using the smtp of gmail.com, if you have not setup email on your server you can use the following note
to get it configured.
Post 27 of series, Setup email on unix server using SMTP of gmail.com(Google Mail) to get alerts from scripts.
Create the following script at your scripts directory. I have it at:
/u01/app/oracle/scripts
Create a log directory where you want to place your logs. I am placing them here:
/u01/app/oracle/product/log
You just need to fix the following parameter of this script as per your environment:
===========================================================
GG_HOME=/u01/app/oracle/product/goldengate; export GG_HOME
GG_BASE="/u01/app/oracle/product"
NOTIFY="xxxxx@yahoo.com"
LOG="${GG_BASE}/log/ggs_monitor_${DT}.log"
===========================================================
Here starts the script:
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$ cat goldengate_monitor.sh
#!/bin/ksh
#
# Created by : Gurpartap Singh
# Date: March 03, 2011
#
# Purpose: Monitoring script for Golden Gate processes
#
# Notes: At the very least, the "Setup variables" section will need to be modified to
# reflect your environment.
#
if test "${1}" = "RBA"
then
CHECK_RBA="YES"
else
CHECK_RBA="NO"
fi
# Setup variables:
DT=`date +%d%H%M%S`
GG_HOME=/u01/app/oracle/product/goldengate; export GG_HOME
GG_BASE="/u01/app/oracle/product"
NOTIFY="xxxxxxxx@yahoo.com"
LOG="${GG_BASE}/log/ggs_monitor_${DT}.log"
echo "======================================================================" >> $LOG 2>&1
echo "+ Date: $(date)" >> $LOG 2>&1
echo "+ Script: $0" >> $LOG 2>&1
# Setup sub-routines here.
notfyMe() {
MESG=${1} # argument passed to sub-routine
}
checkReplicat() {
cd $GG_HOME
./ggsci <> $LOG 2>&1
echo "Type: ${TYPE[$x]}" >> $LOG 2>&1
echo "Name: ${NAME[$x]}" >> $LOG 2>&1
echo "Status: ${STATUS[$x]}" >> $LOG 2>&1
echo "Lag: ${LAG[$x]}" >> $LOG 2>&1
echo "Checkpoint: ${CHKPT[$x]}" >> $LOG 2>&1
MYSTATUS="${STATUS[$x]}"
case ${STATUS[$x]} in
"ABENDED") echo "Action: ${NAME[$x]} ABENDED - Sending mail" >> $LOG 2>&1
echo "Group Name = ${NAME[$x]} has ABENDED" /bin/mailx -s "OGG Abend" ${NOTIFY}
;;
"STOPPED") echo "Action: ${NAME[$x]} STOPPED - Sending mail" >> $LOG 2>&1
echo "Group Name = ${NAME[$x]} has been STOPPED" /bin/mailx -s "OGG Stopped" ${NOTIFY}
;;
"RUNNING") echo "Action: None required!" >> $LOG 2>&1
;;
*) echo "Optional case" >> $LOG 2>&1
esac
# Next check for lag
# Continue, even if the process is STOPPED or ABENDED!!!
MYLAG="${LAG[$x]}"
if [[ $MYLAG > "01:00:00" ]]
then
echo "Group Name = ${NAME[$x]} is LAGGING" /bin/mailx -s "OGG Lag" ${NOTIFY}
fi
if [ $CHECK_RBA == "YES" ]
then
# Now check to make sure the RBA numbers are changing... stupid pos!!!
curr=$(checkReplicat ${NAME[$x]}grep -v grepgrep RBAawk {'print $4}')
sleep $delay
next=$(checkReplicat ${NAME[$x]}grep -v grepgrep RBAawk {'print $4}')
if [ $curr -eq $next ]
then
echo "RBA has not change in $delay minutes"
echo "Group Name = ${NAME[$x]} RBA is stuck" /bin/mailx -s "OGG RBA" ${NOTIFY}
fi
fi
let x=x+1
done
exit 0;
oracle : rac1.rac.meditate.com : @crs : /u01/app/oracle/scripts
$
After creating it just change its permisssins to 755 like:
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$ chmod 755 goldengate_monitor.sh
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$ ls -lart
total 12
drwxr-xr-x 11 oracle oinstall 4096 Mar 3 21:57 ..
-rwxr-xr-x 1 oracle oinstall 3157 Mar 3 22:09 goldengate_monitor.sh
drwxr-xr-x 2 oracle oinstall 4096 Mar 3 22:09 .
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$
Now lets execute it as :
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$ goldengate_monitor.sh
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$
You will have an email for each process that the status of "ABENDED".
You can also see the logfiles generated at following location for your each run with time stamp.
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/log
$ ls -lart
total 32
drwxr-xr-x 6 oracle oinstall 4096 Mar 3 21:24 ..
-rw-r--r-- 1 oracle oinstall 1418 Mar 3 22:04 ggs_monitor_03220438.log
-rw-r--r-- 1 oracle oinstall 1418 Mar 3 22:06 ggs_monitor_03220650.log
-rw-r--r-- 1 oracle oinstall 1403 Mar 3 22:18 ggs_monitor_03221828.log
-rw-r--r-- 1 oracle oinstall 1371 Mar 3 22:19 ggs_monitor_03221901.log
-rw-r--r-- 1 oracle oinstall 1418 Mar 3 22:29 ggs_monitor_03222951.log
drwxr-xr-x 2 oracle oinstall 4096 Mar 3 22:29 .
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/log
$
cat on the logfile will look as:
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/log
$ cat ggs_monitor_03222951.log
======================================================================
+ Date: Thu Mar 3 22:29:51 PST 2011
+ Script: ./goldengate_monitor.sh
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: ECREATOR
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1708:36:18
Action: ECREATOR ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: ELORD1
Status: ABENDED
Lag: 00:00:00
Checkpoint: 2663:42:41
Action: ELORD1 ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: EPLACE
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1708:36:27
Action: EPLACE ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: ETRUTH
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1708:36:27
Action: ETRUTH ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: PCREATOR
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1832:31:42
Action: PCREATOR ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: PPLACE
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1832:31:41
Action: PPLACE ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: PTRUTH
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1832:31:43
Action: PTRUTH ABENDED - Sending mail
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/log
$
Have fun and don't forget to do meditation in the name of GOD as thats the only savior in Kaliyuga and pray for me that I do more and more meditation in rememberance of all mighty.
On that path where the miles cannot be
counted, there, the Name of the Lord shall be your sustenance. On that journey of total, pitch-black darkness, the Name of the
Lord shall be the Light with you. On that journey where no one knows you, with the Name of the Lord, you shall be
recognized. Where there is awesome and terrible heat and blazing sunshine, there, the Name of the Lord will give you shade.
Where thirst, O my mind, torments you to cry out, there, O Nanak, the Ambrosial Name, Har, Har, shall rain down upon you
Where there is no mother, father, children, friends or siblings - O my
mind, there, only the Naam, the Name of the Lord, shall be with you as your help and support. Where the great and terrible
Messenger of Death shall try to crush you, there, only the Naam shall go along with you. Where the obstacles are so very
heavy, the Name of the Lord shall rescue you in an instant. By performing countless religious rituals, you shall not be saved.
The Name of the Lord washes off millions of sins. As Gurmukh, chant the Naam, O my mind.
Index of all the posts of Gurpartap Singh's Blog
Here is the script that can be used to monitor the goldengate extract/replicat process. It will email you to the specified
email address or you can configure remedy to cut the ticket by letting its agent read the logfile.
This script will send the email for every extract that has a status of "ABENDED". Then you should go in and look at the
problem that caused it and fix it and restart it. You can schedule this script in crontab for every 30 minutes or as
much frequency as you want.
If you are using it on a home network this script will send email only if you have configured email. On my home network
I have configured email using the smtp of gmail.com, if you have not setup email on your server you can use the following note
to get it configured.
Post 27 of series, Setup email on unix server using SMTP of gmail.com(Google Mail) to get alerts from scripts.
Create the following script at your scripts directory. I have it at:
/u01/app/oracle/scripts
Create a log directory where you want to place your logs. I am placing them here:
/u01/app/oracle/product/log
You just need to fix the following parameter of this script as per your environment:
===========================================================
GG_HOME=/u01/app/oracle/product/goldengate; export GG_HOME
GG_BASE="/u01/app/oracle/product"
NOTIFY="xxxxx@yahoo.com"
LOG="${GG_BASE}/log/ggs_monitor_${DT}.log"
===========================================================
Here starts the script:
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$ cat goldengate_monitor.sh
#!/bin/ksh
#
# Created by : Gurpartap Singh
# Date: March 03, 2011
#
# Purpose: Monitoring script for Golden Gate processes
#
# Notes: At the very least, the "Setup variables" section will need to be modified to
# reflect your environment.
#
if test "${1}" = "RBA"
then
CHECK_RBA="YES"
else
CHECK_RBA="NO"
fi
# Setup variables:
DT=`date +%d%H%M%S`
GG_HOME=/u01/app/oracle/product/goldengate; export GG_HOME
GG_BASE="/u01/app/oracle/product"
NOTIFY="xxxxxxxx@yahoo.com"
LOG="${GG_BASE}/log/ggs_monitor_${DT}.log"
echo "======================================================================" >> $LOG 2>&1
echo "+ Date: $(date)" >> $LOG 2>&1
echo "+ Script: $0" >> $LOG 2>&1
# Setup sub-routines here.
notfyMe() {
MESG=${1} # argument passed to sub-routine
}
checkReplicat() {
cd $GG_HOME
./ggsci <
echo "Type: ${TYPE[$x]}" >> $LOG 2>&1
echo "Name: ${NAME[$x]}" >> $LOG 2>&1
echo "Status: ${STATUS[$x]}" >> $LOG 2>&1
echo "Lag: ${LAG[$x]}" >> $LOG 2>&1
echo "Checkpoint: ${CHKPT[$x]}" >> $LOG 2>&1
MYSTATUS="${STATUS[$x]}"
case ${STATUS[$x]} in
"ABENDED") echo "Action: ${NAME[$x]} ABENDED - Sending mail" >> $LOG 2>&1
echo "Group Name = ${NAME[$x]} has ABENDED" /bin/mailx -s "OGG Abend" ${NOTIFY}
;;
"STOPPED") echo "Action: ${NAME[$x]} STOPPED - Sending mail" >> $LOG 2>&1
echo "Group Name = ${NAME[$x]} has been STOPPED" /bin/mailx -s "OGG Stopped" ${NOTIFY}
;;
"RUNNING") echo "Action: None required!" >> $LOG 2>&1
;;
*) echo "Optional case" >> $LOG 2>&1
esac
# Next check for lag
# Continue, even if the process is STOPPED or ABENDED!!!
MYLAG="${LAG[$x]}"
if [[ $MYLAG > "01:00:00" ]]
then
echo "Group Name = ${NAME[$x]} is LAGGING" /bin/mailx -s "OGG Lag" ${NOTIFY}
fi
if [ $CHECK_RBA == "YES" ]
then
# Now check to make sure the RBA numbers are changing... stupid pos!!!
curr=$(checkReplicat ${NAME[$x]}grep -v grepgrep RBAawk {'print $4}')
sleep $delay
next=$(checkReplicat ${NAME[$x]}grep -v grepgrep RBAawk {'print $4}')
if [ $curr -eq $next ]
then
echo "RBA has not change in $delay minutes"
echo "Group Name = ${NAME[$x]} RBA is stuck" /bin/mailx -s "OGG RBA" ${NOTIFY}
fi
fi
let x=x+1
done
exit 0;
oracle : rac1.rac.meditate.com : @crs : /u01/app/oracle/scripts
$
After creating it just change its permisssins to 755 like:
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$ chmod 755 goldengate_monitor.sh
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$ ls -lart
total 12
drwxr-xr-x 11 oracle oinstall 4096 Mar 3 21:57 ..
-rwxr-xr-x 1 oracle oinstall 3157 Mar 3 22:09 goldengate_monitor.sh
drwxr-xr-x 2 oracle oinstall 4096 Mar 3 22:09 .
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$
Now lets execute it as :
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$ goldengate_monitor.sh
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/scripts
$
You will have an email for each process that the status of "ABENDED".
You can also see the logfiles generated at following location for your each run with time stamp.
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/log
$ ls -lart
total 32
drwxr-xr-x 6 oracle oinstall 4096 Mar 3 21:24 ..
-rw-r--r-- 1 oracle oinstall 1418 Mar 3 22:04 ggs_monitor_03220438.log
-rw-r--r-- 1 oracle oinstall 1418 Mar 3 22:06 ggs_monitor_03220650.log
-rw-r--r-- 1 oracle oinstall 1403 Mar 3 22:18 ggs_monitor_03221828.log
-rw-r--r-- 1 oracle oinstall 1371 Mar 3 22:19 ggs_monitor_03221901.log
-rw-r--r-- 1 oracle oinstall 1418 Mar 3 22:29 ggs_monitor_03222951.log
drwxr-xr-x 2 oracle oinstall 4096 Mar 3 22:29 .
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/log
$
cat on the logfile will look as:
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/log
$ cat ggs_monitor_03222951.log
======================================================================
+ Date: Thu Mar 3 22:29:51 PST 2011
+ Script: ./goldengate_monitor.sh
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: ECREATOR
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1708:36:18
Action: ECREATOR ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: ELORD1
Status: ABENDED
Lag: 00:00:00
Checkpoint: 2663:42:41
Action: ELORD1 ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: EPLACE
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1708:36:27
Action: EPLACE ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: ETRUTH
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1708:36:27
Action: ETRUTH ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: PCREATOR
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1832:31:42
Action: PCREATOR ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: PPLACE
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1832:31:41
Action: PPLACE ABENDED - Sending mail
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Type: EXTRACT
Name: PTRUTH
Status: ABENDED
Lag: 00:00:00
Checkpoint: 1832:31:43
Action: PTRUTH ABENDED - Sending mail
oracle : rac1.rac.meditate.com : @simar1 : /u01/app/oracle/product/log
$
Have fun and don't forget to do meditation in the name of GOD as thats the only savior in Kaliyuga and pray for me that I do more and more meditation in rememberance of all mighty.
On that path where the miles cannot be
counted, there, the Name of the Lord shall be your sustenance. On that journey of total, pitch-black darkness, the Name of the
Lord shall be the Light with you. On that journey where no one knows you, with the Name of the Lord, you shall be
recognized. Where there is awesome and terrible heat and blazing sunshine, there, the Name of the Lord will give you shade.
Where thirst, O my mind, torments you to cry out, there, O Nanak, the Ambrosial Name, Har, Har, shall rain down upon you
Monday, February 14, 2011
Part 32 of series - Integrate Goldengate Manager Process with 11gr2 Grid Infrastructure, use GI to start and stop Mgr Process and so extract/replict
The rulers of the all the world are unhappy; one who chants the Name of the Lord becomes happy. Acquiring hundreds of thousands and millions, your desires shall not be contained. Chanting the Name of the Lord, you shall find release. By the countless pleasures of Maya, your thirst shall not be quenched. Chanting the Name of the Lord, you shall
be satisfied. Upon that path where you must go all alone, there, only the Lord’s Name shall go with you to sustain you.
Index of all the posts of Gurpartap Singh's Blog
Here are the steps I used to add Oracle Goldengate Manager process to Grid Infrastructure. After adding this only Grid Infrastructure should be used to stop/start/relocate Oracle Goldengate Manger Process. This will start/stop manger process and manager process in turn will start all the extracts/replicats using parameter AUTOSTART in manager process parameter file. Like add the following file in the end of the file as:
AUTOSTART extract *truth
This will make all the extracts ending with truth to start once manager process starts. You can configure similar thing for all the porcesses you have for extracts/replicats.
Step 1 is to add a new vip. Here are the steps:
Adding a vip.
/u02/app/11.2.0.1/grid/bin/appvipcfg create -network=1 \
-ip=10.10.2.199 \
-vipname=mvggatevip \
-user=root
with:
• GRID_HOME as the oracle home in which Oracle 11g Release 2 Grid infrastructure
components have been installed (e.g. /u02/app/11.2.0.1/grid).
• -network refers to the network number that you want to use. With Oracle Clusterware
11.2.0.1 you can find the network number using the command:
crsctl stat res -p grep -ie .network -ie subnet grep -ie name -ie subnet
Sample output is:
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/crsctl stat res -p grep -ie .network -ie subnet grep -ie name -ie subnet
NAME=ora.net1.network
USR_ORA_SUBNET=10.10.2.0
[root@rac1 ~]#
net1 in NAME=ora.net1.network indicates this is network 1, and the second line
indicates the subnet on which the VIP will be created.
10.10.2.199 - Is a ip on the above mentioned subnet and which router can support but is not allocated to any other machine and is not pingable as of now. Grid infrastrucure will bring it up and then it will become pingable.
• mvggatevip is the name of the application VIP that you will create.
Oracle recommends the use of the appvipcfg utility to define applications VIPs. The VIP is
created with a set of pre-defined settings and dependencies. Please refer to the Oracle
Clusterware documentation for further details:
http://download.oracle.com/docs/cd/E11882_01/rac.112/e10717/crschp.htm#BGBJHJHC
Oracle White Paper—Oracle GoldenGate high availability with Oracle Clusterware
Step 2:
As root, allow the Oracle Grid infrastructure software owner (e.g. oracle) to run the script
to start the VIP.
/u02/app/11.2.0.1/grid/bin/crsctl setperm resource mvggatevip -u user:oracle:r-x
Step 3:
Then, as oracle, start the VIP:
/u02/app/11.2.0.1/grid/bin/crsctl start resource mvggatevip
To validate whether the VIP is running and on which node it is running, execute:
GRID_HOME/bin/crsctl status resource mvggatevip
For example:
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/crsctl status resource mvggatevip
NAME=mvggatevip
TYPE=app.appvip.type
TARGET=ONLINE
STATE=ONLINE on rac2
[root@rac1 ~]#
At this point you can also connect to another server in the subnet and ping the VIP's IP address.
You should get a reply from this IP address.
oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ ping -c5 10.10.2.199
PING 10.10.2.199 (10.10.2.199) 56(84) bytes of data.
64 bytes from 10.10.2.199: icmp_seq=1 ttl=64 time=0.165 ms
64 bytes from 10.10.2.199: icmp_seq=2 ttl=64 time=0.168 ms
64 bytes from 10.10.2.199: icmp_seq=3 ttl=64 time=0.172 ms
64 bytes from 10.10.2.199: icmp_seq=4 ttl=64 time=0.172 ms
64 bytes from 10.10.2.199: icmp_seq=5 ttl=64 time=0.188 ms
--- 10.10.2.199 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.165/0.173/0.188/0.008 ms
oracle : rac1.rac.meditate.com : @crs : /home/oracle
$
Step 4:
We will use the following script to tell Grid Infrastructure to start and stop manager process I am using ACFS. These steps can also be used for NFS mount points.
Place this script at the shared location where you have installed the goldengate binaries.
I am placing at the following location:
oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
84G 35G 45G 44% /
/dev/sda1 99M 16M 78M 17% /boot
tmpfs 1.6G 429M 1.2G 28% /dev/shm
/dev/asm/shared_vol1-106
3.0G 424M 2.6G 14% /u01/app/oracle/acfsmounts/data1_shared_vol1
oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ cd /u01/app/oracle/acfsmounts/data1_shared_vol1
oracle : rac1.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1
$ cd script
oracle : rac1.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1/script
$ lst
total 12
drwxrwx--- 7 root oinstall 4096 Feb 10 22:32 ..
-rwxr-xr-x 1 oracle oinstall 3278 Feb 12 15:00 11gr2_gg_action.scr
drwxr-xr-x 2 oracle oinstall 4096 Feb 12 15:06 .
oracle : rac1.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1/script
$
oracle : rac1.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1/script
$ cat 11gr2_gg_action.scr
#!/bin/sh
##GGS_HOME=
export GGS_HOME=/u01/app/oracle/acfsmounts/data1_shared_vol1/goldengate
##DBFS_MOUNT_POINT=
##DBFS_FILE_SYSTEM=
##specify delay after start before checking for successful start
start_delay_secs=5
##Include the GoldenGate home in the library path to start GGSCI
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${GGS_HOME}
##set the oracle home to the database to ensure GoldenGate will get the
##right environment settings to be able to connect to the database
export ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1
##check_process validates that a manager process is running at the PID
##that GoldenGate specifies.
check_process () {
if ( [ -f "${GGS_HOME}/dirpcs/MGR.pcm" ] )
then
pid=`cut -f8 "${GGS_HOME}/dirpcs/MGR.pcm"`
if [ ${pid} = `ps -e grep ${pid} grep mgr cut -d " " -f2` ]
then
##manager process is running on the PID . exit success
exit 0
else
if [ ${pid} = `ps -e grep ${pid} grep mgr cut -d " " -f1` ]
then
##manager process is running on the PID . exit success
exit 0
else
##manager process is not running on the PID
exit 1
fi
fi
else
##manager is not running because there is no PID file
exit 1
fi
}
##call_ggsci is a generic routine that executes a ggsci command
call_ggsci () {
ggsci_command=$1
ggsci_output=`${GGS_HOME}/ggsci << start_dependencies="'hard(mvggatevip,ora.asm)" start_dependencies="'hard(mvggatevip)" action_script="/u01/app/oracle/acfsmounts/data1_shared_vol1/script/11gr2_gg_action.scr," check_interval="30," start_dependencies="'hard(mvggatevip,ora.asm)" stop_dependencies="'hard(mvggatevip)'">
(In my case I am using oracle user to do goldengate stuff so its not required)
Step 6:
[root@rac2 ~]# /u02/app/11.2.0.1/grid/bin/crsctl start resource ggateapp -f
CRS-2673: Attempting to stop 'mvggatevip' on 'rac2'
CRS-2677: Stop of 'mvggatevip' on 'rac2' succeeded
CRS-2672: Attempting to start 'mvggatevip' on 'rac3'
CRS-2676: Start of 'mvggatevip' on 'rac3' succeeded
CRS-2672: Attempting to start 'ggateapp' on 'rac3'
CRS-2674: Start of 'ggateapp' on 'rac3' failed
CRS-2679: Attempting to clean 'ggateapp' on 'rac3'
CRS-2681: Clean of 'ggateapp' on 'rac3' succeeded
CRS-2673: Attempting to stop 'mvggatevip' on 'rac3'
CRS-2677: Stop of 'mvggatevip' on 'rac3' succeeded
CRS-2672: Attempting to start 'mvggatevip' on 'rac2'
CRS-2676: Start of 'mvggatevip' on 'rac2' succeeded
CRS-2563: Attempt to start resource 'ggateapp' on 'rac3' has failed. Will re-retry on 'rac2' now.
CRS-2672: Attempting to start 'ggateapp' on 'rac2'
CRS-2676: Start of 'ggateapp' on 'rac2' succeeded
[root@rac2 ~]#
oracle : rac2.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1/script
$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ggateapp clus...esource 0/1 0/0 ONLINE ONLINE rac2
mvggatevip app....ip.type 0/0 0/0 ONLINE ONLINE rac2
ora.DATA1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.OCR1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora.RECV1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ ONLINE ONLINE rac1
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora.prabgun.db ora....se.type 0/2 0/2 ONLINE OFFLINE
ora....sv3.svc ora....ce.type 0/0 0/0 ONLINE OFFLINE
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE rac3
ora....C3.lsnr application 0/5 0/0 ONLINE ONLINE rac3
ora.rac3.gsd application 0/5 0/0 ONLINE ONLINE rac3
ora.rac3.ons application 0/3 0/0 ONLINE ONLINE rac3
ora.rac3.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac3
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1
ora.simar.db ora....se.type 0/2 0/1 ONLINE ONLINE rac1
ora....sv1.svc ora....ce.type 0/0 0/0 ONLINE ONLINE rac1
oracle : rac2.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1/script
$
[root@rac2 ~]# /u02/app/11.2.0.1/grid/bin/crsctl status resource ggateapp
NAME=ggateapp
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on rac2
[root@rac2 ~]#
Logs of goldengate:
2011-02-12 15:12:07 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): start manager.
2011-02-12 15:12:12 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop er *.
2011-02-12 15:12:12 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop er *.
2011-02-12 15:12:12 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop manager.
2011-02-12 15:12:16 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): start manager.
2011-02-12 15:12:21 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop er *.
2011-02-12 15:12:21 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop er *.
2011-02-12 15:12:21 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop manager.
2011-02-12 15:12:27 GGS INFO 330 Oracle GoldenGate Manager for Oracle, mgr.prm: Manager started (port 7909).
2011-02-12 15:12:47 GGS INFO 330 Oracle GoldenGate Manager for Oracle, mgr.prm: Manager started (port 7909).
Step 7:
If one ASM instance goes down it should not effect our manager process and for that use the following entries of ASM on every node of RAC:
ASM entry on 3 nodes should look like:
Node 1:
RAC1ASM =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = +ASM1)
)
)
Node 2:
RAC1ASM =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = +ASM2)
)
)
Node 3:
RAC1ASM =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = +ASM3)
)
)
You shall not be saved by hundreds of thousands and millions of helping hands. Chanting the Naam, you shall be lifted up and carried across. Where countless misfortunes threaten to destroy you, the Name of the Lord shall rescue you in an instant. Through countless incarnations, people are born and die. Chanting the Name of the Lord, you shall come to rest in peace. The ego is polluted by a filth which can never be washed off. The Name of the Lord erases millions of sins.
Monday, February 7, 2011
Part 31 of series - Move/Migrate/Relocate OCR and Vote disk in 11gr2 Grid infrastructure from one ASM diskgroup to another ASM diskgroup
Remembering God, one's works are accomplished. Remembering God, one never grieves. Remembering God, one speaks the Glorious Praises of the Lord. Remembering God, one is absorbed into the state of intuitive ease. Remembering God, one attains the unchanging position. Remembering God, the heart-lotus blossoms forth. Remembering God, the unstruck melody vibrates. The peace of the meditative remembrance of God has no end or limitation.
They alone remember Him, upon whom God bestows His Grace.
Index of all the posts of Gurpartap Singh's Blog
I had a bad experience while downgrading grid infrastructure from 11.2.0.2 to 11.2.0.1
I had a 7 node cluster with multiple databases and only 2 disk groups. One data and one recv.
Data diskgroup had OCR and vote disk. When executed deinstall on last node with last node option it deletes OCR and Vote disk and past of it is to clean up the header of the disk having OCR and VOTE disk. Since headers of the disk were cleaned as the past of deinstall script with last node option, I was not able to mount DATA diskgrup and so lost all the data and had to restore all the databases. Then came to know that the best practice is to have OCR and Vote disk
in different diskgroup. Here are the steps to move or migrate OCR and vote disk to new diskgroup in 11gr2 grid infrastructure.
All the following steps can be done while all databases are online. I created a new diskgroup +OCR1 using asmca. You can create this from comandline also. It can be 1GB or 2GB lun only. This diskggroup is mounted on all the 3 nodes.
Document I used is following:
http://download.oracle.com/docs/cd/E18283_01/rac.112/e16794.pdf
Page 61,65:
Here we go:
oracle : rac1.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid
$ su -
Password:
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3200
Available space (kbytes) : 258920
ID : 174947145
Device/File Name : +DATA1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@rac1 ~]#
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/ocrconfig -add +OCR1
Logs:
2011-02-07 22:19:12.726
[crsd(7242)]CRS-1007:The OCR/OCR mirror location was replaced by +OCR1.
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3200
Available space (kbytes) : 258920
ID : 174947145
Device/File Name : +DATA1
Device/File integrity check succeeded
Device/File Name : +OCR1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@rac1 ~]#
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/ocrconfig -delete +DATA1
2011-02-07 22:21:17.963
[crsd(7242)]CRS-1010:The OCR mirror location +DATA1 was removed.
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3200
Available space (kbytes) : 258920
ID : 174947145
Device/File Name : +OCR1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@rac1 ~]#
Logs on node 2:
2011-02-07 22:19:15.521
[crsd(6789)]CRS-1007:The OCR/OCR mirror location was replaced by +OCR1.
2011-02-07 22:21:18.611
[crsd(6789)]CRS-1010:The OCR mirror location +DATA1 was removed.
Logs on node 3:
2011-02-07 22:19:16.395
[crsd(6914)]CRS-1007:The OCR/OCR mirror location was replaced by +OCR1.
2011-02-07 22:20:09.459
[ctssd(6539)]CRS-2408:The clock on host rac3 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2011-02-07 22:20:25.467
[ctssd(6539)]CRS-2408:The clock on host rac3 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2011-02-07 22:21:18.967
[crsd(6914)]CRS-1010:The OCR mirror location +DATA1 was removed.
--------------------------------------------------
page 65:
If there were multiple OCR locations then following command cold be used:
ocrconfig -replace current_OCR_location -replacement new_OCR_location
=============================================================================
http://download.oracle.com/docs/cd/E18283_01/rac.112/e16794.pdf
Page 279:
Moving vote disk:
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 44d575b60a4b4fb5bf3cf8b51d001778 (/dev/sdc) [DATA1]
Located 1 voting disk(s).
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/crsctl replace votedisk +OCR1
Successful addition of voting disk 28657f1692a04fd3bf80dd7451fc1817.
Successful deletion of voting disk 44d575b60a4b4fb5bf3cf8b51d001778.
Successfully replaced voting disk group with +OCR1.
CRS-4266: Voting file(s) successfully replaced
In cluster alert log:
rac1:
2011-02-07 22:39:47.924
[cssd(6548)]CRS-1605:CSSD voting file is online: /dev/sdl; details in /u02/app/11.2.0.1/grid/log/rac1/cssd/ocssd.log.
2011-02-07 22:39:49.104
[cssd(6548)]CRS-1626:A Configuration change request completed successfully
2011-02-07 22:39:49.608
[cssd(6548)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 rac3 .
rac2:
2011-02-07 22:39:12.120
[ctssd(6300)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2011-02-07 22:39:48.614
[cssd(6166)]CRS-1605:CSSD voting file is online: /dev/sdl; details in /u02/app/11.2.0.1/grid/log/rac2/cssd/ocssd.log.
2011-02-07 22:39:49.186
[cssd(6166)]CRS-1626:A Configuration change request completed successfully
2011-02-07 22:39:49.608
[cssd(6166)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 rac3 .
2011-02-07 22:43:04.315
[ctssd(6300)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2011-02-07 22:43:20.321
[ctssd(6300)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
rac3:
2011-02-07 22:39:48.621
[cssd(6444)]CRS-1605:CSSD voting file is online: /dev/sdl; details in /u02/app/11.2.0.1/grid/log/rac3/cssd/ocssd.log.
2011-02-07 22:39:49.100
[cssd(6444)]CRS-1626:A Configuration change request completed successfully
2011-02-07 22:39:49.604
[cssd(6444)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 rac3 .
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 28657f1692a04fd3bf80dd7451fc1817 (/dev/sdl) [OCR1]
Located 1 voting disk(s).
[root@rac1 ~]#
------------------------------------
crsctl add css votedisk
------------------------------------
File Universal Id (FUID) obtained from (crsctl query css votedisk)
crsctl delete css votedisk 28657f1692a04fd3bf80dd7451fc1817
Remembering the Lord, His devotees are famous and radiant. Remembering the Lord, the Vedas were composed. Remembering the Lord, we become Siddhas, celibates and givers. Remembering the Lord, the lowly become known in all four directions. For the remembrance of the Lord, the whole world was established. Remember, remember in meditation the Lord, the Creator, the Cause of causes. For the remembrance of the Lord, He created the whole creation. In the remembrance of the Lord, He Himself is Formless. By His Grace, He Himself bestows understanding.
They alone remember Him, upon whom God bestows His Grace.
Index of all the posts of Gurpartap Singh's Blog
I had a bad experience while downgrading grid infrastructure from 11.2.0.2 to 11.2.0.1
I had a 7 node cluster with multiple databases and only 2 disk groups. One data and one recv.
Data diskgroup had OCR and vote disk. When executed deinstall on last node with last node option it deletes OCR and Vote disk and past of it is to clean up the header of the disk having OCR and VOTE disk. Since headers of the disk were cleaned as the past of deinstall script with last node option, I was not able to mount DATA diskgrup and so lost all the data and had to restore all the databases. Then came to know that the best practice is to have OCR and Vote disk
in different diskgroup. Here are the steps to move or migrate OCR and vote disk to new diskgroup in 11gr2 grid infrastructure.
All the following steps can be done while all databases are online. I created a new diskgroup +OCR1 using asmca. You can create this from comandline also. It can be 1GB or 2GB lun only. This diskggroup is mounted on all the 3 nodes.
Document I used is following:
http://download.oracle.com/docs/cd/E18283_01/rac.112/e16794.pdf
Page 61,65:
Here we go:
oracle : rac1.rac.meditate.com : @crs : /u02/app/11.2.0.1/grid
$ su -
Password:
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3200
Available space (kbytes) : 258920
ID : 174947145
Device/File Name : +DATA1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@rac1 ~]#
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/ocrconfig -add +OCR1
Logs:
2011-02-07 22:19:12.726
[crsd(7242)]CRS-1007:The OCR/OCR mirror location was replaced by +OCR1.
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3200
Available space (kbytes) : 258920
ID : 174947145
Device/File Name : +DATA1
Device/File integrity check succeeded
Device/File Name : +OCR1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@rac1 ~]#
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/ocrconfig -delete +DATA1
2011-02-07 22:21:17.963
[crsd(7242)]CRS-1010:The OCR mirror location +DATA1 was removed.
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 3
Total space (kbytes) : 262120
Used space (kbytes) : 3200
Available space (kbytes) : 258920
ID : 174947145
Device/File Name : +OCR1
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
[root@rac1 ~]#
Logs on node 2:
2011-02-07 22:19:15.521
[crsd(6789)]CRS-1007:The OCR/OCR mirror location was replaced by +OCR1.
2011-02-07 22:21:18.611
[crsd(6789)]CRS-1010:The OCR mirror location +DATA1 was removed.
Logs on node 3:
2011-02-07 22:19:16.395
[crsd(6914)]CRS-1007:The OCR/OCR mirror location was replaced by +OCR1.
2011-02-07 22:20:09.459
[ctssd(6539)]CRS-2408:The clock on host rac3 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2011-02-07 22:20:25.467
[ctssd(6539)]CRS-2408:The clock on host rac3 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2011-02-07 22:21:18.967
[crsd(6914)]CRS-1010:The OCR mirror location +DATA1 was removed.
--------------------------------------------------
page 65:
If there were multiple OCR locations then following command cold be used:
ocrconfig -replace current_OCR_location -replacement new_OCR_location
=============================================================================
http://download.oracle.com/docs/cd/E18283_01/rac.112/e16794.pdf
Page 279:
Moving vote disk:
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 44d575b60a4b4fb5bf3cf8b51d001778 (/dev/sdc) [DATA1]
Located 1 voting disk(s).
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/crsctl replace votedisk +OCR1
Successful addition of voting disk 28657f1692a04fd3bf80dd7451fc1817.
Successful deletion of voting disk 44d575b60a4b4fb5bf3cf8b51d001778.
Successfully replaced voting disk group with +OCR1.
CRS-4266: Voting file(s) successfully replaced
In cluster alert log:
rac1:
2011-02-07 22:39:47.924
[cssd(6548)]CRS-1605:CSSD voting file is online: /dev/sdl; details in /u02/app/11.2.0.1/grid/log/rac1/cssd/ocssd.log.
2011-02-07 22:39:49.104
[cssd(6548)]CRS-1626:A Configuration change request completed successfully
2011-02-07 22:39:49.608
[cssd(6548)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 rac3 .
rac2:
2011-02-07 22:39:12.120
[ctssd(6300)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2011-02-07 22:39:48.614
[cssd(6166)]CRS-1605:CSSD voting file is online: /dev/sdl; details in /u02/app/11.2.0.1/grid/log/rac2/cssd/ocssd.log.
2011-02-07 22:39:49.186
[cssd(6166)]CRS-1626:A Configuration change request completed successfully
2011-02-07 22:39:49.608
[cssd(6166)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 rac3 .
2011-02-07 22:43:04.315
[ctssd(6300)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
2011-02-07 22:43:20.321
[ctssd(6300)]CRS-2408:The clock on host rac2 has been updated by the Cluster Time Synchronization Service to be synchronous with the mean cluster time.
rac3:
2011-02-07 22:39:48.621
[cssd(6444)]CRS-1605:CSSD voting file is online: /dev/sdl; details in /u02/app/11.2.0.1/grid/log/rac3/cssd/ocssd.log.
2011-02-07 22:39:49.100
[cssd(6444)]CRS-1626:A Configuration change request completed successfully
2011-02-07 22:39:49.604
[cssd(6444)]CRS-1601:CSSD Reconfiguration complete. Active nodes are rac1 rac2 rac3 .
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 28657f1692a04fd3bf80dd7451fc1817 (/dev/sdl) [OCR1]
Located 1 voting disk(s).
[root@rac1 ~]#
------------------------------------
crsctl add css votedisk
------------------------------------
File Universal Id (FUID) obtained from (crsctl query css votedisk)
crsctl delete css votedisk 28657f1692a04fd3bf80dd7451fc1817
Remembering the Lord, His devotees are famous and radiant. Remembering the Lord, the Vedas were composed. Remembering the Lord, we become Siddhas, celibates and givers. Remembering the Lord, the lowly become known in all four directions. For the remembrance of the Lord, the whole world was established. Remember, remember in meditation the Lord, the Creator, the Cause of causes. For the remembrance of the Lord, He created the whole creation. In the remembrance of the Lord, He Himself is Formless. By His Grace, He Himself bestows understanding.
Thursday, February 3, 2011
Part 30 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Upgrade Additional Grid Control , OMS
Those who remember God are wealthy. Those who remember God are honorable. Those who remember God are approved. Those who remember God are the most distinguished persons. Those who remember God are not lacking. Those who remember God are the rulers of all. Those who remember God dwell in peace. Those who remember God are immortal and eternal. They alone hold to the remembrance of Him, unto whom He Himself shows His Mercy.
Index of all the posts of Gurpartap Singh's Blog
---------------------------------------------------------------
To do a Fresh install of 11gR1 Look at posts 12,13 and 14.
Part 12 of series, Install JDK for Enterprise Manager Grid COntrol 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control.html
Part 13 of series, Install Weblogic wls10.3.2 for Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_29.html
Part 14 of series, Install Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_2064.html
To upgrade existing Enterprise Manager Grid control from 10.2.0.4/10.2.0.5 to 11.1.0.1
Part 30 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Upgrade Additional Grid Control , OMS
http://gurpartapblog.blogspot.com/2011/02/part-30-of-series-upgrade-enterprise.html
Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS
http://gurpartapblog.blogspot.com/2011/02/part-29-of-series-upgrade-enterprise.html
Part 28 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install JDK and Weblogic
http://gurpartapblog.blogspot.com/2011/02/part-28-of-series-upgrade-enterprise.html
---------------------------------------------------------------
Now we will upgrade the Additional OMS.
Primary OMS is up and running on sever grid01. Keep it up and running and start extending (upgrading the additional oms server)
oracle : grid02 : @agent10g : /home/oracle/software/GridContol/11.1
$ ./runInstaller
On first screen uncheck "I wish to receive security updates via MY Oracle Support" and click next.
On next pop up click Yes to "Do you wish to remain uninformed of critical security issues ….."
On next screen make sure "Skip Software Updates is clicked and click next.
On next screen make sure "Upgrade to Enterprise Manager 11g" and click next.
On next screen make sure the current agent/oms base and version is correct and is checked/selected and click next.
There are prerequisites on next screen and make sure all succeed and click next.
At next screen if it doesn’t accept the sys password and error at invalid username/password, just stop instance 2 on server grid02 and make sure all the services are running and try again and it will move to next screen.
And if the connect descriptor is incorrect, you have to change the $ORACLE_HOME/sysman/config/emoms.properties
oracle.sysman.eml.mntr.emdRepConnectDescriptor=(DESCRIPTION\=(ADDRESS_LIST\=(FAILOVER\=ON)(ADDRESS\=(PROTOCOL\=TCP)(HOST\=grid01)(PORT\=41521)))(CONNECT_DATA\=(SERVICE_NAME\=GRIDsv1)(SERVER\=DEDICATED)))
And retry connectivity.
On next screen "Change the OMS instance base location to end in "11.1" instead of "gc_inst"
and make sure the Middleware and the OMS base location is correct and click next. Here installer checks for space and
will error out if we don’t have enough space.
On the next screen Admin Server Host grid01 (where existing upgraded OMS is hosted)
Admin Server Port : 7102 (Check the range from the primay OMS install snapshot that we took)
Enter weblogic username and password that you used in the primay oms upgrade.
On next keep the ports default and click next.
On next screen click "Install" and the install will starts.
At the end of the installation you will be prompted to run a script for the root.
If prompted, accept all defaults and it should finish without errors and after that click ok
and some Configuration Assistants will run. The process will take at least 2-3 hours so be prepared to wait:
(The Repository Upgrade portion takes the longest)
After keeping the snapshot of last window that has the url to connect click close.
Update the oratab with the new agent and OMS home:
#oms10g:/ora/fs0000/app/oracle/product/10.2/oms10g:N
#agent10g:/ora/fs0000/app/oracle/product/10.2/agent10g:N
agent11g:/ora/fs0000/app/oracle/product/Middleware/oms11g:N
agent11g:/ora/fs0000/app/oracle/product/Middleware/agent11g:N
Now apply PSU2 patch to OMS and agent.
Last step:
Submit the Refresh Weblogic job
Follow document How to Enable job "Refresh WebLogic Domain" ? [ID 1088537.1]
Thats it !!!!!
-------------------------------------------------------------------
To do a Fresh install of 11gR1 Look at posts 12,13 and 14.
Part 12 of series, Install JDK for Enterprise Manager Grid COntrol 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control.html
Part 13 of series, Install Weblogic wls10.3.2 for Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_29.html
Part 14 of series, Install Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_2064.html
To upgrade existing Enterprise Manager Grid control from 10.2.0.4/10.2.0.5 to 11.1.0.1
Part 30 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Upgrade Additional Grid Control , OMS
http://gurpartapblog.blogspot.com/2011/02/part-30-of-series-upgrade-enterprise.html
Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS
http://gurpartapblog.blogspot.com/2011/02/part-29-of-series-upgrade-enterprise.html
Part 28 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install JDK and Weblogic
http://gurpartapblog.blogspot.com/2011/02/part-28-of-series-upgrade-enterprise.html
-------------------------------------------------------------------
Remembering God, one's works are accomplished. Remembering God, one never grieves.
Remembering God, one speaks the Glorious Praises of the Lord. Remembering God, one is absorbed into the state of intuitive ease. Remembering God, one attains the unchanging position. Remembering God, the heart-lotus blossoms forth. Remembering God, the unstruck melody vibrates. The peace of the meditative remembrance of God has no end or limitation.
They alone remember Him, upon whom God bestows His Grace.
Index of all the posts of Gurpartap Singh's Blog
---------------------------------------------------------------
To do a Fresh install of 11gR1 Look at posts 12,13 and 14.
Part 12 of series, Install JDK for Enterprise Manager Grid COntrol 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control.html
Part 13 of series, Install Weblogic wls10.3.2 for Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_29.html
Part 14 of series, Install Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_2064.html
To upgrade existing Enterprise Manager Grid control from 10.2.0.4/10.2.0.5 to 11.1.0.1
Part 30 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Upgrade Additional Grid Control , OMS
http://gurpartapblog.blogspot.com/2011/02/part-30-of-series-upgrade-enterprise.html
Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS
http://gurpartapblog.blogspot.com/2011/02/part-29-of-series-upgrade-enterprise.html
Part 28 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install JDK and Weblogic
http://gurpartapblog.blogspot.com/2011/02/part-28-of-series-upgrade-enterprise.html
---------------------------------------------------------------
Now we will upgrade the Additional OMS.
Primary OMS is up and running on sever grid01. Keep it up and running and start extending (upgrading the additional oms server)
oracle : grid02 : @agent10g : /home/oracle/software/GridContol/11.1
$ ./runInstaller
On first screen uncheck "I wish to receive security updates via MY Oracle Support" and click next.
On next pop up click Yes to "Do you wish to remain uninformed of critical security issues ….."
On next screen make sure "Skip Software Updates is clicked and click next.
On next screen make sure "Upgrade to Enterprise Manager 11g" and click next.
On next screen make sure the current agent/oms base and version is correct and is checked/selected and click next.
There are prerequisites on next screen and make sure all succeed and click next.
At next screen if it doesn’t accept the sys password and error at invalid username/password, just stop instance 2 on server grid02 and make sure all the services are running and try again and it will move to next screen.
And if the connect descriptor is incorrect, you have to change the $ORACLE_HOME/sysman/config/emoms.properties
oracle.sysman.eml.mntr.emdRepConnectDescriptor=(DESCRIPTION\=(ADDRESS_LIST\=(FAILOVER\=ON)(ADDRESS\=(PROTOCOL\=TCP)(HOST\=grid01)(PORT\=41521)))(CONNECT_DATA\=(SERVICE_NAME\=GRIDsv1)(SERVER\=DEDICATED)))
And retry connectivity.
On next screen "Change the OMS instance base location to end in "11.1" instead of "gc_inst"
and make sure the Middleware and the OMS base location is correct and click next. Here installer checks for space and
will error out if we don’t have enough space.
On the next screen Admin Server Host grid01 (where existing upgraded OMS is hosted)
Admin Server Port : 7102 (Check the range from the primay OMS install snapshot that we took)
Enter weblogic username and password that you used in the primay oms upgrade.
On next keep the ports default and click next.
On next screen click "Install" and the install will starts.
At the end of the installation you will be prompted to run a script for the root.
If prompted, accept all defaults and it should finish without errors and after that click ok
and some Configuration Assistants will run. The process will take at least 2-3 hours so be prepared to wait:
(The Repository Upgrade portion takes the longest)
After keeping the snapshot of last window that has the url to connect click close.
Update the oratab with the new agent and OMS home:
#oms10g:/ora/fs0000/app/oracle/product/10.2/oms10g:N
#agent10g:/ora/fs0000/app/oracle/product/10.2/agent10g:N
agent11g:/ora/fs0000/app/oracle/product/Middleware/oms11g:N
agent11g:/ora/fs0000/app/oracle/product/Middleware/agent11g:N
Now apply PSU2 patch to OMS and agent.
Last step:
Submit the Refresh Weblogic job
Follow document How to Enable job "Refresh WebLogic Domain" ? [ID 1088537.1]
Thats it !!!!!
-------------------------------------------------------------------
To do a Fresh install of 11gR1 Look at posts 12,13 and 14.
Part 12 of series, Install JDK for Enterprise Manager Grid COntrol 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control.html
Part 13 of series, Install Weblogic wls10.3.2 for Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_29.html
Part 14 of series, Install Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_2064.html
To upgrade existing Enterprise Manager Grid control from 10.2.0.4/10.2.0.5 to 11.1.0.1
Part 30 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Upgrade Additional Grid Control , OMS
http://gurpartapblog.blogspot.com/2011/02/part-30-of-series-upgrade-enterprise.html
Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS
http://gurpartapblog.blogspot.com/2011/02/part-29-of-series-upgrade-enterprise.html
Part 28 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install JDK and Weblogic
http://gurpartapblog.blogspot.com/2011/02/part-28-of-series-upgrade-enterprise.html
-------------------------------------------------------------------
Remembering God, one's works are accomplished. Remembering God, one never grieves.
Remembering God, one speaks the Glorious Praises of the Lord. Remembering God, one is absorbed into the state of intuitive ease. Remembering God, one attains the unchanging position. Remembering God, the heart-lotus blossoms forth. Remembering God, the unstruck melody vibrates. The peace of the meditative remembrance of God has no end or limitation.
They alone remember Him, upon whom God bestows His Grace.
Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS
The remembrance of God is the highest and most exalted of all. In the remembrance of God, many are saved. In the remembrance of God, thirst is quenched. In the remembrance of God, all things are known. In the remembrance of God, there is no fear of death. In the remembrance of God, hopes are fulfilled. In the remembrance of God, the filth of the mind is removed. The Ambrosial Naam, the Name of the Lord, is absorbed into the heart. God abides upon the tongues of His Saints.
Index of all the posts of Gurpartap Singh's Blog
----------------------------------------------------------
To do a Fresh install of 11gR1 Look at posts 12,13 and 14.
Part 12 of series, Install JDK for Enterprise Manager Grid COntrol 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control.html
Part 13 of series, Install Weblogic wls10.3.2 for Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_29.html
Part 14 of series, Install Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_2064.html
To upgrade existing Enterprise Manager Grid control from 10.2.0.4/10.2.0.5 to 11.1.0.1
Part 30 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Upgrade Additional Grid Control , OMS
http://gurpartapblog.blogspot.com/2011/02/part-30-of-series-upgrade-enterprise.html
Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS
http://gurpartapblog.blogspot.com/2011/02/part-29-of-series-upgrade-enterprise.html
Part 28 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install JDK and Weblogic
http://gurpartapblog.blogspot.com/2011/02/part-28-of-series-upgrade-enterprise.html
---------------------------------------------------------
INSTALLING and Upgrading GRID CONTROL
Grid Control 11.1.0.1.0 Install and Upgrade (from 10.2.0.4+)
I have already downloaded and extracted Grid Control 11g for Sun64 and placed it in the software directory at “/home/oracle/software/GridContol/11.1” After unzipping the downloaded files it will be approximately 4.7 GB
Prerequisites :
Your existing OMS must be 10.2.0.4 or higher. If not, you cannot upgrade to 11g until it’s at least 10.2.0.4. The first step is to stop all OMS servers and upgrade 1 OMS first as this will also upgrade the repository.
You must have at LEAST 8.4 GB free to install 11g OMS.
Verify the OS package prerequisites at: http://download.oracle.com/docs/cd/E11857_01/install.111/e15838/appdx_package_reqs.htm#BABIJAJB
For Solaris 10 we only needed: SUNWbtool
Database Init parameters:
cluster_database
(Only for Oracle RAC Database) TRUE
Compatible 10.2.0.1.0 or higher
max_commit_propogation_delay
(Only for Oracle RAC Database) 0
remote_login_passwordfile SHARED or EXCLUSIVE
statistics_level TYPICAL
timed_statistics TRUE
undo_management AUTO
workarea_size_policy AUTO
db_block_size 8192 or greater
job_queue_processes (For Installation) 10 or greater
(For Upgrade) If you are upgrading Enterprise Manager 10g Grid Control Release 4 (10.2.0.4) or higher, then ensure that you set the value to 0 before upgrading. And after upgrading, reset it to the original value that existed before upgrading the product.
log_buffer 10485760 or greater
open_cursors 300 or greater
Processes 500 or greater
session_cached_cursors 200 or greater
Verify the database has the Partitioning option enabled. If this is an upgrade, it would already be there.
SQL> select value from v$option where parameter = 'Partitioning';
VALUE
----------------------------------------------------------------
TRUE
Ensure all SYSMAN objects are valid
SQL> select object_name, object_type from all_objects where owner='SYSMAN' and status <> 'VALID';
no rows selected
If there are invalid objects, recompile them.
@admin_recompile_invalid.sql SYSMAN
Verify all SYS objects are valid:
SQL> select object_name, object_type from all_objects where status<>'VALID' and object_name like 'DBMS%';
no rows selected
If there are invalid objects, also compile them.
Get the value of the current job queue processes in all the instances and make a note of it as we will be changing it temporarily:
SQL> select a.instance_name as sid, b.value as jobqueue from gv$instance a, gv$parameter b where a.inst_id = b.inst_id and b.name='job_queue_processes';
SID
----------------
JOBQUEUE
--------------------------------------------------------------------------------
REPR2
20
REPR1
20
Here starts the downtime.
First, verify the repository database is not in QUIESCE mode:
SQL> select active_state from v$instance;
ACTIVE_ST
---------
NORMAL
Set job_queue_processes to 0 during the upgrade and remove the em jobs (we’ll add them back later).
Connect to the repository as SYSMAN.
SQL> conn sysman
Enter password:
Connected.
SQL> execute emd_maintenance.remove_em_dbms_jobs;
PL/SQL procedure successfully completed.
SQL> alter system set job_queue_processes=0;
System altered.
SQL> commit;
Commit complete.
SQL>
Ensure there are no active jobs:
SQL> select l.id2 job, l.sid, to_char(last_date, 'DD-MON-YYYY:HH24.MI.SS') last_date, to_char(this_date, 'DD-MON-YYYY:HH24.MI.SS') this_date, l.inst_id instance from sys.job$ j, gv$lock l where l.type = 'JQ' and j.job (+) = l.id2 order by 5, 4;
no rows selected
Disable dbms scheduler jobs:
SQL> execute dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED','TRUE');
PL/SQL procedure successfully completed.
Verify (wait if necessary) until there are no more running jobs
SQL> select count(*) from dba_scheduler_running_jobs where SLAVE_PROCESS_ID IS NOT NULL;
COUNT(*)
----------
0
Log in as SYS and stop the jobs
SQL> conn / as sysdba
Connected.
SQL> execute dbms_scheduler.disable('GATHER_STATS_JOB',TRUE);
PL/SQL procedure successfully completed.
SQL> execute dbms_scheduler.stop_job('GATHER_STATS_JOB',TRUE);
BEGIN dbms_scheduler.stop_job('GATHER_STATS_JOB',TRUE); END;
*
ERROR at line 1:
ORA-27366: job "SYS.GATHER_STATS_JOB" is not running
ORA-06512: at "SYS.DBMS_ISCHED", line 164
ORA-06512: at "SYS.DBMS_SCHEDULER", line 483
ORA-06512: at line 1
Ensure SYSMAN and DBSNMP users have EXECUTE on DBMS_RANDOM.
SQL> GRANT EXECUTE ON DBMS_RANDOM TO SYSMAN;
SQL> GRANT EXECUTE ON DBMS_RANDOM TO DBSNMP;
Now stop all OMS servers even if you are upgrading just one.
oracle : grid01 : @oms10g : /home/oracle/software
$ $ORACLE_HOME/opmn/bin/opmnctl stopall
opmnctl: stopping opmn and all managed processes...
oracle : grid01 : @oms10g : /home/oracle/software
$ emctl stop oms
Oracle Enterprise Manager 10g Release 5 Grid Control
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Oracle Management Server is Down.
OPMN may not be up.
Now stop on othernode and if you have more oms servers, stop them too.
oracle : grid02 : @oms10g : /home/oracle/software
$ $ORACLE_HOME/opmn/bin/opmnctl stopall
opmnctl: stopping opmn and all managed processes...
oracle : grid02 : @oms10g : /home/oracle/software
$ emctl stop oms
Oracle Enterprise Manager 10g Release 5 Grid Control
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Oracle Management Server is Down.
OPMN may not be up.
Create a rollback point for OMS repository + binaries
Actually grid control has so many bugs and I want to have my own backout plan. I am creating a flashback point in the repository and back up the OMS binaries and agent binaries as these three components will be upgraded.
Tar up the binaries where there is space, you will need at least 5-6 GB.
oracle : grid01 : @oms10g : /home/oracle/software/OMS10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/oms10g
Do the same command on the other server/servers. I only have two oms servers so will do only on servers grid01/grid02.
oracle : grid02 : @oms10g : /home/oracle/software/OMS10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/oms10g
Also tar the agent binaries as:
oracle : grid01 : @oms10g : /home/oracle/software/Agent10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/agnet10g
Do the same command on the other server/servers. I only have two oms servers so will do only on servers grid01/grid02.
oracle : grid02 : @oms10g : /home/oracle/software/ Agent10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/agent10g
Now create a guaranteed flashback point on database level. Here we are enabling flashback in the database too.
SQL> select flashback_on from v$database;
FLASHBACK_ON
------------------
NO
SQL> show parameter recovery
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest string
db_recovery_file_dest_size big integer 0
SQL> alter system set db_recovery_file_dest_size=30G;
System altered.
SQL> alter system set db_recovery_file_dest="+ RECV1 " ;
System altered.
oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$ srvctl stop database -d REPR
Log back in and startup mount + start flashback
SQL> startup mount
ORACLE instance started.
Total System Global Area 3758096384 bytes
Fixed Size 2045064 bytes
Variable Size 1342180216 bytes
Database Buffers 2399141888 bytes
Redo Buffers 14729216 bytes
Database mounted.
SQL> alter database flashback on;
Database altered.
SQL> shutdown immediate
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$ srvctl start database -d REPR
oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$ srvctl start service -d REPR
oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$sqlplus / as sysdba
SQL> create restore point before_OMS11g_upgrade guarantee flashback database;
Restore point created.
Check that everything is enabled:
SQL> select FLASHBACK_ON from v$database;
FLASHBACK_ON
------------------
YES
SQL> ALTER SYSTEM SWITCH LOGFILE;
System altered.
SQL> select estimated_flashback_size,flashback_size from v$flashback_database_log;
ESTIMATED_FLASHBACK_SIZE FLASHBACK_SIZE
------------------------ --------------
0 31866880
Now lets Install/upgrade Grid Control 11g
Now stop the Management agent on the OMS you are upgrading. I am doing that on first node i.e. grid01.
. oraenv
agent10g
oracle : grid01 : @agent10g : /home/oracle/software
$ emctl stop agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Agent is Not Running
Now we will install.
Check your DISPLAY property is set and start your X Window app.
oracle : grid01 : @agent10g : /home/oracle/software
$ env |grep DISPLAY
DISPLAY=192.168.1.140:0.0
cd /home/oracle/software/GridContol/11.1
oracle : grid01 : @agent10g : /home/oracle/software/GridContol/11.1
$ ./runInstaller
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 150 MB. Actual 87354 MB Passed
Checking swap space: must be greater than 150 MB. Actual 94539 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
...
oracle : grid01 : @agent10g : /home/oracle/software/GridContol/11.1
$ ./runInstaller
On first screen uncheck “I wish to receive security updates via MY Oracle Support” and click next.
On next pop up click Yes to “Do you wish to remain uninformed of critical security issues …..”
On next screen make sure “Skip Software Updates is clicked and click next.
On next screen make sure “Upgrade to Enterprise Manager 11g” and click next.
On next screen make sure the current agent/oms base and version is correct and is checked/selected and click next.
There are prerequisites on next screen and make sure all succeed and click next.
At next screen if it doesn’t accept the sys password and error at invalid username/password, just stop instance 2 on server grid02 and make sure all the services are running and try again and it will move to next screen.
And if the connect descriptor is incorrect, you have to change the $ORACLE_HOME/sysman/config/emoms.properties
oracle.sysman.eml.mntr.emdRepConnectDescriptor=(DESCRIPTION\=(ADDRESS_LIST\=(FAILOVER\=ON)(ADDRESS\=(PROTOCOL\=TCP)(HOST\=grid01)(PORT\=41521)))(CONNECT_DATA\=(SERVICE_NAME\=GRIDsv1)(SERVER\=DEDICATED)))
And retry connectivity.
On next screen “Change the OMS instance base location to end in “11.1” instead of “gc_inst”
and make sure the Middleware and the OMS base location is correct and click next. Here installer checks for space and will error out if we don’t have enough space.
On next screen enter the password of GCDomain user. Username is “weblogic” and password whatever you want. On this screen also enter the password of “nodemanager”. The password you create has to be 8 digits and contain alphanumeric. Also for “JVM diagnostic data tablespace” enter following +DATA1/REPR/DATAFILE/mgmt_ad4j.dbf. Now click next.
On next keep the ports default and click next. (Remember to take and save the snapshot of this window, will help you in checking port no.s in extending ths OMS to new server)
On next screen click “Install” and the install will start.
At the end of the installation you will be prompted to run a script for the root.
If prompted, accept all defaults and it should finish without errors and after that click ok and some Configuration Assistants will run. The process will take at least 2-3 hours so be prepared to wait:
It may fail at step oms configuration, just execute the script in the error popup window as sysman and click retry and then it should be ok. This error may be due to some invalid objects.
(The Repository Upgrade portion takes the longest)
After keeping the snapshot of last window that has the url to connect click close.
Now some post steps:
Change the job_queue_processes parameter back to the original value
SQL> sho parameter job_queue
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
job_queue_processes integer 0
SQL> alter system set job_queue_processes=20;
System altered.
SQL> sho parameter job_queue
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
job_queue_processes integer 20
Start DBMS Scheduler as sys:
SQL> execute dbms_scheduler.set_scheduler_attribute ('SCHEDULER_DISABLED','FALSE');
PL/SQL procedure successfully completed.
SQL> conn sysman
Enter password:
Connected.
SQL> execute emd_maintenance.submit_em_dbms_jobs;
Reconnect as sys and submit the gather stats jobs:
SQL> conn / as sysdba
Connected.
SQL> execute dbms_scheduler.enable('GATHER_STATS_JOB');
PL/SQL procedure successfully completed.
PL/SQL procedure successfully completed.
Update the oratab with the new agent and OMS home:
#oms10g:/ora/fs0000/app/oracle/product/10.2/oms10g:N
#agent10g:/ora/fs0000/app/oracle/product/10.2/agent10g:N
agent11g:/ora/fs0000/app/oracle/product/Middleware/oms11g:N
agent11g:/ora/fs0000/app/oracle/product/Middleware/agent11g:N
And test you can query them:
oracle : grid01 : @agent11g : /home/oracle
$ emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /ora/fs0000/app/oracle/product/Middleware/agent11g
Agent binaries : /ora/fs0000/app/oracle/product/Middleware/agent11g
Agent Process ID : 21210
Parent Process ID : 21195
Agent URL : https://grid01:3872/emd/main/
Repository URL : https://grid01:1159/em/upload
Started at : 2011-01-15 02:49:16
Started by user : oracle
Last Reload : 2011-01-22 02:57:35
Last successful upload : 2011-01-22 05:15:38
Total Megabytes of XML files uploaded so far : 37.80
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 17.72%
Last successful heartbeat to OMS : 2011-01-22 05:24:30
---------------------------------------------------------------
Agent is Running and Ready
oracle : grid01 : @oms11g : /home/oracle
$ emctl status oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
WebTier is Up
Oracle Management Server is Up
Next remove old 10g Middleware targets
Now log into your new 11.1 Grid control OMS and we’ll have to do some cleanup
In particular, remove
_home
_OC4J_EM
_OC4J_EMPROV
_HTTP Server
Web Cache.
Some have to be removed one at a time for proper cleanup.
Other issues you may notice:
"EMGC_ADMINSERVER/FMW Welcome Page Application(11.1.0.0.0)" -Target Down [ID 1100553.1]
Apply patch 9431704 to fix this.
At the time of writing this note we had PSU 2 released a few days ago. So, apply it on top of this 11gr1 OMS. It may take 2-3 hours to apply this PSU. Before applying this PSU upgrade the OPatch to latest OPatch for 11gR1 and not 11gR2 for both OMS and Agent.
-------------------------------------------------------------------
To do a Fresh install of 11gR1 Look at posts 12,13 and 14.
Part 12 of series, Install JDK for Enterprise Manager Grid COntrol 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control.html
Part 13 of series, Install Weblogic wls10.3.2 for Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_29.html
Part 14 of series, Install Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_2064.html
To upgrade existing Enterprise Manager Grid control from 10.2.0.4/10.2.0.5 to 11.1.0.1
Part 30 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Upgrade Additional Grid Control , OMS
http://gurpartapblog.blogspot.com/2011/02/part-30-of-series-upgrade-enterprise.html
Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS
http://gurpartapblog.blogspot.com/2011/02/part-29-of-series-upgrade-enterprise.html
Part 28 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install JDK and Weblogic
http://gurpartapblog.blogspot.com/2011/02/part-28-of-series-upgrade-enterprise.html
-----------------------------------------------------------------------
Those who remember God generously help others. Those who remember God -- to them, I am forever a sacrifice. Those who remember God -- their faces are beautiful. Those who
remember God abide in peace. Those who remember God conquer their souls. Those who remember God have a pure and spotless lifestyle. Those who remember God experience all sorts of joys. Those who remember God abide near the Lord. By the Grace of the Saints, one remains awake and aware, night and day.
Index of all the posts of Gurpartap Singh's Blog
----------------------------------------------------------
To do a Fresh install of 11gR1 Look at posts 12,13 and 14.
Part 12 of series, Install JDK for Enterprise Manager Grid COntrol 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control.html
Part 13 of series, Install Weblogic wls10.3.2 for Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_29.html
Part 14 of series, Install Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_2064.html
To upgrade existing Enterprise Manager Grid control from 10.2.0.4/10.2.0.5 to 11.1.0.1
Part 30 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Upgrade Additional Grid Control , OMS
http://gurpartapblog.blogspot.com/2011/02/part-30-of-series-upgrade-enterprise.html
Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS
http://gurpartapblog.blogspot.com/2011/02/part-29-of-series-upgrade-enterprise.html
Part 28 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install JDK and Weblogic
http://gurpartapblog.blogspot.com/2011/02/part-28-of-series-upgrade-enterprise.html
---------------------------------------------------------
INSTALLING and Upgrading GRID CONTROL
Grid Control 11.1.0.1.0 Install and Upgrade (from 10.2.0.4+)
I have already downloaded and extracted Grid Control 11g for Sun64 and placed it in the software directory at “/home/oracle/software/GridContol/11.1” After unzipping the downloaded files it will be approximately 4.7 GB
Prerequisites :
Your existing OMS must be 10.2.0.4 or higher. If not, you cannot upgrade to 11g until it’s at least 10.2.0.4. The first step is to stop all OMS servers and upgrade 1 OMS first as this will also upgrade the repository.
You must have at LEAST 8.4 GB free to install 11g OMS.
Verify the OS package prerequisites at: http://download.oracle.com/docs/cd/E11857_01/install.111/e15838/appdx_package_reqs.htm#BABIJAJB
For Solaris 10 we only needed: SUNWbtool
Database Init parameters:
cluster_database
(Only for Oracle RAC Database) TRUE
Compatible 10.2.0.1.0 or higher
max_commit_propogation_delay
(Only for Oracle RAC Database) 0
remote_login_passwordfile SHARED or EXCLUSIVE
statistics_level TYPICAL
timed_statistics TRUE
undo_management AUTO
workarea_size_policy AUTO
db_block_size 8192 or greater
job_queue_processes (For Installation) 10 or greater
(For Upgrade) If you are upgrading Enterprise Manager 10g Grid Control Release 4 (10.2.0.4) or higher, then ensure that you set the value to 0 before upgrading. And after upgrading, reset it to the original value that existed before upgrading the product.
log_buffer 10485760 or greater
open_cursors 300 or greater
Processes 500 or greater
session_cached_cursors 200 or greater
Verify the database has the Partitioning option enabled. If this is an upgrade, it would already be there.
SQL> select value from v$option where parameter = 'Partitioning';
VALUE
----------------------------------------------------------------
TRUE
Ensure all SYSMAN objects are valid
SQL> select object_name, object_type from all_objects where owner='SYSMAN' and status <> 'VALID';
no rows selected
If there are invalid objects, recompile them.
@admin_recompile_invalid.sql SYSMAN
Verify all SYS objects are valid:
SQL> select object_name, object_type from all_objects where status<>'VALID' and object_name like 'DBMS%';
no rows selected
If there are invalid objects, also compile them.
Get the value of the current job queue processes in all the instances and make a note of it as we will be changing it temporarily:
SQL> select a.instance_name as sid, b.value as jobqueue from gv$instance a, gv$parameter b where a.inst_id = b.inst_id and b.name='job_queue_processes';
SID
----------------
JOBQUEUE
--------------------------------------------------------------------------------
REPR2
20
REPR1
20
Here starts the downtime.
First, verify the repository database is not in QUIESCE mode:
SQL> select active_state from v$instance;
ACTIVE_ST
---------
NORMAL
Set job_queue_processes to 0 during the upgrade and remove the em jobs (we’ll add them back later).
Connect to the repository as SYSMAN.
SQL> conn sysman
Enter password:
Connected.
SQL> execute emd_maintenance.remove_em_dbms_jobs;
PL/SQL procedure successfully completed.
SQL> alter system set job_queue_processes=0;
System altered.
SQL> commit;
Commit complete.
SQL>
Ensure there are no active jobs:
SQL> select l.id2 job, l.sid, to_char(last_date, 'DD-MON-YYYY:HH24.MI.SS') last_date, to_char(this_date, 'DD-MON-YYYY:HH24.MI.SS') this_date, l.inst_id instance from sys.job$ j, gv$lock l where l.type = 'JQ' and j.job (+) = l.id2 order by 5, 4;
no rows selected
Disable dbms scheduler jobs:
SQL> execute dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED','TRUE');
PL/SQL procedure successfully completed.
Verify (wait if necessary) until there are no more running jobs
SQL> select count(*) from dba_scheduler_running_jobs where SLAVE_PROCESS_ID IS NOT NULL;
COUNT(*)
----------
0
Log in as SYS and stop the jobs
SQL> conn / as sysdba
Connected.
SQL> execute dbms_scheduler.disable('GATHER_STATS_JOB',TRUE);
PL/SQL procedure successfully completed.
SQL> execute dbms_scheduler.stop_job('GATHER_STATS_JOB',TRUE);
BEGIN dbms_scheduler.stop_job('GATHER_STATS_JOB',TRUE); END;
*
ERROR at line 1:
ORA-27366: job "SYS.GATHER_STATS_JOB" is not running
ORA-06512: at "SYS.DBMS_ISCHED", line 164
ORA-06512: at "SYS.DBMS_SCHEDULER", line 483
ORA-06512: at line 1
Ensure SYSMAN and DBSNMP users have EXECUTE on DBMS_RANDOM.
SQL> GRANT EXECUTE ON DBMS_RANDOM TO SYSMAN;
SQL> GRANT EXECUTE ON DBMS_RANDOM TO DBSNMP;
Now stop all OMS servers even if you are upgrading just one.
oracle : grid01 : @oms10g : /home/oracle/software
$ $ORACLE_HOME/opmn/bin/opmnctl stopall
opmnctl: stopping opmn and all managed processes...
oracle : grid01 : @oms10g : /home/oracle/software
$ emctl stop oms
Oracle Enterprise Manager 10g Release 5 Grid Control
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Oracle Management Server is Down.
OPMN may not be up.
Now stop on othernode and if you have more oms servers, stop them too.
oracle : grid02 : @oms10g : /home/oracle/software
$ $ORACLE_HOME/opmn/bin/opmnctl stopall
opmnctl: stopping opmn and all managed processes...
oracle : grid02 : @oms10g : /home/oracle/software
$ emctl stop oms
Oracle Enterprise Manager 10g Release 5 Grid Control
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Oracle Management Server is Down.
OPMN may not be up.
Create a rollback point for OMS repository + binaries
Actually grid control has so many bugs and I want to have my own backout plan. I am creating a flashback point in the repository and back up the OMS binaries and agent binaries as these three components will be upgraded.
Tar up the binaries where there is space, you will need at least 5-6 GB.
oracle : grid01 : @oms10g : /home/oracle/software/OMS10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/oms10g
Do the same command on the other server/servers. I only have two oms servers so will do only on servers grid01/grid02.
oracle : grid02 : @oms10g : /home/oracle/software/OMS10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/oms10g
Also tar the agent binaries as:
oracle : grid01 : @oms10g : /home/oracle/software/Agent10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/agnet10g
Do the same command on the other server/servers. I only have two oms servers so will do only on servers grid01/grid02.
oracle : grid02 : @oms10g : /home/oracle/software/ Agent10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/agent10g
Now create a guaranteed flashback point on database level. Here we are enabling flashback in the database too.
SQL> select flashback_on from v$database;
FLASHBACK_ON
------------------
NO
SQL> show parameter recovery
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest string
db_recovery_file_dest_size big integer 0
SQL> alter system set db_recovery_file_dest_size=30G;
System altered.
SQL> alter system set db_recovery_file_dest="+ RECV1 " ;
System altered.
oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$ srvctl stop database -d REPR
Log back in and startup mount + start flashback
SQL> startup mount
ORACLE instance started.
Total System Global Area 3758096384 bytes
Fixed Size 2045064 bytes
Variable Size 1342180216 bytes
Database Buffers 2399141888 bytes
Redo Buffers 14729216 bytes
Database mounted.
SQL> alter database flashback on;
Database altered.
SQL> shutdown immediate
SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options
oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$ srvctl start database -d REPR
oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$ srvctl start service -d REPR
oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$sqlplus / as sysdba
SQL> create restore point before_OMS11g_upgrade guarantee flashback database;
Restore point created.
Check that everything is enabled:
SQL> select FLASHBACK_ON from v$database;
FLASHBACK_ON
------------------
YES
SQL> ALTER SYSTEM SWITCH LOGFILE;
System altered.
SQL> select estimated_flashback_size,flashback_size from v$flashback_database_log;
ESTIMATED_FLASHBACK_SIZE FLASHBACK_SIZE
------------------------ --------------
0 31866880
Now lets Install/upgrade Grid Control 11g
Now stop the Management agent on the OMS you are upgrading. I am doing that on first node i.e. grid01.
. oraenv
agent10g
oracle : grid01 : @agent10g : /home/oracle/software
$ emctl stop agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Agent is Not Running
Now we will install.
Check your DISPLAY property is set and start your X Window app.
oracle : grid01 : @agent10g : /home/oracle/software
$ env |grep DISPLAY
DISPLAY=192.168.1.140:0.0
cd /home/oracle/software/GridContol/11.1
oracle : grid01 : @agent10g : /home/oracle/software/GridContol/11.1
$ ./runInstaller
Starting Oracle Universal Installer...
Checking Temp space: must be greater than 150 MB. Actual 87354 MB Passed
Checking swap space: must be greater than 150 MB. Actual 94539 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
...
oracle : grid01 : @agent10g : /home/oracle/software/GridContol/11.1
$ ./runInstaller
On first screen uncheck “I wish to receive security updates via MY Oracle Support” and click next.
On next pop up click Yes to “Do you wish to remain uninformed of critical security issues …..”
On next screen make sure “Skip Software Updates is clicked and click next.
On next screen make sure “Upgrade to Enterprise Manager 11g” and click next.
On next screen make sure the current agent/oms base and version is correct and is checked/selected and click next.
There are prerequisites on next screen and make sure all succeed and click next.
At next screen if it doesn’t accept the sys password and error at invalid username/password, just stop instance 2 on server grid02 and make sure all the services are running and try again and it will move to next screen.
And if the connect descriptor is incorrect, you have to change the $ORACLE_HOME/sysman/config/emoms.properties
oracle.sysman.eml.mntr.emdRepConnectDescriptor=(DESCRIPTION\=(ADDRESS_LIST\=(FAILOVER\=ON)(ADDRESS\=(PROTOCOL\=TCP)(HOST\=grid01)(PORT\=41521)))(CONNECT_DATA\=(SERVICE_NAME\=GRIDsv1)(SERVER\=DEDICATED)))
And retry connectivity.
On next screen “Change the OMS instance base location to end in “11.1” instead of “gc_inst”
and make sure the Middleware and the OMS base location is correct and click next. Here installer checks for space and will error out if we don’t have enough space.
On next screen enter the password of GCDomain user. Username is “weblogic” and password whatever you want. On this screen also enter the password of “nodemanager”. The password you create has to be 8 digits and contain alphanumeric. Also for “JVM diagnostic data tablespace” enter following +DATA1/REPR/DATAFILE/mgmt_ad4j.dbf. Now click next.
On next keep the ports default and click next. (Remember to take and save the snapshot of this window, will help you in checking port no.s in extending ths OMS to new server)
On next screen click “Install” and the install will start.
At the end of the installation you will be prompted to run a script for the root.
If prompted, accept all defaults and it should finish without errors and after that click ok and some Configuration Assistants will run. The process will take at least 2-3 hours so be prepared to wait:
It may fail at step oms configuration, just execute the script in the error popup window as sysman and click retry and then it should be ok. This error may be due to some invalid objects.
(The Repository Upgrade portion takes the longest)
After keeping the snapshot of last window that has the url to connect click close.
Now some post steps:
Change the job_queue_processes parameter back to the original value
SQL> sho parameter job_queue
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
job_queue_processes integer 0
SQL> alter system set job_queue_processes=20;
System altered.
SQL> sho parameter job_queue
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
job_queue_processes integer 20
Start DBMS Scheduler as sys:
SQL> execute dbms_scheduler.set_scheduler_attribute ('SCHEDULER_DISABLED','FALSE');
PL/SQL procedure successfully completed.
SQL> conn sysman
Enter password:
Connected.
SQL> execute emd_maintenance.submit_em_dbms_jobs;
Reconnect as sys and submit the gather stats jobs:
SQL> conn / as sysdba
Connected.
SQL> execute dbms_scheduler.enable('GATHER_STATS_JOB');
PL/SQL procedure successfully completed.
PL/SQL procedure successfully completed.
Update the oratab with the new agent and OMS home:
#oms10g:/ora/fs0000/app/oracle/product/10.2/oms10g:N
#agent10g:/ora/fs0000/app/oracle/product/10.2/agent10g:N
agent11g:/ora/fs0000/app/oracle/product/Middleware/oms11g:N
agent11g:/ora/fs0000/app/oracle/product/Middleware/agent11g:N
And test you can query them:
oracle : grid01 : @agent11g : /home/oracle
$ emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /ora/fs0000/app/oracle/product/Middleware/agent11g
Agent binaries : /ora/fs0000/app/oracle/product/Middleware/agent11g
Agent Process ID : 21210
Parent Process ID : 21195
Agent URL : https://grid01:3872/emd/main/
Repository URL : https://grid01:1159/em/upload
Started at : 2011-01-15 02:49:16
Started by user : oracle
Last Reload : 2011-01-22 02:57:35
Last successful upload : 2011-01-22 05:15:38
Total Megabytes of XML files uploaded so far : 37.80
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 17.72%
Last successful heartbeat to OMS : 2011-01-22 05:24:30
---------------------------------------------------------------
Agent is Running and Ready
oracle : grid01 : @oms11g : /home/oracle
$ emctl status oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
WebTier is Up
Oracle Management Server is Up
Next remove old 10g Middleware targets
Now log into your new 11.1 Grid control OMS and we’ll have to do some cleanup
In particular, remove
Some have to be removed one at a time for proper cleanup.
Other issues you may notice:
"EMGC_ADMINSERVER/FMW Welcome Page Application(11.1.0.0.0)" -Target Down [ID 1100553.1]
Apply patch 9431704 to fix this.
At the time of writing this note we had PSU 2 released a few days ago. So, apply it on top of this 11gr1 OMS. It may take 2-3 hours to apply this PSU. Before applying this PSU upgrade the OPatch to latest OPatch for 11gR1 and not 11gR2 for both OMS and Agent.
-------------------------------------------------------------------
To do a Fresh install of 11gR1 Look at posts 12,13 and 14.
Part 12 of series, Install JDK for Enterprise Manager Grid COntrol 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control.html
Part 13 of series, Install Weblogic wls10.3.2 for Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_29.html
Part 14 of series, Install Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_2064.html
To upgrade existing Enterprise Manager Grid control from 10.2.0.4/10.2.0.5 to 11.1.0.1
Part 30 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Upgrade Additional Grid Control , OMS
http://gurpartapblog.blogspot.com/2011/02/part-30-of-series-upgrade-enterprise.html
Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS
http://gurpartapblog.blogspot.com/2011/02/part-29-of-series-upgrade-enterprise.html
Part 28 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install JDK and Weblogic
http://gurpartapblog.blogspot.com/2011/02/part-28-of-series-upgrade-enterprise.html
-----------------------------------------------------------------------
Those who remember God generously help others. Those who remember God -- to them, I am forever a sacrifice. Those who remember God -- their faces are beautiful. Those who
remember God abide in peace. Those who remember God conquer their souls. Those who remember God have a pure and spotless lifestyle. Those who remember God experience all sorts of joys. Those who remember God abide near the Lord. By the Grace of the Saints, one remains awake and aware, night and day.
Subscribe to:
Posts (Atom)