Part 32 of series - Integrate Goldengate Manager Process with 11gr2 Grid Infrastructure, use GI to start and stop Mgr Process and so extract/replict

The rulers of the all the world are unhappy; one who chants the Name of the Lord becomes happy. Acquiring hundreds of thousands and millions, your desires shall not be contained. Chanting the Name of the Lord, you shall find release. By the countless pleasures of Maya, your thirst shall not be quenched. Chanting the Name of the Lord, you shall
be satisfied. Upon that path where you must go all alone, there, only the Lord’s Name shall go with you to sustain you.


Index of all the posts of Gurpartap Singh's Blog

Here are the steps I used to add Oracle Goldengate Manager process to Grid Infrastructure. After adding this only Grid Infrastructure should be used to stop/start/relocate Oracle Goldengate Manger Process. This will start/stop manger process and manager process in turn will start all the extracts/replicats using parameter AUTOSTART in manager process parameter file. Like add the following file in the end of the file as:

AUTOSTART extract *truth

This will make all the extracts ending with truth to start once manager process starts. You can configure similar thing for all the porcesses you have for extracts/replicats.


Step 1 is to add a new vip. Here are the steps:

Adding a vip.


/u02/app/11.2.0.1/grid/bin/appvipcfg create -network=1 \
-ip=10.10.2.199 \
-vipname=mvggatevip \
-user=root



with:
• GRID_HOME as the oracle home in which Oracle 11g Release 2 Grid infrastructure
components have been installed (e.g. /u02/app/11.2.0.1/grid).
• -network refers to the network number that you want to use. With Oracle Clusterware
11.2.0.1 you can find the network number using the command:
crsctl stat res -p grep -ie .network -ie subnet grep -ie name -ie subnet

Sample output is:
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/crsctl stat res -p grep -ie .network -ie subnet grep -ie name -ie subnet
NAME=ora.net1.network
USR_ORA_SUBNET=10.10.2.0
[root@rac1 ~]#

net1 in NAME=ora.net1.network indicates this is network 1, and the second line
indicates the subnet on which the VIP will be created.

10.10.2.199 - Is a ip on the above mentioned subnet and which router can support but is not allocated to any other machine and is not pingable as of now. Grid infrastrucure will bring it up and then it will become pingable.

• mvggatevip is the name of the application VIP that you will create.
Oracle recommends the use of the appvipcfg utility to define applications VIPs. The VIP is
created with a set of pre-defined settings and dependencies. Please refer to the Oracle
Clusterware documentation for further details:
http://download.oracle.com/docs/cd/E11882_01/rac.112/e10717/crschp.htm#BGBJHJHC
Oracle White Paper—Oracle GoldenGate high availability with Oracle Clusterware


Step 2:

As root, allow the Oracle Grid infrastructure software owner (e.g. oracle) to run the script
to start the VIP.

/u02/app/11.2.0.1/grid/bin/crsctl setperm resource mvggatevip -u user:oracle:r-x


Step 3:

Then, as oracle, start the VIP:

/u02/app/11.2.0.1/grid/bin/crsctl start resource mvggatevip


To validate whether the VIP is running and on which node it is running, execute:

GRID_HOME/bin/crsctl status resource mvggatevip

For example:
[root@rac1 ~]# /u02/app/11.2.0.1/grid/bin/crsctl status resource mvggatevip
NAME=mvggatevip
TYPE=app.appvip.type
TARGET=ONLINE
STATE=ONLINE on rac2

[root@rac1 ~]#

At this point you can also connect to another server in the subnet and ping the VIP's IP address.

You should get a reply from this IP address.

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ ping -c5 10.10.2.199
PING 10.10.2.199 (10.10.2.199) 56(84) bytes of data.
64 bytes from 10.10.2.199: icmp_seq=1 ttl=64 time=0.165 ms
64 bytes from 10.10.2.199: icmp_seq=2 ttl=64 time=0.168 ms
64 bytes from 10.10.2.199: icmp_seq=3 ttl=64 time=0.172 ms
64 bytes from 10.10.2.199: icmp_seq=4 ttl=64 time=0.172 ms
64 bytes from 10.10.2.199: icmp_seq=5 ttl=64 time=0.188 ms

--- 10.10.2.199 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4001ms
rtt min/avg/max/mdev = 0.165/0.173/0.188/0.008 ms

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$



Step 4:

We will use the following script to tell Grid Infrastructure to start and stop manager process I am using ACFS. These steps can also be used for NFS mount points.
Place this script at the shared location where you have installed the goldengate binaries.

I am placing at the following location:

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
84G 35G 45G 44% /
/dev/sda1 99M 16M 78M 17% /boot
tmpfs 1.6G 429M 1.2G 28% /dev/shm
/dev/asm/shared_vol1-106
3.0G 424M 2.6G 14% /u01/app/oracle/acfsmounts/data1_shared_vol1

oracle : rac1.rac.meditate.com : @crs : /home/oracle
$ cd /u01/app/oracle/acfsmounts/data1_shared_vol1

oracle : rac1.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1
$ cd script

oracle : rac1.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1/script
$ lst
total 12
drwxrwx--- 7 root oinstall 4096 Feb 10 22:32 ..
-rwxr-xr-x 1 oracle oinstall 3278 Feb 12 15:00 11gr2_gg_action.scr
drwxr-xr-x 2 oracle oinstall 4096 Feb 12 15:06 .

oracle : rac1.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1/script
$




oracle : rac1.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1/script
$ cat 11gr2_gg_action.scr
#!/bin/sh
##GGS_HOME=
export GGS_HOME=/u01/app/oracle/acfsmounts/data1_shared_vol1/goldengate
##DBFS_MOUNT_POINT=
##DBFS_FILE_SYSTEM=
##specify delay after start before checking for successful start
start_delay_secs=5
##Include the GoldenGate home in the library path to start GGSCI
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:${GGS_HOME}
##set the oracle home to the database to ensure GoldenGate will get the
##right environment settings to be able to connect to the database
export ORACLE_HOME=/u01/app/oracle/product/11.2.0.1/dbhome_1
##check_process validates that a manager process is running at the PID
##that GoldenGate specifies.
check_process () {
if ( [ -f "${GGS_HOME}/dirpcs/MGR.pcm" ] )
then
pid=`cut -f8 "${GGS_HOME}/dirpcs/MGR.pcm"`
if [ ${pid} = `ps -e grep ${pid} grep mgr cut -d " " -f2` ]
then
##manager process is running on the PID . exit success
exit 0
else
if [ ${pid} = `ps -e grep ${pid} grep mgr cut -d " " -f1` ]
then
##manager process is running on the PID . exit success
exit 0
else
##manager process is not running on the PID
exit 1
fi
fi
else
##manager is not running because there is no PID file
exit 1
fi
}
##call_ggsci is a generic routine that executes a ggsci command
call_ggsci () {
ggsci_command=$1
ggsci_output=`${GGS_HOME}/ggsci << start_dependencies="'hard(mvggatevip,ora.asm)" start_dependencies="'hard(mvggatevip)" action_script="/u01/app/oracle/acfsmounts/data1_shared_vol1/script/11gr2_gg_action.scr," check_interval="30," start_dependencies="'hard(mvggatevip,ora.asm)" stop_dependencies="'hard(mvggatevip)'">
(In my case I am using oracle user to do goldengate stuff so its not required)



Step 6:

[root@rac2 ~]# /u02/app/11.2.0.1/grid/bin/crsctl start resource ggateapp -f
CRS-2673: Attempting to stop 'mvggatevip' on 'rac2'
CRS-2677: Stop of 'mvggatevip' on 'rac2' succeeded
CRS-2672: Attempting to start 'mvggatevip' on 'rac3'
CRS-2676: Start of 'mvggatevip' on 'rac3' succeeded
CRS-2672: Attempting to start 'ggateapp' on 'rac3'
CRS-2674: Start of 'ggateapp' on 'rac3' failed
CRS-2679: Attempting to clean 'ggateapp' on 'rac3'
CRS-2681: Clean of 'ggateapp' on 'rac3' succeeded
CRS-2673: Attempting to stop 'mvggatevip' on 'rac3'
CRS-2677: Stop of 'mvggatevip' on 'rac3' succeeded
CRS-2672: Attempting to start 'mvggatevip' on 'rac2'
CRS-2676: Start of 'mvggatevip' on 'rac2' succeeded
CRS-2563: Attempt to start resource 'ggateapp' on 'rac3' has failed. Will re-retry on 'rac2' now.
CRS-2672: Attempting to start 'ggateapp' on 'rac2'
CRS-2676: Start of 'ggateapp' on 'rac2' succeeded
[root@rac2 ~]#



oracle : rac2.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1/script
$ crs_stat -t -v
Name Type R/RA F/FT Target State Host
----------------------------------------------------------------------
ggateapp clus...esource 0/1 0/0 ONLINE ONLINE rac2
mvggatevip app....ip.type 0/0 0/0 ONLINE ONLINE rac2
ora.DATA1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora....ER.lsnr ora....er.type 0/5 0/ ONLINE ONLINE rac1
ora....N1.lsnr ora....er.type 0/5 0/0 ONLINE ONLINE rac1
ora.OCR1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora.RECV1.dg ora....up.type 0/5 0/ ONLINE ONLINE rac1
ora.asm ora.asm.type 0/5 0/ ONLINE ONLINE rac1
ora.eons ora.eons.type 0/3 0/ ONLINE ONLINE rac1
ora.gsd ora.gsd.type 0/5 0/ ONLINE ONLINE rac1
ora....network ora....rk.type 0/5 0/ ONLINE ONLINE rac1
ora.ons ora.ons.type 0/3 0/ ONLINE ONLINE rac1
ora.prabgun.db ora....se.type 0/2 0/2 ONLINE OFFLINE
ora....sv3.svc ora....ce.type 0/0 0/0 ONLINE OFFLINE
ora....SM1.asm application 0/5 0/0 ONLINE ONLINE rac1
ora....C1.lsnr application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.gsd application 0/5 0/0 ONLINE ONLINE rac1
ora.rac1.ons application 0/3 0/0 ONLINE ONLINE rac1
ora.rac1.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac1
ora....SM2.asm application 0/5 0/0 ONLINE ONLINE rac2
ora....C2.lsnr application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.gsd application 0/5 0/0 ONLINE ONLINE rac2
ora.rac2.ons application 0/3 0/0 ONLINE ONLINE rac2
ora.rac2.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac2
ora....SM3.asm application 0/5 0/0 ONLINE ONLINE rac3
ora....C3.lsnr application 0/5 0/0 ONLINE ONLINE rac3
ora.rac3.gsd application 0/5 0/0 ONLINE ONLINE rac3
ora.rac3.ons application 0/3 0/0 ONLINE ONLINE rac3
ora.rac3.vip ora....t1.type 0/0 0/0 ONLINE ONLINE rac3
ora....ry.acfs ora....fs.type 0/5 0/ ONLINE ONLINE rac1
ora.scan1.vip ora....ip.type 0/0 0/0 ONLINE ONLINE rac1
ora.simar.db ora....se.type 0/2 0/1 ONLINE ONLINE rac1
ora....sv1.svc ora....ce.type 0/0 0/0 ONLINE ONLINE rac1

oracle : rac2.rac.meditate.com : @crs : /u01/app/oracle/acfsmounts/data1_shared_vol1/script
$


[root@rac2 ~]# /u02/app/11.2.0.1/grid/bin/crsctl status resource ggateapp
NAME=ggateapp
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on rac2

[root@rac2 ~]#


Logs of goldengate:

2011-02-12 15:12:07 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): start manager.
2011-02-12 15:12:12 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop er *.
2011-02-12 15:12:12 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop er *.
2011-02-12 15:12:12 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop manager.
2011-02-12 15:12:16 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): start manager.
2011-02-12 15:12:21 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop er *.
2011-02-12 15:12:21 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop er *.
2011-02-12 15:12:21 GGS INFO 399 Oracle GoldenGate Command Interpreter for Oracle: GGSCI command (root): stop manager.
2011-02-12 15:12:27 GGS INFO 330 Oracle GoldenGate Manager for Oracle, mgr.prm: Manager started (port 7909).
2011-02-12 15:12:47 GGS INFO 330 Oracle GoldenGate Manager for Oracle, mgr.prm: Manager started (port 7909).


Step 7:

If one ASM instance goes down it should not effect our manager process and for that use the following entries of ASM on every node of RAC:

ASM entry on 3 nodes should look like:

Node 1:

RAC1ASM =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = +ASM1)
)
)


Node 2:

RAC1ASM =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = +ASM2)
)
)


Node 3:

RAC1ASM =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1)(PORT = 1521))
)
(CONNECT_DATA =
(SERVICE_NAME = +ASM3)
)
)



You shall not be saved by hundreds of thousands and millions of helping hands. Chanting the Naam, you shall be lifted up and carried across. Where countless misfortunes threaten to destroy you, the Name of the Lord shall rescue you in an instant. Through countless incarnations, people are born and die. Chanting the Name of the Lord, you shall come to rest in peace. The ego is polluted by a filth which can never be washed off. The Name of the Lord erases millions of sins.