Thursday, February 3, 2011

Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS

The remembrance of God is the highest and most exalted of all. In the remembrance of God, many are saved. In the remembrance of God, thirst is quenched. In the remembrance of God, all things are known. In the remembrance of God, there is no fear of death. In the remembrance of God, hopes are fulfilled. In the remembrance of God, the filth of the mind is removed. The Ambrosial Naam, the Name of the Lord, is absorbed into the heart. God abides upon the tongues of His Saints.




Index of all the posts of Gurpartap Singh's Blog


----------------------------------------------------------

To do a Fresh install of 11gR1 Look at posts 12,13 and 14.

Part 12 of series, Install JDK for Enterprise Manager Grid COntrol 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control.html

Part 13 of series, Install Weblogic wls10.3.2 for Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_29.html

Part 14 of series, Install Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_2064.html



To upgrade existing Enterprise Manager Grid control from 10.2.0.4/10.2.0.5 to 11.1.0.1


Part 30 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Upgrade Additional Grid Control , OMS
http://gurpartapblog.blogspot.com/2011/02/part-30-of-series-upgrade-enterprise.html

Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS
http://gurpartapblog.blogspot.com/2011/02/part-29-of-series-upgrade-enterprise.html

Part 28 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install JDK and Weblogic
http://gurpartapblog.blogspot.com/2011/02/part-28-of-series-upgrade-enterprise.html


---------------------------------------------------------

INSTALLING and Upgrading GRID CONTROL

Grid Control 11.1.0.1.0 Install and Upgrade (from 10.2.0.4+)

I have already downloaded and extracted Grid Control 11g for Sun64 and placed it in the software directory at “/home/oracle/software/GridContol/11.1” After unzipping the downloaded files it will be approximately 4.7 GB

Prerequisites :
Your existing OMS must be 10.2.0.4 or higher. If not, you cannot upgrade to 11g until it’s at least 10.2.0.4. The first step is to stop all OMS servers and upgrade 1 OMS first as this will also upgrade the repository.
You must have at LEAST 8.4 GB free to install 11g OMS.
Verify the OS package prerequisites at: http://download.oracle.com/docs/cd/E11857_01/install.111/e15838/appdx_package_reqs.htm#BABIJAJB
For Solaris 10 we only needed: SUNWbtool

Database Init parameters:
cluster_database
(Only for Oracle RAC Database) TRUE

Compatible 10.2.0.1.0 or higher
max_commit_propogation_delay
(Only for Oracle RAC Database) 0

remote_login_passwordfile SHARED or EXCLUSIVE
statistics_level TYPICAL
timed_statistics TRUE
undo_management AUTO
workarea_size_policy AUTO
db_block_size 8192 or greater
job_queue_processes (For Installation) 10 or greater
(For Upgrade) If you are upgrading Enterprise Manager 10g Grid Control Release 4 (10.2.0.4) or higher, then ensure that you set the value to 0 before upgrading. And after upgrading, reset it to the original value that existed before upgrading the product.
log_buffer 10485760 or greater
open_cursors 300 or greater
Processes 500 or greater
session_cached_cursors 200 or greater


Verify the database has the Partitioning option enabled. If this is an upgrade, it would already be there.

SQL> select value from v$option where parameter = 'Partitioning';

VALUE
----------------------------------------------------------------
TRUE

Ensure all SYSMAN objects are valid

SQL> select object_name, object_type from all_objects where owner='SYSMAN' and status <> 'VALID';

no rows selected

If there are invalid objects, recompile them.

@admin_recompile_invalid.sql SYSMAN

Verify all SYS objects are valid:

SQL> select object_name, object_type from all_objects where status<>'VALID' and object_name like 'DBMS%';

no rows selected

If there are invalid objects, also compile them.

Get the value of the current job queue processes in all the instances and make a note of it as we will be changing it temporarily:

SQL> select a.instance_name as sid, b.value as jobqueue from gv$instance a, gv$parameter b where a.inst_id = b.inst_id and b.name='job_queue_processes';

SID
----------------
JOBQUEUE
--------------------------------------------------------------------------------
REPR2
20

REPR1
20



Here starts the downtime.

First, verify the repository database is not in QUIESCE mode:

SQL> select active_state from v$instance;

ACTIVE_ST
---------
NORMAL

Set job_queue_processes to 0 during the upgrade and remove the em jobs (we’ll add them back later).
Connect to the repository as SYSMAN.

SQL> conn sysman
Enter password:
Connected.
SQL> execute emd_maintenance.remove_em_dbms_jobs;

PL/SQL procedure successfully completed.

SQL> alter system set job_queue_processes=0;

System altered.

SQL> commit;

Commit complete.

SQL>

Ensure there are no active jobs:

SQL> select l.id2 job, l.sid, to_char(last_date, 'DD-MON-YYYY:HH24.MI.SS') last_date, to_char(this_date, 'DD-MON-YYYY:HH24.MI.SS') this_date, l.inst_id instance from sys.job$ j, gv$lock l where l.type = 'JQ' and j.job (+) = l.id2 order by 5, 4;

no rows selected

Disable dbms scheduler jobs:
SQL> execute dbms_scheduler.set_scheduler_attribute('SCHEDULER_DISABLED','TRUE');

PL/SQL procedure successfully completed.

Verify (wait if necessary) until there are no more running jobs

SQL> select count(*) from dba_scheduler_running_jobs where SLAVE_PROCESS_ID IS NOT NULL;

COUNT(*)
----------
0


Log in as SYS and stop the jobs

SQL> conn / as sysdba
Connected.
SQL> execute dbms_scheduler.disable('GATHER_STATS_JOB',TRUE);

PL/SQL procedure successfully completed.

SQL> execute dbms_scheduler.stop_job('GATHER_STATS_JOB',TRUE);
BEGIN dbms_scheduler.stop_job('GATHER_STATS_JOB',TRUE); END;

*
ERROR at line 1:
ORA-27366: job "SYS.GATHER_STATS_JOB" is not running
ORA-06512: at "SYS.DBMS_ISCHED", line 164
ORA-06512: at "SYS.DBMS_SCHEDULER", line 483
ORA-06512: at line 1

Ensure SYSMAN and DBSNMP users have EXECUTE on DBMS_RANDOM.

SQL> GRANT EXECUTE ON DBMS_RANDOM TO SYSMAN;

SQL> GRANT EXECUTE ON DBMS_RANDOM TO DBSNMP;

Now stop all OMS servers even if you are upgrading just one.

oracle : grid01 : @oms10g : /home/oracle/software
$ $ORACLE_HOME/opmn/bin/opmnctl stopall
opmnctl: stopping opmn and all managed processes...


oracle : grid01 : @oms10g : /home/oracle/software
$ emctl stop oms
Oracle Enterprise Manager 10g Release 5 Grid Control
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Oracle Management Server is Down.
OPMN may not be up.


Now stop on othernode and if you have more oms servers, stop them too.

oracle : grid02 : @oms10g : /home/oracle/software
$ $ORACLE_HOME/opmn/bin/opmnctl stopall
opmnctl: stopping opmn and all managed processes...

oracle : grid02 : @oms10g : /home/oracle/software
$ emctl stop oms
Oracle Enterprise Manager 10g Release 5 Grid Control
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Oracle Management Server is Down.
OPMN may not be up.


Create a rollback point for OMS repository + binaries

Actually grid control has so many bugs and I want to have my own backout plan. I am creating a flashback point in the repository and back up the OMS binaries and agent binaries as these three components will be upgraded.

Tar up the binaries where there is space, you will need at least 5-6 GB.

oracle : grid01 : @oms10g : /home/oracle/software/OMS10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/oms10g

Do the same command on the other server/servers. I only have two oms servers so will do only on servers grid01/grid02.

oracle : grid02 : @oms10g : /home/oracle/software/OMS10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/oms10g


Also tar the agent binaries as:
oracle : grid01 : @oms10g : /home/oracle/software/Agent10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/agnet10g

Do the same command on the other server/servers. I only have two oms servers so will do only on servers grid01/grid02.

oracle : grid02 : @oms10g : /home/oracle/software/ Agent10g_backup
$ tar cvf oms10g.tar /ora/fs0000/app/oracle/product/10.2/agent10g



Now create a guaranteed flashback point on database level. Here we are enabling flashback in the database too.

SQL> select flashback_on from v$database;

FLASHBACK_ON
------------------
NO

SQL> show parameter recovery

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest string
db_recovery_file_dest_size big integer 0

SQL> alter system set db_recovery_file_dest_size=30G;

System altered.

SQL> alter system set db_recovery_file_dest="+ RECV1 " ;

System altered.

oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$ srvctl stop database -d REPR

Log back in and startup mount + start flashback

SQL> startup mount
ORACLE instance started.

Total System Global Area 3758096384 bytes
Fixed Size 2045064 bytes
Variable Size 1342180216 bytes
Database Buffers 2399141888 bytes
Redo Buffers 14729216 bytes
Database mounted.
SQL> alter database flashback on;

Database altered.

SQL> shutdown immediate

SQL> exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options

oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$ srvctl start database -d REPR

oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$ srvctl start service -d REPR

oracle : grid01 : @REPR1 : /home/oracle/software/ Agent10g_backup
$sqlplus / as sysdba
SQL> create restore point before_OMS11g_upgrade guarantee flashback database;

Restore point created.

Check that everything is enabled:

SQL> select FLASHBACK_ON from v$database;

FLASHBACK_ON
------------------
YES

SQL> ALTER SYSTEM SWITCH LOGFILE;

System altered.

SQL> select estimated_flashback_size,flashback_size from v$flashback_database_log;

ESTIMATED_FLASHBACK_SIZE FLASHBACK_SIZE
------------------------ --------------
0 31866880






Now lets Install/upgrade Grid Control 11g

Now stop the Management agent on the OMS you are upgrading. I am doing that on first node i.e. grid01.

. oraenv
agent10g

oracle : grid01 : @agent10g : /home/oracle/software
$ emctl stop agent
Oracle Enterprise Manager 10g Release 5 Grid Control 10.2.0.5.0.
Copyright (c) 1996, 2009 Oracle Corporation. All rights reserved.
Agent is Not Running

Now we will install.

Check your DISPLAY property is set and start your X Window app.

oracle : grid01 : @agent10g : /home/oracle/software
$ env |grep DISPLAY
DISPLAY=192.168.1.140:0.0

cd /home/oracle/software/GridContol/11.1

oracle : grid01 : @agent10g : /home/oracle/software/GridContol/11.1
$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 150 MB. Actual 87354 MB Passed
Checking swap space: must be greater than 150 MB. Actual 94539 MB Passed
Checking monitor: must be configured to display at least 256 colors. Actual 16777216 Passed
...
oracle : grid01 : @agent10g : /home/oracle/software/GridContol/11.1
$ ./runInstaller

On first screen uncheck “I wish to receive security updates via MY Oracle Support” and click next.
On next pop up click Yes to “Do you wish to remain uninformed of critical security issues …..”
On next screen make sure “Skip Software Updates is clicked and click next.
On next screen make sure “Upgrade to Enterprise Manager 11g” and click next.
On next screen make sure the current agent/oms base and version is correct and is checked/selected and click next.
There are prerequisites on next screen and make sure all succeed and click next.

At next screen if it doesn’t accept the sys password and error at invalid username/password, just stop instance 2 on server grid02 and make sure all the services are running and try again and it will move to next screen.
And if the connect descriptor is incorrect, you have to change the $ORACLE_HOME/sysman/config/emoms.properties

oracle.sysman.eml.mntr.emdRepConnectDescriptor=(DESCRIPTION\=(ADDRESS_LIST\=(FAILOVER\=ON)(ADDRESS\=(PROTOCOL\=TCP)(HOST\=grid01)(PORT\=41521)))(CONNECT_DATA\=(SERVICE_NAME\=GRIDsv1)(SERVER\=DEDICATED)))

And retry connectivity.

On next screen “Change the OMS instance base location to end in “11.1” instead of “gc_inst”
and make sure the Middleware and the OMS base location is correct and click next. Here installer checks for space and will error out if we don’t have enough space.
On next screen enter the password of GCDomain user. Username is “weblogic” and password whatever you want. On this screen also enter the password of “nodemanager”. The password you create has to be 8 digits and contain alphanumeric. Also for “JVM diagnostic data tablespace” enter following +DATA1/REPR/DATAFILE/mgmt_ad4j.dbf. Now click next.

On next keep the ports default and click next. (Remember to take and save the snapshot of this window, will help you in checking port no.s in extending ths OMS to new server)
On next screen click “Install” and the install will start.

At the end of the installation you will be prompted to run a script for the root.
If prompted, accept all defaults and it should finish without errors and after that click ok and some Configuration Assistants will run. The process will take at least 2-3 hours so be prepared to wait:

It may fail at step oms configuration, just execute the script in the error popup window as sysman and click retry and then it should be ok. This error may be due to some invalid objects.
(The Repository Upgrade portion takes the longest)
After keeping the snapshot of last window that has the url to connect click close.


Now some post steps:

Change the job_queue_processes parameter back to the original value

SQL> sho parameter job_queue

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
job_queue_processes integer 0

SQL> alter system set job_queue_processes=20;

System altered.

SQL> sho parameter job_queue

NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
job_queue_processes integer 20

Start DBMS Scheduler as sys:

SQL> execute dbms_scheduler.set_scheduler_attribute ('SCHEDULER_DISABLED','FALSE');

PL/SQL procedure successfully completed.

SQL> conn sysman
Enter password:
Connected.

SQL> execute emd_maintenance.submit_em_dbms_jobs;

Reconnect as sys and submit the gather stats jobs:

SQL> conn / as sysdba
Connected.
SQL> execute dbms_scheduler.enable('GATHER_STATS_JOB');

PL/SQL procedure successfully completed.



PL/SQL procedure successfully completed.


Update the oratab with the new agent and OMS home:

#oms10g:/ora/fs0000/app/oracle/product/10.2/oms10g:N
#agent10g:/ora/fs0000/app/oracle/product/10.2/agent10g:N
agent11g:/ora/fs0000/app/oracle/product/Middleware/oms11g:N
agent11g:/ora/fs0000/app/oracle/product/Middleware/agent11g:N

And test you can query them:

oracle : grid01 : @agent11g : /home/oracle
$ emctl status agent
Oracle Enterprise Manager 11g Release 1 Grid Control 11.1.0.1.0
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
---------------------------------------------------------------
Agent Version : 11.1.0.1.0
OMS Version : 11.1.0.1.0
Protocol Version : 11.1.0.0.0
Agent Home : /ora/fs0000/app/oracle/product/Middleware/agent11g
Agent binaries : /ora/fs0000/app/oracle/product/Middleware/agent11g
Agent Process ID : 21210
Parent Process ID : 21195
Agent URL : https://grid01:3872/emd/main/
Repository URL : https://grid01:1159/em/upload
Started at : 2011-01-15 02:49:16
Started by user : oracle
Last Reload : 2011-01-22 02:57:35
Last successful upload : 2011-01-22 05:15:38
Total Megabytes of XML files uploaded so far : 37.80
Number of XML files pending upload : 0
Size of XML files pending upload(MB) : 0.00
Available disk space on upload filesystem : 17.72%
Last successful heartbeat to OMS : 2011-01-22 05:24:30
---------------------------------------------------------------
Agent is Running and Ready

oracle : grid01 : @oms11g : /home/oracle
$ emctl status oms
Oracle Enterprise Manager 11g Release 1 Grid Control
Copyright (c) 1996, 2010 Oracle Corporation. All rights reserved.
WebTier is Up
Oracle Management Server is Up

Next remove old 10g Middleware targets

Now log into your new 11.1 Grid control OMS and we’ll have to do some cleanup

In particular, remove

_home
_OC4J_EM
_OC4J_EMPROV
_HTTP Server
Web Cache.

Some have to be removed one at a time for proper cleanup.

Other issues you may notice:

"EMGC_ADMINSERVER/FMW Welcome Page Application(11.1.0.0.0)" -Target Down [ID 1100553.1]

Apply patch 9431704 to fix this.


At the time of writing this note we had PSU 2 released a few days ago. So, apply it on top of this 11gr1 OMS. It may take 2-3 hours to apply this PSU. Before applying this PSU upgrade the OPatch to latest OPatch for 11gR1 and not 11gR2 for both OMS and Agent.

-------------------------------------------------------------------

To do a Fresh install of 11gR1 Look at posts 12,13 and 14.

Part 12 of series, Install JDK for Enterprise Manager Grid COntrol 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control.html

Part 13 of series, Install Weblogic wls10.3.2 for Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_29.html

Part 14 of series, Install Enterprise Manager Grid Control 11gR1
http://gurpartapblog.blogspot.com/2010/09/install-oracle-install-grid-control_2064.html



To upgrade existing Enterprise Manager Grid control from 10.2.0.4/10.2.0.5 to 11.1.0.1


Part 30 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Upgrade Additional Grid Control , OMS
http://gurpartapblog.blogspot.com/2011/02/part-30-of-series-upgrade-enterprise.html

Part 29 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install and upgrade Grid Control, OMS
http://gurpartapblog.blogspot.com/2011/02/part-29-of-series-upgrade-enterprise.html

Part 28 of series - Upgrade Enterprise Manager Grid Control from 10.2.0.5 to 11gr1 (11.1.0.1) on Solairs/Unix, Install JDK and Weblogic
http://gurpartapblog.blogspot.com/2011/02/part-28-of-series-upgrade-enterprise.html


-----------------------------------------------------------------------

Those who remember God generously help others. Those who remember God -- to them, I am forever a sacrifice. Those who remember God -- their faces are beautiful. Those who
remember God abide in peace. Those who remember God conquer their souls. Those who remember God have a pure and spotless lifestyle. Those who remember God experience all sorts of joys. Those who remember God abide near the Lord. By the Grace of the Saints, one remains awake and aware, night and day.


1 comment:

  1. Great ... Excellent Quotes from Bani ..What is your email id ?

    ReplyDelete