When You are on my side, Lord, what do I need to worry about? You entrusted everything to me, when I became Your slave. My wealth is inexhaustible, no matter how much I spend and consume. The 8.4 million species of beings all work to serve me. All these enemies have become my friends, and no one wishes me ill. No one calls me to account, since God is my forgiver. I have become blissful, and I have found peace, meeting with the Guru, the Lord of the Universe. All my affairs have been resolved, since You are pleased with me.
Index of all the posts of Gurpartap Singh's Blog
Purpose
This document will provide step by step document to upgrade 11g Enterprise Manager Grid Control 11g to Enterprise Manager Grid Contrlol 12c with 1- system upgrade approach. The audience of this
document should understand the install and working of the Enterprise Manager 11g as well as 12c and at least should know the high level overview of 1-System upgrade.
Documentation
For this upgrade I am following upgrade guide located at the following link:
http://docs.oracle.com/cd/E24628_01/upgrade.121/e22625.pdf
Please note due to our security compliance we cannot use these steps as it is and we have to do some teaks and played around with sudos and credential setup and went through series of notes on metalink. All is documented here.
Glossary
The following is a list of terms and abbreviations that will be used throughout this document:
OMS – Oracle Management Service
Agent – Oracle Management Agent
Also instead of user gnepal you can use any user who is part of the “dba” group on all the servers.
Overview
In 1 – System upgrade we have an outage from the point we switch agents to the time we upgrade Oracle Management Service to 12c.
The 12.1.0.1 Enterprise Manager uses the 10.3.5 WebLogic Server environment for Grid Control web application deployment instead of the 10.3.2 WLS deployment environment utilized with version 11.1, so the existing Weblogic Server domain will be extended in the process of the upgrade from 11.1.0.1 to 12.1.0.1. The installer will place an 10.3.5 WLS installation with the requisite jdk1.6.0_24 in the 1-System Upgrade method from version 11.1.0.1.
Additionally, in Enterprise Manager 12.1.0.1, there is no backwards agent version compatibility (the 12.1.0.1 agent is java-based) so all agents of interest must be converted to 12.1.0.1 to work with the 12.1.0.1 Enterprise Manager console. Likewise, versions of the OMS prior to 12.1.0.1 will not be able to work with the 12.1.0.1 agent.
Summay of the upgrade process
The typical upgrade from 11.1.0.1 with the 1-System Method is this sequence of steps:
1. The 11.1.0.1 Enterprise Manager console is patched with the capability to manage the pre-upgrade actions for the OMS and all agents intended for upgrade in the monitored enterprise.
2. In 12.1.0.1, the agent has been defined as an agent core, which represents only the agent and host targets, and plug-ins, which add the ability to monitor all other targets types, including databases, application servers, etc. For any target host, the agent core is required as a minimum agent configuration for agent and host monitoring, but it will require additional plug-ins for any other hosted targets. This 12.1.0.1 agent core and plug-in software must be obtained and staged for deployment through the upgrade-patched 11.1.0.1 OMS.
3. In the 1-System upgrade method from version 11.1.0.1, 12.1.0.1 agents and the associated plug-in software must be pre-deployed to all target hosts of interest through the Upgrade Console in 11.1.0.1 Enterprise Manager using the Enterprise Manager job system. Any agents required after OMS upgrade will necessarily be introduced as new installations, without prior monitoring history.
4. The pre-deployed agents are evaluated and a Health Report is generated in a second job action initiated from the 11.1.0.1 Upgrade Console. The Health report requires user verification in order to proceed.
5. At a time close to the planned OMS/repository upgrade, the pre-deployed and verified 12.1.0.1 agents are switched over in still another job initiated form the 11.1.0.1 Upgrade Console. In switch over, the 12.1.0.1 agents are activated, ready to report into the 12.1.0.1 OMS when it is available, and the original, pre-12c agents are disabled..
6. The OMS and repository are upgraded to 12.1.0.1 in the final step of the 1-System upgrade. This upgrade consists of a 12.1.0.1 OMS installation on the same host as the original OMS, using different Middleware and Oracle Home directories. The 12.1.0.1 OMS points to the same repository as the original OMS, and the installer will upgrade the repository for 12.1.0.1 compatibility. This action disables the 11.1.0.1 OMS for further use, and it may be decommissioned once it is disabled
In the transition from 11.1.0.1 to 12.1.0.1, the deployment environment changes from the 10.3.2 WebLogic Server to the 10.3.5 WebLogic Server. The 10.3.5 WebLogic Server requires a platform-compatible jdk in version 1.6.0_24 for installation. The upgrade installer will install both the 1.6.0_24 jdk and the 10.3.5 WLS in the Linux versions in a new Middleware directory location determined by the user, and an additional Oracle Home directory for the 12.1.0.1 OMS will be placed in that location. The 12.1.0.1 agent location was independently was already determined by the user in the deployment action executed in the 11.1.0.1 Upgrade Console.
7. As the 12.1.0.1 OMS is started up during configuration in the upgrade installation, the pre-deployed 12.1.0.1 agents begin reporting automatically as the OMS begins to ping them, initiating monitoring/management in the 12.1.0.1 environment.
Required Software
The required Enterprise Manager software components for upgrade to 12.1.0.1:
- Upgrade Console patch (13503844) for the 11.1.0.1 OMS
- 12.1.0.1 agent base product installation source that can be deployed
- Plug-in files
The above three can be downloaded form:
http://www.oracle.com/technetwork/oem/grid-control/downloads/oem-upgrade-console-502238.html
- 12.1.0.1 Enterprise Manager base product installer
OEM12c software mentioned above can be downloaded from:
http://www.oracle.com/technetwork/oem/grid-control/downloads/linuxx8664soft-085949.html
Patch OEM11g OMS to create Upgrade Console
You need to apply patch 13503844 to all 11.1.0.1 Oracle Management Services in your environment. We had two and I applied this patch. To apply this patch just follow the readme accompanies with this patch.
Once the patch application and post-installation scripts have been completed successfully, start the OMS and view the Deployments page by accessing its tab. Note the presence of a new feature, the Enterprise Manager 12c Upgrade Console at the top of the feature stack under the Upgrade heading.
Note the state of the OMS installation prior to patching shows no evidence of the upgrade console in the Deployments page.
As described in the README.txt file, for upgrade from version 11.1.0.1, run the first two post-patch rcuJDBCEngine scripts with sys credentials and the third with sysman credentials.
These scripts are required so the Upgrade Console will be functional. If these scripts are not run, the link to the Upgrade Console will be available in the Deployments main page, but an error will be returned when it is accessed.
Once the patch was applied our lsinventory will look like following:
17:20 : oracle : xxxxxx : @OMS11G : /opt/app/oracle/product/Middleware/oms11g/OPatch
$ ./opatch lsinventory
Oracle Interim Patch Installer version 11.1.0.9.4
Copyright (c) 2011, Oracle Corporation. All rights reserved.
Oracle Home : /opt/app/oracle/product/Middleware/oms11g
Central Inventory : /opt/app/oracle/product/oraInventory
from : /opt/app/oracle/product/Middleware/oms11g/oraInst.loc
OPatch version : 11.1.0.9.4
OUI version : 11.1.0.8.0
Log file location : /opt/app/oracle/product/Middleware/oms11g/cfgtoollogs/opatch/opatch2012-02-15_17-20-43PM_1.log
OPatch detects the Middleware Home as "/opt/app/oracle/product/Middleware"
Lsinventory Output file location : /opt/app/oracle/product/Middleware/oms11g/cfgtoollogs/opatch/lsinv/lsinventory2012-02-15_17-20-43PM.txt
--------------------------------------------------------------------------------
Installed Top-level Products (1):
Oracle Enterprise Manager Grid Console 11.1.0.1.0
There are 1 products installed in this Oracle Home.
Interim patches (4) :
Patch 13503844 : applied on Fri Jan 27 14:31:15 PST 2012
Unique Patch ID: 14378035
Created on 16 Dec 2011, 02:16:47 hrs PST8PDT
Bugs fixed:
10044087, 13503844
Patch 11727299 : applied on Thu May 05 17:09:58 PDT 2011
Unique Patch ID: 13669556
Created on 11 Mar 2011, 08:29:30 hrs PST8PDT
Bugs fixed:
9804442, 10122270, 9930146, 9576468, 10108019, 9535382, 9928480, 9543582
9276193, 10278756, 9892455, 9476313, 9911851, 9506062, 9876632, 9386076
9803798, 10043790, 9646681, 9965939, 9500255, 9390531, 9662869, 9462098
9544428, 9747671, 9489355, 9706552, 9562462, 9570911, 10638043, 9937784
7340407, 9723689, 9575427, 10204587, 9935898, 9734399, 9504581, 9558551
9446422, 9874327, 9586285, 9548105, 9313608, 9534200, 9684768, 9341486
10122223, 10307544, 9849222, 9537948, 10118817, 9919614, 9903280, 9888852
9342290, 10410743, 9151748, 9221090, 10213026, 9952713, 9491872, 9559264
10015330, 10036568, 10042225
Patch 9367763 : applied on Wed Oct 27 20:33:08 PDT 2010
Unique Patch ID: 12255254
Created on 1 Mar 2010, 22:34:50 hrs PST8PDT
Bugs fixed:
9367763
Patch 9207217 : applied on Wed Oct 27 20:33:01 PDT 2010
Unique Patch ID: 12151251
Created on 27 Jan 2010, 15:49:36 hrs US/Pacific
Bugs fixed:
9207217
--------------------------------------------------------------------------------
OPatch succeeded.
17:20 : oracle : xxxxx0 : @OMS11G : /opt/app/oracle/product/Middlewa
OMS/Repository Upgrade Preparation: Managing Agent Software
8.1 Accessing the Upgrade Console
The view from the Upgrade Console allows access to the Overview link, which takes the user to a description of the upgrade process for reference, the Agent Upgrade Status view, which summarizes the state of the agent deployment and preparation activity and the Manage Software view, which allows association of agent core product and plug-in install stages with the console for deployment purposes.
Note that the steps involved in the upgrade will change with the upgrade type selection that must be made before continuing. Described here is the 1-System upgrade (same host).
Note also the presence of the Refresh Agent List button which will allow recognition of any agents discovered
After the Upgrade Console patch was applied. The list is not automatically updated, so refreshing from the Upgrade Console will be required in the event of new agent additions.
When you click on the upgrade console you will see the following screen:
8.2 Manage Software to Get an Appraisal of the Enterprise Targets
The task sequence in the conversion to 12.1.0.1 Enterprise Manager monitoring in the 1-System method is to deploy and configure 12.1.0.1 agents to all monitored hosts, evaluate the health of the monitoring environment and switch over to the newly deployed 12.1.0.1 agents. To prepare for the deployment of 12.1.0.1 agents, staged agent software must be recognized by the Upgrade Console.
8.2.1 Download and Deploy Agent Core and required Plugins
On the OMS server download and deploy the plugins and agent core software at the same location.
Download them from following location:
http://www.oracle.com/technetwork/oem/grid-control/downloads/oem-upgrade-console-502238.html
and deploy on both nodes at this location, please note that zip file of agent core is not unzipped:
/m2/scratch/install/12c/software/AgentCore_226
Here is ls –lart looks like in this directory on both servers:
18:41 : oracle : xxxx : @OMS11G : /m2/scratch/install/12c/software/AgentCore_226
$ ls -lart
total 1433092
-rw-r--r-- 1 oracle oinstall 8980435 Feb 9 15:42 12.1.0.1.0_oracle.sysman.mos_2000_0.opar
-rw-r--r-- 1 oracle oinstall 297231 Feb 9 15:42 12.1.0.1.0_oracle.sysman.emrep_2000_0.opar
-rw-r--r-- 1 oracle oinstall 3925371 Feb 9 15:42 12.1.0.1.0_oracle.em.soee_2000_0.opar
-rw-r--r-- 1 oracle oinstall 3505186 Feb 9 15:43 12.1.0.1.0_oracle.sysman.xa_2000_0.opar
-rw-r--r-- 1 oracle oinstall 621251038 Feb 9 15:43 12.1.0.1.0_oracle.sysman.db_2000_0.opar
-rw-r--r-- 1 oracle oinstall 8780464 Feb 9 15:43 12.1.0.1.0_oracle.sysman.vt_2000_0.opar
-rw-r--r-- 1 oracle oinstall 580159 Feb 9 15:43 12.1.0.1.0_oracle.em.sidb_2000_0.opar
-rw-r--r-- 1 oracle oinstall 28076693 Feb 9 15:43 12.1.0.1.0_oracle.sysman.emfa_2000_0.opar
-rw-r--r-- 1 oracle oinstall 2246995 Feb 9 15:43 12.1.0.1.0_oracle.sysman.ssa_2000_0.opar
-rw-r--r-- 1 oracle oinstall 495016 Feb 9 15:43 12.1.0.1.0_oracle.sysman.oh_2000_0.opar
-rw-r--r-- 1 oracle oinstall 6555873 Feb 9 15:43 12.1.0.1.0_oracle.sysman.empa_2000_0.opar
-rw-r--r-- 1 oracle oinstall 2775862 Feb 9 15:43 12.1.0.1.0_oracle.sysman.emct_2000_0.opar
-rw-r--r-- 1 oracle oinstall 440206388 Feb 9 15:43 12.1.0.1.0_oracle.sysman.emas_2000_0.opar
-rw-r--r-- 1 oracle oinstall 499849 Feb 9 15:43 12.1.0.1.0_oracle.sysman.csa_2000_0.opar
-rw-r--r-- 1 oracle oinstall 106452968 Feb 9 15:43 12.1.0.1.0_oracle.sysman.beacon_2000_0.opar
-rw-r--r-- 1 oracle oinstall 607976 Feb 9 15:43 12.1.0.1.0_oracle.em.ssad_2000_0.opar
-rwxr-xr-x 1 oracle oinstall 230699506 Feb 9 15:43 12.1.0.1.0_AgentCore_226.zip
drwxr-xr-x 3 oracle oinstall 4096 Feb 9 15:43 .
drwxr-xr-x 8 oracle oinstall 4096 Feb 10 11:18 ..
drwxr----- 18 oracle oinstall 4096 Feb 10 16:15 plugin_bin
18:41 : oracle : xxxx : @OMS11G : /m2/scratch/install/12c/software/AgentCore_226
$
Recognizing the agent upgrade software
To associate the 12.1.0.1 agent software with the Upgrade Console, select the Manage Software link in the Upgrade Console. On initial access, without any software association, the Manage Software view will only show that agents are not upgradeable. This status should change as valid staged agent deployment software is recognized by the Upgrade Console. Note the status of the agent and plug-in software in the Agent Software and Plug-in Software tables near the bottom of the page.
In the Software Location field, enter the file path of the directory containing the agent core (i.e., 12.1.0.0.0_AgentCore_226.zip for the Linux x86_64 agent) and the plug-in files (i.e. 12.1.0.0.0_oracle.sysman.db_2000_0.opar for the generic db plug-in) on the OMS.
The screen will look like this:
Validating the Agent Software Location
Select Validate on the Manage Software page when the file path has been entered and the Upgrade Console will recognize a properly staged file set, associate this file set with the appropriate agents and update the view of the agent status to show ability to upgrade.
Click Validate and you will see the screen below, make sure that you check “Click if the software location is shared across all OMSes”
This should how 100% green but if not we can continue with upgrade with agents that are ok with plugins but which need plugins we have to install and deploy them. We had problem with exadata plugins and Oracle thinks it’s a bug and they are working on it.
Validation Results
Note the change in agent and target status shown in the pie charts and the tables on the Manage Software page. Review the results, especially the agents or targets in not upgradeable status, and correct any reported shortcomings, if the agent or target is one that must be monitored in Enterprise Manager 12.1.0.1. Validate once more, once the corrections have been made to accumulate the corrected agents and targets in the upgradeable category. Any change to the staged software will require re-validation.
The file location of the designated agent software will have added a /plugin_bin directory at the same level as the agent file, housing the plug-in software extracted in the validation step.
Note that an agent core .zip file is required for every agent platform represented in the current monitoring arrangement.
Push and configure Oracle Management Agent Software to Targets
Pre-step is to do host credentials test. As here in Sony we don’t have Oracle user password so, I had to work with security team to get the push of agents working for me.
There are 5 pre-steps and 3 of them are for SA.
1. Make sure we have package param_devel installed on the target server - SA
2. Take the backup of existing file /etc/pam.d/emagent if it exists and replace it with the contents of /etc/pam.d/sshd. If file /etc/pam.d/emagent doesn’t exist just create a new one as a copy of /etc/pam.d/sshd - SA
3. Update sudoers file as given in Apendix
4. Update .profile under home directory of user ORACLE and add following entry - DBA
export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH
5. Edit file commonenv in directory $AGENT_HOME/bin and in the end of that file add following text in blue
if [ "$LD_LIBRARY_PATH" = "" ] ; then
LD_LIBRARY_PATH=/usr/lib64/libpamc.so
else
LD_LIBRARY_PATH=/usr/lib64/libpamc.so:$LD_LIBRARY_PATH
fi
Set Manage Privilege Delegation Setting Templates
Go to Preferences -> Preferred Credentials (left part of Screen) -> Clcik Set Crendentials blue icon in front of Host -> In search bar find the target server
Now you will see the following screen as:
Here click url “Manage Privilege Delegation Settings” and search for your target host again and you will see the following screen:
Click on the pen on most right of the line to edit the privileges as and then on next scren click in a radio button in front of “Sudo” and then clcik update and you will see the following screen.
Here in “Sudo Command” box copy paste the following line in blue:
/usr/bin/sudo -u %RUNAS% %COMMAND%
Note – Here /usr/bin/sudo is the location of sudo binaries on your Operating system
And your screen now looks like following:
Click update now and on next screen click “Yes” and you will see the following, i.e. under type it will say “Sudo::
9.2 Set Prefered Credentials of the server
Go to Preferences -> Preferred Credentials (left part of Screen) -> Clcik Set Crendentials blue icon in front of Host -> In search bar find the target server
Now you will see the following screen as:
Fill the form as:
Normal Username – gnepal
Normal Password –
Run as - oracle
Privileged User - gnepal
Privileged Password -
Run as - root
Now your screen looks like following screen:
Now click test and check the result as “Credentials successfully verified for xxxxxxx:
Push the Agent from Upgrade console
Click Deployments ->
Click “Enterprise Manager 12c Upgrade Console” under heading upgrade and click “Target Upgradability Status” under heading “Other Links”
And you will see following and search for host “lv-dnp-db106.sonynei.net” or where you want to push the software:
You will see the following, i.e. target is upgradable:
Now on the home screen of Upgrade console under Deploymets -> Enterprise Manager 12c Upgrade Console, Click “Deploy and Configure Agents” under “Agent Upgrade Steps” you will see the following screen:
Add agent that you want to upgrade by clicking add:
Type host name of the target server and in our case its “xxxxx” and click go.
Check the box in front of the agent name and click select and on your orignal screen you will that the agent has been select, just click the check box in front of it.
Under “Agent Base Directory” and “Agent Instance Name” make sure the path is:
/opt/app/oracle/product/agent12c/root.sh – Sony Standards
and not
/opt/app/oracle/product/agent12g/root.sh
and you will have the following screen:
Under “Agent Credentials” click “Override Oracle Home Preferred Credentials” and the page will refresh and you will again have to do the check mark in front of agent name. Now fill in the “User Name” as gnepal and also enter its password twice and check “SUDO” under “Run Privilege” and in “Run as” fill “oracle” . The screen will look like following, just match your screen.
Click “NEXT” and you will see the following screen , just fill in “User Name” gnepal and enter the password twice. Make sure radio button in front of SUDO is checked and “Run As” is greyed out and is “root”
Now click Submit and you will get the follwoing screen :
Click on the job and you will see the following:
Just keep refreshing the page and see the progress:
Just keep watching it till the status changes from “Running” to “Succeeded”.
Make sure the status of all the componenets is also “Succeeded” and if any is failed please fix it and then proceed from this point onwards.
Click on “Sign off health Report of Deployed Agents” and you will see the following screen:
Select one agent at a time and click view detailed report and then on the same page go through the report. If there are ptroblems it will show warnings in red. If there are warnings just fix them and comeback and view detailed report again and if all is well just click “Verify and Sing Off Report”.
Report in beginning looks like:
And in end looks like:
Once all have been verified you will see something like following:
Switch agents
Outage for the agent will start here. Just click and select the agent and fill the form as shown below and submit the job.
Click submit and you will get the following:
and
OMS upgraded is continued in next part.
Why, O mind, do you plot and plan, when the Dear Lord Himself provides for your care?
From rocks and stones He created living beings; He places their nourishment before them. O my Dear Lord of souls, one who joins the Sat Sangat, the True Congregation, is saved. By Guru's Grace, the supreme status is obtained, and the dry wood blossoms forth again in lush greenery. Pause Mothers, fathers, friends, children and spouses no one is the support of anyone else. For each and every person, our Lord and Master provides sustenance. Why are you so afraid, O mind? The flamingoes fly hundreds of miles, leaving their young ones behind. Who feeds them, and who teaches them to feed themselves? Have you ever thought of this in your mind? All the nine treasures, and the eighteen supernatural powers are held by our Lord and Master in the Palm of His Hand. Servant Nanak is devoted, dedicated, forever a sacrifice to You, Lord. Your Expanse has no limit, no boundary.
Click on the job name and then you will get the following page:
Once all Switchover is done we will have a following screens:
No comments:
Post a Comment