Monday, October 31, 2016

13.1 to 13.2 Agent upgrade and cleanup OLD agents from OEM 13c

AGENTS UPGRADE:

Setup à “Manage Cloud Control” à “Upgrade Agents”









Click ‘OK’ and continue…..




Verify:


[oracle@oemserver bin]$ cd ../../agent_13.2.0.0.0/bin/
[oracle@oemserver bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.2.0.0.0
OMS Version            : 13.2.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /opt/oracle/product/agent13c/agent_inst
Agent Log Directory    : /opt/oracle/product/agent13c/agent_inst/sysman/log
Agent Binaries         : /opt/oracle/product/agent13c/agent_13.2.0.0.0
Core JAR Location      : /opt/oracle/product/agent13c/agent_13.2.0.0.0/jlib
Agent Process ID       : 6218
Parent Process ID      : 6141
Agent URL              : https://oemserver.domain.com:3872/emd/main/
Local Agent URL in NAT : https://oemserver.domain.com:3872/emd/main/
Repository URL         : https://oemserver.domain.com:4903/empbs/upload
Started at             : 2016-10-27 16:08:44
Started by user        : oracle
Operating System       : Linux version 3.10.0-327.28.3.el7.x86_64 (amd64)
Number of Targets      : 36
Last Reload            : 2016-10-27 16:10:02
Last successful upload                       : 2016-10-27 16:23:52
Last attempted upload                        : 2016-10-27 16:23:52
Total Megabytes of XML files uploaded so far : 0.36
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 12.31%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2016-10-27 16:26:19
Last successful heartbeat to OMS             : 2016-10-27 16:26:19
Next scheduled heartbeat to OMS              : 2016-10-27 16:27:19

---------------------------------------------------------------
Agent is Running and Ready




Verify Agent upgrade before Cleaning up OLD agents:




Click on “Agent Upgrade Results”




Above are the 2 jobs (agent upgrade jobs) and the status is Success, click and see the upgrade results


Now, check the agent versions:

From the Setup menu, select Manage Cloud Control, then select Agents. Click the name of a Management Agent that you want to verify the upgrade for, and verify the Management Agent version. The Management Agent version after the upgrade must be the same as the OMS version. 




Postupgrade Cleanup of Old Management Agents

From the Setup menu, select Manage Cloud Control, then select Upgrade Agents.


Click “ Post Agent Upgrade Tasks



Click “ADD” and add the agents to cleanup and “submit”



Click on Job name to see the results





Friday, October 28, 2016

Oracle OMS upgrade from 13.1 to 13.2 version



Currently my OEM version is on 13.1 version and i'm going to upgrade to latest 13.2 version

OMS: 13.1 on RHEL 7.2
OMS repo: 12c database on RHEL 6.8

Login to OMS server and perform below

 [oracle@oemserver product]$ cd $OMS_HOME
[oracle@oemserver oms]$ cd bin
[oracle@oemserver bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server is Up


 [oracle@oemserver bin]$ ./emctl stop oms -all
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping Oracle Management Server...
WebTier Successfully Stopped
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down
Stopping BI Publisher Server...
BI Publisher Server Successfully Stopped
AdminServer Successfully Stopped
BI Publisher Server is Down

 [oracle@oemserver bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
WebTier is Down
Oracle Management Server is Down
JVMD Engine is Down
BI Publisher Server is Down





[oracle@oemserver bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.1.0.0.0
OMS Version            : 13.1.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /opt/oracle/product/agent13c/agent_inst
Agent Log Directory    : /opt/oracle/product/agent13c/agent_inst/sysman/log
Agent Binaries         : /opt/oracle/product/agent13c/agent_13.1.0.0.0
Core JAR Location      : /opt/oracle/product/agent13c/agent_13.1.0.0.0/jlib
Agent Process ID       : 10420
Parent Process ID      : 6697
Agent URL              : https://oemserver.ux.domain.com:3872/emd/main/
Local Agent URL in NAT : https://oemserver.domain.com:3872/emd/main/
Repository URL         : https://oemserver.domain.com:4903/empbs/upload
Started at             : 2016-09-24 22:26:56
Started by user        : oracle
Operating System       : Linux version 3.10.0-327.28.3.el7.x86_64 (amd64)
Number of Targets      : 33
Last Reload            : (none)
Last successful upload                       : 2016-10-24 12:42:09
Last attempted upload                        : 2016-10-24 12:49:23
Total Megabytes of XML files uploaded so far : 50.87
Number of XML files pending upload           : 26
Size of XML files pending upload(MB)         : 0.02
Available disk space on upload filesystem    : 20.43%
Collection Status                            : Collections enabled
Heartbeat Status                             : OMS is unreachable
Last attempted heartbeat to OMS              : 2016-10-24 12:49:15
Last successful heartbeat to OMS             : 2016-10-24 12:45:33
Next scheduled heartbeat to OMS              : 2016-10-24 12:49:45

---------------------------------------------------------------
Agent is Running and Ready

[oracle@oemserver bin]$ ./emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.


 [oracle@oemserver bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent is Not Running



Backup the 3 below things (Mandatory):

1.Backup OMS home
    tar -czvf /opt/oracle/OMS_HOME.tar.gz /opt/oracle/product/middleware/oms

2. Backup  13.1 agent
    tar -czvf /opt/oracle/agent13c.tar.gz agent13c

3. Take RMAN full backup of Repo database:


run {
allocate channel c1 type disk;
BACKUP format '/pkg/oem/CB/FULL_emprod_%u_%s_%p_%T' database
plus archivelog  format '/pkg/oem/CB/emprod_Arch_%u_%s_%p_%T';
release channel c1;
}

To start the GUI:
New putty terminal à expand “SSH” à click X11 à Select “ENABLE X11 Forwarding”

I've already downloaded software binaries to the below directory from OTN

[oracle@oemserver oms]$ cd /opt/oracle/product/OMS_13.2/soft
[oracle@oemserver soft]$  chmod 777 em13200_linux64.bin
[oracle@oemserver soft]$ ls -l
total 7727668
-rw-r--r--. 1 oracle dba 2118110455 Oct 20 14:55 em13200_linux64-2.zip
-rw-r--r--. 1 oracle dba  553563556 Oct 20 14:46 em13200_linux64-3.zip
-rw-r--r--. 1 oracle dba 2084299941 Oct 20 14:55 em13200_linux64-4.zip
-rw-r--r--. 1 oracle dba 2128070240 Oct 20 14:56 em13200_linux64-5.zip
-rw-r--r--. 1 oracle dba  474441637 Oct 20 14:44 em13200_linux64-6.zip
-rwxrwxrwx. 1 oracle dba  554607894 Oct 20 14:46 em13200_linux64.bin


[oracle@oemserver soft]$ ./em13200_linux64.bin -J-Djava.io.tmpdir=/opt/oracle/tmp 0%.........................................................................100%
Launcher log file is /opt/oracle/tmp/OraInstall2016-10-25_04-21-28PM/launcher2016-10-25_04-21-28PM.log.
Starting Oracle Universal Installer

Checking if CPU speed is above 300 MHz.   Actual 2700.000 MHz    Passed
Checking monitor: must be configured to display at least 256 colors.   Actual 16777216    Passed
Checking swap space: must be greater than 512 MB.   Actual 3999 MB    Passed
Checking if this platform requires a 64-bit JVM.   Actual 64    Passed (64-bit not required)



























 If you see the above screen, I don’t see an option to select “upgrade”. It is because of installer cannot able to find the “oraInst.loc” file


NOTE:
By default, on Linux environment tmdir=/tmp and oraInst= /etc/oraInst.loc
Since my /tmp has only 7G so I want to redirect to another temp location so used “-J-Djava.io.tmpdir
Also my central inventory does not exist so I need to specify manually using “-invPtrLoc”


[oracle@oemserver soft]$ ./em13200_linux64.bin -J-Djava.io.tmpdir=/opt/oracle/tmp -invPtrLoc /home/oracle/oraInst.loc


















Deferred Data Migration (DDMP) is a post-upgrade activity to migrate the format of the data stored in an earlier release of Enterprise Manager to the format compatible with the upgraded Enterprise Manager system.






Click ‘OK’ and it will keep asking for few below and make sure you satisfy these conditions






  









This Emkey should be done as a prereq before you down the OMS, but I forgot to do this and now I’m doing it.

[oracle@oemserver bin]$ ./emctl status emkey
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Error occurred. Check the log /opt/oracle/product/middleware/gc_inst/em/EMGC_OMS1/sysman/log/secure.log
[oracle@oemserver bin]$ vi /opt/oracle/product/middleware/gc_inst/em/EMGC_OMS1/sysman/log/secure.log


 [oracle@oemserver bin]$ ./emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Error occurred. Check the log /opt/oracle/product/middleware/gc_inst/em/EMGC_OMS1/sysman/log/secure.log


[oracle@oemserver bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up
Starting BI Publisher Server ...

BI Publisher Server Successfully Started
BI Publisher Server is Up


[oracle@oemserver bin]$ ./emctl config emkey -copy_to_repos
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
The EMKey has been copied to the Management Repository. This operation will cause the EMKey to become unsecure.
After the required operation has been completed, secure the EMKey by running "emctl config emkey -remove_from_repos".







Click ‘YES’




Click “OK” and click ‘Next”




















Error Log:

INFO: oracle.sysman.top.oms:1 error(s) happened in performing the action:
INFO: oracle.sysman.top.oms:1):
ORA-20001: No Resource Provider Type Exist with Target Type=oracle_nimbula_site.
ORA-06512: at "SYSMAN.CFW_SERVICE_TYPE", line 26
ORA-06512: at line 20
File:/opt/oracle/product/OMS_13.2/middleware/plugins/oracle.sysman.csm.oms.plugin_13.2.1.0.0/sql/csm_mds/latest/CfwServiceType/jcsservicetype_post_creation.sql
Statement:DECLARE

BEGIN

DECLARE
l_version_number number;
BEGIN
l_version_number := CFW_SERVICE_TYPE.get_service_type_version('java_cloud_service' );
if l_version_number < 0 then
DECLARE
l_service_id NUMBER;
l_temp_id NUMBER;
l_pref_id NUMBER;
l_service_inst_type_id NUMBER;
l_setting_id NUMBER;
l_param_id NUMBER;
BEGIN
l_service_id := CFW_SERVICE_TYPE.add_service_type('JAVA_CLOUD_SERVICE' , 'oracle.sysman.csm' , 1.0 , 'OPC_SERVICE_FAMILY' , '/images/jcs.png' , 'Java Cloud Service' , 'Oracle Public Cloud - Java Cloud Service.' , 'JAVA_CLOUD_SERVICE_NAME' , 'JAVA_CLOUD_SERVICE_DESC' , '' , 'jcs' , 1 , 'Middleware' );
l_service_inst_type_id := CFW_SERVICE_TYPE.add_service_inst_type(l_service_id , 'oracle_cloud_jcs' , 'goto_cfw-service-instance-homepage' , 0 );
CFW_SERVICE_TYPE.add_service_rp_mapping('oracle_nimbula_site' , l_service_id );
l_temp_id := CFW_SVC_COMMON.add_entity_instance('SERVICE_TYPE' , l_service_id );
END;

elsif l_version_number < 1.0 then
DECLARE
l_service_id NUMBER;
l_temp_id NUMBER;
l_pref_id NUMBER;
l_service_inst_type_id NUMBER;
l_setting_id NUMBER;
l_param_id NUMBER;
BEGIN
l_service_id := CFW_SERVICE_TYPE.update_service_type('JAVA_CLOUD_SERVICE' , 'oracle.sysman.csm' , 1.0 , 'OPC_SERVICE_FAMILY' , '/images/jcs.png' , 'Java Cloud Service' , 'Oracle Public Cloud - Java Cloud Service.' , 'JAVA_CLOUD_SERVICE_NAME' , 'JAVA_CLOUD_SERVICE_DESC' , '' , 'jcs' , 1 , 'Middleware' );
l_service_inst_type_id := CFW_SERVICE_TYPE.add_service_inst_type(l_service_id , 'oracle_cloud_jcs' , 'goto_cfw-service-instance-homepage' , 0 );
CFW_SERVICE_TYPE.remove_service_rp_mapping(l_service_id );
CFW_SERVICE_TYPE.add_service_rp_mapping('oracle_nimbula_site' , l_service_id );
l_temp_id := CFW_SVC_COMMON.get_entity_instance_id('SERVICE_TYPE' , l_service_id );
CFW_SVC_COMMON.remove_characteristic_ref(l_temp_id );
END;

elsif l_version_number > 1.0 then
 raise_application_error(-20002,'Downgrade is not Supported');
else
 DBMS_OUTPUT.put_line('Same version of entity already exist.');
end if;

END;


END;


INFO: oracle.sysman.top.oms:The plug-in Repository Upgrade has failed its perform method





Contacted oracle support and was told that its because of the plugin "Cloud service management" which i have selected, when we select this plugin we also need to select plugin "Oracle Virtual Infrastructure" (mandatory) but I didn’t select this plugin and gave the above error
he said we might need to rollback and restore the repo DB and try again with that option

Performed RMAN database Point in time recovery and performed all the steps again but this time I didn’t check in any additional plugins
















Once Done run Root.sh script and click “OK’





Verify:


Go to OEM “SYSMAN” à “About Enterprise Manager”




Now, Login to OMS server and check the status

[oracle@oemserver bin]$ ./emctl status oms -details
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Enter Enterprise Manager Root (SYSMAN) Password :
Console Server Host        : oemserver.domain.com
HTTP Console Port          : 7788
HTTPS Console Port         : 7803
HTTP Upload Port           : 4889
HTTPS Upload Port          : 4903
EM Instance Home           : /opt/oracle/product/OMS_13.2/gc_inst/em/EMGC_OMS1
OMS Log Directory Location : /opt/oracle/product/OMS_13.2/gc_inst/em/EMGC_OMS1/sysman/log
OMS is not configured with SLB or virtual hostname
Agent Upload is locked.
OMS Console is locked.
Active CA ID: 1
Console URL: https://oemserver.domain.com:7803/em
Upload URL: https://oemserver.domain.com:4903/empbs/upload

WLS Domain Information
Domain Name            : GCDomain
Admin Server Host      : oemserver.domain.com
Admin Server HTTPS Port: 7102
Admin Server is RUNNING

Oracle Management Server Information
Managed Server Instance Name: EMGC_OMS1
Oracle Management Server Instance Host: oemserver.domain.com
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up

BI Publisher Server Information
BI Publisher Managed Server Name: BIP
BI Publisher Server is Up

BI Publisher HTTP Managed Server Port   : 9701
BI Publisher HTTPS Managed Server Port  : 9803
BI Publisher HTTP OHS Port              : 9751
BI Publisher HTTPS OHS Port             : 9852
BI Publisher is locked.
BI Publisher Server named 'BIP' running at URL: https://oemserver.domain.com:9852/xmlpserver
BI Publisher Server Logs: /opt/oracle/product/OMS_13.2/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/
BI Publisher Log        : /opt/oracle/product/OMS_13.2/gc_inst/user_projects/domains/GCDomain/servers/BIP/logs/bipublisher/bipublisher.log




[oracle@oemserver bin]$ ./emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Starting agent ................................................................ started.

[oracle@oemserver bin]$ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.1.0.0.0
OMS Version            : 13.2.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /opt/oracle/product/agent13c/agent_inst
Agent Log Directory    : /opt/oracle/product/agent13c/agent_inst/sysman/log
Agent Binaries         : /opt/oracle/product/agent13c/agent_13.1.0.0.0
Core JAR Location      : /opt/oracle/product/agent13c/agent_13.1.0.0.0/jlib
Agent Process ID       : 120943
Parent Process ID      : 120764
Agent URL              : https://oemserver.domain.com:3872/emd/main/
Local Agent URL in NAT : https://oemserver.domain.com:3872/emd/main/
Repository URL         : https://oemserver.domain.com:4903/empbs/upload
Started at             : 2016-10-27 15:50:11
Started by user        : oracle
Operating System       : Linux version 3.10.0-327.28.3.el7.x86_64 (amd64)
Number of Targets      : 33
Last Reload            : (none)
Last successful upload                       : 2016-10-27 15:51:08
Last attempted upload                        : 2016-10-27 15:51:08
Total Megabytes of XML files uploaded so far : 0.04
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 13.87%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2016-10-27 15:50:47
Last successful heartbeat to OMS             : 2016-10-27 15:50:47
Next scheduled heartbeat to OMS              : 2016-10-27 15:51:48

---------------------------------------------------------------
Agent is Running and Ready


AGENTS UPGRADE:


Setup à “Manage Cloud Control” à “Upgrade Agents”

Steps for upgrading agents from OEM interface :  Agent upgrade 13c







Auto Scroll Stop Scroll