Saturday, August 18, 2018

13c OEM Status Pending (Target addition is in Progress)



After discovering and adding database to my 13.2 OEM, target database is still in “status Pending (Target addition is in Progress)” even after few days

From OMS repo database……

SQL> select entity_name, entity_type, host_name, manage_status , promote_status, emd_url ,DISPLAY_NAME from SYSMAN.EM_MANAGEABLE_ENTITIES where entity_name like '%DEVDB%' and emd_url like '%oradev01%';

ENTITY_NAME            ENTITY_TYPE                         HOST_NAME            MANAGE_STATUS     PROMOTE_STATUS
---------------------- ------------------------------------------------------ -------------------- ------------- --------------
EMD_URL
-------------------------------------------------------------
DISPLAY_NAME
------------------------------------------------------------DEVDB                 oracle_database                                                  oradev01.domain.com             2              1
https://oradev01.domain.com:3872/emd/main/
DEVDB

From the above output we can see that
managed_status=2 and promote_status=1 indicates that the target is not yet promoted to the agent side


Now from Target database (DEVDB) go to the agent and check below target status

oradev01:DEVDB:/home/oracle $ cd /opt/oracle/product/oemagent/agent13c/agent_13.2.0.0.0/bin

oradev01:DEVDB:/opt/oracle/product/oemagent/agent13c/agent_13.2.0.0.0/bin $ ./emctl config agent listtargets
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
[oradev01.domain.com, host]                                 
[oradev01.domain.com:3872, oracle_emd]

If you check the above output, it doesn’t listed any of the databases existing ion the server and according to (Doc ID 2286993.1) root.sh has not been executed after the installation of Agent.
To check whether root.sh was executed on target DB

$ cd <AGENT_BASE_DIR>/sbin

oradev01:DEVDB:/opt/oracle/product/oemagent/agent13c/agent_13.2.0.0.0/sbin $ ls -ltr nm*
-rw-r----- 1 oracle dba    9615 Aug  1  2016 nmr_macro_list
-rwx--x--x 1 oracle dba 4786905 Sep 30  2016 nmo.0
-rwx--x--x 1 oracle dba 4703504 Sep 30  2016 nmoconf
-rwx--x--x 1 oracle dba   53058 Sep 30  2016 nmb.0
-rwx--x--x 1 oracle dba   50457 Sep 30  2016 nmgsshe.0
-rwx------ 1 oracle dba  104650 Sep 30  2016 nmr.0
-rwx------ 1 oracle dba   34522 Sep 30  2016 nmrconf
-rwx--x--x 1 oracle dba 4708480 Sep 30  2016 nmopdpx.0
-rwx--x--x 1 oracle dba  105929 Sep 30  2016 nmhs.0
-rwx--x--x 1 oracle dba  585242 Feb 10  2017 nmosudo.0


Hmmm looks like it is NOT

Solution:

Run Root.sh script on target server (oradev01)  as a ROOT user
Verify again owner for nm* files

[root@oradev01 root] # /opt/oracle/product/oemagent/agent13c/agent_13.2.0.0.0/root.sh
Finished product-specific root actions.
/etc exist

oradev01:DEVDB:/opt/oracle/product/oemagent/agent13c/agent_13.2.0.0.0/sbin $ ls -lrth
total 30M
-rw-r----- 1 root   dba  9.4K Aug  1  2016 nmr_macro_list
-rwx--x--x 1 oracle dba  4.6M Sep 30  2016 nmo.0
-rwx--x--x 1 oracle dba  4.5M Sep 30  2016 nmoconf
-rwx--x--x 1 oracle dba   52K Sep 30  2016 nmb.0
-rwx--x--x 1 oracle dba   50K Sep 30  2016 nmgsshe.0
-rwx------ 1 oracle dba  103K Sep 30  2016 nmr.0
-rwx------ 1 oracle dba   34K Sep 30  2016 nmrconf
-rwx--x--x 1 oracle dba  4.5M Sep 30  2016 nmopdpx.0
-rwx--x--x 1 oracle dba  104K Sep 30  2016 nmhs.0
-rwx--x--x 1 oracle dba  572K Feb 10  2017 nmosudo.0
-rwx--x--x 1 root   root 4.6M Aug  6 13:11 nmo.new.bak
-rw-r----- 1 root   dba   188 Aug  6 13:11 nmo_public_key.txt
-rwsr-x--- 1 root   dba  4.6M Aug  6 13:11 nmo
-rwsr-x--- 1 root   dba  104K Aug  6 13:11 nmhs
-rwsr-x--- 1 root   dba   52K Aug  6 13:11 nmb
-rwx------ 1 root   root 103K Aug  6 13:11 nmr.new.bak
-rwsr-x--- 1 root   dba  103K Aug  6 13:11 nmr
-rwxr-xr-x 1 root   dba  572K Aug  6 13:11 nmosudo
-rwxr-xr-x 1 root   dba   50K Aug  6 13:11 nmgsshe
-rwxr-xr-x 1 root   dba  4.5M Aug  6 13:11 nmopdpx


Now Delete the Target database from OEM and rediscover again and this time it will show as up and Running Cheers !!!!
EM 13c : After Discovery, Targets Are Stuck In "Status Pending (Target Addition In Progress)" Status (Doc ID 2286993.1)




Sunday, July 15, 2018

ERROR OGG-08221 Cannot register or unregister EXTRACT because of the following SQL error: OCI Error 1,653.



DB Version:  12.1.0.2.0
GG Version:  12.2

SET Database environment
Login to database from ggsci using goldengate alias (or) with user/password if Alias not created

mylocaldev.domain.com : dbstage:/goldengate/home $ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.2.0.1.1 OGGCORE_12.2.0.1.0_PLATFORMS_151211.1401_FBO
Linux, x64, 64bit (optimized), Oracle 12c on Dec 12 2015 02:56:48
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2015, Oracle and/or its affiliates. All rights reserved.

GGSCI (mylocaldev.domain.com as GGADMIN@dbstage) 1> dblogin useridalias ggadmin
Successfully logged into database.

GGSCI (mylocaldev.domain.com as GGADMIN@dbstage) 2> register extract EMYDBST database

2017-08-23 13:10:58  ERROR   OGG-08221  Cannot register or unregister EXTRACT EMYDBST because of the following SQL error: OCI Error 1,653.


In DB alert log:

Wed Aug 23 13:28:44 2017
ORA-1653: unable to extend table SYS.AUD$ by 128 in tablespace SYSTEM
ORA-1653: unable to extend table SYS.AUD$ by 1024 in tablespace SYSTEM

Ok now we know the issue and once the tablespace space issue is resolved then GG register command is success, Cheers !!


Tuesday, May 29, 2018

WARNING: NMO not setuid-root (Unix-only) 13c OEM


For the past Few days my OEM jobs on my DEV server are getting SUSPENDED , tried to schedule multiple time after failure and still they are NOT running they are all in SUSPENDED State.

Agent is up and running and I tried stop, start ,clear, secure, upload, reload all sort of agent commands and still my jobs getting SUSPENDED

From the server all looks good.

mydev01:NOTSET:/opt/oracle/product/oemagent/agent13c_dev/agent_inst/bin $ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.2.0.0.0
OMS Version            : 13.2.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /opt/oracle/product/oemagent/agent13c_dev/agent_inst
Agent Log Directory    : /opt/oracle/product/oemagent/agent13c_dev/agent_inst/sysman/log
Agent Binaries         : /opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0
Core JAR Location      : /opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0/jlib
Agent Process ID       : 261231
Parent Process ID      : 261232
Agent URL              : https://mydev01.domain.local:3872/emd/main/
Local Agent URL in NAT : https://mydev01.domain.local:3872/emd/main/
Repository URL         : https://mydevoms.domain.local:4903/empbs/upload
Started at             : 2018-03-21 12:25:23
Started by user        : oracle
Operating System       : Linux version 2.6.32-696.23.1.el6.x86_64 (amd64)
Number of Targets      : 23
Last Reload            : (none)
Last successful upload                       : 2018-03-21 12:59:32
Last attempted upload                        : 2018-03-21 12:59:32
Total Megabytes of XML files uploaded so far : 0.37
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 38.65%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2018-03-21 13:00:45
Last successful heartbeat to OMS             : 2018-03-21 13:00:45
Next scheduled heartbeat to OMS              : 2018-03-21 13:01:45

---------------------------------------------------------------
Agent is Running and Ready

mydev01:NOTSET:/opt/oracle/product/oemagent/agent13c_dev/agent_inst/bin $ ./emctl upload agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
EMD upload completed successfully

FROM my OEM I see Agent is fine and up and running (but with additional warning)


WARNING: 
NMO not setuid-root (Unix-only). Run root.sh script to correct this problem. The next OS Command Job submitted will clear this alert.

According to (Doc ID 1465278.1) this is what I have followed

mydev01:NOTSET:/opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0/sbin $ ls -lrht
total 33M
-rw-r----- 1 oracle dba 9.4K Aug  1  2016 nmr_macro_list
-rwx--x--x 1 oracle dba 4.6M Sep 30  2016 nmo.0
-rwx--x--x 1 oracle dba 4.5M Sep 30  2016 nmoconf
-rwx--x--x 1 oracle dba  52K Sep 30  2016 nmb.0
-rwx--x--x 1 oracle dba  50K Sep 30  2016 nmgsshe.0
-rwx------ 1 oracle dba 103K Sep 30  2016 nmr.0
-rwx------ 1 oracle dba  34K Sep 30  2016 nmrconf
-rwx--x--x 1 oracle dba 4.5M Sep 30  2016 nmosudo.0
-rwx--x--x 1 oracle dba 4.5M Sep 30  2016 nmopdpx.0
-rwx--x--x 1 oracle dba 104K Sep 30  2016 nmhs.0
-rwxr-x--- 1 oracle dba 103K Feb 23  2017 nmr
-rwxr-xr-x 1 oracle dba 4.5M Feb 23  2017 nmosudo
-rw-r----- 1 oracle dba  188 Feb 23  2017 nmo_public_key.txt
-rwxr-xr-x 1 oracle dba 4.5M Feb 23  2017 nmopdpx
-rwxr-x--- 1 oracle dba 4.6M Feb 23  2017 nmo
-rwxr-x--- 1 oracle dba 104K Feb 23  2017 nmhs
-rwxr-xr-x 1 oracle dba  50K Feb 23  2017 nmgsshe
-rwxr-x--- 1 oracle dba  52K Feb 23  2017 nmb


mydev01:NOTSET:/opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0 $ ls -lrth root*
-rwxr-xr-x 1 oracle dba 4.7K Feb 23  2017 root.sh.template
-rwxr-xr-x 1 oracle dba 4.8K Feb 23  2017 root.sh


[root@mydev01 root] # /opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0/root.sh
Finished product-specific root actions.
/etc exist


Once Root.sh script ran, I see permissions now changed.

mydev01:NOTSET:/opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0 $ cd sbin/
mydev01:NOTSET:/opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0/sbin $ ls -lrht
total 37M
-rw-r----- 1 root   dba  9.4K Aug  1  2016 nmr_macro_list
-rwx--x--x 1 oracle dba  4.6M Sep 30  2016 nmo.0
-rwx--x--x 1 oracle dba  4.5M Sep 30  2016 nmoconf
-rwx--x--x 1 oracle dba   52K Sep 30  2016 nmb.0
-rwx--x--x 1 oracle dba   50K Sep 30  2016 nmgsshe.0
-rwx------ 1 oracle dba  103K Sep 30  2016 nmr.0
-rwx------ 1 oracle dba   34K Sep 30  2016 nmrconf
-rwx--x--x 1 oracle dba  4.5M Sep 30  2016 nmosudo.0
-rwx--x--x 1 oracle dba  4.5M Sep 30  2016 nmopdpx.0
-rwx--x--x 1 oracle dba  104K Sep 30  2016 nmhs.0
-rwx--x--x 1 root   root 4.6M Mar 21 13:20 nmo.new.bak
-rw-r----- 1 root   dba   188 Mar 21 13:20 nmo_public_key.txt
-rwsr-x--- 1 root   dba  4.6M Mar 21 13:20 nmo
-rwsr-x--- 1 root   dba  104K Mar 21 13:20 nmhs
-rwsr-x--- 1 root   dba   52K Mar 21 13:20 nmb
-rwx------ 1 root   root 103K Mar 21 13:20 nmr.new.bak
-rwsr-x--- 1 root   dba  103K Mar 21 13:20 nmr
-rwxr-xr-x 1 root   dba  4.5M Mar 21 13:20 nmosudo
-rwxr-xr-x 1 root   dba   50K Mar 21 13:20 nmgsshe
-rwxr-xr-x 1 root   dba  4.5M Mar 21 13:20 nmopdpx

mydev01:NOTSET:/opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0/sbin $ cd ../bin
mydev01:NOTSET:/opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0/bin $ ./emctl status agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
---------------------------------------------------------------
Agent Version          : 13.2.0.0.0
OMS Version            : 13.2.0.0.0
Protocol Version       : 12.1.0.1.0
Agent Home             : /opt/oracle/product/oemagent/agent13c_dev/agent_inst
Agent Log Directory    : /opt/oracle/product/oemagent/agent13c_dev/agent_inst/sysman/log
Agent Binaries         : /opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0
Core JAR Location      : /opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0/jlib
Agent Process ID       : 261231
Parent Process ID      : 261232
Agent URL              : https://mydev01.domain.local:3872/emd/main/
Local Agent URL in NAT : https://mydev01.domain.local:3872/emd/main/
Repository URL         : https://mydevoms.domain.local:4903/empbs/upload
Started at             : 2018-03-21 12:25:23
Started by user        : oracle
Operating System       : Linux version 2.6.32-696.23.1.el6.x86_64 (amd64)
Number of Targets      : 23
Last Reload            : (none)
Last successful upload                       : 2018-03-21 13:19:32
Last attempted upload                        : 2018-03-21 13:19:32
Total Megabytes of XML files uploaded so far : 0.41
Number of XML files pending upload           : 0
Size of XML files pending upload(MB)         : 0
Available disk space on upload filesystem    : 38.65%
Collection Status                            : Collections enabled
Heartbeat Status                             : Ok
Last attempted heartbeat to OMS              : 2018-03-21 13:20:46
Last successful heartbeat to OMS             : 2018-03-21 13:20:46
Next scheduled heartbeat to OMS              : 2018-03-21 13:21:47

---------------------------------------------------------------
Agent is Running and Ready

mydev01:NOTSET:/opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0/bin $ ./emctl stop agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Stopping agent ... stopped.

mydev01:NOTSET:/opt/oracle/product/oemagent/agent13c_dev/agent_13.2.0.0.0/bin $ ./emctl start agent
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
Starting agent ............. started.

Now I see Warning message is Gone in OEM.

Sunday, April 1, 2018

Fully qualified table name is not found in the Oracle GoldenGate configuration



Today on one of my Goldengate server i want to see the transaction statistics for a particular table on source side and when I queried below using ggsci stats command I see “Fully qualified table name MYCUSTOMER.CUST_CLEAR_HIST is not found in the Oracle GoldenGate configuration“ in the output

Reason:
Yes this is NOT an issue and it is expected when there are no transactions performed on this table ever since on source side then the output prints this message.

According to (Doc ID 1941158.1), If no transaction log records for the specific table have been read and processed by the extract process, the internal mapping table has not been built.


myproddemo:custdb1:/goldengate/home $ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.2.0.1.1 OGGCORE_12.2.0.1.0_PLATFORMS_151211.1401_FBO
Linux, x64, 64bit (optimized), Oracle 12c on Dec 12 2015 02:56:48
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2015, Oracle and/or its affiliates. All rights reserved.

GGSCI (myproddemo) 1> stats ECUSTDBP totalsonly MYCUSTOMER.CUST_CLR_HIST

Sending STATS request to EXTRACT ECUSTDBP ...

Start of Statistics at 2018-03-30 10:19:28.

DDL replication statistics (for all trails):

*** Total statistics since extract started     ***
        Operations                                         9.00

Output to /goldengate/home/dirdat/SRC/custdb/trgdb/ct:

Cumulative totals for specified table(s):

*** Total statistics since 2018-03-05 06:47:25 ***

        No database operations have been performed.

*** Daily statistics since 2018-03-30 00:00:00 ***

        No database operations have been performed.

*** Hourly statistics since 2018-03-30 10:00:00 ***

        No database operations have been performed.

*** Latest statistics since 2018-03-05 06:47:25 ***

        No database operations have been performed.

Fully qualified table name MYCUSTOMER.CUST_CLR_HIST is not found in the Oracle GoldenGate configuration.

End of Statistics.


Auto Scroll Stop Scroll