Tuesday, December 18, 2018

ggsci: error while loading shared libraries: libclntsh.so.12.1: cannot open shared object file: No such file or directory


I have a shell script which Kicks off automatically when server reboots but that script when trying to login to ggsci giving below error

/goldengate/home/./ggsci: error while loading shared libraries: libclntsh.so.12.1: cannot open shared object file: No such file or directory

Shell Script:
oradev01:NOTSET:/home/oracle $ cat /goldengate/DBA/start_mgr_rc.sh
sudo su - oracle -c "export ORACLE_HOME=/opt/oracle/product/12.1.0.2.64;export PATH=$ORACLE_HOME/bin:$PATH;export LD_LIBRARY_PATH=$ORACLE_HOME/lib;/goldengate/home/./ggsci" <<EOF
sh sleep 30m
start mgr
EOF
exit;

Reason:
ggsci is unable to locate lib files which are available under database ORACLE_HOME, tried to verify login manually and same issue

oradev01:NOTSET:/goldengate/home $ ./ggsci
/goldengate/home/./ggsci: error while loading shared libraries: libclntsh.so.12.1: cannot open shared object file: No such file or directory

oradev01:NOTSET:/goldengate/home $  ldd ./ggsci
        linux-vdso.so.1 =>  (0x00007ffc0b5e3000)
        librt.so.1 => /lib64/librt.so.1 (0x0000003504000000)
        libdl.so.2 => /lib64/libdl.so.2 (0x0000003503c00000)
        lib./ggscilog.so => /goldengate/home/./lib./ggscilog.so (0x00007f07cf2a8000)
        lib./ggscirepo.so => /goldengate/home/./lib./ggscirepo.so (0x00007f07cf036000)
        libdb-6.1.so => /goldengate/home/./libdb-6.1.so (0x00007f07cec51000)
        lib./ggsciperf.so => /goldengate/home/./lib./ggsciperf.so (0x00007f07cea21000)
        lib./ggsciparam.so => /goldengate/home/./lib./ggsciparam.so (0x00007f07cd917000)
        libicui18n.so.48 => /goldengate/home/./libicui18n.so.48 (0x00007f07cd527000)
        libicuuc.so.48 => /goldengate/home/./libicuuc.so.48 (0x00007f07cd1a6000)
        libicudata.so.48 => /goldengate/home/./libicudata.so.48 (0x00007f07cb9e1000)
        libpthread.so.0 => /lib64/libpthread.so.0 (0x0000003503800000)
        libxerces-c.so.28 => /goldengate/home/./libxerces-c.so.28 (0x00007f07cb41a000)
        libantlr3c.so => /goldengate/home/./libantlr3c.so (0x00007f07cb201000)
        libnnz12.so => not found
        libclntsh.so.12.1 => not found
        libons.so => not found
        libclntshcore.so.12.1 => not found
        lib./ggscinnzitp.so => /goldengate/home/./lib./ggscinnzitp.so (0x00007f07ca721000)
        libm.so.6 => /lib64/libm.so.6 (0x0000003504800000)
        libc.so.6 => /lib64/libc.so.6 (0x0000003503400000)
        /lib64/ld-linux-x86-64.so.2 (0x0000003503000000)
        libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x000000350f000000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x000000350a400000)

oradev01:NOTSET:/goldengate/home $ exit

As ORACLE user Create Soft links

oradev01:NOTSET:/opt/oracle/product/12.1.0.2.64/lib $ ls -l libnnz*
-rw-r--r-- 1 oracle dba  1752834 Jul  5  2016 libnnz12.a
-rwxr-xr-x 1 oracle dba  6318657 Jul  5  2016 libnnz12.so
-rw-r--r-- 1 oracle dba 19160504 Jul  5  2016 libnnzst12.a

oradev01:NOTSET:/opt/oracle/product/12.1.0.2.64/lib $ cd -
/goldengate/home

oradev01:NOTSET:/goldengate/home $ ln -s /opt/oracle/product/12.1.0.2.64/lib/libnnz12.so libnnz12.so

oradev01:NOTSET:/goldengate/home $ ls -l libnnz12.so
lrwxrwxrwx 1 oracle dba 47 Sep 22 10:04 libnnz12.so -> /opt/oracle/product/12.1.0.2.64/lib/libnnz12.so

oradev01:NOTSET:/goldengate/home $ ./ggsci
/goldengate/home/./ggscisci: error while loading shared libraries: libclntsh.so.12.1: cannot open shared object file: No such file or directory

oradev01:NOTSET:/goldengate/home $ ln -s /opt/oracle/product/12.1.0.2.64/lib/libclntsh.so.12.1 libclntsh.so.12.1

oradev01:NOTSET:/goldengate/home $ ./ggsci
/goldengate/home/./ggscisci: error while loading shared libraries: libons.so: cannot open shared object file: No such file or directory

oradev01:NOTSET:/goldengate/home $ ln -s /opt/oracle/product/12.1.0.2.64/lib/libons.so libons.so

oradev01:NOTSET:/goldengate/home $ ./ggsci
/goldengate/home/./ggscisci: error while loading shared libraries: libclntshcore.so.12.1: cannot open shared object file: No such file or directory

oradev01:NOTSET:/goldengate/home $ ln -s /opt/oracle/product/12.1.0.2.64/lib/libclntshcore.so.12.1 libclntshcore.so.12.1

oradev01:NOTSET:/goldengate/home $ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 12.2.0.1.170221 O./GGSCICORE_12.2.0.1.0O./GGSCIBP_PLATFORMS_170123.1033_FBO
Linux, x64, 64bit (optimized), Oracle 12c on Jan 23 2017 23:22:14
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2017, Oracle and/or its affiliates. All rights reserved.



GGSCI (oradev01.domain.local) 1>exit

Yes now I can login after doing softlinks

oradev01:NOTSET:/goldengate/home $ ls -lrth lib*
-rw-r----- 1 oracle dba 1.7K Jan 30  2017 libxml2.txt
-rwxr-x--- 1 oracle dba 4.7M Jan 30  2017 libxerces-c.so.28
-rwxr-x--- 1 oracle dba 1.8M Jan 30  2017 libicuuc.so.48.1
-rwxr-x--- 1 oracle dba 1.8M Jan 30  2017 libicuuc.so.48
-rwxr-x--- 1 oracle dba 2.5M Jan 30  2017 libicui18n.so.48.1
-rwxr-x--- 1 oracle dba 2.5M Jan 30  2017 libicui18n.so.48
-rwxr-x--- 1 oracle dba  22M Jan 30  2017 libicudata.so.48.1
-rwxr-x--- 1 oracle dba  22M Jan 30  2017 libicudata.so.48
-rwxr-x--- 1 oracle dba 3.0M Jan 30  2017 lib./ggscirepo.so
-rwxr-x--- 1 oracle dba 1.6M Jan 30  2017 lib./ggsciperf.so
-rwxr-x--- 1 oracle dba  43M Jan 30  2017 lib./ggsciparam.so
-rwxr-x--- 1 oracle dba  11M Jan 30  2017 lib./ggscinnzitp.so
-rwxr-x--- 1 oracle dba  14M Jan 30  2017 lib./ggscilog.so
-rwxr-x--- 1 oracle dba 2.1M Jan 30  2017 libdb-6.1.so
-rwxr-x--- 1 oracle dba 124K Jan 30  2017 libantlr3c.so
lrwxrwxrwx 1 oracle dba   47 Sep 22 10:04 libnnz12.so -> /opt/oracle/product/12.1.0.2.64/lib/libnnz12.so
lrwxrwxrwx 1 oracle dba   53 Sep 22 10:04 libclntsh.so.12.1 -> /opt/oracle/product/12.1.0.2.64/lib/libclntsh.so.12.1
lrwxrwxrwx 1 oracle dba   45 Sep 22 10:05 libons.so -> /opt/oracle/product/12.1.0.2.64/lib/libons.so
lrwxrwxrwx 1 oracle dba   57 Sep 22 10:05 libclntshcore.so.12.1 -> /opt/oracle/product/12.1.0.2.64/lib/libclntshcore.so.12.1

  
(OR)
To Avoid this issue include $ORACLE_HOME/lib, $OGG_HOME in bash profile

Sunday, November 4, 2018

WARNING OGG-01519 in Oracle GoldenGate Replicat



Today my GG Checkpoint table not updating and my Replicat NOT moving at all but the lag is ZERO
I tried even bouncing the replicat but still NO Luck

DB Version:  12.1.0.2.0
GG Version:  12.2

GGSCI (trgprod.domain.com) 4> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING
REPLICAT    RUNNING     REPSTDB    00:00:00      00:00:00
REPLICAT    RUNNING     REPUPDB    00:00:00      00:00:01
REPLICAT    RUNNING     REPTGDB      00:00:00      00:00:05


GGSCI (trgprod.domain.com) 5> info REPTGDB

REPLICAT   REPTGDB    Last Started 2018-10-24 14:35   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:13 ago)
Process ID           9255
Log Read Checkpoint  File /opt/oracle/GG/home/dirdat/reptgdb/MT000001301
                     2018-10-11 06:58:48.821076  RBA 16434843


GGSCI (trgprod.domain.com) 6> send REPTGDB status

Sending STATUS request to REPLICAT REPTGDB ...
  Current status: At EOF
  Sequence #: 1301
  RBA: 16434843
  0 records in current transaction

From above send command we can see that the GG is at the end of the trail file and processed all records in it

Ggserr.log:

2018-10-31 10:22:27  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): info REPTGDB.
2018-10-31 10:22:49  WARNING OGG-01519  Oracle GoldenGate Delivery for Oracle, REPTGDB.prm:  Waiting at EOF on input trail file /opt/oracle/GG/home/dirdat/reptgdb/MT000001301, which is not marked as complete; but succeeding trail file /opt/oracle/GG/home/dirdat/reptgdb/MT000001302 exists. If ALTER ETROLLOVER has been performed on source extract, ALTER EXTSEQNO must be performed on each corresponding downstream reader.

Trails Files are Accumulating but GG not processing and moving forward

trgprod:NOTSET:/home/oracle $ ls -lrth /opt/oracle/GG/home/dirdat/reptgdb/MT*

-rw-r----- 1 oracle dba  16M Oct 20 11:16 /opt/oracle/GG/home/dirdat/reptgdb/MT000001301
-rw-r----- 1 oracle dba  48M Oct 22 08:25 /opt/oracle/GG/home/dirdat/reptgdb/MT000001302
-rw-r----- 1 oracle dba  48M Oct 22 08:33 /opt/oracle/GG/home/dirdat/reptgdb/MT000001303
-rw-r----- 1 oracle dba  48M Oct 22 08:57 /opt/oracle/GG/home/dirdat/reptgdb/MT000001304
-rw-r----- 1 oracle dba  48M Oct 22 15:34 /opt/oracle/GG/home/dirdat/reptgdb/MT000001305
-rw-r----- 1 oracle dba  48M Oct 23 07:33 /opt/oracle/GG/home/dirdat/reptgdb/MT000001306
-rw-r----- 1 oracle dba  48M Oct 23 15:30 /opt/oracle/GG/home/dirdat/reptgdb/MT000001307
-rw-r----- 1 oracle dba  48M Oct 24 07:13 /opt/oracle/GG/home/dirdat/reptgdb/MT000001308
-rw-r----- 1 oracle dba  48M Oct 24 13:43 /opt/oracle/GG/home/dirdat/reptgdb/MT000001309
-rw-r----- 1 oracle dba  915 Oct 24 14:30 /opt/oracle/GG/home/dirdat/reptgdb/MT000001311
-rw-r----- 1 oracle dba 6.7M Oct 24 14:30 /opt/oracle/GG/home/dirdat/reptgdb/MT000001310
-rw-r----- 1 oracle dba  48M Oct 25 08:12 /opt/oracle/GG/home/dirdat/reptgdb/MT000001312
-rw-r----- 1 oracle dba  48M Oct 25 16:27 /opt/oracle/GG/home/dirdat/reptgdb/MT000001313
-rw-r----- 1 oracle dba  48M Oct 26 07:00 /opt/oracle/GG/home/dirdat/reptgdb/MT000001314
-rw-r----- 1 oracle dba  48M Oct 26 10:13 /opt/oracle/GG/home/dirdat/reptgdb/MT000001315
-rw-r----- 1 oracle dba  48M Oct 26 13:00 /opt/oracle/GG/home/dirdat/reptgdb/MT000001316
-rw-r----- 1 oracle dba  48M Oct 26 14:23 /opt/oracle/GG/home/dirdat/reptgdb/MT000001317
-rw-r----- 1 oracle dba  48M Oct 29 06:12 /opt/oracle/GG/home/dirdat/reptgdb/MT000001318
-rw-r----- 1 oracle dba  48M Oct 29 06:15 /opt/oracle/GG/home/dirdat/reptgdb/MT000001319
-rw-r----- 1 oracle dba  48M Oct 29 08:07 /opt/oracle/GG/home/dirdat/reptgdb/MT000001320

Reason:

This is due the reason that on the source side we performed etrollover for the EXTRACT/Pump but not done on target due to which replicat still waiting on EOF on the current trail file(MT000001301) but pump already started writing to next trail file(MT000001302) on Target side due to etrollover issued on source so this is expected on target side

FIX:

1.       Check whether GG processed all the records in the current trail file (MT000001301)

GGSCI (trgprod.domain.com) 6> send REPTGDB status

Sending STATUS request to REPLICAT REPTGDB ...
  Current status: At EOF
  Sequence #: 1301
  RBA: 16434843
  0 records in current transaction
 (Or)
   send <rep_name> logend

2.       STOP replicat
3.       Alter replicat with new sequence number & RBA

GGSCI (trgprod.domain.com) 10> Alter REPLICAT REPTGDB extseqno 1302, extrba 00

2018-10-31 10:35:52  INFO    OGG-06594  Replicat REPTGDB has been altered through GGSCI. Even the start up position might be updated, duplicate suppression remains active in next startup. To override duplicate suppression, start REPTGDB with NOFILTERDUPTRANSACTIONS option.

REPLICAT altered.

4.       Start replicat

GGSCI (trgprod.domain.com) 11> start REPTGDB

Sending START request to MANAGER ...
REPLICAT REPTGDB starting

Now it started moving and processing records


Same on Oracle DOC:

WARNING OGG-01519 in Oracle GoldenGate Replicat (Doc ID 2383240.1)



Sunday, August 19, 2018

Wohooo 1M Views Today


Time for the Celebration folks that my Blog reached 1 Million hits, Thanks Every one who are visiting my blog and i hope my posts are helpful at least 1 in a million and not going in vain :)

As Always i'll keep blogging and share my experiences of my oracle world , Cheers !!

Saturday, August 18, 2018

13c OEM Status Pending (Target addition is in Progress)



After discovering and adding database to my 13.2 OEM, target database is still in “status Pending (Target addition is in Progress)” even after few days

From OMS repo database……

SQL> select entity_name, entity_type, host_name, manage_status , promote_status, emd_url ,DISPLAY_NAME from SYSMAN.EM_MANAGEABLE_ENTITIES where entity_name like '%DEVDB%' and emd_url like '%oradev01%';

ENTITY_NAME            ENTITY_TYPE                         HOST_NAME            MANAGE_STATUS     PROMOTE_STATUS
---------------------- ------------------------------------------------------ -------------------- ------------- --------------
EMD_URL
-------------------------------------------------------------
DISPLAY_NAME
------------------------------------------------------------DEVDB                 oracle_database                                                  oradev01.domain.com             2              1
https://oradev01.domain.com:3872/emd/main/
DEVDB

From the above output we can see that
managed_status=2 and promote_status=1 indicates that the target is not yet promoted to the agent side


Now from Target database (DEVDB) go to the agent and check below target status

oradev01:DEVDB:/home/oracle $ cd /opt/oracle/product/oemagent/agent13c/agent_13.2.0.0.0/bin

oradev01:DEVDB:/opt/oracle/product/oemagent/agent13c/agent_13.2.0.0.0/bin $ ./emctl config agent listtargets
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
[oradev01.domain.com, host]                                 
[oradev01.domain.com:3872, oracle_emd]

If you check the above output, it doesn’t listed any of the databases existing ion the server and according to (Doc ID 2286993.1) root.sh has not been executed after the installation of Agent.
To check whether root.sh was executed on target DB

$ cd <AGENT_BASE_DIR>/sbin

oradev01:DEVDB:/opt/oracle/product/oemagent/agent13c/agent_13.2.0.0.0/sbin $ ls -ltr nm*
-rw-r----- 1 oracle dba    9615 Aug  1  2016 nmr_macro_list
-rwx--x--x 1 oracle dba 4786905 Sep 30  2016 nmo.0
-rwx--x--x 1 oracle dba 4703504 Sep 30  2016 nmoconf
-rwx--x--x 1 oracle dba   53058 Sep 30  2016 nmb.0
-rwx--x--x 1 oracle dba   50457 Sep 30  2016 nmgsshe.0
-rwx------ 1 oracle dba  104650 Sep 30  2016 nmr.0
-rwx------ 1 oracle dba   34522 Sep 30  2016 nmrconf
-rwx--x--x 1 oracle dba 4708480 Sep 30  2016 nmopdpx.0
-rwx--x--x 1 oracle dba  105929 Sep 30  2016 nmhs.0
-rwx--x--x 1 oracle dba  585242 Feb 10  2017 nmosudo.0


Hmmm looks like it is NOT

Solution:

Run Root.sh script on target server (oradev01)  as a ROOT user
Verify again owner for nm* files

[root@oradev01 root] # /opt/oracle/product/oemagent/agent13c/agent_13.2.0.0.0/root.sh
Finished product-specific root actions.
/etc exist

oradev01:DEVDB:/opt/oracle/product/oemagent/agent13c/agent_13.2.0.0.0/sbin $ ls -lrth
total 30M
-rw-r----- 1 root   dba  9.4K Aug  1  2016 nmr_macro_list
-rwx--x--x 1 oracle dba  4.6M Sep 30  2016 nmo.0
-rwx--x--x 1 oracle dba  4.5M Sep 30  2016 nmoconf
-rwx--x--x 1 oracle dba   52K Sep 30  2016 nmb.0
-rwx--x--x 1 oracle dba   50K Sep 30  2016 nmgsshe.0
-rwx------ 1 oracle dba  103K Sep 30  2016 nmr.0
-rwx------ 1 oracle dba   34K Sep 30  2016 nmrconf
-rwx--x--x 1 oracle dba  4.5M Sep 30  2016 nmopdpx.0
-rwx--x--x 1 oracle dba  104K Sep 30  2016 nmhs.0
-rwx--x--x 1 oracle dba  572K Feb 10  2017 nmosudo.0
-rwx--x--x 1 root   root 4.6M Aug  6 13:11 nmo.new.bak
-rw-r----- 1 root   dba   188 Aug  6 13:11 nmo_public_key.txt
-rwsr-x--- 1 root   dba  4.6M Aug  6 13:11 nmo
-rwsr-x--- 1 root   dba  104K Aug  6 13:11 nmhs
-rwsr-x--- 1 root   dba   52K Aug  6 13:11 nmb
-rwx------ 1 root   root 103K Aug  6 13:11 nmr.new.bak
-rwsr-x--- 1 root   dba  103K Aug  6 13:11 nmr
-rwxr-xr-x 1 root   dba  572K Aug  6 13:11 nmosudo
-rwxr-xr-x 1 root   dba   50K Aug  6 13:11 nmgsshe
-rwxr-xr-x 1 root   dba  4.5M Aug  6 13:11 nmopdpx


Now Delete the Target database from OEM and rediscover again and this time it will show as up and Running Cheers !!!!
EM 13c : After Discovery, Targets Are Stuck In "Status Pending (Target Addition In Progress)" Status (Doc ID 2286993.1)




Auto Scroll Stop Scroll