Saturday, September 5, 2015

OGG-01496 Failed to open target trail file


My pump process is down for so many days and when I tried to start the process it is getting ABENDED because it is trying to write in the remote trail file which is MISSING



GGSCI (oracleqa011.domain.com) 7> start PUMPQA

Sending START request to MANAGER ...
EXTRACT PUMPQA starting


GGSCI (oracleqa011.domain.com) 8> info PUMPQA

EXTRACT    PUMPQA  Last Started 2015-08-20 19:59   Status ABENDED
Checkpoint Lag       00:00:00 (updated 00:00:28 ago)
Log Read Checkpoint  File ./dirdat/cb000011  --> This is  source trail file
                     2015-08-20 19:59:30.000000  RBA 36679


GGSCI (oracleqa011.domain.com) 9> view report PUMPQA


2015-08-20 19:59:44  ERROR   OGG-01496  Failed to open target trail file /u01/oracle/TARGET/cb000008, at RBA 8448365.

2015-08-20 19:59:44  ERROR   OGG-01668  PROCESS ABENDING.


Here pump process is trying to write to the file “cb000008” where is has left before, since it’s been long ago this file doesn’t exist in the target location


Tried doing begin now command, but doesn’t work

GGSCI (oracleqa011.domain.com) 10> alter PUMPQA begin now
EXTRACT altered.

Still “ABENDED”    -->  because it will begin now but try to write to the same trail file


Solution:

Do “ETROLLOVER” for pump process, this will roll over to the next trail file in the sequence (here cb000009)

GGSCI (oracleqa011.domain.com) 10> ALTER EXTRACT PUMPQA ETROLLOVER

2015-08-20 20:02:22  INFO    OGG-01520  Rollover performed.  For each affected output trail of Version 10 or higher format, after starting the source extract, issue ALTER EXTSEQNO for that trail's reader (either pump EXTRACT or REPLICAT) to move the reader's scan to the new trail file;  it will not happen automatically.
EXTRACT altered.


GGSCI (oracleqa011.domain.com) 11> start PUMPQA

Sending START request to MANAGER ...
EXTRACT PUMPQA starting


GGSCI (oracleqa011.domain.com) 12> info PUMPQA

EXTRACT    PUMPQA  Last Started 2015-08-20 20:02   Status RUNNING
Checkpoint Lag       00:03:09 (updated 00:00:08 ago)
Log Read Checkpoint  File ./dirdat/cb000011
                     2015-08-20 19:59:30.000000  RBA 36679



In the target server I see “cb000009” has been created and writing into it.


ERROR OGG-01224 Address already in use.


DB version: 11.2.0.4 2-Node RAC
OS:  RHEL 6
GG version :  11.2.1.0.3

Today when i'm trying to start my manager process it is not starting 

GGSCI (oracledev01) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED
JAGENT      STOPPED
EXTRACT     ABENDED     EOECBP      291:56:23     27:02:50

GGSCI (oracledev01) 2> start mgr

Manager started.

But when I do info mgr it says manager is DOWN…..

GGSCI (oracledev01) 3> info mgr

Manager is DOWN!


GGSCI (oracledev01) 4> view params mgr

PORT 7809
DYNAMICPORTLIST 7840-9860


GGSCI (oracledev01) 5> view report mgr


***********************************************************************
                 Oracle GoldenGate Manager for Oracle
 Version 11.2.1.0.14 17547423 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_131022.0605
   Linux, x64, 64bit (optimized), Oracle 11g on Oct 22 2013 07:38:49

Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.


                    Starting at 2015-08-11 19:56:22
***********************************************************************

Operating System Version:
Linux
Version #1 SMP Fri May 29 10:16:43 EDT 2015, Release 2.6.32-504.23.4.el6.x86_64
Node: sl73orcdbdbq005
Machine: x86_64
                         soft limit   hard limit
Address Space Size   :    unlimited    unlimited
Heap Size            :    unlimited    unlimited
File Size            :    unlimited    unlimited
CPU Time             :    unlimited    unlimited

Process id: 36788

Parameters...

PORT 7809
DYNAMICPORTLIST 7840-9860


***********************************************************************
**                     Run Time Messages                             **
***********************************************************************


Source Context :
  SourceModule            : [mgr.main]
  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150267/oggcore/OpenSys/src/app/mgr/mgr.c]
  SourceFunction          : [init_functions]
  SourceLine              : [3390]
  ThreadBacktrace         : [8] elements
                          : [/gg/GG11/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7ff4595509fe]]
                          : [/gg/GG11/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x2cc) [0x7ff45954974c]]
                          : [/gg/GG11/libgglog.so(_MSG_ERR_TCP_GENERIC(CSourceContext*, char const*, CMessageFactory::MessageDisposition)+0x31) [0x7ff4595318a5]]
                          : [./mgr(init_functions(int, char**)+0x7f5) [0x4511c5]]
                          : [./mgr(main_loop(int, char**)+0x4c) [0x454aec]]
                          : [./mgr(main+0xf2) [0x455362]]
                          : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3d9361ed5d]]
                          : [./mgr(__gxx_personality_v0+0x142) [0x43efca]]

2015-08-11 19:56:22  ERROR   OGG-01224  Address already in use.

2015-08-11 19:56:22  ERROR   OGG-01668  PROCESS ABENDING.


Error in ggserr.log:

2015-08-11 19:56:10  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): info mgr.
2015-08-11 19:56:19  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): info all.
2015-08-11 19:56:22  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): start mgr.
2015-08-11 19:56:22  ERROR   OGG-01224  Oracle GoldenGate Manager for Oracle, mgr.prm:  Address already in use.
2015-08-11 19:56:22  ERROR   OGG-01668  Oracle GoldenGate Manager for Oracle, mgr.prm:  PROCESS ABENDING.


CAUSE:

Previous mgr process is still running, which used port 7809 according to parameter file.


SOLUTION:

Use other port and start
OR
To release the port perform below steps


As a root or Goldengate owner check the port

[root@oracledev01~]#  netstat -nap | grep 7809
tcp        0      0 0.0.0.0:7809                0.0.0.0:*                   LISTEN      44402/./mgr

From the above output we see mgr is already running on port 7809 so kill this old process and start again

[root@oracledev01~]# kill -9 44402
[root@oracledev01~]# netstat -nap | grep 7809
Nothing displays

[oracle@oracledev01 GG11]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.14 17547423 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_131022.0605_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Oct 22 2013 11:03:39

Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.


GGSCI (oracledev01) 1> start mgr

Manager started.


GGSCI (oracledev01) 2> info mgr


Manager is running (IP port oracledev01.7809).


Auto Scroll Stop Scroll