OGG-01184 Expected 4 bytes, but got 0 bytes

 While starting my extract  it is getting ABENDED with below error

DB version  : 19c

OGG version : 19c

OS version : RHEL 7

GGSCI (myoraqe.domain.com) 4> start EXTQE

Sending START request to MANAGER ...

EXTRACT EXTQE starting

GGSCI (myoraqe.domain.com) 6> info EXTQE

EXTRACT    EXTQE   Last Started 2019-02-13 12:06   Status ABENDED

Checkpoint Lag       00:00:07 (updated 30:48:03 ago)

Log Read Checkpoint  Oracle Integrated Redo Logs

                     2019-02-12 05:18:53

                     SCN 88.3221809052 (381178931100)

2019-02-13 12:06:27  ERROR   OGG-01184  Expected 4 bytes, but got 0 bytes, in trail /goldengate/home/dirdat/SRC/sxtqe/tgxqe/mt000001148, seqno 1148, reading record trailer token at RBA 1525.

2019-02-13 12:06:27  ERROR   OGG-01668  PROCESS ABENDING.

Extract Abends with "Expected <n> bytes, but got <n> bytes, in trail " (Doc ID 1968574.1)

GGSCI (myoraqe.domain.com) 10> info PXTQE

EXTRACT    PXTQE   Last Started 2019-02-08 16:43   Status RUNNING

Checkpoint Lag       00:00:00 (updated 00:00:08 ago)

Process ID           170397

Log Read Checkpoint  File /goldengate/home/dirdat/SRC/sxtqe/tgxqe/mt000001148

                     First Record  RBA 0

GGSCI (myoraqe.domain.com) 11> send PXTQE logend

Sending LOGEND request to EXTRACT PXTQE ...

YES.

GGSCI (myoraqe.domain.com) 12> stop PXTQE

Sending STOP request to EXTRACT PXTQE ...

STOP request pending. Recovery is not complete.

This normal stop will wait and checkpoint recovery's work when recovery has finished.

To force Extract to stop now, use the SEND EXTRACT PXTQE, FORCESTOP command..

GGSCI (myoraqe.domain.com) 13> info PXTQE

EXTRACT    PXTQE   Last Started 2019-02-08 16:43   Status RUNNING

Checkpoint Lag       00:00:00 (updated 00:00:06 ago)

Process ID           170397

Log Read Checkpoint  File /goldengate/home/dirdat/SRC/sxtqe/tgxqe/mt000001148

                     First Record  RBA 0

GGSCI (myoraqe.domain.com) 14> !

info PXTQE

EXTRACT    PXTQE   Last Started 2019-02-08 16:43   Status RUNNING

Checkpoint Lag       00:00:00 (updated 00:00:06 ago)

Process ID           170397

Log Read Checkpoint  File /goldengate/home/dirdat/SRC/sxtqe/tgxqe/mt000001148

                     First Record  RBA 0

GGSCI (myoraqe.domain.com) 15> send PXTQE logend

Sending LOGEND request to EXTRACT PXTQE ...

YES.

GGSCI (myoraqe.domain.com) 16> stop PXTQE !

Sending FORCESTOP request to EXTRACT PXTQE ...

STOP request will be executed immediately (recovery aborted).

GGSCI (myoraqe.domain.com) 17> alter EXTQE ETROLLOVER

2019-02-13 12:11:38  INFO    OGG-01520  Rollover performed.  For each affected output trail of Version 10 or higher format, after starting the source extract, issue ALTER EXTSEQNO for that trail's reader (either pump EXTRACT or REPLICAT) to move the reader's scan to the new trail file;  it will not happen automatically.

EXTRACT altered.

GGSCI (myoraqe.domain.com) 18> start EXTQE

Sending START request to MANAGER ...

EXTRACT EXTQE starting

GGSCI (myoraqe.domain.com) 19> info EXTQE

EXTRACT    EXTQE   Last Started 2019-02-13 12:11   Status RUNNING

Checkpoint Lag       25:04:57 (updated 00:00:03 ago)

Process ID           268862

Log Read Checkpoint  Oracle Integrated Redo Logs

                     2019-02-12 11:06:51

                     SCN 88.3221882653 (381179004701)

GGSCI (myoraqe.domain.com) 20> alter PXTQE extseqno 1149 extrba 00

EXTRACT altered.

GGSCI (myoraqe.domain.com) 21> start PXTQE

Sending START request to MANAGER ...

EXTRACT PXTQE starting

GGSCI (myoraqe.domain.com) 22> info PXTQE

EXTRACT    PXTQE   Last Started 2019-02-13 12:12   Status RUNNING

Checkpoint Lag       00:00:00 (updated 00:00:02 ago)

Process ID           270788

Log Read Checkpoint  File /goldengate/home/dirdat/SRC/sxtqe/tgxqe/mt000001149

                     First Record  RBA 0

GGSCI (myoraqe.domain.com) 23> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT      RUNNING

EXTRACT     RUNNING     ESXTQEQ1     00:00:05      00:00:07

EXTRACT     RUNNING     ESXTQEQ2     00:00:07      00:00:00

EXTRACT     RUNNING     EXTQE     00:00:05      00:00:02

EXTRACT     RUNNING     PSXTQEQ1     00:00:00      00:00:10

EXTRACT     STOPPED     PSXTQEQ2     00:00:00      339:08:25

EXTRACT     RUNNING     PXTQE     00:00:00      00:00:05

Goldengate Tracing

To enable trace for EXT/PUMP/REPLICAT we can achieve in 2 ways

1.        Using trace/trace2 command

2.        Using Activity Logging

GGSCI (myoradev.domain.com) 2> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

REPLICAT    RUNNING     RTRG1DB    82:10:06      00:00:00

REPLICAT    RUNNING     RTRG2DB    00:00:00      00:00:08

Add below trace command in your param file and the restart and wait for some time and stop the process then you see trace file with the path you specified is generated and you can view the file

TRACE /goldengate/home/dirrpt/trace.trc

(or)

TRACE2 /goldengate/home/dirrpt/trace2.trc

where

TRACE    :   Provides step-by-step processing information.

TRACE2  :   Identifies the code segments on which Extract or Replicat is spending the most time.

Or if you want to enable trace ONLINE without stopping the GG Process then use the below send command which will enable trace and then turn off whenever you want

GGSCI (myoradev.domain.com) 3> send RTRG1DB trace RTRG1DB_trace.txt

Sending trace request to REPLICAT RTRG1DB ...

Trace file /goldengate/home/RTRG1DB_trace.txt opened.

To Stop/Disable Trace :

GGSCI (myoradev.domain.com) 4> send RTRG1DB trace off

Sending trace request to REPLICAT RTRG1DB ...

Closing all trace files..

Activity tracing :

1. Please save the below xml file with your process name like gglog-EXTNAME.xml , the process name should be CAPITAL letters .

2. Copy gglog-EXTNAME.xml to the Oracle GoldenGate Installation Directory.

3. Start the process and perform the DML/DDL operations

4. Move the gglog-EXTNAME.xml file to different location. Then it will stop the logging.

5. There will be a log generated in the same Oracle GoldenGate installation directory with name gglog-EXTNAME.log

<?xml version="1.0"?>

<configuration reset="true">

<appender name="myAppender" class="RollingFileAppender">

<param name="File" value="gglog-%I.log"/>

<param name="MaxBackupIndex" value="9"/>

<param name="BufferedIO" value="false"/>

<param name="MaxFileSize" value="10MB" />

<layout class="PatternLayout">

<param name="Pattern" value="%d{%Y-%m-%d %H:%M:%S} %-5p|%-30.30c|%5L %-24.24C{2} | %m%n"/>

</layout>

</appender>

<root>

<level value="all"/>

<appender-ref name="myAppender"/>

</root>

</configuration>

Oracle Reference Doc: 

OGG Activity Logging Tracing (Doc ID 1204284.1)

Release/Shrink UNDO tablespace

  

Sometimes there will be situations where we run out of DB storage due to huge undo growth and we wanted to shrink it to release some space for other datafiles in the database and The best way to shrink Undo tablespace is to switch to a new Undo tablespace and drop the old Undo tablespace

Create New UNDOTBS2 tablespace

SQL> create undo tablespace UNDOTBS2 datafile '+DATAC1' size 500M;

Tablespace created.

SQL> ALTER SYSTEM SET UNDO_TABLESPACE = UNDOTBS2 SCOPE=BOTH;

System altered.

SQL>  select tablespace_name, status, count(*) from dba_rollback_segs group by tablespace_name, status;

TABLESPACE_NAME                STATUS             COUNT(*)

------------------------------ ---------------- ----------

SYSTEM                         ONLINE                    1

UNDOTBS1                       OFFLINE                  12

UNDOTBS2                       ONLINE                   10

If there are Undo segments with a status other than OFFLINE in the tablespace to be dropped, we need to wait till they become OFFLINE.

Check any segments other than OFFLINE status using below

SQL>  select status,segment_name from dba_rollback_segs where status not in ('OFFLINE') and tablespace_name='UNDOTBS1';

no rows selected

Since there are no segments then we can easily drop the tablespace and release space

SQL> drop tablespace UNDOTBS1 including contents and datafiles;

Tablespace dropped.

SQL>  select tablespace_name, status, count(*) from dba_rollback_segs group by tablespace_name, status;

TABLESPACE_NAME                STATUS             COUNT(*)

------------------------------ ---------------- ----------

SYSTEM                         ONLINE                    1

UNDOTBS2                       ONLINE                   10

UNDOTBS3                       OFFLINE                  22

Now reset to same tablespace by creating same tablespace again

SQL> create undo tablespace UNDOTBS1 datafile '+DATAC1' size 500M;

Tablespace created.

SQL> ALTER SYSTEM SET UNDO_TABLESPACE =UNDOTBS1 SCOPE=BOTH;

System altered.

SQL>  select tablespace_name, status, count(*) from dba_rollback_segs group by tablespace_name, status;

TABLESPACE_NAME                STATUS             COUNT(*)

------------------------------ ---------------- ----------

SYSTEM                         ONLINE                    1

UNDOTBS1                       ONLINE                   15

UNDOTBS2                       OFFLINE                  10

Sometimes dropping the old tablespace may give ORA-30013: undo tablespace '%s' is currently in use. This error indicates you must wait for the undo tablespace to become unavailable. In other words, you must wait for existing transaction to commit or rollback.   

Also, According to ORACLE DOC:

On some platforms, disk space is not freed to the OS until the database is restarted.  The disk space will remain "allocated" from the OS perspective until the database restart.

expdp ORA-01775: looping chain of synonyms

While doing export on my 12.1 database i get below error

oracledev.domain.com:REPODB:/home/oracle $ expdp directory=STATSDIR dumpfile=REPODB_Triggers.dmp logfile=exp_REPODB_Triggers.log include=TRIGGER metrics=y schemas= Workers

Export: Release 12.1.0.2.0 - Production on Wed Oct 23 10:48:44 2019

Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

Username: / as sysdba

Connected to: Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
ORA-39001: invalid argument value
ORA-01775: looping chain of synonyms


FIX:


SQL> Select owner, object_name, object_type, status   from dba_objects  where object_name like '%SYS_EXPORT%';

OWNER
--------------------------------------------------------------------------------
OBJECT_NAME
--------------------------------------------------------------------------------
OBJECT_TYPE             STATUS
----------------------- -------
PUBLIC
SYS_EXPORT_SCHEMA_01
SYNONYM                 VALID

SYSTEM
SYS_EXPORT_TABLE_01
TABLE                   VALID

connect / as sysdba
drop public synonym sys_export_schema_01;

Now rerun export again

(OR)

If you dont want to drop the table now then give some job name ( job_name=<JOB_NAME>" ) and run the export 

Referrence:
DataPump Import Results In ORA-39001 Invalid Argument Value ORA-1775 Looping Chain Of Synonyms (Doc ID 459151.1)

IMPDP Job Hanging On STATISTICS/MARKER

DB Version : 12.1
OS Version : RHEL 6

While doing import for one of the Env my impdp job just waiting at STATISTICS/MARKER step and taking very very long though i'm doing small set of data import, and if i check the JOB status and it shows job is 100% Done and there is no movement at all even in DB Alert log

Impdp job at “Processing object type SCHEMA_EXPORT/STATISTICS/MARKER”

After doing some research online and according to oracle looks like this is Due to the BUG and below is the work around/Solutions

Data pump Import Job Is Hanging On STATISTICS/MARKER (Doc ID 2135702.1)

Find table in trail file using Goldengate logdump utility

Check TABLE (CUST.SALES) in Trail file using LOGDUMP

cd  $GG_HOME

$./logdump

Logdump 1 >ghdr on

Logdump 2 >detail on

Logdump 3 >detail data

Logdump 4 >open /goldengate/home/dirdat/xxxx/CS000023799

Current LogTrail is /goldengate/home/dirdat/xxxx/CS000023799

Logdump 6 >FILTER INCLUDE FILENAME CUST.sales

Logdump 7 >count

Scanned     10000 records, RBA    7356269, 2019/05/03 14:44:45.000.000

Scanned     20000 records, RBA   14846486, 2019/05/03 14:46:49.000.000

Scanned     30000 records, RBA   23290032, 2019/05/03 14:56:03.000.000

Scanned     40000 records, RBA   32171981, 2019/05/03 15:04:39.000.000

Scanned     50000 records, RBA   41262061, 2019/05/03 15:09:51.000.000

Partial record at RBA 46133384

Format 550 record, total len = 3268

LogTrail /goldengate/home/xxxx/hp000023799 has 0 records

Filtering matched            0 records

          suppressed     56004 records

 Logdump 8 >n