Friday, October 7, 2016

13c OMS 13.1.0.0.160719 PSU (Bug # 23134365)


NOTE:  Beginning with this PSU, the fifth digit version number has changed to a new format YYMMDD. For more information, see My Oracle Support Note 2061926.1 - "Oracle Database, Enterprise Manager and Middleware - Change to Patch Numbering from Nov 2015 on wards"

OMS : Standalone server on RHEL 7
Make sure we set the PATH environment variables and I downloaded patch to directory /opt/oracle/CB/PSU_JULY_OMS/23134365


[oracle@OMSHOST6 OPatch]$ echo $OMS_HOME
/opt/oracle/product/middleware/oms
[oracle@OMSHOST6 ~]$ export ORACLE_HOME=/opt/oracle/product/middleware/oms
[oracle@OMSHOST6 ~]$ export PATH=$ORACLE_HOME/OMSPatcher:$PATH
[oracle@OMSHOST6 ~]$ export PATH=$ORACLE_HOME/bin:$PATH

OMSPatcher is a new feature of EM 13c to apply patches to OMS home.
To use the OMSPatcher functionality, you must set the WebLogic Administration Server credential access. If you do not want to set the credentials every time you are prompted while patching the OMS, you can update the property file

Applying patch to OMS can be done in two ways:         
          1. Applying patch without using property file
          2. Applying patch using property file  (I’m using this option)

Creating property file:

[oracle@OMSHOST6 ]$ mkdir -p /opt/oracle/product/middleware/oms/OMSPatcher/property_file
[oracle@OMSHOST6 ]$ cd  $OMS_HOME/OMSPatcher/wlskeys

[oracle@OMSHOST6 wlskeys]$ ./createkeys.sh –oh /opt/oracle/product/middleware/oms -location /opt/oracle/product/middleware/oms/OMSPatcher/property_file
Please enter weblogic admin server username: weblogic
Please enter weblogic admin server password:
Warning: weblogic.Admin is deprecated and will be removed in a future release. WLST should be used instead of weblogic.Admin.
Creating the key file can reduce the security of your system if it is not kept in a secured location after it is created. Creating new key...

Trying to get configuration and key files for the given inputs...
This operation will take some time. Please wait for updates...
User configuration file created: /opt/oracle/product/middleware/oms/OMSPatcher/property_file/config
User key file created: /opt/oracle/product/middleware/oms/OMSPatcher/property_file/key
'createkeys' succeeded.



[oracle@OMSHOST6 wlskeys]$ cd /opt/oracle/product/middleware/oms/OMSPatcher/property_file
[oracle@OMSHOST6 property_file]$ ll
total 8
-rw-r--r--. 1 oracle dba 227 Oct  6 15:38 config
-rw-r--r--. 1 oracle dba  64 Oct  6 15:37 key
[oracle@OMSHOST6 ~]$ cd /opt/oracle/CB/PSU_JULY_OMS/23134365

[oracle@OMSHOST6 23134365]$ omspatcher apply –analyze  –property_file /opt/oracle/product/middleware/oms/OMSPatcher/property_file
OMSPatcher Automation Tool
Copyright (c) 2015, Oracle Corporation.  All rights reserved.

Syntax Error... Unrecognized Option for Apply
Please refer 'omspatcher apply -help' for details

OMSPatcher failed with error code 14

 [oracle@OMSHOST6 23134365]$ omspatcher apply  –property_file /opt/oracle/product/middleware/oms/OMSPatcher/property_file   –analyze
OMSPatcher Automation Tool
Copyright (c) 2015, Oracle Corporation.  All rights reserved.

Syntax Error... Unrecognized Option for Apply
Please refer 'omspatcher apply -help' for details

OMSPatcher failed with error code 14


Tried many different ways…………but still same error then I thought that this is the command copied from oracle support read me file so instead of that I typed the entire command manually then it worked…!   Damn…. never copy as is from oracle DOC (Lesson learned J)


[oracle@OMSHOST6 23134365]$ omspatcher apply -analyze -property_file /opt/oracle/product/middleware/oms/OMSPatcher/property_file
OMSPatcher Automation Tool
Copyright (c) 2015, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.6.0.0.1
OUI version        : 13.6.0.0.0
Running from       : /opt/oracle/product/middleware/oms
Log file location  : /opt/oracle/product/middleware/oms/cfgtoollogs/omspatcher/opatch2016-10-06_16-03-24PM_1.log

OMSPatcher log file: /opt/oracle/product/middleware/oms/cfgtoollogs/omspatcher/23134365/omspatcher_2016-10-06_16-03-30PM_analyze.log

Please enter OMS weblogic admin server URL(t3s://OMSHOST6.domain.com:7102):> enter
Please enter OMS weblogic admin server username(weblogic):> enter
Please enter OMS weblogic admin server password:>


Configuration Validation: Success

Running apply prerequisite checks for sub-patch(es) "23134365" and Oracle Home "/opt/oracle/product/middleware/oms"...
Sub-patch(es) "23134365" are successfully analyzed for Oracle Home "/opt/oracle/product/middleware/oms"


Complete Summary
================


All log file names referenced below can be accessed from the directory "/opt/oracle/product/middleware/oms/cfgtoollogs/omspatcher/2016-10-06_16-03-24PM_SystemPatch_23134365_1"

Prerequisites analysis summary:
-------------------------------

The following sub-patch(es) are applicable:

             Featureset   Sub-patches                                          Log file
             ----------   -----------                                          --------
  oracle.sysman.top.oms      23134365   23134365_ApplyPrereq2016-10-06_16-04-12PM_2.log



Log file location: /opt/oracle/product/middleware/oms/cfgtoollogs/omspatcher/23134365/omspatcher_2016-10-06_16-03-30PM_analyze.log

OMSPatcher succeeded.



[oracle@OMSHOST6 23134365]$ cd /opt/oracle/product/middleware/oms/bin
[oracle@OMSHOST6 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server is Up

[oracle@OMSHOST6 bin]$ ./emctl stop oms
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Stopping Oracle Management Server...
Oracle Management Server Successfully Stopped
Oracle Management Server is Down
JVMD Engine is Down

[oracle@OMSHOST6 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Down
JVMD Engine is Down
BI Publisher Server is Up


[oracle@OMSHOST6 23134365]$ omspatcher apply -property_file /opt/oracle/product/middleware/oms/OMSPatcher/property_file
OMSPatcher Automation Tool
Copyright (c) 2015, Oracle Corporation.  All rights reserved.


OMSPatcher version : 13.6.0.0.1
OUI version        : 13.6.0.0.0
Running from       : /opt/oracle/product/middleware/oms
Log file location  : /opt/oracle/product/middleware/oms/cfgtoollogs/omspatcher/opatch2016-10-06_16-10-56PM_1.log

OMSPatcher log file: /opt/oracle/product/middleware/oms/cfgtoollogs/omspatcher/23134365/omspatcher_2016-10-06_16-11-02PM_deploy.log

Please enter OMS weblogic admin server URL(t3s://OMSHOST6.domain.com:7102):>
Please enter OMS weblogic admin server username(weblogic):>
Please enter OMS weblogic admin server password:>


Configuration Validation: Success

Running apply prerequisite checks for sub-patch(es) "23134365" and Oracle Home "/opt/oracle/product/middleware/oms"...
Sub-patch(es) "23134365" are successfully analyzed for Oracle Home "/opt/oracle/product/middleware/oms"

To continue, OMSPatcher will do the following:
[Patch and deploy artifacts]   : Apply sub-patch(es) [ 23134365 ] to component "oracle.sysman.top.oms" and version "13.1.0.0.0";
                                 Apply RCU artifact with patch "/opt/oracle/product/middleware/oms/.omspatcher_storage/23134365_Jul_4_2016_05_05_46/original_patch";
                                 Register MRS artifact "targetType";
                                 Register MRS artifact "storeTargetType";
                                 Register MRS artifact "default_collection"


Do you want to proceed? [y|n]
y
User Responded with: Y

Applying sub-patch "23134365 " to component "oracle.sysman.top.oms" and version "13.1.0.0.0"...

Updating repository with RCU reference file "/opt/oracle/product/middleware/oms/.omspatcher_storage/23134365_Jul_4_2016_05_05_46/original_patch"

Registering service "targetType" with register file "/opt/oracle/product/middleware/oms/sysman/metadata/targetType/oldVersions/host/12.1.0.5/host.xml"...

Registering service "storeTargetType" with register file "/opt/oracle/product/middleware/oms/sysman/metadata/targetType/oldVersions/host/12.1.0.5/host.xml"...

Registering service "default_collection" with register file "/opt/oracle/product/middleware/oms/sysman/metadata/default_collection/oldVersions/host/12.1.0.5/host.xml"...

Complete Summary
================


All log file names referenced below can be accessed from the directory "/opt/oracle/product/middleware/oms/cfgtoollogs/omspatcher/2016-10-06_16-10-56PM_SystemPatch_23134365_1"

Patching summary:
-----------------

Binaries of the following sub-patch(es) have been applied successfully:

                        Featureset   Sub-patches                                     Log file
                        ----------   -----------                                     --------
  oracle.sysman.top.oms_13.1.0.0.0      23134365   23134365_opatch2016-10-06_16-11-35PM_3.log

Deployment summary:
-------------------

The following artifact(s) have been successfully deployed:

               Artifacts                                                      Log file
               ---------                                                      --------
                     SQL       rcu_applypatch_original_patch_2016-10-06_16-12-39PM.log
          MRS-targetType           emctl_register_targetType_2016-10-06_16-21-39PM.log
     MRS-storeTargetType      emctl_register_storeTargetType_2016-10-06_16-22-12PM.log
  MRS-default_collection   emctl_register_default_collection_2016-10-06_16-22-21PM.log


Log file location: /opt/oracle/product/middleware/oms/cfgtoollogs/omspatcher/23134365/omspatcher_2016-10-06_16-11-02PM_deploy.log

OMSPatcher succeeded.



[oracle@OMSHOST6 bin]$ ./emctl start oms
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
Starting Oracle Management Server...
WebTier Successfully Started
Oracle Management Server Successfully Started
Oracle Management Server is Up
JVMD Engine is Up
Starting BI Publisher Server ...
BI Publisher Server Already Started
BI Publisher Server is Up


[oracle@OMSHOST6 bin]$ ./emctl status oms
Oracle Enterprise Manager Cloud Control 13c Release 1
Copyright (c) 1996, 2015 Oracle Corporation.  All rights reserved.
WebTier is Up
Oracle Management Server is Up
JVMD Engine is Up
BI Publisher Server is Up



Refer Oracle doc:
Patch Set Update 13.1.0.0.160719 for Oracle Management Service (Doc ID 2158215.1)



Friday, September 23, 2016

FAL[client]: Failed to request gap sequence

Primary and standby DB : 11.2.0.3 on RHEL 6

Primary DB: PROD
Standby DB: STANDBY

We got an alert for one of our production standby database is out of sync

Standby Alert log :

FAL[client]: Failed to request gap sequence
 GAP - thread 1 sequence 32651-32656
 DBID 3477518933 branch 814361045
FAL[client]: All defined FAL servers have been attempted.
------------------------------------------------------------
Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization
parameter is defined to a value that's sufficiently large
enough to maintain adequate log switch information to resolve
archivelog gaps.
------------------------------------------------------------
Fri Sep 23 15:43:48 2016
db_recovery_file_dest_size of 25600 MB is 0.00% used. This is a
user-specified limit on the amount of space that will be used by this
database for recovery-related files, and does not reflect the amount of
space available in the underlying filesystem or ASM diskgroup.
Fri Sep 23 15:54:41 2016

Then checked for the archive gap….

standby_server:STANDBY:/opt/oracle/ARCH/standby $ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Fri Sep 23 15:29:26 2016

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.Sfference" FROM (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROAD#)) ARCH,  (SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HIS  WHERE ARCH.THREAD# = APPL.THREAD# ORDER BY 1;

    Thread Last Sequence Received Last Sequence Applied   Difference
---------- ----------------------   ---------------------       ----------
         1                  32687                   32650               37         <<-----    37 arch logs gap

SQL> select message, timestamp from v$dataguard_status where severity in ('Error','Fatal')
order by timestamp; 

no rows selected

SQL> SELECT PROCESS, STATUS FROM V$MANAGED_STANDBY;

PROCESS   STATUS
--------- ------------
ARCH      CONNECTED
ARCH      CONNECTED
MR(fg)    WAIT_FOR_GAP             --->>  Recovery says WAITING FOR GAP
RFS       IDLE

NOTE : Before going for any type of troubleshooting for standby databases first thing we need to check is where standby have required archivelogs or not.

standby_server:STANDBY:/opt/oracle/ARCH/standby $ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Fri Sep 23 16:14:25 2016

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL> archive log list
Database log mode              Archive Mode
Automatic archival             Enabled
Archive destination            /opt/oracle/ARCH/standby/
Oldest online log sequence     32685
Next log sequence to archive   0
Current log sequence           32690


SQL>  SELECT * FROM V$ARCHIVE_GAP;

   THREAD# LOW_SEQUENCE# HIGH_SEQUENCE#
---------- ------------- --------------
         1         32651          32656


Check last sequence applied:

SQL> SELECT thread#,   Max(sequence#) "Last Standby Seq Applied" FROM   v$archived_log   WHERE  applied = 'YES'  GROUP  BY thread#   ORDER  BY 1;
 

   THREAD#   Last Standby Seq Applied
----------     ------------------------------
1                                  32650

$ cd /opt/oracle/ARCH/standby
standby_server:STANDBY:/opt/oracle/ARCH/standby $ ls -lrth
-rw-r----- 1 oracle dba 2.4M Sep 21 19:11 TS0001_0000032647_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  29M Sep 21 23:22 TS0001_0000032648_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  15K Sep 21 23:22 TS0001_0000032649_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  78K Sep 21 23:29 TS0001_0000032650_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  38M Sep 23 12:51 TS0001_0000032657_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  39M Sep 23 12:51 TS0001_0000032658_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  66M Sep 23 12:52 TS0001_0000032659_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  52M Sep 23 12:52 TS0001_0000032660_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  34M Sep 23 12:52 TS0001_0000032661_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  38M Sep 23 12:52 TS0001_0000032662_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  66M Sep 23 12:53 TS0001_0000032663_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  85M Sep 23 12:53 TS0001_0000032664_STANDBY0814361045.ARC
-rw-r----- 1 oracle dba  29M Sep 23 12:53 TS0001_0000032665_STANDBY0814361045.ARC

From the above output we can clearly see that from Seq# 32651   -  seq# 32656 are MISSING, uummmm now we know what need to be done to resolve

NOTE: Luckily i have archivelogs exists in primary side if doesn't exists any more then you need to rebuild standby again(ex: rman incremental backups,etc.,)


Primary:

oracleprimaryserver:PROD:/opt/oracle/ARCH/Primary $ sftp oracle@standby_server:/opt/oracle/ARCH/standby
Connecting to standby_server...
oracle@standby_server's password:
Changing to: /opt/oracle/ARCH/standby
sftp> put TS0001_0000032651_STANDBY0814361045.ARC
Uploading TS0001_0000032651_STANDBY0814361045.ARC to /opt/oracle/ARCH/standby/TS0001_0000032651_STANDBY0814361045.ARC
TS0001_0000032651_STANDBY0814361045.ARC                                      100%  278KB 278.1KB/s   00:0

Similarly, all……………….

Once copied then STOP Recover process on standby:


Note: open another Terminal and watch the alert log

standby_server:STANDBY:/opt/oracle/ARCH/standby $ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Fri Sep 23 16:09:08 2016

Copyright (c) 1982, 2011, Oracle.  All rights reserved.


Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SQL>  Recover managed standby database cancel;

SQL> alter database register logfile '/FULL_PATH/TS0001_0000032651_STANDBY0814361045.ARC';
Database altered.

Similarly register all the 6 logfiles and start the recovery process

SQL> recover managed standby database;

Now check again the gap

SQL> SELECT PROCESS, STATUS FROM V$MANAGED_STANDBY;

PROCESS   STATUS
--------- ------------
ARCH      CONNECTED
ARCH      CONNECTED
MR(fg)    APPLYING_LOG                   <<-- See recovery started applying
RFS       IDLE


SQL> SELECT ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference"
FROM (SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH,
(SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL
WHERE ARCH.THREAD# = APPL.THREAD# ORDER BY 1;

    Thread Last Sequence Received Last Sequence Applied Difference
---------- ---------------------- --------------------- ----------
         1                  32689                 32689          0



Hope this helps ….!!


Thursday, March 3, 2016

ERROR OGG-01705 Input checkpoint position for input trail file is greater than the size of the file

Pump, Extract or Replicat may get stuck on a trail file even if there are more trail files available in machine for the reader object to process. This issue occurs when the pump extract / Replicat read checkpoint RBA is larger than the local trail file size that it reads.
Here my Replicat got abended with the following error message …….

ERROR   OGG-01705  Input checkpoint position 33144618 for input trail file './dirdat/c4001256' is greater than the size of the file (33079717).  Please consult Oracle Knowledge Management Doc ID 1138409.1. for instructions.


GGSCI (oracledev01) 9> info REPDEV

REPLICAT   REPDEV  Last Started 2016-03-01 19:10   Status ABENDED
Checkpoint Lag       00:00:00 (updated 240:44:11 ago)
Log Read Checkpoint  File ./dirdat/c4001256
                     2016-02-20 19:19:49.000207  RBA 33144618

Now let’s check the trail file size…………

 [oracle@oracledev01 GG11]$ ls -l dirdat/c4001256
-rw-r----- 1 oracle oinstall 33079717 Feb 22 11:48 dirdat/c4001256

From above we can see that RBA is greater than the trail file size

FILTERDUPTRANSACTIONS
When using this option to start the replicat, it will use the information already stored in the checkpoint table and filter out transactions already applied.
This option is valid only for Replicat in nonintegrated mode. Integrated mode handles duplicate transactions transparently.

[oracle@oracledev01 GG11]$ ./ggsci

Oracle GoldenGate Command Interpreter for Oracle
Version 11.2.1.0.14 17547423 OGGCORE_11.2.1.0.0OGGBP_PLATFORMS_131022.0605_FBO
Linux, x64, 64bit (optimized), Oracle 11g on Oct 22 2013 11:03:39
Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.

GGSCI (oracledev01) 1> start replicat REPDEV FILTERDUPTRANSACTIONS
Sending START request to MANAGER ...
REPLICAT REPDEV starting

GGSCI (oracledev01) 2> info REPDEV
REPLICAT   REPDEV  Last Started 2016-03-01 20:06   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:04 ago)
Log Read Checkpoint  File ./dirdat/c4001257
                                         First Record  RBA 0


GGSCI (oracledev01) 3> info REPDEV
REPLICAT   REPDEV  Last Started 2016-03-01 20:06   Status RUNNING
Checkpoint Lag       240:46:27 (updated 00:00:03 ago)
Log Read Checkpoint  File ./dirdat/c4001257
                   2016-02-20 19:19:57.964167  RBA 276602

From the above we can see my replicat has moved on to the next trail file “c4001257” and it is up and running fine J

Refer:

OGG Extract / Replicat Checkpoint RBA Is Larger than Local Trail Size (Doc ID 1138409.1)

Auto Scroll Stop Scroll