Saturday, July 15, 2023

Recover Standby DB when standby_file_management is MANUAL when new files added in primary 19c

 
 
When datafiles are added on primary I see 2 datafiles are in WRONG Location in standby database after the new standby build
 
On Standby :
09:52:55 SYSTEM@devdb_std >select file_name from dba_data_files order by 1;
 
FILE_NAME
--------------------------------------------------------------------------------
/acfs_data/devdb_std/datafile/DEVDB_STD/datafile/o1_mf_it2_lbtw9k_.dbf
/acfs_data/devdb_std/datafile/DEVDB_STD/datafile/o1_mf_it2_lbtymt_.dbf
/acfs_data/devdb_std/datafile/data_D-DEVDB_TS-A_FNO-8
/acfs_data/devdb_std/datafile/data_D-DEVDB_TS-AA_FNO-9
/acfs_data/devdb_std/datafile/data_D-DEVDB_TS-AD_FNO-10
…… Trimmed ….
 
09:54:26 SYSTEM@devdb_std >sho parameter convert
 
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
db_file_name_convert                 string      +DATAC1, /acfs_data/devdb_std/datafile
log_file_name_convert                string      +RECOC1, /acfs_reco/devdb_std/onlinelog
pdb_file_name_convert                string
 
09:53:19 SYSTEM@devdb_std >sho parameter standby
 
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
enabled_PDBs_on_standby              string      *
standby_db_preserve_states           string      NONE
standby_file_management              string      AUTO
standby_pdb_source_file_dblink       string
standby_pdb_source_file_directory    string
 
Reason :
 
The reason being by default when “standby_file_management” is AUTO OMF creates files in “db_file_name_convert” parameter in below location
 
<db_file_name_convert PATH>/<CAPS DBNAME>/datafile/
 
FIX:
 
Check the file number and status
 
10:31:33 SYSTEM@devdb_std > select file#, error, name from v$datafile_header where ERROR='FILE MISSING';
 
     FILE# ERROR
---------- -----------------------------------------------------------------
NAME
------------------------------------------------------------------------------------------------------------------------------------------
       781 FILE MISSING
 
 
 
10:34:39 SYSTEM@devdb_std >select name from v$datafile where file#=781 ;
 
NAME
-------------------------------------------------------------------------------------------------------------------
/u02/app/oracle/product/19.0.0.0/dbhome_1/dbs/UNNAMED00781
 
10:34:53 SYSTEM@devdb_std >alter database create datafile '/u02/app/oracle/product/19.0.0.0/dbhome_1/dbs/UNNAMED00781' as NEW;
alter database create datafile '/u02/app/oracle/product/19.0.0.0/dbhome_1/dbs/UNNAMED00781' as NEW
*
ERROR at line 1:
ORA-01275: Operation CREATE DATAFILE is not allowed if standby file management is automatic.
 
Cannot run when standby_file_management is AUTO so disable and rerun
 
10:36:13 SYSTEM@devdb_std >alter system set standby_file_management=MANUAL scope=both;
 
System altered.
 
10:36:25 SYSTEM@devdb_std >alter database create datafile '/u02/app/oracle/product/19.0.0.0/dbhome_1/dbs/UNNAMED00781' as NEW;
 
Database altered.
 
10:36:31 SYSTEM@devdb_std >alter system set standby_file_management=AUTO scope=both;
 
System altered.
 
 
Check the paths again
 
SQL >   select file_name from dba_data_files order by 1;
 
SQL >   select name from v$datafile where file#=779;
 
set the path correctly based on your location so that we cannot ran into this issue again
 
10:17:19 SYSTEM@devdb_std >alter system set db_create_file_dest='/acfs_data';
 
System altered.
 
10:23:41 SYSTEM@devdb_std >sho parameter file_dest
 
NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
audit_file_dest                      string      /u02/app/oracle/product/19.0.0.0/dbhome_1/rdbms/audit
db_create_file_dest                  string      /acfs_data
db_recovery_file_dest                string      /acfs_reco/devdb_std/archivelog
db_recovery_file_dest_size           big integer 3500G
remote_recovery_file_dest            string
 
 
Now add couple of datafiles in primary and verify.
 
10:52:44 SYSTEM@devdb_std >select name from v$datafile where file#=779;
 
NAME
------------------------------------------------------------------------------------------------------------------------------------------
/acfs_data/DEVDB_STD/datafile/o1_mf_imm2_lc2vtbcd_.dbf
 
 
More like this

Friday, June 23, 2023

Goldengate PMSRVR - Monitoring is not enabled in the GLOBALS parameter file

 
 
ENV :  OGG  19c (Classic Architecture)
 
On my new server I wanted to setup PMSRVR
Port should be open and listening  (default is 9004)
 
cd $GG_HOME
vi GLOBALS and add below entries
 
dev01.domain.com:NOTSET:/goldengate/home $ cat GLOBALS
GGSCHEMA ggadmin
ENABLEMONITORING UDP
 
 
dev01.domain.com:NOTSET:/goldengate/home $ GG
 
Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.
 
Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.
 
 
 
GGSCI (dev01.domain.com) 1> info all
 
Program     Status      Group       Lag at Chkpt  Time Since Chkpt
 
MANAGER     RUNNING                                          
EXTRACT     RUNNING     EDEV01     00:00:00      00:00:08   
EXTRACT     RUNNING     PDEV01     00:00:00      00:00:10   
REPLICAT    RUNNING     RDEV01     00:00:00      00:00:04   
 
 
GGSCI (dev01.domain.com) 2> start pmsrvr
ERROR: Monitoring is not enabled in the GLOBALS parameter file..
 
But GLOBALS file has entries already.
 
FIX :
 
dev01.domain.com:NOTSET:/goldengate/home $ GG
 
Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.
 
Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.
 
 
 
GGSCI (dev01.domain.com) 1> edit params ./GLOBALS  à This was EMPTY
GGSCHEMA ggadmin
ENABLEMONITORING UDP
 
 
-        Added below entries and save
 
GGSCI (dev01.domain.com) 3> exit
 
dev01.domain.com:NOTSET:/goldengate/home $ GG
 
Oracle GoldenGate Command Interpreter for Oracle
Version 19.1.0.0.4 OGGCORE_19.1.0.0.0_PLATFORMS_191017.1054_FBO
Linux, x64, 64bit (optimized), Oracle 19c on Oct 17 2019 21:16:29
Operating system character set identified as UTF-8.
 
Copyright (C) 1995, 2019, Oracle and/or its affiliates. All rights reserved.
 
 
 
GGSCI (dev01.domain.com) 1> info all
 
Program     Status      Group       Lag at Chkpt  Time Since Chkpt
 
MANAGER     STOPPED                                          
JAGENT      STOPPED                                          
PMSRVR      STOPPED                                          
EXTRACT     RUNNING     EDEV01     00:00:00      00:00:05    
EXTRACT     RUNNING     PDEV01     00:00:00      00:00:11    
REPLICAT    RUNNING     RDEV01     00:00:00      00:00:04   
 
 
GGSCI (dev01.domain.com) 2> start PMSRVR
PMSRVR started
 
 
Refer more on PMSRVR here :
http://chandu208.blogspot.com/search?q=pmsrvr
 
OMC: Discovery of Oracle GoldenGate in Oracle Management Cloud Fails with Error: Monitoring is not enabled in the GLOBALS parameter file (Doc ID 2514856.1)
 
   

Tuesday, June 13, 2023

Refresh 19c physical standby database using service

 
Primary DB (19c) : TESTDB_PRIM (standby)
Standby DB (19c) : TESTDB_STD (cascade standby)
 
Standby has a GAP and waiting on logs which are deleted from primary
 
NOTE : here my primary is also a STANDBY from where I setup another standby (cascade)
 
SQL> select inst_id,process, thread#, sequence#, status from gv$managed_standby where process='MRP0';
 
   INST_ID PROCESS      THREAD#  SEQUENCE# STATUS
---------- --------- ---------- ---------- ------------
         1 MRP0               1     973244 WAIT_FOR_GAP
 
 
 
SQL> SELECT name "Database name", status, ARCH.THREAD# "Thread", ARCH.SEQUENCE# "Last Sequence Received", APPL.SEQUENCE# "Last Sequence Applied", (ARCH.SEQUENCE# - APPL.SEQUENCE#) "Difference" FROM (select name from v$database) name, (select status from v$instance) status,(SELECT THREAD# ,SEQUENCE# FROM V$ARCHIVED_LOG WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$ARCHIVED_LOG GROUP BY THREAD#)) ARCH, (SELECT THREAD# ,SEQUENCE# FROM V$LOG_HISTORY WHERE (THREAD#,FIRST_TIME ) IN (SELECT THREAD#,MAX(FIRST_TIME) FROM V$LOG_HISTORY GROUP BY THREAD#)) APPL WHERE ARCH.THREAD# = APPL.THREAD# ORDER BY 1;
 
Database  STATUS           Thread Last Sequence Received Last Sequence Applied Difference
--------- ------------ ---------- ---------------------- --------------------- ----------
TESTDB    MOUNTED               1                 975607                973243       2364
TESTDB    MOUNTED               3                 945126                942906       2220
TESTDB    MOUNTED               2                 995543                993282       2261
 
 ***** Switch some logfile on PRIMARY *****
 
SQL>   /
 
Database  STATUS           Thread Last Sequence Received Last Sequence Applied Difference
--------- ------------ ---------- ---------------------- --------------------- ----------
TESTDB    MOUNTED               1                 975607                973243       2364
TESTDB    MOUNTED               3                 945126                942906       2220
TESTDB    MOUNTED               2                 995545                993282       2263
 
 
I see logfiles are flowing to standby but MRP is waiting on GAP
 
Standby Alert log gives the GAP Message
 
2023-06-13T11:48:36.422200-05:00
PR00 (PID:397954): FAL: Failed to request gap sequence
PR00 (PID:397954):  GAP - thread 1 sequence 973244-973343
PR00 (PID:397954):  DBID 3401882687 branch 943793862
PR00 (PID:397954): FAL: All defined FAL servers have been attempted
PR00 (PID:397954): -------------------------------------------------------------------------
PR00 (PID:397954): Check that the CONTROL_FILE_RECORD_KEEP_TIME initialization
PR00 (PID:397954): parameter is defined to a value that's sufficiently large
PR00 (PID:397954): enough to maintain adequate log switch information to resolve
PR00 (PID:397954): archived redo log gaps.
PR00 (PID:397954): -------------------------------------------------------------
 
 
Solution :
 
OCI - Roll Forward A Standby Database Using Recover Database From Service (Doc ID 2931070.1)
How to Roll Forward a Standby Database Using Recover Database From Service (Doc ID 2850185.1)
 
From Oracle 18c and higher. Single command (RED) will do all the work
 
1.      STOP MRP
                     SQL>  recover managed standby database cancel;   ( or )
 
                     DGMGRL>  EDIT DATABASE '<standby>' SET STATE='APPLY-OFF';
 
 
2.      Stop all standby instances (if RAC) and mount only 1 node
3.       Run the shell script
HOSTNAME:NOTSET:/export/scripts $ cat TESTDB_INC_RECOVER_CB.sh
 
export NLS_DATE_FORMAT='DD-MON-YY HH24:MI:SS'
rman target / trace /var/logs/RMAN_INC_recover_01jun2023.log << EOF
alter session set NLS_DATE_FORMAT='DD-MON-YY HH24:MI:SS';
select sysdate from dual;
 
run
{
allocate channel c1 type disk;
allocate channel c2 type disk;
allocate channel c3 type disk;
allocate channel c4 type disk;
allocate channel c5 type disk;
allocate channel c6 type disk;
allocate channel c7 type disk;
allocate channel c8 type disk;
allocate channel c9 type disk;
allocate channel c10 type disk;
recover database from service TESTDB_PRIM section size 5g;
}
 
alter session set NLS_DATE_FORMAT='DD-MON-YY HH24:MI:SS';
select sysdate from dual;
 
EOF
 
 
4.      Once recovery is done then start MRP
 
 
 
 
Then logs will start apply normally.
 
 
 
 
 
 
 
 
 
 
 

Friday, June 9, 2023

ORA-38777: database must not be started in any other instance

 

I have 19c DB converted to SNAPSHOT STANDBY and now i wanted to revert back to Physical standby

SQL> select NAME, OPEN_MODE, GUARD_STATUS, DATABASE_ROLE from v$database;

NAME      OPEN_MODE            GUARD_S DATABASE_ROLE
--------- -------------------- ------- ----------------
MYPROD    READ WRITE           NONE    SNAPSHOT STANDBY

While Converting Snapshot standby back to Physical standby while both RAC Nodes are up and running we get this error


SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;

ALTER DATABASE CONVERT TO PHYSICAL STANDBY

*

ERROR at line 1:

ORA-38777: database must not be started in any other instance


srvctl status database -d myprod_std -v
Instance myprod_std1 is running on node hostname1. Instance status: Mounted (Closed).
Instance myprod_std2 is running on node hostname2. Instance status: Mounted (Closed).


NOTE : 

   Instance should open in only 1 node to Convert back

 $ srvctl stop instance -i myprod2 -d myprod_std
 
hostname:myprod1:/export/scripts $ sqlplus / as sysdba
 
SQL*Plus: Release 19.0.0.0.0 - Production on Fri Jun 9 13:11:19 2023
Version 19.17.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
 
SQL> ALTER DATABASE CONVERT TO PHYSICAL STANDBY;
 
Database altered.
 
SQL> shut immediate
ORA-01109: database not open
 
 
Database dismounted.
ORACLE instance shut down.
SQL> exit
 
$ srvctl start database -d myprod_std -v
 
$ srvctl status database -d myprod_std -v

Instance myprod_std1 is running on node hostname1. Instance status: Mounted (Closed).

Instance myprod_std2 is running on node hostname2. Instance status: Mounted (Closed).


 
hostname:myprod_std:/export/scripts $ sqlplus / as sysdba
 
SQL*Plus: Release 19.0.0.0.0 - Production on Fri Jun 9 13:13:41 2023
Version 19.17.0.0.0
 
Copyright (c) 1982, 2022, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 19c EE Extreme Perf Release 19.0.0.0.0 - Production
Version 19.17.0.0.0
 
SQL> alter database open;
 
Database altered.
 
SQL> select NAME, OPEN_MODE, GUARD_STATUS, DATABASE_ROLE from v$database;
 
NAME      OPEN_MODE            GUARD_S  DATABASE_ROLE
--------- -------------------- ------- ----------------
MYPROD    READ ONLY            NONE     PHYSICAL STANDBY
 
 Start MRP .......

SQL> alter database recover managed standby database disconnect from session;
 
Database altered.





Friday, May 12, 2023

Oracle goldengate Encrypt TRAIL File using ENCRYPTTRAIL parameter

 

https://docs.oracle.com/en/middleware/goldengate/core/21.3/reference/encrypttrail-noencrypttrail.html#GUID-F9C77C5E-500A-4B1D-9326-8385EECE531D

 ENCRYPTTRAIL and NOENCRYPTTRAIL parameters are used to enable/disable encryption on OGG TRAIL files

      1.      Wallet (or)  master key Method

      2.     ENCKEY Method

Extract param: (wallet method)

For wallet method you need to create wallets before using the ENCRYPTTRAIL in extract


 GGSCI (oracledev) 5> view params EXT

EXTRACT ext

INCLUDE /tmp/GG_ENV.mac

#E_GG_env()

GETUPDATEBEFORES

NOCOMPRESSDELETES

NOCOMPRESSUPDATES

REPORTCOUNT EVERY 1 HOUR, RATE

WARNLONGTRANS 1D, CHECKINTERVAL 10M, SKIPEMPTYTRANS

EXTTRAIL <path>/aa

TABLE SCOTT.EMPL GETBEFORECOLS(ON UPDATE ALL, ON DELETE ALL) ;

 

$ cat /goldengate/home/GG_ENV.mac

MACRO #E_GG_env

BEGIN
SETENV (ORACLE_HOME='/u01/app/oracle/product/19.0.0.0/dbhome_1')
SETENV (ORACLE_SID='oradb1')
USERIDALIAS ggadmin_dev
ENCRYPTTRAIL
END;

Reference:

How to Encrypt/Decrypt Oracle GoldenGate Trail File (Doc ID 1287578.1)
How to Use ENCRYPTTRAIL and DECRYPTTRAIL Parameters in Extract, Pump and Replicat (Doc ID 1397104.1)

 

Wednesday, December 28, 2022

Goldengate 19c Switchover shell script

 #######################################################
#!/bin/ksh
#  Purpose          : This Script will do Actual GG Failover to another DC
#  NOTE            : Always SHOULD run from OLD Primary
#  Hard Coded   : Target server, Target DB
#  ASSUMING   : SSH works from Source to Target servers
#                          :  DB Switchover already completed
#  AUTHOR       : CB
########################################################
# Changes :
# CB - Aug 2018   - version 1 - Initial Draft
 
echo -e "\n*************************************"
echo  "DATE = `date`"
echo  "HOSTNAME = " `hostname`
echo -e "*************************************\n"

if [ "$2" = "" ]; then
  echo " "
  echo -e "Syntax: $0 <DB name>  <New Prod Server (node 1 ONLY)> \n"
  exit 1
fi

DBTGT=`echo $1 | tr "[:upper:]" "[:lower:]"`
export GG_HOME=/goldengate/home
export GG_DBA_DIR=/goldengate/SCRITPS/
TRG_DB=<TARGET_DB_NAME>
TARGET_SERVER=`$GG_DBA_DIR/GET_HOST.sh $TRG_DB`
SWITCH_TO=`echo $2 | tr "[:upper:]" "[:lower:]"`
#DBTAIL=`echo $DBTGT |awk '{print substr($0,3)}'`
#echo "DBTAIL :" $DBTAIL
#echo "DBTGT : " $DBTGT
SOURCE_SERVER=`hostname`
TODAY=`date '+%b%d%y'`
#END_MOD="\e[0m"
# Checking logic for NODE1 input because my mgr always runs on node1
if [[ "${SWITCH_TO:0:6}" == "server_name" || "${SWITCH_TO:0:6}" == "server_name" ]]; then
echo -e "\n Starting Switchover of  ${DBTGT} to ${SWITCH_TO}  ..........  \n"
else
echo -e "\n Entered Server name is NOT a PROD NODE 1 RAC server \n"
exit 1
fi

ORAENV_ASK=NO ; export ORAENV_ASK; export ORACLE_SID=$DBTGT
. oraenv $ORACLE_SID
 
check_prompt_3()
{
while true
do
  # (1) prompt and read command line argument
  read -p "Do you want to proceed ...?? " answer3
  # (2) handle the input we were given
  case $answer3 in
   [yY]* )
           echo -e "\nEntered YES, Continuing script........\n"
           break;;
   [nN]* ) echo -e "\nEntered NO ........"
           echo -e "\nexiting script ........\n"
           exit;;
   * )     echo "Dude, just enter Y or N, please.";;
  esac
done
}
GG_INFO () {
$GG_HOME/ggsci -s <<EOF
info all
exit
EOF
}
GG_PROCESS () {
$GG_HOME/ggsci -s <<EOF
info ${PROCESS_NAME}
exit
EOF
}
STOP_EXTRACT () {
$GG_HOME/ggsci -s <<EOF > ${GG_DBA_DIR}/spool_EXT.log
info all
exit
EOF
cat ${GG_DBA_DIR}/spool_EXT.log | egrep 'EXTRACT' | grep -i E$DBTGT| tr ":" " " | while read LINE
do
  case $LINE in
    *)
    PROCESS_TYPE=`echo $LINE | awk -F" " '{print $1}'`
    PROCESS_STATUS=`echo $LINE | awk -F" " '{print $2}'`
    PROCESS_NAME=`echo $LINE | awk -F" " '{print $3}'`
if [ "$PROCESS_TYPE" != "MANAGER" ]
  then
    if [ "$PROCESS_STATUS" == "RUNNING" ]
       then
#Making sure Only Extracts should be STOPPED NOT PUMP
PUMP_CHAR=`echo $PROCESS_NAME |awk '{print substr($0,1,1)}'`
#echo -e "\nPUMP_CHAR = " $PUMP_CHAR
      if [[ "$PUMP_CHAR" != "P" ]]; then
        echo -e "\n  *******************************************"
        echo -e "  Executing for $PROCESS_TYPE :  $PROCESS_NAME "
        echo -e "  ********************************************\n"
        echo -e "STOPPING EXTRACT $PROCESS_NAME ........\n"
$GG_HOME/ggsci <<EOF
STOP $PROCESS_NAME
sh sleep 15
info $PROCESS_NAME
sh echo -e "\ninfo ${PROCESS_NAME} SHOWCH"
info ${PROCESS_NAME} SHOWCH
EOF
       fi
  else
     echo -e "\n   *******************************************"
     echo  "    EXTRACT ${PROCESS_NAME} NOT RUNNING......."
     echo -e "   *******************************************\n"
     GG_PROCESS;
    fi
if [[ -f ${GG_DBA_DIR}/EXT_SHOWCH.txt ]]; then
rm ${GG_DBA_DIR}/EXT_SHOWCH.txt
fi
touch ${GG_DBA_DIR}/EXT_SHOWCH.txt
$GG_HOME/ggsci -s <<EOF > ${GG_DBA_DIR}/EXT_SHOWCH.txt
info ${PROCESS_NAME} SHOWCH
exit
EOF
EXT_SEQ_NO=`grep -i "Sequence #" ${GG_DBA_DIR}/EXT_SHOWCH.txt|awk -F" " '{print $3}'`
EXT_RBA_NO=`grep -i "RBA" ${GG_DBA_DIR}/EXT_SHOWCH.txt|awk -F" " '{print $2}'`
   echo -e "\n\n          *************************************************"
   echo -e "           Extract STOPPED at SEQ No. =" ${EXT_SEQ_NO} ", RBA No. ="  ${EXT_RBA_NO}
   echo -e "          **************************************************\n"
if [[ -f ${GG_DBA_DIR}/EXT_SHOWCH.txt ]]; then
rm ${GG_DBA_DIR}/EXT_SHOWCH.txt
fi
 fi
esac
done
if [[ -f ${GG_DBA_DIR}/spool_EXT.log ]]; then
rm ${GG_DBA_DIR}/spool_EXT.log
fi
}


STOP_PUMP () {
$GG_HOME/ggsci -s <<EOF > ${GG_DBA_DIR}/spool_PUMP.log
info all
exit
EOF
#DB_TYPE=$DBTGT
DB_TYPE=`echo ${DBTGT} |awk '{print substr($0,1,2)}'`
echo -e "\nDB_TYPE = " $DB_TYPE
cat ${GG_DBA_DIR}/spool_PUMP.log | egrep 'EXTRACT' | grep -i P${DBTGT}| tr ":" " " | while read LINE2
do
  case $LINE2 in
    *)
    PROCESS_TYPE=`echo $LINE2 | awk -F" " '{print $1}'`
    PROCESS_STATUS=`echo $LINE2 | awk -F" " '{print $2}'`
    PROCESS_NAME=`echo $LINE2 | awk -F" " '{print $3}'`
if [ "$PROCESS_TYPE" != "MANAGER" ]
  then
    if [ "$PROCESS_STATUS" == "RUNNING" ]
       then
#Making sure Only Extracts should be STOPPED NOT PUMP
PUMP_CHAR=`echo $PROCESS_NAME |awk '{print substr($0,1,1)}'`
#echo -e "\nPUMP_CHAR = " $PUMP_CHAR
      if [[ "$PUMP_CHAR" == "P" ]]; then
echo -e "\nChecking .......SEND ${PROCESS_NAME} LOGEND"
send_pump_fn ()
{
$GG_HOME/ggsci <<EOF
SEND P${DB_TYPE}* LOGEND
EXIT
EOF
}
if [[ -f ${GG_DBA_DIR}/READ_SEND_PUMP ]]; then
rm ${GG_DBA_DIR}/READ_SEND_PUMP
fi
if [[ -f ${GG_DBA_DIR}/send_pump.tmp ]]; then
rm ${GG_DBA_DIR}/send_pump.tmp
fi
touch ${GG_DBA_DIR}/READ_SEND_PUMP
touch ${GG_DBA_DIR}/send_pump.tmp
until [ -s ${GG_DBA_DIR}/READ_SEND_PUMP ]
  do
      send_pump_fn > ${GG_DBA_DIR}/send_pump.tmp
      grep -i YES ${GG_DBA_DIR}/send_pump.tmp > ${GG_DBA_DIR}/READ_SEND_PUMP
         sleep 5
      echo -e "\nWaiting for Pump to get .......YES"
  done
echo -e "\n ******    SEND Pump STATUS ==>  YES and it processed all trail records  ******"
echo -e "\n  ---------- [ SEND PUMP Output  ] ------------"
echo -e "  *********************************************\n"
cat ${GG_DBA_DIR}/send_pump.tmp
echo -e "\n  ********************************************\n"
echo -e "\nSTOPPING PUMP $PROCESS_NAME Process ............\n"
$GG_HOME/ggsci <<EOF
SEND $PROCESS_NAME STATUS
STOP $PROCESS_NAME
sh sleep 5
info $PROCESS_NAME
EOF
fi
   else
   echo -e "\n  ****************************"
   echo -e "  PUMP $PROCESS_NAME NOT running"
   echo -e "  *****************************\n"
    fi
fi
esac
done
if [[ -f ${GG_DBA_DIR}/spool_PUMP.log ]]; then
rm ${GG_DBA_DIR}/spool_PUMP.log
fi
}
###   MAIN    ####
GG_INFO;
STOP_EXTRACT;
GG_INFO;
#check_prompt_3
STOP_PUMP;
GG_INFO;
#check_prompt_3
$GG_HOME/ggsci <<EOF > EXT_TRAIL
info EXTTRAIL *${DBTGT}*
EXIT
EOF

EXT_TRAIL_PATH=`grep -i "SRC" EXT_TRAIL |awk -F" " '{print $3}' `
echo "   EXT_TRAIL_PATH=" $EXT_TRAIL_PATH
PUMP_TRAIL_PATH=`grep -i TRG EXT_TRAIL |awk -F" " '{print $3}'`
echo "   PUMP_TRAIL_PATH =" $PUMP_TRAIL_PATH
OLD_REP_SEQNO=`grep  "Seqno:" EXT_TRAIL |awk -F" " '{print $2}' |tail -1`
if [[ -f ${GG_DBA_DIR}/READ_SEND_PUMP ]]; then
rm ${GG_DBA_DIR}/READ_SEND_PUMP
fi
if [[ -f ${GG_DBA_DIR}/send_pump.tmp ]]; then
rm ${GG_DBA_DIR}/send_pump.tmp
fi
if [[ -f ${GG_DBA_DIR}/EXT_TRAIL ]]; then
rm ${GG_DBA_DIR}/EXT_TRAIL
fi
echo -e "\nChecking manager........\n"
$GG_HOME/ggsci <<EOF
sh echo "info mgr"
info Mgr
EXIT
EOF
echo -e "\n\n    ================[   Current SOURCE `hostname -s` Trail files  ]================== "
ls -lrth ${EXT_TRAIL_PATH}* | tail
## TARGET REPLICAT Activities
#echo -e "\n##### ---------------------------[ START NEW SSH ]------------------------########"
echo -e "###############################################################################################"
#echo -e "\n First :  ssh oracle@${TARGET_SERVER} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} TRG_DB=${TRG_DB} PUMP_TRAIL_PATH=${PUMP_TRAIL_PATH}  'bash -ls' <<'ENDSSH' \n"
#ssh oracle@${TARGET_SERVER} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} TRG_DB=${TRG_DB} PUMP_TRAIL_PATH=${PUMP_TRAIL_PATH}  'bash -ls' <<'ENDSSH'
echo -e "\n  ************************************"
echo -e "  Logged into Target server : `hostname -s`"
echo -e "  ************************************\n"
$GG_HOME/ggsci -s <<EOF
info all
exit
EOF
$GG_HOME/ggsci -s <<EOF > ${GG_DBA_DIR}/spool_targetDB.log
info all
exit
EOF
cat ${GG_DBA_DIR}/spool_targetDB.log | egrep 'REPLICAT' | egrep -i $DB_TYPE| egrep -i ${DBTGT} | tr ":" " " | while read LINE
do
  case $LINE in
    *)
    PROCESS_TYPE=`echo $LINE | awk -F" " '{print $1}'`
    PROCESS_STATUS=`echo $LINE | awk -F" " '{print $2}'`
    PROCESS_NAME=`echo $LINE | awk -F" " '{print $3}'`
if [ "$PROCESS_TYPE" != "MANAGER" ]
  then
    if [ "$PROCESS_TYPE" == "REPLICAT" ]
      then
echo -e "\n  *******************************************"
echo -e "   Executing for $PROCESS_TYPE :  $PROCESS_NAME"
echo -e "  ********************************************\n"
        if [ "$PROCESS_STATUS" == "RUNNING" ]
          then
echo -e "DB_TYPE =" $DB_TYPE
echo  "DBTGT = " $DBTGT
#REP_PROCESS_NAME=R${DB_TYPE}*D2
echo "REP_PROCESS_NAME =" $PROCESS_NAME
#echo -e "\ninfo ${PROCESS_NAME}"
$GG_HOME/ggsci -s <<EOF
sh echo "info ${PROCESS_NAME}"
info ${PROCESS_NAME}
exit
EOF

send_replicat_fn ()
{
$GG_HOME/ggsci <<EOF
SEND ${PROCESS_NAME} LOGEND
EXIT
EOF
}
if [[ -f ${GG_DBA_DIR}/READ_SEND_REPLICAT ]]; then
rm ${GG_DBA_DIR}/READ_SEND_REPLICAT
fi
if [[ -f ${GG_DBA_DIR}/send_replicat.tmp ]]; then
rm ${GG_DBA_DIR}/send_replicat.tmp
fi
touch ${GG_DBA_DIR}/READ_SEND_REPLICAT
touch ${GG_DBA_DIR}/send_replicat.tmp
until [ -s ${GG_DBA_DIR}/READ_SEND_REPLICAT ]
  do
     send_replicat_fn > ${GG_DBA_DIR}/send_replicat.tmp
     grep -i YES ${GG_DBA_DIR}/send_replicat.tmp > ${GG_DBA_DIR}/READ_SEND_REPLICAT
     echo -e "\nWaiting for Replicat to get .......YES"
      sleep 5
  done
echo -e "\nSEND REPLICAT STATUS ==>  is YES and it processed all trail records"
echo -e "\n*********** [ SEND REPLICAT status OUTPUT ] *********"
  cat ${GG_DBA_DIR}/send_replicat.tmp
echo -e "\n********************************************\n"
echo -e "\n  STOPPING $PROCESS_NAME Process ............\n"
$GG_HOME/ggsci -s <<EOF
sh echo "SEND ${PROCESS_NAME} STATUS"
SEND ${PROCESS_NAME} STATUS
STOP ${PROCESS_NAME}
sh sleep 5
info ${PROCESS_NAME}
exit
EOF
     else
     echo -e "\n   ********************************"
     echo -e "    Replicat ${PROCESS_NAME} NOT RUNNING........"
     echo -e "   *********************************\n"
$GG_HOME/ggsci -s <<EOF
info ${PROCESS_NAME}
exit
EOF
        fi
fi
fi
ORAENV_ASK=NO ; export ORAENV_ASK; export ORACLE_SID=${TRG_DB}
. oraenv
GETPW=<password>
#echo "Password -> "${GETPW}
echo " ORACLE_SID = " $ORACLE_SID
echo " ORACLE_HOME = " $ORACLE_HOME
DB_SEQNO=`${ORACLE_HOME}/bin/sqlplus -s /nolog << EOF
connect system/$GETPW@${TRG_DB}
set verify off trimspool on feed off echo off termout off head off
select seqno from ggadmin.checkpoint_target where group_name='${PROCESS_NAME}';
exit
EOF
`
DB_RBANO=`${ORACLE_HOME}/bin/sqlplus -s /nolog << EOF
connect system/$GETPW@${TRG_DB}
set verify off trimspool on feed off echo off termout off head off
select RBA from ggadmin.checkpoint_target where group_name='${PROCESS_NAME}';
exit
EOF
`
     echo -e "\n      *************************************************************"
     echo -e "      Replicat ${PROCESS_NAME} Last Applied SEQ No. = " ${DB_SEQNO} ", RBA No. =" ${DB_RBANO}
     echo -e "      **************************************************************\n"
esac
done
echo -e "\n  -------------[  List of Trail Files on TARGET `hostname -s` Server ]-------------"
echo -e "ls -lrth ${PUMP_TRAIL_PATH}*| tail"
ls -lrth ${PUMP_TRAIL_PATH}*| tail
if [[ -f ${GG_DBA_DIR}/READ_SEND_REPLICAT ]]; then
rm ${GG_DBA_DIR}/READ_SEND_REPLICAT
fi
if [[ -f ${GG_DBA_DIR}/send_replicat.tmp ]]; then
rm ${GG_DBA_DIR}/send_replicat.tmp
fi
#ENDSSH
#if [ $? != 0 ]
#then
#echo -e "\n  *********************************************"
#    echo -e "\n  Found errors on Target SSH script .......Check the Error\n"
#    echo -e "  *******************************************\n"
#exit 1
#fi
echo -e "\n##### ---------------------------[ END SSH ]------------------------########"
###check_prompt_3;
#####################  NEW PRIMARY  (Was Standby)  #############
##### ---------------------------[ START NEW SSH ]------------------------########
## Checking just EXTRACT info and Cleaning OLD files
echo -e "\nSecond:  ssh ${SWITCH_TO} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} EXT_TRAIL_PATH=${EXT_TRAIL_PATH} TRG_DB=${TRG_DB} 'bash -ls' <<'ENDSSH' \n"
ssh ${SWITCH_TO} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} EXT_TRAIL_PATH=${EXT_TRAIL_PATH} TRG_DB=${TRG_DB}  'bash -ls' <<'ENDSSH'
sleep 5
echo -e "\n\n  ---------[ List of checkpoint files... ]--------"
ls -lcrth $GG_HOME/dirchk | tail
echo -e "\n  ---------[ List of param files... ]--------"
ls -lcrth $GG_HOME/dirprm | tail -5
echo -e "\n        ************************************"
echo -e "\n           Logged into `hostname -s` Server\n"
echo -e "         ************************************\n"
$GG_HOME/ggsci -s <<EOF
sh echo "Starting Manager ......."
info mgr
start mgr
sh sleep 5
info mgr
exit
EOF


SOURCE_SERVER=`hostname -s`
TODAY=`date '+%b%d%y'`
ORAENV_ASK=NO ; export ORAENV_ASK; export ORACLE_SID=$DBTGT
. dbaenv $ORACLE_SID
 
GG_INFO_DETAIL () {
$GG_HOME/ggsci -s <<EOF
sh echo "dblogin useridalias ggadmin_prod"
dblogin useridalias ggadmin_prod
info E${DBTGT^^}* DETAIL
exit
EOF
}
GG_INFO_DETAIL;
if [[ -f ${GG_DBA_DIR}/EXT_SHOWCH.txt ]]; then
rm ${GG_DBA_DIR}/EXT_SHOWCH.txt
fi
touch ${GG_DBA_DIR}/EXT_SHOWCH.txt
$GG_HOME/ggsci -s <<EOF > ${GG_DBA_DIR}/EXT_SHOWCH.txt
info E${DBTGT^^}* SHOWCH
exit
EOF
EXT_SEQ_NO=`grep -i "Sequence #" ${GG_DBA_DIR}/EXT_SHOWCH.txt|awk -F" " '{print $3}'`
EXT_RBA_NO=`grep -i "RBA" ${GG_DBA_DIR}/EXT_SHOWCH.txt|awk -F" " '{print $2}'`
   echo -e "\n\n          *************************************************"
   echo -e "         OLD Extract CURRENT SEQ No. =" ${EXT_SEQ_NO} ", RBA No. ="  ${EXT_RBA_NO}
   echo -e "          **************************************************\n"
if [[ -f ${GG_DBA_DIR}/EXT_SHOWCH.txt ]]; then
rm ${GG_DBA_DIR}/EXT_SHOWCH.txt
fi
echo -e "\n ##################################"
echo -e " Cleaning OLD Files ............."
echo -e " ##################################\n"
echo -e " EXTRACT Cleanup ...........\n"
mkdir -p ${GG_HOME}/dirprm/Switchover_$TODAY
mkdir -p ${GG_HOME}/dirchk/Switchover_$TODAY
mkdir -p ${GG_HOME}/dirrpt/Switchover_$TODAY
echo -e "\nBacking up Existing param files "
echo "mv ${GG_HOME}/dirprm/e$DBTGT*.prm ${GG_HOME}/dirprm/p$DBTGT*.prm ${GG_HOME}/dirprm/Switchover_$TODAY"
mv ${GG_HOME}/dirprm/e$DBTGT*.prm ${GG_HOME}/dirprm/p$DBTGT*.prm ${GG_HOME}/dirprm/Switchover_$TODAY
ls -lrth ${GG_HOME}/dirprm | tail -5
echo -e "\nBacking up Existing DIRCHK directory "
echo "mv ${GG_HOME}/dirchk/*${DBTGT^^}*.cpb ${GG_HOME}/dirchk/*${DBTGT^^}*.cpe  ${GG_HOME}/dirchk/Switchover_$TODAY"
mv ${GG_HOME}/dirchk/*${DBTGT^^}*.cpb ${GG_HOME}/dirchk/*${DBTGT^^}*.cpe  ${GG_HOME}/dirchk/Switchover_$TODAY
ls -lrth ${GG_HOME}/dirchk
mv ${GG_HOME}/dirrpt/E*${DBTGT^^}* ${GG_HOME}/dirrpt/e*${DBTGT}* ${GG_HOME}/dirrpt/Switchover_$TODAY
mv ${GG_HOME}/dirrpt/P*${DBTGT^^}* ${GG_HOME}/dirrpt/p*${DBTGT}* ${GG_HOME}/dirrpt/Switchover_$TODAY
echo -e "\nBacking up Existing Trail Files...."
mkdir -p ${GG_HOME}/dirdat/SRC/${DBTGT}/${TRG_DB}/Switchover_$TODAY
echo "mv ${GG_HOME}/dirdat/SRC/${DBTGT}/${TRG_DB}/${DB_TYPE}* ${GG_HOME}/dirdat/SRC/${DBTGT}/${TRG_DB}/Switchover_$TODAY"
mv ${GG_HOME}/dirdat/SRC/${DBTGT}/${TRG_DB}/${DB_TYPE}* ${GG_HOME}/dirdat/SRC/${DBTGT}/${TRG_DB}/Switchover_$TODAY
#if [[ "$DO_EXT" == "Y" ]]; then
echo -e "\n REPLICAT Cleanup ...........\n"
#### Below are for Replicat
echo -e "\nBacking up Existing param files "
echo "mv ${GG_HOME}/dirprm/r$DBTGT*.prm ${GG_HOME}/dirprm/Switchover_$TODAY"
mv ${GG_HOME}/dirprm/r$DBTGT*.prm  ${GG_HOME}/dirprm/Switchover_$TODAY
ls -lrth ${GG_HOME}/dirprm | tail -5
echo -e "\nBacking up Existing DIRCHK directory "
echo "mv ${GG_HOME}/dirchk/*${DBTGT^^}*.cpr ${GG_HOME}/dirchk/Switchover_$TODAY"
mv ${GG_HOME}/dirchk/*${DBTGT^^}*.cpr ${GG_HOME}/dirchk/Switchover_$TODAY
ls -lrth ${GG_HOME}/dirchk
mv ${GG_HOME}/dirrpt/R*${DBTGT^^}* ${GG_HOME}/dirrpt/r*${DBTGT}* ${GG_HOME}/dirrpt/Switchover_$TODAY
echo -e "\nBacking up Existing Trail Files...."
mkdir -p ${GG_HOME}/dirdat/TRG/${DBTGT}/${TRG_DB}/Switchover_$TODAY
echo "mv ${GG_HOME}/dirdat/TRG/${DBTGT}/${TRG_DB}/${DB_TYPE^^}* ${GG_HOME}/dirdat/TRG/${DBTGT}/${TRG_DB}/Switchover_$TODAY"
mv ${GG_HOME}/dirdat/TRG/${DBTGT}/${TRG_DB}/${DB_TYPE^^}* ${GG_HOME}/dirdat/TRG/${DBTGT}/${TRG_DB}/Switchover_$TODAY
#fi

echo "ORACLE_SID=" $ORACLE_SID
$GG_HOME/ggsci -s <<EOF
info all
exit
EOF
echo -e "\n\n ---------[ List of Latest trail files on NEW PRIMARY ]--------- "
#echo -e "\nls -lrth ${EXT_TRAIL_PATH}* | tail"
ls -lrth ${EXT_TRAIL_PATH}* | tail
ENDSSH
##### ---------------------------[ END SSH ]------------------------########
###check_prompt_3;
echo -e "  ########################## SCP files to NEW PRIMARY : ${SWITCH_TO} ##############"
sleep 5
echo "scp -r ${GG_HOME}/dirchk/E${DBTGT^^}*  $GG_HOME/dirchk/P${DBTGT^^}* oracle@${SWITCH_TO}:$GG_HOME/dirchk "
scp -r ${GG_HOME}/dirchk/E${DBTGT^^}*.* $GG_HOME/dirchk/P${DBTGT^^}*.* oracle@${SWITCH_TO}:$GG_HOME/dirchk
scp -r ${GG_HOME}/dirprm/e${DBTGT}*.prm $GG_HOME/dirprm/p${DBTGT}*.prm oracle@${SWITCH_TO}:$GG_HOME/dirprm
echo "scp -r ${GG_HOME}/dirchk/R${DBTGT^^}*   oracle@${SWITCH_TO}:$GG_HOME/dirchk "
scp -r  ${GG_HOME}/dirchk/R${DBTGT^^}* oracle@${SWITCH_TO}:$GG_HOME/dirchk
scp -r ${GG_HOME}/dirprm/r${DBTGT}*.prm  oracle@${SWITCH_TO}:$GG_HOME/dirprm
if [ $? != 0 ]
then
    echo -e "\n    **********************************************"
    echo -e "\n     SCP to ${SWITCH_TO} Failed.......Check the Error\n"
    echo -e "     **********************************************\n"
  exit 1
else
echo -e "\n    *************************************"
    echo -e "    SCP to ${SWITCH_TO} ............SUCCESS"
    echo -e "   **************************************\n"
fi
###check_prompt_3;
##### ---------------------------[ START NEW SSH ]------------------------########
# EXTRACT Activities on NEW Primary
echo -e "\n3rd : ssh ${SWITCH_TO} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} TARGET_SERVER=${TARGET_SERVER} EXT_TRAIL_PATH=${EXT_TRAIL_PATH} EXT_SEQ_NO=${EXT_SEQ_NO} EXT_RBA_NO=${EXT_RBA_NO}  'bash -ls' <<'ENDSSH' \n"
ssh ${SWITCH_TO} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} TARGET_SERVER=${TARGET_SERVER} EXT_TRAIL_PATH=${EXT_TRAIL_PATH} EXT_SEQ_NO=${EXT_SEQ_NO} EXT_RBA_NO=${EXT_RBA_NO}  'bash -ls' <<'ENDSSH'
echo -e "\n  ************************************"
echo -e "\n  Logged into `hostname -s` Server\n"
echo -e "  ************************************\n"
ls -lcrth $GG_HOME/dirchk | tail
echo -e "\n  Updating RMTHOST in p${DBTGT}p paramfile ......."
RMT_HOST_NAME=`grep RMTHOST ${GG_HOME}/dirprm/p${DBTGT}*.prm  | awk '{print $2}'`
RMT_HOST_NAME=${RMT_HOST_NAME%%?}
echo -e "\nOLD RMTHOST = " $RMT_HOST_NAME
echo "s/${RMT_HOST_NAME}/${TARGET_SERVER}/g"
eval sed -i 's/${RMT_HOST_NAME}/${TARGET_SERVER}/g' ${GG_HOME}/dirprm/p${DBTGT}*.prm
cat ${GG_HOME}/dirprm/p${DBTGT}*.prm
RMT_HOST_NAME=`grep RMTHOST ${GG_HOME}/dirprm/p${DBTGT}*.prm  | awk '{print $2}'`
RMT_HOST_NAME=${RMT_HOST_NAME%%?}
#echo -e  "\nNEW RMTHOST = " $RMT_HOST_NAME
if [[  ${RMT_HOST_NAME} == ${TARGET_SERVER} ]]; then
 echo -e  "\nNEW RMTHOST = " $RMT_HOST_NAME
else
 echo -e  "\nNEW RMTHOST = " $RMT_HOST_NAME
 echo -e " WRONG HOSTNAME in pump param file ..... Modify to ${TARGET_SERVER} name and continue. "
 exit 1
fi
SOURCE_SERVER=`hostname -s`
$GG_HOME/ggsci -s <<EOF > ${GG_DBA_DIR}/INFO_EXT
info all
exit
EOF
#INFO_EXT_VAR= `cat ${GG_DBA_DIR}/INFO_EXT | egrep 'EXTRACT' | grep -i E${DB_TYPE}|grep -i ${DBTGT}| tr ":" " "`
EXTRACT_NAME=`cat ${GG_DBA_DIR}/INFO_EXT | egrep 'EXTRACT' | grep -i E${DB_TYPE}|grep -i ${DBTGT}| tr ":" " " | awk -F" " '{print $3}'`
echo "EXTRACT_NAME =" $EXTRACT_NAME
if [[ -f ${GG_DBA_DIR}/EXT_SHOWCH_BEFORE.txt ]]; then
rm ${GG_DBA_DIR}/EXT_SHOWCH_BEFORE.txt
fi
touch ${GG_DBA_DIR}/EXT_SHOWCH_BEFORE.txt
$GG_HOME/ggsci -s <<EOF > ${GG_DBA_DIR}/EXT_SHOWCH_BEFORE.txt
info ${EXTRACT_NAME} SHOWCH
exit
EOF
SEQ_NO_BEFORE=`grep -i "Sequence #" ${GG_DBA_DIR}/EXT_SHOWCH_BEFORE.txt|awk -F" " '{print $3}'`
RBA_NO_BEFORE=`grep -i "RBA" ${GG_DBA_DIR}/EXT_SHOWCH_BEFORE.txt|awk -F" " '{print $2}'`



ORAENV_ASK=NO ; export ORAENV_ASK; export ORACLE_SID=$DBTGT
. dbaenv $ORACLE_SID
 
echo "ORACLE_SID=" $ORACLE_SID
$GG_HOME/ggsci -s <<EOF
dblogin useridalias ggadmin_prod
info all
info ${EXTRACT_NAME} DETAIL
sh echo "ALTER EXTRACT ${EXTRACT_NAME} ETROLLOVER"
ALTER EXTRACT ${EXTRACT_NAME} ETROLLOVER
info ${EXTRACT_NAME} DETAIL
sh echo "start ${EXTRACT_NAME}"
start ${EXTRACT_NAME}
sh sleep 5
info ${EXTRACT_NAME} showch
sh sleep 5
send ${EXTRACT_NAME} status
info all
exit
EOF
echo -e "\n\n  ------------[ Extract Trail file after ALTER  ]----------"
echo "ls -lrth ${EXT_TRAIL_PATH}* | tail"
ls -lrth ${EXT_TRAIL_PATH}* | tail
if [[ -f ${GG_DBA_DIR}/EXT_SHOWCH.txt ]]; then
rm ${GG_DBA_DIR}/EXT_SHOWCH.txt
fi
touch ${GG_DBA_DIR}/EXT_SHOWCH.txt
$GG_HOME/ggsci -s <<EOF > ${GG_DBA_DIR}/EXT_SHOWCH.txt
info ${EXTRACT_NAME} SHOWCH
exit
EOF
SEQ_NO=`grep -i "Sequence #" ${GG_DBA_DIR}/EXT_SHOWCH.txt|awk -F" " '{print $3}'`
echo -e "\n        ***************************************************"
echo -e "        EXTRACT ${EXTRACT_NAME} Latest SEQ no. After ETROLLOVER : " $SEQ_NO
echo -e "        ***************************************************\n"
ENDSSH
##### ---------------------------[ END 3rd SSH ]------------------------########
check_prompt_3;
##### ---------------------------[ START NEW SSH for PUMP ]------------------------########
#PUMP Activities
echo -e "\n -------[ Starting PUMP Activities ]-------- \n"
sleep 5
echo -e "\n4th : ssh ${SWITCH_TO} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} TARGET_SERVER=${TARGET_SERVER} EXT_TRAIL_PATH=${EXT_TRAIL_PATH} PUMP_TRAIL_PATH=${PUMP_TRAIL_PATH} 'bash -ls' <<'ENDSSH' \n"
ssh ${SWITCH_TO} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} TARGET_SERVER=${TARGET_SERVER} EXT_TRAIL_PATH=${EXT_TRAIL_PATH} PUMP_TRAIL_PATH=${PUMP_TRAIL_PATH} 'bash -ls' <<'ENDSSH'
echo -e "\n  ************************************"
echo -e "\n  Logged into `hostname -s` Server\n"
echo -e "  ************************************\n"

$GG_HOME/ggsci -s <<EOF > ${GG_DBA_DIR}/INFO_EXT_PUMP
info all
exit
EOF
# USE "P" in below Grep for PUMP
EXTRACT_NAME=`cat ${GG_DBA_DIR}/INFO_EXT_PUMP | egrep 'EXTRACT' | grep -i P${DB_TYPE}|grep -i ${DBTGT}| tr ":" " " | awk -F" " '{print $3}'`
echo -e "\nEXTRACT_NAME =" $EXTRACT_NAME

SEQ_NO=`grep -i "Sequence #" ${GG_DBA_DIR}/EXT_SHOWCH.txt|awk -F" " '{print $3}'`
echo "Extract SEQ_NO : " ${SEQ_NO}
#EXTRACT_NAME=P${DB_TYPE}*D2
#EXT_EXTRACT_NAME=E${DB_TYPE}*D2
$GG_HOME/ggsci -s <<EOF
info ${EXTRACT_NAME}
sh echo "ALTER EXTRACT ${EXTRACT_NAME} extseqno ${SEQ_NO} , EXTRBA 00"
ALTER EXTRACT ${EXTRACT_NAME} extseqno ${SEQ_NO} , EXTRBA 00
info ${EXTRACT_NAME}
sh echo "ALTER EXTRACT ${EXTRACT_NAME} ETROLLOVER"
ALTER EXTRACT ${EXTRACT_NAME} ETROLLOVER
sh echo "start ${EXTRACT_NAME}"
start ${EXTRACT_NAME}
EOF
echo -e "\n "
echo -e "\n -------------[ Pump SOURCE Latest Trail file after alter PUMP  ]------------"
echo -e "\nls -lrth ${EXT_TRAIL_PATH}* | tail"
ls -lrth ${EXT_TRAIL_PATH}* | tail
echo -e "\n -------------[  List of new REPLICAT Trail file on ${TARGET_SERVER} After PUMP ALTER SEQ  ]------------"
echo -e "\nssh oracle@${TARGET_SERVER} ls -lrth ${PUMP_TRAIL_PATH}* | tail"
ssh oracle@${TARGET_SERVER} ls -lrth ${PUMP_TRAIL_PATH}* | tail
if [[ -f ${GG_DBA_DIR}/EXT_SHOWCH.txt ]]; then
rm ${GG_DBA_DIR}/EXT_SHOWCH.txt
fi
if [[ -f ${GG_DBA_DIR}/INFO_EXT_PUMP ]]; then
rm ${GG_DBA_DIR}/INFO_EXT_PUMP
fi
ENDSSH
echo -e "\n ##### ---------------------------[ END SSH ]------------------------########\n"
###check_prompt_3;
echo -e "\n ##### ---------------------------[ START NEW SSH for REPLICAT ]------------------------########"
#REPLICAT Activities
sleep 5
echo -e "\n 5th :  ssh ${TARGET_SERVER} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} TRG_DB=${TRG_DB} PUMP_TRAIL_PATH=${PUMP_TRAIL_PATH} OLD_REP_SEQNO=${OLD_REP_SEQNO} 'bash -ls' <<'ENDSSH' \n"
ssh ${TARGET_SERVER} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} TRG_DB=${TRG_DB} PUMP_TRAIL_PATH=${PUMP_TRAIL_PATH} OLD_REP_SEQNO=${OLD_REP_SEQNO} 'bash -ls' <<'ENDSSH'
echo -e "\n  ************************************"
echo -e "\n  Logged into `hostname -s` Server\n"
echo -e "  ************************************\n"
echo -e "\n   -----------[ List of latest REPLICAT trail Files on TARGET `hostname -s`  server ]---------"
ls -lrth  ${PUMP_TRAIL_PATH}* | tail

GG_INFO (){
$GG_HOME/ggsci -s <<EOF
info all
exit
EOF
}
GG_INFO;
$GG_HOME/ggsci -s <<EOF > ${GG_DBA_DIR}/spool_targetDB.log
info all
exit
EOF
cat ${GG_DBA_DIR}/spool_targetDB.log | egrep 'REPLICAT' | egrep -i $DB_TYPE| egrep -i ${DBTGT}| tr ":" " " | while read LINE
do
  case $LINE in
    *)
    PROCESS_TYPE=`echo $LINE | awk -F" " '{print $1}'`
    PROCESS_STATUS=`echo $LINE | awk -F" " '{print $2}'`
    PROCESS_NAME=`echo $LINE | awk -F" " '{print $3}'`
if [ "$PROCESS_TYPE" != "MANAGER" ]
  then
    if [ "$PROCESS_TYPE" == "REPLICAT" ]
      then
#echo -e "DB_TYPE =" $DB_TYPE
#exit 1
echo -e "\n  *******************************************"
echo -e "   Executing for $PROCESS_TYPE :  $PROCESS_NAME"
echo -e "  ********************************************\n"

ORAENV_ASK=NO ; export ORAENV_ASK; export ORACLE_SID=${TRG_DB}
 
GETPW=<password>
#echo "Password -> "${GETPW}
echo " ORACLE_SID = " $ORACLE_SID
echo " ORACLE_HOME = " $ORACLE_HOME
DB_SEQNO=`$ORACLE_HOME/bin/sqlplus -s /nolog << EOF
connect system/$GETPW@${TRG_DB}
set verify off trimspool on feed off echo off termout off head off
select seqno from ggadmin.checkpoint_target where group_name='${PROCESS_NAME}';
exit
EOF
`
DB_RBA=`$ORACLE_HOME/bin/sqlplus -s /nolog << EOF
connect system/$GETPW@${TRG_DB}
set verify off trimspool on feed off echo off termout off head off
select rba from ggadmin.checkpoint_target where group_name='${PROCESS_NAME}';
exit
EOF
`
     echo -e "\n********************************************************"
     echo -e " Replicat ${PROCESS_NAME} Current OLD SEQ No. = " ${DB_SEQNO} , RBA No. = ${DB_RBA}
     echo -e "*********************************************************\n"
echo -e "OLD_REP_SEQNO =" ${OLD_REP_SEQNO}
echo -e "DB_SEQNO =" ${DB_SEQNO}
DB_SEQNO=`expr ${DB_SEQNO} + 1`
echo "        NEW REP DB_SEQNO need to set =  " ${DB_SEQNO}
if [[ "$DB_SEQNO" -eq 1 ]]; then
echo " **********************************************"
echo -e  "  ERROR ....... New REPLICAT Seq number is WRONG !!  \n"
echo -e " **********************************************\n"
exit 1
fi
$GG_HOME/ggsci -s <<EOF
sh echo "info ${PROCESS_NAME}"
info ${PROCESS_NAME}
sh echo "Alter REPLICAT ${PROCESS_NAME} extseqno $DB_SEQNO  extrba 00"
Alter REPLICAT ${PROCESS_NAME} extseqno $DB_SEQNO  extrba 00
info ${PROCESS_NAME}
exit
EOF
fi
fi
 
     echo -e "\n  **************************************************"
     echo -e "   Replicat ${PROCESS_NAME} SEQ No. After ALTER = " ${DB_SEQNO}
     echo -e "  ***************************************************\n"
esac
done
GG_INFO;
## ENDSSH
##### ---------------------------[ END SSH ]------------------------########
### check_prompt_3;
##### ---------------------------[ START NEW SSH for REPLICAT ]------------------------########
## REPLICAT Activities
##echo -e "\n 6th :  ssh ${TARGET_SERVER} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} TRG_DB=${TRG_DB} PUMP_TRAIL_PATH=${PUMP_TRAIL_PATH}  'bash -ls' <<'ENDSSH' \n"
### ssh ${TARGET_SERVER} GG_HOME=${GG_HOME} GG_DBA_DIR=${GG_DBA_DIR} DB_TYPE=${DB_TYPE} DBTGT=${DBTGT} TRG_DB=${TRG_DB} PUMP_TRAIL_PATH=${PUMP_TRAIL_PATH} 'bash -ls' <<'ENDSSH'
## echo -e "\n  ************************************"
## echo -e "\n   Logged into `hostname -s` Server"
## echo -e "  ************************************\n"
echo -e "\n List of Latest trail files for ${DBTGT}"
ls -lrth ${PUMP_TRAIL_PATH}* | tail -5
#GG_INFO (){
#$GG_HOME/ggsci -s <<EOF
#info all
#exit
#EOF
#}
GG_INFO;
$GG_HOME/ggsci -s <<EOF > ${GG_DBA_DIR}/spool_targetDB.log
info all
exit
EOF
cat ${GG_DBA_DIR}/spool_targetDB.log | egrep 'REPLICAT' | egrep -i $DB_TYPE| egrep -i ${DBTGT}| tr ":" " " | while read LINE
do
  case $LINE in
    *)
    PROCESS_TYPE=`echo $LINE | awk -F" " '{print $1}'`
    PROCESS_STATUS=`echo $LINE | awk -F" " '{print $2}'`
    PROCESS_NAME=`echo $LINE | awk -F" " '{print $3}'`
if [ "$PROCESS_TYPE" != "MANAGER" ]
  then
    if [ "$PROCESS_TYPE" == "REPLICAT" ]
      then
#echo -e "DB_TYPE =" $DB_TYPE
#exit 1
echo -e "\n  *******************************************"
echo -e "  Executing for $PROCESS_TYPE :  $PROCESS_NAME"
echo -e "  ********************************************\n"
$GG_HOME/ggsci -s <<EOF
sh echo "info ${PROCESS_NAME}"
info ${PROCESS_NAME}
sh echo " START REPLICAT ${PROCESS_NAME}"
START REPLICAT ${PROCESS_NAME}
sh sleep 5
exit
EOF
GG_INFO;
fi
fi

$GG_HOME/ggsci -s <<EOF > REP_STATUS
sh echo " SEND REPLICAT ${PROCESS_NAME} STATUS"
SEND REPLICAT ${PROCESS_NAME} status
exit
EOF
DB_SEQNO=`grep -i "Sequence #" REP_STATUS |awk -F" " '{print $3}'`
     echo -e "\n  **************************************************"
     echo -e "   Replicat ${PROCESS_NAME} New CURRENT SEQ No. = " ${DB_SEQNO}
     echo -e "  ***************************************************\n"
esac
done
ENDSSH
##### ---------------------------[ END 6th SSH ]------------------------########
################################################
echo -e "\n       Shell Script COMPLETED : `date`"
exit 0

-- 

Saturday, December 3, 2022

Set OEM page to NOT Timeout


OEM page login session expires to the time what ever you have set in OEM properties and if we want to change to the time depending on your Organization policies then please follow below step

Here i'm trying to set it to NOT expire / Unlimited to do so 

Check the current value from OMS server 

[oracle@OMSHOSTSRVR bin]$ ./emctl get property -name oracle.sysman.eml.maxInactiveTime
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
SYSMAN password:
Value for property oracle.sysman.eml.maxInactiveTime for oms All Management Servers is null

Set the property value with "-1" which is unlimited

[oracle@OMSHOSTSRVR bin]$ ./emctl set property -name oracle.sysman.eml.maxInactiveTime -value -1
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
SYSMAN password:
Property oracle.sysman.eml.maxInactiveTime has been set to value -1 for all Management Servers
OMS restart is not required to reflect the new property value

[oracle@OMSHOSTSRVR bin]$ ./emctl get property -name oracle.sysman.eml.maxInactiveTime
Oracle Enterprise Manager Cloud Control 13c Release 2
Copyright (c) 1996, 2016 Oracle Corporation.  All rights reserved.
SYSMAN password:
Value for property oracle.sysman.eml.maxInactiveTime at Global level is -1

BOUNCE the OMS to take affect the changes made

$ emctl stop oms -all
$ emctl start oms 

Referrence:
EM 12c : How to Change the Timeout Window of an emcli Session (Doc ID 1589497.1)
How to Change the Default Login (Idle) Timeout Value for the Enterprise Manager Cloud Console Connections? (Doc ID 1385996.1)

Wednesday, July 6, 2022

OEM 13c New feature - SYSTEM BROADCAST

 

There are times where OEM admin wants to inform and send notification banner to all the users or particular user who are connected to the OEM due to some maintenance/Downtime .

 So OEM 13c has one new feature introduced – “SYSTEM BROADCAST” , where you can send a broadcast message (upto 200 Char) using EMCLI command


Login to OMS server

 Cd $OMS/bin

 Syntax :

emcli publish_message -toOption="ALL|SPECIFIC"
           [-to="comma separated user names"]
           [-messageType="INFO|CONF|WARN|ERROR|WARNING" (default is INFO)]
          -message="message details" 
           [ ]  indicates that the parameter is optional.

 

 

[oracle@omssrvr bin]$ ./emcli send_system_broadcast -toOption="ALL" -messageType="INFO" -message="OEM is going down for Maintenance this Friday 10AM CST”

Status:Unauthorized 401

 

To send system broadcast to ALL users connected to OEM

[oracle@omssrvr bin]$ ./emcli login -username=sysman
 
Enter password :

 Login successful


 [oracle@omssrvr  bin]$ ./emcli send_system_broadcast -toOption="ALL" -messageType="INFO" -message= "OEM is going down for Maintenance this Friday 10AM CST”

Successfully requested to send System Broadcast to users.

 

To send system broadcast to Specific user connected to OEM 

./emcli send_system_broadcast  -toOption="SPECIFIC" -to="USERNAME" -messageType="WARN" -message="OEM is going down for Maintenance this Friday 10AM CST”

 

https://docs.oracle.com/en/enterprise-manager/cloud-control/enterprise-manager-cloud-control/13.5/emcli/send_system_broadcast.html

 

 


Auto Scroll Stop Scroll