Goldengate ENABLEMONITORING in 12.3 and above

In our Current env where we have OGG 12.2 version we use REST API to access/View the GG processes using below url

http://<my_GGserver_name>:7809/groups

           where < my_GGserver_name>  --- > is the server where my OGG is running

                            7809                           ---> is the Port on which my MGR is running

    

But after upgrading OGG from 12.2 to 18.1 above url stopped working which is due to the new feature introduce in 12.3 version called  Performance Metric Server (PMSRVR)

GGSCI (oradev.domain.com) 9> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT      RUNNING

PMSRVR      STOPPED

EXTRACT     RUNNING     EXT1    00:00:04      00:00:09

EXTRACT     RUNNING     EXT2    00:00:02      00:00:05

EXTRACT     RUNNING     PXT1    00:00:03      00:00:04

EXTRACT     RUNNING     PXT2    00:26:22      00:00:04

GSCI (oradev.domain.com) 6> start PMSRVR

PMSRVR started

GGSCI (oradev.domain.com) 9> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT      RUNNING

PMSRVR      RUNNING

EXTRACT     RUNNING     EXT1    00:00:04      00:00:09

EXTRACT     RUNNING     EXT2    00:00:02      00:00:05

EXTRACT     RUNNING     PXT1    00:00:03      00:00:04

EXTRACT     RUNNING     PXT2    00:26:22      00:00:04

But still the url not working because from 12.3 where PMSRVR is introduced for monitoring using rest API you need to modify GLOBALS param and add below param

ENABLEMONITORING [ UDP ]

ENABLEMONITORING [ UDP ] [ UDPPORT portnumber ] [ HTTPPORT portnumber ]

If you Don’t give any PORT number then it will use default port which is 9004

GGSCI (oradev.domain.com) 3> status PMSRVR

PM Server is Running (HTTP Port oradev.domain.com.9004, Process ID 79277)

After Modifying GLOBALS param just bounce all the GG params including MGR and use the below url

http://<my_GGserver_name>:9004/groups

You will see all the GG processes and some new additional processes which is shown BELOW which shows DOWN since I’m not using GG Micro Services Architecture

Check more on PMSRVR here : 

http://chandu208.blogspot.com/2019/01/goldengate-performance-metric-server.html

OGG The requested URL is not serviced by Manager in UDP Mode

OGG Version: 18.1

After upgrading OGG to 18c from 12.2 then when Trying to access below url which was working before upgrade we are seeing below in the web page

http://<my_GGserver_name>:7809/groups

The requested URL is not serviced by Manager in UDP Mode

GGSCI (oradev.domain.com) 9> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT      RUNNING

PMSRVR      STOPPED

EXTRACT     RUNNING     EXT1    00:00:04      00:00:09

EXTRACT     RUNNING     EXT2    00:00:02      00:00:05

EXTRACT     RUNNING     PXT1    00:00:03      00:00:04

EXTRACT     RUNNING     PXT2    00:00:22      00:00:04

Cause:

I see somehow GLOBALS param contains mgr default port 7809 which is already in use

$ cat GLOBALS

GGSCHEMA ggadmin

ENABLEMONITORING UDP HTTPPORT 7809

If you use 7809 in GLOBALS (HTTPPORT 7809) then when you start PMSRVR you will see below error since mgr already uses this port

GGSCI (oradev.domain.com) 7> Attempt 1 - Bind failed for address 'ANY:7809'[AF_INET] on 'ANY' interface with global scope.

'Address already in use' (98)

Attempt 2 - Bind failed for address 'ANY:7809'[AF_INET] on 'ANY' interface with global scope.

'Address already in use' (98)

Attempt 3 - Bind failed for address 'ANY:7809'[AF_INET] on 'ANY' interface with global scope.

'Address already in use' (98)

Attempt 4 - Bind failed for address 'ANY:7809'[AF_INET] on 'ANY' interface with global scope.

'Address already in use' (98)

Attempt 5 - Bind failed for address 'ANY:7809'[AF_INET] on 'ANY' interface with global scope.

SOLUTION :

So in order to fix it stop all OGG processes and we need to use other open PORT in GLOBALS file which is not already in use and then restart the PMSRVR and then all other remaining OGG processes including mgr, then issue will be resolved

$ cat GLOBALS

GGSCHEMA ggadmin

ENABLEMONITORING UDP

Here I removed 7809 and didn’t gave any port number which means I’m using default 7809

Now, STOP and STARTED all process

stop er*

stop mgr

stop jagent

GGSCI (oradev.domain.com) 2> start PMSRVR

PMSRVR started

start mgr

start jagent

stop er*

GGSCI (oradev.domain.com) 9> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT      RUNNING

PMSRVR      RUNNING

EXTRACT     RUNNING     EXT1    00:00:04      00:00:09

EXTRACT     RUNNING     EXT2    00:00:02      00:00:05

EXTRACT     RUNNING     PXT1    00:00:03      00:00:04

EXTRACT     RUNNING     PXT2    00:00:22      00:00:04

The monitoring functionality previously provided by the Manager process is now provided using the new Performance Metrics server process (PMSRVR)

Check about PMSRVR here: 

http://chandu208.blogspot.com/2019/01/goldengate-performance-metric-server.html

Goldengate Performance Metric Server OGG 12.3

The Performance Metrics Server (PMSRVR) introduced in OGG 12.3 version and uses the metrics service to collect and store instance deployment performance results

We can use Performance Metrics Server in both CLASSIC and MICROSERVICES Architecture modes

When PMSRVR is started for the first time, the sequence of commands must be:

START PMSRVR

START MANAGER

START *

START JAGENT

Command                                          Description 

INFO PMSRVR                        Returns information about the PMSRVR. 

START PMSRVR                    Starts the PMSRVR. 

STATUS PMSRVR                  Returns the state of the PMSRVR. 

STOP PMSRVR                      Stops the PMSRVR.

START PMSRVR                    Starts the PMSRVR.

GGSCI (prod01.domain.com) 4> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT       RUNNING

PMSRVR    STOPPED

EXTRACT    RUNNING     EXTPSTDB    00:00:00      00:00:00

EXTRACT    RUNNING     PMPUPDB    00:00:00      00:00:01

REPLICAT    RUNNING     REPTGDB      00:00:00      00:00:05

GGSCI (prod01.domain.com) 4> start pmsrvr

PMSRVR started

GGSCI (prod01.domain.com) 4> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT       RUNNING

PMSRVR    RUNNING

EXTRACT    RUNNING     EXTPSTDB    00:00:00      00:00:00

EXTRACT    RUNNING     PMPUPDB    00:00:00      00:00:00

REPLICAT    RUNNING     REPTGDB      00:00:00      00:00:01

ALL "datastore" commands are deprecated in Oracle GoldenGate 12.3.x and above.

If using GoldenGate 12.3.x and above then DO NOT execute delete datastore, create datastore commands if you do so you will

receive "command not found" type errors.

"start pmsrvr" will create and manage $OGG_HOME/dirbdb

Enable monitoring options are changed

ENABLEMONITORING [ UDP ]

ENABLEMONITORING [ UDP ] [ UDPPORT portnumber ] [ HTTPPORT portnumber ]

default port is 9004

Oracle Goldengate 18.1 Install

Oracle released 18.1.0 in Oct 2018 and here i'll explaining on how to do new install of this release and i see no difference from 12c version and its same and very simple.

Before you install/Upgrade oracle goldengate to 18.1 you need to check the compatibility for your environment, you can check the compatible matrix below in oracle page

Oracle Fusion Middleware Supported System Configurations 

Also, According to oracle if you have multiple RDBMS versions like Oracle Database 11.2 and 12.1, you must have separate Oracle GoldenGate installations for each one.

The Classic Extract (capture) is deprecated as of Oracle GoldenGate 18c (18.1.0). This means that there will be no further enhancements to the Classic Capture (Extract) process.  

Download New 18c (18.1) goldengate version from oracle, here I'm NOT using Microservices Architecture

Download traditional one (highlighted) and copy to some location on server

mkdir -p /goldengate/ggsw/18.1_soft

devhost01:NOTSET:/goldengate/ggsw/18.1_soft $ ll

total 383516

-rw-r--r-- 1 oracle dba 392717701 Jan 15 17:01 181000_fbo_ggs_Linux_x64_shiphome.zip

devhost01:NOTSET:/goldengate/ggsw/18.1_soft $ unzip 181000_fbo_ggs_Linux_x64_shiphome.zip

devhost01:NOTSET:/goldengate/ggsw/18.1_soft $  ls -lrth

total 375M

drwxr-xr-x 3 oracle dba 4.0K Sep 29 14:13 fbo_ggs_Linux_x64_shiphome

-rw-r--r-- 1 oracle dba 196K Oct 10 11:22 OGG_WinUnix_Rel_Notes_18.1.0.0.0.pdf

-rw-r--r-- 1 oracle dba 1.4K Oct 10 11:28 OGG-18.1.0.0-README.txt

-rw-r--r-- 1 oracle dba 375M Jan 15 17:01 181000_fbo_ggs_Linux_x64_shiphome.zip

devhost01:NOTSET:/goldengate/ggsw/18.1_soft/fbo_ggs_Linux_x64_shiphome/Disk1 $ ls -lrth

total 16K

drwxr-xr-x  4 oracle dba 4.0K Sep 29 14:12 install

drwxr-xr-x 11 oracle dba 4.0K Sep 29 14:13 stage

-rwxr-xr-x  1 oracle dba  918 Sep 29 14:13 runInstaller

drwxrwxr-x  2 oracle dba 4.0K Sep 29 14:13 response

devhost01:NOTSET:/goldengate/ggsw/18.1_soft/fbo_ggs_Linux_x64_shiphome/Disk1 $ export DISPLAY=localhost:0.0

devhost01:NOTSET:/goldengate/ggsw/18.1_soft/fbo_ggs_Linux_x64_shiphome/Disk1 $ ./runInstaller

Once GUI is opened you can follow the steps as required for your environment by clicking "Next"

Oracle Goldengate 12c Installation can be found here : Goldengate12c_install

OEM unable to discover Goldengate targets

OEM Version : 13.2 
OGG Version     : 12.2

My OEM has goldengate plugin already installed and the Jagent is installed on the Sserver where my Goldenagte processes are running and When trying to discover Goldengate Targets from 13c OEM its not able to discover and below are the messages i see in the Goldengate monitoring agent instance log

JAGENT_INSTANCE_HOME/logs $ tail -20f ogg_agent.log

[2019-01-02T13:11:51.868-05:00] [JAGENT] [ERROR] [OGGMON-20603] [com.goldengate.monitor.jagent.comm.ws.ManagerService] [tid: quickUpdates] [ecid: 0000MWFh2__1j^^5xV9DiW1SBFfa000004,0] RESTful Web Service with name mpoints has become unresponsive

[2019-01-02T13:11:53.578-05:00] [JAGENT] [ERROR] [OGGMON-20610] [com.goldengate.monitor.jagent.comm.ws.ManagerService] [tid: MessageCollector] [ecid: 0000MWFh1Nk1j^^5xV9DiW1SBFfa000001,0] Could not connect to Message Web Service. Query String: messages/last

[2019-01-02T13:11:53.578-05:00] [JAGENT] [ERROR] [OGGMON-20603] [com.goldengate.monitor.jagent.comm.ws.ManagerService] [tid: MessageCollector] [ecid: 0000MWFh1Nk1j^^5xV9DiW1SBFfa000001,0] RESTful Web Service with name messages/last has become unresponsive

[2019-01-02T13:11:53.595-05:00] [JAGENT] [ERROR] [OGGMON-20610] [com.goldengate.monitor.jagent.comm.ws.ManagerService] [tid: MessageCollector] [ecid: 0000MWFh1Nk1j^^5xV9DiW1SBFfa000001,0] Could not connect to Message Web Service. Query String: messages/last

FIX:

GGSCI (prod01.domain.com) 1> info datastore

2019-01-02T18:11:00Z  WARNING OGG-06307  Datastore does not exist.

GGSCI (prod01.domain.com) 4> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT       RUNNING

EXTRACT    RUNNING     EXTPSTDB    00:00:00      00:00:00

EXTRACT    RUNNING     PMPUPDB    00:00:00      00:00:01

REPLICAT    RUNNING     REPTGDB      00:00:00      00:00:05

Stop all GG processes including mgr and jagent

Stop *

Stop mgr

Stop jagent

GGSCI (prod01.domain.com) 6> create datastore

Profile 'Trace' added.

2019-01-02T18:12:26Z  INFO    OGG-06489  Datastore created

Start mgr

Start jagent

Start *

GGSCI (prod01.domain.com) 1> info datastore

2019-01-02T18:20:08Z  INFO    OGG-06309  Datastore uses shared memory (SHM) environment files with a starting ID of 0x00000b00.

Now go to OEM and do the discovery again by adding GG discovery module parameters

Also Refer:

Not Able To Discover The Goldengate Targets On OEM12c (Doc ID 2277992.1)

Happy New Year 2019

Hello everyone i wish you all a wonderful Happy New Year 2019

As always i'll keep Blogging my Tech Experiences/Challenges to help myself and Visitors, Thank you all wish you Success and Good Luck, Cheers !!!