WARNING OGG-01932 Datastore already exists.

After Server Reboot where OGG is running my goldengate manager is unable to start and getting below error

GGSCI (oradev.domain.com) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED

JAGENT      STOPPED

EXTRACT     RUNNING     EXTDEVQ1    00:00:02      00:00:07

EXTRACT     RUNNING     EXTDEVQ1    00:00:04      00:00:09

EXTRACT     RUNNING     EXTDEVQ1    00:00:01      00:00:09

EXTRACT     RUNNING     EXTDEVQ1    00:00:02      00:00:03

EXTRACT     RUNNING     PXTDEVQ1    00:00:00      00:00:08

EXTRACT     RUNNING     PXTDEVQ1    00:00:00      00:00:03

EXTRACT     RUNNING     PXTDEVQ1    00:00:00      00:00:05

EXTRACT     RUNNING     PXTDEVQ1    00:00:00      00:00:02

GGSCI (oradev.domain.com) 1> start mgr

Manager started.

2019-01-17 10:37:29  INFO    OGG-00987  Oracle GoldenGate Command Interpreter for Oracle:  GGSCI command (oracle): start mgr.

2019-01-17 10:37:29  WARNING OGG-01877  Oracle GoldenGate Manager for Oracle, mgr.prm:  Missing explicit accessrule for server collector.

2019-01-17 10:37:29  INFO    OGG-00960  Oracle GoldenGate Manager for Oracle, mgr.prm:  Access granted (rule #7).

2019-01-17 10:37:29  INFO    OGG-00983  Oracle GoldenGate Manager for Oracle, mgr.prm:  Manager started (port 7809).

2019-01-17 10:37:29  WARNING OGG-01934  Oracle GoldenGate Manager for Oracle, mgr.prm:  Datastore repair failed.

2019-01-17 10:37:29  WARNING OGG-06530  Oracle GoldenGate Manager for Oracle, mgr.prm:  Berkeley Database encountered a Critical error.

2019-01-17 10:37:29  WARNING OGG-00543  Oracle GoldenGate Manager for Oracle, mgr.prm:  Unexpected threading library failure. Error code 16 (Device or resource busy).

GGSCI (oradev.domain.com) 13> delete datastore

Are you sure you want to delete the datastore? y

BDB0118 shmat: id 1679097930: unable to attach to shared system memory region: Invalid argument

BDB0118 shmat: id 1679097930: unable to attach to shared system memory region: Invalid argument

2019-01-17T15:37:57Z  INFO    OGG-06492  Datastore deleted

GGSCI (oradev.domain.com) 14>

GGSCI (oradev.domain.com) 14> info datastore

2019-01-17T15:38:03Z  WARNING OGG-06307  Datastore does not exist.

GGSCI (oradev.domain.com) 15> Create datastore mmap

2019-01-17T15:38:13Z  WARNING OGG-01933  Datastore create failed.

GGSCI (oradev.domain.com) 18> Create datastore

2019-01-17T15:39:44Z  WARNING OGG-01933  Datastore XTeate failed.

oradev:NOTSET:/home/oracle $ cd /goldengate/home/

oradev:NOTSET:/goldengate/home $ cd dirbdb

-bash: cd: dirbdb: No such file or directory

Looks like 'dirdbd' Directory is missing or Lost

oradev:NOTSET:/goldengate/home $ mkdir dirbdb

GGSCI (oradev.domain.com) 1> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED

JAGENT      STOPPED

EXTRACT     STOPPED     EXTDEVQ1    00:00:46      00:17:59

EXTRACT     STOPPED     EXTDEVQ1    00:00:43      00:17:59

EXTRACT     STOPPED     EXTDEVQ1    00:00:44      00:17:58

EXTRACT     ABENDED     EXTDEVQ1    00:00:02      00:18:39

EXTRACT     STOPPED     PXTDEVQ1    00:00:00      00:17:57

EXTRACT     STOPPED     PXTDEVQ1    00:00:00      00:17:56

EXTRACT     STOPPED     PXTDEVQ1    00:00:00      00:17:55

EXTRACT     STOPPED     PXTDEVQ1    00:00:00      00:17:55

GGSCI (oradev.domain.com) 2> Create datastore mmap

2019-01-17T15:55:06Z  WARNING OGG-01932  Datastore already exists.

GGSCI (oradev.domain.com) 3> info datastore

2019-01-17T15:55:14Z  INFO    OGG-06308  Datastore uses memory mapped (MMAP) environment files.

GGSCI (oradev.domain.com) 5> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     STOPPED

JAGENT      STOPPED

EXTRACT     STOPPED     EXTDEVQ1    00:00:46      00:18:40

EXTRACT     STOPPED     EXTDEVQ1    00:00:43      00:18:39

EXTRACT     STOPPED     EXTDEVQ1    00:00:44      00:18:39

EXTRACT     ABENDED     EXTDEVQ1    00:00:02      00:19:20

EXTRACT     STOPPED     PXTDEVQ1    00:00:00      00:18:38

EXTRACT     STOPPED     PXTDEVQ1    00:00:00      00:18:37

EXTRACT     STOPPED     PXTDEVQ1    00:00:00      00:18:36

EXTRACT     STOPPED     PXTDEVQ1    00:00:00      00:18:35

GGSCI (oradev.domain.com) 6> start mgr

Manager started.

GGSCI (oradev.domain.com) 7> info mgr

Manager is running (IP port oradev.domain.com.7809, Process ID 139824).

Goldengate ENABLEMONITORING in 12.3 and above

In our Current env where we have OGG 12.2 version we use REST API to access/View the GG processes using below url

http://<my_GGserver_name>:7809/groups

           where < my_GGserver_name>  --- > is the server where my OGG is running

                            7809                           ---> is the Port on which my MGR is running

    

But after upgrading OGG from 12.2 to 18.1 above url stopped working which is due to the new feature introduce in 12.3 version called  Performance Metric Server (PMSRVR)

GGSCI (oradev.domain.com) 9> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT      RUNNING

PMSRVR      STOPPED

EXTRACT     RUNNING     EXT1    00:00:04      00:00:09

EXTRACT     RUNNING     EXT2    00:00:02      00:00:05

EXTRACT     RUNNING     PXT1    00:00:03      00:00:04

EXTRACT     RUNNING     PXT2    00:26:22      00:00:04

GSCI (oradev.domain.com) 6> start PMSRVR

PMSRVR started

GGSCI (oradev.domain.com) 9> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT      RUNNING

PMSRVR      RUNNING

EXTRACT     RUNNING     EXT1    00:00:04      00:00:09

EXTRACT     RUNNING     EXT2    00:00:02      00:00:05

EXTRACT     RUNNING     PXT1    00:00:03      00:00:04

EXTRACT     RUNNING     PXT2    00:26:22      00:00:04

But still the url not working because from 12.3 where PMSRVR is introduced for monitoring using rest API you need to modify GLOBALS param and add below param

ENABLEMONITORING [ UDP ]

ENABLEMONITORING [ UDP ] [ UDPPORT portnumber ] [ HTTPPORT portnumber ]

If you Don’t give any PORT number then it will use default port which is 9004

GGSCI (oradev.domain.com) 3> status PMSRVR

PM Server is Running (HTTP Port oradev.domain.com.9004, Process ID 79277)

After Modifying GLOBALS param just bounce all the GG params including MGR and use the below url

http://<my_GGserver_name>:9004/groups

You will see all the GG processes and some new additional processes which is shown BELOW which shows DOWN since I’m not using GG Micro Services Architecture

Check more on PMSRVR here : 

http://chandu208.blogspot.com/2019/01/goldengate-performance-metric-server.html

OGG The requested URL is not serviced by Manager in UDP Mode

OGG Version: 18.1

After upgrading OGG to 18c from 12.2 then when Trying to access below url which was working before upgrade we are seeing below in the web page

http://<my_GGserver_name>:7809/groups

The requested URL is not serviced by Manager in UDP Mode

GGSCI (oradev.domain.com) 9> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT      RUNNING

PMSRVR      STOPPED

EXTRACT     RUNNING     EXT1    00:00:04      00:00:09

EXTRACT     RUNNING     EXT2    00:00:02      00:00:05

EXTRACT     RUNNING     PXT1    00:00:03      00:00:04

EXTRACT     RUNNING     PXT2    00:00:22      00:00:04

Cause:

I see somehow GLOBALS param contains mgr default port 7809 which is already in use

$ cat GLOBALS

GGSCHEMA ggadmin

ENABLEMONITORING UDP HTTPPORT 7809

If you use 7809 in GLOBALS (HTTPPORT 7809) then when you start PMSRVR you will see below error since mgr already uses this port

GGSCI (oradev.domain.com) 7> Attempt 1 - Bind failed for address 'ANY:7809'[AF_INET] on 'ANY' interface with global scope.

'Address already in use' (98)

Attempt 2 - Bind failed for address 'ANY:7809'[AF_INET] on 'ANY' interface with global scope.

'Address already in use' (98)

Attempt 3 - Bind failed for address 'ANY:7809'[AF_INET] on 'ANY' interface with global scope.

'Address already in use' (98)

Attempt 4 - Bind failed for address 'ANY:7809'[AF_INET] on 'ANY' interface with global scope.

'Address already in use' (98)

Attempt 5 - Bind failed for address 'ANY:7809'[AF_INET] on 'ANY' interface with global scope.

SOLUTION :

So in order to fix it stop all OGG processes and we need to use other open PORT in GLOBALS file which is not already in use and then restart the PMSRVR and then all other remaining OGG processes including mgr, then issue will be resolved

$ cat GLOBALS

GGSCHEMA ggadmin

ENABLEMONITORING UDP

Here I removed 7809 and didn’t gave any port number which means I’m using default 7809

Now, STOP and STARTED all process

stop er*

stop mgr

stop jagent

GGSCI (oradev.domain.com) 2> start PMSRVR

PMSRVR started

start mgr

start jagent

stop er*

GGSCI (oradev.domain.com) 9> info all

Program     Status      Group       Lag at Chkpt  Time Since Chkpt

MANAGER     RUNNING

JAGENT      RUNNING

PMSRVR      RUNNING

EXTRACT     RUNNING     EXT1    00:00:04      00:00:09

EXTRACT     RUNNING     EXT2    00:00:02      00:00:05

EXTRACT     RUNNING     PXT1    00:00:03      00:00:04

EXTRACT     RUNNING     PXT2    00:00:22      00:00:04

The monitoring functionality previously provided by the Manager process is now provided using the new Performance Metrics server process (PMSRVR)

Check about PMSRVR here: 

http://chandu208.blogspot.com/2019/01/goldengate-performance-metric-server.html