Skip to content

Oracle Pod not coming UP after restart in active/standby deployment #233

@alinaqvion-hash

Description

@alinaqvion-hash

Once we have active and standby oracle pods deployed along with dataguard.

We were trying to patch with new image and db pod didn't come up.

Then, we again deploy fresh everything and just doing standby pod restart. it's not coming UP.

Pod status ##########

[root@master-node-02 ~]# kubectl get pod -n testing
NAME READY STATUS RESTARTS AGE
oracle-db-dg-wce2d 1/1 Running 0 3d22h
testing-oracle-db-ali4-fl2x4 1/1 Running 0 2d15h
testing-oracle-db-test-02-fg8xw 1/1 Running 0 3d23h
testing-oracle-db-test-141jv 1/1 Running 0 4d

SIDB status ##########

[root@master-node-02 ~]# kubectl get singleinstancedatabase -n testing
NAME EDITION STATUS ROLE VERSION CONNECT STR TCPS CONNECT STR OEM EXPRESS URL
testing-oracle-db-test Enterprise Healthy PRIMARY 19.27.0.0.0 172.16.0.114:30048/PSMF Unavailable https://172.16.0.55:3 1830/em
testing-oracle-db-test-02 Enterprise Healthy PHYSICAL_STANDBY 19.27.0.0.0 172.16.0.174:30059/PSMF2 Unavailable https://172.16.0.55:3 1830/em

Standby Pod deleted ##########

[root@master-node-02 ~]# kubectl delete pod testing-oracle-db-test-02-fg8xw -n testing
pod "testing-oracle-db-test-02-fg8xw" deleted from testing namespace
[root@master-node-02 ~]# kubectl get pod -n testing
NAME READY STATUS RESTARTS AGE
ali-chart-1-oracle-pvc-untar--test-nm8sz 0/1 Completed 0 16d
ali-chart-2-post-install-health-mrt28 0/1 Completed 0 16d
ali-osp-oraclepre1-oracle-pvc-untar-p267m 0/1 Pending 0 3d20h
oracle-db-dg-wce2d 1/1 Running 0 3d22h
python-rman-backup-29581965-s4hsm 0/1 Completed 0 10d
python-rman-backup-29581980-8n5qn 0/1 Completed 0 10d
python-rman-backup-29581995-wvgf9 0/1 Completed 0 10d
smf-chart-1-oracle-pvc-untar-z2dgw 0/1 Completed 0 4d
smf-chart-2-oracle-post-install-import-s7j47 0/1 Completed 0 3d23h
smf-chart-2-post-install-health-cjbfp 0/1 Error 0 4d
smf-chart-2-post-install-health-qhd9j 0/1 Completed 0 3d23h
smf-chart-3-osp-oraclestby-post-upgrade-job-mtnxt 0/1 Completed 0 3d23h
smf-chart-4-osp-oracledg-post-upgrade-job-mwdnp 0/1 Completed 0 3d22h
testing-oracle-db-ali4-fl2x4 1/1 Running 0 2d15h
testing-oracle-db-test-02-39lft 0/1 Running 0 6s
testing-oracle-db-test-141jv 1/1 Running 0 4d

standby pod startup logs ##########

[root@master-node-02 ~]# kubectl logs -f testing-oracle-db-test-02-39lft -n testing
Defaulted container "testing-oracle-db-test-02" out of: testing-oracle-db-test-02, init-wallet (init)
[2026:04:10 08:37:21]: Acquiring lock .PSMF2.create_lck with heartbeat 30 secs
[2026:04:10 08:37:21]: Lock acquired
[2026:04:10 08:37:21]: Starting heartbeat
[2026:04:10 08:37:21]: Lock held .PSMF2.create_lck
ORACLE EDITION: ENTERPRISE

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Apr 10 08:37:21 2026
Version 19.27.0.0.0

Copyright (c) 1982, 2024, Oracle. All rights reserved.

Connected to an idle instance.

SQL> ORACLE instance shut down.
SQL> ORACLE instance started.

Total System Global Area 1.6106E+10 bytes
Fixed Size 18663088 bytes
Variable Size 2181038080 bytes
Database Buffers 1.3892E+10 bytes
Redo Buffers 14888960 bytes
SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
[2026:04:10 08:37:34]: Releasing lock .PSMF2.create_lck
[2026:04:10 08:37:34]: Lock released .PSMF2.create_lck
[2026:04:10 08:37:34]: Acquiring lock .PSMF2.exist_lck with heartbeat 30 secs
[2026:04:10 08:37:37]: Lock acquired
[2026:04:10 08:37:37]: Starting heartbeat
[2026:04:10 08:37:37]: Lock held .PSMF2.exist_lck

LSNRCTL for Linux: Version 19.0.0.0.0 - Production on 10-APR-2026 08:37:37

Copyright (c) 1991, 2025, Oracle. All rights reserved.

Starting /opt/oracle/product/19c/dbhome_1/bin/tnslsnr: please wait...

TNSLSNR for Linux: Version 19.0.0.0.0 - Production
System parameter file is /opt/oracle/product/19c/dbhome_1/network/admin/listener.ora
Log messages written to /opt/oracle/diag/tnslsnr/testing-oracle-db-test-02-39lft/listener/alert/log.xml
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
Listening on: (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))

Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=EXTPROC1)))
STATUS of the LISTENER

Alias LISTENER
Version TNSLSNR for Linux: Version 19.0.0.0.0 - Production
Start Date 10-APR-2026 08:37:37
Uptime 0 days 0 hr. 0 min. 0 sec
Trace Level off
Security ON: Local OS Authentication
SNMP OFF
Listener Parameter File /opt/oracle/product/19c/dbhome_1/network/admin/listener.ora
Listener Log File /opt/oracle/diag/tnslsnr/testing-oracle-db-test-02-39lft/listener/alert/log.xml
Listening Endpoints Summary...
(DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=EXTPROC1)))
(DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=0.0.0.0)(PORT=1521)))
Services Summary...
Service "DATAGUARD" has 1 instance(s).
Instance "PSMF2", status UNKNOWN, has 1 handler(s) for this service...
Service "PSMF2" has 1 instance(s).
Instance "PSMF2", status UNKNOWN, has 1 handler(s) for this service...
Service "PSMF2_DGMGRL" has 1 instance(s).
Instance "PSMF2", status UNKNOWN, has 1 handler(s) for this service...
The command completed successfully

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Apr 10 08:37:37 2026
Version 19.27.0.0.0

Copyright (c) 1982, 2024, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0

SQL> SQL>
Database altered.

SQL>
Database altered.

SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 1 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Apr 10 08:37:49 2026
Version 19.27.0.0.0

Copyright (c) 1982, 2024, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0

SQL> SQL> alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted

SQL> alter database open
*
ERROR at line 1:
ORA-01531: a database already open by the instance

SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 2 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Apr 10 08:37:51 2026
Version 19.27.0.0.0

Copyright (c) 1982, 2024, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0

SQL> SQL> alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted

SQL> alter database open
*
ERROR at line 1:
ORA-01531: a database already open by the instance

SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 3 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Apr 10 08:37:54 2026
Version 19.27.0.0.0

Copyright (c) 1982, 2024, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0

SQL> SQL> alter database mount
*
ERROR at line 1:
ORA-01100: database already mounted

SQL> alter database open
*
ERROR at line 1:
ORA-01531: a database already open by the instance

SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 4 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Apr 10 08:37:59 2026
Version 19.27.0.0.0

Copyright (c) 1982, 2024, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0

SQL> SQL> alter database mount
*
ERROR at line 1:
ORA-01154: database busy. Open, close, mount, and dismount not allowed now

SQL> alter database open
*
ERROR at line 1:
ORA-01531: a database already open by the instance

SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 5 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Apr 10 08:38:04 2026
Version 19.27.0.0.0

Copyright (c) 1982, 2024, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0

SQL> SQL> alter database mount
*
ERROR at line 1:
ORA-01154: database busy. Open, close, mount, and dismount not allowed now

SQL> alter database open
*
ERROR at line 1:
ORA-01531: a database already open by the instance

SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 6 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Apr 10 08:38:11 2026
Version 19.27.0.0.0

Copyright (c) 1982, 2024, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0

SQL> SQL> alter database mount
*
ERROR at line 1:
ORA-01154: database busy. Open, close, mount, and dismount not allowed now

SQL> alter database open
*
ERROR at line 1:
ORA-01531: a database already open by the instance

SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 7 sec(s) before restarting Oracle processes and opening the database

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Apr 10 08:38:19 2026
Version 19.27.0.0.0

Copyright (c) 1982, 2024, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0

SQL> SQL> alter database mount
*
ERROR at line 1:
ORA-01154: database busy. Open, close, mount, and dismount not allowed now

SQL> alter database open
*
ERROR at line 1:
ORA-01531: a database already open by the instance

SQL>
System altered.

SQL> Disconnected from Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0
The Oracle base remains unchanged with value /opt/oracle
Waiting for 8 sec(s) before restarting Oracle processes and opening the database

DG status ##########

[root@master-node-02 ~]# kubectl describe dataguardbroker oracle-db-dg -n testing
Name: oracle-db-dg
Namespace: testing
Labels: app.kubernetes.io/managed-by=Helm
Annotations: Environment: K8s Lab in BO.
OSP-POD-Desc:
This is a TEST Oracle POD for OSP.
It is used to perform PoC for Oracle POD Deployments.
Please do not use in production.
Owner: OSP DB Team - Faisal, Parveen
Use-Case: Deploy TWO Data Guard Configuration in site1 and site2 namespaces
meta.helm.sh/release-name: smf-chart-4
meta.helm.sh/release-namespace: testing
API Version: database.oracle.com/v4
Kind: DataguardBroker
Metadata:
Creation Timestamp: 2026-04-06T09:52:47Z
Finalizers:
database.oracle.com/dataguardbrokerfinalizer
Generation: 1
Resource Version: 13831943
UID: 1a1ba3ed-1777-4bb8-9c2c-cdc6d2d3f6e2
Spec:
Fast Start Failover: true
Primary Database Ref: testing-oracle-db-test
Protection Mode: MaxAvailability
Standby Database Refs:
testing-oracle-db-test-02
Status:
Cluster Connect String: oracle-db-dg.testing:1521/DATAGUARD
Databases In Dataguard Config:
PSMF: testing-oracle-db-test
PSMF2: testing-oracle-db-test-02
External Connect String: 172.16.0.114:30191/DATAGUARD
Fast Start Failover: true
Primary Database: PSMF
Protection Mode: MaxAvailability
Standby Databases: PSMF2
Status: Healthy
Events:
Type Reason Age From Message


Normal Enabling FastStartFailover successful 5m22s (x274 over 158m) DataguardBroker Enabling FastStartFailover for the dataguard broker oracle-db-dg successful
Normal Waiting 2m40s (x5 over 4m51s) DataguardBroker Waiting for testing-oracle-db-test-02 to be Ready

standby pod status in DB after restart ############

[root@master-node-02 ~]# kubectl exec -it testing-oracle-db-test-02-39lft -n testing -- bash
Defaulted container "testing-oracle-db-test-02" out of: testing-oracle-db-test-02, init-wallet (init)
bash-5.1$ sqlplus / as sysdba

SQL*Plus: Release 19.0.0.0.0 - Production on Fri Apr 10 09:01:05 2026
Version 19.27.0.0.0

Copyright (c) 1982, 2024, Oracle. All rights reserved.

Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.27.0.0.0

SQL> select open_mode, database_role from v$database;

OPEN_MODE DATABASE_ROLE


READ ONLY WITH APPLY PHYSICAL STANDBY

SQL>

DG configuration status ##########

[root@master-node-02 ~]# kubectl exec -it testing-oracle-db-test-141jv -n testing -- bash
Defaulted container "testing-oracle-db-test" out of: testing-oracle-db-test, init-wallet (init)
Copyright (c) 1982, 2019, Oracle and/or its affiliates. All rights reserved.

Welcome to DGMGRL, type "help" for information.
Connected to "PSMF"
Connected as SYSDBA.
DGMGRL> show configuration;
show database PSMF;
show database PSMF2;
Configuration - dg_config

Protection Mode: MaxAvailability
Members:
psmf - Primary database
psmf2 - (*) Physical standby database

Fast-Start Failover: Enabled in Zero Data Loss Mode

Configuration Status:
SUCCESS (status updated 54 seconds ago)

DGMGRL>
Database - psmf

Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s):
PSMF

Database Status:
SUCCESS

DGMGRL>

DGMGRL> show database PSMF2;

Database - psmf2

Role: PHYSICAL STANDBY
Intended State: APPLY-ON
Transport Lag: 0 seconds (computed 1 second ago)
Apply Lag: 0 seconds (computed 1 second ago)
Average Apply Rate: 0 Byte/s
Real Time Query: ON
Instance(s):
PSMF2

Database Status:
SUCCESS

DGMGRL> show database PSMF;

Database - psmf

Role: PRIMARY
Intended State: TRANSPORT-ON
Instance(s):
PSMF

Database Status:
SUCCESS

DGMGRL>

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions