19c Database Upgrade (DBUA or manual) fails with ORA-01017, ORA-00210, ORA-15001, ORA-00210 errors.

19c Database Upgrade (DBUA or manual) fails with ORA-01017, ORA-00210, ORA-15001, ORA-00210 errors 

Doc ID 2684747.1

During the database upgrade (using DBUA or a manual upgrade), the instance alert log shows below errors:

From alert log

2020-04-29T13:06:31.996208-04:00

ALTER DATABASE   MOUNT

2020-04-29T13:06:32.829188-04:00

WARNING: failed to start ASMB (connection failed) state=0x1 sid='+ASM'

2020-04-29T13:06:32.829387-04:00

Errors in file <ORACLE BASE>/diag/rdbms/<db name>/<db name>/trace/<db name>_asmb_11348.trc:

ORA-01017: invalid username/password; logon denied

...

2020-04-29T13:06:43.250267-04:00

Errors in file <ORACLE BASE>/diag/rdbms/<db name>/<db name>/trace/<db name>_mz00_11458.trc:

ORA-00202: control file: '+FRA01/<db name>/CONTROLFILE/control02.ctl'

ORA-17503: ksfdopn:2 Failed to open file +FRA01/<db name>/CONTROLFILE/control02.ctl

ORA-15001: diskgroup "FRA01" does not exist or is not mounted

ORA-01017: invalid username/password; logon denied

ORA-00210: cannot open the specified control file

The trace file (<db name>_mz00_11458.trc) shows:

020-04-24 12:31:43.330*:800003D5:KFNC:kgfn.c@5263:kgfnConnect4():1101:41568: kgfnConnect: bequeath connection

2020-04-24 12:31:43.330*:800003D6:KFNC:kgfn.c@6688:kgfnConnect2Int():1101:41568: kgfnConnect2Int: sysasm=0 envflags=0x1000005 srvrflags=0x3 unam=NULL password is NULL pstr=_asmb_<db name>dr srvrcallback=0x7349560 retrynoguid=0

2020-04-24 12:31:43.330*:800003D7:KFNC:kgfn.c@6929:kgfnConnect2Int():1101:41568: kgfnConnect2Int: cstr=(DESCRIPTION=(ADDRESS=(PROTOCOL=beq)(PROGRAM=<ORACLE HOME>/bin/oracle)(ARGV0=oracle+ASM_asmb_<db name>dr)(ENVS='ORACLE_HOME=<ORACLE HOME>,ORACLE_SID=+ASM,ORA_SERVER_BROKER_MODE=NONE')(ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))'))(CONNECT_DATA=(ORACLE_HOME=<ORACLE HOME>)(SID=+ASM))(SECURITY=(AUTHENTICATION_SERVICE=beq))(enable=setuser))

2020-04-24 12:31:44.357*:800003E3:KFNC:kgfn.c@7107:kgfnConnect2Int():1101:41568: kgfnConnect2Int: OCISessionBegin failed

2020-04-24 12:31:44.357*:800003E4:KFNC:kgfn.c@1738:kgfnRecordErrPriv():1101:41568: kgfnRecordErrPriv: status=-1  at kgfn.c:7223

2020-04-24 12:31:44.357*:800003E5:KFNC:kgfn.c@1784:kgfnRecordErrPriv():1101:41568: kgfnRecordErrPriv: 1017 error=ORA-01017: invalid username/password; logon denied

2020-04-24 12:31:44.357*:800003E6:KFNC:kgfn.c@1820:kgfnRecordErrPriv():1101:41568: kgfnRecordErrPriv: rec=1

2020-04-24 12:31:44.357*:800003E7:KFNC:kgfn.c@5461:kgfnConnect4():1101:41568: kgfnConnect2: failed to connect

There is no role separation. Both RDBMS and GRID binaries are owned by oracle.

id -a

uid=1100(oracle) gid=1000(oinstall) groups=1000(oinstall),1200(dba)

The "/etc/group" is not showing the  "oracle" user corresponding to "oinstall" group

[root@<hostname> ~]# grep oracle /etc/group
oinstall:x:54321:==>
dba:x:54322:oracle
racdba:x:54330:oracle

This was reviewed in internal bug 29821687 and the bug was closed as a OS configuration issue. 

SOLUTION 

As a workaround, modify the user "oracle" to reflect "oinstall" in primary and secondary group.

 usermod -g oinstall -G oinstall,dba oracle

After the change,

[root@<hostname> ~]# grep oracle /etc/group
oinstall:x:54321:oracle==>
dba:x:54322:oracle
racdba:x:54330:oracle