12.2 ADOP Prepare Phase Fails with message " [ERROR]: Domain might be locked by some other WLS user process"

12.2 ADOP Prepare Phase Fails with message " [ERROR]: Domain might be locked by some other WLS user process" and "[UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script" (Doc ID 2042785.1)

 E-Business Suite 12.2 Applications DBA, Online Patching ADOP Utilities issues

On EBS 12.2 When running adop, prepare phase fails with the below error:

Validating configuration on node: [MYSERV]...
    Log: /u01/oracle/MYINST/fs_ne/EBSapps/log/adop/MYINST_MYSERV
        [WARNING]: There could be issues while validating the ports used for E-Business Suite instance against ports used in /etc/services. Refer the log file for more details.
        [WARNING]: Either some of the required entries in /etc/hosts file might be missing (e.g. localhost or hostname) OR the file /etc/hosts could not be read.
        [ERROR]: Domain might be locked by some other WLS user process
    [UNEXPECTED]Error occurred while executing "perl /u01/oracle/MYINST/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkADOPValidations.pl  -contextfile=/u01/oracle/MYINST/fs2/inst/apps/MYINST_MYSERV/appl/admin/MYINST_MYSERV.xml -patchctxfile=/u01/oracle/MYINST/fs1/inst/apps/MYINST_MYSERV/appl/admin/MYINST_MYSERV.xml -phase=prepare -logloc=/u01/oracle/MYINST/fs_ne/EBSapps/log/adop/14/prepare_20150811_185137/MYINST_MYSERV -promptmsg=hide"
    [UNEXPECTED]Error 1 occurred while Executing txkADOPValidation script on MYSERV

Reviewing the file $INST_TOP/apps/MYINST_MYSERV/logs/appl/rgf/TXK/txkValidateDomainInRCOut.txt, the below error message is recorded:

 cmd : java weblogic.WLST /u01/oracle/MYINST/fs2/EBSapps/appl/ad/12.0.0/patch/115/bin/txkValidateDomainInRC.py --adminuser=weblogic --verify=domainEditModeEnabled --adminurl=t1://hostname.domain:adminport> /u01/oracle/MYINST/fs2/inst/apps/MYINST_MYSERV/logs/appl/rgf/TXK/txkValidateDomainInRCOut.txt
ERROR: Domain is in edit mode.

CAUSE

If the EBS Weblogic domain is in EDIT mode, it can cause adop to fail.

Someone with administrator privileges on WLS Admin Console may have locked the Weblogic domain for making changes, and has not released the checkout.

This leaves the domain in edit mode, resulting in the issue.

SOLUTION

 To resolve the issue test the following steps in a development instance and then migrate accordingly:

1. Start the admin server on the run file system (if not started already).

2. Login in to WLS Admin Console (http(s)://hostname.domain:<adminport>/console)

3. Click on the "Activate Changes" or the "Release Configuration" button (depending on the version) in the "Change Center" region on the top left corner.

4. Logout of the WLS console.

5. Rerun the adop phase=prepare command and confirm adop completes successfully.

PGA MEMORY USAGE

How to monitor pga usage from within the database?

Queries can be run against the instance to show pga memory usage. Depending on the time frame of the problem occurrence these queries need to be run at different time intervals to review for trends.

The queries are best run through a cron job, Oracle scheduler job or on MS-Windows using the task scheduler. An example UNIX shell script is contained in attachment memleak.sh that can be used to kick off the SQL script at specified intervals. The script can be used to run pga_2010script1.sql containing PGA memory queries.
For example start meamleak.sh collecting at 20 second increments:

$ ./memleak.sh 20 > memleak.out

memleak.sh

#!/bin/sh

until test 0 -eq 1

do

sqlplus /nolog <<END

connect / as sysdba

@ pga_2010script1.sql

END

sleep $1

done

 ----

vi pga_2010script1.sql

set linesize 120

set pagesize 120

select to_char(sysdate, 'dd-MON-yyyy hh24:mi:ss') "Script Run Time" from dual;

select INSTANCE_NUMBER,INSTANCE_NAME,HOST_NAME from v$instance;

PROMPT

PROMPT V$SESSSTAT MEMORY INFO

PROMPT -------------------------

--Monitor the pga usage for all processes related to an instance from v$sesstat.

--Look at trends of individual processes growing in size or high number of processes

REM v$sesstat  pga/uga memory size

select p.spid, s.sid, substr(n.name,1,25) memory, s.value as Bytes from v$sesstat s, v$statname n, v$process p, v$session vs

where s.statistic# = n.statistic#

/* this query currently looks at both uga and pga, if only one of these is desired modify the like clause to pga or uga */

and n.name like '%ga memory%'

and s.sid=vs.sid

and vs.paddr=p.addr

/* --remove comment delimiters to view only certain sizes, i.e. over 10Mbytes */

/* and s.value > 10000000 */

order by s.value asc;

PROMPT

PROMPT LARGEST PGA_ALLOC_MEM PROCESS NOT LIKE LGWR

PROMPT -------------------------

REM List Largest process.

/* Do Not eliminate all background process because certain background processes do need to be monitored at times */

select pid,spid,substr(username,1,20) "USER" ,program,PGA_USED_MEM,PGA_ALLOC_MEM,PGA_FREEABLE_MEM,PGA_MAX_MEM

from v$process

where pga_alloc_mem=(select max(pga_alloc_mem) from v$process

where program not like '%LGWR%');

PROMPT

PROMPT SELECT SUM(PGA_ALLOC_MEM) sum(PGA_USED_MEM) FROM V$PROCESS

PROMPT -------------------------

REM Summation of pga based on v$process

REM allocated includes free PGA memory not yet released to the operating system by the server process

select sum(pga_alloc_mem)/1024/1024 as "Mbytes Alloc", sum(PGA_USED_MEM)/1024/1024 as "Mbytes used" from v$process;

PROMPT

PROMPT SELECT SUM PGA MEMORY FROM V$SESSTAT

PROMPT -------------------------

REM Summation of pga memory based on v$sesstat

select sum(value)/1024/1024 as "Mbytes" from v$sesstat s, v$statname n

        where

        n.STATISTIC# = s.STATISTIC# and

        n.name = 'session pga memory';

PROMPT

PROMPT SELECT * FROM V$PGASTAT

PROMPT -------------------------

REM PGA stats from v$pgastat

select substr(name,1,30), value, unit from v$pgastat;

PROMPT

PROMPT SHOW INFO on ALL PROCESSES

PROMPT -------------------------

--List all processes including pga size from v$process

--Outer join will show if any defunct processes exist without associated session.

set linesize 120

set pagesize 120

column spid heading 'OSpid' format a8

column pid heading 'Orapid' format 999999

column sid heading 'Sess id' format 99999

column serial# heading 'Serial#' format 999999

column status heading 'Status' format a8

column pga_alloc_mem heading 'PGA alloc' format 99,999,999,999

column pga_used_mem heading 'PGA used' format 99,999,999,999

column username heading 'oracleuser' format a12

column osuser heading 'OS user' format a12

column program heading 'Program' format a20

SELECT

p.spid,

p.pid,

s.sid,

s.serial#,

s.status,

p.pga_alloc_mem,

p.PGA_USED_MEM,

s.username,

s.osuser,

s.program

FROM

v$process p,

v$session s

WHERE s.paddr ( + ) = p.addr

and p.BACKGROUND is null /* Comment out this line if need to monitor background processes */

Order by p.pga_alloc_mem desc;

PROMPT

PROMPT SUM of PGA and SGA FROM V$SESSTAT,V$SGA

PROMPT -------------------------

--Summation of pga and sga gives a value of total memory usage by oracle instance

--look at total memory used by instance SGA and PGA

select sum(bytes)/1024/1024 as "Total PGA+SGA Mbytes" from

        (select value as bytes from v$sga

        union all

        select value as bytes from

        v$sesstat s,

        v$statname n

        where

        n.STATISTIC# = s.STATISTIC# and

        n.name = 'session pga memory'

        );

clear breaks

execute

 ./memleak.sh 20 > memleak.out