RC-50013: Fatal: Instantiate driver did not complete successfully

Problem:

While cloning recently upgraded EBS environment from [Release 11 to 12.1]
 using Rapid Clone with dbTechStack option, the following error occurs:

RC-50013: Fatal: Instantiate driver did not complete successfully

Contents of log file:

[AutoConfig Error Report] The following report lists errors AutoConfig encountered during each phase of its execution.  Errors are grouped by directory and phase. The report format is:       <filename>  <phase>  <return code where appropriate>   [APPLY PHASE]   AutoConfig could not successfully execute the following scripts:     Directory: /clone/oracle/product/12101/perl/bin/perl -I /clone/oracle/product/12101/perl/lib/5.14.1 -I /clone/oracle/product/12101/perl/lib/site_perl/5.14.1 -I /clone/oracle/product/12101/appsutil/perl /clone/oracle/product/12101/appsutil/clone       ouicli.pl               INSTE8_APPLY       255 AutoConfig is exiting with status 1 RC-50013: Fatal: Instantiate driver did not complete successfully. /clone/oracle/product/12101appsutil/driver/regclone.drv

Cause:

As the log file states that autoconfig failed as it could not execute the following perl script "ouicl.pl"

The possible reason could be:

1.     Either the script is not found at the location

2.     If the script is present, try to run the script manually.

3.     If it fails with the same error then the perl program "ouicli.pl" is unable to locate the JDK installation.

Solution:

·         If the script "ouicli.pl" is not found under clone directory, I would suggest to copy appsutil from source to target system and unzip appsutil under< ORACLE_HOME>

            $ unzip -o appsutil.zip

·         Check if the JDK/JRE installed under the "<RDBMS ORACLE_HOME>/appsutil/jre/bin" directory. If there is no "java" under "<RDBMS ORACLE_HOME>/appsutil/jre/bin", you will need to replace the "jre" directory and all sub-directories with a full JDK/JRE installation. Generally, you can find this under "$ORACLE_HOME/jdk"

                        $ cp -r $ORACLE_HOME/jdk/ $ORACLE_HOME/appsutil/jre/

·         Now check  that the java" executable under "<RDBMS ORACLE_HOME>/appsutil/jre/bin" is working fine.

·         If the java executable is working fine, make sure it is referenced directly in the Context File for DB tier. Check it with the below command:

                        $ cd $ORACLE_HOME/appsutil

                        $ grep s_adjvaprg< CONTEXT_NAME>.xml

                                    <ADJVAPRG oa_var="s_adjvaprg" osd="UNIX">/<ORACLE_HOME>/appsutil/jre/bin/java</ADJVAPRG>

·         Again run the cloning process,

                        perl adcfgclone.pl dbTechStack

                                    ApplyDatabase Completed Successfully.

Rapid Clone Ouicli.pl Fails: "Finished OUI CLI cloning for s_db_oh with return code: 16777215" (Doc ID 802464.1)

Cause

The perl program "ouicli.pl" is unable to locate the JDK installation. This could be due to any of the following reasons:

1) There is no JDK/JRE installed under the "<RDBMS ORACLE_HOME>/appsutil/jre/bin" directory.
2) There is a problem, such as file corruption, with the JDK.
3) The value of "s_adjvaprg" in the Context File is set to an invalid location.

Solution

1. Ensure there is a working "java" executable under "<RDBMS ORACLE_HOME>/appsutil/jre/bin":

$ cd $ORACLE_HOME/appsutil/jre/bin
$ ./java -version
java version "1.6.0_07"
Java(TM) SE Runtime Environment (build 1.6.0_07-b06)
Java HotSpot(TM) Server VM (build 10.0-b23, mixed mode)

If the version information is returned, the JDK/JRE is working. This "java" executable file should be referenced directly in the Context File, as noted in step #2.

2. Check the value of "s_adjvaprg" in the Context File:

$ cd $ORACLE_HOME/appsutil/
$ grep s_adjvaprg *.xml   (assumes there is only 1 ".xml" file)
<ADJVAPRG oa_var="s_adjvaprg" osd="UNIX">/d01/R1211/db/tech_st/11.1.0/appsutil/jre/bin/java</ADJVAPRG>

Note that the value of "s_adjvaprg" points to the full path to the java executable and includes the name of the executable, "java", as well.

3. If there is no "java" under "<RDBMS ORACLE_HOME>/appsutil/jre/bin", you will need to replace the "jre" directory and all sub-directories with a full JDK/JRE installation. Generally, you can find this under "$ORACLE_HOME/jdk"

$ cp -r $ORACLE_HOME/jdk/ $ORACLE_HOME/appsutil/jre/

4. If any changes have been made, now re-try the "adcfgclone.pl" command.

12.2 Patch Edition Auto config error

12.2 Patch Edition Auto config error.

Oracle 12.2.4 EBS
File system: PATCH

jtfictx.sh              INSTE8_PRF         1

      okladmprf.sh            INSTE8_PRF         1

      txkJavaMailerCfg.sh     INSTE8_PRF         1

      txkWebServicescfg.sh    INSTE8_PRF         1

  [APPLY PHASE]

  AutoConfig could not successfully execute the following scripts:

    Directory: /app/dev/appldev/SID/fs1/FMW_Home/webtier/perl/bin/perl -I
 /app/dev/appldev/SID/fs1/FMW_Home/webtier/perl/lib/5.10.0 -I
/app/dev/appldev/SID/fs1/FMW_Home/webtier/perl/lib/site_perl/5.10.0 -I
/app/dev/appldev/SID/fs1/EBSapps/appl/au/12.0.0/perl -I
/app/dev/appldev/SID/fs1/FMW_Home/webtier/ohs/mod_perl
/lib/site_perl/5.10.0/sun4-solaris-thread-multi-64
 /app/dev/appldev/SID/fs1/inst/apps/SID_CONTEXTNAME/admin/install

      txkGenADOPWrapper.pl    INSTE8_APPLY       1

AutoConfig is exiting with status 28

AutoConfig execution completed on Sat Jan 10 12:16:44 2017

Time taken for AutoConfig execution to complete : 2 mins  51 secs

Solution:
To run Autoconfig from the patch file system

 you must disable trigger ebs_login prior to running autoconfig

Disable the ebs_login trigger for patch file system

 "alter trigger ebs_logon disable" - Ensure you are logged in as the system user
Now run autoconfig with the patch env sourced
Make sure Autoconfig completes ok 

Enable the login trigger

 "alter trigger ebs_logon enable"
. EBSapps.env as Patch option
connect database using system or system/equivalent privileged user 


-------------STEPS--------------------------------------------------
sqlplus system/<password>  

SQL>alter trigger ebs_logon disable;
Trigger altered.

Then Run Auto config with in patch env

ensure that autconfig completed successfully
and

SQL> alter trigger ebs_logon enable;
------------------------------------------------------------------------

On a R12 Multi-Node Install -- Why Do All Middle Tiers Run All Services? (Doc ID 415385.1)

On a R12 Multi-Node Install -- Why Do All Middle Tiers Run All Services? (Doc ID 415385.1)

APPLIES TO:

Oracle Applications Manager - Version 12.0.1 to 12.1.3 [Release 12 to 12.1]
Information in this document applies to any platform.

GOAL

You have created a Multi-Node installation of the E-Business-Suite Release 12 using Rapid Install with the "Edit Services" feature. You changed the services available on each application node, however after the install is complete, you notice that all your application tiers are configured to run all services rather than those specified during rapid install.

Why are all application tier nodes configured to run all services ?

Here are the details to illustrate this situation more clearly :

You have 2 Application Tiers (APPS1 and APPS2). APPS1 is running Admin- and Concurrent Manager- Services and APPS2 is running Forms- and Web Services.

During the Rapid Install, you select the above configuration via the "Edit Services" button as follows :

• APPS1 : enable "Root Service Group", "Batch Processing Services"
• APPS2 : enable "Root Service Group", "Web Entry Point Services", "Web Application Services", "Other Service Group"

Configuring Applications Node Services in Oracle E-Business Suite Release 12 Note 406558.1

The logic behind these service classification is as follows :

1. Root Service Group (Runs services on AS, 10.1.3 OH)
   
   • Oracle Process Manager (OPM) : adopmnctl.sh
2. Web Entry Point Services (Runs services on AS, 10.1.3 OH)
   
   • HTTP Server : adapcctl.sh
3. Web Application Services (Runs services on AS, 10.1.3 OH)
   
   • OACORE OC4J : adoacorectl.sh
   • FORMS OC4J : adformsctl.sh
   • OAFM OC4J : adoafmctl.sh 
4. Batch Processing Services (Concurrent Managers and Apps Listener)
   
   • APPS TNS Listener : adalnctl.sh
   • Concurrent Managers : adcmctl.sh, adsvcm.sh, ieoicsm.sh, ieosvicsm.sh
   • Fulfillment Server : jtffmctl.sh, jtfsvfm.sh 
5. Other Service Group: (Runs services for Forms on 10.1.2)
   
   • Oracle Forms Server : adformsrvctl.sh
   • Oracle MWA Service : mwactlwrpr.sh

After the installation you see that the Autoconfig XML file is generated with the following entries for APPS1 and APPS2 :

• s_isDB=NO
• s_isAdmin=YES
• s_isForms=YES
• s_isConc=YES
• s_isWeb=YES

Why are all services enabled on the two application tier nodes instead of the configuration that was selected via the "Edit Services" feature ?

SOLUTION

In R12, the concept for Applications Nodes has changed.  When installing R12 with multiple nodes. all the nodes are now set as 'Y' in FND_NODES.

This occurs because in R12, concept of unified APPL_TOP is introduced which means everything is laid down on all servers.

From the APPL_TOP perspective, all the Servers on a Multi-Node Environment will have the same files and can now potentially start any Service if needed.  In some cases, additional configuration will be required before this can be done since there can be profiles, etc associated with each Server.

For R12, the only difference between the Servers, are the Services that have been activated on each Node.
The Services are identified by the variables on the /service_group/ section in the APPS Context File:

• Root Service Group : s_root_status
• Web Entry Point Services : s_web_entry_status
• Web Application Services : s_web_applications_status
• Batch Processing Services : s_batch_status
• Other Service Group : s_other_service_group_status

Depending on the value of these variables (enabled or disabled), adstrtal.sh / adstpall.sh will only start/stop the Services associated with them, ignoring the rest.

For example, if a node has only /s_batch_status/ "enabled" and the rest of the services are disabled, when you run adstrtall.sh on that Server and it will only start the Concurrent Managers and the TNS Listener for Apps.

If you experience the problem where all of the service_group variables are enabled on all application tiers make sure you are using the latest version of the Rapid Install StartCD (12.0.0.21) shipped with Patch 5766801.

Important variables in eBusiness Suite in R12.2

Important new variable in R12.2
$FILE_EDITION	shows which file edition you are using, run or patch
$RUN_BASE	shows absolute path to run file system
$PATCH_BASE	shows absolute path to patch file system
$NE_BASE	shows absolute path to non-edition file system
$APPL_TOP_NE	non-editioned appl_top path. Equivalent to $NE_BASE/EBSapps/appl
$LOG_HOME	Application Instance Specific Log Directory
$ADOP_LOG_HOME	Online patching Specific Log Directory. Equivalent to $NE_BASE/EBSapps/log/adop
$IAS_ORACLE_HOME	FMW Web Tier Home Directory
$ORACLE_HOME	10.1.2 ORACLE_HOME
$CONTEXT_FILE	Source for information populating template files (autoconfig)
$EBS_DOMAIN_HOME	WLS Deployment of Oracle E-Business Suite 12.2 Domain (instance specific
$ADMIN_SCRIPTS_HOME	Shell scripts to control processes associated to the Applications Instance
$EBS_ORACLE_HOME	Oracle E-Business Suite 12.2 FMW Deployment directory
$RW	10.1.2 reports directory
$APPS_VERSION	to get the EBS version

Oracle ASM Installation

Oracle ASM Installation
------------------------------
Very nice explanation how to install

ASM INSTALLATION STEPS

ASM UPGRADE

Current Applied Patch & Patch level status

Check Current Applied Patch & Patch level status of all modules in oracle

Check Current Applied Patch

set linesize 1000
set pages 9999
col patch_name format a10
col patch_type format a10
col maint_pack_level format a15
set colsep |

  SELECT   patch_name,           patch_type,           maint_pack_level,           creation_date    FROM   applsys.ad_applied_patches ORDER BY   creation_date DESC

Query used to view the patch level status of all modules

SELECT   a.application_name, DECODE (b.status, 'I', 'Installed', 'S', 'Shared', 'N/A') status,           patch_level  FROM   apps.fnd_application_vl a, apps.fnd_product_installations b WHERE   a.application_id = b.application_id;

Application Short Name of a module

Query to find Application Short Name of a module

The following query lists all the applications related information. This query can be used to find the APPLICATION_SHORT_NAME of a module (eg. Payables, Receivables, Order Management, etc.) that are often used for downloading FNDLOAD LDT files, adding responsibility to a user and many more.

You can uncomment the FAT.APPLICATION_NAME condition (very bottom line of the query) to learn about a particular module. In this case, I used "Payables".

SELECT fa.application_id           "Application ID",

       fat.application_name        "Application Name",

       fa.application_short_name   "Application Short Name",

       fa.basepath                 "Basepath"

  FROM fnd_application     fa,

       fnd_application_tl  fat

 WHERE fa.application_id = fat.application_id

   AND fat.language      = USERENV('LANG')

   -- AND fat.application_name = 'Payables'  -- <change it>

 ORDER BY fat.application_name;

 

ADOP CUTOVER (Reducing Time)

What can I do to reduce the time required for cutover?

It is important to distinguish between the time needed for the whole cutover phase, and the downtime period within the phase.

The actual downtime (during which users cannot log in) is significantly shorter than the whole phase.

To help reduce the overall time taken by cutover, you can do three things:

• Run the finalize phase explicitly, to obviate the need for cutover to do so.
• Shut down the concurrent managers before running cutover, to avoid having to wait for concurrent requests to complete. Alternatively, ensure no long-running concurrent jobs are submitted while a patching cycle is in progress.
• Ensure you are using the maximum number of parallel workers your system will support.

ADOP DOWNTIME MODE

What is downtime mode and when can it be used?

To optimize the process of upgrading to E-Business Suite Release 12.2, the AD Delta 5 Release Update Pack introduced downtime mode, which is used as follows:

• $ adop phase=apply patches=<patch_number> apply_mode=downtime

Downtime mode does not use an online patching cycle. 

The process of applying a patch in downtime mode completes more quickly than in online mode, but at the cost of increased system downtime. 

When applying Oracle E-Business Suite patches in this mode, adop will first confirm that the application tier services are down, and will then proceed to apply the patch to the run edition of the Oracle E-Business Suite database and file system.

Downtime mode is supported for:

• All patching (including post-upgrade patching) that forms part of the Release 12.2 upgrade process and is completed before the system is scaled up, the application tier services are started, and users log in to the upgraded system.
• Single-node development or test environments, where production support and high availability are not required.

Downtime mode allows the 12.2 upgrade process to be completed as quickly as possible.

 Once the upgrade is complete and users are online, all subsequent patching on a production system should use online mode, not downtime mode, unless the patch readme states otherwise. 

Several restrictions apply to the use of downtime mode:

• You cannot validate successful patch application before cutover to the updated code takes place.
• There is no capability to abort a failed patch and return to the existing run edition.
• Release 12.2 patches are not normally tested in downtime mode.
• Use of downtime mode in a multi-node application tier environment is not tested or supported.

Note: Downtime mode is only supported for production use where explicitly documented, or otherwise directed by Oracle.

BACKUPSET DETAILS

Backupset Details

set lines 220
set pages 1000
col backup_type for a4 heading "TYPE"
col controlfile_included heading "CF?"
col incremental_level heading "INCR LVL"
col pieces for 999 heading "PCS"
col elapsed_seconds heading "ELAPSED|SECONDS"
col device_type for a10 trunc heading "DEVICE|TYPE"
col compressed for a4 heading "ZIP?"
col output_mbytes for 9,999,999 heading "OUTPUT|MBYTES"
col input_file_scan_only for a4 heading "SCAN|ONLY"
select
  d.bs_key, d.backup_type, d.controlfile_included, d.incremental_level, d.pieces,
  to_char(d.start_time, 'yyyy-mm-dd hh24:mi:ss') start_time,
  to_char(d.completion_time, 'yyyy-mm-dd hh24:mi:ss') completion_time,
  d.elapsed_seconds, d.device_type, d.compressed, (d.output_bytes/1024/1024) output_mbytes, s.input_file_scan_only
from V$BACKUP_SET_DETAILS d
  join V$BACKUP_SET s on s.set_stamp = d.set_stamp and s.set_count = d.set_count
where session_recid = &SESSION_RECID
  and session_stamp = &SESSION_STAMP
order by d.start_time;