How to Synchronize FND_NODES, ADOP_VALID_NODES, and FND_OAM_CONTEXT_FILES in 12.2 When ADOP Phase=Prepare Fails with Error 'not able to detect any valid application tier nodes in ADOP_VALID_NODES table. ' (Doc ID 2064223.1)

Information in this document applies to any platform.

After a recent change or addition to the 12.2 E-Business Suite application tiers, adop phase=prepare fails with the following error:

ERROR:

$ adop phase=prepare

  Enter the APPS password:
  Enter the SYSTEM password:
  Enter the WLSADMIN password:

Validating credentials...

Initializing...
  Run Edition context : /appl/admin/PROD_prodapp1.xml
  Patch edition context: /appl/admin/PROD_prodapp1.xml
*******FATAL ERROR*******
PROGRAM : (/appl/ad/12.0.0/bin/adzdoptl.pl)
TIME : Fri Oct 2 22:06:45 2015
FUNCTION: ADOP::GlobalVars::_GetMandatoryArgs [ Level 1 ]
ERRORMSG: adop is not able to detect any valid application tier nodes in ADOP_VALID_NODES table. 

Ensure autoconfig is run on all nodes.

[STATEMENT] Please run adopscanlog utility, using the command
"adopscanlog -latest=yes"
to get the list of the log files along with snippet of the error message corresponding to each log file.
adop exiting with status = 255 (Fail)

CAUSE

There is a synchronization error between fnd_nodes, adop_valid_nodes, and fnd_oam_context_files.

Evidence
----------
The fnd_nodes table appears corrupt. It has a NULL domain entry for a valid server.

Example Data Supporting the Conclusion
---------------

[ fnd_nodes_OSS ]

NODE     PLAT    D C A F W    NODE_NAME    SERVER_ADDRESS     DOMAIN    WEBHOST                              VIRTUAL_IP        S
-------  -----    ----------   -------------   ------------------   ---------   ----------------------------   ---------------   --
12,121     226    N Y Y Y Y    NEW122APP1   ###.###.###.103   NULL         new122app1.somewhere.com    NULL                 Y

SOLUTION

 

Due to the method required for "cleaning out" / "re-synchronizing" the following tables, it is EXPECTED / REQUIRED that the Applications have been shutdown.
The only thing running should be the Database Tier.
Note:
A full backup should also be taken before any testing begins.

Test the following steps in a development instance, and then migrate accordingly once the desired result is confirmed:

1. Backup the fnd_oam_context_files, fnd_nodes, and adop_valid_nodes tables in the EBS env nodes:
      sqlplus applsys/pwd

      create table fnd_oam_context_files_bkp as select * from fnd_oam_context_files;
      create table fnd_nodes_bk as select * from fnd_nodes;
        create table adop_valid_nodes_bk as select * from adop_valid_nodes;

2. Truncate the following tables:

      truncate table fnd_oam_context_files;

      truncate table fnd_nodes;

      truncate table adop_valid_nodes;

3.  Run AutoConfig on the DB tier
        Confirm Autoconfig completes successfully

4.  Run Autoconfig on the run file system.
        Confirm Autoconfig completes successfully

5.  Run Autoconfig on the patch file system

Due to the method required for "cleaning out" / "re-synchronizing" the following tables, it is EXPECTED / REQUIRED that the Applications have been shutdown.
The only thing running should be the Database Tier.

Before running Autoconfig on the patch file system the ebs_login trigger MUST be disabled
After the successful completion of Autoconfig the ebs_login trigger MUST be re-enabled.

This needs to be done as the SYSTEM schema user.

a. Disable the ebs_login trigger using the following SQL.

    SQL> alter trigger ebs_logon disable;

•        At this time Run autoconfig with the patch env sourced.
•        Make sure Autoconfig completes ok

b. Enable the ebs_login trigger using the following SQL.

    SQL> alter trigger ebs_logon enable;

6.  After Autoconfig has been run successfully on all nodes, run the following two (2) queries in order to verify the tables have been correctly populated:

SQL>

        set pagesize 5
        set linesize 132
        col node_name format a15
        col server_id format a8
        col server_address format a15
        col platform_code format a4
        col webhost format a12
        col domain format a20
        col virtual_ip format a12

        select  node_id,  platform_code, support_db D, support_cp C, support_admin A,
                     support_forms F, support_web W, node_name, server_id,
                     server_address, domain, webhost, virtual_ip, status
          from fnd_nodes
         order by node_id;

SQL>

        set pagesize 5
        set linesize 132

        col NAME format A20
        col VERSION format A12
        col PATH format A110
        col STATUS format A10

        select NAME,VERSION,PATH, STATUS
          from FND_OAM_CONTEXT_FILES;