CRS-6706: Oracle Clusterware Release patch level ('nnn') does not match  Software patch level ('mmm') (Doc ID 1639285.1)

To Bottom

---

In this Document Purpose Details Case1: Discrepancy found in the list of applied patches on different nodes: Case2: All nodes have same patches but different patch level: References APPLIES TO: Oracle Database - Enterprise Edition - Version 12.1.0.1 and later Information in this document applies to any platform. PURPOSE This note lists causes/solutions for the following issue: Oracle 12c Grid Infrastructure fails to start after patching: CRS-6706: Oracle Clusterware Release patch level ('xxxxxxxx') does not match Software patch level ('xxxxxxxxx'). Oracle Clusterware cannot be started This can be encountered - for example - in cases where the post patch application instruction  to run 'rootcrs.pl -postpatch' is not followed. DETAILS   Case1: Discrepancy found in the list of applied patches on different nodes: Node 1: <GI_HOME>/bin/kfod op=patches --------------- List of Patches =============== 15994107 16495244   ====>> this one doesn't exist on the other node 17027533 17303297 <GI_HOME>/bin/kfod op=patchlvl ------------------- Current Patch level =================== 1581137133 Node 2: <GI_HOME>/bin/kfod op=patches --------------- List of Patches =============== 15994107 17027533 17077442         ====>> this one doesn't exist on the other node 17303297 <GI_HOME>/bin/kfod op=patchlvl ------------------- Current Patch level =================== 3400658801 crsd.trc ====== 2015-06-23 15:06:00.260076 :  OCRMAS:1797244672: th_hub_verify_master_pubdata: Shutdown CacheLocal. Patch Levels don't match.  Local Patch Level [3467666221] != Cache Writer Patch Level [4137922036] 2015-06-23 15:06:00.260101 :  OCRAPI:1797244672: procr_ctx_set_invalid: ctx is in state [6]. 2015-06-23 15:06:00.260103 :  OCRAPI:1797244672: procr_ctx_set_invalid: ctx set to invalid 2015-06-23 15:06:00.260193 :  OCRAPI:1797244672: procr_ctx_set_invalid: Aborting... In 12c, GI home must have identical patches for the clusterware to start unless during rolling patching. After applied the same patches on all nodes, GI started fine. Case2: All nodes have same patches but different patch level: Node1 $ /data01/12.1.0/grid/bin/kfod op=patches --------------- List of Patches =============== 17303297 18365185 19121550 $ /data01/12.1.0/grid/bin/kfod op=patchlvl ------------------- Current Patch level =================== 2061097728 Node2 $ /data01/12.1.0/grid/bin/kfod op=patches --------------- List of Patches =============== 17303297 18365185 19121550 $ /data01/12.1.0/grid/bin/kfod op=patchlvl ------------------- Current Patch level =================== 2061097728                                   ====>> same patch level However opatch lsinventory shows the different patch level: Patch level status of Cluster nodes : Patching Level Nodes -------------- ----- 1216323972 node2                          ====>> different patch level 2061097728 node1 For 12.1.0.1 on-wards: Execute"<GI_HOME>/crs/install/rootcrs.sh -patch" as root  user on the problematic node and the patch level should be corrected. For Oracle Grid Infrastructure for a standalone server Log in as root # <GI_HOME>/crs/install/roothas.sh -unlock # <GI_HOME>/crs/install/roothas.sh -patch

How to drop diskgroup giving ORA-15027

Problem

While removing all the contents of a database using ASM you may encounter error from acmca or sqlplus:

ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15027: active use of diskgroup “DATA” precludes its dismount

Cause

The reason behind is the parameter file residing in the ASM disk:

Solution

SQL> SELECT name, type, total_mb, free_mb, required_mirror_free_mb, usable_file_mb FROM V$ASM_DISKGROUP;

NAME                           TYPE     TOTAL_MB    FREE_MB REQUIRED_MIRROR_FREE_MB USABLE_FILE_MB
—————————— —— ———- ———- ———————– ————–
DATA                           EXTERN    7168000    7167858                       0        7167858
FRA                            EXTERN     512000     511222                       0         511222

SQL>

SQL> column HEADER_STATUS format a20
SQL> set pages 100
SQL> col PATH format a25

SQL> r
1* SELECT name, header_status, path FROM V$ASM_DISK

NAME                           HEADER_STATUS        PATH
—————————— ——————– ————————-
DATA_0004                      MEMBER               /dev/rdisk/disk38
DATA_0005                      MEMBER               /dev/rdisk/disk39
DATA_0006                      MEMBER               /dev/rdisk/disk40
DATA_0007                      MEMBER               /dev/rdisk/disk41
DATA_0008                      MEMBER               /dev/rdisk/disk42
DATA_0009                      MEMBER               /dev/rdisk/disk43
DATA_0010                      MEMBER               /dev/rdisk/disk44
DATA_0011                      MEMBER               /dev/rdisk/disk45
DATA_0012                      MEMBER               /dev/rdisk/disk46
DATA_0000                      MEMBER               /dev/rdisk/disk60
DATA_0001                      MEMBER               /dev/rdisk/disk61
DATA_0002                      MEMBER               /dev/rdisk/disk62
DATA_0003                      MEMBER               /dev/rdisk/disk63
DATA_0013                      MEMBER               /dev/rdisk/disk64
FRA_0000                       MEMBER               /dev/rdisk/disk65

15 rows selected.

SQL>
SQL>
SQL> DROP DISKGROUP data INCLUDING CONTENTS;
DROP DISKGROUP data INCLUDING CONTENTS
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15027: active use of diskgroup “DATA” precludes its dismount

SQL>
SQL> DROP DISKGROUP data FORCE INCLUDING CONTENTS;
DROP DISKGROUP data FORCE INCLUDING CONTENTS
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15230: diskgroup 'DATA' does not require the FORCE option

SQL> create pfile='/tmp/init.ora' from spfile;

File created.

SQL> shutdown immediate;
ASM diskgroups dismounted
ASM instance shutdown
SQL> startup pfile=’/tmp/init.ora’;
ASM instance started

Total System Global Area  283930624 bytes
Fixed Size                  2169104 bytes
Variable Size             256595696 bytes
ASM Cache                  25165824 bytes
ORA-15110: no diskgroups mounted

SQL>  DROP DISKGROUP data INCLUDING CONTENTS;
DROP DISKGROUP data INCLUDING CONTENTS
*
ERROR at line 1:
ORA-15039: diskgroup not dropped
ORA-15001: diskgroup “DATA” does not exist or is not mounted

SQL> DROP DISKGROUP data FORCE INCLUDING CONTENTS;

Diskgroup dropped.

SQL>

APPLSYS password using ALTER command in database directly

FNDCPASS SCENARIOS

--------------------------------

 There can be situation where users has

Updated APPLSYS password using ALTER command in database directly and also you dont have backup of those tables.

Under such situation,

 it is very difficult to recover the application and make it working. Still following methodology is proposed which might help you to restore the password back and make your application work fine.

For this to work you should have some other application (may be debug or UAT) which is having the same passwords or default passwords for schemas.

If you have such application the following the below steps in the application which is affected by password mismatch.

This method is for resetting apps and applsys passwords. Below are the SQL statements that will help you

reset

-the APPS and APPLSYS passwords to APPS,

-the APPLSYSPUB password to PUB, and

-the SYSADMIN password to SYSADMIN.

WARNING: This procedure will cause all user passwords to become invalid. ALL users passwords will need to be reset through the sysadmin responsibility.

Step 1) Reset the Oracle User IDs

Open a SQL*Plus as SYSTEM and reset the passwords for the APPS, APPLSYS, and the APPLSYSPUB Oracle user ID:

ALTER USER apps IDENTIFIED BY apps;

ALTER USER applsys IDENTIFIED BY apps;

ALTER USER applsyspub IDENTIFIED BY pub;

Step 2) Backup the FND_ORACLE_USERID and FND_USER tables (even though these tables are right now corrupted, do take a backup. You can restore the same when ever you want).

Open a SQL*Plus session as APPLSYS and backup the tables:

create table FND_ORACLE_USERID_BAK as (select * from FND_ORACLE_USERID);

create table FND_USER_BAK as (select * from FND_USER);

Step 3) Reset the APPS and APPLSYS application encrypted passwords

Open a SQL*Plus session as APPLSYS and update the FND_ORACLE_USERID table.

update FND_ORACLE_USERID

set ENCRYPTED_ORACLE_PASSWORD = ‘ZGA34EA20B5C4C9726CC95AA9D49EA4DBA8EDB705CB7673E645EED570D5447161491D78D444554655B87486EF537ED9843C8′

where ORACLE_USERNAME in (‘APPS’, ‘APPLSYS’);

commit;

This encrypted string we are updating is the default encrypted string for apps. So if your application is having apps password the encrypted string will look like this. We are updating this encrypted string here directly.

Verify the table update:

select ENCRYPTED_ORACLE_PASSWORD

from FND_ORACLE_USERID

where ORACLE_USERNAME IN (‘APPS’, ‘APPLSYS’);

Step 4) Reset the APPLSYSPUB application encrypted password

Open a SQL*Plus session as APPLSYS and update the FND_ORACLE_USERID table.

update FND_ORACLE_USERID

set ENCRYPTED_ORACLE_PASSWORD = ‘ZG31EC3DD2BD7FB8AD2628CE87DDDF148C1D2F248BE88BE987FDF82830228A88EF44BC78BC7A9FAD4BFB8F09DAD49DF7280E’

where ORACLE_USERNAME = (‘APPLSYSPUB’);

commit;

The above encrypted string is the encrypted string for password pub. If your applsyspub password is pub then the encrypted string in FND_ORACLE_USERID will look like this.

Verify the table update:

select ENCRYPTED_ORACLE_PASSWORD

from FND_ORACLE_USERID

where ORACLE_USERNAME = ‘APPLSYSPUB’;

Once these updates are done, try your luck by running FNDCPASS and it should work fine.

Hope this help !!!

References

Metalink note ID 445153.1

Metalink note ID 429244.1

FNDCPASS SCENARIOS

Issue with APPLSYS and APPS password

Scenario 1:

As you know that apps and applsys password should be in sync and should be changed using FNDCPASS.

There can be situation where a novice user changes applsys password from the backend database. 

In that case when you try to start the services it will show following error

APP-FND-01496: Cannot access application ORACLE password
Cause: Application Object Library was unable access your ORACLE password.

You can even reproduce this issue (ofcourse after taking the backup of FND_USER and FND_ORACLE_USERID table) using the following steps

1. Use the ALTER USER command to change the APPLSYS password

2. Try to run the adstrall.sh script to start Apps services.

3. You will get an error “Cannot complete applications logon. You may have entered an invalid applications password, or there may have been a database connect error.”

4. Then try FNDCPASS to fix password and you will get the error the APP-FND-01496 error.

If this situation happens then you cannot access the application. Infact the services even wont start.

Resolution to such problem is to rollback the 2 tables FND_USER and FND_ORACLE_USERID. 

Once you rollback the tables, apps and applsys passwords will be in sync and password will be older one. 

You can then run FNDCPASS and change the password.

Scenario 2:

Some times when you run FNDCPASS, you get following error

APP-FND-01502: Cannot encrypt application ORACLE password
Cause: Application Object Library was unable encrypt your ORACLE password.
Action: Contact your support representative. (ORACLEUSER=APPS_SERV)

The error comes because the table fnd_oracle_userid contain rows for schemas that does not exist. Those rows must be deleted from the table.

Use the following query to get the details of the schema that doest not exists

select * from fnd_oracle_userid
where oracle_username not in
(select username from all_users);

The rows returned by this query can be deleted from FND_ORACLE_USERID table. This will resolve this issue.

FNDCPASS

FNDCPASS

In Oracle Application R12, 

we have an FND functionality for changing the passwords for either application user, or product schema password or most important – 

the “APPS” password. The FND binary which will help us is doing these things is FNDCPASS.

This is present in $FND_TOP/bin directory.

Using FNDCPASS

Below is the usage for FNDCPASS

-bash-2.05b$ FNDCPASS

FNDCPASS logon 0 Y system/password mode username new_password
where logon is username/password[@connect]

system/password is password of the system account of that database
mode is SYSTEM/USER/ORACLE
username is the username where you want to change its password
new_password is the new password in unencrypted format

example 

FNDCPASS apps/apps 0 Y system/manager SYSTEM APPLSYS WELCOME

FNDCPASS apps/apps 0 Y system/manager ORACLE GL GL1

FNDCPASS apps/apps 0 Y system/manager USER VISION WELCOME

You can just type FNDCPASS and press enter, it will give you these details.

STEP 1 

---------

Usage A

FNDCPASS apps/apps 0 Y system/manager SYSTEM APPLSYS WELCOME

is for changing the password for apps and applsys. These are the database schema users (most important for application to work).

 Password for both these users should be in synch. You can change the password of these users using this command. Note that this is the only way to change the password for apps and applsys. Please do not try any other method for changing apps and applsys password. Oracle recomends using FNDCPASS only to change apps and applsys password. Also note that using this command will change the password for both apps and applsys.

Following activities will take place

(1) applsys validation. (make sure APPLSYS name is correct)
(2) re-encrypt all password in FND_USER
(3) re-encrypt all password in FND_ORACLE_USERID
(4) update applsys’s password in FND_ORACLE_USERID table.
(5) Update apps password in FND_ORACLE_USERID table.

Also changes are made in DBA_USERS table.

Usage B

FNDCPASS apps/apps 0 Y system/manager ORACLE GL GL1

is for changing password for any other product schema like MSC, GL etc.
Following activities will take place

(1) update GL’s password in FND_ORACLE_USERID table. The new password is re-encrypted with the current applsys password.

If GL does not exists, step (2) below does not happen. Message for invalid oracle user is written in the log file.

(2) alter user to change GL’s password.

USAGE C

FNDCPASS apps/apps 0 Y system/manager USER   VISION  WELCOME

is for changing the application level passwords like sysadmin etc used for logging into application.

Following activities will take place

(1) update VISION’s password in FND_USER table. 

The new password is re-encrypted with the current applsys password.

If VISION does not exist, message for invalid application user is written in the log file.
No products affected by the patch

When you run FNDCPASS command it will check the integrity of all schema password in the application. 

If any of the password is corrupt then this will through and error and will not change the password.

The tables that it uses is FND_USER and FND_ORACLE_USERID. All the application passwords and schema passwords are stored in these two tables. Ofcourse DBA_USERS  will have the schema users and password stored as well.

When we run FNDCPASS it will update all the above 3 tables.

Best practices for using FNDCPASS

Before using FNDCPASS and changing the passwords from default to some thing else, always follow the following best practices.

1) Always, Always, Always keep the back of tables FND_USER and FND_ORACLE_USERID. 

You can take back of these tables using CREATE TABLE — AS SELECT * FROM —.
You must have backup of these tables before running FNDCPASS.

 In case if FNDCPASS fails then it might corrupt the passwords of your application and worst can happen that the application wont come up. So always be cautions about this command.

2) If possible also keep an export dump of these two tables.

3) verify each arguement you are providing to FNDCPASS. Like verify that apps and system passwords you are providing is correct.

4) Never update apps, applsys or any schema password directly from database using the alter command. Always use FNDCPASS. System password can be set directly using ALTER command in database.

Gather Schema Stats (Oracle Apps)

Gather Schema Statistics (Oracle Applications)

Gather Schema Statistics program generates statistics that quantify the data distribution and storage characteristics of tables, columns, indexes, and partitions.

The cost-based optimization (CBO) uses these statistics to calculate the selectivity of prediction and to estimate the cost of each execution plan.

We run Gather Schema Statistics under the following circumstances:

1. There is a Significant change in data in either content or volume.
2. After importing data.
3. End-users notice deterioration in performance in routine day-to-day business transactions
4. When running concurrent programs is taking longer than usual.
5. Run on a regular basis (weekly at a minimum) and any time after application of patch, conversion, etc.
6. By default GSS will not run on Custom schema

Estimate Percentage / Modification threshold defines the percentage which should be used to initiate gather stats for those objects which have actually changed beyond the threshold.
The default is 10% (any table which has changed via DML more than 10%, stats will be collected, otherwise it will be skipped).
ow to run Gather Schema Statistics concurrent program:

1.     Log on to Oracle Applications with

Responsibility = System Administrator

2. Submit Request Window
Navigate to: Concurrent > Requests

3. Query for the Gather Schema Statistics

4. Enter the appropriate parameters. This can be run for specific schemas by specifying the schema name or entering ‘ALL’ to gather statistics for every schema in the database

5. Submit the Gather Schema Statistics program

Parameters :
——————
Schema Name: Schema for which statistics are to be gathered. Specify ALL for all Oracle Applications schemas
Percent: The sampling percentage. If left blank, the default value of 10 is used. The valid range is from 0 to 100
Degree: The degree of parallelism to be used for gathering statistics. If a Degree is not provided, it defaults to the minimum of parallel_max_servers and cpu_count.
Backup Flag: NOBACKUP is used, then the GATHER_SCHEMA_STATS procedure will not backup the current statistics. This way the GATHER_SCHEMA_STATS procedure will run faster.
Restart Request ID: In the case where the Gather Schema Statistics run fails due to whatever reasons, the concurrent request can be re-submitted and it will pick up where the failed run left off, if you provide the concurrent request_id of the failed run.
History Mode: Last Run – History records for each object are maintained only for the last gather statistics run. Each subsequent run will overwrite the previous history record for the object. This is the default behaviour
Gather Options: All tables and indexes of the schema schema name are selected for stats gathering. This is the default
Modifications Threshold: Applicable only to GATHER AUTO and LIST AUTO Options
Invalidate Dependent Cursors: This flag indicates whether cursors dependent on the table being analysed should be invalidated or not. By default, dependent cursors are invalidated.

Gather statistics in Oracle apps from backend

Use the following command to gather schema statistics:

exec fnd_stats.gather_schema_statistics('ONT') < For a specific schema >
exec fnd_stats.gather_schema_statistics('ALL') < For all schemas > 

Changing Oracle Apps user password for Oracle E-Business Suite R12 Application.

We can follow the below steps  for changing or resetting Oracle Application user password:

Step 1: Login with 'sysadmin' user into Oracle E-Business Suite frontend page.

Step 2: In the Oracle Applications Home Page go to Navigator section, select the User Managementresponsibility

Step 3: Click the User Management Responsibility, under User Management, click Users

Step 4: In the Oracle User Management of page, we find lot of search options available for use, 

we can use %also to find the name of the Oracle Application user. 
If you now the employee id than you can directly enter here.

Let us say we want to change password for Employee id 'A0067' ,than I will enter 'A0067' 
in the place

 where User Name is present

Step 5: Search the user
After entering the User Name, click Go

We can see the details as specified below:
Last Name First Name Email User Name Status Create User Reset Password Update

Step 6: Reset the password
Click the Reset Password tab and Choose Enter Manually option of resetting the password.

 Now, we can enter the password and confirm the password and Click Submit.

Step 7: Verify and inform the UserAfter resetting or changing the password, 
we can verify  by logging from the Oracle E-Business Suite front end page.

Let us say we have reset the password of 'A0067' to welcome123,
than we can login successfully and inform the user for the change.

HOW TO RESET SYSADMIN PASSWORD

How to Reset only the SYSADMIN Password

PROBLEM DESCRIPTION 
------------------- 
You have forgotten the SYSADMIN applications password and cannot login.
 You know the apps and applsys passwords but does not want to follow the steps to 
reset all passwords since resetting applsys would require re-registering all applications user passwords. 

How can you reset only the SYSADMIN password? 

SOLUTION DESCRIPTION 
-------------------- 
The applsys password provides the basis for all user passwords 
so resetting applsys will require re-registering all user passwords. 
If there are any other users with System Administrator privileges, then they can reset 
the SYSADMIN user's password from within applications. 
Otherwise SYSADMIN can be reset without affecting other users as follows: 
1. Log in to SQL*Plus as applsys 
2. Backup FND_USER and FND_ORACLE_USERID tables using 
create table FND_USER_BAK as select * from FND_USER; 
create table FND_ORACLE_USERID_BAK as select * from FND_ORACLE_USERID; 
3. update FND_USER 
set ENCRYPTED_FOUNDATION_PASSWORD 
='2DF3E509EB6A33F9607959C0976E25D997166FAB694ACDDAE466414791A44411', 
ENCRYPTED_USER_PASSWORD
='D2FCA9810D86BCA9BE944D3E2E7A4A9E6CDF89AAD633179B701774083F907C13' 
where user_name='SYSADMIN'; 
sql>commit; 

The SYSADMIN password is now set to WELCOME.

Reset the AdminServer Password in WebLogic 11g and 12c

If you forget the AdminServer password for your WebLogic 11g domain, you can reset it from the command line using the following process.

• Set up the following environment variables. They are not necessary for the process itself, but will help you navigate. In this case my domain is called "ClassicDomain". Remember to change the value to match your domain.

  export MW_HOME=/u01/app/oracle/middleware
  export DOMAIN_HOME=$MW_HOME/user_projects/domains/ClassicDomain

• Shut down the WebLogic domain.

  $ $DOMAIN_HOME/bin/stopWebLogic.sh

• Rename the data folder.

  $ mv $DOMAIN_HOME/servers/AdminServer/data $DOMAIN_HOME/servers/AdminServer/data-old

• Set the environment variables.

  $ . $DOMAIN_HOME/bin/setDomainEnv.sh

• Reset the password using the following command. Remember to substitute the appropriate username and password.

  $ cd $DOMAIN_HOME/security
  $ java weblogic.security.utils.AdminAccount <username> <password> .

• Update the "$DOMAIN_HOME/servers/AdminServer/security/boot.properties" file with the new username and password. The file format is shown below.

  username=<username>
  password=<password>

• Start the WebLogic domain.

  $ $DOMAIN_HOME/bin/startWebLogic.sh

For more information see:

• How to Change the WebLogic Server Administrator Password [ID 1082299.1]