Apply Patch 22191577 latest GI PSU to RAC and DB homes using Opatch auto or manual steps

Patch 22191577: GRID INFRASTRUCTURE PATCH SET UPDATE 11.2.0.4.160119 (JAN2016)

Unzip the patch 22191577
Unzip latest Opatch Version in oracle home and crs home

Once this completed successfully make sure OPatch folder on both CRS and ORACLE HOME are owned by ORACLE:DBA

Opatch Version check: ./opatch version

Opatch Inventory check: ./opatch lsinventory

Opatch conflict checking: ./opatch prereq CheckConflictAmongPatchesWithDetail -phBaseDir /u02/patch/22191577

After this create OCM response file. You can create either in grid opatch folder or db home opatch folder This will be used if you are using opatch auto

In my case I created the file in both

/u01/app/oracle/product/11.2.0/dbhome_2/OPatch/ocm/bin/emocmrsp
/u01/app/11204/OPatch/ocm/bin/emocmrsp

Skip the email ID and select Y. A file name ocm.rsp will be created.



We have main patch folder 22191577

In that folder we have 3 patch folders

21948348 - DB and Grid Home
21948347 - DB and Grid Home
21948355 - Only Grid Home

You can find which patch is for which home in Patch read me document

In my case

ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_2/
GRID_HOME=/u01/app/11204/


************************************************************************

If using Opatch Auto user must be root. You dont need to shutdown anything OPatch does it automatically

opatch auto
Root user
#./opatch auto /u02/patch/22191577 -ocmrf /u01/app/oracle/product/11.2.0/dbhome_2/OPatch/ocm/bin/ocm.rsp

************************************************************************

If you choose to do it manually you need to shutdown instance and CRS manually

Shutdown instances on node1 as oracle user
srvctl stop instance -d prod -i prod1

Shutdown crs on node1 as root user
crsctl stop crs

Root user
cd $GRID_HOME/crs/install
./rootcrs.pl -unlock   (If you don't execute this you get some permission errors related to patch storage and it complains not able to read and write files)


Oracle user
Grid home
/u01/app/11204/OPatch/opatch napply -oh /u01/app/11204 -local /u02/patch/22191577/21948348

/u01/app/11204/OPatch/opatch napply -oh /u01/app/11204 -local /u02/patch/22191577/21948355

/u01/app/11204/OPatch/opatch apply -oh /u01/app/11204 -local /u02/patch/22191577/21948347
 
Oracle user
DB home
/u02/patch/22191577/21948348/custom/server/21948348/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/product/11.2.0/dbhome_2

/u01/app/oracle/product/11.2.0/dbhome_2/OPatch/opatch napply -oh /u01/app/oracle/product/11.2.0/dbhome_2/. -local /u02/patch/22191577/21948348/custom/server/21948348

/u01/app/oracle/product/11.2.0/dbhome_2/OPatch/opatch apply -oh /u01/app/oracle/product/11.2.0/dbhome_2/ -local /u02/patch/22191577/21948347 

/u02/patch/22191577/21948348/custom/server/21948348/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/product/11.2.0/dbhome_2

After completion
Root user
# /u01/app/oracle/product/11.2.0/dbhome_2/rdbms/install/rootadd_rdbms.sh
# /u01/app/11204/crs/install/rootcrs.pl -patch


Once you are done with patching first node1
Start the instance
srvctl start instance -d prod -i prod1

***********************************************************************

Once 1st instance is started repeat the same steps on node2

***********************************************************************

After the two instances started up

Refer the read me file to see post patch instructions. In this psu i have to do the following.

We should do this only in 1 instance.

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT

After the above command is complete it will show if the log file location. You can grep it for errors.

You can also run @utlrp.sql and compile invalid objects in the database.

************************************************************************


Rollback instructions
If using opatch auto

# opatch auto /u02/patch/22191577/ -rollback -ocmrf /u01/app/oracle/product/11.2.0/dbhome_2/OPatch/ocm/bin/ocm.rsp

If doing manually..shutdown instance and CRS manually

Shutdown instances on node1 as oracle user
srvctl stop instance -d prod -i prod1

Shutdown crs on node1 as root user
crsctl stop crs

Root user
cd $GRID_HOME/crs/install
./rootcrs.pl -unlock   (If you don't execute this you get some permission errors related to patch storage and it complains not able to read and write files)


Oracle user
Grid home
/u01/app/11204/OPatch/opatch rollback -local -id 21948348 -oh /u01/app/11204

/u01/app/11204/OPatch/opatch rollback -local -id 21948355 -oh /u01/app/11204 
 
/u01/app/11204/OPatch/opatch rollback -local -id 21948347 -oh /u01/app/11204
 
Oracle user
DB home
/u02/patch/22191577/21948348/custom/server/21948348/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/product/11.2.0/dbhome_2

/u01/app/oracle/product/11.2.0/dbhome_2/OPatch/opatch rollback -local -id 21948348 -oh /u01/app/oracle/product/11.2.0/dbhome_2

/u01/app/oracle/product/11.2.0/dbhome_2/OPatch/opatch rollback -local -id 21948347  -oh /u01/app/oracle/product/11.2.0/dbhome_2

/u02/patch/22191577/21948348/custom/server/21948348/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/product/11.2.0/dbhome_2

After completion
Root user
# /u01/app/oracle/product/11.2.0/dbhome_2/rdbms/install/rootadd_rdbms.sh
# /u01/app/11204/crs/install/rootcrs.pl -patch


Once you are done with patching first node1
Start the instance
srvctl start instance -d prod -i prod1

Repeat same steps on Node2

After both instances are UP. Run the following only on one instance

cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT

Also run @utlrp.sql

After everything is completed you can verify if patches are applied properly for both homes

Go to Opatch folder

cd /u01/app/oracle/product/11.2.0/dbhome_2/OPatch

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_2/

./opatch lsinventory

The above lists patches to oracle home.

Now change ORACLE_HOME path to grid home

export ORACLE_HOME=/u01/app/11204

./opatch lsinventory

Now it will list patches applied to grid home.

To further check in DB. Run the following sql statements

select comments, version, bundle_series
from sys.registry$history
where bundle_series = 'PSU'
order by action_time;

select comments Last_patch from dba_registry_history order by action_time desc;