Patches

Patch :

Overlay PSU :
Previous PSU becomes a baseline and all future PSUs are released as “overlay PSUs”.The overlay PSU is cumulative down to the base PSU
i.e by installing 10.2.0.4.4 it gets all of the fixes in 10.2.0.4.3 and below, then by installing 10.2.0.4.10 all fixes in 10.2.0.4.9 down to 10.2.0.4.5 are also included.

Overlay Patch :

PSU is a prerequisite to apply  the one-off patch. i.e It cannot be applied if the PSU is not first applied. PSU is a prerequisite of the one-off patch.

http://edba.blogspot.in/2017/12/rac-rolling-psu-patch.html

http://oracle-help.com/articles/applying-psu-patch-12c-rac-solaris/

http://oracle-help.com/oracle-12c/oracle-12cr2/applying-first-ru-oracle-grid-infrastructure-12-2-0-1/

Starting with 11g Release 2 (11.2.0.2), Oracle database patch sets are full installations of the Oracle Database software i.e they do  not replaced files in existing installation but are itself complete installation of the whole software. So even if you want to upgrade, you can install the patch set in the new ORACLE_HOME (Out-of-place upgrade) instead of disturbing the old software binaries. However, you must ensure that you have sufficient free disk space.

  • Patches that do not change the data dictionary qualify for rolling upgrades.
  • To check if patch is rolling : $ORACLE_HOME/OPatch/opatch query -is_rolling_patch | grep rolling
  • If oracle binaries are kept in shared file system like OCFS, rolling upgrades may not be possible.
  • The CRS Bundles, CPU and PSU patches are rolling patches.
  • RDBMS Bundles are only rolling upgradable in case a previous CPU or BUNDLE was installed once, otherwise view recompilation requires a downtime
  • The ASM RDBMS Bundle installations are always rolling upgradable.
  • One off and cumulative patches may or may not be rolling upgrades.


CVE explantion : http://www.first.org/cvss/cvss-guide.html

https://docs.oracle.com/cd/B16240_01/doc/em.102/e15294/options.htm
https://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home/Patching_Grid_and_Database_Software
http://samiora.blogspot.com/2012/07/applying-rolling-patch-on-rac-11g-r2.html



Patch Sets             :  from 11.2.0.1.0 to 11.2.0.2.0 ( out of place full installation)
Patch Set  Update :  from 11.2.0.1.0 to 11.2.0.2.2  (use opatch).Contain CPU as well
Interim Patch        :  specific patch for  bug fix
Upgrade release    :  from 11.2.0.1 to 12.0.1


Patchsets for Grid Infrastructure and RDBMS are delivered separately from 11gR2 onwards.They are are now full releases ie. install without first installing base release.
Grid Infrastructure  can be applied in rolling fashion and not the database patchset.

A patch can be a GI patch  or Database patch .
1.GI patch (interim, bundle or PSU) applies to all homes (GI and Database),
2.Database patch could be  applicable to GI/ASM home as well if the fix applies to ASM.


PSU is applied on top of a given patchset release.It can be independently released as GI PSU or DB PSU.

1.GI PSUs(11gR2) contain both GI PSU and Database PSU.It's referred as  CRS PSUs in pre-11.2 release.GI PSUs are cumulative i.e no need to install the lower version first.11.2.0.2.2 GI PSU can be applied on 11.2.0.2.0.GRID PSU can/should be applied to both GRID and RDBMS home.

2. Database PSU may require a lower PSU to be applied first.Database PSUs always contain the CPU.


CRS patch comes in two parts:  Install first the CRS part, then the ASM part, then the RDBMS part
1. one part must be applied to the CRS_HOME
2. The other part must be applied to the RDBMS home if it is the same version(an ASM_HOME is considered as being a RDBMS_HOME)
3. RDBMS part of CRS patches and bundles can not(and must not) be applied to the RDBMS home if the version is lower than the CRS

prerootpatch.sh  - Script run on $CRS_HOME only .unlocks CRS home
prepatch.sh         - Script run on $CRS_HOME and $RDBMS_HOME.
postpatch.sh       - Script run on $CRS_HOME and $RDBMS_HOME.
postrootpatch.sh - Copies the CRS init script to init.d  , Locks the CRS home , Starts the CRS stack


Patches :

opatch

check ssh :  cluvfy comp admprv -n all -o user_equiv –verbose

Central inventory pointed by /etc/oraInst.loc can be recreated. Local inventory $ORACLE_HOME/inventory can’t


<Path_to_OPatch>/opatch query -all
oh
invPtrLoc
local : patch the local node only and update the inventory of the local node
                      If Auto option used ,  -local mode ensure it does not attempt an automatic rolling install across the entire cluster.
local_node : Specifies the local node
report : Print without executing it.
silent : defaults any answers to "yes."
minimize_downtime : Specifies the order of nodes to be patched
prereq         :  10.2
util              :  11.1
auto            :  11.2

opatch napply -skip_subset -skip_duplicate -local -oh $ORACLE_HOME


If old existing patch is superset   : new patch is not installed
If old existing patch is subset   : Old patch is rolled back and new patch is installed.
If conflict scenario                       : User can decide which patch to keep and which patch to                                                                         rollback.A patch conflict occurs when multiple fixes in different patches touch the same files but  have not been tested together as a single entity.


No  non-security fixes in CPUs.
In N-apply concept, a cpu is divided into one or more molecules.If molecule is subset , skip it and skip if it's duplicate.

Patch Set Updates can be applied on the base release version or on any earlier Patch Set Update.


$ORACLE_HOME/.patch_storage -- Old copies of the files

opatch apply -report -- actions OPatch will perform without actually executing them

#opatch napply - -phBaseFile  /tmp/patches/patches.txt -- multiple patches location in .txt file

OPatchauto apply /scratch/patches/12345678 -binary -target_type cluster -oh <GI_HOME>  -- applies a single patch on a selected Oracle home






http://askdba.org/weblog/oracle11g/11g-rac/applying-psu-11-2-0-3-5-to-grid-infrastructure-and-db-home/

https://startupforce.wordpress.com/tag/rac-patch/



10gR2 Interim http://abcdba.com/abcdbarachowtoapplyrollingopatch


11gR1  PSU :  http://select-star-from.blogspot.in/2013/10/patch-13621679-1110711-patch-set-update.html


11gR2 PSU  :  https://en.wikibooks.org/wiki/RAC_Attack_-_Oracle_Cluster_Database_at_Home/Patching_Grid_and_Database_Software (ocm response file)


11gR2 Bundle : http://samiora.blogspot.in/2012/07/applying-rolling-patch-on-rac-11g-r2.html



Patching Methods


1. Downtime OPatch  : All instances are down , patch applied in rolling )

If the the variable online_rac_installable is set to true in <Patch No>/etc/config/inventory file then the patch is a rolling patch.


On each node, shutdown the instance
Apply patch from the primary node  ( else ACTION: Restart the patch process on the primary node. )
It will be  patching in rolling mode. Select each node at a time.
On each node, start the instance.
crs_stat -t

10g : http://abcdba.com/abcdbarachowtoapplymindowntimeopatch ( downtime and rolling)


2. No Downtime  or Minimum downtime Opatch : 

    On the primary node in the cluster ,   $HOME/DBA/patches/6239052/opatch apply
    First it will patch local node1 , then it will ask for node2 readiness, patch node2 and so on for other nodes i.e  in the rolling mode
    All nodes will  be patched from the primary node. So user equivalence should be working i.e ssh


    Shutdown instance 1  and patch it. start instance 1
    Shutdown instance 2  and patch it. start instance 2
    Shutdown instance 3 ,and patch it. start instance 3


Shutdown instance 1  and patch it.
    Shutdown instance 2  and patch it.
    Shutdown instance 3 , start instance 1 and 2  ,  and then continue with  patching  3 ( from shutdown of instance 3 to start of  instance 1 and 2 there's downtime)
Note :  All nodes will  be patched from the primary node. So user equivalence should be working i.e ssh


3. No Downtime Opatch : One instance down at a time , patch applied one one node at a time 

    https://www.virtualobjectives.com.au/oracle/rolling-patch-rac.htm (no downtime and rolling)
    http://abcdba.com/abcdbarachowtoapplyrollingopatch                    (no downtime and rolling)
 
    STEPS 10g:
oracle:> opatch lsinventory -detail -oh /u01/crs/oracle/product/10/crs
oracle:> opatch lsinventory -detail -oh /u01/app/oracle/product/db10.2.0/


% $ORACLE_HOME/bin/srvctl stop instance -d dbname -i instance_name
% $ORACLE_HOME/bin/srvctl stop asm -n <node_name>
% $ORACLE_HOME/bin/srvctl stop nodeapps -n <node_name>
root # $CRS_HOME/bin/crsctl stop crs



root #   <patch directory>/123456/custom/scripts/prerootpatch.sh -crshome /u01/crs/oracle/product/10/crs -crsuser oracle

oracle:> <patch directory>/123456/custom/scripts/prepatch.sh -crshome /u01/crs/oracle/product/10/crs
oracle:> <patch directory>/123456/custom/server/123456/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/product/db10.2.0


oracle:> opatch apply -local -oh /u01/crs/oracle/product/10/crs
oracle:> opatch apply -local -oh /u01/app/oracle/product/db10.2.0

oracle:> <patch directory>/123456/custom/server/123456/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/product/db10.2.0
root #   <patch directory>/123456/custom/scripts/postrootpatch.sh -crshome /u01/crs/oracle/product/10/crs



Patch Auto Apply(11gR2)


$ORACLE_HOME/OPatch/opatch lsinventory -detail -oh $ORACLE_HOME

$ORACLE_HOME/bin/emctl stop dbconsole/agent

[grid]# $ORACLE_HOME/OPatch/ocm/bin/emocmrsp   -no_banner

[grid]# $ORACLE_HOME/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -phBaseDir /home/oracle/psupatch/

[root]# /oragrid/11.2.0.3/grid/OPatch/opatch auto /home/oracle/psupatch/ -ocmrf  /home/oracle/ocm.rsp  -oh  /oragrid/11.2.0.3/grid/

@catbundle.sql psu apply

https://learnwithme11g.wordpress.com/2011/05/05/applying-psu-patch-11-2-0-1-2-to-a-two-node-rac/
http://select-star-from.blogspot.com/2013/10/patch-13621679-1110711-patch-set-update.html











PSU patch on Oracle 11gr2 (Single Instance)

Stop crs 

#crsctl stop resources

oracle# <$ORACLE_HOME>/bin/srvctl stop home -o <ORACLE_HOME> -s <status file location> -n <node name>

Pre CRs patch :

root# /oradata/app/oracle/product/11.2.0.3/grid/crs/install/rootcrs.pl -unlock
root# /oradata/app/oracle/product/11.2.0.3/grid/crs/install/roothas.pl –unlock

# <GI_HOME>/crs/install/rootcrs.pl -unlock
# <GI_HOME>/crs/install/roothas.pl -unlock

CRS Patch :

grid# $/oradata/app/oracle/product/11.2.0.3/grid/OPatch/opatch napply -oh /oradata/app/oracle/product/11.2.0.3/grid –local /oradata/PSUOCT2013/17076717       
grid# $/oradata/app/oracle/product/11.2.0.3/grid/OPatch/opatch  apply -oh /oradata/app/oracle/product/11.2.0.3/grid -local /oradata/PSUOCT2013/16902043 

$ <GI_HOME>/OPatch/opatch napply -oh <GI_HOME> -local <UNZIPPED_PATCH_LOCATION>/<GI_components_number>

$ <GI_HOME>/OPatch/opatch  apply -oh <GI_HOME> -local<UNZIPPED_PATCH_LOCATION>/<DB_PSU_number>

Pre DB Patch :

oracle# $/oradata/PSUOCT2013/17076717/custom/server/17076717/custom/scripts/prepatch.sh -dbhome /oradata/app/oracle/product/11.2.0.3/db

$<UNZIPPED_PATCH_LOCATION>/<GI_components_number>/custom/server/<GI_components_number>/custom/scripts/prepatch.sh -dbhome <ORACLE_HOME>

DB Patch :

$/oradata/app/oracle/product/11.2.0.3/db/OPatch/opatch napply –oh /oradata/app/oracle/product/11.2.0.3/db –local /oradata/PSUOCT2013/17076717/custom/server/17076717
$/oradata/app/oracle/product/11.2.0.3/db/OPatch/opatch  apply -oh /oradata/app/oracle/product/11.2.0.3/db -local /oradata/PSUOCT2013/16902043

$ <ORACLE_HOME>/OPatch/opatch napply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/<GI_components_number>/custom/server/<GI_components_number>

$ <ORACLE_HOME>/OPatch/opatch  apply -oh <ORACLE_HOME> -local <UNZIPPED_PATCH_LOCATION>/<DB_PSU_number> 

Post DB Patch :

oracle# /oradata/PSUOCT2013/17076717/custom/server/17076717/custom/scripts/postpatch.sh -dbhome /oradata/app/oracle/product/11.2.0.3/db

<UNZIPPED_PATCH_LOCATION>/<GI_components_number>/custom/server/<GI_components_number>/custom/scripts/postpatch.sh -dbhome <ORACLE_HOME>

Post CRs patch :

root# <GI_HOME>/rdbms/install/rootadd_rdbms.sh
root# <GI_HOME>/crs/install/rootcrs.pl -patch

root# /oradata/app/oracle/product/11.2.0.3/grid/crs/install/roothas.pl -patch

Start crs

#crsctl start resource -all
oracle# <$ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location> -n <node name>

oracle# <$ORACLE_HOME>/bin/srvctl start home -o <ORACLE_HOME> -s <status file location>


SQL> STARTUP

SQL> @catbundle.sql psu apply

http://sandeepnandhadba.blogspot.in/2014/05/step-by-step-procedure-for-applying.html
http://sandeepnandhadba.blogspot.in/2014/05/steps-for-apply-patch-set-from-112030.html




New in 12c patch

Check If PSU applied

SQL> set pagesize 0
SQL> set long 1000000
SQL> set serverout on

SQL> exec dbms_qopatch.get_sqlpatch_status;

SQL> select xmltransform(dbms_qopatch.get_opatch_lsinventory, dbms_qopatch.get_opatch_xslt) from dual;

SQL> select xmltransform(dbms_qopatch.is_patch_installed('&patchno'), dbms_qopatch.get_opatch_xslt) "Patch installed?" from dual;




STEPS to Apply DB April 2018 PSU to 12cR1 two node RAC


Note : OPatch 12.2.0.1.5 or later, -ocmrf <ocm response file> option does not need to be provided.

Install new Opatch :

Node1 # mv /u01/app/12.1.0.2/grid/OPatch  /u01/app/12.1.0.2/grid/OPatch/OPatch_bkp
Node1 # unzip p6880880_121010_LINUX.zip -d /u01/app/oracle/12.1.0.2/grid
Node1 # export ORACLE_SID=+ASM1
Node1 # export PATH=/u01/app/12.1.0.2/grid/OPatch:$PATH
Node1 # /u01/app/oracle/12.1.0.2/grid/OPatch/opatch lsinventory detail -oh /u01/app/oracle/12.1.0.2/grid
Node1 # /u01/app/oracle/12.1.0.2/grid/OPatch/opatch version

Node1 # mv /u01/app/oracle/product/12.1.0.2/db_1/OPatch  /u01/app/oracle/product/12.1.0.2/db_1/OPatch_bkp
Node1 # unzip  /tmp/p6880880_121010_Linux-x86-64.zip -d /u01/app/oracle/product/12.1.0.2/db_1
Node1 # export ORACLE_SID=prod1
Node1 # export PATH=/u01/app/oracle/product/12.1.0.2/db_1/OPatch:$PATH
Node1 # /u01/app/oracle/product/12.1.0.2/db_1/OPatch/opatch lsinventory detail -oh /u01/app/oracle/product/12.1.0.2/db_1
Node1 # /u01/app/oracle/product/12.1.0.2/db_1/OPatch version

Repeat on Node2

1. Pre-Checks : Analyze Conflict Detection

Node1 # export PATH=/u01/app/12.1.0.2/grid/OPatch:$PATH
Node1 # /u01/app/12.1.0.2/grid/OPatch/opatchauto apply /u01/27468957 -oh /u01/app/12.1.0.2/grid -analyze

Node1 # export PATH=/u01/app/oracle/product/12.1.0.2/db_1/OPatch:$PATH
Node1 # /u01/app/oracle/product/12.1.0.2/db_1/OPatch/opatchauto apply /u01/27468957 -oh /u01/app/oracle/product/12.1.0.2/db_1 -analyze

Repeat on Node2

2. Apply patch :

Node1 # export PATH=/u01/app/12.1.0.2/grid/OPatch:$PATH
Node1 # /u01/app/12.1.0.2/grid/OPatch/opatchauto apply /u01/27468957 -oh /u01/app/12.1.0.2/grid

Node1 # export PATH=/u01/app/oracle/product/12.1.0.2/db_1/OPatch:$PATH
Node1 #  /u01/app/oracle/product/12.1.0.2/db_1/OPatch/opatchauto apply /u01/27468957 -oh /u01/app/oracle/product/12.1.0.2/db_1

Repeat on Node2


3. Post Checks : 

  Check dba_registry_sqlpatch


To apply on all databases in one command

# opatchauto apply /u01/27468957  -- opatchauto automatically down the CRS and database services and restart

Ref : https://oracledbwr.com/step-by-step-apply-12c-grid-and-db-april-2018-psu-to-oracle-12cr1-2-node-rac/


DataPatch :


Like catbundle in 11g, 12c has datapatch. Refer for MOS 1585822.1

Datapatch determines the requisite apply/rollback actions by matching an internal repository with the patch inventory.
Datapatch should be invoked when the database is restarted after a patching session.
Enterprise Manager and OPatchAuto call datapatch automatically during every patching session.
If OPatch is used to install RDBMS patches then datapatch has to be explictly called to complete any patching actions after database restart.
Datapatch will automatically determine the set of patches that need to be installed and a set of patches that need to be rolled back.
Datapatch ensures that a patch has been installed/rolled back in all RAC instances before to initiate any post-patch SQL actions on the database.
In Oracle Multitenant environment datapatch will patch the root and any pluggable databases that are opened.
In order to patch all pluggable databases, it should be ensured that before to invoke datapatch all pluggable databases are opened.
If an unpatched pluggable database is opened in Oracle Multitenant then calling datapatch will complete the pending patch actions.

To update DBA_REGISTRY_SQLPATCH with the PSU information run the following.

/u01/app/oracle/product/12.1.0.2/db_1/OPatch/datapatch -force -apply 22291127 -bundle_series PSU

$ opatch lsinv -oh /u01/app/oracle/product/12.1.0.2/db_1 | egrep "22291127|Unique"

/u01/app/oracle/product/12.1.0.2/db_1/OPatch/datapatch -apply 22291127/19694308 -force -verbos

select action_time, patch_id, patch_uid, version, status, bundle_series, description from dba_registry_sqlpatch;