Howto upgrade Oracle Grid 11g to 11.2.0.3

This post is also available in: Português

In this article we will upgrade a database running on Oracle RAC version 11.2.0.1 to version 11.2.0.3.
The steps to upgrade from the version 11.2.0.2 are very similar, only changing the dependent packages and patches.
Our RAC database is running on a Red Hat, with the following specifications:

  • O.S: Red Hat Enterprise Linux Server release 5.8 (Tikanga) – x86_64
  • Kernel: 2.6.18-308.16.1.el5
  • VMWare: VMware ESXi 5.0
  • Oracle 11.2.0.3
  • Individual Oracle homes and shared disks for RAC via VMFS

To perform this upgrade, you will need the following Oracle packages:

  • Oracle and Grid Installer: p10404530_112030_Linux-x86-64 (1,2,3 files only)
  • Patch for update bug fix: p9413827_112010_Linux-x86-64 (if the source version is the 11.2.0.1)
  • Latest Patch Set update

This tutorial is divided into 7 stages macros:

  1. Transfer the installation files to the server and unzip them.
  2. Apply patch 9413827.
  3. Install the dependency package cvuqdisk.
  4. Install and upgrade Oracle Grid to the latest version.
  5. Install and upgrade Oracle Database to the latest version.
  6. Apply the latest Patch Set Update in Oracle Database and Grid.

So, let's start:

1 - Transfer the installation files to the server and unzip them.

First part, we will transfer the files to a server where they have available. If you do not have them, download these files from Metalink.

[root@patchserver ~]# scp -r p10404530_112030_Linux-x86-64/ patches/ root@10.20.1.2:/u01/oracle_install
root@10.20.1.2's password:
p10404530_112030_Linux-x86-64_1of7.zip             100% 1296MB  21.2MB/s   01:01
p10404530_112030_Linux-x86-64_2of7.zip             100% 1089MB  21.8MB/s   00:50
p10404530_112030_Linux-x86-64_3of7.zip             100%  934MB  21.2MB/s   00:44
p9413827_112010_Linux-x86-64.zip                   100%  285MB  21.9MB/s   00:13
[root@patchserver oracle_install]#

Then, let's log into RAC node 1 where we transfered the files and check:

[root@oraclebdsrv01 ~]# ls -laR /u01/oracle_install/
/u01/oracle_install/:
total 16
drwxr-xr-x 4 root root     4096 Jun 14 16:46 .
drwxr-xr-x 5 root oinstall 4096 Jun 14 16:41 ..
drwxr-xr-x 2 root root     4096 Jun 14 16:45 p10404530_112030_Linux-x86-64
drwxrwxr-x 2 root root     4096 Jun 14 16:47 patches

/u01/oracle_install/p10404530_112030_Linux-x86-64:
total 3401636
drwxr-xr-x 2 root root       4096 Jun 14 16:45 .
drwxr-xr-x 4 root root       4096 Jun 14 16:46 ..
-rw-r--r-- 1 root root 1358454646 Jun 14 16:43 p10404530_112030_Linux-x86-64_1of7.zip
-rw-r--r-- 1 root root 1142195302 Jun 14 16:44 p10404530_112030_Linux-x86-64_2of7.zip
-rw-r--r-- 1 root root  979195792 Jun 14 16:46 p10404530_112030_Linux-x86-64_3of7.zip

/u01/oracle_install/patches:
total 747360
drwxrwxr-x 2 root root      4096 Jun 14 16:47 .
drwxr-xr-x 4 root root      4096 Jun 14 16:46 ..
-rw-r--r-- 1 root root 298972922 Jun 14 16:47 p9413827_112010_Linux-x86-64.zip
[root@oraclebdsrv01 ~]#

Now unzip the files as the oracle user:

[root@oraclebdsrv01 ~]# su - oracle
[oracle@oraclebdsrv01 ~]$ cd /u01/oracle_install/p10404530_112030_Linux-x86-64/
[oracle@oraclebdsrv01 p10404530_112030_Linux-x86-64]$ ls -la
total 3401636
drwxr-xr-x 2 oracle oinstall       4096 Jun 14 16:45 .
drwxr-xr-x 5 oracle oinstall       4096 Jun 14 17:13 ..
-rw-r--r-- 1 oracle oinstall 1358454646 Jun 14 16:43 p10404530_112030_Linux-x86-64_1of7.zip
-rw-r--r-- 1 oracle oinstall 1142195302 Jun 14 16:44 p10404530_112030_Linux-x86-64_2of7.zip
-rw-r--r-- 1 oracle oinstall  979195792 Jun 14 16:46 p10404530_112030_Linux-x86-64_3of7.zip
[oracle@oraclebdsrv01 p10404530_112030_Linux-x86-64]$ unzip "*.zip"
Archive:  p10404530_112030_Linux-x86-64_1of7.zip
   creating: database/
   .....

Archive:  p10404530_112030_Linux-x86-64_2of7.zip
   creating: database/stage/Components/oracle.ctx/
   .....

Archive:  p10404530_112030_Linux-x86-64_3of7.zip
   creating: grid/
   .....

3 archives were successfully processed.
[oracle@oraclebdsrv01 p10404530_112030_Linux-x86-64]$

The first part was easy.

2 - Unzip the files.

The second part will be a patch to fix the update for this database version. If skipped, you would face the following message during the installation of Grid in step 3:

Attention to this step, READ the README.txt file inside the patch folder.
In my case, the installation folder of the Oracle Home is not shared across the intances. So, I used the option "Configuration B" specified in the manual.

You must run the whole steps below a node of your RAC at a time. So, perform all the steps in a node, then another, successively.

We will divide this part into 10 sub-steps:

### 1 Make sure the opatch is set up to the ORACLE_HOME and GRID_HOME.

As oracle, run:
% opatch lsinventory -detail -oh <rdbms_home>

[root@oraclebdsrv01 ~]# su - oracle
[oracle@oraclebdsrv01 ~]$ echo $DB_HOME
/u01/app/oracle/product/11.1.0/db_1
[oracle@oraclebdsrv01 ~]$ opatch lsinventory -detail -oh $DB_HOME
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.1.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.1.0/db_1/oui
Log file location : /u01/app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/opatch2013-06-20_15-26-49PM.log

Patch history file: /u01/app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2013-06-20_15-26-49PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.

Installed Products (134):

Agent Required Support Files                                         10.2.0.4.2
Assistant Common Files                                               11.2.0.1.0
Bali Share                                                           1.1.18.0.0
Buildtools Common Files                                              11.2.0.1.0
Character Set Migration Utility                                      11.2.0.1.0
Cluster Verification Utility Common Files                            11.2.0.1.0
Database Configuration and Upgrade Assistants                        11.2.0.1.0
Database SQL Scripts                                                 11.2.0.1.0
Database Workspace Manager                                           11.2.0.1.0
Deinstallation Tool                                                  11.2.0.1.0
Enterprise Edition Options                                           11.2.0.1.0
Enterprise Manager Agent                                             10.2.0.4.2
Enterprise Manager Agent Core Files                                  10.2.0.4.2
Enterprise Manager Common Core Files                                 10.2.0.4.2
Enterprise Manager Common Files                                      10.2.0.4.2
Enterprise Manager Database Plugin -- Agent Support                  11.2.0.1.0
Enterprise Manager Database Plugin -- Repository Support             11.2.0.1.0
Enterprise Manager Grid Control Core Files                           10.2.0.4.2
Enterprise Manager Minimal Integration                               11.2.0.1.0
Enterprise Manager plugin Common Files                               11.2.0.1.0
Enterprise Manager Repository Core Files                             10.2.0.4.2
Exadata Storage Server                                               11.2.0.1.0
Expat libraries                                                       2.0.1.0.1
Generic Connectivity Common Files                                    11.2.0.1.0
HAS Common Files                                                     11.2.0.1.0
HAS Files for DB                                                     11.2.0.1.0
Installation Common Files                                            11.2.0.1.0
Installation Plugin Files                                            11.2.0.1.0
Installer SDK Component                                              11.2.0.1.0
JAccelerator (COMPANION)                                             11.2.0.1.0
LDAP Required Support Files                                          11.2.0.1.0
OLAP SQL Scripts                                                     11.2.0.1.0
Oracle 11g Warehouse Builder Required Files                          11.2.0.1.0
Oracle Advanced Security                                             11.2.0.1.0
Oracle Application Express                                           11.2.0.1.0
Oracle Call Interface (OCI)                                          11.2.0.1.0
Oracle Clusterware RDBMS Files                                       11.2.0.1.0
Oracle Code Editor                                                   1.2.1.0.0I
Oracle Configuration Manager                                         10.3.1.1.0
Oracle Configuration Manager Deconfiguration                         10.3.1.0.0
Oracle Containers for Java                                           11.2.0.1.0
Oracle Core Required Support Files                                   11.2.0.1.0
Oracle Data Mining RDBMS Files                                       11.2.0.1.0
Oracle Database 11g                                                  11.2.0.1.0
Oracle Database 11g                                                  11.2.0.1.0
Oracle Database 11g Multimedia Files                                 11.2.0.1.0
Oracle Database Deconfiguration                                      11.2.0.1.0
Oracle Database Gateway for ODBC                                     11.2.0.1.0
Oracle Database User Interface                                       2.2.13.0.0
Oracle Database Utilities                                            11.2.0.1.0
Oracle Database Vault J2EE Application                               11.2.0.1.0
Oracle Database Vault option                                         11.2.0.1.0
Oracle DBCA Deconfiguration                                          11.2.0.1.0
Oracle Display Fonts                                                  9.0.2.0.0
Oracle Enterprise Manager Console DB                                 11.2.0.1.0
Oracle Extended Windowing Toolkit                                    3.4.47.0.0
Oracle Globalization Support                                         11.2.0.1.0
Oracle Globalization Support                                         11.2.0.1.0
Oracle Help For Java                                                  4.2.9.0.0
Oracle Help for the  Web                                             2.0.14.0.0
Oracle Ice Browser                                                    5.2.3.6.0
Oracle Internet Directory Client                                     11.2.0.1.0
Oracle Java Client                                                   11.2.0.1.0
Oracle JDBC Server Support Package                                   11.2.0.1.0
Oracle JDBC/OCI Instant Client                                       11.2.0.1.0
Oracle JDBC/THIN Interfaces                                          11.2.0.1.0
Oracle JFC Extended Windowing Toolkit                                4.2.36.0.0
Oracle JVM                                                           11.2.0.1.0
Oracle Label Security                                                11.2.0.1.0
Oracle LDAP administration                                           11.2.0.1.0
Oracle Locale Builder                                                11.2.0.1.0
Oracle Message Gateway Common Files                                  11.2.0.1.0
Oracle Multimedia                                                    11.2.0.1.0
Oracle Multimedia Annotator                                          11.2.0.1.0
Oracle Multimedia Client Option                                      11.2.0.1.0
Oracle Multimedia Java Advanced Imaging                              11.2.0.1.0
Oracle Multimedia Locator                                            11.2.0.1.0
Oracle Multimedia Locator RDBMS Files                                11.2.0.1.0
Oracle Net                                                           11.2.0.1.0
Oracle Net Listener                                                  11.2.0.1.0
Oracle Net Required Support Files                                    11.2.0.1.0
Oracle Net Services                                                  11.2.0.1.0
Oracle Netca Client                                                  11.2.0.1.0
Oracle Notification Service                                          11.2.0.0.0
Oracle Notification Service (eONS)                                   11.2.0.1.0
Oracle ODBC Driver                                                   11.2.0.1.0
Oracle ODBC Driverfor Instant Client                                 11.2.0.1.0
Oracle OLAP                                                          11.2.0.1.0
Oracle OLAP API                                                      11.2.0.1.0
Oracle OLAP RDBMS Files                                              11.2.0.1.0
Oracle One-Off Patch Installer                                       11.2.0.0.2
Oracle Partitioning                                                  11.2.0.1.0
Oracle Programmer                                                    11.2.0.1.0
Oracle Quality of Service Management (Client)                        11.2.0.1.0
Oracle RAC Deconfiguration                                           11.2.0.1.0
Oracle RAC Required Support Files-HAS                                11.2.0.1.0
Oracle Real Application Testing                                      11.2.0.1.0
Oracle Recovery Manager                                              11.2.0.1.0
Oracle Security Developer Tools                                      11.2.0.1.0
Oracle Spatial                                                       11.2.0.1.0
Oracle SQL Developer                                                 11.2.0.1.0
Oracle Starter Database                                              11.2.0.1.0
Oracle Text                                                          11.2.0.1.0
Oracle Text Required Support Files                                   11.2.0.1.0
Oracle UIX                                                           2.2.24.5.0
Oracle Universal Connection Pool                                     11.2.0.1.0
Oracle Universal Installer                                           11.2.0.1.0
Oracle Wallet Manager                                                11.2.0.1.0
Oracle XML Development Kit                                           11.2.0.1.0
Oracle XML Query                                                     11.2.0.1.0
Parser Generator Required Support Files                              11.2.0.1.0
Perl Interpreter                                                     5.10.0.0.1
Perl Modules                                                         5.10.0.0.1
PL/SQL                                                               11.2.0.1.0
PL/SQL Embedded Gateway                                              11.2.0.1.0
Platform Required Support Files                                      11.2.0.1.0
Precompiler Common Files                                             11.2.0.1.0
Precompiler Required Support Files                                   11.2.0.1.0
Provisioning Advisor Framework                                       10.2.0.4.2
RDBMS Required Support Files                                         11.2.0.1.0
RDBMS Required Support Files for Instant Client                      11.2.0.1.0
regexp                                                                2.1.9.0.0
Required Support Files                                               11.2.0.1.0
Sample Schema Data                                                   11.2.0.1.0
Secure Socket Layer                                                  11.2.0.1.0
SQL*Plus                                                             11.2.0.1.0
SQL*Plus Files for Instant Client                                    11.2.0.1.0
SQL*Plus Required Support Files                                      11.2.0.1.0
SQLJ Runtime                                                         11.2.0.1.0
SSL Required Support Files for InstantClient                         11.2.0.1.0
Sun JDK                                                              1.5.0.17.0
XDK Required Support Files                                           11.2.0.1.0
XML Parser for Java                                                  11.2.0.1.0
XML Parser for Oracle JVM                                            11.2.0.1.0
There are 134 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

Rac system comprising of multiple nodes
  Local node = oraclebdsrv01
  Remote node = oraclebdsrv02

--------------------------------------------------------------------------------

OPatch succeeded.
[oracle@oraclebdsrv01 ~]$

As grid, run:
% opatch lsinventory -detail -oh <crs_home>

[root@oraclebdsrv01 ~]# su - grid
[grid@oraclebdsrv01 ~]$ echo $GRID_HOME
/u01/app/11.2.0/grid
[grid@oraclebdsrv01 ~]$ opatch lsinventory -detail -oh $GRID_HOME
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/11.2.0/grid/oui
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2013-06-20_15-27-03PM.log

Patch history file: /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2013-06-20_15-27-03PM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.

Installed Products (87):

Agent Required Support Files                                         10.2.0.4.2
Assistant Common Files                                               11.2.0.1.0
Automatic Storage Management Assistant                               11.2.0.1.0
Bali Share                                                           1.1.18.0.0
Buildtools Common Files                                              11.2.0.1.0
Character Set Migration Utility                                      11.2.0.1.0
Cluster Ready Services Files                                         11.2.0.1.0
Cluster Verification Utility Common Files                            11.2.0.1.0
Cluster Verification Utility Files                                   11.2.0.1.0
Database SQL Scripts                                                 11.2.0.1.0
Deinstallation Tool                                                  11.2.0.1.0
Enterprise Manager Common Core Files                                 10.2.0.4.2
Enterprise Manager Common Files                                      10.2.0.4.2
Enterprise Manager Minimal Integration                               11.2.0.1.0
Enterprise Manager plugin Common Files                               11.2.0.1.0
Expat libraries                                                       2.0.1.0.1
HAS Common Files                                                     11.2.0.1.0
HAS Files for DB                                                     11.2.0.1.0
Installation Common Files                                            11.2.0.1.0
Installation Plugin Files                                            11.2.0.1.0
Installer SDK Component                                              11.2.0.1.0
LDAP Required Support Files                                          11.2.0.1.0
OLAP SQL Scripts                                                     11.2.0.1.0
Oracle Clusterware RDBMS Files                                       11.2.0.1.0
Oracle Configuration Manager Deconfiguration                         10.3.1.0.0
Oracle Containers for Java                                           11.2.0.1.0
Oracle Core Required Support Files                                   11.2.0.1.0
Oracle Database 11g                                                  11.2.0.1.0
Oracle Database 11g Multimedia Files                                 11.2.0.1.0
Oracle Database Deconfiguration                                      11.2.0.1.0
Oracle Database User Interface                                       2.2.13.0.0
Oracle Database Utilities                                            11.2.0.1.0
Oracle DBCA Deconfiguration                                          11.2.0.1.0
Oracle Extended Windowing Toolkit                                    3.4.47.0.0
Oracle Globalization Support                                         11.2.0.1.0
Oracle Globalization Support                                         11.2.0.1.0
Oracle Grid Infrastructure                                           11.2.0.1.0
Oracle Help For Java                                                  4.2.9.0.0
Oracle Ice Browser                                                    5.2.3.6.0
Oracle Internet Directory Client                                     11.2.0.1.0
Oracle Java Client                                                   11.2.0.1.0
Oracle JDBC/OCI Instant Client                                       11.2.0.1.0
Oracle JDBC/THIN Interfaces                                          11.2.0.1.0
Oracle JFC Extended Windowing Toolkit                                4.2.36.0.0
Oracle JVM                                                           11.2.0.1.0
Oracle LDAP administration                                           11.2.0.1.0
Oracle Locale Builder                                                11.2.0.1.0
Oracle Multimedia                                                    11.2.0.1.0
Oracle Multimedia Client Option                                      11.2.0.1.0
Oracle Multimedia Java Advanced Imaging                              11.2.0.1.0
Oracle Multimedia Locator                                            11.2.0.1.0
Oracle Multimedia Locator RDBMS Files                                11.2.0.1.0
Oracle Net                                                           11.2.0.1.0
Oracle Net Listener                                                  11.2.0.1.0
Oracle Net Required Support Files                                    11.2.0.1.0
Oracle Netca Client                                                  11.2.0.1.0
Oracle Notification Service                                          11.2.0.0.0
Oracle Notification Service (eONS)                                   11.2.0.1.0
Oracle One-Off Patch Installer                                       11.2.0.0.2
Oracle Quality of Service Management (Client)                        11.2.0.1.0
Oracle Quality of Service Management (Server)                        11.2.0.1.0
Oracle RAC Deconfiguration                                           11.2.0.1.0
Oracle RAC Required Support Files-HAS                                11.2.0.1.0
Oracle Recovery Manager                                              11.2.0.1.0
Oracle Security Developer Tools                                      11.2.0.1.0
Oracle Text Required Support Files                                   11.2.0.1.0
Oracle Universal Installer                                           11.2.0.1.0
Oracle Wallet Manager                                                11.2.0.1.0
Parser Generator Required Support Files                              11.2.0.1.0
Perl Interpreter                                                     5.10.0.0.1
Perl Modules                                                         5.10.0.0.1
PL/SQL                                                               11.2.0.1.0
PL/SQL Embedded Gateway                                              11.2.0.1.0
Platform Required Support Files                                      11.2.0.1.0
Precompiler Required Support Files                                   11.2.0.1.0
RDBMS Required Support Files                                         11.2.0.1.0
RDBMS Required Support Files for Instant Client                      11.2.0.1.0
Required Support Files                                               11.2.0.1.0
Secure Socket Layer                                                  11.2.0.1.0
SQL*Plus                                                             11.2.0.1.0
SQL*Plus Files for Instant Client                                    11.2.0.1.0
SQL*Plus Required Support Files                                      11.2.0.1.0
SSL Required Support Files for InstantClient                         11.2.0.1.0
Sun JDK                                                              1.5.0.17.0
Universal Storage Manager Files                                      11.2.0.1.0
XDK Required Support Files                                           11.2.0.1.0
XML Parser for Java                                                  11.2.0.1.0
There are 87 products installed in this Oracle Home.

There are no Interim patches installed in this Oracle Home.

Rac system comprising of multiple nodes
  Local node = oraclebdsrv01
  Remote node = oraclebdsrv02

--------------------------------------------------------------------------------

OPatch succeeded.
[grid@oraclebdsrv01 ~]$

### 2 Unzip the files.

As oracle, run:
% unzip p9413827_11201_LINUX.X64.zip

[root@oraclebdsrv01 ~]# su - oracle
[oracle@oraclebdsrv01 ~]$ cd /u01/oracle_install/patches
[oracle@oraclebdsrv01 patches]$ ls -la
total 747360
drwxrwxr-x 2 oracle oinstall      4096 Jun 14 16:47 .
drwxr-xr-x 5 oracle oinstall      4096 Jun 14 17:13 ..
-rw-r--r-- 1 oracle oinstall 298972922 Jun 14 16:47 p9413827_112010_Linux-x86-64.zip
[oracle@oraclebdsrv01 patches]$ unzip p9413827_112010_Linux-x86-64.zip
Archive:  p9413827_112010_Linux-x86-64.zip
   creating: 9413827/
   .....
[oracle@oraclebdsrv01 patches]$

### 3 Stop the "oracle_home" service.

As oracle, run:
% srvctl stop home -o <rdbms_home> -s -n <node_name>#

[root@oraclebdsrv01 ~]# su - oracle
[oracle@oraclebdsrv01 ~]$ srvctl stop home -o $DB_HOME -s /u01/oracle_install/patches/status.file -n oraclebdsrv01
[oracle@oraclebdsrv01 ~]$

### 4 Unprotect the blocked installation files.

As root, run:
# <crs_home>/crs/install/rootcrs.pl -unlock

ATTENTION: If you have many audit files, this step will take much! So, delete them before!

In my case, I never knew I had so many and my command "rootcrs.pl -unlock" got stuck. I opened another parallel session to solve the problem.

[root@oraclebdsrv01 ~]# ls -1 /u01/app/oracle/admin/SGF/adump/ | wc -l
127153
[root@oraclebdsrv01 ~]# ls -1 /u01/app/11.2.0/grid/rdbms/audit/ | wc -l
3175278

One of the fastest ways to delete files using rsync is, as can be seen in: http://linuxnote.net/jianingy/en/linux/a-fast-way-to-remove-huge-number-of-files.html

[grid@oraclebdsrv01 ~]# find /u01/app/11.2.0/grid/rdbms/audit/ ! -name "*.aud"
/u01/app/11.2.0/grid/rdbms/audit/
[grid@oraclebdsrv01 ~]$ mkdir /tmp/empty
[grid@oraclebdsrv01 ~]$ rsync -a --delete /tmp/empty/ /u01/app/11.2.0/grid/rdbms/audit/

Running the script:

[root@oraclebdsrv01 ~]# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -unlock
2013-06-21 10:17:54: Parsing the host name
2013-06-21 10:17:54: Checking for super user privileges
2013-06-21 10:17:54: User has super user privileges
Using configuration parameter file: crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.crsd' on 'oraclebdsrv01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.OCR_VOTE.dg' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.registry.acfs' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.ARCHIVE.dg' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.DATA1.dg' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.oraclebdsrv01.vip' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.oraclebdsrv01.vip' on 'oraclebdsrv01' succeeded
CRS-2672: Attempting to start 'ora.oraclebdsrv01.vip' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.registry.acfs' on 'oraclebdsrv01' succeeded
CRS-2676: Start of 'ora.oraclebdsrv01.vip' on 'oraclebdsrv02' succeeded
CRS-2677: Stop of 'ora.OCR_VOTE.dg' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.ARCHIVE.dg' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.DATA1.dg' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.asm' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.eons' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.ons' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.ons' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.net1.network' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.eons' on 'oraclebdsrv01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oraclebdsrv01' has completed
CRS-2677: Stop of 'ora.crsd' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.evmd' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.asm' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.cssdmonitor' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.asm' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.cssd' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.diskmon' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.gipcd' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.gipcd' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'oraclebdsrv01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oraclebdsrv01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
Successfully unlock /u01/app/11.2.0/grid
[root@oraclebdsrv01 ~]#

### 5 Save the current settings of the database home.

As oracle, run:
% custom/server/9413827/custom/scripts/prepatch.sh -dbhome <rdbms_home>

[root@oraclebdsrv01 ~]# su - oracle
[oracle@oraclebdsrv01 ~]$ cd /u01/oracle_install/patches/9413827/
[oracle@oraclebdsrv01 9413827]$ custom/server/9413827/custom/scripts/prepatch.sh -dbhome $DB_HOME
custom/server/9413827/custom/scripts/prepatch.sh completed successfully.
[oracle@oraclebdsrv01 9413827]$

### 6 Let's apply the patch on CRS and after on RDBMS.

As grid, run:
% opatch napply -local -oh <crs_home> -id 9413827

[root@oraclebdsrv01 ~]# su - grid
[grid@oraclebdsrv01 ~]$ cd /u01/oracle_install/patches/
[grid@oraclebdsrv01 patches]$ ls -la
total 747368
drwxrwxr-x 3 oracle oinstall      4096 Jun 21 10:14 .
drwxr-xr-x 5 oracle oinstall      4096 Jun 14 17:13 ..
drwxr-xr-x 5 oracle oinstall      4096 Oct 25  2010 9413827
-rw-r--r-- 1 oracle oinstall 298972922 Jun 14 16:47 p9413827_112010_Linux-x86-64.zip
-rw-r--r-- 1 oracle oinstall         7 Jun 21 10:14 status.file
[grid@oraclebdsrv01 patches]$ echo $GRID_HOME
/u01/app/11.2.0/grid
[grid@oraclebdsrv01 patches]$ opatch napply -local -oh $GRID_HOME -id 9413827
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/11.2.0/grid/oui
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2013-06-21_13-17-02PM.log

Patch history file: /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch_history.txt

Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9413827

Do you want to proceed? [y|n]
y
User Responded with: Y

Running prerequisite checks...

You selected -local option, hence OPatch will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/11.2.0/grid')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...

Applying patch 9413827...

ApplySession applying interim patch '9413827' to OH '/u01/app/11.2.0/grid'
Backing up files affected by the patch '9413827' for rollback. This might take a while...

Patching component oracle.crs, 11.2.0.1.0...
Copying file to "/u01/app/11.2.0/grid/bin/appagent.bin"
.....
Copying file to "/u01/app/11.2.0/grid/srvm/mesg/sclsus.msg"
ApplySession adding interim patch '9413827' to inventory

Verifying the update...
Inventory check OK: Patch ID 9413827 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9413827 are present in Oracle Home.

The local system has been patched and can be restarted.

UtilSession: N-Apply done.

OPatch succeeded.
[grid@oraclebdsrv01 patches]$

As oracle, run:
% opatch napply custom/server/ -local -oh <rdbms_home> -id 9413827

[root@oraclebdsrv01 ~]# su - oracle
[oracle@oraclebdsrv01 ~]$ cd /u01/oracle_install/patches/
[oracle@oraclebdsrv01 patches]$ ls -la
total 747368
drwxrwxr-x 3 oracle oinstall      4096 Jun 21 10:14 .
drwxr-xr-x 5 oracle oinstall      4096 Jun 14 17:13 ..
drwxr-xr-x 5 oracle oinstall      4096 Oct 25  2010 9413827
-rw-r--r-- 1 oracle oinstall 298972922 Jun 14 16:47 p9413827_112010_Linux-x86-64.zip
-rw-r--r-- 1 oracle oinstall         7 Jun 21 10:14 status.file
[oracle@oraclebdsrv01 patches]$ echo $DB_HOME
/u01/app/oracle/product/11.1.0/db_1
[oracle@oraclebdsrv01 patches]$ opatch napply 9413827/custom/server/ -local -oh $DB_HOME -id 9413827
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

UTIL session

Oracle Home       : /u01/app/oracle/product/11.1.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.1.0/db_1/oui
Log file location : /u01/app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/opatch2013-06-21_13-55-07PM.log

Patch history file: /u01/app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/opatch_history.txt

Invoking utility "napply"
Checking conflict among patches...
Checking if Oracle Home has components required by patches...
Checking conflicts against Oracle Home...
OPatch continues with these patches:   9413827

Do you want to proceed? [y|n]
y
User Responded with: Y

Running prerequisite checks...

You selected -local option, hence OPatch will patch the local system only.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/11.1.0/db_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files affected by the patch 'NApply' for restore. This might take a while...

Applying patch 9413827...

ApplySession applying interim patch '9413827' to OH '/u01/app/oracle/product/11.1.0/db_1'
Backing up files affected by the patch '9413827' for rollback. This might take a while...

Patching component oracle.rdbms, 11.2.0.1.0...
Copying file to "/u01/app/oracle/product/11.1.0/db_1/bin/cluvfy"
.....
Copying file to "/u01/app/oracle/product/11.1.0/db_1/srvm/mesg/sclsus.msg"
ApplySession adding interim patch '9413827' to inventory

Verifying the update...
Inventory check OK: Patch ID 9413827 is registered in Oracle Home inventory with proper meta-data.
Files check OK: Files from Patch ID 9413827 are present in Oracle Home.
Running make for target install
Running make for target install

The local system has been patched and can be restarted.

UtilSession: N-Apply done.

OPatch succeeded.
[oracle@oraclebdsrv01 patches]$

### 7 Some configuration settings need to be done.

As root, run:
# chmod +w <crs_home>/log//agent
# chmod +w <crs_home>/log//agent/crsd

[root@oraclebdsrv01 ~]# chmod +w /u01/app/11.2.0/grid/log/oraclebdsrv01/agent/
[root@oraclebdsrv01 ~]# chmod +w /u01/app/11.2.0/grid/log/oraclebdsrv01/agent/crsd/
[root@oraclebdsrv01 ~]#

As oracle, run:
% custom/server/9413827/custom/scripts/postpatch.sh -dbhome <rdbms_home>

[root@oraclebdsrv01 ~]# su - oracle
[oracle@oraclebdsrv01 ~]$ cd /u01/oracle_install/patches/9413827/
[oracle@oraclebdsrv01 9413827]$ echo $DB_HOME
/u01/app/oracle/product/11.1.0/db_1
[oracle@oraclebdsrv01 9413827]$ custom/server/9413827/custom/scripts/postpatch.sh -dbhome $DB_HOME
Reading /u01/app/oracle/product/11.1.0/db_1/install/params.ora..
Reading /u01/app/oracle/product/11.1.0/db_1/install/params.ora..
Parsing file /u01/app/oracle/product/11.1.0/db_1/bin/racgwrap
Parsing file /u01/app/oracle/product/11.1.0/db_1/bin/srvctl
Parsing file /u01/app/oracle/product/11.1.0/db_1/bin/srvconfig
Parsing file /u01/app/oracle/product/11.1.0/db_1/bin/cluvfy
Verifying file /u01/app/oracle/product/11.1.0/db_1/bin/racgwrap
Verifying file /u01/app/oracle/product/11.1.0/db_1/bin/srvctl
Verifying file /u01/app/oracle/product/11.1.0/db_1/bin/srvconfig
Verifying file /u01/app/oracle/product/11.1.0/db_1/bin/cluvfy
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/racgwrap
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/srvctl
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/srvconfig
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/cluvfy
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/racgmain
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/racgeut
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/diskmon.bin
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/lsnodes
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/osdbagrp
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/rawutl
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/srvm/admin/ractrans
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/srvm/admin/getcrshome
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/gnsd
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/bin/crsdiag.pl
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/lib/libhasgen11.so
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/lib/libclsra11.so
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/lib/libdbcfg11.so
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/lib/libocr11.so
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/lib/libocrb11.so
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/lib/libocrutl11.so
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/lib/libuini11.so
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/lib/librdjni11.so
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/lib/libgns11.so
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/lib/libgnsjni11.so
Reapplying file permissions on /u01/app/oracle/product/11.1.0/db_1/lib/libagfw11.so
[oracle@oraclebdsrv01 9413827]$

### 8 We must restore the security permissions of the application directories.

As root, run:
# <crs_home>/crs/install/rootcrs.pl -patch
# <db_home>/bin/srvctl start home -o <db_home> -s -n

[root@oraclebdsrv01 ~]# /u01/app/11.2.0/grid/crs/install/rootcrs.pl -patch
2013-06-21 14:05:58: Parsing the host name
2013-06-21 14:05:58: Checking for super user privileges
2013-06-21 14:05:58: User has super user privileges
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
CRS-4123: Oracle High Availability Services has been started.
[root@oraclebdsrv01 ~]#
[root@oraclebdsrv01 ~]# /u01/app/oracle/product/11.1.0/db_1/bin/srvctl start home -o /u01/app/oracle/product/11.1.0/db_1 -s /u01/oracle_install/patches/status.file -n oraclebdsrv01
[root@oraclebdsrv01 ~]#

### 9 Checking opatch.

Finally, we will check if the patch is installed and cataloged.

As oracle, run:
% opatch lsinventory -detail -oh <rdbms_home>

[root@oraclebdsrv01 ~]# su - oracle
[oracle@oraclebdsrv01 ~]$ opatch lsinventory -oh $DB_HOME
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/11.1.0/db_1
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/oracle/product/11.1.0/db_1/oui
Log file location : /u01/app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/opatch2013-07-20_10-21-28AM.log

Patch history file: /u01/app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/oracle/product/11.1.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2013-07-20_10-21-28AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Database 11g                                                  11.2.0.1.0
There are 1 products installed in this Oracle Home.

Interim patches (1) :

Patch  9413827      : applied on Fri Jun 21 13:55:34 BRT 2013
Unique Patch ID:  13136265
   Created on 25 Oct 2010, 07:07:02 hrs PST8PDT
   Bugs fixed:
     9262748, 9343627, 9655006, 9783609, 9413827, 9778840

Rac system comprising of multiple nodes
Rac system comprising of multiple nodes
  Local node = oraclebdsrv01
  Remote node = oraclebdsrv02

--------------------------------------------------------------------------------

OPatch succeeded.
[oracle@oraclebdsrv01 ~]$

As grid, run:
% opatch lsinventory -detail -oh <crs_home>

[root@oraclebdsrv01 ~]# su - grid
[grid@oraclebdsrv01 ~]$ opatch lsinventory -oh $GRID_HOME
Invoking OPatch 11.1.0.6.6

Oracle Interim Patch Installer version 11.1.0.6.6
Copyright (c) 2009, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/11.2.0/grid
Central Inventory : /u01/app/oraInventory
   from           : /etc/oraInst.loc
OPatch version    : 11.1.0.6.6
OUI version       : 11.2.0.1.0
OUI location      : /u01/app/11.2.0/grid/oui
Log file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch2013-07-20_10-19-44AM.log

Patch history file: /u01/app/11.2.0/grid/cfgtoollogs/opatch/opatch_history.txt

Lsinventory Output file location : /u01/app/11.2.0/grid/cfgtoollogs/opatch/lsinv/lsinventory2013-07-20_10-19-44AM.txt

--------------------------------------------------------------------------------
Installed Top-level Products (1):

Oracle Grid Infrastructure                                           11.2.0.1.0
There are 1 products installed in this Oracle Home.

Interim patches (1) :

Patch  9413827      : applied on Fri Jun 21 13:18:07 BRT 2013
Unique Patch ID:  13136265
   Created on 25 Oct 2010, 07:07:18 hrs PST8PDT
   Bugs fixed:
     9262748, 9343627, 9655006, 9783609, 9413827, 9262722, 9778840

Rac system comprising of multiple nodes
  Local node = oraclebdsrv01
  Remote node = oraclebdsrv02

--------------------------------------------------------------------------------

OPatch succeeded.
[grid@oraclebdsrv01 ~]$

### 10 Repeat on other servers.

Now repeat the previous steps in the next RAC node.

[root@oraclebdsrv02 ~]# mkdir -p /u01/oracle_install/patches/
[root@oraclebdsrv02 ~]# chown -R oracle:oinstall /u01/oracle_install
[oracle@oraclebdsrv01 ~]$ scp /u01/oracle_install/patches/p9413827_112010_Linux-x86-64.zip oracle@oraclebdsrv02:/u01/oracle_install/patches/
p9413827_112010_Linux-x86-64.zip                                                                                                                                    100%  285MB  35.6MB/s   00:08
[oracle@oraclebdsrv01 ~]$

3 - Install the dependency package cvuqdisk.

Updating the grid brings a new package as a dependency: cvuqdisk. Luckily for us, this package is available in the Oracle installation zip.

There is another way to apply this package which is directly by the graphic interface on the screen of dependency check, where Oracle will generate a script in "/tmp/" to run as root. However, I still prefer the manual approach:

[root@oraclebdsrv01 ~]# find /u01/oracle_install/p10404530_112030_Linux-x86-64/grid/ -name "*.rpm"
/u01/oracle_install/p10404530_112030_Linux-x86-64/grid/stage/cvu/cv/remenv/cvuqdisk-1.0.9-1.rpm
/u01/oracle_install/p10404530_112030_Linux-x86-64/grid/rpm/cvuqdisk-1.0.9-1.rpm
[root@oraclebdsrv01 ~]# yum --disablerepo="*" --nogpgcheck install /u01/oracle_install/p10404530_112030_Linux-x86-64/grid/rpm/cvuqdisk-1.0.9-1.rpm
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Examining /u01/oracle_install/p10404530_112030_Linux-x86-64/grid/rpm/cvuqdisk-1.0.9-1.rpm: cvuqdisk-1.0.9-1.x86_64
Marking /u01/oracle_install/p10404530_112030_Linux-x86-64/grid/rpm/cvuqdisk-1.0.9-1.rpm as an update to cvuqdisk-1.0.7-1.x86_64
Resolving Dependencies
--> Running transaction check
---> Package cvuqdisk.x86_64 0:1.0.9-1 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================================================================================================
 Package                                       Arch                                        Version                                      Repository                                              Size
=====================================================================================================================================================================================================
Updating:
 cvuqdisk                                      x86_64                                      1.0.9-1                                      /cvuqdisk-1.0.9-1                                       11 k

Transaction Summary
=====================================================================================================================================================================================================
Install       0 Package(s)
Upgrade       1 Package(s)

Total size: 11 k
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating       : cvuqdisk                                                                                                                                                                      1/2
  Cleanup        : cvuqdisk                                                                                                                                                                      2/2

Updated:
  cvuqdisk.x86_64 0:1.0.9-1

Complete!
[root@oraclebdsrv01 ~]#

Now on the other node:

[root@oraclebdsrv01 ~]#
[root@oraclebdsrv01 ~]# scp /u01/oracle_install/p10404530_112030_Linux-x86-64/grid/rpm/cvuqdisk-1.0.9-1.rpm root@oraclebdsrv02:/tmp/
root@oraclebdsrv02's password:
cvuqdisk-1.0.9-1.rpm                                                                                                                                               100% 8551     8.4KB/s   00:00
[root@oraclebdsrv01 ~]# ssh oraclebdsrv02
root@oraclebdsrv02's password:
Last login: Thu Jun 20 14:54:08 2013 from 10.20.1.2
[root@oraclebdsrv02 ~]# yum --disablerepo="*" --nogpgcheck install /tmp/cvuqdisk-1.0.9-1.rpm
Loaded plugins: rhnplugin, security
This system is not registered with RHN.
RHN support will be disabled.
Setting up Install Process
Examining /tmp/cvuqdisk-1.0.9-1.rpm: cvuqdisk-1.0.9-1.x86_64
Marking /tmp/cvuqdisk-1.0.9-1.rpm as an update to cvuqdisk-1.0.7-1.x86_64
Resolving Dependencies
--> Running transaction check
---> Package cvuqdisk.x86_64 0:1.0.9-1 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=====================================================================================================================================================================================================
 Package                                       Arch                                        Version                                      Repository                                              Size
=====================================================================================================================================================================================================
Updating:
 cvuqdisk                                      x86_64                                      1.0.9-1                                      /cvuqdisk-1.0.9-1                                       11 k

Transaction Summary
=====================================================================================================================================================================================================
Install       0 Package(s)
Upgrade       1 Package(s)

Total size: 11 k
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Updating       : cvuqdisk                                                                                                                                                                      1/2
  Cleanup        : cvuqdisk                                                                                                                                                                      2/2

Updated:
  cvuqdisk.x86_64 0:1.0.9-1

Complete!
[root@oraclebdsrv02 ~]#

4 - Install and upgrade Oracle Grid to the latest version.

We are now ready to call the Grid installer. Starting with version 11.2.0.3, Oracle does not update existing homes, it creates separate homes.

First, let's just create the folders where will be located the new CRS_HOME:

As root, execute on each node:

[root@oraclebdsrv01 ~]# mkdir -p /u01/app/11.2.3/grid/
[root@oraclebdsrv01 ~]# chown grid:oinstall /u01/app/11.2.3/grid/
[root@oraclebdsrv01 ~]#

[root@oraclebdsrv02 ~]# mkdir -p /u01/app/11.2.3/grid/
[root@oraclebdsrv02 ~]# chown grid:oinstall /u01/app/11.2.3/grid/
[root@oraclebdsrv02 ~]#

Calling the installer. As grid, run:

[grid@oraclebdsrv01 ~]$ cd /u01/oracle_install/p10404530_112030_Linux-x86-64/grid/
[grid@oraclebdsrv01 grid]$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 356122 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 95997 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 65536    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-06-20_02-59-13PM. Please wait ...
[grid@oraclebdsrv01 grid]$

The screen below will appear. Choose the option "Skip software updates".

On the next screen, choose "Upgrade Oracle Grid Infrastructure".

Select the desired languages.

All nodes in your cluster should be selected.

Associate the corresponding OS groups.

At this point, you should choose a different directory for your upgrade and installation. In my case, I use the standard "11.2.3".

On the next screen, a summary appears. Check the data and click "Install".

When the upgrade is complete, it will request you to run the following script as root. Do one at a time on both nodes, starting at the node where the installation was called.

As root, run in the first node:

[root@oraclebdsrv01 ~]# /u01/app/11.2.3/grid/rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.3/grid/crs/install/crsconfig_params
Creating trace directory

ASM upgrade has started on first node.

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.crsd' on 'oraclebdsrv01'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.ossdb.db' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.oraclebdsrv01.vip' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.oraclebdsrv01.vip' on 'oraclebdsrv01' succeeded
CRS-2672: Attempting to start 'ora.oraclebdsrv01.vip' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.scan2.vip' on 'oraclebdsrv01' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.scan3.vip' on 'oraclebdsrv01' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'oraclebdsrv02'
CRS-2676: Start of 'ora.oraclebdsrv01.vip' on 'oraclebdsrv02' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'oraclebdsrv02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'oraclebdsrv02'
CRS-2676: Start of 'ora.scan3.vip' on 'oraclebdsrv02' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'oraclebdsrv02'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'oraclebdsrv02' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'oraclebdsrv02' succeeded
CRS-2677: Stop of 'ora.ossdb.db' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.ARCHIVE.dg' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.ARCHIVE.dg' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.asm' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.ons' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.eons' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.ons' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.net1.network' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.eons' on 'oraclebdsrv01' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oraclebdsrv01' has completed
CRS-2677: Stop of 'ora.crsd' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.ctssd' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.evmd' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.asm' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.cssdmonitor' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.asm' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.cssd' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oraclebdsrv01'
CRS-2673: Attempting to stop 'ora.diskmon' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.gpnpd' on 'oraclebdsrv01' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.gipcd' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'oraclebdsrv01' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oraclebdsrv01' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in inittab
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@oraclebdsrv01 ~]#

As root, run in the second (and last) node:

[root@oraclebdsrv02 ~]# /u01/app/11.2.3/grid/rootupgrade.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/11.2.3/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying oraenv to /usr/local/bin ...
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]: y
   Copying coraenv to /usr/local/bin ...

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.3/grid/crs/install/crsconfig_params
Creating trace directory
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.crsd' on 'oraclebdsrv02'
CRS-2790: Starting shutdown of Cluster Ready Services-managed resources on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.LISTENER.lsnr' on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN2.lsnr' on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN3.lsnr' on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.ossdb.db' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.LISTENER.lsnr' on 'oraclebdsrv02' succeeded
CRS-2673: Attempting to stop 'ora.oraclebdsrv02.vip' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.oraclebdsrv02.vip' on 'oraclebdsrv02' succeeded
CRS-2672: Attempting to start 'ora.oraclebdsrv02.vip' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.LISTENER_SCAN2.lsnr' on 'oraclebdsrv02' succeeded
CRS-2673: Attempting to stop 'ora.scan2.vip' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.LISTENER_SCAN3.lsnr' on 'oraclebdsrv02' succeeded
CRS-2673: Attempting to stop 'ora.scan3.vip' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.scan2.vip' on 'oraclebdsrv02' succeeded
CRS-2672: Attempting to start 'ora.scan2.vip' on 'oraclebdsrv01'
CRS-2677: Stop of 'ora.scan3.vip' on 'oraclebdsrv02' succeeded
CRS-2672: Attempting to start 'ora.scan3.vip' on 'oraclebdsrv01'
CRS-2676: Start of 'ora.oraclebdsrv02.vip' on 'oraclebdsrv01' succeeded
CRS-2676: Start of 'ora.scan2.vip' on 'oraclebdsrv01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN2.lsnr' on 'oraclebdsrv01'
CRS-2676: Start of 'ora.scan3.vip' on 'oraclebdsrv01' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN3.lsnr' on 'oraclebdsrv01'
CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'oraclebdsrv01' succeeded
CRS-2676: Start of 'ora.LISTENER_SCAN3.lsnr' on 'oraclebdsrv01' succeeded
CRS-2677: Stop of 'ora.ossdb.db' on 'oraclebdsrv02' succeeded
CRS-2673: Attempting to stop 'ora.ARCHIVE.dg' on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.DATA.dg' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.ARCHIVE.dg' on 'oraclebdsrv02' succeeded
CRS-2677: Stop of 'ora.DATA.dg' on 'oraclebdsrv02' succeeded
CRS-2673: Attempting to stop 'ora.asm' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.asm' on 'oraclebdsrv02' succeeded
CRS-2673: Attempting to stop 'ora.eons' on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.ons' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.ons' on 'oraclebdsrv02' succeeded
CRS-2673: Attempting to stop 'ora.net1.network' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.net1.network' on 'oraclebdsrv02' succeeded
CRS-2677: Stop of 'ora.eons' on 'oraclebdsrv02' succeeded
CRS-2792: Shutdown of Cluster Ready Services-managed resources on 'oraclebdsrv02' has completed
CRS-2677: Stop of 'ora.crsd' on 'oraclebdsrv02' succeeded
CRS-2673: Attempting to stop 'ora.cssdmonitor' on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.ctssd' on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.evmd' on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.asm' on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.cssdmonitor' on 'oraclebdsrv02' succeeded
CRS-2677: Stop of 'ora.evmd' on 'oraclebdsrv02' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'oraclebdsrv02' succeeded
CRS-2677: Stop of 'ora.asm' on 'oraclebdsrv02' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'oraclebdsrv02' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.cssd' on 'oraclebdsrv02' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'oraclebdsrv02'
CRS-2673: Attempting to stop 'ora.diskmon' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.gpnpd' on 'oraclebdsrv02' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'oraclebdsrv02'
CRS-2677: Stop of 'ora.gipcd' on 'oraclebdsrv02' succeeded
CRS-2677: Stop of 'ora.diskmon' on 'oraclebdsrv02' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'oraclebdsrv02' has completed
CRS-4133: Oracle High Availability Services has been stopped.
OLR initialization - successful
Replacing Clusterware entries in inittab
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Started to upgrade the Oracle Clusterware. This operation may take a few minutes.
Started to upgrade the CSS.
Started to upgrade the CRS.
The CRS was successfully upgraded.
Oracle Clusterware operating version was successfully set to 11.2.0.3.0

ASM upgrade has finished on last node.

Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@oraclebdsrv02 ~]#

Nice. Press OK on the screen prompting you to run the scripts. The upgrade was successful.

Finally, upgrade .bash_profile for all users with the variable pointing to the new GRID HOME. Do not confuse with the ORACLE HOME.

To check if your grid is already running in the new version, run the following commands as grid:

[grid@oraclebdsrv01 ~]$ crsctl query crs releaseversion
Oracle High Availability Services release version on the local node is [11.2.0.3.0]
[grid@oraclebdsrv01 ~]$ crsctl query crs activeversion
Oracle Clusterware active version on the cluster is [11.2.0.3.0]

5 - Install and upgrade Oracle Database to the latest version.

Now that we have the Grid in version 11.2.0.3, the next step is to install the database on a new home and then invoke the DBUA to upgrade it.

As oracle, run:

[oracle@oraclebdsrv01 ~]$ cd /u01/oracle_install/p10404530_112030_Linux-x86-64/database/
[oracle@oraclebdsrv01 database]$ ./runInstaller
Starting Oracle Universal Installer...

Checking Temp space: must be greater than 120 MB.   Actual 25835 MB    Passed
Checking swap space: must be greater than 150 MB.   Actual 32724 MB    Passed
Checking monitor: must be configured to display at least 256 colors.    Actual 65536    Passed
Preparing to launch Oracle Universal Installer from /tmp/OraInstall2013-07-17_12-07-58PM. Please wait ...
[oracle@oraclebdsrv01 database]$

On the screen that appears, fill in the data as desired. In my case, I'm not interested in receiving updates via MOS.

When presenting the alert, confirm.

Now select the option "Skip software updates".

At that moment, just pick the option "Install database software only". We will upgrade it in another step.

Check the "Oracle Real Application Clusters" and select all nodes of your cluster.

Now select the desired language.

Choose the same edition as your original database, according to your license.

At this time, the destination folder will be different from where the current Oracle Database is running. I use the standard "11.2.3".

Check the corresponding OS groups.

Wait for the verification requirements. If all goes well, nothing will be missing.

Wait for the summary and click "Install".

The installation will proceed.

After installation, you are prompted to run the script as the root user. Execute on all nodes, one at a time.

As root, run on all nodes:

[root@oraclebdsrv01 ~]# /u01/app/oracle/product/11.2.3/dbhome_1/root.sh
Performing root user operation for Oracle 11g

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/oracle/product/11.2.3/dbhome_1

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The contents of "dbhome" have not changed. No need to overwrite.
The contents of "oraenv" have not changed. No need to overwrite.
The contents of "coraenv" have not changed. No need to overwrite.

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Finished product-specific root actions.
[root@oraclebdsrv01 ~]#

Finally, the success installation screen is displayed.

Nice. Now, upgrade .bash_profile for all users with the variable pointing to the new HOME of ORACLE. Do not confuse with the GRID HOME.

Next, let's run the DBUA to migrate the database to the new home.

As oracle, run:

[oracle@oraclebdsrv01 ~]$ /u01/app/oracle/product/11.2.3/dbhome_1/bin/dbua

The database upgrade screen will appear.

Select the database you want to update.

Wait for verification.

A screen will appear with the necessary recommendations to be made. Execute them before proceeding.

During the upgrade, Oracle recommends some checks to be performed. Below are some examples to help solve them:

-- List and recompile invalid objects:
select * from dba_objects where status<>'VALID';
EXEC UTL_RECOMP.recomp_serial();
EXEC UTL_RECOMP.recomp_parallel(4);

-- Check if there are refreshes of MVIEW ongoing:
select currmvowner, currmvname from v$mvrefresh;

-- Check if there are items in the trash:
select * from dba_recyclebin;
PURGE DBA_RECYCLEBIN;

-- Check for objects dependent to the system-related package DBMS_LDAP:
select * from dba_dependencies where referenced_name='DBMS_LDAP';

-- Check if the parameter 'event' or '_trace_event' are not default and save list of current parameters:
select * from v$parameter where name in ('event','_trace_event');
create pfile=/tmp/pfile.bkp.txt from spfile;

-- Redo system tables statistics:
exec DBMS_STATS.GATHER_FIXED_OBJECTS_STATS; 
exec DBMS_STATS.GATHER_DICTIONARY_STATS;

On the next screen, I do not recommend disabling the archives so you will be able to safe return if necessary.

When the warning appears, proceed with the installation.

Choose ARCHIVE area, if not already specified.

Uncheck the option to configure the Enterprise Manager. It is better to run it later manually.

Check the report and click "Finish".

The upgrade will start.

At the end, click OK. It is advisable to check the generated log files for "ORA-" messages.

The update will show a final report.

Finally, do not forget to copy the tnsnames.ora file, as well as any ".ora" configuration files that is relevant, from the old home to the new.

6 - Apply the latest Patch Set Update in Oracle Database and Grid.

Up to this point, the updating process is done, but it is always good to have all correction patches installed. So, read the article to apply the latest PSU here!

Do not forget to delete the installation files to save space!

 

Have you enjoyed? Please leave a comment or give a 👍!

Leave a Reply

Your email address will not be published.