How to Backup and Restore a RAC Database (Complete Recovery)
Backup the Database
Use the method you prefer. I used a NOCATALOG, disk-based backup for this example, using the scripts found here.
Simulate the Failure
Shutdown the database and simulate a failure by dropping and/or corrupting a datafile.
# export ORACLE_SID=em1# . oraenv# srvctl stop database -d em# srvctl status database -d emInstance em1 is not running on node rac1Instance em2 is not running on node rac2
# cd /u02/oracle/ora1/em# rm sysaux01.dbfVerify the Failure
# srvctl start instance -d em -i em1
PRKP-1001 : Error starting instance em1 on node rac1CRS-0215: Could not start resource 'ora.em.em1.inst'.You will find something similar to the following in the instance’s alert log:
Sat May 24 16:53:47 2008Errors in file /u02/oracle/logs/em/bdump/em1_dbw0_16947.trc:ORA-01157: cannot identify/lock data file 3 - see DBWR trace fileORA-01110: data file 3: '/u02/oracle/ora1/em/sysaux01.dbf'ORA-27037: unable to obtain file statusLinux Error: 2: No such file or directoryAdditional information: 3ORA-1157 signalled during: ALTER DATABASE OPEN...Sat May 24 16:53:49 2008Shutting down instance (abort)Restore the Database
First, take the database out of cluster mode via:
# sqlplus "/ as sysdba" SQL*Plus: Release 10.2.0.3.0 - Production on Sat May 24 17:02:17 2008 Copyright (c) 1982, 2006, Oracle. All Rights Reserved. Connected to an idle instance. SQL> startup nomount;
ORACLE instance started. Total System Global Area 268435456 bytes
Fixed Size 1261300 bytes
Variable Size 251658508 bytes
Database Buffers 8388608 bytes
Redo Buffers 7127040 bytes
SQL> alter system set cluster_database=false scope=spfile sid='*';
System altered.
SQL> shutdown immediate;
ORA-01507: database not mountedORACLE instance shut down.Then restore the database via RMAN:
# rman target=/ Recovery Manager: Release 10.2.0.3.0 - Production on Sat May 24 17:04:10 2008 Copyright (c) 1982, 2005, Oracle. All rights reserved. connected to target database (not started) RMAN> startup mount; Oracle instance starteddatabase mounted Total System Global Area 268435456 bytes Fixed Size 1261300 bytes
Variable Size 251658508 bytes
Database Buffers 8388608 bytes
Redo Buffers 7127040 bytes
RMAN> restore database;
Starting restore at 24-MAY-08 using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1channel ORA_DISK_1: sid=154 devtype=DISK channel ORA_DISK_1: starting datafile backupset restorechannel ORA_DISK_1: specifying datafile(s) to restore from backup setrestoring datafile 00001 to /u02/oracle/ora1/em/system01.dbfrestoring datafile 00002 to /u02/oracle/ora1/em/undotbs01.dbfrestoring datafile 00003 to /u02/oracle/ora1/em/sysaux01.dbfrestoring datafile 00004 to /u02/oracle/ora1/em/users01.dbfrestoring datafile 00005 to /u02/oracle/ora1/em/example01.dbfrestoring datafile 00006 to /u02/oracle/ora1/em/undotbs02.dbfrestoring datafile 00007 to /u02/oracle/ora2/em/mgmt.dbfrestoring datafile 00008 to /u02/oracle/ora1/em/mgmt_ecm_depot1.dbfchannel ORA_DISK_1: reading from backup piece /u02/oracle/ora3/em/backups/0ijh6j4t_1_1channel ORA_DISK_1: restored backup piece 1piece handle=/u02/oracle/ora3/em/backups/0ijh6j4t_1_1 tag=FHB_EM1200805241630channel ORA_DISK_1: restore complete, elapsed time: 00:06:36Finished restore at 24-MAY-08Since my database is so small, I will simply restore the entire thing, however, since you know which datafile is missing, you could simply restore it and then recover the database as necessary.
Recover the Database
RMAN> recover database;
Starting recover at 24-MAY-08using channel ORA_DISK_1
starting media recoverymedia recovery complete, elapsed time: 00:00:03
Finished recover at 24-MAY-08
RMAN> alter database open;
database opened
RMAN-06900: WARNING: unable to generate V$RMAN_STATUS or V$RMAN_OUTPUT row
RMAN-06901: WARNING: disabling update of the V$RMAN_STATUS and V$RMAN_OUTPUT rowsORACLE error from target database:ORA-06502: PL/SQL: numeric or value error: character to number conversion errorAfterwards, place the database back into cluster mode and startup both instances:
# sqlplus "/ as sysdba"
SQL*Plus: Release 10.2.0.3.0 - Production on Sat May 24 17:16:36 2008
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Connected to:Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - ProductionWith the Partitioning, Real Application Clusters, OLAP and Data Mining options
SQL> alter system set cluster_database=true scope=spfile sid='*';
System altered.
SQL> shutdown immediate;
Database closed.
Database dismounted.
ORACLE instance shut down.
# srvctl start database -d em
[oracle@rac1 bdump]$ srvctl status database -d em
Instance em1 is running on node rac1
Instance em2 is running on node rac2
