An RMAN restore from Hell!

We were recently tasked with cloning a production environment to a UAT server using an RMAN backup, sounds straight forward enough right? Time constraints were very tight, so there was quite a rush to sort out copying the backups across to the UAT server, etc. The restore was to be completed on a UAT environment we had not had access to before, but there was already a previous clone from a production UAT environment on this server, so this should have been straightforward restore task (wrong assumption).

The first restore attempt was done using the RMAN Duplicate command and we found that we couldn’t set the parameter ‘MEMORY_TARGET’ to under 50G and, as there was only 20G of memory on this UAT server, the memory was maxing out once the database restore was started. After several attempts at tweaking parameters we found that someone had set the ‘db_cache_size’ parameter in production to 40G; so we reset this to 0 in the RMAN spfile clause as well as reducing the ‘MEMORY_TARGET’ parameter to 15G.

The RMAN restore was finally ticking along until it hit the following error:

RMAN-03002: failure of Duplicate Db command
RMAN-05501: aborting duplication of target database
RMAN-05541: no archived logs found in target database

We attempted to resolve this by adding the ‘NOREDO’ clause to the backup location and started the process all over again. The theory being that if archived logs were required then we could manually apply these as part of the recovery (wrong assumption). The RMAN restore was now cooking along nicely again, until at about 3am where the server ran out of disk space. It appears that the assumption that we could clone over the previous UAT environment was a mistake and the server was greatly under spec’d, contrary to what we were told. So we contacted the client and got them to add the required disk space and then started the process all over, again!

24 hours later and finally the restore has completed. Unfortunately somewhere along the line (assumingly from the ‘NOREDO’ clause) the target UAT database had been placed into ‘NOARCHIVELOG’ mode so a consistent recovery is not possible, Arrrgghhhh!

We decided to start all over again but instead of use the RMAN duplicate command, we followed the more manual standard recovery process, e.g. restore the control file, catalog the backups, set NEWNAME for the relevant datafile path changes, restore database, etc. Finally, we were set for a clean recovery until the server decides to automatically reboot in the early hours of the morning, due to the OS applying updates. At this point, the remaining hair is now pulled out of my head! This time instead of starting from scratch decide to kick off the same restore command as previously, as Oracle is intelligent enough to know that if an existing file header is as expected then it will not attempt to overwrite it, or in other words if the file already exists it will skip it by default unless the ‘FORCE’ option is included.

Fast forward a few hours, and yet again, we discovered another problem. Although the RMAN restore command hadn’t errored it appeared it wasn’t really progressing. Looking in the alert log found that it has raised the following error:

ORA-19505: failed to identify file “….”
ORA-27041: unable to open file
OSD-04002: unable to open file
OS/Error: (OS 3) The system cannot find the path specified

We then noticed that despite using the same script as before the server reboot, the RMAN command was attempting to read backups from the non-existent backup directory which is the same as the one used in prod. Odd indeed, we therefore stopped the restore, and ran a crosscheck of the backups. So the only backups then known to the UAT restore were the ones that we added using the RMAN catalog command and started the restore process again using the same script as previously (not deleting any files).

48 hours later and we were finally set for a clean restore of the backup, but again, another spanner decided to be thrown into the works!

Find out how this epic RMAN restore finished by contacting us today!

Colin Smith

Contact Us