“All technology should be assumed guilty until proven innocent.”
-David Brower
I searched for quite a while to find a quote that summed up my feelings about my recent experience with a full disaster recovery test. I went into this test with measured confidence and was whipped back to reality within the first 2 hours. The technology was not innocent and neither was I.
Encryption – Secured from external threats but what about your own stupidity?
Yes, backup encryption was the first thing to bite me in the ass during the test. Note to self, when setting up the encryption pass phrases, check and double check that you documented them perfectly. I think I may have mis-typed one of the pass phrases so the backup server restore procedure failed. Luckily for me, I was able to remote in to our main office and copy over the encryption database so that I could proceed with the test. In a real disaster, we would have been stopped dead in our tracks right here. I now have to re-build the encryption restore procedure from scratch with new pass phrases, since I can’t know the correct pass phrases reliably in my old setup. I am also keeping a copy of the encryption database in a safe location (just in case).
One Tape in, One Tape Out
We backup to one tape drive. While this meets our needs on a daily basis (barely), restoring is another matter. We spent almost 18 hours waiting for data to restore from tape. Each job had to run serially since the DR site only had one tape drive as well. We we’re able to restore most of the servers, however, some of the larger archive servers had to be skipped due to time constraints. We are completely re-doing our backup and restore scheme in 2010 so our DR tape restore efficiency will be greatly improved.
Independent Disks, Not So Independent
Apparently, when a virtual disk in a VMware Infrastructure environment is set as Independent, Virtual Consolidated Backup (VCB) ignores the disk when backing up the virtual machine. Of course, Netbackup does not report an error and shows a successful backup. We found out halfway through the test that some of our machines would not fully restore due to this issue. One of them ended up containing very important timekeeping information. Post test, I have done a full visual inspection on all virtual disk configuration in our environment and have turned off the Independent Disk setting. We are now getting proper backups of these problem virtual machines.
Not All of it Was Smoke, Fire and Ashes
Besides the 3 problems previously mentioned, this DR test was a success. Most of our hosts were restored successfully and most of our procedures we’re validated. Of course, the most important lesson learned and the purpose of this exercise was that we found out where our shortcomings we’re and are on the path of fixing them. Personally, I am happy to have done this test because it exposed me to an aspect of business planning that I have not yet experienced. Disaster Recovery felt like a giant mountain that I could never hope to scale to something a bit more manageable but still very formidable.
Related posts:
