System Rescue CD to the.. rescue!

Jared Watkins — Fri, 18 Feb 2011 00:00:00 +0000

Here’s the scenario.. It’s 1 am and I have to shut down a critical linux server to relocate it in a rack to make room for new equipment. It should have been a 5 minute job.. but on powering up the server it refused to boot past printing the word ‘Grub‘ on the screen. This isn’t good.. this server is needed by a couple hundred thousand customers and rebuilding it wasn’t planned or scheduled. On closer examination 3 of the 16 hard drive power lights are not on. It’s extremely unlikely that 3 drives would die like that on a server that isn’t even two years old. Unfortunately I didn’t have a copy of the the System Rescue CD so the fix attempt would have to wait until morning.

I had the CoLo staff burn me a copy which I used to boot the damaged server the next morning. It booted into a live linux environment and correctly detected all the server hardware.. including the raid controller. I was able to check the status of the 3 raid arrays and found them to be all in working order.. the 3 dark drive lights were unrelated. I was then able to chroot into the broken system and reinstall grub onto the primary disk. The server then booted normally and all was well. I still don’t know how or when the MBR got corrupted.. but thanks to the utility of the RescueCD this was an easy fix.

Disaster-Recovery on The Infinite Unknown

System Rescue CD to the.. rescue!