System Rescue CD to the.. rescue!


Here’s the scenario..  It’s 1 am and I have to shut down a critical linux server to relocate it in a rack to make room for new equipment. It should have been a 5 minute job.. but on powering up the server it refused to boot past printing the word ‘Grub‘ on the screen.  This isn’t good..  this server is needed by a couple hundred thousand customers and rebuilding it wasn’t planned or scheduled.  On closer examination 3 of the 16 hard drive power lights are not on. It’s extremely unlikely that 3 drives would die like that on a server that isn’t even two years old.  Unfortunately I didn’t have a copy of the the System Rescue CD so the fix attempt would have to wait until morning.

I had the CoLo staff burn me a copy which I used to boot the damaged server the next morning.  It booted into a live linux environment and correctly detected all the server hardware.. including the raid controller.  I was able to check the status of the 3 raid arrays and found them to be all in working order.. the 3 dark drive lights were unrelated.  I was then able to chroot into the broken system and reinstall grub onto the primary disk. The server then booted normally and all was well. I still don’t know how or when the MBR got corrupted.. but thanks to the utility of the RescueCD this was an easy fix.