Monday, May 25, 2009

Doomsday Averted

Ok, first of all. Yes I know I should have had backups, but I didn't.

My file server, hidden away in the shed, had been randomly rebooting lately. All hardware tests checked out OK, but it always recovered after a reboot and only happened every other week, so I just ignored it. Until FreeBSD-7.2 hit an official release, and it rebooted every time I ran csup. I pulled it out of the shed and hooked it up to the KVM switch in the study so I could investigate what was happening.

It took some time to replicated, but when it did, it was immediately obvious what was wrong; "ufs_dirbad", followed by a panic. The file system had somehow become corrupted, so immediately I unmount the drive and naively run fsck -y and remount it.

Disaster!!! The file system was empty, years of photos, backups of our DVD collection (some of which the original DVDs no longer exist, thanks to the kids), the entire music collection. In all 400Gb of data, gone. Needless to say, when the wife found out, she was a little freaked out. I tried to calm her down, explaining that the data was still there, but just not accessible under conventional means. It seemed to work, I was putting all bets on that I'd lost a few inode references, and praying that I hadn't managed scramble the data along with it.

4 years working as a technician has taught me how to pretend to know what you're doing, fake it till you make it and that if you google hard enough you'll eventually find someone with the same problem, or even just a clue to fix it yourself. After about an hour, I find the clue I was looking for, fsdb.
Fsdb opens fsname (usually a raw disk partition) and runs a command loop allowing manipulation of the file system's inode data.
I began manually inspecting each individual inode, and resigned to the fact that this was a 500Gb hard drive and was going to possibly take a very long time. Fortunately, when I hit inode 128, there was my filesystem. I linked back onto the root inode, remounted the filesystem and there it was, all my precious media, all my music, all my anime, all my DVDs, my entire Console and MAME ROM collection. I suppose the family photos and the wedding video were important too.

I immediately scrounged together some spare hard drives, dumped the data and reformatted the disk. Now to convince the wife to let me by some more hard drives ;)