[kwlug-disc] root filesystem recovery after fscking disaster

[unsolicited]

FWIW, this seems like a worthy kwlug.org blog post.

Chris Irwin wrote, On 06/26/2010 3:20 AM:
> I'm unsure what happened, but some time today I had a filesystem issue.
> 
> I had a few files in /var/lib/dpkg/info that were size 0. I only noticed
> something was amiss as apt refused to work. Interestingly, permissions
> and ownership on these affected files were "?" according to ls. Possibly
> all metadata was missing, but I didn't write it down.
> 
> After booting from external media, I ran an fsck. It fixed many, many
> errors, much than I could count. At the end of it, I have a bunch of
> files in /lost+found. 
> 
> I removed the now proper-looking but still zero-sized files from
> dpkg/info, and verified apt worked. Great, now I have a working system
> that is missing several files from /. Luckily /home and /data were
> unaffected, but unfortunately I don't keep backups of / (sounds like I
> have a job for this weekend). I could easily reinstall, but I decided to
> see if I could recover from this. One of the benefits of using a package
> manager is supposed to be file verification.
> 
> After spending a bit over an hour sifting through a few dozen websites
> with non-working and out-of-date tips, I managed to find a useful one
> detailing a package recovery based on package/file checksums.
> 
> 1. http://arthurdejong.org/recovery.html
> 
> The steps were not perfect, so I had to modify slightly. I'm recording
> here incase anybody else ever comes across the same issue.
> 
> ================== Begin Recovery Steps ==================
> # Install debsums package
> sudo aptitude install debsums
> 
> # Reinstall packages with missing signatures
> sudo aptitude reinstall $( debsums -l )
> 
> # List broken packages into a file. Took about a minute for me.
> sudo debsums -s -a 2> broken-packages
> 
> #Clean up output
> #Note: I needed to manually edit 'restore-packages' to remove
> # a few error messages that were not picked out of my simple regex. 
> # I was hesitant to run the regex from the URL above as it is quite
> # lengthy, and I didn't want to spend more time understanding it than
> # it would take to manually build my package list.
> cat broken-packages  | sed -e 's/.*(from \(.*\) package)/\1/' \
>   | sort | uniq > restore-packages
> 
> #At this point, restore-packages contained five packages. 
> # Had it been >50, you may want to split this next step up.
> sudo aptitude reinstall $(cat restore-packages)
> ==================  End Recovery Steps  ==================
> 
> One thing that concerns me is at the end of his page, he notes that some
> symlinks and files are generated in post-install scripts. Those files
> would not have checksums registered in the package, and would therefore
> still be possibly corrupted or missing. Are there any other steps that I
> can do to find those? I've just about exhausted my google-fu.
> 
> So far, everything *seems* to be working okay...