[kwlug-disc] root filesystem recovery after fscking disaster

[Chris Irwin]

I'm unsure what happened, but some time today I had a filesystem issue.

I had a few files in /var/lib/dpkg/info that were size 0. I only noticed
something was amiss as apt refused to work. Interestingly, permissions
and ownership on these affected files were "?" according to ls. Possibly
all metadata was missing, but I didn't write it down.

After booting from external media, I ran an fsck. It fixed many, many
errors, much than I could count. At the end of it, I have a bunch of
files in /lost+found. 

I removed the now proper-looking but still zero-sized files from
dpkg/info, and verified apt worked. Great, now I have a working system
that is missing several files from /. Luckily /home and /data were
unaffected, but unfortunately I don't keep backups of / (sounds like I
have a job for this weekend). I could easily reinstall, but I decided to
see if I could recover from this. One of the benefits of using a package
manager is supposed to be file verification.

After spending a bit over an hour sifting through a few dozen websites
with non-working and out-of-date tips, I managed to find a useful one
detailing a package recovery based on package/file checksums.

1. http://arthurdejong.org/recovery.html

The steps were not perfect, so I had to modify slightly. I'm recording
here incase anybody else ever comes across the same issue.

================== Begin Recovery Steps ==================
# Install debsums package
sudo aptitude install debsums

# Reinstall packages with missing signatures
sudo aptitude reinstall $( debsums -l )

# List broken packages into a file. Took about a minute for me.
sudo debsums -s -a 2> broken-packages

#Clean up output
#Note: I needed to manually edit 'restore-packages' to remove
# a few error messages that were not picked out of my simple regex. 
# I was hesitant to run the regex from the URL above as it is quite
# lengthy, and I didn't want to spend more time understanding it than
# it would take to manually build my package list.
cat broken-packages  | sed -e 's/.*(from \(.*\) package)/\1/' \
  | sort | uniq > restore-packages

#At this point, restore-packages contained five packages. 
# Had it been >50, you may want to split this next step up.
sudo aptitude reinstall $(cat restore-packages)
==================  End Recovery Steps  ==================

One thing that concerns me is at the end of his page, he notes that some
symlinks and files are generated in post-install scripts. Those files
would not have checksums registered in the package, and would therefore
still be possibly corrupted or missing. Are there any other steps that I
can do to find those? I've just about exhausted my google-fu.

So far, everything *seems* to be working okay...

-- 
Chris Irwin <chris at chrisirwin.ca>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 198 bytes
Desc: This is a digitally signed message part
URL: <http://kwlug.org/pipermail/kwlug-disc_kwlug.org/attachments/20100626/04474daa/attachment.sig>