[kwlug-disc] root filesystem recovery after fscking disaster
chris at chrisirwin.ca
Sat Jun 26 03:20:35 EDT 2010
I'm unsure what happened, but some time today I had a filesystem issue.
I had a few files in /var/lib/dpkg/info that were size 0. I only noticed
something was amiss as apt refused to work. Interestingly, permissions
and ownership on these affected files were "?" according to ls. Possibly
all metadata was missing, but I didn't write it down.
After booting from external media, I ran an fsck. It fixed many, many
errors, much than I could count. At the end of it, I have a bunch of
files in /lost+found.
I removed the now proper-looking but still zero-sized files from
dpkg/info, and verified apt worked. Great, now I have a working system
that is missing several files from /. Luckily /home and /data were
unaffected, but unfortunately I don't keep backups of / (sounds like I
have a job for this weekend). I could easily reinstall, but I decided to
see if I could recover from this. One of the benefits of using a package
manager is supposed to be file verification.
After spending a bit over an hour sifting through a few dozen websites
with non-working and out-of-date tips, I managed to find a useful one
detailing a package recovery based on package/file checksums.
The steps were not perfect, so I had to modify slightly. I'm recording
here incase anybody else ever comes across the same issue.
================== Begin Recovery Steps ==================
# Install debsums package
sudo aptitude install debsums
# Reinstall packages with missing signatures
sudo aptitude reinstall $( debsums -l )
# List broken packages into a file. Took about a minute for me.
sudo debsums -s -a 2> broken-packages
#Clean up output
#Note: I needed to manually edit 'restore-packages' to remove
# a few error messages that were not picked out of my simple regex.
# I was hesitant to run the regex from the URL above as it is quite
# lengthy, and I didn't want to spend more time understanding it than
# it would take to manually build my package list.
cat broken-packages | sed -e 's/.*(from \(.*\) package)/\1/' \
| sort | uniq > restore-packages
#At this point, restore-packages contained five packages.
# Had it been >50, you may want to split this next step up.
sudo aptitude reinstall $(cat restore-packages)
================== End Recovery Steps ==================
One thing that concerns me is at the end of his page, he notes that some
symlinks and files are generated in post-install scripts. Those files
would not have checksums registered in the package, and would therefore
still be possibly corrupted or missing. Are there any other steps that I
can do to find those? I've just about exhausted my google-fu.
So far, everything *seems* to be working okay...
Chris Irwin <chris at chrisirwin.ca>
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 198 bytes
Desc: This is a digitally signed message part
More information about the kwlug-disc