Improving fsck With Regard to Bad Cylinder Groups

While running fsck in FreeBSD, circa 6.X-RELEASE, you may encounter an error message in the form of:

fsck_ffs: cannot alloc 275829428 bytes for inoinfo

Further, you may find that regardless of which superblock you point fsck to, you continue to see a failure to allocate this very large amount of memory.  This indicates that it is not merely bad superblock information that is causing this problem.

Typically, when fsck complains about being unable to alloc, it is due to a system setting that does not allow the fsck process to grow as large as it would like.  The typical solution is to add a line to /boot/loader.conf which will allow individual processes to grow larger:

kern.maxdsiz="1073741824"

However, the error above contains a ridiculously large value of memory for fsck to be allocating, even on a multi-terabyte filesystem with 100 million or more inodes in use.  A sane error message would be something like:

fsck_ufs: cannot alloc 94208 bytes for inoinfo

So, something is wrong.  The next step is to use the dumpfs command and examine the output:

    # dumpfs /filesystem

If you see output like this, containing more ridiculous numbers, and even negative values:

cg 1685:

magic   ee3082ac        tell    5786e2c000      time    Thu Apr 25

01:34:45 1991

cgx     -1882696659     ndblk   851865734       niblk   1634883043

initiblk 1142699181

nbfree  1250395599      ndir    1940678658      nifree  605792534 nffree

-1948694717

rotor   119584674       irotor  829041313       frotor  738456786

frsum   -1091738562     1268980862      -670423794      593122229

-2060720067     -656729973      -1359818993

sum of frsum: -480435948

Segmentation fault (core dumped)

The problem becomes clear.  A cylinder group (in this case 1685) has been badly corrupted.  fsck naively uses this corrupted information and attempts to allocate memory to accommodate these nonsense values.

The fsck that has been distributed with FreeBSD release versions up to, and including, 6.4-RELEASE, is unable to deal with this problem.  No contingency is made to deal with corrupted cylinder groups.  Further, fsck and dumpfs deal with these cylinder groups naively, following information that is clearly out of bounds, usually with the end result being a segfault and core dump.

There was a patch added to CURRENT some time ago that added a -D option (not to be confused with the -d option for debugging which has been extant for some time) that addresses corrupted cylinder groups:

http://www.kozubik.com/code/patches/2009-01-00-freebsd_fsck_incorrect_cylinder_group_patch

However, as of January 2009, this patch, and the -D option it provides, was implemented incorrectly.  If you clear cylinder groups with this intermediate version of fsck, you will further damage your filesystem and greatly complicate further recovery efforts.

In response to this limitation of fsck in general, Kirk McKusick has rewritten this handling of damaged cylinder groups.  Kirk notes:

"I am going to eliminate the -D option and just make the detection and rebuilding of cylinder groups one of the standard things that fsck does when running in manual (non-preen) mode."

He further notes:

"I plan to incorporate the improvements I made to fsck to rebuild cylinder group maps into 8-current."

So, if you are using a 6.x or 7.x RELEASE version of FreeBSD and are encountering these errors, be sure to avoid the intermediate patch that was extant prior to January of 2009, and seek out the new fsck code that is in 8-current.

In the meantime, for convenience, I have made available this intermediate i386 binary of fsck that can be used on a FreeBSD 6.x RELEASE system.  This binary is a preliminary implementation of the new cylinder group map repair code that Kirk has introduced, and does NOT automatically rebuild cylinder groups - you are required to run this binary with the -D option if you want that:

     # fsck_ffs -y -D /dev/da0

The work that has been done since this binary was created is certainly superior, and will no longer require the -D option, but I make this available just in case somebody needs to run this immediately in 6.x:

fsck binary with intermediate fixes for broken cylinder groups

 

Comments

[maple story mesos]

Glad to help. Can't tell you how many times ramdom posts like this have saved mine!

[nx]

John,

Before I can use your good work... can you please post a little howto on using the new fsck on 6.2 for noobs like me?

I have one partition /dev/ad0s1d (/var) that is experiencing these errors when trying to map an alternate super block to it:
fsck_ffs: cannot alloc 2155905152 bytes for inoinfo
and a segmentation fault when running dumpfs.

I have no idea how to 'mount/install/use' the fsck you've provided in Single User Mode that the drive is booting to due to the super block error, but need to as its uber important that I get /var fixed and the FreeBSD install on it booting normally.

Before finding your post I asked for help at:

http://forums.freebsd.org/showthread.php?t=16984

I will link back to your blog post once I have fixed the error.

This is AWESOME work,

thank you!