John Kozubik < john @ kozubik . com >

Free Speech or Stone Age

I am passionate about free speech. To that end, I watch closely the actions that state actors take to restrict freedom of speech (as well as their perceived effectiveness).

The explosion of speech on the Internet has been met with increasingly frequent and sophisticated means of monitoring and censorship. Nationwide blackouts in North Korea, government mandated spyware in China, and pinpointed disabling of infrastructure during the Iranian presidential election of 2009, all lead to the conclusion that state actors can control free speech in the Internet era.

That conclusion is wrong.

Modern commerce is so dependent upon information that removing this infrastructure condemns a society to a "stone age". Further, as commerce rapidly evolves, it is impossible to distinguish between the critical and superfluous aspects of this infrastructure, so selective control is impossible. Even if a state actor takes drastic action, the mere existence of portable computers, open wireless networking standards and encryption guarantee the populace access to free speech - even without the Internet.

The choice state actors face is Free Speech or Stone Age. (full paper - 2 pages)

 

The Question of Commerce

I am now writing about the various commercial endeavors that I undertake.  This will consist mainly of technical and procedural questions related to rsync.net, JohnCompanies and Trent (coming soon).

However, it will also consist of writing that explores deeper questions about commerce itself, and the various questions and conflicts that arise from creating and running these organizations.

What is the specific purpose of a business firm ? To what end should it be optimized for ? What are the activities properly suited to this end ? What lifecycle should such a firm describe ?

I am disturbed by the culture of consumption that pervades the developed world and the warped interactions that occur between producer and consumer. I would like to explore how currently accepted practices in production, sales and marketing, and customer support are often manipulative and degrading. I will discuss my efforts at avoiding this in the context of my own business firms.

I am also disturbed by the notion of a business firm as a social actor with anthropomorphic qualities like "conscience" and "duty", and I find it troubling that business firms participate in activities that are typically undertaken by human beings. It has long been my suspicion that these humanist, extra-commercial activities merely serve to balance the many dehumanizing effects of business firms. I would like to explore how a stricter conception of such firms would obviate the need for these extra-commercial activities by removing the dehumanizing effects they are balancing.

I will explore the position of commerce in the Platonic castes of society. Does commerce belong to the lowest conception, or class, of human activity ? Or are there only certain aspects of commerce that are degrading ? Further, the attitudes of various aristocracies toward work, commerce and money deserve some attention as one explores the forms that commerce should and should not take. What are the archetypes of commerce and to what degree should their purity be pursued in the 21st century ?

These writings on commerce in general will be filed under the "Commerce" category, whereas writings specific to a particular firm will be categorized under that firms name.

I look forward to interacting, via comments, on all of these topics with both my customers and the public at large.

 

Improving fsck With Regard to Bad Cylinder Groups

While running fsck in FreeBSD, circa 6.X-RELEASE, you may encounter an error message in the form of:

fsck_ffs: cannot alloc 275829428 bytes for inoinfo

Further, you may find that regardless of which superblock you point fsck to, you continue to see a failure to allocate this very large amount of memory.  This indicates that it is not merely bad superblock information that is causing this problem.

Typically, when fsck complains about being unable to alloc, it is due to a system setting that does not allow the fsck process to grow as large as it would like.  The typical solution is to add a line to /boot/loader.conf which will allow individual processes to grow larger:

kern.maxdsiz="1073741824"

However, the error above contains a ridiculously large value of memory for fsck to be allocating, even on a multi-terabyte filesystem with 100 million or more inodes in use.  A sane error message would be something like:

fsck_ufs: cannot alloc 94208 bytes for inoinfo

So, something is wrong.  The next step is to use the dumpfs command and examine the output:

    # dumpfs /filesystem

If you see output like this, containing more ridiculous numbers, and even negative values:

cg 1685:

magic   ee3082ac        tell    5786e2c000      time    Thu Apr 25

01:34:45 1991

cgx     -1882696659     ndblk   851865734       niblk   1634883043

initiblk 1142699181

nbfree  1250395599      ndir    1940678658      nifree  605792534 nffree

-1948694717

rotor   119584674       irotor  829041313       frotor  738456786

frsum   -1091738562     1268980862      -670423794      593122229

-2060720067     -656729973      -1359818993

sum of frsum: -480435948

Segmentation fault (core dumped)

The problem becomes clear.  A cylinder group (in this case 1685) has been badly corrupted.  fsck naively uses this corrupted information and attempts to allocate memory to accommodate these nonsense values.

The fsck that has been distributed with FreeBSD release versions up to, and including, 6.4-RELEASE, is unable to deal with this problem.  No contingency is made to deal with corrupted cylinder groups.  Further, fsck and dumpfs deal with these cylinder groups naively, following information that is clearly out of bounds, usually with the end result being a segfault and core dump.

There was a patch added to CURRENT some time ago that added a -D option (not to be confused with the -d option for debugging which has been extant for some time) that addresses corrupted cylinder groups:

http://www.kozubik.com/code/patches/2009-01-00-freebsd_fsck_incorrect_cylinder_group_patch

However, as of January 2009, this patch, and the -D option it provides, was implemented incorrectly.  If you clear cylinder groups with this intermediate version of fsck, you will further damage your filesystem and greatly complicate further recovery efforts.

In response to this limitation of fsck in general, Kirk McKusick has rewritten this handling of damaged cylinder groups.  Kirk notes:

"I am going to eliminate the -D option and just make the detection and rebuilding of cylinder groups one of the standard things that fsck does when running in manual (non-preen) mode."

He further notes:

"I plan to incorporate the improvements I made to fsck to rebuild cylinder group maps into 8-current."

So, if you are using a 6.x or 7.x RELEASE version of FreeBSD and are encountering these errors, be sure to avoid the intermediate patch that was extant prior to January of 2009, and seek out the new fsck code that is in 8-current.

In the meantime, for convenience, I have made available this intermediate i386 binary of fsck that can be used on a FreeBSD 6.x RELEASE system.  This binary is a preliminary implementation of the new cylinder group map repair code that Kirk has introduced, and does NOT automatically rebuild cylinder groups - you are required to run this binary with the -D option if you want that:

     # fsck_ffs -y -D /dev/da0

The work that has been done since this binary was created is certainly superior, and will no longer require the -D option, but I make this available just in case somebody needs to run this immediately in 6.x:

fsck binary with intermediate fixes for broken cylinder groups

 

FreeBSD now has 64-bit quotas

In June of 2008, I posted this to the freebsd-fs mailing list:

It's 2008. 1 TB disk drives cost $160. Quotas are 32-bit.

In summary, I was very upset to find that five years later:

FreeBSD bigdisk to-do list

quotas were still 32-bit.

I am happy to report that this is no longer the case. Kirk McKusick has provided patches for both the kernel and the userland tools:

64-bit quotas for FreeBSD 6

64-bit quotas for FreeBSD 7

His instructions are as follows:

 

If you extract the tar file in your /usr directory, everything will be dropped into its appropriate location.

Once the files are in their appropriate locations, you need to rebuild your kernel (ensuring that you have the QUOTA option defined in your kernel configuration file). You will have to build, install, and boot your updated kernel. You then need to build and install the six quota related utilities: sbin/quotacheck, usr.sbin/edquota, usr.sbin/repquota, usr.sbin/quotaon, usr.bin/quota, and libexec/rpc.rquotad.

The first step to using the new quota code is to convert any existing quota files to the new quota file format. Conversion is done by running quotacheck which will automatically convert any old format files that it encounters. BE SURE TO SAVE OLD COPIES OF YOUR QUOTA FILES, as there is no provision for reversing the conversion process. So, if for whatever reason you need/want to revert to the old quota system, you will need to put back your saved copies of the quota files.

After running `quotacheck -a -v' you should then be able to operate quotas in the same way that you are used to doing.

 

I have tested these patches on both 6.x and 7.x using filesystems and quotas larger than three terabytes. This code is now in production use at rsync.net.

Kirk had this to say about the near-term progress of this code into actual FreeBSD releases:

 

I will proceed to check it into 8-CURRENT. I will then apply to get permission to MFC it to 7-STABLE which should be proforma with about a 1-month waiting period. I can also apply for MFC to 6-STABLE, though I expect that request to be turned down as the criterion for MFCs to 6-STABLE is bug fixes only and not enhancements. And this change would be hard to classify as a bug fix.

 

In addition to the $1000 bounty that I will be paying to Kirk, he will receive a lifetime of free rsync.net service. Please join me in thanking him for all of the work that he has done, and continues to do for the FreeBSD community.

It's 2008. 1 TB disk drives cost $160. Quotas are 32-bit.

(posted to the freebsd-fs mailing list this afternoon)

Subject: It's 2008. 1 TB disk drives cost $160. Quotas are 32-bit.

I needed to set a user quota of greater than 2 TB today. I failed, because FreeBSD does not have 64-bit quota tools.

I wasted a fair amount of time trying to track down what I assumed was my own user error. Surely there is _no way_ that an enterprise operating system, in 2008, has 32-bit quotas.

Now I know better.

I am upset to find that several of my non-technical friends now have larger filesystems _in their living rooms_ than FreeBSD can handle with quotas.

Quotas are a long-standing, core piece of filesystem functionality and have been considered a bedrock of unix operating systems for decades. There is nothing new or experimental in moving quotas from 32 to 64 bit.

This is _as opposed to_ porting ZFS to FreeBSD, and gjournal, and every other shiny bauble that has monopolized freebsd-fs in the last four years. Those are new. Those are experimental.

Apparently those take priority.

I don't have time to monitor the core pieces of FreeBSD to make sure _they still exist_. Further, while I might have volunteered to help with the code back in 2004, when it took 5 hard drives to max out the usefulness of the filesystem, that's not how I'll be spending my time in 2008.

So I'll try this instead:

I will paypal $1000 to whoever can deliver fully clean 64-bit quotas and userland tools in FreeBSD by July 20, 2008.

That is, if you can tear yourself away from ZFS and whatever sexy SMP improvements you're building into FreeBSD 14.0 for a week.

High Definition Content Playback on the Intel Mac Mini

One of the FAQs regarding the current Intel Mac Minis is "will it play HD video", or "will it play 1080p video".  The answers have been all over the place, from emphatic "yes" and "no" answers to hazy "maybes" with few real details.

It's important to have a good answer to this question because many, many people are interested in the Mac Mini as a HTPC.  There is little value, however, in spending time and money on a platform if it isn't powerful enough to run ALL high definition content that one might encounter.

In this post I will discuss how to measure high-definition playback in a useful way, what problems people are currently having with high definition playback, and why the current Mac Mini is probably fast enough to handle all currently available high definition content.

Bounty for Flash9 in Opera in FreeBSD

It's well known that I use FreeBSD as a platform at both JohnCompanies and rsync.net. However, it may be surprising to learn that I also use it as my desktop and laptop operating system.

In the eight years that FreeBSD has been my operating environment, I have had surprisingly few issues with hardware compatibility or application support. This is not surprising, as I run almost nothing but xterms and Opera.

However, two issues are becoming more and more difficult to work around:

1. I need an up to date vmware. I have been using an extremely old Vmware 3 license in Linux binary compatibility mode since 2002. Although I continue to make do with this old version today, I would really like a modern version of Vmware - especially now that Vmware 6 has 3d virtualization.

2. I need Flash 9 to work with Opera. The mob has spoken and apparently every site on the web needs yet another unnecessary layer of complexity and gimmickry, and I just can't work without flash anymore. I am currently using linux-flashplugin-7.0r69 with Opera 9.20 but there are more sites every day that require at least Flash 8.

The first problem was addressed earlier this year when I included "Vmware 6.x on FreeBSD" in the 2007 rsync.net Code Bounties. At the time of this writing, Orlando Bassotto, who was responsible for the original Vmware3 port in FreeBSD, has taken on the project and reports that it is moving forward. I hope to have Vmware 6.x running on FreeBSD (in Linux binary compatibility mode, of course) in the first quarter of 2008.

I will now address the second problem here with a new bounty:

I will pay $200 to whoever can compose a working and stable recipe for running Adobe Flash 9 inside of the FreeBSD native version of Opera 9 on FreeBSD 6.x. This shouldn't be that hard - in fact, there is already a linux-flashplugin9 port. The trouble is, even if you do convince your browser to use the plugin, it crashes frequently and generally "doesn't work". I think a good proof of success would be the ability to play arbitrary content on YouTube without complication, or perhaps use all of google maps / google finance without crashing.

If you would like to contribute funds to the bounty, add your name, email address and a dollar amount as a comment below. The more money there is in the pot, the faster things get done...