John Kozubik < john @ kozubik . com >

With friends like these...

I have been an EFF supporter for a very long time and am excited about the work that they do. To a lesser degree, I am interested in, and support sister organizations of the EFF, such as Electronic Frontier Finland.

Imagine my surprise then, to find this quote of Vili Lehdonvirta while reading an article on bitcoin in the October 10, 2011 issue of the New Yorker:

Still, Lehdonvirta had researched bitcoin and was worried about it. 'The only people who need cash in large denominations right now are criminals ... only anarchists want absolute, unbreakable financial privacy ... we need to have a back door so that law enforcement can intercede.'"

Vili Lehdonvirta is identified in the article as serving on the advisory board of EFFI, although I cannot confirm that. It is clear from their archives and publications that he is closely related to their organization.

He should not be. The nuance of his comments, and the distinctions he makes, are false, and are indistinguishable from the worst forms of statist, authoritarian presumption. He has no business awarding "Big Brother Awards" nor does he have any foundation on which to judge the cash, or privacy needs of others.

 

Peak Internet

In light of current events, I think it is time to begin discussing "Peak Internet" in the same way that we discuss "Peak Oil" or "Peak Credit".

The last two weeks have included Comcast potentially de-peering due to end user traffic, the DNS system compromised with questionable legal authority and a a popular website forced out of US infrastructure.

I believe that 2010 represents the "peak" of the homogenous, globally routed Internet as we have known it. For what it's worth, we're guilty of some fragmentation of our own ...

 

Ignoring The Kill Switch

Much has been said recently of Cyber Warfare and the existential threat it represents.

Hopefully you have been able to ignore this ridiculous meme and the clownish "experts" propagating it.

Unfortunately, providing "solutions" for these problems is far too lucrative a market opportunity to be left unexploited, so you should expect to see this meme grow, and for the proposed policies to become more and more radical.

Enter the kill switch.

Like the more general "cyber warfare" meme, the Internet Kill Switch is demonstrably absurd. Will wireless networking equipment be confiscated ? Will private peering interconnects within datacenters be disconnected ? Will IP traffic over cellular or CATV links be disabled ? The understanding of those that propose such an idea probably begins and ends at the word "switch".

Derision and disgust, however, are not ingredients of a good business continuity plan. Although it appears unlikely to be implemented soon, the (bad) policy has been proposed, and regardless of our technical "understanding", we as service providers must respond and fit our offerings to this possible reality.

And so, in Q1 of 2011, rsync.net will provide a wireless private network at each of its physical locations (San Diego California, Denver Colorado, Zurich Switzerland, and Hong Kong) which will allow customers to physically approach the datacenter and access their stored data without traversing The Internet. This will ensure that bad policy will have to enter the stone age to keep you from accessing your data.

Full technical details will be published on our web site before these wireless links are active. Capabilities will include the ability to tunnel through to other rsync.net locations. So, for instance, if you live in California and store your data in our Zurich location, we may still have routing across the ocean even if you do not - we're right next to our backbone providers, and have several of them, after all. So while we will not transform ourselves into an ISP for you, we will maintain whatever VPNs we can to allow you to reach your data, regardless of which physical location you may be at.

As we've pointed out before, we are not an ISP. We announce today that we are not even an Internet data storage provider ... we are simply a data storage provider and will work with our customers to provide them service regardless of political inefficiencies.

 

The Warrant Canary in 2010 and Beyond

We have been publishing our Warrant Canary weekly at rsync.net for almost five years now. We are happy to report that in this time, no warrants of any kind have been served to us.

Unfortunately, it does not appear that the National Security Letter as an abusive surveillance tool is becoming any less widespread. While the Doe v. Ashcroft ruling was an encouraging step, the reality is that the NSL is alive and well. To wit:

The USA PATRIOT Act is affected only if the limits on NSLs in terrorism cases also apply to non-terrorism cases such as those authorized by the Act. The government was expected to appeal the ruling to the Supreme Court, and until the district court ruling is reviewed, the secrecy procedures of the NSL remain in place.

Further, the limitations of the NSL, and of the USA PATRIOT act in general, are slowly being worked around by successive presidential and agency administrations:

Wrong Direction on Privacy (Huffington Post)

Breaking a Promise on Surveillance (New York Times)

And so we will continue to publish the Warrant Canary. Further, we will begin mirroring our Warrant Canary at our international locations (currently Switzerland and Hong Kong). This will address the most common complaint: that when the rubber hits the road, conventional and unconventional means of coercion could be used to compel persons at our firm to publish false statements. Restriction of our speech will now require cooperation across multiple continents, cultures, and languages.

 

Related to this, let me say that I have never thought of rsync.net as an Internet Service Provider. We have a technical philosophy that places a high value on simplicity and we have constructed our data storage platform with as few processes and services running as possible.

We believe that stability and security are greatly enhanced by keeping the list of moving parts as small as possible.

There are legal and privacy related benefits to this philosophy. We do not currently fit the legal definition of an ISP, and we will take steps to ensure that we continue to fall outside of data collection measures that specifically relate to ISPs

There is some cost associated with this. For instance, we have considered implementing an "email to file" service that allows our customers to email attached files into their rsync.net filesystems. One of our competitors, DropBox, has such a feature. We have eschewed this, however, as providing email service (even in this non-standard way) takes us a big step closer to being classified as an ISP.

 

Finally, I will address the FAQ: "what will happen if you do receive a warrant ?"

The answer is, we will publish the receipt of the warrant, and perhaps the warrant itself, provided that it is not a "secret" warrant. So the current verbage of the warrant canary:

No warrants have ever been served to rsync.net, or rsync.net principals or employees. No searches or seizures of any kind have ever been performed on rsync.net assets

would be changed to read:

rsync.net has received the following warrants during operation:
2010-09-14 Warrant served by Sevier County Court, Sevier County, UT, compelled rsync.net to provide confirmation that John Q. Public was not an rsync.net customer.

NO OTHER warrants have ever been served to rsync.net

As always, a secret warrant causes the canary to cease updating until such time as that warrant can be listed. I hope that this makes clear that the mere receipt of a warrant or order of any kind, which I am surprised we have never received, does not necessarily "kill" the warrant canary.

 

I will conclude by reiterating a key component of our Terms of Service:

No consumer or personal information about our customers of any kind will be divulged to any party for any reason.

It's amazing how simple things can be...

 

Moving (back) to the Finder

I've been running an HTPC of some kind or another for almost ten years now, and my user interface has finally come (almost) full circle.

In the very beginning I had a Windows laptop attached to a television and the user interface was the Windows 98 Explorer. There was no skinning, no scraping and no metadata database. I browsed filenames, double clicked them, and they played.

Then, starting in the fall of 2003, I began using XBMP and then migrated to XBMC - both on the original XBOX hardware. This was attractive because the XBOX was an instant on/off device with a television-ready video output (unlike a PC). (In fact, I would say that XBMC, on an XBOX, right after the spring of 2004 when DVD ISO support was added, was the high water mark of the HTPC. It played everything, there was no on-demand ecosystem (like Netflix) to take part in, and there was almost no high bitrate content that the XBOX CPU could not handle).

Unfortunately, this sweet spot only lasted for about 6-12 months. Every day more and more "high definition" content was extant, and the XBOX did not have the CPU horsepower to play it.

The next step for me (after procrastinating for a year or so) was a mac mini based HTPC which required some form of "HTPC software" - not because I wanted such an interface, but because of the codec landscape, which Quicktime was ill equipped to deal with. If you wanted to play the file formats that you would find in the wild, you needed something more than Quicktime.

So I moved to "HTPC Software" because I needed the codecs, and ended up following the entire arc of OSXBMC and Plex and later returned to the mainline XBMC on OSX.

Through it all though, I never set up any scraping, never installed any plug-ins, and metadata caching (especially of tens of thousands of mp3 files accessed over a network) never did anything but paralyze my system. In fact, all I ever did, from XBMP on the XBOX to Plex on the Mac Mini, was choose filenames and play them.

And so with a recent OS reload on my Mac Mini HTPC, I skipped that software layer entirely. I use the Finder to choose files, and they play in QuickTime Pro with the Perian components. For DVD ISOs, I simply double-click them and after they have mounted, start the built-in OSX DVD Player application.

Finally, for live, or on-demand content, I just use Safari.

So, instead of a Netflix plug-in, like one uses with Plex:

 

 

I just use my web browser (Safari):

 

 

The only downside so far is that I have a fair number of MPEG Transport Stream (.ts) files that are not handled by Quicktime in any way, regardless of plug-ins. I can live with this, as they usually had playback glitches in XBMC/OSXBMC/Plex anyway. I'll just convert them to MPEG-2.

I use a bluetooth Apple Mighty Mouse to control the HTPC along with an Apple Wireless Keyboard

The Spaces virtual desktop interface in OSX makes this work very well - I simply run a full screen Finder window in one space, and a full screen Safari window in another, and have configured the two side buttons (the "squeeze" gesture) on the mouse to engage Spaces.

Other than Perian, no non-default applications need to be added to my OSX HTPC. This is very appealing to me, and more then makes up for the loss of a "sophisticated" user interface. In fact, it is Perian that makes all of this possible, as the default codec support in Quicktime is not sufficient by any means, and it is important to note that this "back to basics" HTPC will only work so long as Perian keeps up with the current codec landscape. It is for this reason that I encourage you to donate to their project.

 

Git and Subversion Support at rsync.net

At long last, git is supported at rsync.net.

We wrestled with the decision to add it for some time, as we place a very, very high value on the simplicity of our systems. We have no intention of turning rsync.net into a development platform, running a single additional network service, or opening a single additional TCP port.

At the same time, there are a number of very straightforward synchronization and archival functions inherent to subversion and git that lend themselves very well to our offsite filesystem.

In addition, my own interest in DVCS in general, and git in particular as a tool for organizing plain old text files and unix file trees, added new urgency to the implementation.

That being said, we will not be adding any further such systems. There will not be cvs or Mercurial support in the future (although I am aware of some ways to run Mercurial over a git, or subversion "transport" and users are free to do so).

 

Some details...

 

Both git and svn can now be run without restrictions on the server side. Previously, we only allowed you to access svnserve through a svn+ssh:// style URL that you would pass to your own applications (like Tortoise, or the command line 'svn' tool). It was impossible, then, to create new repositories or sync from some other repository or view logs, etc. You are now able to do all of these things.

 

First, let's look at subversion.

 

We'll start by creating a remote repository in an rsync.net account:

ssh 1234@usw-s001.rsync.net "svnadmin create svn_repository"

The svn command is now supported and enables a vast set of functionality which previously wasn't available. For example, if we wanted to keep an up-to-date working copy of one of our projects for which a repository was remotely available (we'll use http://example.com/svn/repository as our remote URL), we could issue the following commands:

ssh 1234@usw-s001.rsync.net "svn checkout http://example.com/svn/repository my_checkout"
... list of files checkeChecked out revision 42.

If you'd like to update that checkout later, you can issue a command like:

ssh 1234@usw-s001.rsync.net "svn update my_checkout"

The svnsync command is also available. svnsync allows synchronization of repositories. This is useful to keep a full backup of an existing repository. Again, we'll use the URL of http://example.com/svn/repository as the repository for which we want to create a backup. First, we'll create a repository:

ssh 1234@usw-s001.rsync.net "svnadmin create synced_repository"

svnsync requires that we initialize the repository prior to doing any syncing. Part of this initialization process requires that we change repository attributes that are normally restricted, so we need to allow these properties to be changed by creating a hook script that exits with a zero return code. In order to do this, we'll create a symlink to the "echo" program:

ssh 1234@usw-s001.rsync.net "ln -s /bin/echo synced_repository/hooks/pre-revprop-
change"

With that hook script in place, we can now initialize our repository for syncing. Before we do so, we need to know the exact path to our checkout, so let's discover the path to our home directory:

$ ssh 1234@usw-s001.rsync.net "pwd"
/data2/home/1234

$ ssh 1234@usw-s001.rsync.net "svnsync initialize
file:///data2/home/1234/synced_repository http://example.com/svn/repository"
Copied properties for revision 0 (svn:sync-* properties skipped).

Now, we can issue the following command as often as needed in order to keep the repository up-to-date:

$ ssh 1234@usw-s001.rsync.net "svnsync sync
file:///data2/home/1234/synced_repository"
Committed revision 1.
Copied properties for revision 1.
...
Committed revision 42.
Copied properties for revision 42.

We can also dump a repository to a file by using the 'svnadmin dump' command. Similarly, the database can be restored by using the 'svnadmin load' command. These two commands work in harmony to keep backups of subversion databases, or specific revisions within a database.

To make a single-file backup to my local machine of a repository existing on rsync.net, we can do the following:

$ ssh 1234@usw-s001.rsync.net "svnadmin dump existing_repository" >
repository_backup.dump 2>repository_backup_errors.txt

The above command creates two files on the local machine, a repository_backup.dump file and a repository_backup_errors.txt file. The former is a single-file backup of the subversion repository and the latter is a list of notifications and errors that might be important.

Finally, the svn client can provide information about a checkout by using the "svn info" command. You can use this information to determine the source repository:

$ ssh 1234@usw-s001.rsync.net "svn info existing_repository"
Path: .
URL: http://example.com/directory
Repository Root: http://example.com/directory
Repository UUID: dc7efa32-411c-0410-9537-da5bd19367fc
Revision: 42
Node Kind: directory
Schedule: normal
Last Changed Author: admin
Last Changed Rev: 42
Last Changed Date: 2010-02-02 12:56:03 -0800 (Tue, 02 Feb 2010)

The svnlook command allows inspection of a repository itself. For example, to inspect the subversion repository and find out its UUID:

$ ssh 1234@usw-s001.rsync.net "svnlook uuid svn_repository"
dc7efa32-411c-0410-9537-da5bd19367fc

 

Let's next examine the new git support.

 

All standard git operations that do not require external programs are supported when the repository is hosted by rsync.net. (See note about client support at the end of this blog post)

To create a git repository on rsync.net we can issue the following command:

$ ssh 1234@usw-s001.rsync.net "git init --bare git_repo.git"
Initialized empty Git repository in /data2/home/1234/git_repo.git/

Note that the full path to the repository will be required for some operations.

Once a repository has been created, we would want to clone it so we have a local copy on which we could make changes. We can do so with the git clone command, but in order to do so we need the absolute path that was displayed when we initialized the repository:

$ git clone ssh://1234@usw-s001.rsync.net/data2/home/1234/git_repo.git
Initialized empty Git repository in /home/kibab/git_repo/.git/

Once you've made changes to your local repository, changes can be pushed back to rsync.net using "git push". By default, any changes we make on our local filesystem will be applied to the repository will be applied to the "master" branch. So, when we push changes up to rsync.net, we'll need to specify which branch should be pushed. When the "git clone" command was issued, git recorded the original repository in the .git/config file.

(NOTE: if you happen to have a hosts file alias for rsync.net, you might need to edit $REPO/.git/config so that if git attempts to bypass the hosts file lookup rsync.net may be correctly resolved.) Since git already knows where the origin (i.e. original) repository is, we can simply issue a push on the local filesystem:

$ git push origin master
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 3.33 MiB | 72 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://1234@usw-s003.rsync.net/data2 * [new branch]      master -> master

After pushing the master branch, we'll no longer need to specify the branch we're working with as it will have contents and be selected by default unless otherwise changed.

To pull updates from your git repository, in the event that a different machine pushes some changes up to the git repository, use 'git pull':

$ git pull
remote: Counting objects: 4, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From ssh://usw-s003.rsync.net/data2/home/1234/git_repo
   f4d263b..445595a  master     -> origin/master
Updating f4d263b..445595a
Fast-forward
 TestableJava.pdf |  Bin 0 -> 86224 bytes
 1 files changed, 0 insertions(+), 0 create mode 100644 TestableJava.pdf

Finally, we can make changes with branching. For this example, let's assume that we've been baking up a settings directory and we're about to embark on a system upgrade that may drastically change all of your settings files. We want to be able to upgrade the settings directory without losing any of the prior work and while retaining the ability to go back to the older settings, if necessary. To do this we'll need to create a new branch:

$ git branch upgrade

We can then see the branch using "git branch" without any parameters:

$ git branch
* master
  upgrade

The asterisk (*) indicates that we're still using the master branch, so let's switch to the upgrade branch:

$ git checkout upgrade
Switched to branch 'upgrade'

Now, after performing the upgrade and committing our changes, we can push the changes to the server without losing any of our prior work.

$ git push origin upgrade
Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 1.23 KiB, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://1234@usw-s003.rsync.net/data2/home/1234/git_repo.git
 * [new branch]      upgrade -> upgrade

I can now switch between the two different versions using "git checkout" and use "git diff" to see the differences between the revisions. To revert back to our pre-upgrade configuration, we use "git checkout" one more time:

$ git checkout master
Switched to branch 'master'

 

A special note regarding git client functionality:

git has been configured to be a fully functional server hosted on rsync.net. Although certain operations may work, not all programs have been installed to make it a fully functional client.

 

A further note regarding hook scripts:

There is no hook script support for either subversion or git. However, as we saw in the above subversion example, where svnsync required the pre-revprop-change hook script, you can symlink /bin/echo to the hook script you need, in your own rsync.net filesystem, to satisfy this rare requirement. It simply returns a zero and satisfies svnsync.

 

As always, please contact support@rsync.net if you have any questions.

 

Google DNS

There are many very good reasons to be alarmed by the notion that Google would like the world to migrate to their servers for DNS resolution.  A good discussion on the Nanog mailing list:

http://mailman.nanog.org/pipermail/nanog/2009-December/015787.html

highlighted several of these, mainly from the perspective of a network operator or ISP admin, but the usual questions of privacy/tracking/anonymity were raised.

Certainly a monoculture is a dangerous thing, and this makes the sorry state of the global Internet worse in a fairly profound way.

As you may imagine, I have things to say about this. But today, I'd like to submit what I believe to be the impetus behind this new offering:

I believe that Google will begin delivering search results based not only on PageRank but an amalgam of PageRank and other, increasingly "out of band" information.

While a significant portion of my Internet behavior is funneled, in one way or another, through Google search (I don't use mail/voice/groups/etc.) there is a lot of behavior that isn't. This "out of band" behavior is every web site, and especially every chain of web sites that I visit without searching, or after searching for a single site, but then following subsequent links forward.

If search results, and especially ad placement, is generated not just with PageRank, but with a union of PageRank and other information like "site you are most likely to visit within X minutes of these sites" there is the potential of providing even more "relevant" search results, which lead to more "relevant" ad placements, leading to more ad clicks.

This isn't exactly what I would characterize as a "good" in the world, and all of the fears I have heard voiced are well founded. So even though my proposed motivation for this new service is rather benign, the technical underpinning is not any less foreboding.

If this is indeed the motivation for "Google Public DNS", what would this imply ?

First, Google would have an incentive to reduce DNS caching. If you receive name lookups from inside of your own network, or inside of your own system, this decreases the intelligence that is flowing back to Google. It will be very interesting to see what the default cache settings for name resolution are in Google Chrome OS and Android.

Second, the shady world of "search engine optimization" and Adwords fraud is going to get more interesting. These activities could conceivably shift from a passive collection of fake websites and cross-connected "blog entries" to the active generation of network traffic to influence search results.

Which sounds like a real party.

 

Making the Mac more like Ion

For ten years - roughly 1999 through 2009 - all of my personal workstations ran the Ion window manager on top of X Windows (on FreeBSD, FWIW).  (actually, it was ratpoison at first...)

Modern GUIs are, IMO, extremely inefficient. Almost all of their "features", and arguably all of their graphical elements are at best wastes of time, and at worst major impediments to usability and workflow.

I first started thinking about this, ten years ago, when I noticed myself memorizing every possible keyboard shortcut in Windows. I realized, first subconsciously, that moving the mouse was inefficient. Later, as I switched to X Windows as my primary operating environment, I found that people were writing entire window managers around not touching the mouse. But that was only the beginning - things like overlapping windows, exposed areas of "desktop", and especially clicking to focus started to annoy me. Once you decide to give up the eye candy and optimize for actual work, you quickly shed a lot of things you used to take for granted.

Since all I ever work in is a browser and some xterms, this was a workable solution for a very long time, and I was perfectly happy with Ion on top of X Windows. However, as "web 2.0" evolved, it became increasingly difficult to simply get through the work day. My days of spending a week on configuration and compilation of five different packages just to get PDF reading to work ended a long time ago. I just didn't have time for the pain anymore. When you go to Usenix, or BSDcon, and all of the big name kernel hackers have mac laptops, you start to wonder just what you're trying to prove...

Enter OSX. I would click on a PDF link and it would open a PDF, immediately, in my browser. I would see a Youtube link and could actually watch it without crashing my browser. I could plug in a printer and print. I could use online banking without editing my browser identification. It really had been a slow boil for me over the last few years as I put up with more and more petty inconveniences in the name of "freedom".

The problem was the user interface. I believe that the Mac operating environment is judged by the ability of technically illiterate users to sit down and start using it. By that measure, it probably excels. But I'm not technically illiterate, and I have things like workflows and efficiencies that I need to develop and fine tune, and the Mac OE gives me very little leeway to do that. You can sum it up nicely with the inability to maximize a Window (that is, without manually dragging it to the corners with the mouse in a one-shot attempt to cover the whole screen).

But ... I can't go back to praying every time I go to a news site, or managing five different package installations just to view PDF files. So I decided to make the mac as close to Ion as possible. Lucky for me, this involved only two new package: sizeup and Mondo Mouse.

There are many aspects to the Ion Window Manager, and the most important are probably things I don't even touch here, like keyboard shortcuts and Lua scripting, and so on - but for me, the major usability features were tiling, non-overlapping windows, and focus-follows-mouse.

Sizeup provides the ability to precisely resize a window to one of many pre-determined geometries - things like "the left half of the screen" or "the entire screen" or "the upper right quadrant of the screen". This means you can quickly arrange a completely tiled and non-overlapping workspace. Further, you can, with a keystroke, toggle between tiling a window to some portion of a screen and making it full screen.

Mondo Mouse does all sorts of weird things. I have no idea what they are, as I turn all of them off. Except for focus follows mouse. From my own system build notes: In preferences, turn off everything except for focus follows mouse, set the timer to '0.00' and under the 'appearance' tab, uncheck "display information about the window under the mouse".

And there you have it. It's not perfect. Command-tab and Command-tilde still behave very differently (cmd-tab "remembers" your previous location, whereas cmd-tilde is always sequential) and new window creation on the mac is laughably braindead (especially on multi-head displays) but things are much, much better.

Flat Rate Storage Services vs. rsync.net

One of the most common pre-sales questions we get at rsync.net is:

"Why should I pay a per gigabyte rate for storage when these other providers are offering unlimited storage for a low flat rate?"

The short answer is: paying a flat rate for unlimited storage, or transfer, pits you against your provider in an antagonistic relationship. This is not the kind of relationship you want to have with someone providing critical functions.

 

Now for the long answer...

 

In 1998 I did some consulting for an early, and very successful provider of web hosting. They offered unlimited disk space and traffic for a relatively low monthly price. Of course, the immediate question was how this was feasible - why wouldn't I just pay my $14.99 per month and upload gigabytes upon gigabytes of data, etc.

They chuckled and said (I paraphrase) "the more you upload, the more we throttle you. The little users don't bother us and the big users eventually just go away".

It's hard to say how common this kind of (deplorable) policy is ... but the economics demand that something like this be in place. Bandwidth and disk space are not free, and even the most unsophisticated hosting or storage consumer has the ability to deluge a provider with both.

What this means is that the providers and consumers of "unlimited" services will necessarily have conflicting interests. If your interest is to store as much data per dollar as possible, it will be the interest of your provider for you to store as little data per dollar as possible. The more you utilize, the more antagonistic this relationship will become, as you move from "someone we don't make any money on" to "someone who is killing our business model".

These non-intersecting goals will manifest themselves in any number of ways from bad service and support to "unexplained" poor performance to outright disconnection. As described above, I know firsthand of hosting companies that practice these very measures, and our interaction with prospective customers at rsync.net has exposed us to many more examples.

 

Here are some specific examples:

 

The Backup Provider with Unlimited Storage - This is the most common provider we are asked about, obviously. Examples include Mozy and Carbonite. This is the textbook case of what we are talking about here. Traffic throttling, connection resets, and discriminatory support are all examples of things I have personally seen providers like this do to users that don't perfectly fit their business model.

The Web Host with Unlimited Storage - Once in a while a more technically savvy individual will point out that they could get a $4.99/mo web hosting account from 1 & 1 or GoDaddy or someone like that and use duplicity over FTP or something like that to upload unlimited data there. In cases like this, unlimited web hosts have very specific contract language that stipulates that ALL content you upload to the web host MUST be publicly available web content. So the good news is they aren't playing any tricks with the storage they provide (the bandwidth is a different story) but it is unlikely you want to publish all of your data on the web.

Gmail Drives, etc. - Technically very clever, and an example of "things that make me happy", but not worth discussing in terms of critical data backup. Even for personal use, you're moving beyond simply working at cross purposes to your provider, and into the realm of "this tool isn't even designed for this, and can change at any time".

rsync.net, and other per-gigabyte storage providers - We want you to store as much data as you possibly can with us. We will do whatever we can to help you with whatever solution you require to get your data on our systems. The more data you put here, and the faster you do it, the happier we are. It should be noted that we do offer unlimited transfer, or bandwidth, but this is not evidence of a possible conflict of interest. We offer the unlimited transfer as a complement to the data that we want you to transfer here as fast as possible. Since we do not allow anonymous access (or "web" access, or inline linking, etc.) to an rsync.net account, there is no downside for us in doing this.

 

All of this leads to a general rule that should be applied to all consumption of services: it is imperative that you be aware of the interests of your provider, fully understand your own interests, and make sure that those two things align as well as possible. It is naive to think that any workable solution can be built upon an underlying relationship that is at cross purposes.

 

SwissDisk Doubles Down

We'd heard of SwissDisk here at rsync.net, but they rarely showed up on our radar screen.  We were reminded of their existence a few days ago when their entire infrastructure failed.  It's unclear how much data, if any, was eventually lost ... but my reading of their announcement makes me think "a lot".

I'm commenting on this because I believe their failure was due to an unnecessarily complex infrastructure. Of course, this requires a lot of conjecture on my part about an organization I know little about ... but I'm pretty comfortable making some guesses.

It's en vogue these days to build filesystems across a SAN and build an application layer on top of that SAN platform that deals with data as "objects" in a database, or something resembling a database. All kinds of advantages are then presented by this infrastructure, from survivability and fault tolerance to speed and latency. And cost. That is, when you look out to the great green future and the billions of transactions you handle every day from your millions of customers are all realized, the per unit cost is strikingly low.

It is my contention that, in the context of offsite storage, these models are too complex, and present risks that the end user is incapable of evaluating. I can say this with some certainty, since we have seen that the model presented risks that even the people running it were incapable of evaluating.

This is indeed an indictment of "cloud storage", which may seem odd coming from the proprietor of what seems to be "cloud storage". It makes sense, however, when you consider the very broad range of infrastructure that can be used to deliver "online backup". When you don't have stars in your eyes, and aren't preparing for your IPO filing and the "hockey sticking" of your business model, you can do sensible things like keep regular files on UFS2 filesystems on standalone FreeBSD systems.

This is, of course, laughable in the "real world". You couldn't possibly support thousands and thousands of customers around the globe, for nearly a decade, using such an infrastructure. Certainly not without regular interruption and failure.

Except when you can, I guess:

# uptime
12:48PM  up 350 days, 21:34, 2 users, load averages: 0.14, 0.14, 0.16

(a live storage system, with about a thousand users, that I picked at random)

# uptime
 2:02PM  up 922 days, 18:38, 1 user, load averages: 0.00, 0.00, 0.00

(another system on the same network)

Further, the consensus might be that rsync.net is not providing "cloud storage". Good. We're not. I would be very happy if the colloquial definition of "cloud storage" did indeed imply very "sophisticated" and "modern" infrastructures, like Amazon S3 and Rackspace Cloud Files and so on. Complex, sprawling webs of fault tolerance and survivability that no team of ten people could possibly understand - and certainly not their customers.

I don't know what we'll end up being called. Offsite fileservers, maybe. Or maybe just "human beings that control computer systems" (instead of the other way around).

I have a lot to say about complexity in computer systems and the conclusions I have drawn about "fault tolerance" and things like the "five nines". I'd like to wait and present it more formally, perhaps even write a book about it. I am forced to address it today, however, when I see the worst aspect of the SwissDisk failure - their own response:

"(SwissDisk) has signed a contractual service agreement with a $40B company. For the first time this will enable SwissDisk to provide our customers with a 99.95 per cent service level agreement. SwissDisk users will enjoy the peace of mind that comes with access to a truely world class, internationally deployed, multi-million dollar, extremely resiliant storage infrastructure and internet network."

Their response to their inability to comprehend and master their own environment is to increase the complexity of that environment. Whatever layers of risk that SwissDisk customers could not possibly evaluate will now be increased by at least one order of magnitude - probably more.

We wish them luck as they climb higher up into the stratosphere of "serious" infrastructure.