John Kozubik < john @ kozubik . com >

Ignoring The Kill Switch

Much has been said recently of Cyber Warfare and the existential threat it represents.

Hopefully you have been able to ignore this ridiculous meme and the clownish "experts" propagating it.

Unfortunately, providing "solutions" for these problems is far too lucrative a market opportunity to be left unexploited, so you should expect to see this meme grow, and for the proposed policies to become more and more radical.

Enter the kill switch.

Like the more general "cyber warfare" meme, the Internet Kill Switch is demonstrably absurd. Will wireless networking equipment be confiscated ? Will private peering interconnects within datacenters be disconnected ? Will IP traffic over cellular or CATV links be disabled ? The understanding of those that propose such an idea probably begins and ends at the word "switch".

Derision and disgust, however, are not ingredients of a good business continuity plan. Although it appears unlikely to be implemented soon, the (bad) policy has been proposed, and regardless of our technical "understanding", we as service providers must respond and fit our offerings to this possible reality.

And so, in Q1 of 2011, rsync.net will provide a wireless private network at each of its physical locations (San Diego California, Denver Colorado, Zurich Switzerland, and Hong Kong) which will allow customers to physically approach the datacenter and access their stored data without traversing The Internet. This will ensure that bad policy will have to enter the stone age to keep you from accessing your data.

Full technical details will be published on our web site before these wireless links are active. Capabilities will include the ability to tunnel through to other rsync.net locations. So, for instance, if you live in California and store your data in our Zurich location, we may still have routing across the ocean even if you do not - we're right next to our backbone providers, and have several of them, after all. So while we will not transform ourselves into an ISP for you, we will maintain whatever VPNs we can to allow you to reach your data, regardless of which physical location you may be at.

As we've pointed out before, we are not an ISP. We announce today that we are not even an Internet data storage provider ... we are simply a data storage provider and will work with our customers to provide them service regardless of political inefficiencies.

 

The Warrant Canary in 2010 and Beyond

We have been publishing our Warrant Canary weekly at rsync.net for almost five years now. We are happy to report that in this time, no warrants of any kind have been served to us.

Unfortunately, it does not appear that the National Security Letter as an abusive surveillance tool is becoming any less widespread. While the Doe v. Ashcroft ruling was an encouraging step, the reality is that the NSL is alive and well. To wit:

The USA PATRIOT Act is affected only if the limits on NSLs in terrorism cases also apply to non-terrorism cases such as those authorized by the Act. The government was expected to appeal the ruling to the Supreme Court, and until the district court ruling is reviewed, the secrecy procedures of the NSL remain in place.

Further, the limitations of the NSL, and of the USA PATRIOT act in general, are slowly being worked around by successive presidential and agency administrations:

Wrong Direction on Privacy (Huffington Post)

Breaking a Promise on Surveillance (New York Times)

And so we will continue to publish the Warrant Canary. Further, we will begin mirroring our Warrant Canary at our international locations (currently Switzerland and Hong Kong). This will address the most common complaint: that when the rubber hits the road, conventional and unconventional means of coercion could be used to compel persons at our firm to publish false statements. Restriction of our speech will now require cooperation across multiple continents, cultures, and languages.

 

Related to this, let me say that I have never thought of rsync.net as an Internet Service Provider. We have a technical philosophy that places a high value on simplicity and we have constructed our data storage platform with as few processes and services running as possible.

We believe that stability and security are greatly enhanced by keeping the list of moving parts as small as possible.

There are legal and privacy related benefits to this philosophy. We do not currently fit the legal definition of an ISP, and we will take steps to ensure that we continue to fall outside of data collection measures that specifically relate to ISPs

There is some cost associated with this. For instance, we have considered implementing an "email to file" service that allows our customers to email attached files into their rsync.net filesystems. One of our competitors, DropBox, has such a feature. We have eschewed this, however, as providing email service (even in this non-standard way) takes us a big step closer to being classified as an ISP.

 

Finally, I will address the FAQ: "what will happen if you do receive a warrant ?"

The answer is, we will publish the receipt of the warrant, and perhaps the warrant itself, provided that it is not a "secret" warrant. So the current verbage of the warrant canary:

No warrants have ever been served to rsync.net, or rsync.net principals or employees. No searches or seizures of any kind have ever been performed on rsync.net assets

would be changed to read:

rsync.net has received the following warrants during operation:
2010-09-14 Warrant served by Sevier County Court, Sevier County, UT, compelled rsync.net to provide confirmation that John Q. Public was not an rsync.net customer.

NO OTHER warrants have ever been served to rsync.net

As always, a secret warrant causes the canary to cease updating until such time as that warrant can be listed. I hope that this makes clear that the mere receipt of a warrant or order of any kind, which I am surprised we have never received, does not necessarily "kill" the warrant canary.

 

I will conclude by reiterating a key component of our Terms of Service:

No consumer or personal information about our customers of any kind will be divulged to any party for any reason.

It's amazing how simple things can be...

 

Git and Subversion Support at rsync.net

At long last, git is supported at rsync.net.

We wrestled with the decision to add it for some time, as we place a very, very high value on the simplicity of our systems. We have no intention of turning rsync.net into a development platform, running a single additional network service, or opening a single additional TCP port.

At the same time, there are a number of very straightforward synchronization and archival functions inherent to subversion and git that lend themselves very well to our offsite filesystem.

In addition, my own interest in DVCS in general, and git in particular as a tool for organizing plain old text files and unix file trees, added new urgency to the implementation.

That being said, we will not be adding any further such systems. There will not be cvs or Mercurial support in the future (although I am aware of some ways to run Mercurial over a git, or subversion "transport" and users are free to do so).

 

Some details...

 

Both git and svn can now be run without restrictions on the server side. Previously, we only allowed you to access svnserve through a svn+ssh:// style URL that you would pass to your own applications (like Tortoise, or the command line 'svn' tool). It was impossible, then, to create new repositories or sync from some other repository or view logs, etc. You are now able to do all of these things.

 

First, let's look at subversion.

 

We'll start by creating a remote repository in an rsync.net account:

ssh [email protected] "svnadmin create svn_repository"

The svn command is now supported and enables a vast set of functionality which previously wasn't available. For example, if we wanted to keep an up-to-date working copy of one of our projects for which a repository was remotely available (we'll use http://example.com/svn/repository as our remote URL), we could issue the following commands:

ssh [email protected] "svn checkout http://example.com/svn/repository my_checkout"
... list of files checkeChecked out revision 42.

If you'd like to update that checkout later, you can issue a command like:

ssh [email protected] "svn update my_checkout"

The svnsync command is also available. svnsync allows synchronization of repositories. This is useful to keep a full backup of an existing repository. Again, we'll use the URL of http://example.com/svn/repository as the repository for which we want to create a backup. First, we'll create a repository:

ssh [email protected] "svnadmin create synced_repository"

svnsync requires that we initialize the repository prior to doing any syncing. Part of this initialization process requires that we change repository attributes that are normally restricted, so we need to allow these properties to be changed by creating a hook script that exits with a zero return code. In order to do this, we'll create a symlink to the "echo" program:

ssh [email protected] "ln -s /bin/echo synced_repository/hooks/pre-revprop-
change"

With that hook script in place, we can now initialize our repository for syncing. Before we do so, we need to know the exact path to our checkout, so let's discover the path to our home directory:

$ ssh [email protected] "pwd"
/data2/home/1234

$ ssh [email protected] "svnsync initialize
file:///data2/home/1234/synced_repository http://example.com/svn/repository"
Copied properties for revision 0 (svn:sync-* properties skipped).

Now, we can issue the following command as often as needed in order to keep the repository up-to-date:

$ ssh [email protected] "svnsync sync
file:///data2/home/1234/synced_repository"
Committed revision 1.
Copied properties for revision 1.
...
Committed revision 42.
Copied properties for revision 42.

We can also dump a repository to a file by using the 'svnadmin dump' command. Similarly, the database can be restored by using the 'svnadmin load' command. These two commands work in harmony to keep backups of subversion databases, or specific revisions within a database.

To make a single-file backup to my local machine of a repository existing on rsync.net, we can do the following:

$ ssh [email protected] "svnadmin dump existing_repository" >
repository_backup.dump 2>repository_backup_errors.txt

The above command creates two files on the local machine, a repository_backup.dump file and a repository_backup_errors.txt file. The former is a single-file backup of the subversion repository and the latter is a list of notifications and errors that might be important.

Finally, the svn client can provide information about a checkout by using the "svn info" command. You can use this information to determine the source repository:

$ ssh [email protected] "svn info existing_repository"
Path: .
URL: http://example.com/directory
Repository Root: http://example.com/directory
Repository UUID: dc7efa32-411c-0410-9537-da5bd19367fc
Revision: 42
Node Kind: directory
Schedule: normal
Last Changed Author: admin
Last Changed Rev: 42
Last Changed Date: 2010-02-02 12:56:03 -0800 (Tue, 02 Feb 2010)

The svnlook command allows inspection of a repository itself. For example, to inspect the subversion repository and find out its UUID:

$ ssh [email protected] "svnlook uuid svn_repository"
dc7efa32-411c-0410-9537-da5bd19367fc

 

Let's next examine the new git support.

 

All standard git operations that do not require external programs are supported when the repository is hosted by rsync.net. (See note about client support at the end of this blog post)

To create a git repository on rsync.net we can issue the following command:

$ ssh [email protected] "git init --bare git_repo.git"
Initialized empty Git repository in /data2/home/1234/git_repo.git/

Note that the full path to the repository will be required for some operations.

Once a repository has been created, we would want to clone it so we have a local copy on which we could make changes. We can do so with the git clone command, but in order to do so we need the absolute path that was displayed when we initialized the repository:

$ git clone ssh://[email protected]/data2/home/1234/git_repo.git
Initialized empty Git repository in /home/kibab/git_repo/.git/

Once you've made changes to your local repository, changes can be pushed back to rsync.net using "git push". By default, any changes we make on our local filesystem will be applied to the repository will be applied to the "master" branch. So, when we push changes up to rsync.net, we'll need to specify which branch should be pushed. When the "git clone" command was issued, git recorded the original repository in the .git/config file.

(NOTE: if you happen to have a hosts file alias for rsync.net, you might need to edit $REPO/.git/config so that if git attempts to bypass the hosts file lookup rsync.net may be correctly resolved.) Since git already knows where the origin (i.e. original) repository is, we can simply issue a push on the local filesystem:

$ git push origin master
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 3.33 MiB | 72 KiB/s, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://[email protected]/data2 * [new branch]      master -> master

After pushing the master branch, we'll no longer need to specify the branch we're working with as it will have contents and be selected by default unless otherwise changed.

To pull updates from your git repository, in the event that a different machine pushes some changes up to the git repository, use 'git pull':

$ git pull
remote: Counting objects: 4, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 3 (delta 0), reused 0 (delta 0)
Unpacking objects: 100% (3/3), done.
From ssh://usw-s003.rsync.net/data2/home/1234/git_repo
   f4d263b..445595a  master     -> origin/master
Updating f4d263b..445595a
Fast-forward
 TestableJava.pdf |  Bin 0 -> 86224 bytes
 1 files changed, 0 insertions(+), 0 create mode 100644 TestableJava.pdf

Finally, we can make changes with branching. For this example, let's assume that we've been baking up a settings directory and we're about to embark on a system upgrade that may drastically change all of your settings files. We want to be able to upgrade the settings directory without losing any of the prior work and while retaining the ability to go back to the older settings, if necessary. To do this we'll need to create a new branch:

$ git branch upgrade

We can then see the branch using "git branch" without any parameters:

$ git branch
* master
  upgrade

The asterisk (*) indicates that we're still using the master branch, so let's switch to the upgrade branch:

$ git checkout upgrade
Switched to branch 'upgrade'

Now, after performing the upgrade and committing our changes, we can push the changes to the server without losing any of our prior work.

$ git push origin upgrade
Counting objects: 4, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 1.23 KiB, done.
Total 3 (delta 0), reused 0 (delta 0)
To ssh://[email protected]/data2/home/1234/git_repo.git
 * [new branch]      upgrade -> upgrade

I can now switch between the two different versions using "git checkout" and use "git diff" to see the differences between the revisions. To revert back to our pre-upgrade configuration, we use "git checkout" one more time:

$ git checkout master
Switched to branch 'master'

 

A special note regarding git client functionality:

git has been configured to be a fully functional server hosted on rsync.net. Although certain operations may work, not all programs have been installed to make it a fully functional client.

 

A further note regarding hook scripts:

There is no hook script support for either subversion or git. However, as we saw in the above subversion example, where svnsync required the pre-revprop-change hook script, you can symlink /bin/echo to the hook script you need, in your own rsync.net filesystem, to satisfy this rare requirement. It simply returns a zero and satisfies svnsync.

 

As always, please contact [email protected] if you have any questions.

 

Flat Rate Storage Services vs. rsync.net

One of the most common pre-sales questions we get at rsync.net is:

"Why should I pay a per gigabyte rate for storage when these other providers are offering unlimited storage for a low flat rate?"

The short answer is: paying a flat rate for unlimited storage, or transfer, pits you against your provider in an antagonistic relationship. This is not the kind of relationship you want to have with someone providing critical functions.

 

Now for the long answer...

 

In 1998 I did some consulting for an early, and very successful provider of web hosting. They offered unlimited disk space and traffic for a relatively low monthly price. Of course, the immediate question was how this was feasible - why wouldn't I just pay my $14.99 per month and upload gigabytes upon gigabytes of data, etc.

They chuckled and said (I paraphrase) "the more you upload, the more we throttle you. The little users don't bother us and the big users eventually just go away".

It's hard to say how common this kind of (deplorable) policy is ... but the economics demand that something like this be in place. Bandwidth and disk space are not free, and even the most unsophisticated hosting or storage consumer has the ability to deluge a provider with both.

What this means is that the providers and consumers of "unlimited" services will necessarily have conflicting interests. If your interest is to store as much data per dollar as possible, it will be the interest of your provider for you to store as little data per dollar as possible. The more you utilize, the more antagonistic this relationship will become, as you move from "someone we don't make any money on" to "someone who is killing our business model".

These non-intersecting goals will manifest themselves in any number of ways from bad service and support to "unexplained" poor performance to outright disconnection. As described above, I know firsthand of hosting companies that practice these very measures, and our interaction with prospective customers at rsync.net has exposed us to many more examples.

 

Here are some specific examples:

 

The Backup Provider with Unlimited Storage - This is the most common provider we are asked about, obviously. Examples include Mozy and Carbonite. This is the textbook case of what we are talking about here. Traffic throttling, connection resets, and discriminatory support are all examples of things I have personally seen providers like this do to users that don't perfectly fit their business model.

The Web Host with Unlimited Storage - Once in a while a more technically savvy individual will point out that they could get a $4.99/mo web hosting account from 1 & 1 or GoDaddy or someone like that and use duplicity over FTP or something like that to upload unlimited data there. In cases like this, unlimited web hosts have very specific contract language that stipulates that ALL content you upload to the web host MUST be publicly available web content. So the good news is they aren't playing any tricks with the storage they provide (the bandwidth is a different story) but it is unlikely you want to publish all of your data on the web.

Gmail Drives, etc. - Technically very clever, and an example of "things that make me happy", but not worth discussing in terms of critical data backup. Even for personal use, you're moving beyond simply working at cross purposes to your provider, and into the realm of "this tool isn't even designed for this, and can change at any time".

rsync.net, and other per-gigabyte storage providers - We want you to store as much data as you possibly can with us. We will do whatever we can to help you with whatever solution you require to get your data on our systems. The more data you put here, and the faster you do it, the happier we are. It should be noted that we do offer unlimited transfer, or bandwidth, but this is not evidence of a possible conflict of interest. We offer the unlimited transfer as a complement to the data that we want you to transfer here as fast as possible. Since we do not allow anonymous access (or "web" access, or inline linking, etc.) to an rsync.net account, there is no downside for us in doing this.

 

All of this leads to a general rule that should be applied to all consumption of services: it is imperative that you be aware of the interests of your provider, fully understand your own interests, and make sure that those two things align as well as possible. It is naive to think that any workable solution can be built upon an underlying relationship that is at cross purposes.

 

SwissDisk Doubles Down

We'd heard of SwissDisk here at rsync.net, but they rarely showed up on our radar screen.  We were reminded of their existence a few days ago when their entire infrastructure failed.  It's unclear how much data, if any, was eventually lost ... but my reading of their announcement makes me think "a lot".

I'm commenting on this because I believe their failure was due to an unnecessarily complex infrastructure. Of course, this requires a lot of conjecture on my part about an organization I know little about ... but I'm pretty comfortable making some guesses.

It's en vogue these days to build filesystems across a SAN and build an application layer on top of that SAN platform that deals with data as "objects" in a database, or something resembling a database. All kinds of advantages are then presented by this infrastructure, from survivability and fault tolerance to speed and latency. And cost. That is, when you look out to the great green future and the billions of transactions you handle every day from your millions of customers are all realized, the per unit cost is strikingly low.

It is my contention that, in the context of offsite storage, these models are too complex, and present risks that the end user is incapable of evaluating. I can say this with some certainty, since we have seen that the model presented risks that even the people running it were incapable of evaluating.

This is indeed an indictment of "cloud storage", which may seem odd coming from the proprietor of what seems to be "cloud storage". It makes sense, however, when you consider the very broad range of infrastructure that can be used to deliver "online backup". When you don't have stars in your eyes, and aren't preparing for your IPO filing and the "hockey sticking" of your business model, you can do sensible things like keep regular files on UFS2 filesystems on standalone FreeBSD systems.

This is, of course, laughable in the "real world". You couldn't possibly support thousands and thousands of customers around the globe, for nearly a decade, using such an infrastructure. Certainly not without regular interruption and failure.

Except when you can, I guess:

# uptime
12:48PM  up 350 days, 21:34, 2 users, load averages: 0.14, 0.14, 0.16

(a live storage system, with about a thousand users, that I picked at random)

# uptime
 2:02PM  up 922 days, 18:38, 1 user, load averages: 0.00, 0.00, 0.00

(another system on the same network)

Further, the consensus might be that rsync.net is not providing "cloud storage". Good. We're not. I would be very happy if the colloquial definition of "cloud storage" did indeed imply very "sophisticated" and "modern" infrastructures, like Amazon S3 and Rackspace Cloud Files and so on. Complex, sprawling webs of fault tolerance and survivability that no team of ten people could possibly understand - and certainly not their customers.

I don't know what we'll end up being called. Offsite fileservers, maybe. Or maybe just "human beings that control computer systems" (instead of the other way around).

I have a lot to say about complexity in computer systems and the conclusions I have drawn about "fault tolerance" and things like the "five nines". I'd like to wait and present it more formally, perhaps even write a book about it. I am forced to address it today, however, when I see the worst aspect of the SwissDisk failure - their own response:

"(SwissDisk) has signed a contractual service agreement with a $40B company. For the first time this will enable SwissDisk to provide our customers with a 99.95 per cent service level agreement. SwissDisk users will enjoy the peace of mind that comes with access to a truely world class, internationally deployed, multi-million dollar, extremely resiliant storage infrastructure and internet network."

Their response to their inability to comprehend and master their own environment is to increase the complexity of that environment. Whatever layers of risk that SwissDisk customers could not possibly evaluate will now be increased by at least one order of magnitude - probably more.

We wish them luck as they climb higher up into the stratosphere of "serious" infrastructure.