Saturday, May 31st, 2008, at 3:54 pm

Storage Device: Fast, redundant, fail-safe, easy

This is an idea that has been scratching at me for quite some time. The biggest reason: I need to find something like this!

The problem

Everybody has growing file collections. Some of us definitely have more than others—my photo collection alone is reaching 80 GB! A single recording session can easily reach 20 GB. Acquired media aside (purchased/downloaded music, movies, etc—as it’s replaceable), everyone has a need for an elegant solution to back up this data.

There are a number of products and services popping up that attempt a solution to this problem, but none do it quite right. I have to give Apple a lot of credit here—they figured out a way to make backup so simple, the masses can finally do the right thing! Time Machine is not perfect for everyone, but it’s a huge step in the right direction.

The purpose of this post is not to discuss software solutions, but rather the hardware that accompanies it. While the right software is what gets people backing up, it always requires the right hardware¹ to make it work.

The current solutions

Online

Online services (aka “cloud” storage) allow users to upload their data to online servers. Some of these include backup software and all tout the benefits of being failure-safe and secure.

Pros

• Safe: Redundantly hosted on large server farms so the data itself is safe and secure.
• Portable: You can usually access your data anywhere you have a net connection.
• Sharing: Many of these services let you share certain folders with colleagues, friends, etc.
• Flexible/Scaleable: If the service has storage limits (some are unlimited), you can scale your costs with your direct backup needs.

Cons

• Speed: Backing up a lot of data online is constrained to internet speeds, and residential upload speeds are typically 2Mbps or less. This makes it very poor for users with large data needs and any time a user needs to access some of the data there is a big lag time.
• Software: A number of these solutions lock you into using their software. Often, their software is limited in big ways, e.g. not being able to backup external hard drives (unless they remain attached all the time).

Hardware

Hardware solutions consist of external hard-drives, file servers, and more intelligent products such as the Drobo (which is extremely cool).

Pros

• Fast: Accessing your data through a wired connection—or over a local wireless network—is much faster than through the internet. This makes it ideal for large amounts of data and all around much more accessible.
• Secure: You have absolute control of the security settings, so it’s as secure as you want to make it.

Cons

• Redundancy: Unless you are savvy enough to be able set up a RAID solution, or shell out for a Drobo, making external hard-drives redundant is a real pain.
• Flexibility/Scaleability: With the exception of the Drobo none of these solutions making scaling very easy.
• Physical Security: External hard drives, file-servers, and the like can all be easily stolen if you ever experience a break-in. Server farms often have very tight security: id badges, fingerprint/retina scanners, armed escorts, and all kinds of crazy cool precautions.
• Disaster:² Electronics are extremely susceptible to everything that comes with a house fire: heat, smoke, water, and chemicals. Backing up off-site is always a good idea.

Summary

The cons of both online and hardware solutions exclude any one solution from being ideal for me. I am a heavy data user that wants fast access and redundant, off-site backup. There is nothing that easily and elegantly fills this need. Sure, I could purchase “cloud” storage and manually back things up, but that requires quite a lot of time. I want something I can set and forget in the Time Machine sense.

The ideal solution

This leads me to what I think is the ideal solution to meet all of these needs. There would definitely be challenges in doing it right, but I think the need would make the investment worthwhile.

The Hardware

First, there is a hardware device that acts as the primary solution. This hardware device needs to redundant, easy to use, and smart.

• Like the Drobo³, it must be easy to scale with hot-swappable drive bays and it needs to “take care” of making the data redundant.
• It needs a variety of physical access methods: USB 2, FireWire 400/800+, eSATA, and Gigabit Ethernet.
• It must be cross-platform and therefore accept a variety of protocols such as AFP, Samba, and NFS, and advertise itself using Bonjour and Avahi. Along the same lines, it must be compatible with any file-system thrown at it, such as NTFS, HFS+, and ext3, or even ZFS, WinFS, and ext4.⁴
• It requires an easy-to-use and cross-platform compatible user-interface. A small web application would probably be best.
• Finally, it needs rsync set up as a nightly process.

The Cloud

Behind the hardware device, sits the ever-present protection of an online backup solution. It’s only for the event of a truly catastrophic event.

• It must be inexpensive. It doesn’t need lots of sharing features (though they are nice) and it needs to be priced in such a way that the user is ok with the on-going expense. Probably no more than $10/month, if that.
• It must be scaleable. Since the hardware component must be easy to scale, the online service must be able to keep up, with little or no direct effort from the user.
• It must be absolutely secure. When it comes to security, perhaps nothing is truly absolute, but it must be as secure as possible so that people can trust it. In addition, the data must be physically secure: it should be stored in more than one geographic location so it is impervious to natural disasters.
• It can’t have weird limitations such as a maximum file size of 5 GB That said, perhaps a little intelligence on the hardware side could get around a limitation like that…
• It must be stable and have very limited downtime.

Wrap-up

A solution like this—done correctly—would solve backup problems for some time to come. It gives users access to data quickly, redundant local backup, and off-site protection. The beauty is that the software is irrelevent: users can use any software solution they like, e.g. Time Machine, SuperDuper!, Retrospect, FlyBack, or anything else.

It might not be a product that everyone in the world would need, but it would be great for power users, professional or prosumer photographers, and just about anyone else that doesn’t have needs quite as high as video editor. Most importantly for me, I would buy one.

¹ In this case, “hardware” can mean online backup services as well.
² As someone that has woken up in the middle of a house fire and lived to tell about it, this is a bit of an ever-present threat in the back of my mind. A fire is a terrible thing, but losing all of your photos on top of it is icing on the suck cake. Plus, you never, ever, want to worry about grabbing things like that when evacuating. We are fortunate we live in a time where we can store things like photos off-site and still have access to them!
³ I swear, I’m not being paid for this. That said, offers are always welcome.
⁴ Obviously, it should be as forward-looking as possible. That said, if any potential future file-systems have specific hardware requirements that are unrealistic, they would have to be considered on a case-by-case basis.