I think computer users can be broken down into three camps:
- Computer users who haven’t had a hard disk fail and haven’t yet figured out they need to back up their systems.
- Computer users who have had a disk fail but still don’t back up their systems reliably (or at all), even though they know they should.
- Grouchy old computer geeks who yell at the first two groups because we’re the ones who get that call at 10PM because a disk failed and they need a file back because they’re on deadline and oh my god please help me I don’t have a backup what do I do?
I warn you up front, I am one of that last group. My goal is to convince you to start backing up your computer before it’s too late, because I want those late night on deadline oh my god I’m doomed please help me phone calls to stop. Even though I know it’ll never happen in my lifetime.
A hard drive is a spinning mechanical device with motors and magnets and bearings and a read-write head that flies milimeters away from the surface of the platter where the data is stored. It is inevitable this device will fail. Not IF, but WHEN. Newer computer use SSDs, which are solid state devices instead of spinning mechanic ones, but they, too, fail.
That’s the reality: whatever you store your data on is going to fail some day. If you don’t plan for that, bad things will happen. And when bad things happen, you call your geek friend late at night blubbering and crying and asking for help. Neither of us want that.
You can’t prevent the failure, but you can reduce the chances of it happening, and you can back up your data so that if a disk fails, it’s not a big deal, because that data also exists on another hard disk. Or two. Or three. The more the merrier.
This article will help you understand how to reduce the chance of that failure and to limit the pain and damage when it happens.
The Best Backup is Never Needing your Backup
The best and most reliable backup is never needing to recover data from your backup. You can never guarantee that a drive will never fail — but you can reduce the chances of it happening.
How? Simple: replace your drives before they fail. Backblaze is a company that will back up your data over the internet to their servers. They have lots of data on lots (and lots) of hard drives, and it’s their job for that data to never be missing. They’ve got lots of experience with failing hard drives and how long it takes for one to fail, and they’ve been nice enough to provide the data. If you’re interested in the details, read their study. The executive summary is that after a hard drive is three years old, the failure rate starts to rise rapidly. So the first thing you can do to reduce the chance of a hard drive failing on you is retiring it and replacing it with a new one before it gets to be four years old.
I take this one step further: if you have a laptop that you carry around, that laptop tends to get bounced and jostled. Inside that laptop is a hard drive, which is also getting jostled and bounced around. My experience is that laptop hard drives have a tendency to die younger than hard drives in machines that don’t move around, so if you have a laptop, you really want to replace that hard drive earlier.
My hard drive policy is simple:
- Any hard drive used I use as a working drive (attached to a computer and powered up for use on a daily basis) is replaced when it is between two and three years old.
- Any hard drive installed inside a laptop is replaced earlier: between 18 months and two years.
That doesn’t mean their useful life is over: the drives I used as my day to day drives get turned into backup drives (unless they’re too small). They’re used as backups until they’re around four years old, and then they’re retired.
Backup drives tend to be powered off a lot more, their usage is much lower, and you don’t put them under stress. That reduced stress means they’re less likely to fail. You use a drive hard when it’s new, give it a reduced role as it ages, and retire it before it hits that point in time where failure becomes likely.
If you do that, you will rarely have a drive fail on you. It costs a little money, but the cost of a new laptop drive these days is under $100, so it’s not that expensive. It’s a lot less expensive than the time and stress of recovering from a failure, that’s for sure.
A note on SSDs: As SSD (solid state drives, with no moving parts) mature, they’re rapidly replacing spinning drives for data storage. The failure tendencies of SSDs are a lot different than for hard disks, and it can be much different from one manufacturer to another. So what should you do about replacing aging SSDs? I don’t know yet. My current (tentative) plan is to let the SSD in my laptop go for three years and then replace and retire it rather than make it a backup drive, but that’s subject to change once I do more research. I still think the 3 years and out concept works for them, but I don’t think you need to be as aggressive moving them out of a high use mode.
A note on Hybrid Drives: Apple and some other companies are shipping computers with what they call a hybrid drive, which is both a hard disk and an SSD merged together. My view right now is that you treat them like hard drives and replace them like one, but I haven’t looked into the real-world failure tendencies of them yet.
Setting up backups
Even if you never have a hard drive fail, you still need backups. There are many ways for your data to disappear other than a drive failure: your house or office could burn down. Your computer could fail and scribble Shakespeare’s Sonnets all over your disks and data. You could be sitting in Starbucks and watch as someone grabs your laptop and runs out the door. You could drop your laptop (yes, I know, that never happens, right?). There are many bad things that can happen to your data.
The only way to protect yourself from these bad things is to keep multiple copies of your data. and since if your house burns down it may destroy everything inside it, not just your computer, you need to keep those copies in multiple places. This can turn into a hassle quickly, and one reality of backups is that the more hassle they are to do, the less likely it is you’re going to do them. So we need to keep doing and managing backups as simple as possible (but not too simple to be useful).
The basic goal of your backups is therefore to have at least three copies of your data, and have those copies exist in two independent locations.
My basic setup: back up data do a separate disk on a regular basis, and then swap that drive to an offsite location once a month. This gives you three copies of your data: on your computer, on your backup drive, and on your offsite drive. It minimizes cost, because you only need two backup drives that you swap. It limits the hassle factor, because as long as your backups are run automatically, you only need to intervene once a month to swap drives and take the updated one off-site.
One of the tradeoffs: not all of your data will be in all three places; your newest data won’t get out to the offsite until you swap disks at the end of the month. Remember, though, that the offsite backup is there to recover from catastrophic disasters (house burned down! oops!); the compromise between reduced hassle of constantly swapping that drive and losing some data in that situation is a reasonable one; in reality, you are unlikely to ever need that catastrophic backup. But if you do, you’ll be glad it’s there.
That said, it never hurts to have more copies of your data. You can do this in a number of ways. Using an offsite backup is one — our friends Backblaze, for instance, or Crashplan is another option. There are other companies doing this as well. The downside is that these services use your internet connection and that connection can be slow; if you have a lot of data, it can take a long time to upload them to the remote backup server and if your data fails before it’s backed up, you’re hosed. That’s one reason why I like to use these services as a supplemental backup and not a primary one.
Some ISPs put data caps on your internet connection. If yours does, doing an online backup could cause you to use more data than the cap allows and you can find your network throttled to a really slow speed, or turned off completely. Before you go online, you need to understand how big your data set it you want to back up, how long it will take to upload, how long it might take to recover if you need to, and whether you have a data cap to worry about. I generally recommend that people consider using these online services to back up the important data, but not everything.
Another online option are services like Dropbox or box.net or Google Drive. These services turn a part of your hard drive into a virtual folder that gets copied onto their servers, and then copied down to any other computer that you set up to share that virtual folder. This can be quite useful if you use multiple computers at different times, but it can also act as a kind of backup because the data gets copied to multiple places. It’s not something you should use as your primary backup, and like the other online backup services, slow network connections and data caps may impact its usefulness.
These are all ways to create multiple copies of your important data in relatively painless ways that you don’t need to spend time managing.
How to back up your data
This section assumes you’re using a Macintosh. If you don’t, there are other equivalent tools you can use to back up your computer, but I’m not the person to tell you which one to use.
Backing up a Macintosh can actually be very simple: use Time Machine. For a lot of people, this will work quite well and it’s free with all copies of Mac OS X. I use Time Machine for part of my backups system because I like it’s incremental backups so you can go back and find a file and it’s data at a given time.
Time Machine’s big weakness is large data sets. Because it’s doing incremental backups, it is going to want a backup drive larger than the amount of data you have created. I’ve found that it works best when the backup drive is at least 2X the data being backed up, and I prefer 3X. This means if you have, say, a 500Gb boot drive in a laptop and a firewire drive with 1.2 Terabytes on it, your total data set is 1.7 Terabytes. Time Machine is going to struggle keeping that backed up on a 2 Terabyte drive, so you really need 3TB for your backup at a minimum. If you update large parts of your stored data, you can really give it indigestion (for instance: take 1000 photos in Adobe Lightroom, and assign a new keyword to each, and make sure the updated metadata is flushed to the DNG with an embedded XML sidecar. You just created 60-70 gigabyte backup). The larger the data set, the larger the disk Time Machine needs to back it up and work efficiently, and as your data set continues to grow, this is going to be a challenge.
I am not a big fan of Time Machine to recover a failed disk. I’ve done it, and sometimes it works fine, and sometimes it’s fought me and taken forever to get the data restored. Apple’s done a lot of work improving Time Machine since the early days of Mac OS X so a lot of my reservations about it aren’t true if you’re running Snow Leopard or Mavericks — but I still prefer to have a way to recover an entire disk as well.
For that I use Superduper. This tool makes an exact clone of a disk, one that you can plug into a computer and use without any work; even boot the computer from it. I use it to make bootable copies of my computer’s main drives; so if I lose one, I can clone a copy quickly, or just boot the backup drive and get back to work. And it creates another copy of my data for me (never a bad thing).
Do you need this? How badly do you want to protect your data? How quickly do you want to recover from a drive failure and get back to work? How many hard drives are you willing to buy and manage? If your data is really worth the effort, it’s a good way to create a reliable and quick-to-recover copy of it — but it does entail more time, energy and money. Whether it’s worth it to you is a decision you’ll have to make. It’s worth it to me.
I am not a fan of Apple’s Time Capsule for backups. It’s very simple, but offsite backups are effectively impossible. Recovering a failed drive from it takes time, and it’s hard (to impossible) to replace the drive as it ages. I want the ability to upgrade my WIFI router separately from my backup drives. And Time Capsule is not a good solution if the number of computers to be backed up is two or greater. I do use one in one specific situation: my mother’s house with my mother’s Mac, where absolute simplicity is the prime directive. If your needs are simple and you’re willing to forgo offsite copies of your backup, it’ll do the job, but I think for most uses, it’s not the right solution.
What if I have big data sets?
As your data set grows, it gets more complicated. As the number of computers you need to back up grows, it gets more complicated. As it gets more complicated you’ll need to spend more time (and money) making sure you have good reliable backups and that the backups work. If you’re a serious photographer or a videographer, you’ve probably stopped thinking about gigabytes and now think about terabytes.
You can keep plugging disks into your computer to store all of that data, but that’s expensive, unwieldy, and backing them up is a horror (so chances are, you’ll stop and pray nothing bad happens). That’s a disaster waiting to happen. So at some point, you need to start thinking about disk subsystems, or network-based disks, or some other setup designed to handle large sets of data.
I’ve recently hit that point, and my choice was to go to a NAS, or a Network Attached Storage device. I talk about that in some detail in Should you consider upgrading your home network to a NAS?
Is this an option you need to consider? Here are my general guidelines:
If you’re managing a single computer, a NAS probably doesn’t buy you much, until your data set starts growing past 4Terabytes. At that point, you’re talking about plugging in multiple drives and multiple backup drives and things start getting complex, and the NAS will make your life easier and you’ll end up buying less hardware over time. If you’re someone who is wandering the house/office with a laptop wireless, a NAS starts making sense sooner because your data can live on the network and you don’t need to plug in to work on that project as often.
If you’re a multi-computer environment, the complexity of your data management and keeping your backups going reliably is going to be harder and harder. The NAS helps a lot with that, and so you should consider it. I think a good general metric is when you hit 2-3 computers and your total data you have to manage hits around 4 Terabytes, it’s a good time and cost effective to start considering a NAS. If your data requirements are small, you may not need one, but if you’re a photographer or videographer, your data requirements aren’t small any more.
Once you hit 5-6 computers in the installation, the advantage of centralized online backups to the NAS seem to be overwhelming. you’re an idiot to not consider it. IMHO.
If you’re in a single-computer setup, another option are dedicated disk arrays that connect via Thunderbolt or Firewire like the Drobo. I personally think the NAS is a better option and most of the time will be less expensive and more flexible, because I like the ability to connect to it over WIFI if I grab the laptop and wander around the house. The direct-connect systems like Drobos, on the other hand, will win on pure performance, so if you need absolute max performance, they’re your better option.
My backup strategy
I’m going to close out by documenting my current backup strategy. Not everyone is going to want to implement all of this but I want people to see what I do and understand why, and have the ability to adopt in the pieces that make sense. My data situation is moderately large and I have predicted that growth will accelerate. We’re a three computer family, two of us are photographers and I’m starting to work with some video. My photo collection is well past 30,000 images, and my wife’s is 20,000+. So we have a big hunk o’ data.
I’ve just migrated to using a NAS, and I no longer have a second (or third, or fourth) drive attached to my computer. I have the boot drive on the laptop, which is a 500GB SSD and everything else lives on the NAS.
My wife keeps her data on a mirrored RAID drive (in part because she hasn’t had time to sort out what should get moved to the NAS). All three computers are backed up to the NAS via Time Machine. The NAS has a backup capability, so I back up all of the data onto two external drives, and its those drives that get swapped offside monthly.
Here’s a diagram that shows everything involving data on the home network
Here’s what’s going on:
- Each machine uses Time Machine to back itself up to the NAS. Each machine has its own partition with a quota set on it, because otherwise, Time Machine will grow the backup to infinite size. The quotas are around 3X the size of the backed up data.
- Each machine uses Superduper to write update a disk image on the NAS, kept in the data volume as a Sparse Bundle. I can load that onto drive if I ever need to do a recovery.
- Each machine has access to a personal data volume and a shared data volume we both use. I have my iTunes library out there and shared, and I keep a morgue, which is data I keep but which if I lose, I won’t die, so it doesn’t need to be backed up (I currently do, but as my data set grows, I’ll stop that).
- The NAS backs up to two disks. I don’t need two today, but this gives me breathing room so I don’t have to update this for a while. A second pair of disks lives offsite and is swapped by sneakernet monthly. (for what it’s worth, a full backup of the NAS currently takes about 3 days).
- I have two other disks hanging off my Macbook; these are my travel disks. One is a 500Gb drive that is bus powered (no need for external power); I use that to clone my laptop drive every night when I’m on the road, and I plug it in once every week or two and update the clone via Superduper; one more copy of that data hanging around. The other travel disk is a 500GB mirrored raid that’s bus powered, and I use that to store data on longer trips when the size of my created data is larger than my internal drive can handle. With photos and video, that’s not hard… Both of these drives are from Other World Computing and built like tanks.
What this means is that once it gets copied offsite, all of my data lives in at least three places (NAS, backup, offsite backup). 24 hours after creation, it’s in two places at the minimum. Any data that lives on my laptop drive ends up with at least five copies, and I also use Dropbox for some data, which makes even more copies including at least two computers at work…
That’s my comfort level for trying to prevent data loss. Do you need to do all this? Depends; how bad would it be to lose your data? Choose the pieces that get you to your comfort level. The really good news is that once this is all set up and running, it takes almost no time to keep going; other than swapping the backup disks (which takes up an evening, roughly) on the NAS, it’s all automated.
Setting it up takes time; getting fully running on the NAS too me two and a half weeks. And it takes some money to invest in the gear you need to add to get things going. But those are investments in not having that freak out panic attack later when a disk fails.
And you’ll sleep better at night. I know I do.
How comfortable are you at the thought that someone just grabbed your computer and ran out the front door of that Starbuck’s your sitting in? Will your backups protect you? If not, you have some work to do…