Storing Irreplaceable Data

[fatguy]

I'm curious how everyone is managing their irreplaceable data. Does digital imaging / video recording give consumers comparable data management requirements to the enterprise? RAID is a good start, but what do you use for backups? Doing it 'right' can be extremely expensive - are there any cheap solutions?

[bLack0ut]

RAID is more "High availability" than backup. Backups would be offsite, and offline.

My order of preference goes like this:
RAID-0 on machine with data
Weekly backups to onsite fileserver (separate machine, same network, same power)
Weekly backups to offsite fileserver (separate machine, different network, different power) - this one can be repeated many times
Burning CDs/Tape drives (depending on your budget) stored in a safe or bank. - this step can be repeated many times

How far I go depends on how important the data is. Usually, RAID-0, weekly backups to one onsite fileserver and backups to 2-3 offsite fileservers is sufficient for small businesses, but it's all preference.

[JCLW]

I have a RAID 1 array at home that is backed up to a second RAID 1 located at my parents house (200km away) via FTP.

The majority of my data is images with a tiny percentage being financials, etc. Probably roughly 500GB in data that I would really like to keep all in all.

I use Cerberus FTP Server and Cobian Backup (both are available free for personal use).

[seadave77]

I use Jungle Disk to back up my data to Amazons S3 storage service.

http://www.jungledisk.com/

[KTE]

You'll need to define what you call "cheap". It differs vastly from one to another. ;)

I personally use a very expensive layered approach for backing up my share of such critical data. 5/10 layers are provided by the government for my profession so its feasible. Recently I realized 3 layers was too small for good safety when all 3 backups of the same personal content managed to go out on the same day whilst running basic maintenance on the new drives. Something I had never envisaged nor experienced happening priorly, but, I know how I felt when it did.

[darthjones]

How about SD cards? Throw your data onto 4 or 8 gig ones and then lock them. How's that for cheap storage? Plus, no moving parts so you can store them for years or decades! At least, in theory... Do SD cards lose their data after a few years?

[c627627]

Connect external hard drive to power cord with an on/off switch.

The switch is usually off - physically protecting any unauthorized access to it.

[rainless]

I'm curious how everyone is managing their irreplaceable data. Does digital imaging / video recording give consumers comparable data management requirements to the enterprise? RAID is a good start, but what do you use for backups? Doing it 'right' can be extremely expensive - are there any cheap solutions?

RAID isn't an option for backup.

You want to backup to multiple removable sources: CD, DVD, *TAPE* if necessary...

Small files like photos you can backup on the web. I store a lot of my books, stories, and articles online.

Multiple sources are the key. Just in case you lose a CD or DVD.

Another cheap backup media is 8.5gb Dual Layer DVDs. A few of those can handle even the biggest backup jobs.

If you can afford it you can do 50GB Bluray too. I believe those discs cost around 15-20 bucks. Still reasonable.

But I would still store some stuff online just in case your house burns down.

"Blessed are the pessimists: For they have made backups."

[m²]

Nowadays HDD is my choice for backup.
It's cheap (much cheaper than Bluray / CF), fast, dense, erasable.
From my experience, lives about as long as optic discs, but I admit I never used ones meant for archiving, they cost more than (8GB@HDD now+8GB in 3 years+8GB in 6 years+..+8GB in 27 years).

For critical data I have 4 backups, for other - 2.
The primary one is in my computer and is meant to protect me from my mistakes like accidental deletion or infection. (For critical data there's also a copy at my main disk)
Another one is in my parent's house, 70 km away, synchronized monthly (I intend to synchronize daily / weekly over the net, but I'll have to buy them a server first, which delays the whole thing).
The 4th critical backup is online. I use a customized encryption scheme for it, I don't trust that anything is unbreakable and nobody's gonna search for custom encryption vulnerabilities just to read my private stuff.

All backups are checksummed and scrubbed weekly (at home) / monthly (offsite).
I intend to move to backups with ECC, but currently there's no solution appealing to me.

[Surfrider77]

I use a 1TB RAID 1 array for all of my storage. Every month or so, I back that up to an external 1TB drive via Firewire. Call me paranoid, but I went through the agony of losing ~5 years worth of pics and videos and things of some of the best years of my life living in London in my early-mid 20s. Didnt know any better and stored everything on a single HDD. It failed and I lost everything. Now, I am extra extra cautious. :D

[ihrsetrdr]

I back-up my irreplaceable data to pendrives like these (http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&N=2013240522+1309429390&Configurator=&Subcategory=522&description=&Ntk=&SpeTabStoreType=&srchInDesc=). Then, they go "off site" for safe keeping, in case the house burns down and takes all my RAID arrays with it.

[c627627]

I also keep additional copies stored with family on optical media (DVD R's) just in case one of those magnetic bombs hits and wipes out the data stored on drives... :soda:

[hafa]

All critical data resides in a file server with a RAID 1 array, giving it protection against a single drive failure. This array is synchronized/versioned daily with both an external drive and my laptop, so at any given moment, the most crucial data has 3 copies, one of which (laptop) is with me most all of the time.

[Archer0915]

Read my sig and in case of fire well one drive stays with me.

There is something called carbon or something itr is an online backup (I don't trust systems I dont control). OH yeah almost forgot I am about to install a NAS in my shop for that moment when the drive is not in the safe:)

[Ninety-9 SE-L]

Depends on how much data. If we're talking about <32GB, get a thumb drive or a compact flash and just copy your files over. Encrypt if they're confidential.

IMO, Hard drives are getting very cheap and are by far the cheapest cost per GB.

I like RAID, but RAID only protects against a single HDD failure (in most cases), in which case, that failed HDD must be replaced and the data rebuilt. RAID doesn't protect against human error or software error (say a file gets deleted or corrupted).

There is no replacement for a good backup system. The most cost effective way of backing up files is to only back up the files you need the most. Either by a backup program, moving files manually, or folder syncing. A back-up program would be more redundant in most cases, which would allow you to go back to a specific date and restore a file, manual moving or sync will replace the file with the latest version, preventing you from rolling back to a previous version.

My recommendation, multiple hard drives without RAID, either mount them inside your computer or get one of these:
http://www.newegg.com/Product/Product.aspx?Item=N82E16817816009 (I have a Thermaltake one, but it appears this one is the exact same one, relabeled). I know they've been around for a while, but I never realized how cool they are until I got one.

Pop it in, back up your files, store it in a safe place, maybe even a small fire-proof chest if it's VERY important to you. By using the HDD as a removable device, it preserves the drives MTBF and also protects it against power surges, viruses, etc.

[freeagent]

I also keep additional copies stored with family on optical media (DVD R's) just in case one of those magnetic bombs hits and wipes out the data stored on drives... :soda:


me too :thup:

[gangaskan]

Connect external hard drive to power cord with an on/off switch.

The switch is usually off - physically protecting any unauthorized access to it.


that only helps with part of the problem. one "huge" issue is if a fire breaks and destroy's your data, or some other significant disaster your data is done no matter how you had stored it.


its best to have a offiste for that "just in case" moment.

[cyberfish]

I don't trust that anything is unbreakable and nobody's gonna search for custom encryption vulnerabilities just to read my private stuff.

I would trust open and proven algorithms over "custom" secret encryption any day. There's a paper written by a cryptography expert that went into quite a bit of detail on this topic, but I can't seem to find it now. Essentially, what it says is, popular algorithms are usually a better choice because they are well tested and proven, and thoroughly studied and peer-reviewed. The GPG algorithm, for example, was released more than 10 years ago, and is used in many mission-critical environments (government agencies, businesses, banks...), and has never been broken (there sure are A LOT of people interested in breaking it). The algorithm itself is open, too, anyone can see how it works, yet no one has broken it.

"Proprietary" algorithms need to be secret because objectively and mathematically, their algorithms are not nearly as good. They need to keep it secret because otherwise it will be easily broken. The problem is, no program is truly closed. Your CPU needs to execute those instructions, so the instructions have to be given to you in the binary. It will only take a mediocre hacker (in the original sense) a few hours to disassemble the program. That is to say, they can be possibly easily broken if there's enough interest/reward. Open and proven algorithms are really the way to go.

As for the original topic..

I also keep additional copies stored with family on optical media (DVD R's) just in case one of those magnetic bombs hits and wipes out the data stored on drives...

I have had a few DVD-R backups fail on me... so be aware. I do checksuming (MD5, aka hashing aka file fingerprinting) on all my backup, so I know when they are not exactly the same bit by bit as when I made them. If you have a few thousand files or more, and only a few of them are corrupted (as in my case), you will statistically probably never notice, but I think that's a bad thing, so I always check my backup.

A possible fix would be redundancy - making a few copies of each DVD. But then you will need a way to know which copy is right when 2 differ, and checksuming is the answer.

[Enablingwolf]

For critical backups. I use an external drive left unplugged until it is needed. I physically store it in my home fireproof safe. It is by my thumb drives for other types of stuff. Short of a bomb, I cannot think of a safer/secure way for me to save data. On the cheap.

For easier backing up and doing dailies (give or take.) I use the hot swap caddy on one of my machines. Doing most my backing up over the network.
It to me works like a ROM, but holds much more. I simply store them drives in a cardboard box. Sometimes under my computer on a shelf. I did drop one drive once, but so far it still works 6 months later. it is not a mission critical drive. Just an easier way for my to get images and backups on a large drive.

[m²]

I would trust open and proven algorithms over "custom" secret encryption any day. There's a paper written by a cryptography expert that went into quite a bit of detail on this topic, but I can't seem to find it now. Essentially, what it says is, popular algorithms are usually a better choice because they are well tested and proven, and thoroughly studied and peer-reviewed. The GPG algorithm, for example, was released more than 10 years ago, and is used in many mission-critical environments (government agencies, businesses, banks...), and has never been broken (there sure are A LOT of people interested in breaking it). The algorithm itself is open, too, anyone can see how it works, yet no one has broken it.

"Proprietary" algorithms need to be secret because objectively and mathematically, their algorithms are not nearly as good. They need to keep it secret because otherwise it will be easily broken. The problem is, no program is truly closed. Your CPU needs to execute those instructions, so the instructions have to be given to you in the binary. It will only take a mediocre hacker (in the original sense) a few hours to disassemble the program. That is to say, they can be possibly easily broken if there's enough interest/reward. Open and proven algorithms are really the way to go.
How long password can you remember? For me that's<20 characters. Storing a stronger one on the backup location is pointless. Therefore brutefoce can break any encryption scheme...and CPU power is getting cheaper and cheaper very quickly, at some point cost of breaking all encrypted archives might be feasible for online storage vendors. At this point no common scheme will be secure.

Actually I use double protection: custom encrypted files inside AES encrypted 7zips. I don't trust AES, but it doesn't cost me anything.
My encryption scheme was designed to be easily breakable...but to require intelligence to succeed. That's because I've seen too many people loosing data by some mistakes, I didn't want something I couldn't reverse.

Human breaking my encryption? Sure, possible. But files inside are not worth that much for anybody but me. If they were, they wouldn't go online in the first place.

[cyberfish]

How long password can you remember? For me that's<20 characters. Storing a stronger one on the backup location is pointless. Therefore brutefoce can break any encryption scheme...and CPU power is getting cheaper and cheaper very quickly, at some point cost of breaking all encrypted archives might be feasible for online storage vendors. At this point no common scheme will be secure.

My regular password is just 15 alphanumeric characters.

That is more than enough possibilities.

36^15 = 221073919720733357899776 (36 because 26 letters + 10 digits)

Modern personal computers can do about 1 million "instructions" per second. For simplicity's sake, let's assume they can try one password in one instruction. And then let's assume the fastest supercomputer on Earth can do about 1000 million IPS (highly unlikely, but feel free to look it up). It would take such a computer 221073919720733 seconds, or about 7 million years, to bruteforce it.

By Moore's law (computing power doubles every 1.5 years, which probably won't be the case for long, and is already slowing down), it would take log_2(7000000)*1.5 = 34 years, before the fastest supercomputer on Earth will be able to brute-force my password in 1 year.

That's enough security for me.

For anything more serious, though, a password could be run through a hashing algorithm like SHA, and the resulting hash can be used as the key (a 256-bit key, for example, which is considered short by today's standards for mission-critical applications, would be equivalent to ~32 characters - enough to keep the supercomputer busy until the Sun burns out).

[m²]

My regular password is just 15 alphanumeric characters.

That is more than enough possibilities.

36^15 = 221073919720733357899776 (36 because 26 letters + 10 digits)

Modern personal computers can do about 1 million "instructions" per second. For simplicity's sake, let's assume they can try one password in one instruction. And then let's assume the fastest supercomputer on Earth can do about 1000 million IPS (highly unlikely, but feel free to look it up). It would take such a computer 221073919720733 seconds, or about 7 million years, to bruteforce it.

By Moore's law (computing power doubles every 1.5 years, which probably won't be the case for long, and is already slowing down), it would take log_2(7000000)*1.5 = 34 years, before the fastest supercomputer on Earth will be able to brute-force my password in 1 year.

That's enough security for me.

For anything more serious, though, a password could be run through a hashing algorithm like SHA, and the resulting hash can be used as the key (a 256-bit key, for example, which is considered short by today's standards for mission-critical applications, would be equivalent to ~32 characters - enough to keep the supercomputer busy until the Sun burns out).
Modern cpus do over 12 billion operations/second. 3.2Ghz*4cores. I guess that SIMD extensions further increase this number, but I don't know CPU construction well. GPUs are approaching 1 trillion. But because trying a password takes much more than 1 instruction, your calculations (for CPUs) don't seem far off.

And in recent years supercomputers evolve way faster than Moore's law predicts. Currently the top does 1.6 quadrillion ops in benchmark (not theoretical throughput). 20 quadrillion is scheduled for 2011.

I'm too lazy to do calculations, but 34 years is little anyway. I expect that I have up to 60 years of life left.

[cyberfish]

My "instruction" is as in a general "operation" - just an extremely rough estimate. Not just 1 CPU instruction. Many CPU instructions take more than 1 cycle, too. The integer division, IIRC, takes 24 cycles on the Core 2.

My point, though, is that in the forseable future (unless quantum computers or something kick off and give everyone a supersupercomputer, and make supersupersupercomputers), brute-force won't be a problem. Even a relatively short 512-bit key, due to the exponential nature, would take the supercomputer of year 2100 a few universe's lifetime to brute-force.

[hafa]

...sync will replace the file with the latest version, preventing you from rolling back to a previous version.

The better synchronization programs, such as Vice Versa Pro (http://www.tgrmn.com/) allow you to do file archiving (http://www.tgrmn.com/web/kb/item22.htm), i.e., save former versions of files which are changed or deleted. This is a great tool not only for backup, but to do audit tracking as well.

There is something called carbon or something itr is an online backup (I don't trust systems I dont control).

That would be carbonite (http://www.carbonite.com/).

[fatguy]

Some good comments and experiences here - thanks for sharing!

I have the added requirement of hosting a lot of this data (pictures, videos, etc) on a fileserver for R/W access on my network. (I do NOT want to be the bottleneck for my wife uploading thousands of pictures of our 6 month old son). I run a 500GB Raid 1 array in a separate server, all protected with a UPS synched to shutdown the server after 5 minutes of power loss. I use a tool called "rdiff-backup" to archive segments of the file server I want to protect against accidental deletion (pictures, documents, etc).

This works well, but I'm concerned mostly about theft - somebody breaking into my apartment and walking out with my machines. I like the idea of an "attach for backups only" solution to mitigate this, or possibly archiving to a remote server at a friends house or even S3.

Thanks for sharing your experiences - keep em coming!

[Enablingwolf]

If you have a safe. You can use hotswap caddies. Using data encryption. When your backups are done. Then pull the drives and put them up.

I use a thumb drive for my online banking. When I am done banking. The drive gets pulled and put in the safe. It is the only reason I use that profile on Firefox and that drive. I have another drive for other sensitive data. It also is stored in the same manner. When I use O_O or spreadsheets. It goes right to the thumb and in the safe.

Depending on how sensitive the data is. I just encrypt the drives on the machines themselves. Having secured user log in and such is about all you can do to stop physical exploits.

Removable drives is a cheap and safe way to mitigate theft of physically stored data. You just need a safe way to store them drives. If someone rips out my safe. They are going to get my extra credit cards and SS card, plus birth certs and goodies like that. So it all goes in the secured fireproof safe.