When we back up our business or personal data files, we often think our job is done if our backup utility didn’t inform of us any problems. All is well with that part of our world, we think. But is it really?
Image may be NSFW.
Clik here to view.
Data rot may not be a term you’re familiar with, but if most of your family photos are in digital format, you will want to become very familiar very quickly.
Storage Media is Not Forever
Regardless of the medium, the data we store will eventually suffer decay. You can think of it as the law of entropy conspiring against you. The cause varies:
- Magnetic media – Disk drives and magnetic tape (i.e. the VHS video tape of your wedding) can experience data decay as the individual data bits loose their magnetic orientation. In the normal scheme of things, error-correcting features of the hardware catch most of this, but not all.
- Optical media – Your CD’s and DVDs and BDs (Blue Ray). The physical materials used to store the data degrade with the passing of time. You can slow this down with proper storage, but you can’t eliminate it.
- Solid state media – USB keys, flash drives, the SSD drives in your laptop, etc. These store data as electrical charges, which are subject to leakage due to imperfect manufacturing techniques. In other words, it can’t be helped. Again, the hardware is designed to compensate for this to a degree, but some amount of data decay is unavoidable.
The File System Doesn’t Care
If you are one of my more technically-minded readers, you may be thinking the file system is taking care of data quality for you. Sadly, it isn’t. It’s merely keeping track of where everything is located. If all of the pieces of all your files are in the right places and all in the right order, and it can see and access all of them, the file system is happy. The so-called “modern” file systems we are given to use do not do data integrity checking.
Demo Bunny
A picture is worth a 1,000 words, so here is illustration of data decay using the jpeg photo above that I’d really hate to lose. We can simulate data rot by opening this file in a hex editor and changing just one single bit. Here’s the result:
Image may be NSFW.
Clik here to view.
Not so bad, you say. It’s just a bunny after all, and he’s mostly still there. Let’s change just one more bit in the file. Things start going downhill in a hurry. Now imagine this was a photo of your first born instead of my fuzzy buddy. Two bits of data out of whack, and we have irreparable damage to the family archives. Not Good.
Image may be NSFW.
Clik here to view.
These two tiny changes I manually introduced are exactly the kind of thing that happens on your hard drive without your knowledge. Does it happen every day? Most likely not, but it is happening.
So we know that the file system doesn’t care, and your data will decay without detection or any notification. Consider now that your daily backup program is happily backing up that corruption day after day, overwriting the files that matter to you with corrupted versions. This is a problem you should take seriously.
What You Can Do
Until we have better file systems that check the integrity of data (such as ZFS), the best we can hope to do is be vigilant and stay ahead of the rot. I suggest a three-step strategy.
Step 1 – Make multiple backups of your most important files, stored in different formats or locations. In my own case, I back up my files three times using a combination of SuperDuper, Time Machine, and CrashPlan. SuperDuper provides a daily image-based copy of my system on an 1TB external USB hard drive that I can boot from if my system hard drive fails. Time Machine does a file-based copy on an hourly basis, also to the external USB drive. CrashPlan is a pay-for service that I use to do file-based backup over the Internet into the CrashPlan cloud. This gives me a remote copy of my data in the event that a catastrophic event destroys my home, or a burglar pilfers my gear.
Step 2 – Periodically check your data to verify it’s still correct by actually reading it. For your financial data, actually load the spreadsheet, Quicken file, etc. and run some reports. For photos, you really need to do this visually as proven by the example above. A nice activity, perhaps, for over the holidays when you and your family or friends could “flip through” your digital album.
Step 3 – Copy your most valuable data to multiple different types of media. Take for example the wedding video on VHS tape that I mentioned above. Convert that to DVD format, or better yet to a video format such as MP4 or AVI and copy it to the cloud using a storage service like Dropbox or iCloud.
The post Why You Should Care About Data Rot appeared first on //ScottKantner.