Filed Under (Uncategorized) by Dave Mast on July-13-2007

We recently had an experience here at NewPointe that clearly demonstrated the value of having a good backup/data protection solution.  I’ll keep it as short as I can. 

We recently had a bulk of audio/video files sitting on a RAID 0 array on our editing system.  These files consisted of recorded weekend services, logged/captured footage for a project in the works, and all of our Final Cut Pro project files as well.

A couple Saturday nights ago, I walked into the editing room and heard a loud scratching noise coming from the direction of our editing system.  This scratching noise was in fact coming from one of the 3 hard drives in that RAID 0 array…and was being caused by the heads rubbing against the platters at around 7200 RPM.

For my readers who don’t know a whole lot about hard drives, this basically means that the hard drive was utterly hosed, and because this hard drive was part of a non-redundant array, the data on the entire array (all of our weekend recordings, projects files, etc.) were equally hosed.  The hard drive that died was less than 9 months old.

We had a backup solution in the works, it was actually sitting NEXT TO the editing system, but it had not been completely implemented yet, as we were waiting on parts.  It’s impossible to point a finger to any one thing/person in a situation like this… it was really no one’s fault.  Nonetheless, it was extremely unfortunate, and I’m betting we will still feel this loss a few months from now.

I had to break the news to our video editor and my boss that we had suffered a significant loss of data.  In a situation like this, there’s no amount of off-subject banter or workaround ideas that you can come up with that will make anyone feel better.  No one was happy, including myself.  I was actually sick to my stomach for a few hours.

Of course, I did call around to get pricing for repair on the array.  Problem is, the fact that it’s an array make the repair process VERY complicated (at least that’s what I’ve been told).  Prices ranged anywhere from $4500 to EIGHT THOUSAND DOLLARS.  As one might guess, we ended up not sending the drives in for repair.

The ONE positive thing I was able to pull out of this situation was this:  We were moving in the right direction, and there was no one who was against getting that data backed up.  Unfortunately for us, that failed hard drive caught us before we were ready.  Had we not had anything even close to being in place, I would feel quite different.  Ultimately though, it doesn’t really matter.

So I’ve said all that to say this:  For those of you who are teetering back and forth about getting a backup solution implemented:  Freekin’ DO IT.  If you’re already storing critical data on a single hard drive or a non-redundant array, get a backup solution (or at LEAST a RAID 5 array) going and get it quick.  It costs much less to get a decent backup solution in place than it does to repair a hard disk (much less an array), and you never know when you’ll hear a loud scratching noise coming out of YOUR machine.



Comments
Tom Troyer on July 13th, 2007 at 3:03 pm #

Ouch … I’ve seen this happen before and I’ve seen a backup save a year & 4 months of data too. I know that sick feeling in the stomach you felt. Did you have anything burned to CD or DVD?

Dave Mast on July 13th, 2007 at 8:20 pm #

Unfortunately no… a majority of these files were large HD video files, some as big as 60GB. We thought about chopping the files up and burning them to Blu-ray disc, but a single 50GB Blu-ray was $35 last time I checked.

Jeff and I spent some time this week looking over Final Cut and what we can do to preserve an acceptable amount of picture quality and keep the file size down as much as possible. We think we may have landed on something good, but we’re going to give it a full run-through this weekend.

Austin Spooner on July 13th, 2007 at 8:49 pm #

Dave, I know this seems weird, but trying putting the drive between two packs of dry ice to keep is super cold and then hook it while being between the dry ice. We had this happen and since the drive was so cold it was able to get back up long enough to get all the data off and then it was totally hosed. Just an idea.
Austin

Tony Dye on July 15th, 2007 at 7:25 pm #

Dave, sorry that you had this awful experience. I hurt with you!

One extra comment — near the end of your post you mention “at least get RAID 5″ and I’m fearful someone will misunderstand that to mean if you ahve RAID 5 you don’t need another form of backup. (Because I’ve heard that said before) I don’t believe that was what you were meaning, but someone might get the wrong idea…

Dave Mast on July 16th, 2007 at 6:48 am #

You’re right on, Tony. Having a RAID 5 array certainly doesn’t constitute a safe backup of your data, all you get there is decent fault tolerance.

This poses a big challenge to people (like myself) who work in video and manage files that are several gigabytes (sometimes 50+) in size. How do you do effecetive data protection on files that large?

This is one of those times I wish our video infrastructure wasn’t in HD. The files would be much smaller. ;-)

Post a comment
Name: 
Email: 
URL: 
Comments: 

FireStats iconPowered by FireStats