We recently had an experience here at NewPointe that clearly demonstrated the value of having a good backup/data protection solution.  I’ll keep it as short as I can. 

We recently had a bulk of audio/video files sitting on a RAID 0 array on our editing system.  These files consisted of recorded weekend services, logged/captured footage for a project in the works, and all of our Final Cut Pro project files as well.

A couple Saturday nights ago, I walked into the editing room and heard a loud scratching noise coming from the direction of our editing system.  This scratching noise was in fact coming from one of the 3 hard drives in that RAID 0 array…and was being caused by the heads rubbing against the platters at around 7200 RPM.

For my readers who don’t know a whole lot about hard drives, this basically means that the hard drive was utterly hosed, and because this hard drive was part of a non-redundant array, the data on the entire array (all of our weekend recordings, projects files, etc.) were equally hosed.  The hard drive that died was less than 9 months old.

We had a backup solution in the works, it was actually sitting NEXT TO the editing system, but it had not been completely implemented yet, as we were waiting on parts.  It’s impossible to point a finger to any one thing/person in a situation like this… it was really no one’s fault.  Nonetheless, it was extremely unfortunate, and I’m betting we will still feel this loss a few months from now.

I had to break the news to our video editor and my boss that we had suffered a significant loss of data.  In a situation like this, there’s no amount of off-subject banter or workaround ideas that you can come up with that will make anyone feel better.  No one was happy, including myself.  I was actually sick to my stomach for a few hours.

Of course, I did call around to get pricing for repair on the array.  Problem is, the fact that it’s an array make the repair process VERY complicated (at least that’s what I’ve been told).  Prices ranged anywhere from $4500 to EIGHT THOUSAND DOLLARS.  As one might guess, we ended up not sending the drives in for repair.

The ONE positive thing I was able to pull out of this situation was this:  We were moving in the right direction, and there was no one who was against getting that data backed up.  Unfortunately for us, that failed hard drive caught us before we were ready.  Had we not had anything even close to being in place, I would feel quite different.  Ultimately though, it doesn’t really matter.

So I’ve said all that to say this:  For those of you who are teetering back and forth about getting a backup solution implemented:  Freekin’ DO IT.  If you’re already storing critical data on a single hard drive or a non-redundant array, get a backup solution (or at LEAST a RAID 5 array) going and get it quick.  It costs much less to get a decent backup solution in place than it does to repair a hard disk (much less an array), and you never know when you’ll hear a loud scratching noise coming out of YOUR machine.

I thought it might be nice to mellow out DaveMast.net a little, so as of tonight, the site has a new theme on it.  I’m liking it, and I hope you do too.

For the second time in 6 months, I’m packing boxes and getting ready to move to a new place.  This time, instead of moving my office from Sugarcreek FMC to NewPointe, I’m moving my living space from Sugarcreek to Dover.

This wasn’t a rush deal, really it wasn’t.  I didn’t HAVE to move out immediately, but between my workload and other things, the timing was just right.  I had mentioned yesterday that I already had my eyes on an apartment that has good looks, great location and pretty reasonable price  (and the landlord is pretty cool too…he’s an IT guy!).  Things worked out really well over the past 24 hours, and today I was able to sign a lease agreement for the apartment.

So I’ve been packing for the move.  I knew I was going somewhere, so I began packing before I even knew I had a lock on the apartment.  Some people say “you don’t know what you’ve got until it’s gone.”  I’m saying that “you don’t know what you’ve got until you try to put it all into cardboard boxes!”  GOOD GRIEF!  I’ve got some lessons to learn on excess, for sure.

99% of the packing is done though, and now I’m gearing up for the fun gruesome next stage… transport!

This week was a relatively light week in the IT and Video departments, which has come in real handy.  I had the opportunity to take Friday off to do some apartment hunting.

I drove around the Dover/New Philadelphia area today and checked out about 5 different apartments (and talked to their respective landlords as well), and so far I’ve found one that I really like.  Good price, good living conditions, a good amount of flexibility, and a VERY nice quiet location (right now, the house that I’m renting is off of a state highway, and I would really like to find a quieter dwelling, so this is huge.)

Hopefully, if things work out, I’ll be going into the moving process over the next 1-2 weeks.  I’m hoping I’ll be able to post something decent during that time, but I’ll make no guarantees.

As for the Time-Warner install, they haven’t been around since Wednesday.  Hopefully next week NewPointe will make the jump to cable.

Again, it’s been one of those weeks where the IT/Video guy is doing a lot of running.  It’s not bad, but just enough to keep me away from a computer for anything beyond repairs or email.

• I’m falling into an odd sleep pattern during the weekend that I am starting to like:  I’ve been sleeping on Sunday afternoon and evening, and then in the late evening I’ve been getting up and going to the church to take care of some IT issues.  Some would call that sick and wrong, but I’ve yet to find such a span of time that’s so free of distractions.  During the 8 or so hours I have before the staff rolls in, this gives me a chance to do maintenance on PCs that would be otherwise occupied, and also work on servers that I can’t bring down during the workday.  PLUS, since I’ve napped it up all afternoon, I’m wide awake the whole time.  I’m not suggesting that people do this…. this is just what works for me for the moment, and I justify it by staying at home on Fridays.

• After a few months of doing repairs and upgrades in my cubicle, I finally managed to land a storage space to use for a workbench area.  Trying to work on machines in a cube is just too messy.

• Jason has a great post on his blog regarding the importance of planning and testing backups.

• More NewPointe bloggers have hit the scene:
  • Danny Bunnelle, our music director, has begun blogging on all things creative.  You can find his blog, “iBlogCreativeStuf” right here.
  • Our very own Kevin West has also entered the blogging realm.  You can find his blog, “See The Invisible” over here.

• I realized this past week that warm weather makes me feel profoundly happy.  Winter, although it usually averages out the same, always seems to take longer than it should.

• I’m going to be moving.  I’m sort of looking forward to it, but not the actual moving process…. so much work….and SO much JUNK.

 Hopefully things will calm down soon and I’ll be able to get my brain realigned.

I strolled into the control room this morning to do some editing, and found a note on one of our monitors that cracked me up.  Apparently one of our staff has witnessed OS X’s “jumping dock icon” for the first time. 

[youtube QcJiybmTTgo]

In case you can’t read the handwriting, it says “Coach, pick me!”

Perhaps you had to be there, but I thought it was funny enough to warrant a post.

Sorry for the lack of a post over the past few days.  Things have really piled up fast in the world of video, so I’ve been scrambling hard to get them done.  I’m definitely ready to go back to the normal pace.

Until then, have a Happy DST Weekend if I don’t see you before.

Man, it’s been a crazy week.  I’m going to try to recap the week and get my brain unloaded.  Bear with me.

This Tuesday we attempted a P2V (Physical-to-Virtual) conversion on our domain controller.  It worked, but not as well as I expected.  We wound up losing the SYSVOL share (where all your group policy stuff is kept) during the conversion.  We still had our physical machine present, so we fired it back up and all was well.  As a remedy for the next time we try this, I set up a second domain controller and stuck it in one of the IDFs.  Hopefully the domain data will replicate from the new DC to the old, and if not, we can always start from scratch with a new VM.

Also on Tuesday, we started implementing run logs on our servers.  Nothing fancy, just a simple Wordpad document that we will update any time we do something to the server.  This will come in very handy if we ever need to backtrack in the event of trouble.  I’ve been slacking pretty badly on proper documentation, so this is part of my process to get into the “documentation groove,” if you will…

Wednesday - We moved our QuickBooks install to a separate server.  For those of you who don’t use QuickBooks 2006 or have an earlier version, QB2006 actually uses an agent to access your company file if you host it on a server, and you get some very fast multi-user action as a result.  However, the server that our QB lived on was also (at one time) our Exchange server, our file server, our Antivirus server, and a domain controller all at the same time (yuck).  Over time things really started to get sticky on that machine, so we’ve been sticking our other services inside VMs and just letting the file server host some of those VMs.  As a result, things are running cleaner and more reliably.

Today (Friday) I spent bit of the morning talking to Ed as we tried to get a second firewall to work properly on our network.  For whatever reason, we didn’t have a lot of luck, but we DID narrow it down to the firewall as being the issue, and not something else.  On Ed’s recommendation, I’m going to give ISA a run and see how it flies; perhaps maybe we’ll just go that route instead.  Big props to Ed for going above and beyond to help me out through this process.

I was also VERY pleased when I looked out the window of my house today.  I could see fresh buds starting to form on the trees outside.  YEAH!! :-)  You have no idea how ready I am for spring.  Hopefully we won’t drop back into the 10-20s during March.

This past Friday I had the opportunity to spend the day with Jason Powell and Ed Buford from Granger Community Church.    GCC is about a 5-hour drive from my place and I dig road trips, so I decided to make a one-day gig out of it.

I rolled into the parking lot around 9:30 and met up with Ed and Jason soon after.  We spend the day covering a miriad of different topics, from SANs to backup strategy to VLAN-ing, and a bunch of other things in-between.  I came home that night feeling absolutely exhausted but very inspired and extremely excited about what I do (not that I don’t usually enjoy what I do, but I feel especially pumped).

I can safely say that this was no-doubt the best learning experience I’ve had in the IT field to date, and it was also a great example of what can happen when you get out of the solo gig and start networking yourself with other people who do what you do.  Monster props to Ed and Jason for taking time and allowing me to hang with them for a day.

Tomorrow I’m going to have the privilege of going to Granger Community Church to spend the day with Jason Powell and Ed Buford.  I am PUMPED, man! 

I stumbled onto Jason’s and Ed’s blogs sometime in spring 2006, and then met both Jason and Ed at the 2006 Church IT Roundtable.  The Roundtable was a HUGE eye-opener for me, and I have learned an amazing amount of stuff just from that experience. 

So naturally I’m really looking forward to this trip.  We don’t have an extensive list of pre-planned discussions, but any time you put 2 or more IT geeks in the same room, discussion will always ensue (it’s true!).  I’m taking my laptop and plenty of paper so I can capture whatever we talk about.  This trip will definitely be beneficial.