When our control room for live video was installed, it was installed in a hurry. Now, that’s not a finger-point at anyone that was involved, we really only had about 5 days to make it work before Opening Day in the new building. Since then, equipment, functionality, and — you guessed it — more cable has been added to the mess of audio and video cable running along the wall.

About 6 weeks ago, we began plans for a project to clean this all up.  The big goals were…

• Move our audio operations to the adjacent room, essentially creating the ability to have a broadcast mix.
• Move a majority of our production gear to a rack, and relocate the cabling in order to clean things up.

The big deal of this project was the time window — The control room simply must be operational on the weekends, so we had a limited amount of time to do what we needed to do without putting the service in jeopardy.  We ended up splitting the project across 2 weekends:  The first week would be dedicated to audio work (the smaller half of the project), and the more intense video half of the project would take place on the second week.

During "Audio Week" (June 16-20), we also did some preliminary stuff for "Video Week" (June 23-27) — this included tracking down and labeling every incoming and outgoing video/audio line, as well as installing a new Mid-Atlantic ERK Series rack for the video gear to move into.  Quite a few of our video lines came up to the control room through the floor (the control room is on a mezzanine, so installing the rack also meant cutting some holes in the floor with a core drill so that wires could be pulled back into the floor and up through the new holes to the rack.  The concrete floor of the control room sits over top of a drop ceiling, so this would be easy in theory.

The first step was to prep the floor area where the rack was going and cut the holes that the wire would pass through.  God had a hand in this project from start to finish, and it became evident on the morning I was headed towards NPCC with the core drill that I just rented.  Dave H. — one of our control room operators, was already on the scene and ready to help out with the drilling.  Not only that, but Dave H. has extensive experience with concrete, and this was HUGE because I had absolutely none of that, much less any experience operating a core drill.  The morning would not have gone nearly as smooth had Dave not been there.

After the holes were cut and the rack was secured to the floor, we ran some 22/2 cable between the control room desk, and rack, and the new "broadcast mix room."  Once we did some soldering and verified that all our connections were in good shape, we took a break for the weekend.

The video rebuild was a bit more involved.  There were quite a few lines to move from their old locations, and the goal was to get everything functional by Thursday.  The first step involved gutting the entire control area and moving all our video gear to temporary storage.

Once all the cables were disconnected, they were pulled down through their conduits to the drop ceiling area below…

After all that, we began the process of moving gear and wire into the new rack.

We did run into a couple small snags with some over our CAT5 that is used to send XGA signals to the stage, but no show-stopping problems.  And again, God’s hand was very evident this entire week.  Volunteers made themselves available to help with wire pulling and cleanup and just about everything worked the first time around.  On a side note, I claimed a huge personal victory this week by going up 30 or so feet in a scissor lift to install a lipstick camera on the auditorium ceiling without freaking out (I battle acrophobia and have a tendency to "lock up" at around 15-20 feet).

We ended up finishing the project Thursday night around 6:30pm.

There’s still a little bit of touch-up work to do for aesthetics and friendliness, but functionally everything checks out good and signal is flowing everywhere it needs to be.

Some thoughts from this project:

• I’m getting better at detailed planning.  I’ve always been good at execution, but often times I would jump into something without nailing down the details, and it would result in a lot of problems.  I feel real good about how this project panned out.
• The volunteers that helped in this project were invaluable and absolutely crucial to its completion.  Thank you guys!
• The HD-SDI signal format is my friend.  One 75-Ohm cable for 1080i is sweet.
• Next time we do a big wiring project, we need to rock up some cable trays.
• Zip ties are so out.
• Velcro is so in.
• I used to be able to bust out all sorts of productivity by going all night — not the case anymore.  I really had to pace myself on this project and quit when I was tired.

Looking forward to the next project!

(Want more pictures?  They can be found here.)

I’ve been thinking a bit about multi-campus issues lately.  We’re not there yet, but I think sooner or later we will be, and I want to be ready to roll when it happens.  Being a dude that has no formal IT training and has only managed a single-site network, I’m both excited and a little anxious about the prospect of taking our IT and video infrastructure outside the walls of what would eventually be the "Central Campus."

Last week I had a chance to talk with Jared B. from NewSpring Church and get my mind around what it will take to stream live buffered video from one campus to another.  The tools that they’re using are amazing, and seem to be just the fit for what I like to call "North Point-style video" (IMAG on the sides with a HD image of the communicator in the center). The idea of being able to get our services streamed to another location is insanely cool to me, and next week, I’ll be calling around to see just how big of a data pipe I can bring in to NewPointe, as tihs project will require a HEAP of bandwidth for both locations.  (One of our contractors told us there’s fiber possibly running along our road already, how sweet would it be to ride that.)

Another item to consider will be our network.  When it was set up in our current building, I wasn’t even thinking about multi-site ventures.  As a result, the network isn’t in prime condition to be bridged off just yet.  I want to keep our domain intact across our organization, so eventually I’ll need to convert everything from 10.0.0.0/8 to 10.0.0.0/16 and then begin testing site-to-site stuff.  I really don’t want to spend on on new firewall hardware unless it’s absolutely necessary, so I’m gonna be banking on Brutus (our pfSense box) to make it happen.

Like I said, this stuff has me a little anxious, but VERY psyched at the same time.  We don’t have a set date or anything like that for acquiring a second campus, but if I can get some preparations done before that all hits, I’ll feel real good.

The Operations-to-Documentation ratio in IT and video, at least for us, is very high.  This bothers me.  I know I’ve got on my soapbox about it before, and then I always have this renewed resolve to go and make it happen.  Soon after, some new idea pops into my head for reconfiguring this or implementing that and BOOM…documentation is now out the window for the moment (again).

Now I know that as the person responsible for leading IT at the church, I should be looking out for new and better ways of doing things.  At the same time, the pattern I see myself getting into bothers me … especially in the wake of these past 2 weeks with all the health junk I had to wade through.  What if that was more serious?  What if I missed 3 months of work, rather than 4 days?

After thinking through some of this and talking it over with my boss, I decided that after finishing up a couple lingering projects, I am going to shift my focus temporarily and spend a high percentage of my time focused solely on documenting and training for the systems and processes we already have in place.  Of course, routine things like the helpdesk and backups will still happen, but I look to set aside maybe 1.5 days for that.  As for the length of this venture, I’m going to make the push for probably 2 to 3 months.

This definitely wasn’t how I planned to approach documentation at the start of the year, but more and more I’m seeing that we’ve just got to catch up with it.  While I know that we’re not going to just "arrive" in this area ever, I think this will be a big help in getting us where we need to be.

Over the past couple weeks, we’ve had intermittent (my favorite) issues with one of our cameras — the signal would start getting jumpy and eventually go back to normal after a few seconds.  It looked just like someone had unplugged the genlock connection and put it right back.

I finally had a chance to take a look at it today, and I was able to get a nice glimpse of the problem with my camera:

See how the copper in the center of the left connector is flared out?  That’s the camera’s genlock connection, and most-likely where our issue is.

I found a small needle-nosed pliers and closed the copper up as best I could… it’s looking a little better now.

I thought about getting in there with a small soldering tip and dripping some solder on that copper to brace it a little.  Going in there makes me a little nervous though.  See that white at the back of the connector?  That’s plastic.  I don’t really want to risk melting it, or worse, accidentally shorting the connector out.

So far, the camera is passing all the stress-tests (or different wire-wiggling techniques) that I’m subjecting it to, so I’m feeling good about this being the cause of our signal issues.  We’ll see in a couple weeks…

Our video control room has been pretty change-free for the past few months … the only thing new that we’ve started doing lately is playing DVDs back to our center screen through Pro Presenter.

Lately I’ve been spending some time thinking about what we can do to make things a little smoother in the control room.  It was originally set up in quite a hurry, and although we’ve gotten used to the way it was set up, it doesn’t mean that it can’t be improved.

So after some consideration and rethinking about how to make life easy upstairs, we’re going to do three things…

1. Consolidate
2. Rearrange
3. Spread the love knowledge

Consolidating is going to be the easiest part of what we’re doing.  Since our installation, we’ve used two separate computers to handle our song lyrics and message notes using Pro Presenter and Keynote, respectively.  The more we’ve used Pro Presenter (thank you, Renewed Vision, for version 3) the more we like it, and so now we’re looking to move to a single machine running Pro Presenter for both lyrics and message notes.  Sure we would lose the cute animations and whatnot, but we’ve used them rarely at best.  Why would we do this?  Well, it’s much easier to get familiar with a single software program than it is to get familiar with two, and since we’re already using Pro Presenter around the building it makes sense to standardize on it.  We’ll still keep Keynote around, but it won’t be a primary application anymore.  At the most, we’ll use it to design slides so we can export them to PNG for Pro Presenter.

The next task will be rearranging the control room.  For those who don’t pay attention to our IMAG work (and you SHOULD, because the team does a great job at it), we’ve gotten rid of the remote camera that was mounted on the back wall, thus we now have some extra real estate on the control room desk.  Throw in the fact that we’re consolidating 2 workstations down to 1, and suddenly we’ve got room to make some changes in the layout of our equipment.  The rearrangement isn’t going to be anything monumental, but it’s going to make it easier on the computer operators, because they’ll be able to see the results of their work better.

Finally, spreading the knowledge, which is really just a nice euphemism for — you guessed it — documentation.  My goal in this area is to have a binder for each workstation in the control room that contains an outline of the machine’s vital information, software documentation, area-specific troubleshooting methods, and anything else that’s needed.  This process will take the longest by far, but the benefits will be huge for new volunteers once everything is in place.

Are there any readers to this blog that spend time in video?  I’d love to hear about what you’re doing to keep things efficient/clean/<insert positive adjective here> in your work area.

When we installed our video system last year, we admittedly didn’t think of everything we should have.  As a result, we were left with a large gap in our functionality.  This gap was basically the ability to run Keynote presentations into our HDSDI system smoothly.  Now we were able to do this, but not as well as it needed to be done.  See, we were using an HD interface from Blackmagic Design, which was great, except it wouldn’t allow OS X to use Core Image to send graphics across the HDSDI channel.  As a result, we had a NASTY 4-5 second lag every time we changed slides.  Worse yet, we had no ability to to any smooth transitions or builds of any kind.

Fast-forward 9 months to this past September.  Things have calmed down, and it’s now time to try and resolve this issue without killing our budget.  Basically the solution is to convert the DVI signal from our Mac’s dual-head card to HDSDI, and add sync to it so that we can run it through our live switcher.  I’ve found only 2 products so far that can do this:  The Miranda DVI-RAMP, which we tested 2 weekends ago, and the Matrox MXO, which we tested this weekend.

Both systems did a great job of converting DVI to HDSDI.  The DVI-RAMP has some features on it that we don’t really need at this time, but everything it advertises it performs very well.  The Matrox isn’t as loaded up as the DVI-RAMP, but it also costs about 80% less and still is enough to do what we need.  Our Keynote operator was blown away by how responsive her computer was this morning during the first test.  To add to that, this morning’s message included a LOT of graphic elements that changed much quicker than normal.  It’s safe to say that having this component in place actually improved the morning from a presentation standpoint.

I’m hoping that the evaluation of the MXO comes back positive so that we might be able to hang on to the one we have.  Currently we have it strictly on a trial basis. 

There’s a few other places we could use equipment like this in the control room…

• On CG:  We could finally upgrade to the newest version of Pro Presenter and use their Extended Resolution plugin to output our lyrics over DVI at 1920×1080.
• On Playback:  We could quit using the Blackmagic Deck Control to play back our videos and again, use Pro Presenter with the ER plugin to give us one-click playback of our clips.
• Combined with some BTX/Gefen hardware:  We could put just about any signal we wanted to across our HDSDI framework and get it to wherever it needs to go.

So as you can imagine, I’m pretty stoked.  Being able to add a lot of functionality for a relatively low amount of cash is great, but when you can make life easier for your volunteers at the same time, it’s REALLY a win.  I’ll come back to this subject later as our evaluation continues.

One of the big issues we have in the control room right now is this:  How to scale DVI video from our Keynote and Pro Presenter systems to HD/SDI, add sync to it in the process, and not break the budget?

Why is this a hot issue right now?

Pro Presenter:  Our CG computer is using a Blackmagic DeckLink HD to genlock the system’s output and send it to our switcher in HD/SDI.  I would LOVE to move to version 3 of Pro Presenter (we’re using it in other locations and it rocks), but because there is no way to get it to utilize the DeckLink’s HD/SDI output, I’m stuck using version 2, which, while it is still nice, is miles behind version 3 in terms of functionality and ease of use.

Keynote:  We use Keynote to put graphics (announcements, message notes) up on the screen, keyed over live video by our switcher.  We’re using a Blackmagic product here as well (a Multibridge Pro) to output HD/SDI from the computer, sync it, and feed it to the switcher.  The problem here is that since the Multibridge doesn’t support Core Image, Keynote has to be set up to do software rendering…. at 1920×1080.  This leads to very slow slide changes and makes it impossible to do any sort of smooth transitions or motion graphics on the slide.

Note that this is NOT a knock on Blackmagic Design products.  They are what they are, and they do exactly what they’re designed to do (and they do it well).

I’m looking to bump things up a level in terms of functionality and user-friendliness.  I’ve found 2 items so far that can make this work, and there’s a good chance we’ll get to trial both of them:

Miranda DVI-RAMP 2:  I’ve had my eye on the DVI-RAMP 2 for some time now, and we will actually have a demo of it showing up next Thursday.  The unit is capable of scaling and converting from DVI to HD/SDI.  The unit can also sync to a reference signal and act as a frame sync at the same time.  As an added note, the DVI-RAMP 2 is platform-independent.  Mac, PC, whatever; as long as it’s DVI, you’re good to go.

Matrox MXO: I just stumbled onto this piece of gear a few days ago.  The MXO has got the same basic principle as the Miranda unit, except that it will only work on Macs (which is fine for us).  The one thing that catches my eye is the fact that the MXO is about 22% of the cost of the DVI-RAMP 2.  Does that mean it’s only 22% as good?

Any video people reading this that have gone down this road before?  I’d love to hear any thoughts you have on this matter and what you’ve done to get around it.

I’ll post more on this as things pan out.

It’s been awhile since I talked about this, as things have been moving a little slow.  However, over the past week we have made the final steps in getting out video data protected in a manner that helps me sleep better at night.

About a month ago, we installed a PC and drive chassis in our editing room.  This system has 2 2.8TB RAID5 arrays (with 2 additional hot spares drives) that will do nothing but hold video data, SFX, production music, and final cut project files.  We’ve been slowly moving old projects and new finished work onto these arrays over the past couple of weeks.

Yesterday morning, I installed a new gigabit switch in our IDF that serves the editing and control rooms.  This switch also links back to our MDF, and so we now have a much faster link back to our servers.

Finally, earlier this evening, I was able to get our editing storage server talking to our Galaxy Express backup server using some VLAN voodoo and an extra network card.  Our network is mixed 100/1000, so we opted to do our backing up on a separate network.  Since our video server is the first server NOT to be in the rack, I opened up a new VLAN and routed it back to the switch in the server rack that connects the “backup network.”

Note to self:  Time to update the Visio charts of our network layout.  Yippee.

Seriously, I don’t know why I ever considered NOT buying managed switches.  Sure I would save money, but I would be at a serious disadvantage when it came time to do stuff like this.

My adventures with the Norco disk system have continued throughout the day, and after some more tinkering and some software-aided intervention, our storage array for the editor seems to be back up and running.

Apple’s Disk Utility program failed to do any sort of repair work on the array.  After many repair attempts, all I could get were directory errors.  After browsing around for some Mac disk repair utilities, I landed on one that showed promise:  DiskWarriorby Alsoft.

I fired up DiskWarrior, pointed it to our now-unmounted disk array, and watched it go to work.  It identified the array and cleaned up the directory structure, various file attributes, and various other things, and in about 5 minutes the array was back online.  SWEET!  This was WELL worth the price tag (about $80).

Thinking things were back to normal, I went ahead and shut everything down so I could install a UPS in front of the Mac and the resurrected RAID box.  After plugging everything in, I turned on the Norco drives, powered the Mac up, and watched in utter astonishment….as the system failed to recognize five of the twelve installed drives.

At this point I was about at my limit with this RAID array.  Not really knowing what else to do, I went ahead and shut both the disks and the Mac completely down, and then restarted the RAID after about a minute.  I let the disk system run for about 2 minutes before powering up the Mac.  My thought on this was that if the backplane or anything else in the RAID box has to run a POST or anything, I’m going to give it plenty of time to do so before restarting the computer.

I pushed the power button on the Mac, and lo and behold, I heard many drives starting to spin up all at one time.  In a few seconds, all 12 drives had spun up, and the RAID seemed to be back on its feet.  A quick look through the Finder revealed that all of our stuff on the array appeared to be intact.

So is this over?  I really don’t know.  I really haven’t felt like testing to see if I can replicate the issue by starting the Mac “too soon” after the RAID array is powered up.  What I am going to do is exchange our eSATA controller card for one that has been tested and is more compatible with the unit.  If that clears things up, then I’ll feel more confident about marking this down as  a hardware issue.  At this point, it makes sense that 2 different chipsets wouldn’t play well together.  But at the same time, you’d think there would be more consistency to it.

Maybe the constant here is just sheer unreliability.  Time will tell.  Until then, I’ll be copying our FCP project files onto a safer hard drive.

Well, after playing around with the RAID array throughout the wee hours of the morning, it’s pretty apparent that something went seriously wrong.  A massive power flux?  A dicey hard drive?  I really don’t know at this point.  S.M.A.R.T. status on all the drives shows that they’re running just fine.  So far 2 ideas are floating around in my head:

1. The system suffered a massive power fluctuation that totally ticked off the Mac, or the RAID unit, or both.
2. There is a major compatibility problem with the RAID unit and the HighPoint Technology card that I had to use in place of the bundled controller card.  The only thing I can think of is that there might be 2 different chipsets between the RAID unit and the controller that don’t like each other at all.

Either way, I’m glad this problem decided to rear its head NOW instead of later, when we’ve got the drive populated with irreplaceable data.

Speaking of which, it just so happens that most of the files that were on that RAID5 array are still sitting in other areas!  THAT is letting me breathe so much easier right now.  However, there were quite a few Final Cut project files that were only on that array, which is still a bummer.  I’m in the process of looking through data recovery software to see if there’s anything decent that I can try.

In the meantime, I’ve got a UPS set hook into that system immediately.  Plus, if I can’t make the array work after another rebuild, I’m doing to set a separate PC up there and connect it to the MacPro.  Since the PC has a PCI slot on it, I can use the Norco’s bundled controller card.  I’m not exactly thrilled about putting a PC up there JUST to act as a bridge between the editing system and the RAID, but I may find that I have no choice.

More updates as the plot unfolds.