When our control room for live video was installed, it was installed in a hurry. Now, that’s not a finger-point at anyone that was involved, we really only had about 5 days to make it work before Opening Day in the new building. Since then, equipment, functionality, and — you guessed it — more cable has been added to the mess of audio and video cable running along the wall.

About 6 weeks ago, we began plans for a project to clean this all up.  The big goals were…

• Move our audio operations to the adjacent room, essentially creating the ability to have a broadcast mix.
• Move a majority of our production gear to a rack, and relocate the cabling in order to clean things up.

The big deal of this project was the time window — The control room simply must be operational on the weekends, so we had a limited amount of time to do what we needed to do without putting the service in jeopardy.  We ended up splitting the project across 2 weekends:  The first week would be dedicated to audio work (the smaller half of the project), and the more intense video half of the project would take place on the second week.

During "Audio Week" (June 16-20), we also did some preliminary stuff for "Video Week" (June 23-27) — this included tracking down and labeling every incoming and outgoing video/audio line, as well as installing a new Mid-Atlantic ERK Series rack for the video gear to move into.  Quite a few of our video lines came up to the control room through the floor (the control room is on a mezzanine, so installing the rack also meant cutting some holes in the floor with a core drill so that wires could be pulled back into the floor and up through the new holes to the rack.  The concrete floor of the control room sits over top of a drop ceiling, so this would be easy in theory.

The first step was to prep the floor area where the rack was going and cut the holes that the wire would pass through.  God had a hand in this project from start to finish, and it became evident on the morning I was headed towards NPCC with the core drill that I just rented.  Dave H. — one of our control room operators, was already on the scene and ready to help out with the drilling.  Not only that, but Dave H. has extensive experience with concrete, and this was HUGE because I had absolutely none of that, much less any experience operating a core drill.  The morning would not have gone nearly as smooth had Dave not been there.

After the holes were cut and the rack was secured to the floor, we ran some 22/2 cable between the control room desk, and rack, and the new "broadcast mix room."  Once we did some soldering and verified that all our connections were in good shape, we took a break for the weekend.

The video rebuild was a bit more involved.  There were quite a few lines to move from their old locations, and the goal was to get everything functional by Thursday.  The first step involved gutting the entire control area and moving all our video gear to temporary storage.

Once all the cables were disconnected, they were pulled down through their conduits to the drop ceiling area below…

After all that, we began the process of moving gear and wire into the new rack.

We did run into a couple small snags with some over our CAT5 that is used to send XGA signals to the stage, but no show-stopping problems.  And again, God’s hand was very evident this entire week.  Volunteers made themselves available to help with wire pulling and cleanup and just about everything worked the first time around.  On a side note, I claimed a huge personal victory this week by going up 30 or so feet in a scissor lift to install a lipstick camera on the auditorium ceiling without freaking out (I battle acrophobia and have a tendency to "lock up" at around 15-20 feet).

We ended up finishing the project Thursday night around 6:30pm.

There’s still a little bit of touch-up work to do for aesthetics and friendliness, but functionally everything checks out good and signal is flowing everywhere it needs to be.

Some thoughts from this project:

• I’m getting better at detailed planning.  I’ve always been good at execution, but often times I would jump into something without nailing down the details, and it would result in a lot of problems.  I feel real good about how this project panned out.
• The volunteers that helped in this project were invaluable and absolutely crucial to its completion.  Thank you guys!
• The HD-SDI signal format is my friend.  One 75-Ohm cable for 1080i is sweet.
• Next time we do a big wiring project, we need to rock up some cable trays.
• Zip ties are so out.
• Velcro is so in.
• I used to be able to bust out all sorts of productivity by going all night — not the case anymore.  I really had to pace myself on this project and quit when I was tired.

Looking forward to the next project!

(Want more pictures?  They can be found here.)

How nifty is that?

The story starts last Wednesday when Brandon (one of our music/video interns) and I went to Big Jim’s for lunch.  For reasons unknown, I took my car keys out of my pocket and put them in the cup holder of his car.  We had lunch, we came back to work, all was well … except that I forgot that my keys were in Brandon’s car. 

I had forgotten all about my keys until after band rehearsal which was around 12:30am.  Brandon had already left, which didn’t matter because I had long forgotten that my keys were still in his car.  After searching everywhere I could think of looking (I had only been in a couple areas all day), I gave up hope of finding my keys.  It was 2:30am, and I was exhausted.  I found my way to a room with a comfy couch and crashed there for the night, hoping to get some decent rest.

I woke up about 6 hours later (I actually had a shower and a change of clothes), and for no apparent reason, updated my Facebook status something to the effect of "Dave has lost his car keys."  About 15 minutes later, Brandon (who is way more into Facebook than I am) showed up at my desk with a grin, plopped my car keys down on my desk and said "Did you lose these?  I saw your Facebook status and remembered the keys in my cup holder."

So there you have it — I experienced a tangible benefit from being in a social network.  I wish I could say that it wouldn’t be the last time, but the jury’s still out on that.

It’s coming up on 3am, and I’m just now starting to wind down a little from tonight’s work night. Although my project for this evening didn’t exactly turn out as I planned, it capped off a pretty productive day.  The highlight of the day has been our IT operations meeting.  We had a good bit of discussion today on multi-site and what we’ll need to implement to be ready.  Some good discussion…

• Network reconfiguration (addressing and masking needs tweaked - already on my list)
• Data link for streaming and network access
• Who provides it?
• What if what we need is not available? (we’re rural, so that’s possible)  Do we call the power company?  Pull fiber in? Load-balance multiple instances of available connections?
• Phone system changeover (we will be dropping our PBX and going VoIP)
• Increasing systems availability (just because the power goes out here doesn’t mean it’s out everywhere)
• Centralized storage - it sure would be nice, but what does it look like for us?

Another big item will be getting our CMS (F1) ready for this transition.  There’s a lot of background stuff that needs to happen ahead-of-time for this one.

I know even this is just the tip of the iceberg when it comes to the big picture … but are there any IT-related items that I’m missing on this?

I’ve been thinking a bit about multi-campus issues lately.  We’re not there yet, but I think sooner or later we will be, and I want to be ready to roll when it happens.  Being a dude that has no formal IT training and has only managed a single-site network, I’m both excited and a little anxious about the prospect of taking our IT and video infrastructure outside the walls of what would eventually be the "Central Campus."

Last week I had a chance to talk with Jared B. from NewSpring Church and get my mind around what it will take to stream live buffered video from one campus to another.  The tools that they’re using are amazing, and seem to be just the fit for what I like to call "North Point-style video" (IMAG on the sides with a HD image of the communicator in the center). The idea of being able to get our services streamed to another location is insanely cool to me, and next week, I’ll be calling around to see just how big of a data pipe I can bring in to NewPointe, as tihs project will require a HEAP of bandwidth for both locations.  (One of our contractors told us there’s fiber possibly running along our road already, how sweet would it be to ride that.)

Another item to consider will be our network.  When it was set up in our current building, I wasn’t even thinking about multi-site ventures.  As a result, the network isn’t in prime condition to be bridged off just yet.  I want to keep our domain intact across our organization, so eventually I’ll need to convert everything from 10.0.0.0/8 to 10.0.0.0/16 and then begin testing site-to-site stuff.  I really don’t want to spend on on new firewall hardware unless it’s absolutely necessary, so I’m gonna be banking on Brutus (our pfSense box) to make it happen.

Like I said, this stuff has me a little anxious, but VERY psyched at the same time.  We don’t have a set date or anything like that for acquiring a second campus, but if I can get some preparations done before that all hits, I’ll feel real good.

For me, blogging is a bit like exercising:  If I stop doing it, I eventually find it very hard to get back in the groove.  There’s been a lot going on lately at NewPointe, and I’ve had a tough time breaking away to write anything even resembling a blog post.

I’m going to make an effort to get back into posting.  There’s a lot of cool stuff going on at NP that’s worth talking about, so bear with me as I try to get my blog on again.

Local admin privileges — to reinstate or keep locked away?

This has been on my mind a lot lately, mainly because I’ve been thinking about ways to better serve our users at NPCC and make things easier for them.  I know the local admin discussion is not a new one by any stretch, and you can also approach it from either side of the fence.  It really comes down to what you determine to be "acceptable risk."

This is what I’m kicking around in my head right now — what if I made each user a local admin for their respective machine?  Currently, only laptop users and one, maybe two desktop users have local admin rights to their machine. 

The advantages of giving back local admin?
1. Users can install software without having to ask me or wait on me to arrive on the scene.
2. Users can update programs on their own (Yeah I’m talking about you, iTunes) without my approval.
3. Those flash drives that require an extra piece of software before they mount (which I hate) can be used without my being on the scene.  In addition, the end-user can be walked through the process of reassigning drive letters if their flash drive somehow manages to interfere with our standard drive mappings. (boo)
4. Users can install fonts on their own without having to ask me.
5. If a user is done with a program and no longer needs it, they can uninstall it on their own without my help.

Now, the disadvantages and dangers of putting local admin back in the users’ hands?
1. Users can install software without having to ask, regardless of whether this software is legit or not.
2. To guard against the above, I will need to implement a monitoring solution that tracks software installation.  Spiceworks might be a good place to start with that.
3. I’m going to have to create a list of software that it supported by NewPointe IT; I don’t have the resources to support every piece of software that gets installed on a machine.  What happens then, when a user installed "unsupported" software and it wrecks their system?  That will need to be spelled out as well.
4. Any process that runs while the user is logged in will run with local admin privs.  Again, machine monitoring and logging will be a must.

I haven’t made a decision on this yet, but I’m very interested to hear anyone’s argument for or against users running with local admin.  What are you doing in your organization, and what factors led you to that decision?

Here’s how you know your week is going to contain 50% less sleep … one of your co-workers walks up to you and asks "Hey, did you know there’s a loud beeping noise coming from your server room?"

I should have gone home right then and gotten my jammies and pillow to prepare for the week.

As it turns out, the beeping was exactly what I thought it would be.  Another hard drive had bitten the dust in our would-be file server, PowerEdge 1400 that up to this point had been rock-solid for us.  With dual 1GHz P3 CPUs and 2GB of RAM, it would have made a great file server.  WOULD HAVE … except for that it had somehow managed to eat 3 hard drives before I could even put it into production.  However, this was the last straw.  3 dead hard drives in 2 months is enough to convince me that I don’t want this machine in the lineup anymore.

The plan this week was supposed to be simple.  Copy our file server and EMS Lite data to the PE1400 and bring it online.  After that, install a second array of disks in the PE1800 (where the file server once was), install Ubuntu on it, and begin using it as our VM host.  Why Ubuntu?  Because I’m budget-tight at the moment, and also because the current install of Win 2k3 Standard wasn’t utilizing all 8GB of RAM as well as the 64-bit CPUs that the 1800 now has.

This plan seemed pretty airtight, except I didn’t plan on losing server hardware just before making this transition.  However, the migration needed to proceed, and so I loaded a working RAID5 array (controller and drives) into a newer desktop box, threw Server 2k3 on it, and began the Robocopying all over again.  By now it’s Tuesday, and tonight I’m scheduled to take all the servers down and transition our PE1800 over to Ubuntu so it can be a big bad 64-bit VM host.  However, in the midst of copying file server data, I forgot about EMS Lite.

EMS our current calendaring software, and the only SQL (MSDE) database we have on-site that gets any end-user interaction.  It figures that this tiny-but-critical program would hold things up for about 24 hours while I learn how to successfully migrate the database from one instance to another without breaking things.  HUGE thanks to Jeremy Marx for taking time out of his day to help me through this.  (That’s the power of the CITRT community!)

That 24-hour period was not wasted though … during that time I did some test runs with Ubuntu 64-bit and also got our new file server straightened out.  By 3:30pm Wednesday (yes, it’s Wednesday now) I had been up for about 30 hours, but I was very please just to have conquered the EMS data issue.  I fell asleep around 4pm Wednesday and didn’t wake up until about 8am Thursday morning.

It’s now Thursday night (almost Friday morning) now, and I’m on the last leg of this transition.  All of our VMs are being copied over to another server, and once that’s done, I’ll take the old array out of our PE1800 and install a new array.  That new array will have Kubuntu 8.04 on it, and will server as our new 64-bit VM host.

I’m already starting to wear down a bit (I wouldn’t ever make it as a Bering Sea crabber), but I’m pumped to see this project finally coming to a close.  It took longer than I thought and it cost a few hours of sleep, but I feel like the benefit will be worth the trouble.

The Operations-to-Documentation ratio in IT and video, at least for us, is very high.  This bothers me.  I know I’ve got on my soapbox about it before, and then I always have this renewed resolve to go and make it happen.  Soon after, some new idea pops into my head for reconfiguring this or implementing that and BOOM…documentation is now out the window for the moment (again).

Now I know that as the person responsible for leading IT at the church, I should be looking out for new and better ways of doing things.  At the same time, the pattern I see myself getting into bothers me … especially in the wake of these past 2 weeks with all the health junk I had to wade through.  What if that was more serious?  What if I missed 3 months of work, rather than 4 days?

After thinking through some of this and talking it over with my boss, I decided that after finishing up a couple lingering projects, I am going to shift my focus temporarily and spend a high percentage of my time focused solely on documenting and training for the systems and processes we already have in place.  Of course, routine things like the helpdesk and backups will still happen, but I look to set aside maybe 1.5 days for that.  As for the length of this venture, I’m going to make the push for probably 2 to 3 months.

This definitely wasn’t how I planned to approach documentation at the start of the year, but more and more I’m seeing that we’ve just got to catch up with it.  While I know that we’re not going to just "arrive" in this area ever, I think this will be a big help in getting us where we need to be.

• At the jewler’s picking up Jess’s ring. #
• Back from the Jewler’s, and feeling weird again. If I’m getting sick again I’m gonna be unhappy. #
• Feels like an unbaked breadstick is stuck between my ears, which mean’s last week’s sinus issue is still present. Time for a Doc appt. #
• Managed to get an appointment in at my doctor today….before lunch, even. How cool is that. #
• Got my prescription from the Doc and I should be good to go. Finally time for lunch. #
• @ebuford - But if you *do* mess it up, @wantmoore and I are available for more work. #

Powered by Twitter Tools.

• I finally feel like I might be over the fever/stuffiness/dizziness that’s plagued me for the past week. #
• Been working on video documentation all day … getting ready to meet with a volunteer after lunch. #
• Great meeting with "other Dave" … it’s great when a volunteer tells you that they’ve found their niche and they love what they’re doing. #

Powered by Twitter Tools.