After some technical difficulties and some long-awaited changes, NewPointe’s podcast is back in good order and ready for your subscription.

You can subscribe to the podcast by clicking below.

Over the past couple weeks, we’ve had intermittent (my favorite) issues with one of our cameras — the signal would start getting jumpy and eventually go back to normal after a few seconds.  It looked just like someone had unplugged the genlock connection and put it right back.

I finally had a chance to take a look at it today, and I was able to get a nice glimpse of the problem with my camera:

See how the copper in the center of the left connector is flared out?  That’s the camera’s genlock connection, and most-likely where our issue is.

I found a small needle-nosed pliers and closed the copper up as best I could… it’s looking a little better now.

I thought about getting in there with a small soldering tip and dripping some solder on that copper to brace it a little.  Going in there makes me a little nervous though.  See that white at the back of the connector?  That’s plastic.  I don’t really want to risk melting it, or worse, accidentally shorting the connector out.

So far, the camera is passing all the stress-tests (or different wire-wiggling techniques) that I’m subjecting it to, so I’m feeling good about this being the cause of our signal issues.  We’ll see in a couple weeks…

Well, it would seem that our internet connection seems to be back in order, although I’m not too thrilled with the manner in which it was ultimately fixed.

Here’s a rundown of what’s been going on:

Friday PM-Sunday PM - Our internet connection starts dropping out for about a minute every 33 minutes, almost to the second.  In addition, outbound browsing is very slow and showing symptoms of packet loss.  I open a ticket with TW tech support, but they claim that my situation can’t be monitored until Monday morning.  I guess 24/7/365 support doesn’t apply to Tier 3.

Sunday PM-Monday AM - Our cable modem stops passing traffic completely.  It still syncs up, but I’m unable to ping any external addresses with it.  Another call to tech support ensues, and the tech finds that something may be wrong with whatever device that provisions are static IP addresses.  Apparently cable modems that use DHCP are still working though, because I — well, let’s just say I found a way to get around this issue for a short time.

Monday AM-PM - We get the pleasure of becoming part of a widespread outage.  No modem sync — at all.  "Everyone who’s got a pulse is working on this one," the tech tells me.  Well that’s good to know.

Monday PM - Wednesday - I finally see sync lights on our cable modem after popping my head in the datacenter for about the 15th time.  Our internet is back, and it’s no longer dropping out every 30 minutes, but it’s still very slow…almost like DNS isn’t resolving properly (the answer is always DNS).  Connections are fast ONCE their made, but it takes forever to get that far.

Wednesday - Slightly frustrated, I call Time-Warner again to see if they can look at the issue and stress-test our modem.  The stress test doesn’t show any issues of packet loss.  After looking at a few other things, this dialog ensues…

Tech:  "What DNS server are you using?"
Me:  "We’re using OpenDNS’s servers" (I rattle off the IP addresses)
Tech:  <paraphrase>"Well, I’m looking at a bulletin sent out by our network team in response to an issue that’s been going on for about 5 days. (hey that sounds like us!)  They’re recommending that anyone seeing speed issues make sure they’re using our DNS servers.
Me:  "Well that’s great, but we use OpenDNS for our content filtering, plus we’ve been using them for a year now with no issues."
Tech:  "Still, try changing your settings and see if that doesn’t help your speed issue." </paraphrase>

So I logged into our DNS servers and changed the forwarder addresses to match Time-Warner’s DNS servers.  BOOM — instant speed increase.  It would seem apparent that using TW’s DNS instead of OpenDNS fixed the issue, but now I’m left with just as many questions as answers.  What’s up with OpenDNS?  Do they have issues of their own?  Is TW doing something on their end to interfere?  If not, did a simple DNS query to a non-TW DNS server take forever?

Needless to say, I’m not too pleased with this situation.  If this is going to be a permanent issue, I now have to find another way to do content filtering.  I will most-likely be giving ScrubIT a try now to see if there are any similar issues.  We’ll see how that goes.

What started out Friday as a every-33-minute interruption has now become a drop-dead outage as of about 5:45pm today (my FTL drive didn’t make the jump in time, Jason).  In an attempt to not go on-site, I’m currently on the phone with Time Warner tech support for the 3rd time in 36 hours.  On a Monday morning, a 1-minute drop every 30 minutes would be quite an inconvenience, but it would be doable, but again we’re an F1 church, and F1 Contributions requires an internet connection to work.

The rep that I talked to was able to reset the modem remotely, but unfortunately no data is passing through.  So, I’m about to go on-site to power-cycle the modem and follow up on the ticket if any further work is needed.  I’m hoping like crazy that this isn’t going to be an issue that bites us on Monday morning.

It doesn’t matter how many situations and problems you’ve dealt with as an IT guy, something new and different will eventually come along to make you scratch your head.

On Friday morning I got a notice that our internet connection had dropped out for a short period of time.  After just a minute of being down, the connection came back up.  No big deal, right?

Throughout Saturday morning, I’ve received another notice stating that our connection was down, and then up again.  This happened twice on Saturday morning.  Slightly concerned, I downloaded a little utility (Uptime Scout) to constantly ping our network and log any timeouts that occurred.  At the same time, I called Time Warner Tech Support and opened a ticket with them so they at least had it on record.

When I returned later in the evening, what I found in the logs was very interesting.  Our internet connection was dropping out every 33 minutes, almost to the second.  The outage didn’t last long … a minute at the very most.  However, we use F1, and if this happens on Sunday morning things could get a little messy.

Now I’m very intrigued about our problem, and so I decide to go on-site to look at our cable modem and firewall.  I’m figuring that SURELY our modem is resetting itself or that there’s something that I’ll be able to see.  However, after sitting in the cold datacenter for 50 minutes and watching 2 more outages happen, it’s quite apparent that there’s nothing wrong with the cable modem.  It’s not flinching, and the traffic indicators are still flashing despite my inability to ping anything on the internet.

Well, that leaves you with 2 possible causes … faulty equipment directly outside our building, or a faulty network card in our pfSense box.  As unlikely as I think it is that our pfSense box would have a bad nic (I just replaced it), it’s definitely not outside the realm of possibility. 

So, for the last 40 minutes, my laptop has been connected directly to the cable modem in an attempt to rule out any faulty component on our network, and just as I started typing this paragraph, my pings began to time out and Uptime Scout started making noises at me.  From what I can see, whatever is puking every 33 minutes is not connected to our network or even in the building.

I called Time Warner back with this extra tidbit of information, and I must say that their tech support has been very nice to work with, even at 1:00am.

Our video control room has been pretty change-free for the past few months … the only thing new that we’ve started doing lately is playing DVDs back to our center screen through Pro Presenter.

Lately I’ve been spending some time thinking about what we can do to make things a little smoother in the control room.  It was originally set up in quite a hurry, and although we’ve gotten used to the way it was set up, it doesn’t mean that it can’t be improved.

So after some consideration and rethinking about how to make life easy upstairs, we’re going to do three things…

1. Consolidate
2. Rearrange
3. Spread the love knowledge

Consolidating is going to be the easiest part of what we’re doing.  Since our installation, we’ve used two separate computers to handle our song lyrics and message notes using Pro Presenter and Keynote, respectively.  The more we’ve used Pro Presenter (thank you, Renewed Vision, for version 3) the more we like it, and so now we’re looking to move to a single machine running Pro Presenter for both lyrics and message notes.  Sure we would lose the cute animations and whatnot, but we’ve used them rarely at best.  Why would we do this?  Well, it’s much easier to get familiar with a single software program than it is to get familiar with two, and since we’re already using Pro Presenter around the building it makes sense to standardize on it.  We’ll still keep Keynote around, but it won’t be a primary application anymore.  At the most, we’ll use it to design slides so we can export them to PNG for Pro Presenter.

The next task will be rearranging the control room.  For those who don’t pay attention to our IMAG work (and you SHOULD, because the team does a great job at it), we’ve gotten rid of the remote camera that was mounted on the back wall, thus we now have some extra real estate on the control room desk.  Throw in the fact that we’re consolidating 2 workstations down to 1, and suddenly we’ve got room to make some changes in the layout of our equipment.  The rearrangement isn’t going to be anything monumental, but it’s going to make it easier on the computer operators, because they’ll be able to see the results of their work better.

Finally, spreading the knowledge, which is really just a nice euphemism for — you guessed it — documentation.  My goal in this area is to have a binder for each workstation in the control room that contains an outline of the machine’s vital information, software documentation, area-specific troubleshooting methods, and anything else that’s needed.  This process will take the longest by far, but the benefits will be huge for new volunteers once everything is in place.

Are there any readers to this blog that spend time in video?  I’d love to hear about what you’re doing to keep things efficient/clean/<insert positive adjective here> in your work area.

We’re getting hammered with snow today, and I’m diggin’ it.

I don’t mind snow if it actually SNOWS … I mean a good nonstop 7-8 inches, and not the stuff that’s half rain either.

Well today we got that — so much so that our offices are closed and might even be closed tomorrow depending on what happens tonight.

Here’s some pictures I took while out today on a quest for some kerosene.

Earlier tonight, I made a blog post referencing how busy I’ve been and how it’s prevented me from blogging.  However, when I was typing it out, I made a small error that could have caused a lot of misunderstandings and phone calls…

Here’s what I should have typed…

Some of you might be wondering where I’ve been the past couple weeks.  I haven’t quit blogging and I haven’t quit my job…

Because of my oversight, here’s how the original sentence read, as published:

Some of you might be wondering where I’ve been the past couple weeks.  I haven’t quit blogging and I quit my job…

You can see where this might have caused some turning-of-heads, and a sudden spike in executive meetings at the NewPointe offices.

Fortunately, Jason Lee happened to be reading through his RSS feeds and noticed my error.  Thanks to the #citrt IRC channel, he was able to contact me about the mishap.  After a mild heart attack, the post was edited, and the offending post was replaced with the corrected version.

Two take-aways from this short story…

1.  Communities such as CITRT are invaluable. 
2.  Make sure you proofread your posts before posting … no matter how long you’ve been blogging.  A single misspelling or missing word can turn a simple post into a press release.

Some of you might be wondering where I’ve been the past couple weeks.  I haven’t quit blogging and I haven’t quit my job — things have just been pretty crazy.

There are a lot of changes underway in both video and IT, and over the past 2 weeks I’ve been in the middle of a lot of testing, communicating and coaching.  Blogging - even reading blogs - has been all but a foreign concept for the past few days … not that I want it that way.

My goal for the next few days is to bring things up to speed a little bit on what I’ve been doing and what I’ve got coming down the pipe.