Well, it would seem that our internet connection seems to be back in order, although I’m not too thrilled with the manner in which it was ultimately fixed.

Here’s a rundown of what’s been going on:

Friday PM-Sunday PM - Our internet connection starts dropping out for about a minute every 33 minutes, almost to the second.  In addition, outbound browsing is very slow and showing symptoms of packet loss.  I open a ticket with TW tech support, but they claim that my situation can’t be monitored until Monday morning.  I guess 24/7/365 support doesn’t apply to Tier 3.

Sunday PM-Monday AM - Our cable modem stops passing traffic completely.  It still syncs up, but I’m unable to ping any external addresses with it.  Another call to tech support ensues, and the tech finds that something may be wrong with whatever device that provisions are static IP addresses.  Apparently cable modems that use DHCP are still working though, because I — well, let’s just say I found a way to get around this issue for a short time.

Monday AM-PM - We get the pleasure of becoming part of a widespread outage.  No modem sync — at all.  "Everyone who’s got a pulse is working on this one," the tech tells me.  Well that’s good to know.

Monday PM - Wednesday - I finally see sync lights on our cable modem after popping my head in the datacenter for about the 15th time.  Our internet is back, and it’s no longer dropping out every 30 minutes, but it’s still very slow…almost like DNS isn’t resolving properly (the answer is always DNS).  Connections are fast ONCE their made, but it takes forever to get that far.

Wednesday - Slightly frustrated, I call Time-Warner again to see if they can look at the issue and stress-test our modem.  The stress test doesn’t show any issues of packet loss.  After looking at a few other things, this dialog ensues…

Tech:  "What DNS server are you using?"
Me:  "We’re using OpenDNS’s servers" (I rattle off the IP addresses)
Tech:  <paraphrase>"Well, I’m looking at a bulletin sent out by our network team in response to an issue that’s been going on for about 5 days. (hey that sounds like us!)  They’re recommending that anyone seeing speed issues make sure they’re using our DNS servers.
Me:  "Well that’s great, but we use OpenDNS for our content filtering, plus we’ve been using them for a year now with no issues."
Tech:  "Still, try changing your settings and see if that doesn’t help your speed issue." </paraphrase>

So I logged into our DNS servers and changed the forwarder addresses to match Time-Warner’s DNS servers.  BOOM — instant speed increase.  It would seem apparent that using TW’s DNS instead of OpenDNS fixed the issue, but now I’m left with just as many questions as answers.  What’s up with OpenDNS?  Do they have issues of their own?  Is TW doing something on their end to interfere?  If not, did a simple DNS query to a non-TW DNS server take forever?

Needless to say, I’m not too pleased with this situation.  If this is going to be a permanent issue, I now have to find another way to do content filtering.  I will most-likely be giving ScrubIT a try now to see if there are any similar issues.  We’ll see how that goes.

It doesn’t matter how many situations and problems you’ve dealt with as an IT guy, something new and different will eventually come along to make you scratch your head.

On Friday morning I got a notice that our internet connection had dropped out for a short period of time.  After just a minute of being down, the connection came back up.  No big deal, right?

Throughout Saturday morning, I’ve received another notice stating that our connection was down, and then up again.  This happened twice on Saturday morning.  Slightly concerned, I downloaded a little utility (Uptime Scout) to constantly ping our network and log any timeouts that occurred.  At the same time, I called Time Warner Tech Support and opened a ticket with them so they at least had it on record.

When I returned later in the evening, what I found in the logs was very interesting.  Our internet connection was dropping out every 33 minutes, almost to the second.  The outage didn’t last long … a minute at the very most.  However, we use F1, and if this happens on Sunday morning things could get a little messy.

Now I’m very intrigued about our problem, and so I decide to go on-site to look at our cable modem and firewall.  I’m figuring that SURELY our modem is resetting itself or that there’s something that I’ll be able to see.  However, after sitting in the cold datacenter for 50 minutes and watching 2 more outages happen, it’s quite apparent that there’s nothing wrong with the cable modem.  It’s not flinching, and the traffic indicators are still flashing despite my inability to ping anything on the internet.

Well, that leaves you with 2 possible causes … faulty equipment directly outside our building, or a faulty network card in our pfSense box.  As unlikely as I think it is that our pfSense box would have a bad nic (I just replaced it), it’s definitely not outside the realm of possibility. 

So, for the last 40 minutes, my laptop has been connected directly to the cable modem in an attempt to rule out any faulty component on our network, and just as I started typing this paragraph, my pings began to time out and Uptime Scout started making noises at me.  From what I can see, whatever is puking every 33 minutes is not connected to our network or even in the building.

I called Time Warner back with this extra tidbit of information, and I must say that their tech support has been very nice to work with, even at 1:00am.

It’s been a very quiet week here in IT Land.  Things have been going smoothly, and I’ve been spending today closing out the week and cleaning up where needed.

Around 2:30 we had a sweet thunderstorm roll through the area, and I sat outside for awhile and took it all in.  Afterwards, I sat down at my desktop and saw that my MSN connection was down.  In fact, everything internet-related on my PC was disconnected.

I went over to our MDF to look at the firewall.  It was off and pressing the power button didn’t bring it back to life.  My first thought was “toasted power supply,” but after doing some diagnosis, Jeff and I actually discovered that it was our cable modem that took the lightning hit.  It made more sense, because the cable connection is the only thing in the server room that wasn’t surge-protected.

The strange (but great) thing?  The strike only took out ONE port on our Ambit cable modem.  The modem has 4 ports, and amazingly enough, the other 3 ports work just fine.

Here’s a shot of the cable modem.  It was plugged in on port 4 when the lightning hit.

Here the ethernet cable that connects the cable modem to our firewall.  And no, that’s not BBQ sauce on the end of the cable.

Finally, the nic that was in our pfSense box.  I can’t believe this didn’t go beyond the nic and take out our whole firewall.

And I’ve gotta hand it to Time Warner, their customer service was excellent.  Within 5 minutes of dialing their number, I was able to arrange for a new modem to show up the next day.  Very nice.

I had mentioned about a week ago that Time Warner had been on the scene to heat up the cable that had been installed in our building during construction.

Well, about 10:00am yesterday, they were back.  This time they were here to install our modem!

The installation tech and I hooked the cable modem straight into my laptop to get a glimpse of how fast the connection was; it looks like it’s everything they advertised and then some.

 

I’m VERY excited to be able to get this new speed to our staff!  Our counting team that works on Monday mornings always suffers the most.  They’re using the Fellowship One contributions module, and it CRAWLS for them almost constantly because of the latency that comes with most wireless connections.  I’m hoping that this will be the fix that they need.

I turned into the driveway at NewPointe this morning and saw something new and exciting near our bridge…

  

The cables (that were installed by Adelphia before they moved out) have been here a long time, but there’s a jumper at end of the cable headed to the road, so someone’s definitely been doing some testing.  Plus that large green hood wasn’t there yesterday.  I’d say it’s a safe bet that Time-Warner has been on-site.  Sa-weeeeet!! 

Hopefully I’ll have more to post on later.  I don’t see any signs that the tech has been in the building yet.  Schedule-wise, this would be a GREAT week for the cable installation to happen.

More on this as today’s events unfold…

Update:  Looks like just the construction is happening today.  By the end of the day though, we’ll have a verified signal coming into the building, so it won’t be long until our internet access is provisioned and our modem arrives.  I am seriously PUMPED about this…for those of you who haven’t followed the story up to now, this is our first hardwire connection to the internet since we moved out of Sugarcreek.

 

Here’s the splitter that was installed in the work room.  I’m half-tempted to hook a TV to it to see if anything will come in.