There’s been a lot going on this week, and things aren’t winding down anytime soon…I’m just gonna throw out some random thoughts here.

It’s official:  We are going to move our primary internet connection over to Time-Warner cable.  6Mb/512Kb for $140/month and a $300 setup fee…I’m still trying to figure out how the price dropped so much.  This is going to be a huge improvement over the WiFi that we’ve been on since December.  We will probably keep the WiFi around for a backup, though.  DSL isn’t available in our location yet, so right now it’s our only option without spending a ton of money.  That extra outbound bandwidth is going to be huge.

Also, we’ve made a decision on our VPN:  SSL-Explorer is what we will be offering to our users.  I received an overwhelming “YES” vote from our staff when I put a poll out on whether or not they would use a VPN to access their computer and files remotely.  We’re going to start with a license for 10 concurrent users (you can have as many users in the database as you want, but only 10 can be connected at once), and we’ll see how that goes.  That’s like 1/3 of our staff, so we ought to be good for awhile.  You never know though.

On Tuesday night, we removed domain controller status from our file server, and now we only have one DC, and that’s all that machine does.  Seems simple enough, yes?  Well, I came in Wednesday to find that our Exchange servers had stopped talking to each other, and though I could reach our file server just fine, I wasn’t able to make any connections at all FROM the file server.  I did some digging and eventually realized that I never changed the DNS server entries for the former domain controller; it was still looking at itself for DNS records (doh!).  After making the DNS changes ans doing some work in ESM, our file server was talking to the primary DC again, and our Exchange servers are playing nice again as well.  The moral of this story?  Make sure you test your systems THOROUGHLY after a big change like this.

I came into the office this morning to the sound of multiple people calling my name…never good when that’s happening.  It seems that no one was able to authenticate to the first Exchange server.  A look at the machine’s event log revealed that it lost contact with our one-and-only domain controller shortly after midnight.  I tried remoting into the DC and got no response.  Not good.  I went to the server rack and pointed the KVM at the seemingly dead server.  The mouse seemed to work, but as soon as I clicked on an item on the desktop, the whole machine froze instantly.  Oh no, I thought, we make this domain controller the only one on the network, and it craps out in 36 hours!  Reluctantly, I hit the reset button to see if the system will boot up, and I then see the culprit:  It turns out that a drive on the server’s RAID5 array failed and brought the system to a halt.  Luckily, I’ve got 9 drives sitting on the shelf for a Ghetto-Tastic** project, so I remove the failed drive and put a fresh one in its place.  The server boots up, and after a minute or so, the array begins to rebuild using the new drive.  Whew!  Within a few minutes, the domain controller is back up, our servers are talking again, and our users can log in! 

Some observations from this:

1. I’m extremely glad that we decided to RAID this server last year.  Had this been a single-drive server at the time of failure, we would have been extremely hosed, and I would most-likely still be at work. :-) 
2. We need to get a hot spare drive for our domain controller.  According to the docs for the RAID controller, a hot spare would have prevented the system from locking up.  I’m no expert on how RAID setups are supposed to work, so I’m going to read more about this.  I already ordered 2 drives for this system; one to replace the failed unit, and another to plug in as a hot spare.  I’m considering getting hot spare drives for our other servers as well.
3. I miss having 2 domain controllers already.  Yes, it was a security risk serving files and mail off of our secondary DC, but if that machine still had DC status, we wouldn’t have missed a beat this morning.  I’m going to start planning for a 1U server to put in our IDF on the other corner of the building so we can avoid a repeat performance.

This has been a very eventful week.  There’s been some good learning experiences too.  I wish they hadn’t come at the expense of our uptime, but sometimes that’s the way it goes.

**Ghetto-Tastic - ©2006 Jason Powell