This past Friday I had the opportunity to spend the day with Jason Powell and Ed Buford from Granger Community Church.    GCC is about a 5-hour drive from my place and I dig road trips, so I decided to make a one-day gig out of it.

I rolled into the parking lot around 9:30 and met up with Ed and Jason soon after.  We spend the day covering a miriad of different topics, from SANs to backup strategy to VLAN-ing, and a bunch of other things in-between.  I came home that night feeling absolutely exhausted but very inspired and extremely excited about what I do (not that I don’t usually enjoy what I do, but I feel especially pumped).

I can safely say that this was no-doubt the best learning experience I’ve had in the IT field to date, and it was also a great example of what can happen when you get out of the solo gig and start networking yourself with other people who do what you do.  Monster props to Ed and Jason for taking time and allowing me to hang with them for a day.

Tomorrow I’m going to have the privilege of going to Granger Community Church to spend the day with Jason Powell and Ed Buford.  I am PUMPED, man! 

I stumbled onto Jason’s and Ed’s blogs sometime in spring 2006, and then met both Jason and Ed at the 2006 Church IT Roundtable.  The Roundtable was a HUGE eye-opener for me, and I have learned an amazing amount of stuff just from that experience. 

So naturally I’m really looking forward to this trip.  We don’t have an extensive list of pre-planned discussions, but any time you put 2 or more IT geeks in the same room, discussion will always ensue (it’s true!).  I’m taking my laptop and plenty of paper so I can capture whatever we talk about.  This trip will definitely be beneficial.

Last week I came into the office to find that a drive on our domain controller had failed.  No big deal, right?  I overnighted 2 drives, and installed them both the next day; one as a replacement for the failed drive, and another as a hot spare.

I was hurried into work this morning by a voicemail from K.  She informed me that noone was able to log into the Exchange server.  I know I had everything running last night before I left, so this really caught me off-guard.

I rush into the office (fortunately I’m only about 5 miles from the church) and sit down to begin diagnosis.  My tests led me once again to the domain controller I had just worked on less than a week ago.  I tried to click on the start menu, and the whole system froze up.  Wow, the same symtoms that I experienced last week.  I winced, hit the reset button on the machine, and sure enough, the RAID controller’s BIOS shows a rebuild in progress.  Once the server was back up and running, I could see in the event log that the failure had taken place at 7:30

I was partially relieved to see that the drive I had just installed wasn’t the culprit, but was another one of the original drives for that array.  Wow…2 drives in under a week’s time.  These were both Maxtor DiamondMax hard drives that had failed on me, and they’ve been in use for less than a year.  The controller is a Promise Technology TX4310, and we’ve had no issues up to this point.

.

 

Another weird point is that the rebuild actually started BEFORE the machine locked up.  The hot spare stepped in just like it was supposed to, but the controller shows 5 device timeouts before the system gave up on it.

Hopefully this is a drive issue and not a controller issue.  Nonetheless, the task of getting a second DC up and running just went up in priority by many notches.

This past weekend we kicked off our new message series - Elements.  The tagline of the series is “6 elements that move your faith beyond periodic.”  Naturally, the phrasing calls for a science-like decor on the stage, no?

Here are some pics from the weekend. (Click to enlarge)

-

Grant and Vince working on the giant test tubes.  Plexiglass with light gels set behind them.  Add light and stare at for awhile.

Yep, that’s a DMC on the grill, as in ”DeLorean Motor Company.”  Where there’s science, there’s Doc Brown, and where there’s Doc Brown, there has to be a DeLorean!

Mike (the owner of the DeLorean) finishes up the detail work.  Those cars are PRETTY! (at least I think so)  This is my first time seeing one up close.

Dwight’s “podium.”  Very colorful. 

 

Super Volunteer Dave B. works on getting the giant test tubes mounted.

Guissepi the dog guards his master’s car ferociously.

The Elements stage from the control room.  I’m diggin’ the colors, man!

Doc Brown makes his entrance on the DeLorean (we actually had Mike drive it in from backstage.)  GREAT SCOTT!

-

The DeLorean is actually a story in itself.  We had been wanting to move forward with this idea for weeks, and we just kept running into one dead end after another.  The creative team was about ready to give up and just send Doc Brown in on an explosion, or at least a lot of smoke.

On Friday night, our control room manager, who is also a teacher at Kent State, was in the building helping the visual arts crew prep the stage.  He got into a conversation with one of the crew about the DeLorean, and said “well hey, I’ve got a student that owns a DeLorean, let me call him.”  Within one hour, we had the DeLorean sitting on our stage.

It’s amazing how God works stuff out sometimes.  I mean yeah, we could’ve gone forward without the car, but it was a good fit and it added a good bit of excitement to the morning.  I mean seriously, how often do you get Doc Brown rolling in on a DeLorean…on the stage of a CHURCH!

But then again, that’s exactly how it should be on a Sunday morning…always exciting, always intriguing (what’s gonna happen today?), and NEVER boring.

Well, here it is.  Today I moved my blog from its wordpress.com host to BlueHost.com.  The blog itself is still a WordPress blog, which made it pretty nice when it came time to move the data.  I don’t know much about other blogging platforms, but I know I was very pleased to see that WordPress imported everything from my other site except the blogroll.  On the other hand, I was a little bummed that WordPress didn’t import my blogroll.  Ah well, I was going to put it on a separate page anyway.

Will the visual style change immediately?  I don’t know.  I’ve been looking at a LOT of other themes, but I really haven’t landed on one that I like enough to pull me away from the Regulus theme I’m on right now.

I’ve got some neat stuff to post later, but I need to get some sleep for tomorrow.

The official snack food of the CITRT.

Jason Powell doesn’t endorse… he tells America how it’s gonna be.

There’s been a lot going on this week, and things aren’t winding down anytime soon…I’m just gonna throw out some random thoughts here.

It’s official:  We are going to move our primary internet connection over to Time-Warner cable.  6Mb/512Kb for $140/month and a $300 setup fee…I’m still trying to figure out how the price dropped so much.  This is going to be a huge improvement over the WiFi that we’ve been on since December.  We will probably keep the WiFi around for a backup, though.  DSL isn’t available in our location yet, so right now it’s our only option without spending a ton of money.  That extra outbound bandwidth is going to be huge.

Also, we’ve made a decision on our VPN:  SSL-Explorer is what we will be offering to our users.  I received an overwhelming “YES” vote from our staff when I put a poll out on whether or not they would use a VPN to access their computer and files remotely.  We’re going to start with a license for 10 concurrent users (you can have as many users in the database as you want, but only 10 can be connected at once), and we’ll see how that goes.  That’s like 1/3 of our staff, so we ought to be good for awhile.  You never know though.

On Tuesday night, we removed domain controller status from our file server, and now we only have one DC, and that’s all that machine does.  Seems simple enough, yes?  Well, I came in Wednesday to find that our Exchange servers had stopped talking to each other, and though I could reach our file server just fine, I wasn’t able to make any connections at all FROM the file server.  I did some digging and eventually realized that I never changed the DNS server entries for the former domain controller; it was still looking at itself for DNS records (doh!).  After making the DNS changes ans doing some work in ESM, our file server was talking to the primary DC again, and our Exchange servers are playing nice again as well.  The moral of this story?  Make sure you test your systems THOROUGHLY after a big change like this.

I came into the office this morning to the sound of multiple people calling my name…never good when that’s happening.  It seems that no one was able to authenticate to the first Exchange server.  A look at the machine’s event log revealed that it lost contact with our one-and-only domain controller shortly after midnight.  I tried remoting into the DC and got no response.  Not good.  I went to the server rack and pointed the KVM at the seemingly dead server.  The mouse seemed to work, but as soon as I clicked on an item on the desktop, the whole machine froze instantly.  Oh no, I thought, we make this domain controller the only one on the network, and it craps out in 36 hours!  Reluctantly, I hit the reset button to see if the system will boot up, and I then see the culprit:  It turns out that a drive on the server’s RAID5 array failed and brought the system to a halt.  Luckily, I’ve got 9 drives sitting on the shelf for a Ghetto-Tastic** project, so I remove the failed drive and put a fresh one in its place.  The server boots up, and after a minute or so, the array begins to rebuild using the new drive.  Whew!  Within a few minutes, the domain controller is back up, our servers are talking again, and our users can log in! 

Some observations from this:

1. I’m extremely glad that we decided to RAID this server last year.  Had this been a single-drive server at the time of failure, we would have been extremely hosed, and I would most-likely still be at work. :-) 
2. We need to get a hot spare drive for our domain controller.  According to the docs for the RAID controller, a hot spare would have prevented the system from locking up.  I’m no expert on how RAID setups are supposed to work, so I’m going to read more about this.  I already ordered 2 drives for this system; one to replace the failed unit, and another to plug in as a hot spare.  I’m considering getting hot spare drives for our other servers as well.
3. I miss having 2 domain controllers already.  Yes, it was a security risk serving files and mail off of our secondary DC, but if that machine still had DC status, we wouldn’t have missed a beat this morning.  I’m going to start planning for a 1U server to put in our IDF on the other corner of the building so we can avoid a repeat performance.

This has been a very eventful week.  There’s been some good learning experiences too.  I wish they hadn’t come at the expense of our uptime, but sometimes that’s the way it goes.

**Ghetto-Tastic - ©2006 Jason Powell

For the past few weeks our management team has been working on realigning our staff and making some role changes in our organization.  This has happened once already since I’ve been on staff.  It’s a healthy thing to do, because you have to change to stay effective.  Sometimes those changes are small, and sometimes those changes are not-so-small.  Being the lone IT guy, I’ve gotten a little used to the organizational changes not having much of an effect on me, which is why I got rocked at our weekly staff meeting today.

Our executive director had our new org chart ready and presented it to us today.  When he got to the Business Operations division (which is where the IT Department sits), my jaw dropped.  We now have a second person working in the IT Department!

K Greenwalt (her real name, I promise) is now working in IT operations and will be doing training and software support for our staff.  K is also our Fellowship One superhero and has a pretty good knowledge of our systems, so this was a pretty logical move.  Her background and strong ability to train is going to be extremely valuable as our staff begins to “live” in Outlook.  This to me is HUGE…not only because we now have a second person on board, but also because they’ve already been on the inside and have a good feel for how our office operates.  I’M PUMPED!!
Hopefully, this will give us the ability to begin moving forward with some projects we’ve had to sit on for awhile.  I’m very excited! (Have I mentioned that already?)  Welcome to the IT Department, K!

I didn’t really feel like troubleshooting between Firefox and IE, so I decided to load up Firefox and use it for the remainder of the evening. 

I was up around 3:00am this morning doing stuff around the house (I’m a total night owl), and I just happened to hear the famous logout.wav coming from my PC.  Windows was apparently shutting down to install updates.  What’s weird is that I had just gone to Windows Update last night (around 7 or 8pm) and loaded all the critical and optional updates for my PC.  Did Microsoft release another update within 7 hours?

I got to my PC this morning and attempted to load Google Reader using IE.  Voila!  Everything is loading just fine, and there are no deaded lock-ups like I was experiencing last night.  But, I don’t see any updates that were installed outside of the ones that I installed manually the previous evening.  Strange…. was this a covert operation?

I’ve mentioned before that I didn’t really take much time in deciding where to begin blogging at. Right now I’m really starting to see the benefits of hosting a Wordpress blog myself. This would allow me to take total control over how I want the blog to look. I’m a control freak, what can I say?

What I don’t want to do is cause inconvenience for people that already have a subscription to the feed here.  Yeah I know it’s only a matter of posting the new URL, I still feel funny about it though.  I’m weird like that.

So…I will probably decide this over the next week.