Archive for the ‘Uncategorized’ Category
I posted a picture earlier today of a Mac that had crashed during an install DVD boot. The machine was slated to get wiped, reloaded, and set up as a Pro Tools rig for the Main Auditorium. Not thinking much of it, I went ahead and cleaned the machine without doing an reinstall of OS X. I installed our audio hardware and Pro Tools on top of it, and loaded a couple sessions to test things out. So far everything seemed to be working real well, and since I was playing drums for the rig’s first time on stage, I’d be able to further test the unit first-hand.
I carted the new rig over to the Warehouse (our Jr./Sr. High environment) and hooked it up. It tested just fine. However, once we started warm-up, things turned ugly. The G5 mac locked up almost instantly when we began playing a Pro Tools-aided song. I rebooted the machine, played the session back to make sure it was ok, and strangely, it was just fine. However, when we went to actually play along to Pro Tools the machine crashed again. What was WEIRD is that the mac would only seemed to crash while we were playing our instruments.
After a few more minutes of "experimentation," it became strangely apparent that the crashing was being caused by my snare drum. Now before you write this off as "the IT guy is on weed" or something, consider the process we took to get here.
- We ran the PT track with no band, and machine did not crash.
- We ran the PT track with everyone in the band except me for a few measures. As soon as I began to play the snare drum, the computer seized up.
- We ran the PT track again, this time with the band, and every drum on my kit except the snare. The machine ran just fine until I started hitting my snare, after which the machine locked up again.
As it turns out, we ended up playing the worship set without and back-up tracks, and it was still good. The whole situation is just weird though. I know there’s an underlying cause for the crashing (why else would an OS X install DVD not boot?), but being able to trigger it with the snare drum? That’s just goofy, and honestly, I don’t even know where to start with this one.
More on this weirdness as the situation unfolds.
|
This is rare, so I thought I’d grab a picture of it as a keepsake. This happened while the system was booting an OS X install DVD.
I’m not rejoicing or anything … like I said, something this rare is worth a picture. 
|
What started out Friday as a every-33-minute interruption has now become a drop-dead outage as of about 5:45pm today (my FTL drive didn’t make the jump in time, Jason). In an attempt to not go on-site, I’m currently on the phone with Time Warner tech support for the 3rd time in 36 hours. On a Monday morning, a 1-minute drop every 30 minutes would be quite an inconvenience, but it would be doable, but again we’re an F1 church, and F1 Contributions requires an internet connection to work.
The rep that I talked to was able to reset the modem remotely, but unfortunately no data is passing through. So, I’m about to go on-site to power-cycle the modem and follow up on the ticket if any further work is needed. I’m hoping like crazy that this isn’t going to be an issue that bites us on Monday morning.
|
We’re getting hammered with snow today, and I’m diggin’ it.
I don’t mind snow if it actually SNOWS … I mean a good nonstop 7-8 inches, and not the stuff that’s half rain either.
Well today we got that — so much so that our offices are closed and might even be closed tomorrow depending on what happens tonight.
Here’s some pictures I took while out today on a quest for some kerosene.
|
No, I’m not 30 years old … I was just trying to solicit your attention.
Most of you know that I’m in the Church IT Biggest Loser contest that Jason has put together. What a lot of folks don’t know is that I already started a weight loss project back in mid-November.
When I started this, I weighed in at 278. After much diet tweaking (and by that I mean "lowering my intake") and a steady amount of exercise, I got a nice reward today when I stepped on the scale…
For those who find that LCD hard to read … it’s showing ‘247.6′ Yep. I’m down 30-plus pounds as of today. While I can’t register that loss in CITBL, it feels good to have reached this mark.
The next goal? Yep, it would be 40. You’ll know when it happens, too.
|
I was expecting to get back to my apartment and have a normal unpacking procedure. Well, “normal” wasn’t in the plans for me.
Now please understand… I’m no slob, but my apartment has not won any awards for cleanliness. Ever. It’s not ever been “beautiful” or even “nice,” but it’s been functional.
You can imagine my surprise then, when I walked into my apartment this afternoon and saw this…
 
Now believe me when I tell you that my apartment did NOT look this good when I left. As it turns out, my girlfriend and mom both collaborated on this project and cleaned, vacuumed and painted the apartment while I was gone.
I am unbelievably blessed…
|
I don’t have anything against Red Roof Inn; they’ve been good to me. However, this is a little idea of how comfortably I slept last night.
It wasn’t that bad though, I had enough sleep to get back to Ohio. As it turns out, the Ohio Turnpike is pretty empty on the weekend, so the drive back was pretty nice.
 
 
It’s good to be home.
|
The official snack food of the CITRT.
Jason Powell doesn’t endorse… he tells America how it’s gonna be.
|
Continuing where we left off…. here’s what is done so far:
- All mail data is backed up.
- New DC and Exchange servers are up
- Mail is flowing. (Thanks to Ed and Justin)
- I’ve got about 10 machines (out of 50) and 3 servers (out of 10) cut over to the new domain.
Friday, 2:00am - I woke from one of the best sleeps I’ve had in a long time, courtesy of my good friends Sealy and Unisom. I’ve been asleep for about 10 hours now and I’m feeling pretty good. There’s no point in going back to sleep, so I decide to clean up and head into work to try and get some things done while no one is around.
I spent a fair amount of the day wrestling with RPC-HTTPS for Exchange. We introduced this to our laptop users, and taking it away for any amount of time is no longer an option. :-) With some help from Experts Exchange and the #citrt crowd, we finally had RPC-HTTP working by the end of the day. It wasn’t perfected, but I was ecstatic that things were moving in the right direction. I also took this time to get OWA and OMA working to my liking. I had plenty of trouble with OMA, too. I ended up re-creating the IIS folders before finally being able to sync Windows Mobile devices up.
The remainder of the day was spent troubleshooting user issues, moving machines to the new domain, and getting our macs reconnected to the "new" file server. Everyone has been EXTREMELY gracious on staff in regards to what’s happening on their computers.
9:30pm - After a wonderful and final Christmas dinner with some family, I came back to move some more machines to the new domain. Jess was with me, and was extremely helpful as we changed 11 more computers over in record time. I’m blessed by the fact that she not only is OK with me being a geek, she understands what I do, and wants to help out as well. (In case you’re wondering, Jess does not have admin rights, except on her iBook.)
Total work time so far: 44 hours
|
I realize that my posting on our domain rebuild ended suddenly in the middle of the night, so I’m going to attempt to fill in what’s happened between then and now, as well as recap from the start.
Wednesday, 3:00pm - Prepping for what’s going to be a very big evening. I’ve got food ready for "lunch," I’ve got water and coffee ready…. I even put a couple live webcams up so that my fellow geeks could keep tabs on me throughout the night. I’m still excited, and still nervous, and why shouldn’t I be? It’s a big job.
5:30pm - Zero hour. First thing to do, get snapshots of the old DC and Exchange boxes. If something goes horribly wrong, I want to be able to fail back. Next, I check and make sure backups are decent. A quick look at CommVault tells me something isn’t right: Our 9TB backup array is reporting FULL, and thus Exchange is unable to run backups. A quick look at my settings shows that our video server is putting WAY too many full backups on the server, and therefore clogging things up royally. I make the adjustments, run a data aging process, and about 2 hours later, things are finally freed up enough for me to start running ExMerge against the Exchange server.
9:20pm - I’ve finally got ExMerge doing what it’s supposed to be doing. There are 46 mailboxes that are being exported off the server, and they amount to about 16GB total. I figured this was a small deal. Little did I know it would be….
Thursday, 5:00am - …EIGHT hours later before ExMerge finishes. CRAP! That’s a 2GB/hour transfer rate. There’s a couple reasons I think it took this long:
RAID 5 - My Exchange server is a VM, and I was writing the PST data to a share on the host machine. This is all happening on a RAID 5 array, so I wouldn’t be surprised if it choked hard on all that data. What I should have done was export the PSTs to an external HD or another machine.
Folder sizes - Yes, the PSTs only made up about 16GB, but there was MUCH room for improvement. There were Deleted Items and Sent Items folders that were WAY too large. I saw one Inbox folder (owner will remain anonymous) that contained over 1900 messages. I should have asked our users to clean/export a majority of this data before the ExMerge operation.
ExMerge - By far the BIGGEST improvement I could have made. I read through the whole ExMerge documentation, and somehow failed to remember that it can do incremental backups! I could have run this a week ago and taken HOURS off the transition time.
In any case, ExMerge is done now, and it’s time to shut down the old sugarcreekfmc.org DCs and Exchange server and get the new ones cranked up. The new DC fired right up on its VM host, and at 5:12am (yes, I looked at the Event Log) I ceremoniously added the first member server to the newpointe.loc domain; our WSUS box. The file server and my desktop PC would follow shortly after.
Now that the new DC and Exchange server are cranked, it was time to bring mailbox data back in. Still feeling the effects of an 8-hour ExMerge session, I decided to move all the PSTs onto an external HD before merging them into the new Exchange box. For the record, this went MUCH faster. However, I knew that I couldn’t get everyone in before 8:30 (when people usually come in). I had requested that those who were coming in on the 27th email me so I could give their machines priority. Thus, I had a list of about 8 people that I needed to get ready for.
These 8 lucky staffers would be the FIRST people (besides me) to log in to the new domain, but more importantly, they had stuff to get done on Thursday, so I wanted to make sure EVERYTHING they needed was ready.
10:00am - Email, check. Profiles moved over… uhm… almost check. File permissions…those didn’t go very well. I spent a LOT of time going back and forth between users’ desks and mine to change file ownership and permissions. To make matters worse, not only were people coming in early, people were coming in that didn’t tell me they would be there. (can I get a "DOH!").
By lunch time, things were still progressing, albeit slower and slower. I was wearing out. The operation was coming up on 18 hours. I had been up for about 26 hours at this point, and I was starting to lose steam quickly.
1:00pm - The afternoon drug on … I was still going back and forth fixing file permissions, and now I’ve also discovered that my GPO for passwords is acting up as well, some users aren’t able to reset their passwords back to what they were. This wasn’t the end of the world, but with my reduced ability to handle stress, it was wearing on me hard.
2:00pm - I was about at my last straw at this point. I’ve started on laptops, and for some reason, RPC-HTTP/S, which WORKED during the initial build and testing is no longer working. I can’t even authenticate! I’m way too tired by now to understand what the problem is, and nothing frustrates me quite like something that I can’t understand. RPC-HTTP/S will have to wait….we’ll use OWA for now.
I’m walking back to my desk feeling utterly defeated. Did I set my expectations too high? Did I not research enough? I’ve experienced nothing but problems all day. The users that are here barely have the functionality they need to do their job, and now I’ve got a broken Exchange server to deal with.
I got back to my desk and saw this pinned to my chair…
One of the staff had put it there, along with a bottle of flavored water, which I promptly drank. The sign was strategically placed so that one of my webcams was aimed right at it. This blessed me more than I can say, and it must’ve been something I needed pretty badly at the time. Spirits lifted, I took care of a couple more issues before finally heading home at 3:30pm for some much needed sleep.
Total work time so far: 22 hours.
|
|
|
Powered by FireStats