Archive for December, 2007
Saturday, 10:30am - The big thing for today? Make sure the rest of the staff that hasn’t come in since Christmas can get their work done when they come in.
For the most part, this involved going from desktop to desktop, adjusting settings and making certain that each user can access their files and print. Even waiting on ExMerge to finish was less painful than this, because I could at least do other things. There’s some things I would have done differently here, but I’ll outline them in a later post.
After 8.5 painful hours of going from desk-to-desk, I finally adjusted and tested every PC that needed it. Things were still not working like I wanted, especially in the way of GPOs. I was tired though, and decided that I would let it ride until Monday and fix things then.
Sure, right…
Sunday, 2:00am - I awoke feeling nasty…VERY nasty. I’m pretty sure it was something I ate, but regardless, I was unable to go back to sleep. Hey… I’m awake, and I have to be at church early to direct video and prepare a movie clip. May as well go in since my mind is pretty clear.
It’s amazing what you can do with some sleep! Throughout the course of the morning, not only was I able to get the last handful of PCs switched over to the new domain, but with some testing, I also managed to iron out the GPOs issues that had plagued me the day before. To make matters better, after doing some "wrenching" in Exchange, I got RPC-HTTP working just how it needs to be. All this because of some stomach nasties … talk about a blessing in disguise. 
So as it now stands, our users are equipped to go back to work tomorrow. Email is flowing everywhere it needs to be, shares and home directories are accessible by all users, printers are ready, and user profiles are switched over. All that’s left is to bring the rest of the servers online, which should be doable within a day.
This has been both a grueling task (partly because of my time limitations) and a wonderful learning experience for me. Expect to see a post coming up on how I would do this if I had to do it again (which I hope doesn’t happen anytime soon).
Total work time so far: 54 hours
|
Continuing where we left off…. here’s what is done so far:
- All mail data is backed up.
- New DC and Exchange servers are up
- Mail is flowing. (Thanks to Ed and Justin)
- I’ve got about 10 machines (out of 50) and 3 servers (out of 10) cut over to the new domain.
Friday, 2:00am - I woke from one of the best sleeps I’ve had in a long time, courtesy of my good friends Sealy and Unisom. I’ve been asleep for about 10 hours now and I’m feeling pretty good. There’s no point in going back to sleep, so I decide to clean up and head into work to try and get some things done while no one is around.
I spent a fair amount of the day wrestling with RPC-HTTPS for Exchange. We introduced this to our laptop users, and taking it away for any amount of time is no longer an option. :-) With some help from Experts Exchange and the #citrt crowd, we finally had RPC-HTTP working by the end of the day. It wasn’t perfected, but I was ecstatic that things were moving in the right direction. I also took this time to get OWA and OMA working to my liking. I had plenty of trouble with OMA, too. I ended up re-creating the IIS folders before finally being able to sync Windows Mobile devices up.
The remainder of the day was spent troubleshooting user issues, moving machines to the new domain, and getting our macs reconnected to the "new" file server. Everyone has been EXTREMELY gracious on staff in regards to what’s happening on their computers.
9:30pm - After a wonderful and final Christmas dinner with some family, I came back to move some more machines to the new domain. Jess was with me, and was extremely helpful as we changed 11 more computers over in record time. I’m blessed by the fact that she not only is OK with me being a geek, she understands what I do, and wants to help out as well. (In case you’re wondering, Jess does not have admin rights, except on her iBook.)
Total work time so far: 44 hours
|
I realize that my posting on our domain rebuild ended suddenly in the middle of the night, so I’m going to attempt to fill in what’s happened between then and now, as well as recap from the start.
Wednesday, 3:00pm - Prepping for what’s going to be a very big evening. I’ve got food ready for "lunch," I’ve got water and coffee ready…. I even put a couple live webcams up so that my fellow geeks could keep tabs on me throughout the night. I’m still excited, and still nervous, and why shouldn’t I be? It’s a big job.
5:30pm - Zero hour. First thing to do, get snapshots of the old DC and Exchange boxes. If something goes horribly wrong, I want to be able to fail back. Next, I check and make sure backups are decent. A quick look at CommVault tells me something isn’t right: Our 9TB backup array is reporting FULL, and thus Exchange is unable to run backups. A quick look at my settings shows that our video server is putting WAY too many full backups on the server, and therefore clogging things up royally. I make the adjustments, run a data aging process, and about 2 hours later, things are finally freed up enough for me to start running ExMerge against the Exchange server.
9:20pm - I’ve finally got ExMerge doing what it’s supposed to be doing. There are 46 mailboxes that are being exported off the server, and they amount to about 16GB total. I figured this was a small deal. Little did I know it would be….
Thursday, 5:00am - …EIGHT hours later before ExMerge finishes. CRAP! That’s a 2GB/hour transfer rate. There’s a couple reasons I think it took this long:
RAID 5 - My Exchange server is a VM, and I was writing the PST data to a share on the host machine. This is all happening on a RAID 5 array, so I wouldn’t be surprised if it choked hard on all that data. What I should have done was export the PSTs to an external HD or another machine.
Folder sizes - Yes, the PSTs only made up about 16GB, but there was MUCH room for improvement. There were Deleted Items and Sent Items folders that were WAY too large. I saw one Inbox folder (owner will remain anonymous) that contained over 1900 messages. I should have asked our users to clean/export a majority of this data before the ExMerge operation.
ExMerge - By far the BIGGEST improvement I could have made. I read through the whole ExMerge documentation, and somehow failed to remember that it can do incremental backups! I could have run this a week ago and taken HOURS off the transition time.
In any case, ExMerge is done now, and it’s time to shut down the old sugarcreekfmc.org DCs and Exchange server and get the new ones cranked up. The new DC fired right up on its VM host, and at 5:12am (yes, I looked at the Event Log) I ceremoniously added the first member server to the newpointe.loc domain; our WSUS box. The file server and my desktop PC would follow shortly after.
Now that the new DC and Exchange server are cranked, it was time to bring mailbox data back in. Still feeling the effects of an 8-hour ExMerge session, I decided to move all the PSTs onto an external HD before merging them into the new Exchange box. For the record, this went MUCH faster. However, I knew that I couldn’t get everyone in before 8:30 (when people usually come in). I had requested that those who were coming in on the 27th email me so I could give their machines priority. Thus, I had a list of about 8 people that I needed to get ready for.
These 8 lucky staffers would be the FIRST people (besides me) to log in to the new domain, but more importantly, they had stuff to get done on Thursday, so I wanted to make sure EVERYTHING they needed was ready.
10:00am - Email, check. Profiles moved over… uhm… almost check. File permissions…those didn’t go very well. I spent a LOT of time going back and forth between users’ desks and mine to change file ownership and permissions. To make matters worse, not only were people coming in early, people were coming in that didn’t tell me they would be there. (can I get a "DOH!").
By lunch time, things were still progressing, albeit slower and slower. I was wearing out. The operation was coming up on 18 hours. I had been up for about 26 hours at this point, and I was starting to lose steam quickly.
1:00pm - The afternoon drug on … I was still going back and forth fixing file permissions, and now I’ve also discovered that my GPO for passwords is acting up as well, some users aren’t able to reset their passwords back to what they were. This wasn’t the end of the world, but with my reduced ability to handle stress, it was wearing on me hard.
2:00pm - I was about at my last straw at this point. I’ve started on laptops, and for some reason, RPC-HTTP/S, which WORKED during the initial build and testing is no longer working. I can’t even authenticate! I’m way too tired by now to understand what the problem is, and nothing frustrates me quite like something that I can’t understand. RPC-HTTP/S will have to wait….we’ll use OWA for now.
I’m walking back to my desk feeling utterly defeated. Did I set my expectations too high? Did I not research enough? I’ve experienced nothing but problems all day. The users that are here barely have the functionality they need to do their job, and now I’ve got a broken Exchange server to deal with.
I got back to my desk and saw this pinned to my chair…
One of the staff had put it there, along with a bottle of flavored water, which I promptly drank. The sign was strategically placed so that one of my webcams was aimed right at it. This blessed me more than I can say, and it must’ve been something I needed pretty badly at the time. Spirits lifted, I took care of a couple more issues before finally heading home at 3:30pm for some much needed sleep.
Total work time so far: 22 hours.
|
ExMerge is finally up and running. Hallelujah.
|
Still waiting on the backup queue to clear out. Just a few more minutes and I should be able to start running ExMerge…
|
Well, things have been going along smoothly until I checked the backup server and realized that there are multiple jobs in the queue.
I’m not going to do this without an up-to-date backup, so it looks like I’m going to pause here and hunt the issue down.
UPDATE: Looks like there was a configuration problem with our video server. It was pumping full backups to our CommVault server WAY too often, and the 9TB array filled right up. A little reconfiguration and purging, and things are back on track.
|
So far so good. The office isn’t empty yet, so I’m taking care of the preliminaries.
- Preparing the external HD that will transport the ExMerge’d mail data to the new server.
- Running through the procedure a couple more times in my head.
- Get some RadioU playing in the office so it’s not so quiet.
|
Yeah, so this might by a little dorky. I don’t care. For your viewing enjoyment, the domain rebuild is going to be presented live on 2 different webcams. You can access them by following the links below.
Camera 1 - My Desk
Camera 2 - Mobile camera attached to my laptop.
Enjoy!
Update: These cameras have long-since been taken offline, so don’t waste your time clicking on the links.
|
If you’re going to work good, you need to eat good.
Just to quell any concerns that I may be blowing my diet to pull an all-nighter tonight (and yes, I do expect to be here all night), I thought I’d give a glimpse of what the menu will be like for the next 20-23 hours:
For starters, we’ve got some boneless skinless chicken breast. That alone is good stuff. I’ve also got some sweet onion, broccoli, zucchini, and mushrooms to throw into the mix for what stands to be a kickin’ stir fry combo, sans the rice. You’ll also notice some eggs and OJ there as well. That’s for breakfast. Yes. Breakfast.
I’m a geek, but I love to cook, and I can be quite serious about it. I might have also brought my wok and some olive oil with me to assist with the stir fry.
It’s going to be a good night.
|
Well, I’ve been talking about it on and off and today it’s finally going to happen: At 17:30 today, I will be taking our domain controller and Exchange servers offline and moving over to a new Windows domain.
I’m going to try posting events as they happen, and who knows…maybe there will be a live webcam in the mix somewhere, just for fun. Webcam or not, it’s going to be an exciting evening.
|
|
|
Powered by FireStats