I realize that my posting on our domain rebuild ended suddenly in the middle of the night, so I’m going to attempt to fill in what’s happened between then and now, as well as recap from the start.
Wednesday, 3:00pm - Prepping for what’s going to be a very big evening. I’ve got food ready for "lunch," I’ve got water and coffee ready…. I even put a couple live webcams up so that my fellow geeks could keep tabs on me throughout the night. I’m still excited, and still nervous, and why shouldn’t I be? It’s a big job.
5:30pm - Zero hour. First thing to do, get snapshots of the old DC and Exchange boxes. If something goes horribly wrong, I want to be able to fail back. Next, I check and make sure backups are decent. A quick look at CommVault tells me something isn’t right: Our 9TB backup array is reporting FULL, and thus Exchange is unable to run backups. A quick look at my settings shows that our video server is putting WAY too many full backups on the server, and therefore clogging things up royally. I make the adjustments, run a data aging process, and about 2 hours later, things are finally freed up enough for me to start running ExMerge against the Exchange server.
9:20pm - I’ve finally got ExMerge doing what it’s supposed to be doing. There are 46 mailboxes that are being exported off the server, and they amount to about 16GB total. I figured this was a small deal. Little did I know it would be….
Thursday, 5:00am - …EIGHT hours later before ExMerge finishes. CRAP! That’s a 2GB/hour transfer rate. There’s a couple reasons I think it took this long:
RAID 5 - My Exchange server is a VM, and I was writing the PST data to a share on the host machine. This is all happening on a RAID 5 array, so I wouldn’t be surprised if it choked hard on all that data. What I should have done was export the PSTs to an external HD or another machine.
Folder sizes - Yes, the PSTs only made up about 16GB, but there was MUCH room for improvement. There were Deleted Items and Sent Items folders that were WAY too large. I saw one Inbox folder (owner will remain anonymous) that contained over 1900 messages. I should have asked our users to clean/export a majority of this data before the ExMerge operation.
ExMerge - By far the BIGGEST improvement I could have made. I read through the whole ExMerge documentation, and somehow failed to remember that it can do incremental backups! I could have run this a week ago and taken HOURS off the transition time.
In any case, ExMerge is done now, and it’s time to shut down the old sugarcreekfmc.org DCs and Exchange server and get the new ones cranked up. The new DC fired right up on its VM host, and at 5:12am (yes, I looked at the Event Log) I ceremoniously added the first member server to the newpointe.loc domain; our WSUS box. The file server and my desktop PC would follow shortly after.
Now that the new DC and Exchange server are cranked, it was time to bring mailbox data back in. Still feeling the effects of an 8-hour ExMerge session, I decided to move all the PSTs onto an external HD before merging them into the new Exchange box. For the record, this went MUCH faster. However, I knew that I couldn’t get everyone in before 8:30 (when people usually come in). I had requested that those who were coming in on the 27th email me so I could give their machines priority. Thus, I had a list of about 8 people that I needed to get ready for.
These 8 lucky staffers would be the FIRST people (besides me) to log in to the new domain, but more importantly, they had stuff to get done on Thursday, so I wanted to make sure EVERYTHING they needed was ready.
10:00am - Email, check. Profiles moved over… uhm… almost check. File permissions…those didn’t go very well. I spent a LOT of time going back and forth between users’ desks and mine to change file ownership and permissions. To make matters worse, not only were people coming in early, people were coming in that didn’t tell me they would be there. (can I get a "DOH!").
By lunch time, things were still progressing, albeit slower and slower. I was wearing out. The operation was coming up on 18 hours. I had been up for about 26 hours at this point, and I was starting to lose steam quickly.
1:00pm - The afternoon drug on … I was still going back and forth fixing file permissions, and now I’ve also discovered that my GPO for passwords is acting up as well, some users aren’t able to reset their passwords back to what they were. This wasn’t the end of the world, but with my reduced ability to handle stress, it was wearing on me hard.
2:00pm - I was about at my last straw at this point. I’ve started on laptops, and for some reason, RPC-HTTP/S, which WORKED during the initial build and testing is no longer working. I can’t even authenticate! I’m way too tired by now to understand what the problem is, and nothing frustrates me quite like something that I can’t understand. RPC-HTTP/S will have to wait….we’ll use OWA for now.
I’m walking back to my desk feeling utterly defeated. Did I set my expectations too high? Did I not research enough? I’ve experienced nothing but problems all day. The users that are here barely have the functionality they need to do their job, and now I’ve got a broken Exchange server to deal with.
I got back to my desk and saw this pinned to my chair…
One of the staff had put it there, along with a bottle of flavored water, which I promptly drank. The sign was strategically placed so that one of my webcams was aimed right at it. This blessed me more than I can say, and it must’ve been something I needed pretty badly at the time. Spirits lifted, I took care of a couple more issues before finally heading home at 3:30pm for some much needed sleep.
Total work time so far: 22 hours.
Powered by FireStats