I was actually relieved to know that my domain problems were caused by a bad hard drive.  Even though it’s time-consuming, it was a halfway easy fix after hitting up some knowledge base goodness.

Since the DC’s hard drive was dead, I had to use ntdsutil to remove the dead computer and all of its metadata from Active Directory (the article I used is here).  While this was going on, I had a clean install of 2003 Server cranking out on a new disk.  Once it was ready, it joined to our domain with no difficulty.  The DC promo process went smoothly, and roughly 14.5 hours after the incident was reported, things were back to normal.

…and I didn’t have to crack open that bean and rice soup, either… although I might anyway.

Well as it turns out, before I could even get a chance to ghost the offending hard drive in our DC, the drive failed entirely.  I now have a smoking hot hard drive on the table.

So now begins the process of not only rebuilding a DC, but also “forcibly removing” the old DC from our domain.  Fortunately, I’ve got a big bowl of rice and bean soup, and 2 different kinds of hot sauce.

It’s gonna be a fun evening.

Last week before I went on a mini vacation, I set up a new user account for a staffer.  I issued them their username and password, got them connected to Exchange, and everything seemed to be just hunky-dorey.

This past Tuesday I got a call from my boss.  Apparently the user wasn’t able to receive email, although they could send it just fine.  I checked out the normal stuff… permissions, time sync across the servers, the usual.  Not knowing what to do yet, I went ahead and backed up the user’s Exchange data and deleted the mailbox with the intent of starting over.  After re-creating the mailbox, I was quite perplexed to find that I could no longer even connect to Exchange with this user’s account.

I’m not sure what let me do to it, but I remoted to both DCs to take a look at their AD structure side-by-side.  Imagine my surprise when I discovered that there were user accounts missing from our #2 DC.  A further look into the event logs shows that replication between the 2 DCs has been stopped due to a bad computer account.  Because of this, not only is Active Directory broken on this DC, but DNS services (which are relying on AD) are broken as well.  Digging further into the event log, I find that the system is getting hardware errors while attempting to write to the hard disk, which is what corrupted the computer account responsible for shutting down AD replication.

Fast-forward a little to Black Friday.  I don’t shop on Black Friday.  Ever.  However, I am looking over the hard disk that our 2nd DC runs off of.  A disk scan is turning up massive amounts of physical errors on this drive, and although I’d like to try ghosting the system onto a new disk, there’s a good chance that I am going to be building “ripping out” this system from the domain with a little help from the MS Knowledge Base.  It’s a little frustrating to be repairing this domain with a rebuild happening at the same time, but I don’t want to take any chances

We’ll see how this works out.

I was chatting with Justin Moore this morning and he pointed me to this link…probably the funniest campaign ad I’ve seen.

…I-O

..it’s like cough syrup… and not that prissy grapetty-tasting stuff either. It’s like that cough syrup that you took as a kid; the kind that doubles as ipecac for when you ingested some cleaning product that mom kept under the sink.

• It sucks. No 2 ways about it.
• You know it’s good for you, and so you do it anyway (and it still sucks).
• You wind up being better off for going through the initial agony of it.

I’ve been working on rebuilding our domain here at NewPointe for about 3 weeks now. I’m taking my time on it because, well, it’s not an uber-urgent matter, and I want to do everything proper. A couple days ago I realized that “doing it proper” requires … yeah, you guessed it … documentation.

I stopped working on the domain stuff and fired up a wiki to document everything that’s happened so far with the new domain, starting with the domain itself, and eventually branching out to cover all the member servers. Once I’ve caught up to where I’m at, I’ll resume work on the rebuild and document as I go.

Is it fun? No, not by a long shot. However, the value of having this information typed and accessible is immeasurable.

What if, after the domain is built, I fall ill for several weeks, get hit by a bus, or find myself jobless for whatever reason? Simply put, we’d be hosed. Managing servers, much less a domain, is VERY difficult to do without proper documentation. With documentation, you can ensure that someone can come after you and keep things running, and in ministry world, that’s EXTREMELY important.

I know I’m preaching to the choir on some levels here, but I feel like this was worth taking a break to talk. Now, I’m going back to take some more cough syrup.

Series begins here.

I haven’t had much of a chance to work on the rebuild until this week; other things have popped up and taken precedence.  Thus, the new DC has been sitting idle in the corner of my office.

So far, things have been smooth, albeit a bit tedious.  I spent part of this week rebuilding and evaluating our current GPOs.  After a look at what some of them did, a few of them didn’t make the move.  Once the GPOs were rebuilt and adjusted accordingly, I applied them to their proper OUs.

The next task (and so far the worst) was recreating all of the user accounts and groups from our existing AD (for the first time since my hire, I’m glad we don’t have a larger staff).  I documented which users were assigned to what groups, and re-assigned them on the new domain.  Now this may be a bit overboard, but I assigned unique passwords to each user and service account that was recreated (I got my random passwords from SafePasswd.).  In addition, I took the time to enter every user’s contact information and staff info.  Was it necessary?  Probably not, but I’m a sucker for complete data, plus it’ll look good in the Exchange address book.

Now that the user base is rebuilt, it’s time to crank up another VM and install Exchange on it.  Once it’s installed and updated, I’ll be testing mailbox exporting/importing with ExMerge.  It will probably be next week (at the earliest) until this happens.