Last week before I went on a mini vacation, I set up a new user account for a staffer.  I issued them their username and password, got them connected to Exchange, and everything seemed to be just hunky-dorey.

This past Tuesday I got a call from my boss.  Apparently the user wasn’t able to receive email, although they could send it just fine.  I checked out the normal stuff… permissions, time sync across the servers, the usual.  Not knowing what to do yet, I went ahead and backed up the user’s Exchange data and deleted the mailbox with the intent of starting over.  After re-creating the mailbox, I was quite perplexed to find that I could no longer even connect to Exchange with this user’s account.

I’m not sure what let me do to it, but I remoted to both DCs to take a look at their AD structure side-by-side.  Imagine my surprise when I discovered that there were user accounts missing from our #2 DC.  A further look into the event logs shows that replication between the 2 DCs has been stopped due to a bad computer account.  Because of this, not only is Active Directory broken on this DC, but DNS services (which are relying on AD) are broken as well.  Digging further into the event log, I find that the system is getting hardware errors while attempting to write to the hard disk, which is what corrupted the computer account responsible for shutting down AD replication.

Fast-forward a little to Black Friday.  I don’t shop on Black Friday.  Ever.  However, I am looking over the hard disk that our 2nd DC runs off of.  A disk scan is turning up massive amounts of physical errors on this drive, and although I’d like to try ghosting the system onto a new disk, there’s a good chance that I am going to be building “ripping out” this system from the domain with a little help from the MS Knowledge Base.  It’s a little frustrating to be repairing this domain with a rebuild happening at the same time, but I don’t want to take any chances

We’ll see how this works out.