It’s now 11:12am, and I just walked back to my desk to check something. It now appears that DC #2 as well as all our PCs on the network are now 1/2 hour off… the other way!

I’m going to add the syncing program to DC #2 and see if that doesn’t resolve the issue.

This is too weird.

Update: After installing TimeSync on the second DC, things seem to be back on track at least for now. Hopefully they will stay that way.

Another Update:  I actually watched DC #1 fall out of sync by 2 hours a little later in the afternoon.  Not knowing what else to do, I went ahead and rebooted both DCs simultaneously.  It’s been about 24 hours now and I haven’t seen any clock skews as of yet.  Isolated incident?  We’ll see…

Yesterday I came in to install updates on all our servers.  I updated every server except our #2 DC.  No particular reason, I just forgot.

I walked into the building this morning and sat down at my desk only to notice that my computer clock (along with everyone else’s) is exactly 1/2 hour off GMT-5 time.  The only clock that isn’t off is the one domain controller that syncs its time externally, which is not the aforementioned #2 DC.

After installing updates on the forgotten DC and rebooting, everything seemed to go back to normal.  In fact, all of the PCs on the network synced back to the correct time almost instantly.

Aside from “Patch your Domain Controllers simultaneously,” I don’t know what the moral of this story is, but it’s definitely an interesting issue.

One of the best pieces of troubleshooting advice I’ve received was something I got from Ed during this ordeal with the video and backup servers:  Write down everything you know about the situation, [no matter how minute] and draw pictures if you can.  Use this knowledge to aid in your troubleshooting.

I noted each and every little detail about the situation that I could think of, and then started hammering away at the variables.  Shortly after lunch I came across an avenue that hadn’t yet been explored:  This whole time I had the same NIC assigned to the backup network, and I had treated the connection to the core network like the “problem area.”  What if that wasn’t the case?  What if the problem wasn’t with the connection to the core network, but the backup network connection instead?

So after thinking through this, I unplugged both NICs from the network, swapped IP address assignments, and plugged both cables back in to their cards according to their new assignments.  Lo and behold, the CommVault server opened a stream to our video box and the data started flowing freely. 

“And there was much rejoicing…”   -Monty Python and the Holy Grail

 

To say that some weight has been lifted off my shoulders would be an understatement.  I was seriously a couple “elimination steps” away from doing a rebuild of the video server.

So what was the problem?  Windows network services was trying to access the “core network” NIC ahead of the backup NIC.  This is why the backup server could initiate a connection from the backup server to start a backup request, but would not open a stream.  Apparently the stream is initiated on the client end, and the video server was trying to do this with the wrong NIC.  There’s a way you can avoid this using Windows settings that I didn’t even know existed until today.

1.) Go to Control Panel -> Network Connections.  Click on the Advanced menu and select Advanced Settings.

2.)  From here, you can set the order in which your network adapters are access by Windows.  Set the order however it suits your needs and click OK.

I felt a little silly for not exploring Windows a little more than I did before uncovering this, but nonetheless I’m very glad to have this new piece of knowledge in my arsenal.  You can bet I’m going to be spending some time later this afternoon checking the rest of the servers that are attached to CommVault and make sure their settings are correct.

David Szpunar from Lakeview Church has a great post on Windows SteadyState and how he’s implementing it at LC.

If you’re looking to set up some kiosk-style stations in your building, or any other “public access” PC, you need to check this out.

I have to admit, I screwed something up.

A few posts back I had written about a supposedly successful P2V conversion on one of our domain controllers.  I want to be the first (hopefully) to say that I was wrong on this.  After further testing, replication simply wasn’t happening between both DCs, even though the event viewer showed that it was so.

As such, I’m probably going to end up building a new DC from scratch on one of our VMware hosts.  We haven’t got any servers set for deployment at this time, so this will be an opportunity to sysprep a Server 2003 installation and store it off to the side for later.

I mat take another crack at P2V-ing a domain controller in the future, but right now there are bigger fish to fry.

[Note:  As of 4/25/2007 The procedures in this entry were found to be incorrect, and are NOT suitable for a domain controller P2V conversion.  Please read only if you are bored.]

Some time ago, we attempted a Physical-to-Virtual conversion on our domain controller with no luck (seems the SYSVOL tree doesn’t make it through the process).  Since then we’ve brought another DC online, so I’ve been itching to give this another try.

This past Tuesday was a scheduled work night for me, so I decided to see if we could make this work.

Before I started the VMware converter program, I made a backup of the System State of the DC (still in its physical form), that way we would have a copy of Active Directory, SYSVOL, all our active policies, etc. 

After the backup was complete, I went to the VM Host machine that the Domain Controller would live on and started the P2V using VMware Converter.  After an hour or so, the conversion was done and I had an image of the domain controller sitting on the host machine.

After shutting down the physical version of the domain controller, made the proper setting changes for the VM (brought the memory allocation down and disconnected the floppy and CDROM drives), held by breath, and clicked “Start.”

The machine seemed to start up ok, but soon I started to see notices that “services have failed during startup.”  A peek into the event log revealed the problem:  The VM didn’t have a static IP address like it was supposed to.  DOH!  During the conversion process, the physical network card goes away, and with it goes your TCP/IP settings.  After adjusting the network settings and rebooting, things smoothed out quite a bit.

However, a look at the File Replication events showed that things were still not cool.  The SYSVOL tree was missing and therefore replication was not happening.

Ok..reboot again…this time I booted into Directory Services Restore mode and restored the system state that I had saved before doing the conversion.  After doing this, you’ll have to reconfigure the network adapter in your VM again to get your TCP/IP settings back.

WHEW!!  A look into the event log showed that replication was happening, and thus the system was now functioning as a domain controller again.  The only weirdness I saw was that the Net Logon service was paused.  Hm?!  I got it going, rebooted a couple times to make certain it wouldn’t pause again, and all seems to be well.

The next step?  Make double-sure that any changes you make to Active Directory get replicated to your other domain controllers.  I went ahead and brought down the second domain controller (the one that didn’t get converted) to make sure that the VM could service logons and run scripts and such.  So far, everything appears to be good!

Right now, the only “What If” I can think of is this… what if we had put the physical DC in Directory Service Restore mode BEFORE the conversion?  Would I have had to restore the SYSVOL tree from a backup?  Is there anything else that could have been avoided?  I’m no expert in this area, but it intrigues me enough that I may try the whole thing over again just to see the results.  I don’t have immediate plans for the physical box, so if it flops, I can always switch it back on for a fix.

Now I just have to find the spare time to try it out.