(Part 1 and Part 2)

Shortly after services were done on Sunday, I had some downtime while I was waiting on KidStuf to finish up, so I figured “what the heck…may as well try that new NIC out.”

I had gone to Staples the previous day and purchased a Linksys NIC for the video server so that we didn’t have 2 identical cards in the box.  I know it’s a long shot, but I’ve seen weird stuff happen with 2 like-branded printers trying to coexist on the same PC…maybe Intel NICs to the same thing?  I popped the new adapter into the box, fired it up, installed the drivers, set IP addresses, and…..

*insert chirping cricket SFX here*

…still nothing.  The system acts just as it previously did.  Backups still won’t run unless the video server is disconnected from our core network.

So by now you can probably guess that I’m getting frustrated.  I’m pretty good at understanding things, even if I just understand that I don’t know enough about the issue.  But everything I see from this experience tells me that our video and backup servers SHOULD BE TALKING.

1. Both systems reply to pings that are sent on the proper network, and display the proper IP addresses.
2. Both systems can run a successful traceroute, which again, shows the proper IP addresses.
3. A netstat list shows actual ESTABLISHED connections between the 2 machines…there are also TIME_WAIT-status connections present also.
4. Changing NICs out did not affect this issue, so my having 2 PRO 1000/GT cards in the video server is not a problem.
5. Moving the backup network off of its VLAN and onto a separate piece of copper did not resolve the issue either, so switch/VLAN configurations are ruled out.

I had a chance to talk to Ed Buford about this issue (who by the way is not only a gentleman’s gentleman, he’s also a network admin’s gentleman).  His response was the same as mine… “this is just weird.”  It made me feel good to know that I hadn’t missed something stupid in my search.  Out of our conversation, I’ve got some more tricks to try, and since I’m not doing a work night this week, I’ll have a chance to try them first thing this morning:

1. Reinstall the client software.  Is it possible that the CommVault client-side software looks at your installed NICs and binds to them?  I have changed adapters out since the client software was installed.  This won’t take long either.  May as well try it first.
2. Disconnect the backup server from the core network.  This is definitely not my first choice, but I’m ready to try it if we need to.  All backup operations happen on a network that is physically separate from the core (except for the VLAN’ed connection coming from the video box), and we’ve got a KVM in the server rack for when I need to access the backup server.
3. Permanently remove the video server from the core network.  Yet another thing I really don’t want to do, but when it comes down to it, the video server does serve ONLY our editing and playback system for storage purposes.  As long as the the Final Cut rigs and the backup server can talk to it, then it’s ultimately still doing its job.

Another thing I just thought of too… the video server is the ONLY backup client that isn’t a domain member.  Is this a big deal?  In my mind it shouldn’t be, but it’s another thing to explore…

I’ve had 4 cups of coffee and 2 waffles this morning and I’m feeling pretty psyched up.  I want to see this problem go away before the end of this week.

Stay tuned for the outcome.