One of the best pieces of troubleshooting advice I’ve received was something I got from Ed during this ordeal with the video and backup servers:  Write down everything you know about the situation, [no matter how minute] and draw pictures if you can.  Use this knowledge to aid in your troubleshooting.

I noted each and every little detail about the situation that I could think of, and then started hammering away at the variables.  Shortly after lunch I came across an avenue that hadn’t yet been explored:  This whole time I had the same NIC assigned to the backup network, and I had treated the connection to the core network like the “problem area.”  What if that wasn’t the case?  What if the problem wasn’t with the connection to the core network, but the backup network connection instead?

So after thinking through this, I unplugged both NICs from the network, swapped IP address assignments, and plugged both cables back in to their cards according to their new assignments.  Lo and behold, the CommVault server opened a stream to our video box and the data started flowing freely. 

“And there was much rejoicing…”   -Monty Python and the Holy Grail

 

To say that some weight has been lifted off my shoulders would be an understatement.  I was seriously a couple “elimination steps” away from doing a rebuild of the video server.

So what was the problem?  Windows network services was trying to access the “core network” NIC ahead of the backup NIC.  This is why the backup server could initiate a connection from the backup server to start a backup request, but would not open a stream.  Apparently the stream is initiated on the client end, and the video server was trying to do this with the wrong NIC.  There’s a way you can avoid this using Windows settings that I didn’t even know existed until today.

1.) Go to Control Panel -> Network Connections.  Click on the Advanced menu and select Advanced Settings.

2.)  From here, you can set the order in which your network adapters are access by Windows.  Set the order however it suits your needs and click OK.

I felt a little silly for not exploring Windows a little more than I did before uncovering this, but nonetheless I’m very glad to have this new piece of knowledge in my arsenal.  You can bet I’m going to be spending some time later this afternoon checking the rest of the servers that are attached to CommVault and make sure their settings are correct.