I’ve been thinking a bit about multi-campus issues lately.  We’re not there yet, but I think sooner or later we will be, and I want to be ready to roll when it happens.  Being a dude that has no formal IT training and has only managed a single-site network, I’m both excited and a little anxious about the prospect of taking our IT and video infrastructure outside the walls of what would eventually be the "Central Campus."

Last week I had a chance to talk with Jared B. from NewSpring Church and get my mind around what it will take to stream live buffered video from one campus to another.  The tools that they’re using are amazing, and seem to be just the fit for what I like to call "North Point-style video" (IMAG on the sides with a HD image of the communicator in the center). The idea of being able to get our services streamed to another location is insanely cool to me, and next week, I’ll be calling around to see just how big of a data pipe I can bring in to NewPointe, as tihs project will require a HEAP of bandwidth for both locations.  (One of our contractors told us there’s fiber possibly running along our road already, how sweet would it be to ride that.)

Another item to consider will be our network.  When it was set up in our current building, I wasn’t even thinking about multi-site ventures.  As a result, the network isn’t in prime condition to be bridged off just yet.  I want to keep our domain intact across our organization, so eventually I’ll need to convert everything from 10.0.0.0/8 to 10.0.0.0/16 and then begin testing site-to-site stuff.  I really don’t want to spend on on new firewall hardware unless it’s absolutely necessary, so I’m gonna be banking on Brutus (our pfSense box) to make it happen.

Like I said, this stuff has me a little anxious, but VERY psyched at the same time.  We don’t have a set date or anything like that for acquiring a second campus, but if I can get some preparations done before that all hits, I’ll feel real good.

One of the best pieces of troubleshooting advice I’ve received was something I got from Ed during this ordeal with the video and backup servers:  Write down everything you know about the situation, [no matter how minute] and draw pictures if you can.  Use this knowledge to aid in your troubleshooting.

I noted each and every little detail about the situation that I could think of, and then started hammering away at the variables.  Shortly after lunch I came across an avenue that hadn’t yet been explored:  This whole time I had the same NIC assigned to the backup network, and I had treated the connection to the core network like the “problem area.”  What if that wasn’t the case?  What if the problem wasn’t with the connection to the core network, but the backup network connection instead?

So after thinking through this, I unplugged both NICs from the network, swapped IP address assignments, and plugged both cables back in to their cards according to their new assignments.  Lo and behold, the CommVault server opened a stream to our video box and the data started flowing freely. 

“And there was much rejoicing…”   -Monty Python and the Holy Grail

 

To say that some weight has been lifted off my shoulders would be an understatement.  I was seriously a couple “elimination steps” away from doing a rebuild of the video server.

So what was the problem?  Windows network services was trying to access the “core network” NIC ahead of the backup NIC.  This is why the backup server could initiate a connection from the backup server to start a backup request, but would not open a stream.  Apparently the stream is initiated on the client end, and the video server was trying to do this with the wrong NIC.  There’s a way you can avoid this using Windows settings that I didn’t even know existed until today.

1.) Go to Control Panel -> Network Connections.  Click on the Advanced menu and select Advanced Settings.

2.)  From here, you can set the order in which your network adapters are access by Windows.  Set the order however it suits your needs and click OK.

I felt a little silly for not exploring Windows a little more than I did before uncovering this, but nonetheless I’m very glad to have this new piece of knowledge in my arsenal.  You can bet I’m going to be spending some time later this afternoon checking the rest of the servers that are attached to CommVault and make sure their settings are correct.

(Part 1 and Part 2)

Shortly after services were done on Sunday, I had some downtime while I was waiting on KidStuf to finish up, so I figured “what the heck…may as well try that new NIC out.”

I had gone to Staples the previous day and purchased a Linksys NIC for the video server so that we didn’t have 2 identical cards in the box.  I know it’s a long shot, but I’ve seen weird stuff happen with 2 like-branded printers trying to coexist on the same PC…maybe Intel NICs to the same thing?  I popped the new adapter into the box, fired it up, installed the drivers, set IP addresses, and…..

*insert chirping cricket SFX here*

…still nothing.  The system acts just as it previously did.  Backups still won’t run unless the video server is disconnected from our core network.

So by now you can probably guess that I’m getting frustrated.  I’m pretty good at understanding things, even if I just understand that I don’t know enough about the issue.  But everything I see from this experience tells me that our video and backup servers SHOULD BE TALKING.

1. Both systems reply to pings that are sent on the proper network, and display the proper IP addresses.
2. Both systems can run a successful traceroute, which again, shows the proper IP addresses.
3. A netstat list shows actual ESTABLISHED connections between the 2 machines…there are also TIME_WAIT-status connections present also.
4. Changing NICs out did not affect this issue, so my having 2 PRO 1000/GT cards in the video server is not a problem.
5. Moving the backup network off of its VLAN and onto a separate piece of copper did not resolve the issue either, so switch/VLAN configurations are ruled out.

I had a chance to talk to Ed Buford about this issue (who by the way is not only a gentleman’s gentleman, he’s also a network admin’s gentleman).  His response was the same as mine… “this is just weird.”  It made me feel good to know that I hadn’t missed something stupid in my search.  Out of our conversation, I’ve got some more tricks to try, and since I’m not doing a work night this week, I’ll have a chance to try them first thing this morning:

1. Reinstall the client software.  Is it possible that the CommVault client-side software looks at your installed NICs and binds to them?  I have changed adapters out since the client software was installed.  This won’t take long either.  May as well try it first.
2. Disconnect the backup server from the core network.  This is definitely not my first choice, but I’m ready to try it if we need to.  All backup operations happen on a network that is physically separate from the core (except for the VLAN’ed connection coming from the video box), and we’ve got a KVM in the server rack for when I need to access the backup server.
3. Permanently remove the video server from the core network.  Yet another thing I really don’t want to do, but when it comes down to it, the video server does serve ONLY our editing and playback system for storage purposes.  As long as the the Final Cut rigs and the backup server can talk to it, then it’s ultimately still doing its job.

Another thing I just thought of too… the video server is the ONLY backup client that isn’t a domain member.  Is this a big deal?  In my mind it shouldn’t be, but it’s another thing to explore…

I’ve had 4 cups of coffee and 2 waffles this morning and I’m feeling pretty psyched up.  I want to see this problem go away before the end of this week.

Stay tuned for the outcome.

(Story starts here…)

12:00pm - I took a walk down to the IDF that our video server plugs into and opened the door.  WHOA.  Blinking lights everywhere.  According to the “blinkey light test method,” there appears to be something very wrong here.  BOTH switches in the IDF are completely lit up, and when I checked the second IDF, I found the same thing.  Is it related to the backup server?  Maybe.  It it still an issue?  Absolutely.

12:15pm - After cutting off the links between racks, it’s pretty apparent that our “talker” is connected to the first IDF.  I disconnected the link between the #1 and #2 switches, and bingo… loads of traffic is still pouring in from somewhere on the #2 switch.  Even more curious now, I decided to start disconnecting one link at a time until the switch showed that things were calming down.  As soon as I pulled the first cable, every light on the switch stopped flashing.  What luck.  I plugged it in again, and sure enough, the traffic began to pile up on the switch.  A quick trace showed that the culprit was a wireless access point in one of our venues.

12:40pm - Back at my desk, I attempted to run a backup from our video server with it connected to our primary LAN… still no luck though.  Now it appears we have 2 issues to deal with, though the access point is probably just a configuration issue (one can hope, anyway).  In any case, I’ll deal with it later.

1:20pm - I’ve now reduced the network to only the core switch and the gigabit switch that the video server is connected to.  Still no luck getting backups to run.  This leaves me a little perplexed.  WHAT is causing this miscommunication between the 2 servers.  Furthermore, HOW can the servers ping each other but still not connect?

1:50pm - A trip back to the IDF reveals some interesting stuff.  I am unable to get a link established between an unmanaged switch and the patch to the video server…I’m going to try to move the connection between the backup server and video server to a different physical network, but first I need to be able to get a link.  A look at one of the NICs in the video server reveals the problem:  It was set to a certain link speed instead of auto-negotiation.  While this doesn’t cure the root issue, it does allow me to link the 2 servers together without a VLAN.

2:20 - After putting the video server’s “backup NIC” on a completely different physical network, I am still unable to start a backup stream in Galaxy Express unless I disconnect the server from the primary network.  This pretty much kills my theory about the switch/VLAN configurations being an issue, and also rests the blame back on the video server itself.

So the question now is “what’s the problem with the server?”  It has 2 Intel Pro/1000 GT cards in it.  The IP settings are correct, and the MAC addresses aren’t even close to similar.  A trip to Staples might be in order today to pick up a non-Intel card to see how it functions.  I’ve run into issues before with printers where having two same-brand printers hooked to one machine caused some serious funk.  It this possible with NICs?  For the record, I HOPE that it’s that easy at this point.

So last week I was pretty psyched up about having our video storage server connected to Galaxy Express…so much so that I forgot to monitor a backup job AFTER the server was plugged back in to the primary network.

You can imagine my dismay once I found out that backup jobs from video were failing. The reason? Galaxy Express could not establish a connection back to the video server.

The weird part? If I disconnect the cable connecting the server to the primary network (leaving only the connection VLAN’d to Galaxy Express), then things work just fine. As soon as I plug back into the primary network, Galaxy Express ceases to work, BUT I can still ping either server. VERY weird.

I’ve already tried swapping out NICs on the video server, but to no avail. This tells me that that it probably has to be something in the switching…but how? Everything I see tells me that the VLANs are configured correctly on each switch. And as I said before, pings are going through with no problem, so the connectivity seems to be there.

More on this as the plot unfolds. Possible film at 11.

This past Tuesday I had the opportunity to drive out to Granger Community Church to hang with Jason, Ed, Kyle, and their team of volunteers as we got to see a demonstration of 2 high-powered products from Fluke Networks:  The EtherScope and the OmniView.  Kevin from Fluke did a fantastic job demonstrating what these units are capable of.  The guy knows his stuff.

Jason has a very good write-up on some of the things that we picked up throughout the demo.  You can read the rest yourself, but here’s a small tidbit…

The BIG take away of the night for all of us was the incredible impact bluetooth devices have on access points!  A single bluetooth device in use causes an amazing amount of interference in the .11b/g range.  Kevin said he’s seen as few as 12 bluetooth devices take down an access point!  What?!  We all started turning on our bluetooth phones and doing partner searches and the real-time wifi graphs on the Fluke’s were going nuts!

Just like was everyone else, I was blown away by this.  How in the world has this not been widely publicized?  Did I miss it somewhere?

Are wireless manufacturers going to use this to make .11a and .11n easier to market?  I mean, seriously, was b/g equipment NOT tested against other wireless gear that would be used in like manner?  “What that?  Your b/g wireless products are falling victim to an influx of bluetooth devices?  Here, we’ve got some sweet a and n units that will work MUCH better.”

Sounds like a good forced-upgrade plan.  Maybe not, but it was enough to make my mind slip into conspiracy mode.

The next day I was able to hang with the Jason, Ed and Kyle and basically go through a normal work day with them.  I had my laptop with me, so I VPN’d in to NewPointe and took care of some support issues when we weren’t in discussion about anything.

I finally arrived home around midnight exhausted, but VERY glad to have had the opportunity to hook up with everyone at GCC.  Big thanks to Ed, Kyle, Jason, and their volunteer team for making an Ohio geek feel welcome. :-)  I always leave there wiser than when I show up.

>> I took some time earlier this week and upgraded my Wordpress installation to version 2.2.  I was expecting a possibly a small face lift and some more features, but there really isn’t much to note aside from my theme getting whacked and RSS icons getting plastered all over the top of the header.  I upgraded to the latest version of the theme and that seems to have fixed it, and all my plugins seem to be working as well.  Just for good measure, I added Digg icons at the bottom of the posts.

>> Has anyone else ever dealt with credit card readers on their network?  We’ve got one set up in our cafe area at the moment, and we are getting quite a few read errors during transactions.  It’s a pain because any read error totally voids the transaction and it has to be restarted.  We’ve tested this reader on various locations in our network with the same results (which actually makes me breathe easier).  The only thing I haven’t tried at this point is putting the reader on a separate VLAN, but I haven’t had time to get that far yet.  If anyone has any experience with credit card readers, I’d love to hear any advice you have.

>>  This Thursday the Video Department received some much-needed help!  Mr. Jeff Conn has come aboard for a summer internship and will be working mainly in video post-production.  Jeff has already lent quite a bit to the video team with his editing skills.  It’s going to be a tremendous blessing to have him around!

>>  On Friday I entered into some new territory (for me, anyway):  SSL certificates.  Though it’s a little embarrassing to admit, SSL implementation has been on my to-do list for some time now but had never made it to the front burner.  Between Friday and Saturday, I managed (with some how-to advice from John Dolan) to get a CA set up and get a certificate set up on our Exchange server for OWA and OMA.  OMA wasn’t a huge deal, because we only have 2 Windows Mobile devices checking mail on our server, but it’s still nice to know that it works.  Thanks for your help, John!

Next week my primary goal will be getting SSL-Explorer up and running for our non-notebook-toting staff.  I’ve been having some issues with the community edition, such as the WebDAV URLs not opening correcting on the client end.  Hopefully by the end of the week things can get smoothed out.

Last night was another work night at NewPointe.  Here’s what went down.

–> Data/phone lines were run to the kitchen.  This is funny, because I remember doing the wiring plan thinking When will we ever need phones and data in there?  Well here we are, 6 months into the building and they’re being installed.  It just goes to show you, never say “never” and don’t lay down too many absolutes when planning a network.

–> A 12-drive eSATA array made its way to our Final Cut desk.  Since mid-March, we’ve been making it a point to archive video from our services from the weekend…at least one of them.  The problem comes with the fact that our video system is HD (it looks great, but it’s a double-edged sword).  Recording HD video will stretch your hard drives to the limit.  Compressed HD runs about 60 or so GB per hour (which is great compared to uncompressed HD, which is upwards of about 650GB per hour).

Enter the Norco DS-1220.  It’s got some pretty good reviews on it and so far I’ve really liked what I’ve seen.  We loaded ours up with 12 750GB drives; 10 for the array and 2 for hot spares.  The only drawback I’ve seen with this board is that it comes with a PCI-X eSATA controller, and since our MacPros don’t have PCI-X on them, I ended up buying a HighPoint RR2314 eSATA controller to work with our Mac’s PCI-express.  Other than that, I’m pretty happy with it thus far.

Next Work Night:  It’s gonna be a cable-pulling extravaganza.  We’re putting some much-needed data and phone drops into the control and editing rooms, and also taking care of some AV lines in the process.

There are some days when I come in to the office and get so into a project that I don’t pull myself away from it until I’m near exhaustion.  It doesn’t happen often, but it happened on Wednesday night.

When we put our control room together back in November, we didn’t really have much structure as far as file sharing between the Macs (we run an Apple control room, y’all).  There was no set location for incoming media files to go, and every machine was wide open, always logged on as Admin.  After taking some time to plan out what computers needed access to what, I headed upstairs on Wednesday afternoon to start re-tooling things.  Somewhere between 3:30 and 4:00am Thursday morning, I finally had things working how they needed to be… shares are being auto-mounted at login, the machines automatically log into accounts that don’t have admin privileges, and all the software (Final Cut, Pro Presenter, Keynote) all seem to be working fine.

On Thursday afternoon, we were able to put the new setup to the test.  Cindy and Jane headed to the control room to assemble song lyrics and message graphics.  So far, everything seems to work great!

This is probably the most important thing that happened for me this week.  The more I learn about systems and how they need to function (and I’ve got so much to learn), the more I realize that these systems need to work their best when I’m NOT around!  The Macs in the control room are a huge part of what we do on the weekend, and I’m feeling pretty good that Wednesday night’s project turned out as well as it did.  It was worth every ounce of lost sleep (which I didn’t start to regain until late Thursday afternoon).

I have to admit, I screwed something up.

A few posts back I had written about a supposedly successful P2V conversion on one of our domain controllers.  I want to be the first (hopefully) to say that I was wrong on this.  After further testing, replication simply wasn’t happening between both DCs, even though the event viewer showed that it was so.

As such, I’m probably going to end up building a new DC from scratch on one of our VMware hosts.  We haven’t got any servers set for deployment at this time, so this will be an opportunity to sysprep a Server 2003 installation and store it off to the side for later.

I mat take another crack at P2V-ing a domain controller in the future, but right now there are bigger fish to fry.