This past Saturday night I was able to get a script working on our servers that would take them all down gracefully in the event of a power outage.  This was in response to an previous blunder on my part that had allowed every last server in our building to go down hard (although nothing was damaged in the event).

Fast forward to this Tuesday (earlier this week).  Our area got rocked by a couple of huge storms going through the area.  About 3/4 of the way through the first storm, the lights in the office flickered, dimmed, and finally went out.  I didn’t think the UPS script would get tested this quickly.

When I flipped open the KVM on our server rack, I was pleased to see that a countdown was already running on each machine.  The UPS software had run its script, and now each server was about 60 seconds away from shutting down automatically.  When it was all said and done, every last server shut down on its own, with plenty of battery life to spare.

The only tweak I ended up making to this process so far involved our physical domain controller (we have two, and the other one is a VM).  It resides in one of the IDFs and it shut down too quickly in response to the power outage.  As a result, after the VM-based DC went down, the remaining servers had no DC to talk to, and thus took a longer time to shut down cleanly.  All-in-all though, the real-life test proved successful, and as a result.  I have one more reason to sleep better at night.