Too Hot to Trot

This is a good reminder that the best backup systems can be thwarted by rank stupidity.

Some time back there was a major disaster at The State’s Technology Services Division. Details were sketchy but for two full days, employees of The State were unable to logon to their computers or access email, and that this caused business within The State to grind to a halt. This started when employees of The State came in to work following a three day weekend, they found their workstations overloaded with “cannot logon” and “Exchange communication” error messages. The server room was blistering at 109 degrees Fahrenheit and was lying with dead and dying servers. The initial guesses for the disaster was the failure of primary, secondary and tertiary air conditioners all at once. But after cycling the power the A/C’s were working fine. The employees worked day day and night to order new equipment, build new servers and get everything on track. After hundreds of overtime hours and 200 thousand dollars worth equipment they managed to bring everything back online. When the Exchange servers were finally restored, the following email finally made its way to everyone’s inbox, conveniently answering the “why”

1. Nothing like a little unattributed, vague, unconfirmable urban-legend to get people riled up about - spin the wheel for a topic - those who worry about being “eco-friendly.”

   Not to say that there aren’t clueless people out there, but even if this story were true - and it’s probably not, as you can find variants - the story describes someone who’s stolen a keycard and used it to gain unauthorized entry to part of the building when no one was around.  A server room that size should have a temp alarm, no?  Or when a server fails, it pages a tech?  Or that the average clueless would be able to identify and operate the proper on/off controls for the supposedly triple-redundant A/C?  And that doing so didn’t trip an alarm?

   Want to consider something slightly more relevant?  Why do our Senate and Assembly only retain email backups going back 28 days?  I’ve seen small cities with better rentention policies on their Exchange servers, like installing software that doesn’t really let people delete emails they’ve sent or received.

   Posted by John Foust on July 19, 2007 at 2038 hrs




2. I didn’t even think of the “eco-friendly” angle.  I’ll look closer next time.  I was just thinking of all the stupid things I’ve seen people do to their IT environments.  Heck, I’ve seen people unplug servers because they needs an outlet for the vacuum cleaner.

   Posted by Owen on July 19, 2007 at 2040 hrs




3. Not to mention that when the servers started to fail someone would have been paged by the system.

   Posted by on July 20, 2007 at 0840 hrs




4. Funny, but looks like a whole lot of urban legend to me

   Posted by on July 20, 2007 at 0934 hrs




5. I too am skeptical but for more nebulous reasons.

   Not every organization (not to mention IT worker) that does IT follows best practices. I could probably stroll around cubeville in the office I work at and find a couple of yellow sticky notes with ids & passwords to production boxes and I have not yet left my section of IT!

   Also, an article I saw that talked about bad things IT people do is to neglect the testing of failure modes. Almost everyone is good about testing the applications/systems to make sure the sunny day processing works. However, it is always tempting to not test failure modes.

   Posted by Marcus Aurelius on July 20, 2007 at 1416 hrs