April 13, 2012 1 Comment
Equipment failures in data center environments never happen at opportune times. For example, your UPS isn’t likely to fail at 2:00 in the afternoon on a Tuesday when everyone you need to respond to the causality is conveniently in the building. Unfortunately, Murphy’s Law demands that failures will occur at the worst possible time. Calamity invariably strikes when critical personnel are vacationing, snug in their beds or hopelessly bogged down in cross town traffic.
This unfortunate (and statistically improbable) truth is one of the reasons the Uptime Institute’s Tier Standard: Operational Sustainability requires 7×24 staffing for higher Tier Level data centers. In order to achieve true high uptime, personnel often need to take immediate, appropriate action. Having key personnel respond to alarms by getting in their cars and driving to the facility just doesn’t cut it and is not a recipe for maximum uptime.
True story; a single-shift data center I worked with was struck by a midnight critical power failure during a severe ice storm. The facility manager tried to drive to the data center but ended up sliding off an icy road and into a ditch. The brave facility manager tried to talk a security guard through transferring critical power to an alternate source via cell phone as paramedics loaded him into the back of an ambulance. Needless to say, the facility was dark for a painfully long time on that night.
Unfortunately, around the clock data center staffing is an expensive proposition. Most staffing models require at least seven people to safely staff a 7×24 data center. (With a staff of seven you will still have coverage gaps related to vacations and sick days.) When you factor burden rates for seven well trained data center engineers, 7×24 staffing can easily exceed $1M/year.
Many data centers find these costs prohibitively high and look for ways to have key personnel handy at a reasonable price. For example, the data center that experienced the ice storm now puts key personnel in nearby hotels when there’s a threat of inclement weather. Since only a portion of data center failures are weather related, this strategy is a half step at best.
A few data centers in Europe have found a novel way to tackle the issues of having critical personnel around when they are needed. London based data centers are anticipating a perfect storm of high data usage, paralyzing gridlock and unprecedented strain on electrical transmission infrastructure during the upcoming Olympics. In order to ensure that key personnel are not stranded opposite a wall of tourists and sports fans, colocation provider Interexion has purchased a number of sleep pods and placed them in the data center.
Interexion is not alone in adopting this strategy. PodTime, the manufacturer of the sleeping pods is reporting that they have sold 19 units to 3 colocation providers since February of this year.
These pods look pretty comfy to me. However, I spent months on end sleeping in considerably closer quarters aboard US Navy submarines. What I’m wondering is where these guys clean up. Fortunately, there are probably emergency safety showers in the battery room. Seems reasonable to me. But then, I’ve showered under considerably worse conditions too. ;0) Would you sleep in a pod for your data center?