A novel approach to data center staffing

Equipment failures in data center environments never happen at opportune times.  For example, your UPS isn’t likely to fail at 2:00 in the afternoon on a Tuesday when everyone you need to respond to the causality is conveniently in the building.  Unfortunately, Murphy’s Law demands that failures will occur at the worst possible time.  Calamity invariably strikes when critical personnel are vacationing, snug in their beds or hopelessly bogged down in cross town traffic. 

This unfortunate (and statistically improbable) truth is one of the reasons the Uptime Institute’s Tier Standard: Operational Sustainability requires 7×24 staffing for higher Tier Level data centers.  In order to achieve true high uptime, personnel often need to take immediate, appropriate action.  Having key personnel respond to alarms by getting in their cars and driving to the facility just doesn’t cut it and is not a recipe for maximum uptime.

True story; a single-shift data center I worked with was struck by a midnight critical power failure during a severe ice storm.  The facility manager tried to drive to the data center but ended up sliding off an icy road and into a ditch.  The brave facility manager tried to talk a security guard through transferring critical power to an alternate source via cell phone as paramedics loaded him into the back of an ambulance.  Needless to say, the facility was dark for a painfully long time on that night.

Unfortunately, around the clock data center staffing is an expensive proposition.  Most staffing models require at least seven people to safely staff a 7×24 data center.  (With a staff of seven you will still have coverage gaps related to vacations and sick days.)  When you factor burden rates for seven well trained data center engineers, 7×24 staffing can easily exceed $1M/year.

Many data centers find these costs prohibitively high and look for ways to have key personnel handy at a reasonable price.  For example, the data center that experienced the ice storm now puts key personnel in nearby hotels when there’s a threat of inclement weather.  Since only a portion of data center failures are weather related, this strategy is a half step at best.

A few data centers in Europe have found a novel way to tackle the issues of having critical personnel around when they are needed.  London based data centers are anticipating a perfect storm of high data usage, paralyzing gridlock and unprecedented strain on electrical transmission infrastructure during the upcoming Olympics.  In order to ensure that key personnel are not stranded opposite a wall of tourists and sports fans, colocation provider Interexion has purchased a number of sleep pods and placed them in the data center.

Photo: Interexion

Interexion is not alone in adopting this strategy.  PodTime, the manufacturer of the sleeping pods is reporting that they have sold 19 units to 3 colocation providers since February of this year.     

These pods look pretty comfy to me.  However, I spent months on end sleeping in considerably closer quarters aboard US Navy submarines.  What I’m wondering is where these guys clean up.  Fortunately, there are probably emergency safety showers in the battery room.  Seems reasonable to me.  But then, I’ve showered under considerably worse conditions too.  ;0)  Would you sleep in a pod for your data center?

Advertisements

One Response to A novel approach to data center staffing

  1. Well… I would prefer to have a typical night shift. I cannot imagine me sleeping at work. There is always a lot of things to be done, when you’re at work.

    These pods… I was sleeping in worse conditions too – being an amateur but serious sailor – in claustrophobic cabins, in wet clothes, cold everywhere, wind blowing outside, yacht dancing from wave to wave… Alright! On the sea? – It’s ok. That’s why we’re going there! Such conditions are comfortable enough and you have to rest before your watch somewhere, but on the land? Come on! Isn’t it better to work on typical shifts and use your night shifts for the work you cannot do during the day, when there is a lot of traffic – night shifts are great for the things like documentation updates, writing procedures, analysing the designs, calculating data for optimizations, configuring distributed monitoring systems (if there is some distributed infrastructure to be monitored expect the local facility), etc.?

    The pods are acceptable on some specific occasions, but… Definitely not in the server room, as presented on the photo. This should be definitely a kind of dedicated room. Could you imagine sleeping in a pod placed in the middle of the engine room on a vessel? What about the noise of IT equipment and HVACs? No. It’s too loud.

    And honestly… All the time I see data center as a simplified vessel metaphor. To some extent, it’s like working in the engineering department of the ship crew, except the fact you’re close to your home and on the land all the time. Many procedures and typical operations are so similar in both cases.

    These pods, somehow, remind me… the cabins. Maybe short contracts in remote locations could be reasonable choice for data center operators? Let’s say – 6 months on board the data center, then coming back home for 3 months to see you family. Or – maybe – any other crew rotation we knew form shipping industry? ;-)

    Of course, I’m joking, but the news is really inspiring my imagination. BTW. Great story about brave data center facility manager.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: