12% Chance of Grid Crippling Solar Storm in Next 10 Years

Today another stunning Coronal Mass Ejection (CME) erupted from the Sun’s corona.  Fortunately, the massive flare was not directed at the Earth. 

A CME like this sends a cannonball of superheated plasma, energetic particles and radiation barreling through space.  If the Earth is unlucky enough to be in the path of one of these monsters the consequences can be dire.  Notable effects can include spectacular auroras, widespread communication failures and massive blackouts.

In 1859 a large solar storm struck the earth in an event known as the Carrington Event.  The Carrington flare caused telegraph stations to burst into flame due to induced currents in the copper transmission lines.  Auroras caused by the event were seen as far south as Havana. 

In 1921, another huge solar storm and geomagnetic event struck the Earth.  John Kappenmann, from the Metatech Corporation (and co-author of the NASA/NRC report on the economic impact of space weather), modeled the 1921 storm on the modern power grid.  The map of power system collapse and utility transformer damage is a stark warning of the threat of space weather to power distribution systems. 

 

More recently, in 1989 a large flare caused the collapse of the Hydro-Quebec power grid.

In 2008 solar scientists predicted that a Carrington scale solar event today could cause blackouts effecting 130 million people and result in economic losses of “$1 trillion to $2 trillion during the first year alone…with recovery times of 4 to 10 years.”

So, the consequences of the Earth being struck by a massive solar storm are potentially severe.  But such an event can surely be classified as a High Impact/ Low Probability/ (HILP) event, Right?  Not so fast my friend.  Pat Riley, a predictive science expert, just published a paper in the scientific journal, SPACE WEATHER: THE INTERNATIONAL JOURNAL OF RESEARCH AND APPLICATIONS in February of 2012.  In his paper Riley places the “Probability of a Carrington event occurring over next decade is ~12%”

Around 12%!  Yikes! Those Doomsday Preppers on NatGeo may be on to something.

I outlined the near earth and terrestrial effects of a massive solar flare/geomagnetic storm in my blog here.

I covered the potential effects of space weather on the infrastructure that supports data centers for Mission Critical Magazine here

A novel approach to data center staffing

Equipment failures in data center environments never happen at opportune times.  For example, your UPS isn’t likely to fail at 2:00 in the afternoon on a Tuesday when everyone you need to respond to the causality is conveniently in the building.  Unfortunately, Murphy’s Law demands that failures will occur at the worst possible time.  Calamity invariably strikes when critical personnel are vacationing, snug in their beds or hopelessly bogged down in cross town traffic. 

This unfortunate (and statistically improbable) truth is one of the reasons the Uptime Institute’s Tier Standard: Operational Sustainability requires 7×24 staffing for higher Tier Level data centers.  In order to achieve true high uptime, personnel often need to take immediate, appropriate action.  Having key personnel respond to alarms by getting in their cars and driving to the facility just doesn’t cut it and is not a recipe for maximum uptime.

True story; a single-shift data center I worked with was struck by a midnight critical power failure during a severe ice storm.  The facility manager tried to drive to the data center but ended up sliding off an icy road and into a ditch.  The brave facility manager tried to talk a security guard through transferring critical power to an alternate source via cell phone as paramedics loaded him into the back of an ambulance.  Needless to say, the facility was dark for a painfully long time on that night.

Unfortunately, around the clock data center staffing is an expensive proposition.  Most staffing models require at least seven people to safely staff a 7×24 data center.  (With a staff of seven you will still have coverage gaps related to vacations and sick days.)  When you factor burden rates for seven well trained data center engineers, 7×24 staffing can easily exceed $1M/year.

Many data centers find these costs prohibitively high and look for ways to have key personnel handy at a reasonable price.  For example, the data center that experienced the ice storm now puts key personnel in nearby hotels when there’s a threat of inclement weather.  Since only a portion of data center failures are weather related, this strategy is a half step at best.

A few data centers in Europe have found a novel way to tackle the issues of having critical personnel around when they are needed.  London based data centers are anticipating a perfect storm of high data usage, paralyzing gridlock and unprecedented strain on electrical transmission infrastructure during the upcoming Olympics.  In order to ensure that key personnel are not stranded opposite a wall of tourists and sports fans, colocation provider Interexion has purchased a number of sleep pods and placed them in the data center.

Photo: Interexion

Interexion is not alone in adopting this strategy.  PodTime, the manufacturer of the sleeping pods is reporting that they have sold 19 units to 3 colocation providers since February of this year.     

These pods look pretty comfy to me.  However, I spent months on end sleeping in considerably closer quarters aboard US Navy submarines.  What I’m wondering is where these guys clean up.  Fortunately, there are probably emergency safety showers in the battery room.  Seems reasonable to me.  But then, I’ve showered under considerably worse conditions too.  ;0)  Would you sleep in a pod for your data center?

AFCOM Presentation is online

My presentation, Cybersecurity for Mission Critical Infrastructure from 2012 AFCOM Data Center World is posted on Slideshare.  Despite the fact that I had the last speaking slot of the the conference, I had a terrific turnout.  Thanks to everyone that stuck around to hear it!

Here’s a link to the presentation slide deck

Also had some very good conversations with some of the attendees following my presentation.

Thanks to everyone who attended!