Andrew Pollack's Blog

Technology, Family, Entertainment, Politics, and Random Noise

Data Center explosion in Houston takes down 9000 servers - what happened, and what you should consider about disaster recovery

By Andrew Pollack on 06/03/2008 at 08:10 AM EDT

Having been impacted by the outage at The Planet, obviously I followed it closely. Through many telephone calls and chats with staff there and with people I know personally in the area with professional relationships on staff at The Planet, here's the best take I can give you on what really happened.

What Happened?

At about 17:45 CST on a Saturday, a fire and explosion took down an entire data center in Houston, TX belonging to hosting provider "The Planet". The state of the art facility manages more than 9000 servers in racks on two floors of the huge data center. The facility is known for extremely high reliability and high performance network connectivity to the internet. As such, a large number of its customers ignored best practices and did not have hot standby servers in other data centers. Although the facility has standby generators capable of instantly providing enough power to sustain the facility indefinitely and the fuel for such use to last well over a week, they were unable to use these generators due to the nature of the incident.


Transformer Explosion


The initial incident was an explosion and fire that required (obviously) the immediate evacuation of the entire facility. Initial reports have been that a transformer exploded in their power distribution room. It may be that it was in fact an electrical conduit explosion, and some have suggested that the initial explosion may have caused a burst in a pressurized fire suppression system which did more of the damage. It will be some time before we know for sure what the initial explosion was. We do know, however, that the force of the blast blew out three interior walls of the power distribution room - moving those walls "several feet" from their original position.

Power distribution rooms in large data centers are like the circuit breaker panel at your house, only instead of distributing 20-50 kilowatts to a dozen or so circuits, a data center can be dealing with several megawatts and hundreds or thousands of circuits. Clearly this couldn't have been an explosion of a megawatt sized transformer as that would have left a crater where that half of the building used to be. It could have been one of several smaller transformers or it could more likely have been a conduit explosion.

The damage from the explosion destroyed the connections of nearly every circuit from the distribution room out to the racks of equipment, the cooling equipment, and everything else. Had they started the generators, they'd only have created a dangerous fire.

Even with dozens of electricians at work and vendors from the power company to the networking gear having offices already on site, it was nearly 28 hours before the electrical connections could be made safe enough and enough equipment replaced for the fire marshal to allow them to begin restoring power. Initially, power was restored to the second floor housing 2/3 of the servers. We're talking about Houston in the summer time, so the area first had to be slowly cooled to safe temperatures without cooling too quickly and creating a condensation problem. Racks could then be started in small groups. Each rack drawing its maximum load as servers restarted and rack mounted batteries (designed to carry gear through a transition to generator power) all demanded a full charge at once.

By mid-day on Monday, most of the second floor was powered up. The Planet had brought teams of support people from their Dallas and other Houston data centers to help with the work. Many servers hadn't been restarted in months or years. Some small percent did not start. Drives that had been spinning all that time didn't have strong enough motors to spin up. Configurations that had changed over the months since the last reboot failed to start properly. All of these issues were handled as best they could.

The first floor servers simply had no connection any longer to the power distribution center. Their power conduits were in the concrete floor of the building, and were simply no longer even near the walls that used to hold their connective cables to the distribution room. By the afternoon on Monday, temporary circuits had been created allowing huge cables from out door generators to run directly into the building and connect to the first floor circuits. This is the way it is currently running, and will continue for at least a week while equipment is brought in and the production grid is rebuilt from scratch.

Some Lessons Reinforced --

I would say lessons learned, but anyone in our business should know these already.

1. Any data center, no matter how good, is subject to this kind of rare incident. A data center can be blown up, a plane can fall on one, a meteor could hit one, a sinkhole could swallow one, or flying monkeys could carry one away. I have little sympathy for people screaming at tech support staff about losing thousands of dollars an hour. If the up time is that critical, you should have standby in another data center -- possibly with another vendor entirely. For me, I've already got the fail over machine in place now, and am building it out and configuring today. A bit late, but it can happen again. Especially since right now there is a lot of temporary patchwork in that data center and a lot of mitigation work to be done over the coming weeks.

2. Don't delay your disaster recovery plans. I got caught with my pants down, as I'd been planning a hot standby server in another data center and was months over due setting one up. I own that failure, not The Planet. They do own their own mistake however. The planet had acquired another outfit and was still using the DNS setup that the older firm had used. This left them without a backup DNS system in another data center themselves for those customers. This, combined with the customers themselves not getting their own backup dns providers (which is free in some places) left some customers in other data centers without service. I think that is the only part of this incident where The Planet holds fault. They knew about the issue, but hadn't completed their plans to better redistribute that configuration. Like me, they'd put off what they knew they had to do and they got caught.

3. Don't delay implementing your disaster recovery plans in hopes that you'll be up and running before you could complete the process. If your process takes a long time to change over, you should start right away. If something does happen to bring the primary service back, that's good. In the mean time, the sooner you start the better off you are. Often with these things, it is many hours into the incident before enough facts are known and verified to give an accurate time estimate.


There are  - loading -  comments....



Other Recent Stories...

  1. 05/05/2016Is the growing social-sourced economy the modern back door into socialism?Is the growing social-sourced economy the modern back door into socialism? I read a really insightful post a couple of days ago that suggested the use of social network funding sites like “Go Fund Me” and “Kickstarter” have come about and gained popularity in part because the existing economy in no longer serving its purpose for anyone who isn’t already wealthy. Have the traditional ways to get new ventures funded become closed to all but a few who aren’t already connected to them and so onerous as to make ...... 
  2. 04/20/2016Want to be whitelisted? Here are some sensible rules for web site advertisingAn increasing number of websites are now detecting when users have ad-blocking enabled, and refuse to show content unless you "whitelist" their site (disable your ad-blocking for them). I think that is a fair decision on their part, it's how they pay for the site. However, if you want me (and many others) to white list your site, there are some rules you should follow. If you violate these rules, I won't whitelist your site, I'll just find content elsewhere. 1. The total space taken up by advertisements ...... 
  3. 12/30/2015Fantastic new series on Syfy called “The Expanse” – for people who love traditional science fiction[] “The Expanse” is a new science fiction series being broadcast onthe Syfy channelthis winter. It’s closely based on a series of books by author James S. A. Corey beginning with “Leviathan Wakes”. There are 5 books in the “Expanse” series so far. If you’re a fan of the novels you’ll appreciate how closely the books are followed.TIP: The first five episodes are already available on Syfy.com. If you’re having trouble getting into the characters and plot, use those to get up to speed.The worlds created for ...... 
  4. 10/20/2015My suggestion is to stay away from PayAnywhere(dot)com  
  5. 08/07/2015Here is one for you VMWARE gurus - particularly if you run ESXi without fancy drive arrays 
  6. 08/06/2015The Killer of Orphans (Orphan Documents) 
  7. 06/02/2015Homeopathic Marketing: Traveler on my Android is now calling itself VERSE. Allow me to translate that for the IBM Notes community... 
  8. 03/17/2015A review of British Airways Premium Economy Service – How to destroy customer goodwill all at once 
  9. 02/26/2015There's a bug in how @TextToTime() and @ToTime() process date strings related to international standards and browser settings. 
  10. 01/21/2015Delivering two new presentations at Developer Camp (EntwicklerCamp) 2015 in Germany 
Click here for more articles.....


pen icon Comment Entry
Subject
Your Name
Homepage
*Your Email
* Your email address is required, but not displayed.
 
Your thoughts....
 
Remember Me  

Please wait while your document is saved.