ING Bank’s main data center in Bucharest, Romania, was severely damaged over the weekend during a fire extinguishing test. In what is a very rare but known phenomenon, it was the loud sound of inert gas being released that destroyed dozens of hard drives. The site is currently offline and the bank relies solely on its backup data center, located within a couple of miles’ proximity.
“The drill went as designed, but we had collateral damage”, ING’s spokeswoman in Romania told me, confirming the inert gas issue. Local clients were unable to use debit cards and to perform online banking operations on Saturday between 1PM and 11PM because of the test. “Our team is investigating the incident,” she said.
The purpose of the drill was to see how the data center’s fire suppression system worked. Data centers typically rely on inert gas to protect the equipment in the event of a fire, as the substance does not chemically damage electronics, and the gas only slightly decreases the temperature within the data center.
The gas is stored in cylinders, and is released at high velocity out of nozzles uniformly spread across the data center. According to people familiar with the system, the pressure at ING Bank’s data center was higher than expected, and produced a loud sound when rapidly expelled through tiny holes (think about the noise a steam engine releases).
The bank monitored the sound and it was very loud, a source familiar with the system told us. “It was as high as their equipment could monitor, over 130dB”.
Sound means vibration, and this is what damaged the hard drives. The HDD cases started to vibrate, and the vibration was transmitted to the read/write heads, causing them to go off the data tracks.
“The inert gas deployment procedure has severely and surprisingly affected several servers and our storage equipment,” ING said in a press release.
There is still very little known about how sound can cause hard drive failure. One of the first such experiments was made by engineer Brendan Gregg, in 2008, while he was working for Sun’s Fishworks team. He recorded a video in which he explains how shouting in a data center can result in hard drives malfunction.
In ING Bank’s case, it was “like putting a storage system next to a [running] jet engine,” a source told me.
Researchers at IBM are also investigating data center sound-related inert gas issues. “[T]he HDD can tolerate less than 1/1,000,000 of an inch offset from the center of the data track—any more than that will halt reads and writes”, experts Brian P. Rawson and Kent C. Green wrote in a paper. “Early disk storage had much greater spacing between data tracks because they held less data, which is a likely reason why this issue was not apparent until recently.”
Siemens also published a white paper a year ago saying that its tests show that “excessive noise can have a negative impact on HDD performance”. Researchers said this negative impact may even begin at levels below 110dB.
“It can now be established with a high degree of certainty that the faults in storage systems as a result of an inert gas extinguishing systems discharge were caused by the impact of high noise levels on the hard disk drives,” according to Siemens.
The Bank said it required 10 hours to restart its operation due to the magnitude and the complexity of the damage. A cold start of the systems in the disaster recovery site was needed. “Moreover, to ensure full integrity of the data, we’ve made an additional copy of our database before restoring the system,” ING’s press release reads.
Over the next few weeks, every single piece of equipment will need to be assessed. ING Bank’s main data center is compromised “for the most part”, a source told us.
See our news story regarding Gas Drops and Air Shock effects here: Gas Drops and Air Shock
R3 Data Recovery is an excellent emergency data recovery service that I would have no hesitation in recommending to anyone.
From the initial telephone conversation I had with them I was extremely impressed by their implicit understanding and sensitive handling of my stress and panic, believing I had possibly lost 10 year’s worth of invaluable documentation, imagery and archiving from my highly respected research held on a USB stick.
They worked efficiently and conscientiously to retrieve data urgently for me and made provision for me to receive it and all in time for an important PP presentation, the success of which was crucial to my career.
They were mindful in keeping me informed of their progress with the recovery process, hence putting my mind at rest, and they accomplished this all within a working day/evening. They worked hard and consistently until the job was completed successfully and made themselves available for me to talk to during out-of-work hours. They transferred all the remaining data that was not urgent onto a USB stick and returned it in the post to me, thus very satisfactorily completing an exemplary and outstanding service.
Adrian Howells (Performance Artist and Honorary Research Fellow ) - University of Glasgow
R3 Data Recovery is rated (4.6) by 294 customer reviews on Reviews.co.uk
If your hard drive happens to fail, contact our team of experts. Our skilled professionals will be able to reliably recover data on your hard drive and provide industry-leading turnaround times which give you the confidence that's necessary to develop a solid, effective recovery plan.
Call us today at 0800 999 3282 or fill out our FREE, no-obligation diagnostics form for immediate help!