I remember trying to burn CD-Rs back in the day when 1x burners were still thousands of dollars and there was a club underneath the office doing a sound check at the same time. The vibrations would toast every burn, which was annoying since it took about forty minutes to burn each one.
Shades of trying to do laser holography where everything has to remain perfectly inert for seconds on end.
I don't think of the DCs I've seen have breathing apparatus around - they rely on strictly enforced signin sheets (so when the alarm goes off, a person outside the room immediately looks to see how many people are in the DC), and a 30-45 second delay to evacuate the DC before the suppression system activates.
Halon is not too toxic and you only need a concentration of less than 10 % for it to be effective. And I even once read that Halon helps breathing, i.e. in the presence of Halon humans can survive at a lower oxygen concentration than they normally could but I am unable to find a source right now. All in all it is probably not to bad to be in a room flooded with Halon and the fumes of burning stuff are probably the larger hazard.
Halon is extremely dangerous and most datacentres built in the last twenty years avoid using it for that reason.
The last halon datacentre I was in had breathing apparatus within easy reach no matter where you were standing in the place. Two paces, at most, to grab a mask. You do have seconds if it goes off, and that mask is only to give you enough time to run out of the room.
The newer stuff is less hostile to organic life, so there's no reason for a respirator.
Not halon but there are fire supessant gas mixtures (e.g. Inergen(R)).
These gases have a interesting physiological effects that increases the breathing frequency and therefore allow surviving in a flooded room.
Everything in the article sounds like trigger-happy speculation at best. Their equipment was brought down during a fire drill, but reading the article, it doesn't seem like anyone actually says or knows why.
According to people familiar with the system, the
pressure at ING Bank's data center was higher than
expected, and produced a loud sound when rapidly
expelled through tiny holes.
The bank monitored the sound and it was very loud,
a source familiar with the system told us. “It was
as high as their equipment could monitor, over
130dB”.
Sound means vibration, and this is what damaged the
hard drives.
Those are the words of the author, in that last sentence. It's simply journalism at this point. It's not something worth construing as a technical assessment.
They (the staff at Vice) just want to scoop the article, and get Vice into the action. Was it a siren that was too loud? Was it pressure differential, triggering a head crash in a bernoulli box?
Since we're at least two degrees removed from the actual events, and we'll probably never get direct information from a postmortem report, this article, to me, reads as: Data Center Outage in Eastern Europe, Reason Unknown
Risking an ignorant question - will this accelerate adoption of SSDs? Or is there something about data centers that makes them lean toward spinners?
The issue in the article is that as spinners have become more dense, the track tolerance has decreased. Those small tolerance high density disks must already be expensive, and if they further can't reliably survive the environment (which includes fire suppression), then replacement with SSDs starts to look mandatory on the near horizon.
If you're going to believe every rumor you read you'll be convinced of anything. SSD devices are very durable except if they're subjected to repeated and extreme levels of heating/cooling.
Also I'm not sure what datacentres you work in but I've never heard of one being turned off for six months. You can spin down drives when they're not in use, but that's mostly to save energy. An idle SSD uses almost no power, there's no reason to spin it down.
No you wouldn't switch an in use disk of in a data centre; but you would for archival cold storage; such as legal evidence: (From: https://blog.korelogic.com/blog/2015/03/24 )
> Digital evidence storage for legal matters is a common practice. As the use of Solid State Drives (SSD) in consumer and enterprise computers has increased, so too has the number of SSDs in storage increased. When most, if not all, of the drives in storage were mechanical, there was little chance of silent data corruption as long as the environment in the storage enclosure maintained reasonable thresholds. The same is not true for SSDs.
...
> For client application SSDs, the powered-off retention period standard is one year while enterprise application SSDs have a powered-off retention period of three months. These retention periods can vary greatly depending on the temperature of the storage area that houses SSDs.
...
> The standards change dramatically when you consider JEDEC's standards for enterprise class drives. The storage standard for this class of drive at the same operating temperature as the consumer class drive drops from 2 years under optimal conditions to 20 weeks. Five degrees of temperature rise in the storage environment drops the data retention period to 10 weeks. Overall, JEDEC lists a 3-month period of data retention as the standard for enterprise class drives.
> A check of various drive manufacturers, in this case Samsung, Intel, and Seagate, shows that their ratings for data retention of their consumer class drives are what would be expected for JEDEC's enterprise class drive standards. All three quote a nominal 3-month retention time period. Most likely, the manufacturers are being conservative; however, it demonstrates the potential variability the manufacturers associate with data retention on any SSD in storage.
As SSD explodes in capacity and HDD struggles to keep up look for "archival grade SSD" to emerge as a serious product.
I know there's a ridiculous amount of offline data that must be maintained and tapes aren't always a practical solution.
Manufacturers are being very conservative when it comes to retention times. I haven't heard of anyone losing data because they left their SSD powered off for two long, but if you have any anecdotes or reports to share, by all means.
Early SSDs were temperamental, flaky, and would burn out quickly. The current generation is durable, almost impossible to burn out, and seems to hold data for years even when powered off. It's like the bad rap that plasma screens got for burning in even when that problem was addressed by the manufacturers.
> The site is currently offline and the bank relies solely on its backup data center, located within a couple of miles’ proximity.
> “Moreover, to ensure full integrity of the data, we’ve made an additional copy of our database before restoring the system,” ING’s press release reads.
What? Am I to interpret this to mean that ING has a single backup of its data under normal conditions?
While being directly affected by the outage (could not make card payments on Saturday), I have noticed that none of the communications team was clearly aware of the situation.
Here are some facts:
- the first public announcements - FB news and tweet confirming the issue have been posted at ~17.25-1735 local time, after four and a half hours
- there was a lot of wasted energy on excuses and social-media damage control, while no clear explanation and status being posted
- at 17.41 there was a FB response stating that "one unfunctional server" is the cause of the downtime; all services were down at that time
- the corporate communication director declares for press "we know about the issue, we don't know the cause of it"
- after almost 11 hours, problem is solved
Now the nice part: next day, the bank issues a press release stating that malfunction appeared after a programmed [fire suppression] test.
While accidents can happen, this looks more like "fail to plan is plan to fail" issue, as well as very bad communication.
38 comments
[ 4.5 ms ] story [ 72.0 ms ] threadhttp://www.brendangregg.com/linuxperf.html
Shades of trying to do laser holography where everything has to remain perfectly inert for seconds on end.
Such a shame that Sun was acquired by Oracle.
For many reasons, but what makes you mention it here; what's the relevance?
The last halon datacentre I was in had breathing apparatus within easy reach no matter where you were standing in the place. Two paces, at most, to grab a mask. You do have seconds if it goes off, and that mask is only to give you enough time to run out of the room.
The newer stuff is less hostile to organic life, so there's no reason for a respirator.
http://webcache.googleusercontent.com/search?q=cache%3Awww.i...
If so it's quite a big co-incidence, and goes to show how hard it is to mitigate against every event.
They (the staff at Vice) just want to scoop the article, and get Vice into the action. Was it a siren that was too loud? Was it pressure differential, triggering a head crash in a bernoulli box?
Since we're at least two degrees removed from the actual events, and we'll probably never get direct information from a postmortem report, this article, to me, reads as: Data Center Outage in Eastern Europe, Reason Unknown
The irony is that these systems are maintained by IBM :)
The issue in the article is that as spinners have become more dense, the track tolerance has decreased. Those small tolerance high density disks must already be expensive, and if they further can't reliably survive the environment (which includes fire suppression), then replacement with SSDs starts to look mandatory on the near horizon.
It's only a matter of time before SSD prices cut below HDD and all that spinning rust is thrown in the garbage.
Also I'm not sure what datacentres you work in but I've never heard of one being turned off for six months. You can spin down drives when they're not in use, but that's mostly to save energy. An idle SSD uses almost no power, there's no reason to spin it down.
> Digital evidence storage for legal matters is a common practice. As the use of Solid State Drives (SSD) in consumer and enterprise computers has increased, so too has the number of SSDs in storage increased. When most, if not all, of the drives in storage were mechanical, there was little chance of silent data corruption as long as the environment in the storage enclosure maintained reasonable thresholds. The same is not true for SSDs.
...
> For client application SSDs, the powered-off retention period standard is one year while enterprise application SSDs have a powered-off retention period of three months. These retention periods can vary greatly depending on the temperature of the storage area that houses SSDs.
...
> The standards change dramatically when you consider JEDEC's standards for enterprise class drives. The storage standard for this class of drive at the same operating temperature as the consumer class drive drops from 2 years under optimal conditions to 20 weeks. Five degrees of temperature rise in the storage environment drops the data retention period to 10 weeks. Overall, JEDEC lists a 3-month period of data retention as the standard for enterprise class drives.
> A check of various drive manufacturers, in this case Samsung, Intel, and Seagate, shows that their ratings for data retention of their consumer class drives are what would be expected for JEDEC's enterprise class drive standards. All three quote a nominal 3-month retention time period. Most likely, the manufacturers are being conservative; however, it demonstrates the potential variability the manufacturers associate with data retention on any SSD in storage.
I know there's a ridiculous amount of offline data that must be maintained and tapes aren't always a practical solution.
Manufacturers are being very conservative when it comes to retention times. I haven't heard of anyone losing data because they left their SSD powered off for two long, but if you have any anecdotes or reports to share, by all means.
Early SSDs were temperamental, flaky, and would burn out quickly. The current generation is durable, almost impossible to burn out, and seems to hold data for years even when powered off. It's like the bad rap that plasma screens got for burning in even when that problem was addressed by the manufacturers.
https://m.youtube.com/watch?v=tDacjrSCeq4
> “Moreover, to ensure full integrity of the data, we’ve made an additional copy of our database before restoring the system,” ING’s press release reads.
What? Am I to interpret this to mean that ING has a single backup of its data under normal conditions?
Now the nice part: next day, the bank issues a press release stating that malfunction appeared after a programmed [fire suppression] test.
While accidents can happen, this looks more like "fail to plan is plan to fail" issue, as well as very bad communication.