> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.
This is 100% fault of the hotel, not of Microsoft. It’s unacceptable that the communications regarding such a dangerous thing totally rely on email, without any backup or double check.
Agreed, I'm not sure how it's acceptable to say "Email is exclusive option you can have" vs "We can email you when alarm is going off but light/siren will flash and it must be visible to those using the facility"
I feel like there are things we aren't being told here.. what kind of automatic chlorine dosing system both knows that there's too much chlorine and continues to add enough chlorine to the point of being harmful. Something is fishy here...
It's not the alarm that was dumping in dangerous levels of chlorine. Microsoft is in the line of causes to the lack of warning, but not the actual poisonous conditions.
I do agree with this sentiment. At the same time, it’s still unacceptable for a global service like MS365 to go down for extended periods of time. I wouldn’t be surprised if the hotel chain was sold on some MS guarantee that it would never become unavailable.
Although I would also believe a scenario where the hotel chain "IT” (aka the owners son or grandchild) just used default alert settings and didn’t bother to setup fallback alerting mechanisms.
Hindsight is 20/20? Bullshit. Safety critical systems should not rely on transport layers that are guaranteed to fail. Email especially! Physical alarms are the way here. The email should be for convenience and to inform higher-ups.
Not just that, in this case there was no way for the alert system to even know that the emails were delivered. I can’t speak to everyone but for my company emails were not bouncing, I just got them 12-24 hours late. At least SMS and the like know if the message reached the actual end recipients inbox.
Please chill out. Chlor Level in a swimming pool is now where life and death or as the other user here noted, the user of the pool should be able to notice that something wrong by them self.
You still have think a little bit for your self to prevent self harm.
Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.
There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.
- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault.
- The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault.
- Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault.
- The regulatory system that allowed this design is at fault.
- The health and safety inspection system that didn’t catch this risk is at fault.
I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.
I don't think we ruined our ability to think about things, this was a failure to build a reasonable system. It's just basic planning and trying to consider possible faults. If the only way you will be informed of a potentially deadly situation, a sensor reading out of range - that affects human health - is it okay if it is only via email???
What if the person who reads the email was gone that day? What if they forget to read it? What if the email system is down? What if it was lost? This is just careless and bad design.
At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".
It's not some tricky thing where you had multiple ways to detect it, giving some redundancy. Same thing if it was a text message only or a phone call only.
This is a poor safety design with a single failure mode on the notification.
Just because the alert signal is an email at one point in its journey from sensor to human doesn't mean you're relying on someone using a computer to see it. Automated paging systems can read email too.
Yes, but we have to remember email is not guaranteed to be delivered, it's not reliable. You cannot in general tell if someone even looked at it (people try to build systems on top to provide these things, but it's not universal).
Also, I have a personal domain, some sites decided its not real enough and I can't get email from them.
Email is just not reliable. It could be part of a combined system of other things that are reliable.
The system should be designed to fail-safe during a power loss. If it can't, then it needs to have appropriate battery (and possibly generator) backup.
Your analogies aren’t equivalent because physical security and health and safety are quite hard.
The power failure would presumably shut down the pumps and end in the pool being closed. Then pools should have a power outage protocol they’d follow before opening the pool again. Checking chemistry would be high up on the list.
Smoke detectors need backups because a fire can cause a power outage before there’s enough smoke to trigger the smoke detector.
> At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".
More properly said, you may cause panic. People genuinely don't know how to respond when they hear an alarm in an unfamiliar place.
And that's if you're lucky. I once had to deal with people who did not want to evacuate a building when a fire alarm went off. And when I got them out the door I had trouble convincing them to leave the building's deck. Some people seem to think that it is okay to ignore a fire alarm, even a probable false alarm, in a wood structure with a wood deck.
It wouldn’t need to be that loud. Unlike a fire that becomes dangerous in mere minutes, this type of accident takes days to unfold. Most automatic pool systems will have a light or a beeping noise to let the operator of a failure. The email is just a convenience. The hotel staff should have been checking the system directly every morning to see if there were errors before they opened the pool for the day.
> Decades of “root cause analysis” poisoning have ruined our ability to think about safety and reliability holistically.
This is not the case in aviation. Even in the most egregiously obvious single-cause failures, there's always an analysis of issues in the wider system that allowed that single cause to become the single cause in the first place.
I agree with your points, though. In this case, Microsoft absolutely takes shared responsibility. It is only a matter of time until a similar outage in some other wider networked system with a so-called 'five/seven/nine nines uptime guarantee' goes out for a couple of days, and takes with it some critical life-sustaining system, and people die.
This seems wrong (obviously I don't have any more info than you).
Ford doesn't build all of the modules of their cars. They design specs and outsource manufacturing. But legally speaking, Ford is responsible for design, testing, and support / recalls. It's their responsibility to contract in the terms that both protect their customers' safety and Ford's liability due to spec / contract violations.
When your company is responsible for the health of staff and guests while dealing with deadly gasses, it's your company's responsibility to ensure that all of your vendors detail how their failure modes fail safe.
This is the same exact logic that has been migrating software companies to detail their risk by demanding their vendors all supply a "Software Bill of Materials" (which is very difficult in practice, but is necessary for the downstream users to protect themselves from upstream risk).
the point of root cause analysis isn't to find the one person or entity at fault. the point is to determine what you can fix, so this doesn't happen again next time.
anything that was "at fault" that you can't do anything about doesn't matter. and you can't do anything about a microsoft outage, all you can do is design around it.
Your second sentence is 100% correct. Your first is questionable; Microsoft, a kabillion dollar company has, for years, held itself out to be reliable and safe. And it repeatedly has shown that it is not. That opens them up to some liability here; and frankly deserves to be better held to the standards it (either directly or indirectly) claims.
And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.
It's easy to say this in hindsight, but how many pools/hot tubs have employees monitoring an alert panel around the clock for water quality metrics of the water?
Countless systems are relying on email as a means to alert people to issues, for better or worse. Having a physical alarm for everything is rarely practical for a myriad of reasons.
I worked in a data center 20 years ago that had alarms on the wall for when the halon was preparing to be released for fire suppression. There are likely dozens of things that could be monitored, but it was only the catastrophic thing that once it started, there was no stopping it, that got lights on the wall.
Too many loud alarms make people start ignoring alarms. I would guess that there is a fairly large margin of error in pools before there is a health risk, so it likely wasn't deemed alarm worthy. As someone who's been in a pool before, I'd also imagine that people would notice and say something, or it would be unpleasant enough to not want to spend time in it. That's what I find most surprising... that people weren't immediately turned off just being around it, to the point of staying out.
On every public pool I've ever been to, there was at least one lifeguard on duty. And part of that guard's duties included manually checking the chemical levels. Automation is great, but there still needs to be a human present that can verify those systems are working at the start of their shift, and to be somewhere they can hear or see any alarms if they go off.
100% correct. An email service is not to be used for critical alerting and communication. There are other systems that are up to the task - including infrastructure built on Azure which could have the required level of fault tolerance in it.
Outside tech, the further you are away from Silicon Valley (hint: not physically but industry wise), the more common it is to see such design of systems. Why? Because technology is a cost and IT department is a cost center which doesn't increase revenue.
When an incident like this happened, they might add new "system" to improve the situation, probably in the lines of having someone to patrol and check off a list routinely.
No, there are many Linux users there, and some small high profile government cases of trying to move of office, but like everywhere else it is hugely dominated by Windows, especially this type of smaller business area, hotels, OT systems etc.
Ironically that wouldn't completely do it. You would need to knock out faxes too. Just shutting down Excel would probably work for the Netherlands though.
In the EU there is a lot of noise about the need to not rely on American-controlled platforms, but precious little activity towards actually doing something about it. If anything, I'd say usage of Microsoft platforms is far more prevelant there (at least in the UK, France, Benelux, DACH region, as that's what I'm familiar with) than in the US.
> This was because the automatic dosing system had pumped too much chlorine into the pool.
What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.
I'm not even going to read the article. If your hotel depends on some cloud service to prevent chlorine gas poisoning you're doing something terribly wrong. Full stop. I don't know anything about the laws in Germany but this is dangerously close to libel in the US.
Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.
I don’t understand why someone would design a pool system that can harm swimmers with chlorine and the only safety is an automated email notification, and then blame the email provider and not their own system design for it.
There's no details in the article so you didn't miss anything. TLDR
> So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.
Because the outage didn't lead to the poisoning. Operator negligience and poor design did.
If I built a chlorine tank out of toothpicks and duct tape, and it leaks everywhere and kills people, you don't publish a headline that says DuctTape and Global Toothpicks Inc are responsible for chlorine deaths.
"A leads to B" means that A resulted in B. It doesn't imply that A was the root cause of B or deserves blame for B. For example, "heavy rainfall leads to flooding" can be true as a factual statement, even if the blame goes to poor drainage infrastructure. Also, the headline is in German and translates as "15 guests injured in Phantasialand".
The actual subhead translated from German reads "Due to a Microsoft outage on Friday, Phantasialand was unaware that there was too much chlorine in the pool." That seems like an accurate statement of events. Not that this was competent system design, but that as designed it depended on an email being delivered.
Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoft’s service failed. It doesn’t matter HOW it should be engineered.
We all know that services can fail, and consequently will fail at some moment. It is therefore primarily the responsibility of the designers of this system.
Because Office 365 is specifically marketed and contracted to customers as something not to be relied upon for critical real time controls when life or death matter.
Microsoft has many other offerings for such things, ranging from some services inside Azure to software (like operating systems) intended for real-time use on on-premises hardware.
Chlorine gas is no joke. In high school I was making it by running electricity through salt water and then diffusing the gas through pool water to chlorinate my parents pool. My cousin took a whiff when I was distracted and had to go the emergency room for breathing problems. Fortunately he recovered fully.
Are humans born with innate knowledge that some things are deadly to smell? Or are we born with an instinct to smell things to figure out if it’s good or bad?
Chemistry is literally the science of making deadly substances, smelling them and writing down one's impressions of the odor before the onset of death.
Curiosity. I picked apart that component that made my shoes blink. It was very difficult to get it open, but the silver liquid in it was fun. Today I think that was mercury and that I probably shouldn't have done that.
its interesting in the sense that emails have sort of become the de-facto standard for sending alerts and keeping paper trails. Even for systems like this. However, I agree with what other folks are saying in that email should never have been the ONLY way someone gets notified about an issue.
146 comments
[ 326 ms ] story [ 759 ms ] thread> 15 guests injured at Phantasialand
> Due to the (Microsoft) disruption, no emails were received, as the company informed WDR on Saturday. As a result, no one was aware that the sauna area's maintenance safety system had already issued an email alert. This was because the automatic dosing system had pumped too much chlorine into the pool.
> Subject: Fire
Dear Sir/Madam,
Fire! Fire! Help me 123 Carrendon Road. Looking forward to hearing from you.
All the best, Maurice Moss
Using the unqualified "cause" is misleading click bait and in some places it could get you sued
I do agree with this sentiment. At the same time, it’s still unacceptable for a global service like MS365 to go down for extended periods of time. I wouldn’t be surprised if the hotel chain was sold on some MS guarantee that it would never become unavailable.
Although I would also believe a scenario where the hotel chain "IT” (aka the owners son or grandchild) just used default alert settings and didn’t bother to setup fallback alerting mechanisms.
In short, all parties suck here (MS and hotel)
> the user of the pool should be able to notice that something wrong by them self.
Clearly, their motivations for autonomous self-preservation were insufficient.
Similar to doing regular restores of your data backups. If you never test a restore, you don't have a backup.
There isn’t one root cause here or one person or entity at fault. Many things went wrong and bad choices were made by many people and entities to lead us to this point, and they ought to be all regarded seriously and addressed rigorously.
- Whomever at the hotel that approved this as the sole (or even primary) alert system is at fault. - The lack of training and experience this hotel employee had when in a position to make such a dangerous choice was at fault. - Ditto the alarm company - everyone involved in designing, approving, and implementing this system is at fault, as well as whatever process ensured that nobody had the experience or safety to be able to speak up about this dangerous design is at fault. - The regulatory system that allowed this design is at fault. - The health and safety inspection system that didn’t catch this risk is at fault.
I’m sure there’s more; I’m just spitballing here. But the point is that there’s no one person or thing to blame here, and no one fix. This should kick off a wave of personal and professional soul-searching and improvements.
What if the person who reads the email was gone that day? What if they forget to read it? What if the email system is down? What if it was lost? This is just careless and bad design.
At the very least you need a backup notification, like say a warning siren or noise at the pool? Probably you don't want to do that casually because "it could scare people".
It's not some tricky thing where you had multiple ways to detect it, giving some redundancy. Same thing if it was a text message only or a phone call only.
This is a poor safety design with a single failure mode on the notification.
Also, I have a personal domain, some sites decided its not real enough and I can't get email from them.
Email is just not reliable. It could be part of a combined system of other things that are reliable.
We've been through this with smoke detectors, it's not a new thing.
The power failure would presumably shut down the pumps and end in the pool being closed. Then pools should have a power outage protocol they’d follow before opening the pool again. Checking chemistry would be high up on the list.
Smoke detectors need backups because a fire can cause a power outage before there’s enough smoke to trigger the smoke detector.
More properly said, you may cause panic. People genuinely don't know how to respond when they hear an alarm in an unfamiliar place.
And that's if you're lucky. I once had to deal with people who did not want to evacuate a building when a fire alarm went off. And when I got them out the door I had trouble convincing them to leave the building's deck. Some people seem to think that it is okay to ignore a fire alarm, even a probable false alarm, in a wood structure with a wood deck.
This is not the case in aviation. Even in the most egregiously obvious single-cause failures, there's always an analysis of issues in the wider system that allowed that single cause to become the single cause in the first place.
I agree with your points, though. In this case, Microsoft absolutely takes shared responsibility. It is only a matter of time until a similar outage in some other wider networked system with a so-called 'five/seven/nine nines uptime guarantee' goes out for a couple of days, and takes with it some critical life-sustaining system, and people die.
Ford doesn't build all of the modules of their cars. They design specs and outsource manufacturing. But legally speaking, Ford is responsible for design, testing, and support / recalls. It's their responsibility to contract in the terms that both protect their customers' safety and Ford's liability due to spec / contract violations.
When your company is responsible for the health of staff and guests while dealing with deadly gasses, it's your company's responsibility to ensure that all of your vendors detail how their failure modes fail safe.
This is the same exact logic that has been migrating software companies to detail their risk by demanding their vendors all supply a "Software Bill of Materials" (which is very difficult in practice, but is necessary for the downstream users to protect themselves from upstream risk).
anything that was "at fault" that you can't do anything about doesn't matter. and you can't do anything about a microsoft outage, all you can do is design around it.
"Why did people get gas poisoning"
https://en.wikipedia.org/wiki/Five_whys
yeah, these companies need to be told to stop relying on cloud services for everything.
its wild to me how many criticism things that absolutely do not need the internet are relying on the internet.
much blame to go around here:
- whoever designed the stupid alert system.
- the marketing departments in countless companies ignoring engineers when they market to normie customers “connect everything”
- the hotel company for seemingly doing no dilligence at all. and just blindly listening to some salesman “connect everything”
- accountant types and their “we can save pennie’s if we cut all of these corners. and the stupid corners they choose to cut.
- etc…
on a macro scale it’s amazing to me how many people are victims to slimey car salesman types leading to situations like this.
5/5
Title: 5 nines of uptime
A grand piano is suspended from the ceiling with an electromagnet.
The electromagnet works as long as there is power.
No special battery backups, anything.
Guests are invited to stand underneath it, after signing a waiver.
That’s it. That’s the art.
Your second sentence is 100% correct. Your first is questionable; Microsoft, a kabillion dollar company has, for years, held itself out to be reliable and safe. And it repeatedly has shown that it is not. That opens them up to some liability here; and frankly deserves to be better held to the standards it (either directly or indirectly) claims.
And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.
> And see e.g. "Implied warranty of merchantability" as a refuation of the possibility that you can wave this away with a Terms of Service.
"Implied warranty of merchantability" is a term of art in United States law, and it applies only to tangible goods, not to services.
I'm surprised an e-mail only alert is even legal for this. Or is chlorine level monitoring not something that is regulated?
Countless systems are relying on email as a means to alert people to issues, for better or worse. Having a physical alarm for everything is rarely practical for a myriad of reasons.
I worked in a data center 20 years ago that had alarms on the wall for when the halon was preparing to be released for fire suppression. There are likely dozens of things that could be monitored, but it was only the catastrophic thing that once it started, there was no stopping it, that got lights on the wall.
Too many loud alarms make people start ignoring alarms. I would guess that there is a fairly large margin of error in pools before there is a health risk, so it likely wasn't deemed alarm worthy. As someone who's been in a pool before, I'd also imagine that people would notice and say something, or it would be unpleasant enough to not want to spend time in it. That's what I find most surprising... that people weren't immediately turned off just being around it, to the point of staying out.
Maybe a waterpark resort, but hotels no. These days they barely afford a front desk employee and a cleaner.
When an incident like this happened, they might add new "system" to improve the situation, probably in the lines of having someone to patrol and check off a list routinely.
Anything that could result in that sort of harm should have a more resilient and redundant architecture.
This was bad system design. Hopefully people learn that safety critical services can't rely on Microsoft which they also tell you not to do.
What caused what? There is a story in Mahabharatha where Arjuna was claiming that he achieved killing of Karna. Krishna, the guide, enlightens him about the causal chains and shows how Arjuna' contribution was minimal.
Disclaimer: I work in Azure, this is not a defense of the outage and I didn't read the submitted article.
I’m so sorry.
> So no one noticed that the maintenance safety system of the sauna landscape has long since sounded the alarm by mail. Because the automatic dosing system had pumped too much chlorine into the pool.
If I built a chlorine tank out of toothpicks and duct tape, and it leaks everywhere and kills people, you don't publish a headline that says DuctTape and Global Toothpicks Inc are responsible for chlorine deaths.
Why does it matter that the service someone is paying for is an api or something else versus email? The email service failed. First and foremost Microsoft’s service failed. It doesn’t matter HOW it should be engineered.
Microsoft has many other offerings for such things, ranging from some services inside Azure to software (like operating systems) intended for real-time use on on-premises hardware.
https://how.complexsystems.fail/
Like, what leads a teenager to do such things?
https://www.science.org/content/blog-post/things-i-won-t-wor...
https://www.science.org/content/blog-post/things-i-won-t-wor...
https://www.science.org/content/blog-post/things-i-won-t-wor...
https://www.science.org/content/blog-post/things-i-won-t-wor...
Back in the day, i tried to sniff NH3 because i've heard it smell like piss.
It's a lot of worse.