- cross-posted to:
- tech@feddit.online
- cross-posted to:
- tech@feddit.online
cross-posted from: https://feddit.online/c/tech@feddit.online/p/2115831/microsoft-365-outage-leads-to-chlorine-gas-poisoning-of-15-hotel-guests
cross-posted from: https://feddit.online/c/tech@feddit.online/p/2115831/microsoft-365-outage-leads-to-chlorine-gas-poisoning-of-15-hotel-guests
Yeaaah not MS fault here, no sane person would rely solely on email for something like this
What else should they rely on?
An alarm perhaps, with flashing lights for the deaf and those wearing ear protection? That’s what I’d demand if it was part of a system I was involved with designing
Public pools here are required to have audible alarms and flashing lights on redundant electric circuits in the dosing room and main hall, plus a shitload of ventilation.
Every walk in fridge is required to have CO2 alarms, just like smoke detectors.
Not everything needs to go through a computer. A loud ass alarm is sufficient.
A hotel pool is not a public pool. It’s a private pool.
See that makes too much sense and might cause too much safety
We used to have these things called blinking lights and alarm sirens that required only a few cheap components and no internet at all.
“Used to”. Awesome, what do you have now? Why did you switch from “these things called blinking lights and alarm sirens” and what did you switch to?
Good engineering practices for starters.
Which are? Please tell me and give examples. I’m not an engineer and don’t know what’s code
An important one is something called a fail-safe. It’s quite literal; it dictates that a system must be safe in it’s failure state. In this case when it detected a failed pump, the system should have shut down. Fail safes are usually passive and don’t rely on external systems.
As an example, image a room with a magnetic lock on the door. It should be design so that when the power goes out, people inside can still exit via either a mechanical push, or the latch just releasing from not having power.
https://en.wikipedia.org/wiki/Fail-safe
An actual automation system. Ideally, a self-hosted one.
I wouldn’t advise self-hosting unless you can guarantee the resources to do it right, and with human lives on the line plus negligent management, that’s a lot to ask.
And for modern SMS alerts you probably need someone else’s platform anyways, so you’re halfway to paying for this.
I mean, the pump is self-hosted. The water is self-hosted. Smoke detectors are self-hosted.
Making an exceedingly simple system that sounds an alarm and/or flashy light when a threshold is crossed is hardly new. It doesn’t take that much and if you’re really worried, set up a duplicate system to run in parallel.
Using email for something important is dumb. This is why 911 doesn’t have an email address.
This guy gets it.
Bit of a pickle. If you can’t trust the biggest name in the industry to get it right…
Parallel: Pagerduty, Slack, NTFY.
you can self host NTFY then you have 1/3 under your express control.
and it would alert everyone how?
Notifications on the automation portal, a push notification if you have an app, an sms message, an email, a physical alarm or all of the above. Hell, you could even have a physical alarm at the actual pool to alert customers in case someone from maintenance missed the alarm. I don’t know how this is an actual question for you xd
Imagine being so slop brained that you forget that physical alarms and local system alerts are a thing.
Ok smart ass. Where are the alarms in retrospect to anyone that can hear them? Tell me all about your experience setting up alarm systems for people that aren’t in the building.
You’ve never worked a building engineering or maintenance system at all, have you?
I’m in residential at the moment, but at my last job, there was always someone on call for equipment issues like this. Alarms would pop up on any BAS terminal if it happened during the day. Building I worked at had high pressure steam boilers, so there was always someone on-site as well. You’d get a push notification from the BAS app on your work phone as well if you were away from a workstation. Hell it would automatically text and then call you too. It’s 50/50 if the call was triggered by an email alert or if it would just dial from the main line and do direct text to speech. Don’t remember how it was configured. But I do know the text message alerts were built into the software. There were 3 ways to reach someone, and all the major systems are tied into it, so can be remotely accessed via a laptop for troubleshooting or shutting down equipment.
Actual automation systems have redundancy built in, because they are necessary for safety and thus reduced liability. We had high pressure steam boilers. If the fucking building blew up, I couldn’t exactly go to my boss and say “well you see, Microsoft had an oopsie and that’s why the multi million dollar building doesn’t exist anymore and why we’re on the news, not my fault, boss” of course, systems that are actually designed well will have hardware safeties built in with the monitoring and alerting as well.
Oh and I was also required to do rounds at every shift change and physically check the analog gauges to make sure they matched what the BAS was reporting, and log any discrepancies for investigation if they could not be immediately identified.
Fucking hell you are slop brained. Not everything should or does run though a fucking cloud. Email has always been a backup to a backup for critical systems because sometimes it takes minutes for an email to arrive on a good day. Physical alerts on equipment, electronic notifications on local hardware, and telephone communication for remote alerts have always been the industry standard. I pray to god no one like you ever steps within 50 yards of a boiler with a “cost saving” improvement to how things are actually run.
This is going to get worse as more and more people turn their brains to mush with chatgpt
My god is it ever. Idiots are going to be asking “how can I say something to make it sound like I’m an expert about something when someone asks a question without knowing a thing? Phrase it in a way to get upvotes.”
A buzzer, a bell, a klaxon… Audio saying “caution: chlorine levels too high”
Oh, I thought you were joking
I was literally asking for an answer because at this point in time I don’t know what the trendy monitoring startup company software is I’m supposed to use. Thanks for absolutely nothing.
Auto-shutoff on the pumps side, like it’s standard for even sludge pumps.
So why didn’t they have that? Why isn’t it code?
Chlorine level monitors that staff check before opening the pool? That doesn’t require internet and would only be a simple embedded device. You might be able to have an electricity-free option too, not a chemistry expert but I imagine there are tests for chlorine levels that don’t require electricity?
Yeah they exist. Most people who own home pools use them and then take their water to be tested if something is getting way off.
I helped out at a hotel for a couple months at one point and their setup was simple (to use). Every morning we would check everything manually to ensure it was correct, it was also controlled by machine. Essentially if the machine detects the level of something gets to low, it automatically adds the amount needed. Essentially all of the water is cycled into a room near the pool, and that’s where all water is added and tested automatically from.
The manual tests we would do from 2 different areas of the pool to ensure they matched what the pool sensors detected in the behind the scenes area. If something was way off somehow alarm lights would show on the panel, and no chemicals would be added and a contracted maintenance group would be dispatched to come address the major issues.
Also pools are large amounts of water, so they don’t just suddenly change drastically unless someone is intentionally pouring chemicals of some kind into them.
Id say 98% of chemical issues are dealt with by normal staff and the automated system. The pool company with the contract comes to verify everything working once a month, and maybe once or twice a year you’d call them out for something, most likely after winter when opening the pool just to make sure they clear everything, and once a year when something weird happens.
How much were you paid for this?
I got confused and thought you said this to another comment, paid to help out at the hotel? Not to much it was basically a maintenance job that had this as a morning and afternoon side quest. At the time it was probably like $15/hr,
There are a ton of on-call systems available that have hard contractual uptime reliability requirements.
Every time I’ve been on a team with on call we had a required app that pushed critical notifications to your phone, plus SNS, plus email, plus slack, plus an actual phone call system.
And I’m not even an SRE or anything like that.
Cool what was the name of those systems you personally setup that had all of those messaging targets on by default?
PagerDuty. I have never set it up, so I don’t know the defaults.
Phones? Pagers? Klaxons?
Very rigorous maritime engineering standards.
So why didn’t the maritime code enforcement agency come in and make them fix it?
Idk, any self-hosted system, there are so many stuff one can do…even having displays or alarms…
Cool, good talk. Give me an example of even one and what other than email it uses that is on by default.
MS wants businesses to sign up for their cloud services, promises the world, and says don’t worry. Businesses rely on this nd when it goes down it’s not MS’s fault even tho MS is the service that went down? Come on.
It is their fault for not having a fallback system when lives are on the line. And the whole strategy of “outsource something so we can blame that when it stops working” is transparent, lazy, and negligent (criminally in this case IMO). Downtime has always been a thing and it’s not like MS has a reputation for reliability (I would have said the opposite based on using their software over the last few years).
If you haven’t realized by now that marketers will say whatever they think will get them the sale, that doesn’t say great things about your intelligence, even if there’s a lot of people falling for that BS.
And who are the people that fall for that bs? Spoiler: they’re the same idiots that think they know more than engineers and won’t buy the systems with more contact options because it costs more (or not enable those alert systems because they are extra services and cost more).
It’s not MS fault if you don’t also have a fallback system or any system that can work even offline/dipendently on your network. Regardless of microsoft being down or not, emails for such stuff is a VERY bad idea and should be changed with a better system that cannot be accidentally ignored.
Ok, you got me. What’s the better system and why don’t businesses use it from the get-go instead?
Gross negligence is a thing. What’s the point of moving away from MS if they can just point the finger at them and yell “daddy told me I could?”.