Some servers in GRA still appear to work if that's of any help. All data centres offline at once sounds more like an attack than a power failure in one location. According to them, there was a power failure in SBG but I don't see how that should affect routing in data centres several hundred miles away.
It seems more likely that their data centers aren't quite as isolated as they thought they'd be. The outage also appears to be limited to their locations in Europe.
I always wondered why they promote having 6 datacentres in Roubaix when google maps shows that they're all within 50m. Can't be too much redundancy there.
Then they could call it one DC. A DC can have more than one building. But by saying you have several data centres you imply redundancy, similar to AZs at AWS. And (at least to Amazon), two AZs are far enough away from each other so that one building could blow up without affecting the other one.
The network equipment failure was due to a software error[1]. I would not rule out some kind of cascading failure triggered by the initial outage until the cause of the bug is known, though OVH seems comfortable with such a statement.
How is that even a thing? Don't you have separate routing for every switch? Otherwise you don't have redundancy? I'd even expect data centres to use different hardware for networks to avoid having a single point of failure.
UPDATE: not all datacenters are down, it seems like that in europe because ovh routing hasn't been updated so from our point of view everythign is down but really it is not :)
I think they don't know themselves what works. The CEO said GRA is down while I can access it without issues but that could be depending on where you try to connect from.
Looks like he deleted the tweet but he wrote earlier that GRA was down with the others while BHS was up. I'd guess he is in RBX and couldn't reach GRA so assumed it was down until they realised that the RBX-GRA line is down.
EDIT: He did indeed delete the tweet, [1] is the url in case anyone knows a website archiving them quickly enough.
"SBG: ERDF is trying to find out
the default. 2 separated 20kV lines are down. We are trying to restart 2 generators A+B for SBG1/SG4. 2 others generators A+B work in SBG2. 1 routing room is in SBG1, the second in SBG2. Both are down. "
"An incident is ongoing impacting our network. We are all on the problem. Sorry for the inconvenience."
"SBG: 1 gen restarted."
"RBX: all optical links 100G from RBX to TH2, GSW, LDN, BRU, FRA, AMS are down."
I moved away from OVH after I paid 3 months advance (~$300) for a server which burned down after 1 1/2 months. They did not issue any refunds (data, blood, sweat and tears were lost that day). I have been an OVH customer for 12 years.
Today, I'm glad to have moved away all my production environments as well.
I'm still at OVH (support is reasonable and prices cheap) but would never trust one provider with all my infrastructure. In the end it always turns out that there is a single point of failure and if it's just the billing department. Using two providers protects you from that and if you chose some with good peering and free traffic, keeping both in sync is relatively easy.
That's just the problem with services as AWS, traffic is too costly to have production live with another provider as well.
Mirror your data onto another provider continuously (log shipping/rsync), Switch DNS.
Ansible works for this stuff as it allows you to can the task of “quick get me a production environment up on Linode!”
If you can afford some downtime you don’t need a hot standby just roll out everything into new provider and you’re done.
I’ve done this on very large scale environments and small ones and it’s achievable for even small organisations. The killer is avoid anything you can’t run on bare metal servers.
I also do this approach. Get a decent, from your infrastructure independent DNS provider and take care of your Ansible scripts. This way in emergency you do a one liner, have a new production server running and change DNS settings.
In addition I'd suggest getting a second domain in a TLD operated by another company in another country than the primary TLD is, and teaching your customers/users that both are valid. This protects you from three things:
2) legal issues, when one of your domains gets seized or the provider gets pressured into cancelling, just as has happened with Pirate bay, SciHub and friends, gambling or sites with user generated content that may be illegal or frowned upon in some countries or for the latter to the Nazi site Daily Stormer (although I'm glad for it being down, it's a perfect example what can happen in a very short time frame)
3) (edit, after suggestion below) the entire TLD going down because the TLD DNS provider has issues, which also happens from time to time.
Just an good advice: Never ever trust a single point/server without backups. Do backups, do snapshots (check if and how you can do that), store them somewhere else. Try to do automated reinstall with e.g. Ansible so that you can move easily. It's an investment at the beginning and during running the server but if the stuff is critical you should always do that.
My experience with OVH VPSes is that they are cheap and reasonably reliable. But when one starts failing support never helped me to get it fixed.
Just fire up another one, install everything on there, switch off the old one and forget about it. This means never paying for more than one month in advance.
It happens to all of them. Whenever I read comments like yours I wonder where you moved your servers to, and if you'll move them again when that goes down too.
It started with all our SBG servers going down simultaneously. Approximately 1h later all our RBX servers went down as well including the OVH status page and all other OVH web applications. Either their SBG and RBX data centers are somehow connected or those are indeed two independent incidents.
SBG was a power failure, followed by a generator failure. Not sure if they set up their network infrastructure in a way that this could spread but I find it hard to imagine that the outage in SBG triggered RBX going completely black. Unless of course they store their configuration files in SBG.
"guys we have a single point of failure in our architecture with SBG, maybe we should...
- naaah it's fine, we do not have time nor resources"
Then shit happens.
edit: I have no idea what is happening exactly, but OVH being what it is, it seems extremely weird that all datacenters "can" get down at the same time, and it looks like a serious architecture problem to me (or backup systems, like generators, not being correctly tested... whatever). I am really curious about the future explanation with what happened exactly
edit2: Why all the downvotes? Even the status page of OVH is down, do not tell me it is good design. We are not here to be charitable, but realist.
If the SBG issue really triggered the outage for the whole network I find it hard to believe that no one saw that problem beforehand. They probably thought that this was too unlikely to happen or that there are other failovers but never tested them properly.
No expert on the field but that's the first time I can remember that a provider of that size loses connection to most of their data centres at once. That can happen with one product (eg S3 failure) but datacentre switches should work even if the rest is on fire.
It's supposed to be two separate incidents: power going down in Strasbourg, and fiber network equipment going down in Roubaix (the main center of OVH's network) due to a "software bug".
In this case, I wouldn't be to hard on them. As it appears they lost their main power line, the backup power line and both generators failed and one generator has been restarted now.
So what? Losing main power is a standard case for any DC. That's why you have generators. Even a generator failure is nothing out of the ordinary. But that no generators in a DC work kind of indicates that they don't test them as often as you would expect.
They just announced that they want to be a "hypercloud" provider on the scale of AWS and Google Cloud. I really hope that a power failure in Virginia couldn't bring down all of AWS.
When I've seen things like this before, it's often been the switchover hardware that fails, not the generator as such. it's much harder to test that as you don't want to tell your customers "sorry your sever went down, we were just testing if the switchover worked and it didn't"
Exactly that, we too suffered a power loss at our DC that was due to faults in the power supervising and switch-over circuits; we test both generators weekly.
I worked for a company doing mostly on-premises (wireless) carrier software years back but they still wanted our own server room to be 3 nines for reasons lost to me now. They had installed new electrical circuits high on the walls so that we could survive minor flooding incidents.
So our lead Ops guy is unplugging half the redundant power supplies and plugging them into the new circuits, but a few critical servers are single PSU still, on UPS units. This is how he discovers that one of our PSUs has rotted and only has about five seconds of reserve power in it. Big outage, no bueno.
Didn’t AWS have a cascading failure last year because of a missing interlock on a script to take part of a data center offline? They discovered that it takes a lot longer than they remembered to reintroduce that many machines to the cluster. They came away from that incident with two or three action items.
What is the point of backup generators if you do not verify that they work every so often? I have a very hard time believing that they actually tested that they worked, because a failure of not one, but both of them.
To be fair people do test generators on a monthly schedule usually. Problem you find is it’s getting colder now so any problems are amplified suddenly. Might have been entirely tested a couple of weeks ago.
The problem is usually not "not reliable in cold" but rather, the generators are X years old and the temperature is now changing in Europe from "mostly warm" to "warm over the day and icecold in the night" and finally aiming for "icecold all day", which means any equipment exposed will go through rather severe temperature changes.
While generators are usually able to handle this with sufficiently low failure risk, the risk is increase due to the changing temperature
It wasn't even freezing overnight at Strasbourg [1], I doubt that the cold could cause any effects at 5C. Maybe they were run once a month but never tested under load? 2x20kv lines failing will have put them at near maximum load immediately, perhaps they weren't designed for that.
Improbable. Even in a mostly-normal office building, the monthly generator test consisted essentially of "cut the mains power and see that no impact is perceived as UPS, batteries, and autostarted generators bear the load, in their turn for ~2 hrs total."
Their generators are almost certainly tested frequently. But there could be any number of causes underlying the failure, and unfortunately sometimes failure does happen.
Generators in critical situations like this are usually provided with crankcase heaters which keeps the oil at temperature and makes them easy to start. Also, any emergency power system should be (and usually is) tested monthly if only to prevent the fuel lines going bad.
Data centers can go down while testing the generators too.
There was a DC in California a few years back that had three generators fail during a scheduled test and one of their server rooms had a blackout because of it.
its OVH:
The Hardware is good
DDoS protection is good
The Prices are high
but
support/administration does not work well, i have a lot of really weird story's with them, from them plugging in a keyboard in our server to reboot it (without any reason) to taking down a server for a requested maintenance only to notice after 4 hours of downtime that they did not ask their bosses if they were allowed to even perform the maintenance requested (and then not getting permission to do so after another 2 hours ..)
For me it feels like there are some really deep issues somewhere in the whole administration that make incidents like this no real surprise
Problem is most other providers dont work any better, so ...
Everyone makes mistakes, let's just hope they learn from it.
I've tried Hetzner but the Network Peering to Telekom is subpar compared to OVH. On Hetz I got about 40Mbps up/down to my local computer while on OVH I can easily load my DSL 100% without issues.
OVH is one of the few with direct peering to Telekom, most don't want to pay for that. Otherwise Hetzner is good, their own network is limited but peering works reasonably well. But apart from them I'm not aware of anyone with cheaper dedicated prices than soyoustart
Their prices are pretty amazing for some of the configurations you can put together, especially since you can pay a large setup fee in exchange for a lower monthly rate.
Just the bandwidth alone would cost us 3x what we pay at OVH if we were with one of the big cloud providers.
> - naaah it's fine, we do not have time nor resources"
Yup, been there multiple times in smaller hosting companies.
It's basically how it goes. They don't get serious about outages until revenue is severely affected and the brand damaged, they don't get serious about security until there's been a big breach or sales are lost because of lack of certification.
The person you're replying to was relating their experiences at hosting companies that are smaller than OVH. How does that imply that OVH is a smaller hosting company?
This happens all the time. Every single thing that you see in software development happens in network engineering and data center engineering, except that where in development in general senior people who write software are capable of at least guestimating complexities to provide a marginally unified front against the unreasonable expectations of execs, it is pretty much never the case in neteng or dcops as those that develop software cannot write their heads around the complexities of working with physical hardware.
It is rather counter-intuitive. In neteng and dcops, "I don't know and I cannot find out. I can only attempt to mitigate what i think might have caused it for next time" is a very reasonable answer to 99% of the "why this happened?" questions because in order to replicate the situation to test the theory one needs to recreate the same problem again on the same scale.
This also means that certain things cannot be tested. Most of generator tests are garbage - turning on generator and running it without production load delivered over the transfer switch does not test anything other than that one can turn on a generator and run it. The problem typically happens not because the generator ( also is there the generator or the first and the second generator? Why is there no generator bank for a non monkey-sized company? ) does not start - the problem is because over time transfer switch develops a problem and unlike generators it is not possible to test a transfer switch where in the event of a test failure the customers won't lose power unless the data center is designed from the beginning to deliver A and B powers over separate circuits to every single customer and every single customer has per system ( not per rack ) transfer switches.
Of course it costs a lot more money, something that companies are reluctant to spend.
I think I'm a pretty decent senior network engineer but I've been hit by firmware bugs more than once. No matter how much redundancy you build in, there's always shit that can go wrong and there will always be things that you just can't foresee (like the bug in this case). The software on these network devices and optical gear is written by humans, like all software, and is not perfect, like all software.
(In one case I experienced, the vendor reassured me that what I reported was not even technically possible -- until one of their engineers flew out and witnessed it firsthand.)
wow, yesterday I was playing with their public cloud because considering choosing them. I had some connection problem with my private networking there (deleted it more than once) and opened a ticket. If it was me... sorry, haha. Not good advertisement but it can happen to everyone.
Huh, I was just looking at them too. Contrary to popular opinion, I kinda prefer it when these things happen before I sign up so that in the post-mortem usually whatever architectural failure lead to the outage is corrected and you get a stronger service.
Wouldn't this require that there be a very limited amount of things that can go wrong?
Unless by stronger you mean some kind of a change in mentality over care for the service, but that would probably have a limited lifespan until it's back to normal.
Not saying this is the case here but that's also sometimes where you spot amateurism and should run away. I remember a host provider a long time ago (15y) who was storing its backups on the same machine as the main data. Guess how I figured out!
Oh man that sounds bad! You're right, sometimes these events do expose the inability to handle failure and you're right that that means walking away.
Thankfully many times we're reminded that there are good people out there working hard against difficult constraints and they finally get their chance to do things 'correctly' in the wake of the SHTF.
OVH are very open about their outages and root causes. Even routine maintenance tasks are cataloged with updates at http://status.ovh.net
As a more technical user it's nice to have providers that give this information rather than the boilerplate "Issue with an upstream provider" over and over.
Hey I would suggest not completely disregarding OVH. Their prices are good and their network is usually very reliable and fast. You simply won't find a North American hosting provider with those prices and reliability. You won't find one at double OVH prices, either (and I have tried).
no, I'm not completely disregarding OVH, I now about their network and reliability in the past. Will rethink only about my disaster strategy in the future.
It's in the Gravelines datacenter, which is not actually down contrary to what the initial reports said (only Strasbourg and Roubaix are, and for two different reasons).
Yes, very important. I have two dedicated machines in different racks in Canada that were serving requests all night. I was also logged into both machines via SSH all night with no interruption.
We selected their three data center EU region precisely because they were three separate data centers, so not happy. This is clearly bad design.
I think we're now going to have to look into multi-provider options. The only way to be solidly up is to be hosted by more than one company at more than one data center.
I've also heard stories of billing nightmares where you get locked out of a cloud provider account, so that's another thing.
I guess this is already a reason by its own. It, among other problems, is what happens when we go from small "local" providers you can actually call to automated global providers that cannot provide immediate support even if they tried.
That's the problem with this trade off. The small providers tend to have good support and someone you can reach if anything goes wrong but they won't have experts on site 24/7. The large ones have dozens of them on call at all times but lack in support (unless you pay a lot).
The Canadian data center is also a very low cost way to serve the US. I'm an OVH fan. They have their quirks, but the pricing is great. You just make sure you compensate for their quirks with backups and DR plans.
My data is with me in Europe and the company that has my data is in Europe with me too. If I was using DO or AWS, then my data may be in Europe but the control over the data is in the US, free for all to the three letter agencies and lacking privacy laws there.
OVH is a giant hosting company in Europe, providing dedicated servers to quite a lot of companies. Half the internet here was borked. A translation service, a few developer tools, etc...
Sure, it's less dramatic than AWS going down, but it still hits, and hard.
We're nerds, so how about checking the facts (?). Please explain me if there has been some kind of time anomaly lately or?
My submit timestamp: 07:21:25
Your submit timestamp: 07:28:23
About the title. I did consider the title for a while, because I wasn't sure how bad the situation was. But from my own independent monitoring system I did see that RBX and SBG servers were unavailable. Of course I also did some basic trouble shooting and confirmation work before posting.
Btw. Right now, there's some network traffic present on SBG network. Let's hope that the systems are soon up'n'running.
I'm waiting for the postmortem. I'm very curious to see, what was the root cause of all this mess. As usual, there probably were several overlapping causes.
206 comments
[ 2.3 ms ] story [ 227 ms ] threadhttps://twitter.com/olesovhcom/status/928541667283623936
EDIT: Maybe related to the Cisco issue?
https://blogs.cisco.com/security/cisco-psirt-mitigating-and-...
Their explanation will be really interesting. All those data centres are pretty useless if they have a single point of failure.
One datacenter suffered massive power failure while the other hand network equipment fail at roughly the same time.
[1]: http://travaux.ovh.net/?do=details&id=28256
According to them, they do their routing in SBG, so it's plausible that it could lead to all of their network being down.
EDIT: He did indeed delete the tweet, [1] is the url in case anyone knows a website archiving them quickly enough.
[1] https://twitter.com/olesovhcom/status/928536233311076353
and on https://twitter.com/ovh_support_en
"SBG: ERDF is trying to find out the default. 2 separated 20kV lines are down. We are trying to restart 2 generators A+B for SBG1/SG4. 2 others generators A+B work in SBG2. 1 routing room is in SBG1, the second in SBG2. Both are down. "
"An incident is ongoing impacting our network. We are all on the problem. Sorry for the inconvenience."
"SBG: 1 gen restarted."
"RBX: all optical links 100G from RBX to TH2, GSW, LDN, BRU, FRA, AMS are down."
https://twitter.com/olesovhcom/status/928552251458818048
(edit: parent's post was edited after I posted)
http://status.ovh.com/
Pro-tip: self-hosting status page is maybe not the best idea.
[1] https://twitter.com/awscloud/status/836656664635846656?lang=...
Lister: What's the damage Hol?
Holly: I don't know Dave. The damage report machine has been damaged.
Today, I'm glad to have moved away all my production environments as well.
That's just the problem with services as AWS, traffic is too costly to have production live with another provider as well.
Having one AWS account scares the crap out of me as well. It’s never a good thing if all your eggs are in one basket.
My money is on stuff spread across Bytemark, Linode and DigitalOcean with a DR plan involving mostly automatic recovery.
AWS doesn’t get a look in as it is extremely costly to port away from anything that isn’t bare metal and pipes.
Ansible works for this stuff as it allows you to can the task of “quick get me a production environment up on Linode!”
If you can afford some downtime you don’t need a hot standby just roll out everything into new provider and you’re done.
I’ve done this on very large scale environments and small ones and it’s achievable for even small organisations. The killer is avoid anything you can’t run on bare metal servers.
1) your DNS provider having issues (even Route53 sometimes has them, https://mwork.io/2017/03/14/aws-route53-dns-outage-impacts-l...)
2) legal issues, when one of your domains gets seized or the provider gets pressured into cancelling, just as has happened with Pirate bay, SciHub and friends, gambling or sites with user generated content that may be illegal or frowned upon in some countries or for the latter to the Nazi site Daily Stormer (although I'm glad for it being down, it's a perfect example what can happen in a very short time frame)
3) (edit, after suggestion below) the entire TLD going down because the TLD DNS provider has issues, which also happens from time to time.
Just fire up another one, install everything on there, switch off the old one and forget about it. This means never paying for more than one month in advance.
Servers break. Providers go down.
It happens to all of them. Whenever I read comments like yours I wonder where you moved your servers to, and if you'll move them again when that goes down too.
SBG going down due to a quadruple power failure (both grid connections and both generators) is quite spectacular.
"guys we have a single point of failure in our architecture with SBG, maybe we should...
- naaah it's fine, we do not have time nor resources"
Then shit happens.
edit: I have no idea what is happening exactly, but OVH being what it is, it seems extremely weird that all datacenters "can" get down at the same time, and it looks like a serious architecture problem to me (or backup systems, like generators, not being correctly tested... whatever). I am really curious about the future explanation with what happened exactly
edit2: Why all the downvotes? Even the status page of OVH is down, do not tell me it is good design. We are not here to be charitable, but realist.
No expert on the field but that's the first time I can remember that a provider of that size loses connection to most of their data centres at once. That can happen with one product (eg S3 failure) but datacentre switches should work even if the rest is on fire.
So, that one datacenter caused all other datacenters to die..?
It's explained here https://twitter.com/olesovhcom/status/928587258583748609 in French, they might post an English-language translation soon.
They just announced that they want to be a "hypercloud" provider on the scale of AWS and Google Cloud. I really hope that a power failure in Virginia couldn't bring down all of AWS.
I can not imagine they weren't tested.
But even the most rigorous testing can never reduce the total failure risk to 0. It seems OVH just got very very unlucky.
So our lead Ops guy is unplugging half the redundant power supplies and plugging them into the new circuits, but a few critical servers are single PSU still, on UPS units. This is how he discovers that one of our PSUs has rotted and only has about five seconds of reserve power in it. Big outage, no bueno.
I'm much more interested in why the failure cascaded to other data centres – that's exactly what shouldn't happen.
While generators are usually able to handle this with sufficiently low failure risk, the risk is increase due to the changing temperature
[1] https://www.accuweather.com/en/fr/strasbourg/131836/daily-we...
Their generators are almost certainly tested frequently. But there could be any number of causes underlying the failure, and unfortunately sometimes failure does happen.
There was a DC in California a few years back that had three generators fail during a scheduled test and one of their server rooms had a blackout because of it.
https://twitter.com/internetofshit
It happened to GCE last year[1], though it only lasted 18 minutes.
[1]: https://status.cloud.google.com/incident/compute/16007?post-...
but
support/administration does not work well, i have a lot of really weird story's with them, from them plugging in a keyboard in our server to reboot it (without any reason) to taking down a server for a requested maintenance only to notice after 4 hours of downtime that they did not ask their bosses if they were allowed to even perform the maintenance requested (and then not getting permission to do so after another 2 hours ..)
For me it feels like there are some really deep issues somewhere in the whole administration that make incidents like this no real surprise
Problem is most other providers dont work any better, so ...
Everyone makes mistakes, let's just hope they learn from it.
The prices are high? Compared to what? Cheap is their raison d'être.
Sry if that caused confusion
Just the bandwidth alone would cost us 3x what we pay at OVH if we were with one of the big cloud providers.
Yup, been there multiple times in smaller hosting companies.
It's basically how it goes. They don't get serious about outages until revenue is severely affected and the brand damaged, they don't get serious about security until there's been a big breach or sales are lost because of lack of certification.
> in smaller hosting companies [like ovh]
> in smaller hosting companies [than ovh]
> Yup, been there multiple times, in smaller hosting companies.
I find it funny that anyone would think I would or could refer to them as small (and get away with it)
EDIT: The title on HN in misleading, summary from their CEO here – https://twitter.com/olesovhcom/status/928592231807713280
It is rather counter-intuitive. In neteng and dcops, "I don't know and I cannot find out. I can only attempt to mitigate what i think might have caused it for next time" is a very reasonable answer to 99% of the "why this happened?" questions because in order to replicate the situation to test the theory one needs to recreate the same problem again on the same scale.
This also means that certain things cannot be tested. Most of generator tests are garbage - turning on generator and running it without production load delivered over the transfer switch does not test anything other than that one can turn on a generator and run it. The problem typically happens not because the generator ( also is there the generator or the first and the second generator? Why is there no generator bank for a non monkey-sized company? ) does not start - the problem is because over time transfer switch develops a problem and unlike generators it is not possible to test a transfer switch where in the event of a test failure the customers won't lose power unless the data center is designed from the beginning to deliver A and B powers over separate circuits to every single customer and every single customer has per system ( not per rack ) transfer switches.
Of course it costs a lot more money, something that companies are reluctant to spend.
If someone doesn't know why a failure occured and they can't find out then they aren't looking hard enough
(In one case I experienced, the vendor reassured me that what I reported was not even technically possible -- until one of their engineers flew out and witnessed it firsthand.)
Usually.
Unless by stronger you mean some kind of a change in mentality over care for the service, but that would probably have a limited lifespan until it's back to normal.
If a service never fails then I don't know how well they can recover. I don't know anything about their failure mode.
In this case, I'm learning how OVH handles failure modes, how well they handle it, etc.
I can observe how they will treat such things in the future.
Thankfully many times we're reminded that there are good people out there working hard against difficult constraints and they finally get their chance to do things 'correctly' in the wake of the SHTF.
As a more technical user it's nice to have providers that give this information rather than the boilerplate "Issue with an upstream provider" over and over.
ovh.com looks down for me too.
You can check it's hosted by OVH:
$ whois $(dig sudokugarden.de +short)
Anybody knows ETA?
ETA: 15 min
Even if they didn't, clearly many services are up and running normally, so saying "all datacenters are down" is just a lie.
EDIT: Hosted in EU.
https://twitter.com/hashtag/OVHGATE?src=hash
I think we're now going to have to look into multi-provider options. The only way to be solidly up is to be hosted by more than one company at more than one data center.
I've also heard stories of billing nightmares where you get locked out of a cloud provider account, so that's another thing.
I guess this is already a reason by its own. It, among other problems, is what happens when we go from small "local" providers you can actually call to automated global providers that cannot provide immediate support even if they tried.
If your CD pipeline deploys corrupt apps, even your production database can get compromised,forcing a full restore. No matter how long that might take
Thankfully,I didn't have to witness that yet.
@DNS_BORAT https://twitter.com/DNS_BORAT
@InfoSecBorat https://twitter.com/InfoSecBorat
@KanbanBorat https://twitter.com/KanbanBorat
@mysqlborat https://twitter.com/mysqlborat
@NetEng_Borat https://twitter.com/NetEng_Borat
@secure_borat https://twitter.com/secure_borat
@SecurityBorat https://twitter.com/SecurityBorat
@Sysadm_Borat https://twitter.com/Sysadm_Borat
Opened up Age of Empires II....no connection. Go to website for game servers..."Our provider, OVH, is down...."
Go figure.
My data is with me in Europe and the company that has my data is in Europe with me too. If I was using DO or AWS, then my data may be in Europe but the control over the data is in the US, free for all to the three letter agencies and lacking privacy laws there.
Sure, it's less dramatic than AWS going down, but it still hits, and hard.
06:15 UTC SBG serves failed.
OVH network weathermap: http://weathermap.ovh.net
Btw. First post: https://news.ycombinator.com/item?id=15660524
My submit timestamp: 07:21:25
Your submit timestamp: 07:28:23
About the title. I did consider the title for a while, because I wasn't sure how bad the situation was. But from my own independent monitoring system I did see that RBX and SBG servers were unavailable. Of course I also did some basic trouble shooting and confirmation work before posting.
Btw. Right now, there's some network traffic present on SBG network. Let's hope that the systems are soon up'n'running.
Yeah, I expected this to clear up in a matter of minutes.
Now it seems to be a shitstorm of historic proportions...
Just in case you wonder why your sites don't work, even if you host them somewhere else.
My OVH servers in France are all inaccessible.