For me it was Revolut. No access to my main account, no withdrawals, no card payments since the morning. App doesn't work, there's no phone number to call, nobody to talk to. Dark side of neobanking, I guess.
The stand-in system has a limited subset of features, i.e. no bells and whistles, but enough that you can get any actual banking done.
(Not an ad / I have no connection to Monzo, I just had some appreciation for a bank doing this properly as normally banking IT is a disaster. Plus the idea of keeping a separate codebase for almost an entire bank's functionality seems very invested.)
I opened up a Revolut account a couple of years ago with the intention of switching my current account to it (i.e. wages going into it etc), but I got spooked after hearing that they weren't registered as a UK bank and thus subject to the various regulations and protections. I tried using Chase instead and have been happy with it so far.
I used to use Starling as my main and they were fine, until I started getting issues with random card declines due to "fraud prevention" and nobody to talk to. I switched to First Direct and like the peace of mind of having a human on the phone if needed (when I wanted to make a huge transfer for my house deposit it was nice to have!) and I still get perks like free foreign card transactions.
Yep, I probably won't bother switching to them now as Chase provides a good service as far as I know, though I do get a bit twitchy about its U.S. links. However, it's headquartered in the UK and is a division of J P Morgan Europe, so I hope that provides some level of distancing from the inevitable economic collapse of U.S. A.I. companies.
For what it's worth, JP Morgan specifically is probably the best positioned of all the major banks to endure any political and/or economic instability that might arise in the near-ish future. Not that this should necessarily drive your decision on who to bank with, but having spent the last decade or so working at various large US banks I have a lot of respect for the way JPMC operates
- The vast majority of their infra (at least as of ~3 years ago) is self-hosted in data centers that they own, and a lot of their other key cloud infra is/was multi-cloud (i.e. operations could fail over to GCP in the event of an AWS outage)
- Financially they have the healthiest balance sheet of any non-state-owned bank (like, in the entire world). Over the last ~20 years JPMC has acted as a sort of lender of last resort several times, most recently when they bought out First Republic.
- Politically, after navigating the aforementioned instability Jamie Dimon has almost carte blanche support from the shareholders, and everyone knows it. As a result, he can afford to be a lot less... compliant with the current US regime than other major players have been recently.
Thank you, this is great insider info. I only wish they had free checking accounts; they have all sorts of requirements in order to waive their monthly fee. Then I’d switch in a heartbeat because of your post.
Starling also has a disaster recovery system across cloud providers, although it is implemented differently to Monzo's (I used to work there on multi-cloud).
Multi-cloud is really tough, tbh. It's easier said than done, especially if you care about data consistency at all.
I have built these system for financial institutions since before cloud providers. It is hard/impossible if not well constrained at a level that supports data availability and consistency well (ie using a proper database with the correct setup across all locations with the underlying OS configuration/patches applied uniformly - the latter lesson cost IBM a lot of money due to system downtime for a financial institution that I brought back online). Many software vendors offering on-prem versions don’t know how this works well exactly, so I’ve had to help them along the journey and get them to sign-off for their contractual obligations to be upheld in the setup.
TLDR: I was called in several hours after the downtime started. I architected and built the full system so I knew things were incredibly consistent and how every layer worked (each layer of the software/database, load balancers, the quirky legacy protocols, network setup, etc). IBM was saying it was an issue on the financial institution side, which I validated was not the case practically first (things were down so I was given freedom by the executive team to do as I needed). Since I sanity checked the easy parts first and had high confidence this was something much more wild after scanning through more stack traces than a human should, I decompiled and reverse engineered their code to show the IBM teams where the bug was in their switch statements triggered by the Redhat patch difference and its dependencies that triggered the edge case that stopped the highly available cluster setup from functioning. I created a script to override the system properties being read in on startup to trick the switch statement, show it worked, and got the system back online. IBM made an official hot patch that fixed the issue officially along with paying a lot of money back to the financial institution for their troubles (ie the double contractual obligations for the premium IBM support with included SLA clauses that were overrun and the proprietary IBM software itself).
Help me understand, is the bad vibes you get from stories from the company culture? Or rather that the media put a target on their back (at at least in my opinion this is because they have a Russian born founder? Revolut is one of the best products I've ever used, I've honestly nothing but good things to say about them.
Its a great product when it works. When it doesn't, for example if they decide to freeze your money for no reason, it's not fun anymore. And there's plenty of reports of that happening to people.
You mean because they have automated AML/anti-fraud systems that all banks must have; every bank wrongly freezes accounts. To your mind, Revolut does that worse than others. I disagree, but without hard evidence, it's just our hunches at odds.
Yes, sure. But at my brick-and mortar bank I can go and talk to a flesh and bones human, instead of to an AI robot. Which is a much better service, when the anti-fraud thing does happen.
The revolut anti-fraud thing also seems to be less well done, being easier to trigger than other "real" banks.
I couldn't pay for groceries with Revolut in a store. Staff asked if I was using Revolut, but luckily adding a card from a different bank to Apple Pay was quick and painless (I had neither my wallet nor cash with me).
Another reminder that we need redundancy for all essential services that we rely on.
This is why we ought to have redundancy for critical services, I'm talking both from Revolut and from the client side. I have several cards in different banks for this situation exactly. And Revolut should be multi-zone for this situation exactly.
It's crazy. My main account is with Starling and I have a Lloyds account for a saving an account and some free cinema tickets. I don't think I've ever had an issue with Starling, it's great - Lloyds fails to sign in about 50% of the time consistently.
A lot of the information people are leveraging here comes from Downdetector which is mostly users reporting that stuff is down, not whether they are actually down. It's a really bad source.
It's a pretty good source ime (at least, massive spikes tend to correlate well with issues). Traditional organisations aren't as transparent as SF tech companies about service downtime. Plus, if the service isn't working for its users, I wouldn't say it's actually up.
It's a really bad source compared to what? Downdetector is not perfect, but is usually correct when I check it. Official status pages on the other hand are usually wrong when I check them.
“Correct” here presumably means “some other people are also experiencing problems.” For companies like large banks, with tens of millions of customers, that’s not necessarily very meaningful except for the people who happen to be experiencing the issue.
That's why downdetector shows a chart, which is basically a chart of how many people clicked "I'm having a problem with this service". Of course you get a small amount of people having issues, but if you suddenly see a big spike on the graph, it's a good sign that it's broken for many more people than usual.
Or at least: It's a sign that people were compelled to push a button.
I've seen it happen over and over again when a wireless provider like Verizon or a docsis provider like Spectrum has a widespread outage that is isolated to their service: People push the wrong button.
Verizon issue, other providers fine? Verizon spikes on downdetector, but so do reports of AT&T and T-Mobile problems.
Same with Spectrum: When it's screwed up for whatever reason, reports for other ISPs also spike.
This happens even when those other systems are completely unaffected and working just fine.
Downdetector does provide useful data, but it's not necessarily clean data that establishes anything clear on its own.
I've found that it's better than most service status pages -- Downdetector shows when users are reporting problems, while most service status pages only post significant confirmed outages, but there can still be service affecting issues that don't show up on the status page yet (or ever)
We are experiencing an issue with multiple services in
europe-west2 beginning on Monday, 2026-10-19 23:08 PDT.
Due to a power event we had suffered an outage within
europe-west2-b zone. The power event is mitigated. Partial
mitigation to infrastructure services will be observed within the
next 30 minutes and the remaining services will be mitigated there
after.
So some kind of power failure happened to bring down one zone. Unfortunate, but this kind of stuff happens. That is why all the cloud providers have multiple zones, and I'm amazed the big companies reported as down weren't using multiple zones.
(Also reporting the times in PDT is quite unhelpful for users of this zone who are likely based in the UK or EU)
> I'm amazed the big companies reported as down weren't using multiple zones
Much as with backups - if you don't test them, they don't exist.
Everywhere I've worked has had multi-AZ infrastructure. I have never worked anywhere where the failure of an AZ has been simulated regularly, with second-order effects mapped out.
We set up our applications and services to be multi-AZ natively. It's not particularly difficult with how many AWS services are regional or global (Admittedly, I still expect a potential DNS failure to cause issues if and when that occurs). However we are just part of a workflow, we integrate with other applications that are not multi-AZ and would be impacted. That's just out of our hands.
It's like our on-prem systems were ordered to plan out and exercise DR plans to account for a data center going offline, but we can't exercise a DR plan fully if part of the plan is 'Adapt to how application X, which is hosted in the same datacenter, also chose to fail over.' We do not own application X. There is no appetite in the business to co-ordinate the DR exercise across the entire company at the same time.
We can't even do it all in a lower environment because there are dependencies on systems that are shared across environments. Some massive, redundant filesystem that is sync'd across our data centers, but the sync is a boolean. One datacenter instance of it is the primary and the others are read-only until they get promoted. We have one of these and lower environments are just a different root folder on the system with different access patterns, but they can't make DataCenter A the primary in prod and DataCenter B the primary in UAT.
Question who's never relied on this: how do you test whether your multi-AZ setup can withstand an AZ dropping? It sounds like your third party apps make it kind of pointless to test anyway, but I'm curious!
Ops type here, generally you have to fail an AZ and see what happens. If you have Dev/Staging environment that is exactly like production with plenty of tests, you can use that. I've seen only one environment actually like that though.
Since most companies would have to do it in production and they don't want to have customer impacting outage, they tend to do it during an actual outage.
Thans! So do you have a staging environment for everything, then cut off the network or kill the instances, Aurora DB, etc.? Simulate some load while its going on, and then check for data loss? Then if that works try it in prod?
It seems like a lot of work, and it seems that as architectures evolve it would be hard to ensure that the multi-AZ failover works well.
Yes, which means it's expensive which means companies don't usually pay for it so we should name and shame companies who aren't willing to pay that expense just to pad their bottom line.
I'd imagine (maybe optimistically) most companies do think about using multiple zones. And even test for DR. Regularly. And successfully. Problem is, all that works just fine on a sunny Monday morning. Then shit hits the fan and all of a sudden everyone is forced to, at the same time. It's a different kettle of fish
I'm not saying it's the case here, but sometimes the cloud providers themselves have inter-zone dependencies, or even inter-region dependencies that can cause outages in other zones and regions.
If you mean Google: Yes, extensive. One of the techniques they use is DiRT (Disaster Recovery Testing) which is a scheduled time-frame in which Google purposefully breaks systems that are exceeding their SLAs to verify that automated recovery executes as intended. In general, a DiRT test would catch something like "This service has a key dependency on us-east-1.
Now, in the past (also a Xoogler)... Some pieces of Cloud were moving at a cadence that they got away with a bit of a relaxed DiRT awhile back, so if someone tells me "I remember when a us-east-1 outage took down a key Cloud service" I would believe it. DiRT is also planned testing, it's not slapping a chaos monkey on the cluster fabric and crossing one's fingers, so it's entirely possible for key outage modalities to get missed (engineers are human and even though Google aims for a "no haunted closets" engineering practice, people develop a sixth sense of where the weak parts are of your infra and it takes courage to commit to pushing on those until they break).
honestly more often than not the customer has a weird setup that doesn't handle it more than it being a GCP issue. I've seen some customers with awesomely redundant systems with their...cache all in one zone.
Also keep in mind GCP has a global network so it's possible some other regions are seeing disconnects/packet drops, though the source of the issue is contained to a single datacenter.
People who don't work on cloud services are usually very sloppy with their wording which I think is leading to some disconnect here. A "zone" is not really the unit of availability for most things people consider the "cloud" for hyperscalers. A zone is like 1-3 DCs and you are very vulnerable to stuff like localized power events. For that reason most services are not offered at the zonal level and if they are you get big disclaimers saying so and lower 9s of SLA.
Most things are at a regional level which is multiple zones. Your AWS us-east-1 is one such region. A lot of datacenters spread across a decently large physical area have to have an issue for a whole region to start having failures.
For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)
This is why Microsoft calls it an Availability Zone, each zone is made up of one or more data centers equipped with entirely independent power, cooling, and networking infrastructure. One zone should not knock out other zones. Zones are also often nearby each other just enough that they have very efficient fiber optic connections between them.
That said, that's how Microsoft handles it, I'm not sure how GCP compares.
It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events.
GCP has some regions where all of the AZs are located in the same physical building. AFAIK they’ve stopped doing this in newer regions and have been migrating existing regions to have physically separate sites for each AZ.
AWS has always guaranteed that different AZs are physically separate sites, but that hasn’t always been the case with GCP.
You are generally right, except that you don't seem to know the history of major AWS outages. Failures in AWS in us-east-1, which is the first AWS cloud region, often cause cascading worldwide failures in the whole of AWS - especially failures related to DNS and to DynamoDB, if I remember correctly. That is, AWS workloada running in, say, eu-west-1 (and in fact in the whole of AWS) often start failing or at least having greatly degraded performance/availability because of a failure in AWS infra in their core region.
If AWS themselves can't handle cross region isolation and availability well, my expectations for most companies running on top of AWS are quite modest.
It's because AWS runs their control plane out of us-east-1, particularly lambda. There are also some very popular services that can only be run out of us-east-1. It's not really a technological constraint, it's a design decision.
AWS does not run “their control plane” out of us-east-1. Every single service has its own control plane independently hosted in its own region, with the exceptions of IAM, Route53, Cloudfront, and a small number of others like Global Accelerator.
IAM is usually the culprit when us-east-1 issues affect other regions, but Lambda runs independently in every region.
Explain why how why when aws lambda when out in us-east-1 a year or so ago every other region had intense issues they couldn’t quickly recover from? and their incident report explained this exact fact? where’s your info coming from? I follow this stuff very closely because my livelihood depends on it.
"While the event started at 11:48 PM PDT on October 19 and ended at 2:20 PM PDT on October 20, there were three distinct periods of impact to customer applications. First, between 11:48 PM on October 19 and 2:40 AM on October 20, Amazon DynamoDB experienced increased API error rates in the N. Virginia (us-east-1) Region. Second, between 5:30 AM and 2:09 PM on October 20, Network Load Balancer (NLB) experienced increased connection errors for some load balancers in the N. Virginia (us-east-1) Region. This was caused by health check failures in the NLB fleet, which resulted in increased connection errors on some NLBs. Third, between 2:25 AM and 10:36 AM on October 20, new EC2 instance launches failed and, while instance launches began to succeed from 10:37 AM, some newly launched instances experienced connectivity issues which were resolved by 1:50 PM."
At this point we’re playing semantic nittery. us-east-1 is their control plane. critical services that only run in us-east-1 were impacted badly. this has a cascading effect to services that run in other regions that rely on those services - even if those services themselves aren’t technically out, it’s a cascading effect. the whole fucking internet went down that day.
my original claim was that their control plane runs in us-east-1. this is a fact.
Neither the post you linked nor the actual AWS post mortem say what you are saying. The Plainenglish link is wrong in some ways, but you’re also misunderstanding what is there.
There was a “control plane failure” in us-east-1 that affected us-east-1. It did not affect other regions. There is nothing in the link you posted that says it affected other regions.
The “whole fucking internet went down” because us-east-1 is big enough that even if it’s the only region that goes down, it seems like the entire internet did.
AWS does not have a single “control plane”, so no it’s not a fact to say “their control plane” is in any single place.
They had a comparable problem in europe-west9 in 2023. Apparently they actually run into trouble if they lose a whole zone through a physical event. It's either thundering herd or simply insufficient capacity in the other two zones to take over all the load from the third zone.
> For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)
Why do you say it's not supposed to happen? It's the predictable outcome of not having unlimited scale. The hyperscalers are big, but to be able to accommodate a tectonic shift in demand in a very compressed amount of time is very costly--costs that will be passed to their customers--if it can be done at all given current constraints. Today's hardware crunch, even for hyperscalers, is very real; customers are hoovering up spare capacity like never before.
As a former guidance counselor (specialist SA) for AWS, I regularly told customers that they shouldn't expect to be able to pack up and relocate into a new AZ once an incident commenced. They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.
> They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.
Do you blame them? You not only have to pay double for the infra but also be charged for inter-region data transfers. And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.
> You not only have to pay double for the infra but also be charged for inter-region data transfers.
Well, yeah. Redundancy isn't free. I don't see how this is controversial.
> And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.
What do you mean, "IAM doesn't work"? IAM has always worked, even during control plane impairments. You cannot depend on control plane operations to work during incidents; you should plan in advance for failure and have your orchestration already set up so that you don't have to touch a thing when you have to weigh away from an AZ or region.
This points to what I keep repeating about the control plane vs. the data plane. IAM faulted to a steady state. While you couldn’t make changes during the impairment, it didn’t cause downstream failures like blocking rule evaluations (or worse, failing open and allowing denied operations to be permitted). That’s the important part.
As the page you linked to stated:
> IAM (Identity and Access Management) configuration was unavailable, preventing updates to roles and policies, but cached credentials allowed most running workloads to continue.
Perhaps I should have been clearer about what I meant about IAM still working. That's on me.
1. AWS has no cross-regional or cross-zonal data plane[1] dependencies. Customers might couple them, in violation of best practices, but AWS itself does not. It's strictly forbidden.
2. That article you linked to does not support your claim. That is a report of a single-zone incident impacting us-east-1 only. Five other AZs in the same region were still fully operational.
[1] These are not the same as control plane dependencies: you might not be able to make certain mutations, but everything should still work in the steady state.
While what you say is true for 1, I wholly disagree that it is always customers that do the coupling.
One more than one occasion there are AWS URLs, including redirects inside the console, that include US East 1 in the URL that were only spotted during an outage. Manually changing it to an different region allowed us to get around it but to wholely blame customers when you don't know the context is bad form.
> One more than one occasion there are AWS URLs, including redirects inside the console, that include US East 1 in the URL that were only spotted during an outage. Manually changing it to an different region allowed us to get around it
Yeah, that tracks. The Console has often been wonky like that.
> I've seen US East 1 outages cause autoscaling issues in other regions including failing to start new EC2s
Do you mean scaling failed due to insufficient capacity, or something else? ICE errors I get (thundering herd of attempted capacity shifting); but otherwise, that sounds implausible, and I respectfully believe you are mistaken about cause and effect. Happy to evaluate a causal chain of evidence that proves you correct, though.
Often there are irritating legal geographic limitations on where cloud hardware can exist for a given country. I can understand why, but there is this particular drawback of having all eggs baskets close together.
GCP is kind of odd in that some hardware is only available in certain zones, and some hardware is technically available in multiple zones but the capacity is not evenly spread. I've had to implement infrastructure code for GCP that interrogated hardware availability by region and zone to place VMs, trading off multi-AZ availability where it made sense.
Yes. The area on the Venn diagram where services, hardware and regulatory compliance intersect is surprisingly small.
I had a client that was contracting with a very large bank and there were only something like four zones globally that we could possibly deploy in to.
Google also is unlike AWS in that their zone abstraction is very virtual, and two zones might be in the same physical building. If you have requirements that your deployment zones be physically separated by some minimal distance then it can make things pretty challenging.
And afaik there is no way to get an exact physical location (DC, Hall, rack) for the hardware you are actually using. If there was you could at least verify if there wasn't some kind of hiddent dependency.
There's a GCP API that lists the hardware advertised in each zone as a starting point. Then you have to actually attempt to provision hardware in each zone to try to get a sense of what capacity is available there. Combine the two and you can kind of map out a topology.
Other clouds are similarly odd (or even worse). I have experienced in Azure that if you provision a resource group with some basic SKU for a given service, the only way to later upgrade to the latest and greatest SKU is to tear down the whole resource group and re-provision everything from scratch. All attempts at scale-up in place will fail with cryptic errors. Seems like you are assigned in an old corner of the DC and can't get out of it.
Extra fun if there is a key vault involved, then you have to manually delete every key in it, otherwise you can't create new keys with the same names in the new vault.
It comes across as really uncaring to the probably 99.9999% of people outside of the US who have no idea offhand what UTC offset it is, it almost feels hostile.
If they at least included “(UTC +<amount>)” after PDT, while still a bit annoying, at least people could work out what time that is without having to lookup PDT from somewhere first.
It shouldn't matter. Florence is antiopodal to somewhere slightly east of the Chatham Islands; on the same longtitude as the Diomede Islands.
The argument for Florence Mean Time is that when you spread the map out, you get a nice split of the Pacific on either side. It's quite beautiful actually.
- Chuktoka isn't split and stays all together on the right side of the map
- Zealandia all stays together on the right side, along with Samoa and Tokelau
It's also kinda poetic having the 'centre of time' being basically equidistant from Rome and Carthage.
MTV time is fine if it's consistent and says that it's MTV time. A lot of stuff at Google doesn't clarify the time zone, and there are some things that are UTC.
When I started at a large company in 2020, I began to notice that they advertised all times: meetings, deadlines, events, in terms of "EST". Even when it was Daylight Savings Time.
So, being in Arizona where it's never DST, I called them out on it, and I continually bothered them when they used this "EST" notation. I told them that they were simply wrong to use it during Daylight Savings Time. I told them that MST and Eastern Time did not have a year-round relationship. They didn't care.
It turns out that a lot of very large corporations use "EST" (and other "standard time" notation) all year round. Including FedEx and UPS, for some reason.
It is obviously ridiculous, when you could simply use "ET" instead, and always be right, no matter what your DST status is. But they insisted on using this "EST" thing.
I had a conversation or two with some LLMs about it, and they rationalized it. But it is crazy to subject people to this "corporate fiction" that is obviously wrong/inaccurate for an entire half of the year, and it leads to a lot of problems.
The fact they didn't immediately fail over doesn't mean they didn't had a plan.
The plan might've just been "if it gets bombed, we restore backups into other zone". Just waiting for it to fix itself for few hours mightv'e been acceptable risk.
> ... I'm amazed the big companies reported as down weren't using multiple zones.
Yeah well, they drank the cloud kool-aid. The promise was that they'd go "cloud", and that everything would be taken care of.
Turns out: availability is worse than when those companies used to self-host and major cloud outages are happening all the time (at this point it's basically a meme).
And now the promise that "the cloud is simpler than doing it yourself" turns out to be not true?
I'm not surprised at all: these companies have been sold a lie.
>> That is why all the cloud providers have multiple zones....and I'm amazed the big companies reported as down weren't using multiple zones.
Well then you should know that a Google Zone is not the same as an AWS AZ and many of Google regions are just firewalls across the same data center.... Despite all their SREs and Books and Phds...they dont know how to do infrastructure.
For sure Google zones aren't the same as AWS AZs, and I remember complaining about this years ago during that fire in a France DC. But you also said their regions are firewalls across a datacenter, suggesting a multi-region DC, which I'm pretty sure they don't have.
From the documentation of the Google Cloud Platform.
> To protect against the loss of an entire region due to natural disaster, have a disaster recovery plan and know how to bring up your application in the unlikely event that your primary region is lost.
> The regions Stockholm, Osaka, and Montreal have three zones housed in one or two physical data centers.
Interesting take on the timezone. I imagine a lot of people have their data there for only residency reasons. Even if it is GMT people still use something like worldtimebuddy to sync up.
> Makes me wonder why an EU bank is keeping data outside the EU (from my understanding, only the London data center is down)?
Let me tell you a story about an idiot politician who woke up one day and thought Brexit would be a fantastic idea, and then the stupid dim-witted "ultra-hard brexit" manner in which it was implemented by a certain Mr Johnson.
The same reason UK people can't open EU bank accounts is the same reason EU banks have separate operating companies and infrastructure in the UK.
Given they operate in a heavily-regulated environment it almost certainly made life easier for them to keep UK stuff in the UK because they had no idea how (or if) the regulatory environment would ever progress beyond ultra-hard brexit.
If you log in onto machine and make an error, a machine is down.
If you put everything into configuration management, make scalable, redundant and interconnected servers and services built on top of it, you unlock amazing possibility of one error bringing hundreds if not thousands machines down.
Nice lil reminder that adding more things for redundancy only increases it if they are entirely unrelated to eachother and independent.
Which INCLUDES not running same software. You can have redundant switches, doesn't matter if they have software bug that triggers on all of them (or just one in a way that prevents network from working correctly).
You can have database cluster that can tolerate node dying just fine but still be brought down by a bug despise redundancy.
It's not "going down", the servers just went on strike in solidarity with europe-west9 (Paris) to protest against too much server uptime past 5 PM, thermal abuse, low voltages and mass PID killings. Americans won't understand. Additionally the failover data packets got stuck in customs for inspection and had to form a line back to Dover. So yeah, bad day innit.
It's a little worrying that banking with an institution from one of the world's financial capitals is so vulnerable. I would've thought London would've been priding itself on security, stability, safety, etc. Confidence in their systems is their bread and butter
Afaik, even Russia and Ukraine aren't having such issues while dealing with data centres getting droned, lol.
> If one power failure can take down more than 5 banks, what you are describing is a valuable target. It's ironic that the cloud services that promised to harden systems have delivered another layer of vulnerability.
Fair, but if all of those 5 banks bought their own servers and put them into a datacenter that happens to be same one because it's the fancy one that meets all the bank regulations and required certs, that datacenter going down would still make 5 banks go down.
You would got some resiliency by fact only datacenter related issues (power, internet access) would affect it and not say "cloud company throwing AI at their code that proceeds to explode" (which already happened), but still, if you want to fix that, cloud or not, you have to have multiple locations/regions and that's both costly and difficult.
Anyone have a read on the implications for Digital Operational Resilience Act (DORA) compliance here? Google Cloud seems to be a critical third-party vendor for these financial institutions. The top penalties for DORA can be very expensive (% of worldwide revenue fines).
Depends on where the EU-licensed institution is headquartered, as monitoring is done by the states. But I'll say it's very likely that there will be an angry call or a review at most. Some manager will have to write up how they want to mitigate problems in the future, and this plan is maybe implemented in the next few years. If this is the first problem of an institution, a fine is rather unlikely.
Generally with acts like these: You don't have to be fully compliant in practice. You need to know how compliant you are, have a plan to improve things, and prove that you are working on it. But if you are "close enough" (depending on who audits you), non-compliance isn't a direct route to fines.
185 comments
[ 0.24 ms ] story [ 12.8 ms ] threadThey have a stand-in system that uses a separate codebase and separate cloud provider. They have a blog post about it here: https://monzo.com/blog/tolerating-full-cloud-outages-with-mo...
The stand-in system has a limited subset of features, i.e. no bells and whistles, but enough that you can get any actual banking done.
(Not an ad / I have no connection to Monzo, I just had some appreciation for a bank doing this properly as normally banking IT is a disaster. Plus the idea of keeping a separate codebase for almost an entire bank's functionality seems very invested.)
I have always got bad vibes from Revolut...
I opened up a Revolut account a couple of years ago with the intention of switching my current account to it (i.e. wages going into it etc), but I got spooked after hearing that they weren't registered as a UK bank and thus subject to the various regulations and protections. I tried using Chase instead and have been happy with it so far.
They are reigistered now
- The vast majority of their infra (at least as of ~3 years ago) is self-hosted in data centers that they own, and a lot of their other key cloud infra is/was multi-cloud (i.e. operations could fail over to GCP in the event of an AWS outage)
- Financially they have the healthiest balance sheet of any non-state-owned bank (like, in the entire world). Over the last ~20 years JPMC has acted as a sort of lender of last resort several times, most recently when they bought out First Republic.
- Politically, after navigating the aforementioned instability Jamie Dimon has almost carte blanche support from the shareholders, and everyone knows it. As a result, he can afford to be a lot less... compliant with the current US regime than other major players have been recently.
Multi-cloud is really tough, tbh. It's easier said than done, especially if you care about data consistency at all.
It sounds like that story could make a great blog post
Help me understand, is the bad vibes you get from stories from the company culture? Or rather that the media put a target on their back (at at least in my opinion this is because they have a Russian born founder? Revolut is one of the best products I've ever used, I've honestly nothing but good things to say about them.
The revolut anti-fraud thing also seems to be less well done, being easier to trigger than other "real" banks.
Another reminder that we need redundancy for all essential services that we rely on.
You will probably have issues logging into Llyods even when the cloud is back up.
The usability of their mobile app is atrocious though. Good thing they have a website.
A lot of the information people are leveraging here comes from Downdetector which is mostly users reporting that stuff is down, not whether they are actually down. It's a really bad source.
If I had a nickel for every solid-green status page I shook my fist at while the actual site faceplanted on every request...
"We would like to apologise for any inconvenience and continue to monitor the situation, which is also affecting other organisations."
I've seen it happen over and over again when a wireless provider like Verizon or a docsis provider like Spectrum has a widespread outage that is isolated to their service: People push the wrong button.
Verizon issue, other providers fine? Verizon spikes on downdetector, but so do reports of AT&T and T-Mobile problems.
Same with Spectrum: When it's screwed up for whatever reason, reports for other ISPs also spike.
This happens even when those other systems are completely unaffected and working just fine.
Downdetector does provide useful data, but it's not necessarily clean data that establishes anything clear on its own.
(Also reporting the times in PDT is quite unhelpful for users of this zone who are likely based in the UK or EU)
Much as with backups - if you don't test them, they don't exist.
Everywhere I've worked has had multi-AZ infrastructure. I have never worked anywhere where the failure of an AZ has been simulated regularly, with second-order effects mapped out.
It's like our on-prem systems were ordered to plan out and exercise DR plans to account for a data center going offline, but we can't exercise a DR plan fully if part of the plan is 'Adapt to how application X, which is hosted in the same datacenter, also chose to fail over.' We do not own application X. There is no appetite in the business to co-ordinate the DR exercise across the entire company at the same time.
We can't even do it all in a lower environment because there are dependencies on systems that are shared across environments. Some massive, redundant filesystem that is sync'd across our data centers, but the sync is a boolean. One datacenter instance of it is the primary and the others are read-only until they get promoted. We have one of these and lower environments are just a different root folder on the system with different access patterns, but they can't make DataCenter A the primary in prod and DataCenter B the primary in UAT.
Since most companies would have to do it in production and they don't want to have customer impacting outage, they tend to do it during an actual outage.
It seems like a lot of work, and it seems that as architectures evolve it would be hard to ensure that the multi-AZ failover works well.
For example, AWS us-east-1 outages have caused global outages on multiple occasions, such as https://www.networkworld.com/article/4168878/aws-hit-by-us-e...
Disclaimer: I work for Google Cloud.
If you mean Google: Yes, extensive. One of the techniques they use is DiRT (Disaster Recovery Testing) which is a scheduled time-frame in which Google purposefully breaks systems that are exceeding their SLAs to verify that automated recovery executes as intended. In general, a DiRT test would catch something like "This service has a key dependency on us-east-1.
Now, in the past (also a Xoogler)... Some pieces of Cloud were moving at a cadence that they got away with a bit of a relaxed DiRT awhile back, so if someone tells me "I remember when a us-east-1 outage took down a key Cloud service" I would believe it. DiRT is also planned testing, it's not slapping a chaos monkey on the cluster fabric and crossing one's fingers, so it's entirely possible for key outage modalities to get missed (engineers are human and even though Google aims for a "no haunted closets" engineering practice, people develop a sixth sense of where the weak parts are of your infra and it takes courage to commit to pushing on those until they break).
Also keep in mind GCP has a global network so it's possible some other regions are seeing disconnects/packet drops, though the source of the issue is contained to a single datacenter.
When did that change? I'm glad it did, but it seems like that was long in coming.
Most things are at a regional level which is multiple zones. Your AWS us-east-1 is one such region. A lot of datacenters spread across a decently large physical area have to have an issue for a whole region to start having failures.
For GCP it seems like one power event in a DC led to a thundering herd on the rest of the region which was the actual issue here. (which is a pretty significant problem and isn't supposed to happen)
That said, that's how Microsoft handles it, I'm not sure how GCP compares.
It is an interesting conundrum though, what if all your nearby availability zones come down at once due to power loss? I wonder which major cloud provider has the best setup for these sort of events.
AWS has always guaranteed that different AZs are physically separate sites, but that hasn’t always been the case with GCP.
If AWS themselves can't handle cross region isolation and availability well, my expectations for most companies running on top of AWS are quite modest.
(who would be motivated to check if anything is restricted to only us-east-1)
IAM is usually the culprit when us-east-1 issues affect other regions, but Lambda runs independently in every region.
here’s one third party, it was pretty well documented as a control plane failure in us-east-1
https://aws.amazon.com/message/101925/
This, too, was contained to us-east-1:
"While the event started at 11:48 PM PDT on October 19 and ended at 2:20 PM PDT on October 20, there were three distinct periods of impact to customer applications. First, between 11:48 PM on October 19 and 2:40 AM on October 20, Amazon DynamoDB experienced increased API error rates in the N. Virginia (us-east-1) Region. Second, between 5:30 AM and 2:09 PM on October 20, Network Load Balancer (NLB) experienced increased connection errors for some load balancers in the N. Virginia (us-east-1) Region. This was caused by health check failures in the NLB fleet, which resulted in increased connection errors on some NLBs. Third, between 2:25 AM and 10:36 AM on October 20, new EC2 instance launches failed and, while instance launches began to succeed from 10:37 AM, some newly launched instances experienced connectivity issues which were resolved by 1:50 PM."
my original claim was that their control plane runs in us-east-1. this is a fact.
There was a “control plane failure” in us-east-1 that affected us-east-1. It did not affect other regions. There is nothing in the link you posted that says it affected other regions.
The “whole fucking internet went down” because us-east-1 is big enough that even if it’s the only region that goes down, it seems like the entire internet did.
AWS does not have a single “control plane”, so no it’s not a fact to say “their control plane” is in any single place.
Why do you say it's not supposed to happen? It's the predictable outcome of not having unlimited scale. The hyperscalers are big, but to be able to accommodate a tectonic shift in demand in a very compressed amount of time is very costly--costs that will be passed to their customers--if it can be done at all given current constraints. Today's hardware crunch, even for hyperscalers, is very real; customers are hoovering up spare capacity like never before.
As a former guidance counselor (specialist SA) for AWS, I regularly told customers that they shouldn't expect to be able to pack up and relocate into a new AZ once an incident commenced. They rarely listened, as it was often deemed too costly to maintain AZ redundancy (let alone regional redundancy) in advance of an incident.
Do you blame them? You not only have to pay double for the infra but also be charged for inter-region data transfers. And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.
Well, yeah. Redundancy isn't free. I don't see how this is controversial.
> And, when the worst comes, you realize it was all for nothing as IAM doesn't work, for example.
What do you mean, "IAM doesn't work"? IAM has always worked, even during control plane impairments. You cannot depend on control plane operations to work during incidents; you should plan in advance for failure and have your orchestration already set up so that you don't have to touch a thing when you have to weigh away from an AZ or region.
What do you mean, "IAM always worked"? The last failure was very recent, only a year ago[0]
[0] https://www.dataminded.com/resources/how-to-protect-against-...
As the page you linked to stated:
> IAM (Identity and Access Management) configuration was unavailable, preventing updates to roles and policies, but cached credentials allowed most running workloads to continue.
Perhaps I should have been clearer about what I meant about IAM still working. That's on me.
2. That article you linked to does not support your claim. That is a report of a single-zone incident impacting us-east-1 only. Five other AZs in the same region were still fully operational.
[1] These are not the same as control plane dependencies: you might not be able to make certain mutations, but everything should still work in the steady state.
One more than one occasion there are AWS URLs, including redirects inside the console, that include US East 1 in the URL that were only spotted during an outage. Manually changing it to an different region allowed us to get around it but to wholely blame customers when you don't know the context is bad form.
Yeah, that tracks. The Console has often been wonky like that.
> I've seen US East 1 outages cause autoscaling issues in other regions including failing to start new EC2s
Do you mean scaling failed due to insufficient capacity, or something else? ICE errors I get (thundering herd of attempted capacity shifting); but otherwise, that sounds implausible, and I respectfully believe you are mistaken about cause and effect. Happy to evaluate a causal chain of evidence that proves you correct, though.
https://news.ycombinator.com/item?id=50163858
I had a client that was contracting with a very large bank and there were only something like four zones globally that we could possibly deploy in to.
Google also is unlike AWS in that their zone abstraction is very virtual, and two zones might be in the same physical building. If you have requirements that your deployment zones be physically separated by some minimal distance then it can make things pretty challenging.
How do you to do that ?
Extra fun if there is a key vault involved, then you have to manually delete every key in it, otherwise you can't create new keys with the same names in the new vault.
If they at least included “(UTC +<amount>)” after PDT, while still a bit annoying, at least people could work out what time that is without having to lookup PDT from somewhere first.
Southwest airlines does it.
Rackspace used to do it, to the point of needing to ask them pretty please stop provisioning hardware on central time. Obnoxious.
The argument for Florence Mean Time is that when you spread the map out, you get a nice split of the Pacific on either side. It's quite beautiful actually.
- Chuktoka isn't split and stays all together on the right side of the map - Zealandia all stays together on the right side, along with Samoa and Tokelau
It's also kinda poetic having the 'centre of time' being basically equidistant from Rome and Carthage.
It doesn't sound like it.
So, being in Arizona where it's never DST, I called them out on it, and I continually bothered them when they used this "EST" notation. I told them that they were simply wrong to use it during Daylight Savings Time. I told them that MST and Eastern Time did not have a year-round relationship. They didn't care.
It turns out that a lot of very large corporations use "EST" (and other "standard time" notation) all year round. Including FedEx and UPS, for some reason.
It is obviously ridiculous, when you could simply use "ET" instead, and always be right, no matter what your DST status is. But they insisted on using this "EST" thing.
I had a conversation or two with some LLMs about it, and they rationalized it. But it is crazy to subject people to this "corporate fiction" that is obviously wrong/inaccurate for an entire half of the year, and it leads to a lot of problems.
The plan might've just been "if it gets bombed, we restore backups into other zone". Just waiting for it to fix itself for few hours mightv'e been acceptable risk.
1) the ability to view everyone's bank data, in secret
2) the ability to change EU bank data at will, in secret
(see US Cloud act)
There are much worse things you can do to data than bomb it.
Yeah well, they drank the cloud kool-aid. The promise was that they'd go "cloud", and that everything would be taken care of.
Turns out: availability is worse than when those companies used to self-host and major cloud outages are happening all the time (at this point it's basically a meme).
And now the promise that "the cloud is simpler than doing it yourself" turns out to be not true?
I'm not surprised at all: these companies have been sold a lie.
Well then you should know that a Google Zone is not the same as an AWS AZ and many of Google regions are just firewalls across the same data center.... Despite all their SREs and Books and Phds...they dont know how to do infrastructure.
"The regions Stockholm, Osaka, and Montreal have three zones housed in one or two physical data centers." - https://docs.cloud.google.com/docs/geography-and-regions
"This means the multiple zones within each of these regions may not be located in physically separate data center campuses." - https://docs.cloud.google.com/backup-disaster-recovery/docs/...
"A region is a set of buildings operated by Google in a geographic location, such as a data center campus." - https://cloud.google.com/vpc/network-pricing
> To protect against the loss of an entire region due to natural disaster, have a disaster recovery plan and know how to bring up your application in the unlikely event that your primary region is lost.
> The regions Stockholm, Osaka, and Montreal have three zones housed in one or two physical data centers.
See here: https://docs.cloud.google.com/docs/geography-and-regions
2023 europe-west9 incident: https://www.datacenterdynamics.com/en/news/water-leak-at-par...
Can someone ask Claude, Codex, Copilot, Antigravity, Devin to resolve this outage?
No Googling or help from humans this time.
Let me tell you a story about an idiot politician who woke up one day and thought Brexit would be a fantastic idea, and then the stupid dim-witted "ultra-hard brexit" manner in which it was implemented by a certain Mr Johnson.
The same reason UK people can't open EU bank accounts is the same reason EU banks have separate operating companies and infrastructure in the UK.
Given they operate in a heavily-regulated environment it almost certainly made life easier for them to keep UK stuff in the UK because they had no idea how (or if) the regulatory environment would ever progress beyond ultra-hard brexit.
Russia woke up
- There's AWS
- There's Azure
- There's everything else
And I don't even use Azure, I just hear it's good (and they don't charge cross-AZ traffic)
AWS behind, but with us-east-1 outages.
Azure is extremely far behind those two, and mostly for non-tech legacy shops that got suckered.
I won't touch oracle cloud.
If you put everything into configuration management, make scalable, redundant and interconnected servers and services built on top of it, you unlock amazing possibility of one error bringing hundreds if not thousands machines down.
Nice lil reminder that adding more things for redundancy only increases it if they are entirely unrelated to eachother and independent.
Which INCLUDES not running same software. You can have redundant switches, doesn't matter if they have software bug that triggers on all of them (or just one in a way that prevents network from working correctly).
You can have database cluster that can tolerate node dying just fine but still be brought down by a bug despise redundancy.
Sounds like most of these companies were not designed to handle outages in a single availability zone, and so they went down.
Afaik, even Russia and Ukraine aren't having such issues while dealing with data centres getting droned, lol.
It's really quite poor performance.
The government should be resilience testing everything. Did Covid teach no lessons? Is the defence budget just for shits and giggles?
Will be a great day when Europe can finally say goodbye to the America taxes on most digital payments.
Not having issues is overstatment, but Ukraine has a cheat of having a backup DC on NATO soil, which helps a lot.
Fair, but if all of those 5 banks bought their own servers and put them into a datacenter that happens to be same one because it's the fancy one that meets all the bank regulations and required certs, that datacenter going down would still make 5 banks go down.
You would got some resiliency by fact only datacenter related issues (power, internet access) would affect it and not say "cloud company throwing AI at their code that proceeds to explode" (which already happened), but still, if you want to fix that, cloud or not, you have to have multiple locations/regions and that's both costly and difficult.
They need to talk about some updates to the maintenance interact pricing...
Generally with acts like these: You don't have to be fully compliant in practice. You need to know how compliant you are, have a plan to improve things, and prove that you are working on it. But if you are "close enough" (depending on who audits you), non-compliance isn't a direct route to fines.