I wonder how they ensure durability. Is it possible that a cell going down would roll back a payment after it has occurred. Or do they depend on a non cell database?
American Express tech is some of the worst in the world among big companies. All of the value in the company is just in the branding. They put some work into the mobile app and the website, but other than that, its a facade.
Nobody uses Amex for payments, so the system isn't ever under high load.
Just kidding!
I find the idea quite good, and have to assume that the amount of payment fails they experience due to partitions/outages isn't very high and that the post-payment reconciliation and reclamation process gives them the liberty to rank availability a bit higher than correctness.
One thing that looked a bit shaky was the interplay between the global transaction router's state of knowing which cells can handle a particular payment and the asynchronous distribution of the "failover data", which I presume it needs to know to route correctly. To me that seems to create a window where it might route to the wrong cell due to an outdated routing state.
It also doesn't go into the HA setup of the global transaction router itself.
Ah yes, the financial services company that runs a travel agency, allows me to book my hotel and rental car weeks in advance, registers a hold for incidentals for both the hotel and car when I check in, then blocks the card when I try to buy dinner that night in that same hotel due to fraud detection.
Last week it required me to take pictures of my face from multiple angles to regain membership privileges. I suspect this may be part Palantir data collection and part Peter Thiel dating service.
Because of the title I was expecting to read about doing payments with a distributed network, like a terrorist cell network, or something like Hawala. Not (as I infer from other comments) Amex using multiple independent systems.
This isn't about payment technologies, it's not about isolating transactions, it's about scaling the middle layer. What's worse it's not even explained what middle layer does.
No info on how routing works, no info on data synchronization.
Folks just learning Kubernetes and write extremely abstract stuff.
microservices / clusters / zones - really all of these are other "cell-based" architectures as well. there is absolutely no written rule that a microservice was just an API or a singular service, it basically can be a independent instance that is testable/usable/gives value on itself.
that said: still a nice write up, learning about some of the architectural choices that AMEX makes is definitely insightful (and relavent/useful to what i am working on right now as well!)
Some of it sounds like it reinvented Erlang supervision trees https://learnyousomeerlang.com/supervisors. As a joke there we’re calling gen_severs “nanoservices”. Granted, that was mostly when microservices were the hot new thing.
As described, if one "cell" crashes, you re-route/retry everything on the other cell. Assuming your system is deterministic and the inputs stay the same, the second cell should now break as well.
This reads to me like an attempt to patch a system that's already fucked beyond belief while pretending you're doing "engineering".
Fancy implementation of a retry loop attempting to minimize downtime.
I worked as an SRE at a well-known monitoring company that used a similar architecture. It worked extremely well, and aside from one software SPOF (which still had a blast radius limited to that cell), we had very few large-scale production incidents compared to everywhere else I’ve worked at.
Even if there was a physical hardware failure (at the time, it ran on-prem, but it’s not like AWS is immune to this), every service modulo the aforementioned SPOF had redundancy, so we would have the datacenter techs replace that blade, which would provision itself and rejoin, zero downtime, just a temporary loss of redundancy. Even then, if we felt it necessary, we could shift customers into a different cell, though that did cause a brief outage for them, which would be coordinated ahead of time.
This matching up with sliced data, either sliced by recipient, or sliced by sender, so it makes sense to have primary state with cool backups. Particularly if the state is race-tolerant.
As long as event orderings are unimportant, or self resolvable this works really well.
e.g. if Events A,B arrive, but A+B => C and B+A => C, then as long as you durably record A, B, the end state is the same.
I'm not sure why "reroute" is a message instead of a response, I would expect it to be a failure response pruning the control flow. With the GTR being "default retry".
There's a lot of learned experience in that doc. Reading between the lines, both the logging and configuration systems have caused global outages (or near misses). Nifty to read.
This style of architecture fits with a "no global changes" and "never lose it all" approach to fault tolerance, accepting that there will be visits from "Mr. Cock-up" [1].
25 comments
[ 0.26 ms ] story [ 47.3 ms ] threadJust kidding!
I find the idea quite good, and have to assume that the amount of payment fails they experience due to partitions/outages isn't very high and that the post-payment reconciliation and reclamation process gives them the liberty to rank availability a bit higher than correctness.
One thing that looked a bit shaky was the interplay between the global transaction router's state of knowing which cells can handle a particular payment and the asynchronous distribution of the "failover data", which I presume it needs to know to route correctly. To me that seems to create a window where it might route to the wrong cell due to an outdated routing state.
It also doesn't go into the HA setup of the global transaction router itself.
But still, I kind of like the design.
Last week it required me to take pictures of my face from multiple angles to regain membership privileges. I suspect this may be part Palantir data collection and part Peter Thiel dating service.
Because of the title I was expecting to read about doing payments with a distributed network, like a terrorist cell network, or something like Hawala. Not (as I infer from other comments) Amex using multiple independent systems.
This isn't about payment technologies, it's not about isolating transactions, it's about scaling the middle layer. What's worse it's not even explained what middle layer does.
No info on how routing works, no info on data synchronization.
Folks just learning Kubernetes and write extremely abstract stuff.
that said: still a nice write up, learning about some of the architectural choices that AMEX makes is definitely insightful (and relavent/useful to what i am working on right now as well!)
https://news.ycombinator.com/item?id=32023863
https://wso2.com/engineering-platform/developer-platform/doc...
This reads to me like an attempt to patch a system that's already fucked beyond belief while pretending you're doing "engineering".
Fancy implementation of a retry loop attempting to minimize downtime.
I worked as an SRE at a well-known monitoring company that used a similar architecture. It worked extremely well, and aside from one software SPOF (which still had a blast radius limited to that cell), we had very few large-scale production incidents compared to everywhere else I’ve worked at.
Even if there was a physical hardware failure (at the time, it ran on-prem, but it’s not like AWS is immune to this), every service modulo the aforementioned SPOF had redundancy, so we would have the datacenter techs replace that blade, which would provision itself and rejoin, zero downtime, just a temporary loss of redundancy. Even then, if we felt it necessary, we could shift customers into a different cell, though that did cause a brief outage for them, which would be coordinated ahead of time.
And much simpler systems with basic retry/redispatch handle that just fine.
As long as event orderings are unimportant, or self resolvable this works really well.
e.g. if Events A,B arrive, but A+B => C and B+A => C, then as long as you durably record A, B, the end state is the same.
I'm not sure why "reroute" is a message instead of a response, I would expect it to be a failure response pruning the control flow. With the GTR being "default retry".
There's a lot of learned experience in that doc. Reading between the lines, both the logging and configuration systems have caused global outages (or near misses). Nifty to read.
This style of architecture fits with a "no global changes" and "never lose it all" approach to fault tolerance, accepting that there will be visits from "Mr. Cock-up" [1].
Very nice writeup.
[1] https://www.youtube.com/watch?v=D5r8xwu0l8w