I've heard a fun story from the old timers in my org about a fiber outage in Brazil. A routine fiber cut occurred. They figure out how far from one end the cut is (there is gear that measures the time to see a light pulse reflect off of the cut end.) Then they pull out a map of where the fiber was laid, count out the distance, and send a technician out to have a look at where they expect the cut to be. All standard practice up until this point.
The technician updates the ticket after a while with "cannot find road." The folks back in the office try to send them directions, but then the technician clarifies, "road is gone." Our fiber, and the road it was buried under was totally demolished in the few hours it took to get someone out there. The developing world can develop at alarming rates.
Other tales from the middle of nowhere: People shoot at arial fiber with guns. Or dig it up and cut it for fun. One time out technician was carjacked on the way to doing a repair.
It actually sounds useful, to the point I wouldn't be surprised if in the near future cloud providers bundled up some chaos monkey stack and offered that with a neat price within their realms (dunno, maybe per VPC or project).
This is called chaos engineering and many companies built tooling to do exactly this. Netflix pioneered/proselytized it years ago. Since you likely don't just rely upon AWS services if your app is in AWS, you want something either on your servers themselves or built into whatever low level HTTP wrapper you use. Use that library to do fault injection like high latency, errors, timeouts, etc.
Came here to say this exact thing. There have been a variety of techniques to achieve this - some intrusive to your binaries (i.e. they require embedding specific libraries) and others that are more "external" (ex: tc/iptables).
The "real" challenge is not creating chaos but managing it and verifying that your apps are resilient to said chaos.
Sorry but this is just strangely naive. Yes, you should have integration test setups that allow you to swap out any network dependency with in-memory stubs, with fault-injected proxies, etc. But all of that can never replace the behavior and chaos of a real outage. On top of that, a lot of the more complicated scenarios are immensely difficult to setup and you first also need to know HOW a region can fail.
I agree with you on one thing though. AWS, and actually every cloud provider should ship a Chaos Engineering toolbox that comes with ready-to-use, realistic failure simulations that you can use in tests. I.e. drop-in replacements for their SDK clients that then just start running through predefined failure scenarios.
It's harder to do chaos engineering if you're not engineering it. What this is really asking for is the service provider to sell chaos engineering as a service (CEAAS?), on the services they provide. I've wanted this kind of thing for testing cloud infrastructure before: you read about various failure states and scenarios you might want to handle from docs but there's no way to trigger them so you just have to hope that they work as described and your code is correct. At the least, let users simulate the effect of the failures that are part of your API.
This would be great for testing the pieces of the stack that the provider is responsible for, but you may still want to inject chaos into the part of your stack that you do control.
A zone not only of sight and sound, but of CPU faults and RAM errors, cache inconsistency and microcode bugs. A zone of the pit of prod's fears and the peak of test's paranoia. Look, up ahead: Your root is now read-only and your page cache has been mapped to /dev/null! You're in the Unavailability Zone!
There are multiple methods for automating AWS EC2 instance recovery for instances in the "system status check failed" or "scheduled for retirement event" cases.
Yet to figure out how to test any of those cloudwatch alerts/rules. I've had them deployed in my dev/test environments for months now, after having to manually deal with a handful of them in a short time period. They've yet to trigger once since.
This is why it seems like it would be good to have explicit fault injection APIs instead of assuming that the normal APIs behave the same as a real failure.
I don't work with the group directly, but one group at our company has set up Gremlin, and the breadth and depth of outages Gremlin can cause is pretty impressive. Chaos Testing FTW.
Failing individual computes isn't hard, some chaos script to kill VMs is enough. Worst are situations when things seem to be up but not acceptable: abnormal network latency, random packet drops, random but repeatable service errors, lagging eventual consistency. Not even mentioning any hardware woes.
No, really, it's the IBM cloud that is the joke. This isn't the first I've heard of it, though I've not used it myself.
I'm a happy AWS user and I'll stay a happy AWS user not for their prices or features, but service. Which was the reason I was a Rackspace fan before it was sold and went down the tube.
I used to use their Citrix Netscaler VPX1000s at a previous job.
They were very reliable, imo. Aside from general Netscaler bullshit, We only ever had issues with them when we'd try to get them to do too much so that the CPU or Memory was overloaded.
We tried on a few occasions to get more cores allocated to them, but no. This made terminating large numbers of SSL connections on them problematic.
I was using their shared loadbalancers, not the run a load balancer in a VM option, because I was hoping for something more reliable than a single computer. For the couple months they were running, it was literally every 30 days, 10 minutes of downtime. So I went back to DNS round robin, cause it was better.
A great idea! I'd love to run stuff in this zone. Rotate through a bunch of errors, unavailability, latency spikes, power outages etc every day, make it a 12 hour torture test cycle.
For those saying "Chaos Engineering", first off, the poster is well aware of Chaos Engineering. He's an AWS Hero and the founder of Tarsnap.
Secondly, this would help make CE better. I actually asked Amazon for an API to do this ten years ago when I was working on Chaos Monkey.
I asked for an API to do a hard power off of an instance. To this day, you can only do a graceful power off. I want to know what happens when the instance just goes away.
I also asked for an API to slow down networking, set a random packet drop rate, EBS failures, etc. All of these things can be simulated with software, but it's still not exactly the same as when it happens outside the OS.
Basically I want an API where I can torture an EC2 instance to see what happens to it, for science!
The main result of this testing is that you'll find bugs in all the abstraction layers that you can't control. You'll then be paranoid but unable to take any action. Have you ever seen the source code that's running on your SSD's CPU? Nope. And it's probably not a small program. My guess is that it works well when everything is OK, and fails catastrophically 0.0001% of the time when everything isn't OK. But you'll never know unless you try failing catastrophically a million times. Did the vendor even try failing catastrophically one million times before they started manufacturing (or at least sent the batch to retailers)? Did they do that, fail 1 time, and mark the bug closed as non-reproducible?
I have no idea! Maybe everything is actually great. Or maybe someone else will be oncall the week you hit that one in a million failure case. Or maybe it will happen to your competitors instead of you! Without testing, all we have is hope.
I think this is too pessimistic. If all you do is test your error detection and backup recovery mechanism (of course you have backups right)? Then that’s an advantage.
Let’s say a hard power off causes data corruption that can’t be fixed. For lots of applications downtime is better than corruption, so in that case you at least will be able to test when you should take the system down and recover from a known good backup.
> You'll then be paranoid but unable to take any action.
I disagree.
While you can't control the bugs in the hardware or drivers or whatever, you can definitely ensure that your application detects corrupted data and can at least warn that something is wrong rather than starting back up and silently serving from corrupted data.
It also ensures that a node suddenly disappearing without a graceful "I'm leaving" message, and potentially with open transactions or other operations can be handled by the clients and nodes that ARE still around.
You've hit on one of my big pet peeves -- I am going to add redundancy and erasure codes at the application layer because the disk could catch on fire or be sucked up by a tornado, so why do I need a bunch of unaudited software adding extra redundancy and erasure codes behind my back?
I get it for desktop users... they want to open some benchmark they downloaded and get a higher number than their friends (RAID-0-esque sharding data across individual flash chips), and they also don't have access to software that can add error correction codes (because the Windows installer can't recognize an SSD with 4 flash chips as 4 drives in a RAID array)... so the disk has to do it itself. But why is it a thing in datacenters? Just so you can install Linux+ext2+MySQL and call it a day? That seems crazy to me when better storage software exists.
Aren't there SSDs that go in the opposite direction and expose a key-value store instead of a linear address space? Seems like that will make reliability testing even harder.
If you're using Docker, yes, you can get a lot more options in testing by poking "from the outside", but that still doesn't test what happens when your docker host just dies.
I'm not the GP, but I assume it is because the GP would like to run his "production" setup (or even production setup itself) under such circumstances. So far as I remember, the origin of the idea came from the Chaos Monkey which would sabotage certain services in a production environment to ensure than everything was redundant and fail safe.
> I asked for an API to do a hard power off of an instance. To this day, you can only do a graceful power off. I want to know what happens when the instance just goes away.
Possibly, but how can we be 100% sure without the ability to compare behavior? If I were following this line of research I'd still want to know if there's any difference in the nature of a failure when it comes from within the OS (possibly simulated by half -f) and the situation the parent OP pointed out where the instance just goes poof without sending any kind of signal to the OS itself.
Though of course then the trouble is, if AWS is simulating specific behaviour, it won't be exactly the same as real problems when they occur. It is a bit better, but hard to say how much better.
I'd think the key on this is being able to simulate very specific partial-failure conditions. e.g. specific packet loss, loss of connections to EBS, etc. Just turning machines off I expect wouldn't be that valuable.
> Though of course then the trouble is, if AWS is simulating specific behaviour, it won't be exactly the same as real problems when they occur.
If Amazon simulated power failure, or network cable disconnect, or potentially even corrupt writes to disk, I personally feel it would be indistinguishable from the event really happening.
The problem is that they can't really realistically simulate that for a single instance unless you've got the whole physical machine to yourself, and of course the worst issues tend to be when it's half-broken, not entirely off/disconnected
Maybe for testing the crash-resilience of software running on the node; but not necessarily for testing how the SDN autoscaling glop you've got configured responds to the node's death.
A panicking instance is still "alive" from its hypervisor's perspective (either it'll hang, sleep, or reboot, but it won't usually turn off its vCPU in a way the hypervisor would register as "the instance is now off"); while if a hypervisor box suffers a power cut, the rest of the compute cluster knows that the instances on that node are now very certainly off.
echo o > /proc/sysrq-trigger
will shutdown a AWS instance immediately. I've used it a couple of times to make real sure my costly spot instance was terminated.
Looks like SysRq-o does a clean poweroff, not a dirty/immediate one — it calls kernel_power_off(), not machine_power_off().
This means, importantly, that can actually take some time to happen, as drivers get a chance to run deinitialization code. This also means that a wedged driver that doesn't respond during deinit can prevent the kernel from halting.
Thus, while SysRq-o might be useful for killing a wedged userland, it's not a panacea — especially, it isn't guaranteed to complete a shutdown for unstable kernels, or kernels with badly-written DKMS drivers attached. It's not truly equivalent to a power-cut.
Not exactly. "halt -f" depends on some variant of power management working. For extra fun, you could also have partial power-offs, like say a power cut to the motherboard, or any combination of other devices with their own power lines from a power supply. (in a current PC these would typically be PCI supplementary power, motherboard, SATA power, 4-pin 12v molex power, supplemental CPU power.)
Granted, most of those are out of scope for cloud development, but the concept of externally cutting off a VM is different than even calling whatever power service you have to cut power. In the 'real' world, enough of those failure triggers above would probably also trigger an automated power cycle if you're in a managed environment.
To elaborate a bit further - when things are fine, they're, well, fine. Unfortunately, the assumptions of how computers actually function go out of the window when enough things go wrong, like lightning striking the wrong thing. Things go weird sometimes, like the accepting TCP connections, but not sending responses. can't ping but can SSH in, or vice versa. In a cluster with automatic failover, you want to be sure the bad node is dead dead dead because a bad machine can cause harm to all the packets in the rack. (Eg HA with floating IPs and ARP takeover.) Thus the requirement is to forcefully disable the bad node, which is where the acronym STONITH comes from - Shoot The Other Node In The Head. (HA clusters with more than two nodes existed, but were rarer than 2-node clusters simply due to cost.) Before cloud computing was so pervasive, high availability was implemented with serial cables between in physical hardware and when one machine stopped responding, the equipment basically yanked its power cable out.
As far as I know it still sends a graceful shutdown at the end of the time block. It sends exactly the same signal as if you use the shutdown API, which is the same as pressing the power button on the front of the machine.
There are lots of ways to get a very close experience to a power pull, but nothing you can do from within the VM is quite the same as doing it outside the VM.
I believe you but I'm curios as to the subtle difference.
To my mind simply turning off the networking, physically cutting a cable, or hardware spontaneously combusting are not discernible events to an outside observer. What am I missing?
on a fully virtualized system, echo o > /proc/sysrq-trigger is going to be as close to a forced shutdown as you can get. that won't test hardware/firmware issues, but neither will a VM stop anyways.
It's AWS; surely half of the work would be finding the server that happens to be running your instance, anonymously sitting in a datacenter with thousands of other servers. There's also the question of how many other customers' instances are located on the same hardware.
I really wish the world could agree on a better term than 'on premis' for 'not in the cloud'. At least 'on premises' is descriptive of the situation, although a bit clunky. On premises doesn't make any sense at all. 'Premis' is not the singular of 'premises', 'premises' is the singular of itself! I prefer 'on site', or 'self hosted' myself
The mere existence of such an api would be an interesting source of problems when used accidentally/via buggy code ...
I wonder the degree to which this functionality would end up becoming relied on as a "in worst case hard kill things to recover" behavior that folks utilize for bad engineering reasons as opposed to good ones ...
Totally. I have added wifi power outlets to installations because process restart, OS restart, and hardware restart did not get a system back up and running. Yes it was a startup, and yes it was remotely installed hardware.
Might be a bit strange as you are saying this is a example of bad planning but my employer was discussing this today. We had a hard time finding options, what did you end up deploying or did you develop your own in house?
When I had this issue I just bought some cheap wifi 'smart sockets' which had some open-source firmware available and reflashed them. They even support reporting the voltage and current draw from the socket (though not particularly accurately) and toggling the power through a POST (with a password if set). Was very useful for power cycling various bits of hardware remotely.
In a previous job we actually had to make a decision like this: our server software was generally stable enough, but we reset it whenever we did an update. Every once in a while it would break when running for long enough.
I opted for forcefully restarting the software every day, because I'd rather relive the pains of restarting, than to discover in production what it is like for the first time to run the system for the 501st day of uptime.
(To be honest, the system in question was mostly written in Erlang, so controlled restarts were already baked into its philosophy.)
I was just about to suggest localstack. I use it religiously in personal projects; can't recommend it enough. I haven't started telling it to induce errors yet, but it definitely has that capability. And if you're running it in docker, some of the network stuff can be simulated that way as well.
I don't pay much heed to AWS Heroes. My city has an "AWS Hero" who hoarded the organization of the official AWS meetup for themself, and who is infamous for making liberal use of Cunningham's Law [1] for everything.
AWS Hero, Microsoft MVP.. something about those sorts of recognition seem to attract certain types of people - types seeking the limelight - often not particularly good, ethical, moral, or pleasant types.
Not a bad script, but it's basically going "shields up!", blocking all traffic, and then taking them down. Definitely useful, but won't test for the hard stuff, like if my DB was in the middle of something and then died, scrambling my data. What's that gonna look like on reboot?
Nearly a decade ago I was working on Xen (or rather XenServer). We had lots of fun implementing these kinds of wonky devices for the virtual machines that would randomly drop network packets or fail to read the hark disk in arbitrarily bad ways.
I want a stress testing feature where I can accidentally misconfigure an instance or resource somehow, then when I get an unexpectedly $10,000 higher bill at the end of the month, I can declare that I was only testing, and I don't have to pay it!
I can see a use case for this being implemented on top of Kubernetes. I've no idea if that's achievable, but could go some way to make your code more resilient.
All jokes aside, I actually asked my google cloud rep about stuff like this; they came back with some solutions but often the problem with that is, what kind of failure condition are you hoping for?
Unless it’s something like s3 giving high error rates then most things can actually be done manually. (And this was the advice I got back because faulting the entire set of apis and tools in unique and interesting ways is quite impossible)
I found that in practice for APIs, delays are often much worse than errors.
Mostly because programmers seem to have an easier time thinking about errors, and their programming language might even encourage them to handle these kinds of errors, but arbitrary delays often slip by unanticipated.
Googles approach to RPC deadline propagation solves this. I'm sad that the vast majority of libraries don't support it.
The simple concept is all services/API's should take a deadline parameter. The call should either complete before that time, or return an error. If you are writing code for a service, and you get an incoming request, you send the same deadline for any requests to other services.
Google has awesome clock sync in its datacenters (sub millisecond), and in general all RPC calls should be idempotent (since the network or either host can fail at any point).
I think one might be able to make this approach work with poor clock sync by having the request contain the number of milliseconds left, rather than an absolute time. It isn't great if network delays dominate, but typically it's queueing delays that dominate most webservice responses.
This (or equivalent) will happen. You can either design your software to handle it correctly, or you can have buggy software.
Clock sync doesn't really help, especially if a request can trigger multiple independent action that can individually succeed or fail.
The simplest example where this goes wrong is posting a post on Reddit. You write your text into a text box and submit. You get an Error 500. You cannot know if your post went through (even if you go check whether your post is visible, sometimes the replica that you got your response from hasn't gotten your post yet). And they don't do any deduping, so if you just retry, you may end up with multiple copies of your reply.
Yeah, us-east-1 is pretty good at failing already. We lost us-east-1c for most of the day about a week ago due to a fiber line being cut. I'd estimate that AWS manages fewer than "three 9s" in us-east-1 on average. Not across the board, but at any given time something has a decent chance of not working, be it an entire AZ, or regional S3, etc. They're still pretty reliable, and I like the idea of a zone with built-in failure for testing things, but your joke about us-east-1 is based in solid fact.
Sort of counter-intuitive, but for small projects, you want resilient hardware systems as much as possible... the larger your scale out, the less reliable you would want them to force that out of hardware into resilient software.
While these are not exclusive, personally I'd look instead into studying my system's reliability in a way that is independent of a cloud provider, or even of performing any side-effectful testing at all.
There's extensive research and works on all things resilience. One could say: if one build a system that is proven to be theoretically resilient, that model should extrapolate to real-world resilience.
This approach is probably intimately related with pure-functional programming, which I feel has been not explored enough in this area.
Simply host on Google Cloud! They will terminate your access for something random, like someone said your name on YouTube while doing something bad! They don't have a number you can call, and their support is ran but the stupidest of all AI algorithms.
I imagine AWS and other clouds have a staging/simulation environment for testing their own services. I seem to recall them discussing that for VPC during re:Invent or something.
I'm on the fence though if I would want a separate region for this with various random failures. I think I'd be more interested in being able to inject faults/latencies/degradation in existing regions, and when I want them to happen for more control and ability to verify any fixes.
Would be interesting to see how they price it as well. High per-API cost depending on the service being affected, combined with a duration. Eg, make these EBS volumes 50% slower for the next 5min.
Then after or in tandem with the API pieces, release their own hosted Chaos Monkey type service.
196 comments
[ 11.2 ms ] story [ 3850 ms ] threadThe technician updates the ticket after a while with "cannot find road." The folks back in the office try to send them directions, but then the technician clarifies, "road is gone." Our fiber, and the road it was buried under was totally demolished in the few hours it took to get someone out there. The developing world can develop at alarming rates.
Other tales from the middle of nowhere: People shoot at arial fiber with guns. Or dig it up and cut it for fun. One time out technician was carjacked on the way to doing a repair.
Though not very relevant to your stories about Brazil, it's a neat technique in its own right:
https://en.wikipedia.org/wiki/Optical_time-domain_reflectome...
The "real" challenge is not creating chaos but managing it and verifying that your apps are resilient to said chaos.
I agree with you on one thing though. AWS, and actually every cloud provider should ship a Chaos Engineering toolbox that comes with ready-to-use, realistic failure simulations that you can use in tests. I.e. drop-in replacements for their SDK clients that then just start running through predefined failure scenarios.
This would be great for testing the pieces of the stack that the provider is responsible for, but you may still want to inject chaos into the part of your stack that you do control.
Yet to figure out how to test any of those cloudwatch alerts/rules. I've had them deployed in my dev/test environments for months now, after having to manually deal with a handful of them in a short time period. They've yet to trigger once since.
Umbrellas when it's raining etc.
I'm a happy AWS user and I'll stay a happy AWS user not for their prices or features, but service. Which was the reason I was a Rackspace fan before it was sold and went down the tube.
They were very reliable, imo. Aside from general Netscaler bullshit, We only ever had issues with them when we'd try to get them to do too much so that the CPU or Memory was overloaded.
We tried on a few occasions to get more cores allocated to them, but no. This made terminating large numbers of SSL connections on them problematic.
As for things like EBS failing, or dropping packages, it's a bit tricky as some things might break at the OS levels
And given sufficient failures, you can't swim anymore, you'll just sink.
Secondly, this would help make CE better. I actually asked Amazon for an API to do this ten years ago when I was working on Chaos Monkey.
I asked for an API to do a hard power off of an instance. To this day, you can only do a graceful power off. I want to know what happens when the instance just goes away.
I also asked for an API to slow down networking, set a random packet drop rate, EBS failures, etc. All of these things can be simulated with software, but it's still not exactly the same as when it happens outside the OS.
Basically I want an API where I can torture an EC2 instance to see what happens to it, for science!
Pretty much every option you have from the OS will let caches flush, will have in-progress writes complete.
Yank the power and none of that will happen. It'll let you actually see what level of fibbing your OS and hardware are telling you.
Oh you got a result from that flush to disk? It completed? Are you sure? Really really sure? Lets find out...
I have no idea! Maybe everything is actually great. Or maybe someone else will be oncall the week you hit that one in a million failure case. Or maybe it will happen to your competitors instead of you! Without testing, all we have is hope.
Let’s say a hard power off causes data corruption that can’t be fixed. For lots of applications downtime is better than corruption, so in that case you at least will be able to test when you should take the system down and recover from a known good backup.
I disagree.
While you can't control the bugs in the hardware or drivers or whatever, you can definitely ensure that your application detects corrupted data and can at least warn that something is wrong rather than starting back up and silently serving from corrupted data.
It also ensures that a node suddenly disappearing without a graceful "I'm leaving" message, and potentially with open transactions or other operations can be handled by the clients and nodes that ARE still around.
I get it for desktop users... they want to open some benchmark they downloaded and get a higher number than their friends (RAID-0-esque sharding data across individual flash chips), and they also don't have access to software that can add error correction codes (because the Windows installer can't recognize an SSD with 4 flash chips as 4 drives in a RAID array)... so the disk has to do it itself. But why is it a thing in datacenters? Just so you can install Linux+ext2+MySQL and call it a day? That seems crazy to me when better storage software exists.
I'm not the GP, but I assume it is because the GP would like to run his "production" setup (or even production setup itself) under such circumstances. So far as I remember, the origin of the idea came from the Chaos Monkey which would sabotage certain services in a production environment to ensure than everything was redundant and fail safe.
Wouldn't just running "halt -f" do the same?
I'd think the key on this is being able to simulate very specific partial-failure conditions. e.g. specific packet loss, loss of connections to EBS, etc. Just turning machines off I expect wouldn't be that valuable.
If Amazon simulated power failure, or network cable disconnect, or potentially even corrupt writes to disk, I personally feel it would be indistinguishable from the event really happening.
echo c > /proc/sysrq-trigger
A panicking instance is still "alive" from its hypervisor's perspective (either it'll hang, sleep, or reboot, but it won't usually turn off its vCPU in a way the hypervisor would register as "the instance is now off"); while if a hypervisor box suffers a power cut, the rest of the compute cluster knows that the instances on that node are now very certainly off.
https://www.kernel.org/doc/html/v4.11/admin-guide/sysrq.html...
Looks like SysRq-o does a clean poweroff, not a dirty/immediate one — it calls kernel_power_off(), not machine_power_off().
This means, importantly, that can actually take some time to happen, as drivers get a chance to run deinitialization code. This also means that a wedged driver that doesn't respond during deinit can prevent the kernel from halting.
Thus, while SysRq-o might be useful for killing a wedged userland, it's not a panacea — especially, it isn't guaranteed to complete a shutdown for unstable kernels, or kernels with badly-written DKMS drivers attached. It's not truly equivalent to a power-cut.
Granted, most of those are out of scope for cloud development, but the concept of externally cutting off a VM is different than even calling whatever power service you have to cut power. In the 'real' world, enough of those failure triggers above would probably also trigger an automated power cycle if you're in a managed environment.
To my mind simply turning off the networking, physically cutting a cable, or hardware spontaneously combusting are not discernible events to an outside observer. What am I missing?
And one day there will be PETA [0] for EC2 instances!
[0]: https://www.peta.org/
https://en.wikipedia.org/wiki/Plurale_tantum
That is, it seems to be grammatically plural but not have a distinctive singular.
The mere existence of such an api would be an interesting source of problems when used accidentally/via buggy code ...
I wonder the degree to which this functionality would end up becoming relied on as a "in worst case hard kill things to recover" behavior that folks utilize for bad engineering reasons as opposed to good ones ...
Edit: Another term is Switched PDU
On top of keeping people who don't want it out, you'd need it to be expensive to offset the lower utilization.
I'd be okay with a garbage-tier, too. It's not failing on purpose, but only because no one gives a shit. Maybe that's just standard AWS?
This would be more effective if so many other AWS features weren't hidden in non-obvious places in the docs.
In a previous job we actually had to make a decision like this: our server software was generally stable enough, but we reset it whenever we did an update. Every once in a while it would break when running for long enough.
I opted for forcefully restarting the software every day, because I'd rather relive the pains of restarting, than to discover in production what it is like for the first time to run the system for the 501st day of uptime.
(To be honest, the system in question was mostly written in Erlang, so controlled restarts were already baked into its philosophy.)
[1] https://localstack.cloud/
Yeah, but did he win the Putnam?
2007 HN was highly entertaining
https://en.wikipedia.org/wiki/William_Lowell_Putnam_Mathemat...
I don't pay much heed to AWS Heroes. My city has an "AWS Hero" who hoarded the organization of the official AWS meetup for themself, and who is infamous for making liberal use of Cunningham's Law [1] for everything.
[1] https://meta.wikimedia.org/wiki/Cunningham%27s_Law
Honestly, I don't either. There's a handful of us who are technical and a huge number whose contributions are primarily marketing.
All jokes aside, I actually asked my google cloud rep about stuff like this; they came back with some solutions but often the problem with that is, what kind of failure condition are you hoping for?
Zonal outage (networking)? Hypervisor outage? Storage outage?
Unless it’s something like s3 giving high error rates then most things can actually be done manually. (And this was the advice I got back because faulting the entire set of apis and tools in unique and interesting ways is quite impossible)
Just firewall off the real s3, and point clients at a proxy which forwards most requests to the real s3 and returns errors or delays to the rest.
Mostly because programmers seem to have an easier time thinking about errors, and their programming language might even encourage them to handle these kinds of errors, but arbitrary delays often slip by unanticipated.
The simple concept is all services/API's should take a deadline parameter. The call should either complete before that time, or return an error. If you are writing code for a service, and you get an incoming request, you send the same deadline for any requests to other services.
Host A calls host B
Host B runs SQL+COMMIT
DB commit finishes
Host B deadline expires
DB success reaches host B
Also would need fanatical time synchronization. Maybe even PTP rather than NTP?
I think one might be able to make this approach work with poor clock sync by having the request contain the number of milliseconds left, rather than an absolute time. It isn't great if network delays dominate, but typically it's queueing delays that dominate most webservice responses.
Clock sync doesn't really help, especially if a request can trigger multiple independent action that can individually succeed or fail.
The simplest example where this goes wrong is posting a post on Reddit. You write your text into a text box and submit. You get an Error 500. You cannot know if your post went through (even if you go check whether your post is visible, sometimes the replica that you got your response from hasn't gotten your post yet). And they don't do any deduping, so if you just retry, you may end up with multiple copies of your reply.
We are forced to use Azure for business reasons where I work, and the frequency of one off failures and outages is insane.
There's extensive research and works on all things resilience. One could say: if one build a system that is proven to be theoretically resilient, that model should extrapolate to real-world resilience.
This approach is probably intimately related with pure-functional programming, which I feel has been not explored enough in this area.
I'm on the fence though if I would want a separate region for this with various random failures. I think I'd be more interested in being able to inject faults/latencies/degradation in existing regions, and when I want them to happen for more control and ability to verify any fixes.
Would be interesting to see how they price it as well. High per-API cost depending on the service being affected, combined with a duration. Eg, make these EBS volumes 50% slower for the next 5min.
Then after or in tandem with the API pieces, release their own hosted Chaos Monkey type service.