Kudos for Amazon sorting out their status page. The last time this happened, the status page didn't show anything until hours after the outage began. I just noticed this one maybe 15 minutes ago, and 5-10 minutes later they acknowledged the problem.
On the other hand, I'm feeling a strong case of "Not this shit again". Wondering if US-East-1 is more trouble than it's worth, as these outages seem to happen mostly there.
It’s primarily just due to the incredible scale of the region. It’s easily twice as large as any other, so any issues relating to scale appear there first and are fixed before they affect any of the other regions.
As in, most global services run primarily within us-east-1, and have only subsystems running in each region.
For example, it's likely that the actual master data stores for IAM are solely within us-east-1, but the key data is cached and services run in each region.
Similarly, Cloudfront is theoretically a global service, but only ACM certificates set up within us-east-1 can be used with Cloudfront.
We launched our production workload in us-west-2 (Oregon) 4 years ago for this exact reason and it was one of the best decisions we made. Our staging servers in us-east-1 have experienced all of the outages between then and now, but production hums right along. Of course we're ready to failover to us-east-1 if us-west-2 has trouble, but it's been far more stable (knock on wood).
Working for a consultancy and we pretty much always default to doing initial (MVP) deploys in Oregon these days. Feels like a no brainer at this point.
If us-east and us-west regions are only relevant to you because of the relative reliability, then I envy your situation because latency doesn't exist in your world.
> Us-east-1 is where all the shiny new features are released
Not true at all. It'll be among the first for newer features, in as much as the US regions usually are among the first, but it's certainly not the first place code is deployed to.
All services in AWS start out software deployments in smaller regions where the blast radius of a software bug is likely to be smaller.
Before the popularity of SSD in the average computer, and the theoretically low downtimes for AWS, telling someone to just restart their computer a few times until the website shows up might actually have "fixed" it.
Definitely affecting S3 bucket ops, static site hosting on s3, and even things like CloudTrail. Anything that uses S3 East VA my guess. Rather severe impact. Maybe less focus on avocados is what's called for...
I'm getting issues as well. Here's to hoping this issue is just a hiccup and not like the last outage.. otherwise, it may be time to seriously consider alternatives. Anyone know of a good solution for a self-hosted s3-esque service?
true, but some internal AWS features themselves rely on S3 and apparently us-east-1. Those are hosed right now, too. You can't even modify bucket metadata via AWS Console or cli tools. So like checking permissions or logging right now is not possible, or CloudTrail.
Not exactly. S3 doesn't have AZs at all, it's only split on regions. Further, a bucket can exist in only one region. You can set up cross-region replication, but you of course need to flip the bucket coordinates in all your applications to fail over. It's not nearly as easy as Multi-AZ support in things like RDS.
I meant that a bucket exists in only one region and that you have to replicate to a different bucket if you want to do anything to improve S3 availability.
If you're serving a static site out of S3, put Cloudflare (ugh) in front of it and enable the feature to serve your cached content when the origin is down [1]. When S3 goes sideways, as everyone is learning, it goes down hard and you're probably not going to be able to make changes to objects, read object metadata, etc.
When serving files from s3, if you care about availability it's worth using your own subdomain rather than pointing directly at S3. You can't have a subdomain point to different buckets in different regions directly, but you can point your subdomain to a CDN and have your CDN's origin point to different buckets in different regions.
I would turn on bucket replication and sync everything to another region, or move your primary bucket to us-west-2 or something other than us-east-1 and replicate somewhere else.
Quick tip for those using static (jekyll, hugo...) sites on s3: If you have cloudflare in front of it, you can turn on aggressive html caching by creating a page rule: example.com/* => Cache level => Cache everything
This downtime made me realize we weren't caching any html on cloudflare. I just turned it on and all our static sites are doing fine now (and our bills are smaller!).
If you're fancy you can even programmatically purge the cache when you do CI deploys using the cloudflare API.
Exactly, it doesn't. That's why you should use the actual cache mgmt options from your CDN provider, or just wait for your (properly configured) cache control headers to expire the cache (bear in mind that intermediate caching proxies don't always obey those TTLs or dates either).
I was just pointing out that some people adjust their menu links (etc) to include some sort of dynamic variable (ie /?ts=xxx). I don't recommend this sort of scheme, though, except in unusual circumstances: IMO, it's fragile, leaky, and inefficient.
Cloudflare and all other modern CDNs do this instantly and for free. Only the legacy ones take longer (usually minutes at most) and might charge a small extra.
This is not true. Purges on akamai take less than ten minutes for entire cp codes or specific objects.
Deploying new cache rules is less than an hour in many cases
How many POPs are you using? I can assure you, I am not wrong here and have internal knowledge of why if you are truly using all POPs it takes so long.
Look, I know what I'm talking about. Just because you can purge faster because of a different deployment doesn't mean that's the case for everyone. "Fast Purge" is not available for every type of deployment.
Thanks for replying, but unless you're willing to explain to the rest of us why so, why bother? I can't make heads or tails of this pissing contest thread.
"Site Delivery" is purged globally within 7 minutes.
"Fast purge" enabled products (site delivery) is purged globally in under 20 seconds.
All assets for websites are site delivery. Only media services family of products are not site delivery. Those purge within a couple of hours. If you are someone who uses media services products, you know how to force Akamai to go back to origin globally instantaneously and how to trigger invalidation. In media services you should never serve stale.
As I said, it depends on your deployment. Thanks for the reply and confirming what I said was correct, that under some deployments it can be a multi-hour purge.
Won't help if your cache is empty and Cloudflare can't access S3. I originally discovered this service outage when an image in cloudflare was returning a 500 error about 40 minutes ago.
Yes and no. In outage periods, S3 access is often degraded because throttling is greatly increased, leading to 503s. Meaning if you have that cache layer on top of it, you dodge the issue.
Where it doesn't help is if Cloudflare itself is having issues (all it does is move the weak link one layer up in this case). But that is easy enough to disable (assuming the cloudflare API or dashboard is up).
In my experience I can set the TTL to whatever value I want, last mile providers don't really give a shit about that and just cache it for how long they want.
> Quick tip for those using static (jekyll, hugo...) sites on s3: If you have cloudflare in front of it, you can turn on aggressive html caching
> If you're fancy you can even programmatically purge the cache when you do CI deploys using the cloudflare API.
You need to be really careful with this.
I'm not sure how you've got things set up but isn't this going to lead to issues unless you have a cache busting or revalidation strategy? You can purge Cloudflare's cache but that's not going to purge the cache on browsers that already visited your site and cached a page already. You might get cases where an old cached HTML page is asking for page resources that don't exist anymore or pages break because the user is seeing the old HTML with the new CSS/JavaScript.
Also, if you can log in to your site, it can potentially cache one user's logged in page content and share it with others if your Cache-Control settings don't include "private".
You could set things up to make sure the browser and then the Cloudflare cache always asks your server first if a page has been updated recently (revalidation) but your server has to be configured right for this (e.g. etag or modified date usage).
Cloudflare is actually meant to have a feature that keeps a static version of your site to show in the event of outages:
You're both sort-of correct. AO kicks in on specific 5xx error codes: 502, 504, and then Cloudflare's 52x codes. It doesn't kick in on 500 or 503, or 4xx errors.
I don't think this is a good idea if you have lots of visits, as Cloudflare might ask you to upgrade to a Business account which is very expensive. It's excellent for small sites and blogs which are static, but for anything visited often, like a Docs site for some popular project, it might get dropped from CF's network. Don't quote me on that, I haven't experienced it, but I heard that they do ask people to upgrade, for example Brian Fritz of omdbapi.com, who had to close down the service to the public because of huge amounts of traffic, and that's just with text, not serving images or video...
This was us! Added CloudFlare in front of every bucket with our own "cdn.example.com" DNS name and changed every reference to s3 in code. Didn't even know S3 had problems today until I saw this on HN. (note: This outage taught us to set up monitoring of the S3 assets separate from the "cdn")
186 comments
[ 2.2 ms ] story [ 235 ms ] threadEB interface is affected too
Console says: Failed to load resource: the server responded with a status of 503 (Slow Down)
On the other hand, I'm feeling a strong case of "Not this shit again". Wondering if US-East-1 is more trouble than it's worth, as these outages seem to happen mostly there.
For example, it's likely that the actual master data stores for IAM are solely within us-east-1, but the key data is cached and services run in each region.
Similarly, Cloudfront is theoretically a global service, but only ACM certificates set up within us-east-1 can be used with Cloudfront.
Not true at all. It'll be among the first for newer features, in as much as the US regions usually are among the first, but it's certainly not the first place code is deployed to.
All services in AWS start out software deployments in smaller regions where the blast radius of a software bug is likely to be smaller.
Looks like November 17 to June 14.
[1] https://aws.amazon.com/about-aws/whats-new/2016/11/amazon-sq...
[2] https://aws.amazon.com/about-aws/whats-new/2017/06/amazon-sq...
Oregon sees almost the same speed of adoption of new features with way better reliability.
[1]: https://twitter.com/AlecSanger/status/908402829349572608
11:58 AM PDT We are investigating increased error rates for Amazon S3 requests in the US-EAST-1 Region.
12:28 PM PDT We are investigating increased error rates for Git Push, Git Pull and API calls in the US-EAST-1 Region.
they have multi-az support if you use it
I think you really meant s3 objects are redundant in each region, which actually is spanned across multiple DCs.
Active/active cross-region is possible, but far more complex.
https://www.openstack.org/software/releases/ocata/components...
https://github.com/ceph
If you're serving a static site out of S3, put Cloudflare (ugh) in front of it and enable the feature to serve your cached content when the origin is down [1]. When S3 goes sideways, as everyone is learning, it goes down hard and you're probably not going to be able to make changes to objects, read object metadata, etc.
[1] https://support.cloudflare.com/hc/en-us/articles/200172256-H...
https://support.cloudflare.com/hc/en-us/articles/200172256-H...
This downtime made me realize we weren't caching any html on cloudflare. I just turned it on and all our static sites are doing fine now (and our bills are smaller!).
If you're fancy you can even programmatically purge the cache when you do CI deploys using the cloudflare API.
EDIT: Oops. Misread cloudflare/CloudFront.
For inner pages, browsers do consider /your-page-url?v=1505418887 to be a completely different page from /your-page-url?v=1505418888.
However, just expiring cache and/or properly setting HTTP cache control headers at most CDN's is a cleaner and more correct option.
I was just pointing out that some people adjust their menu links (etc) to include some sort of dynamic variable (ie /?ts=xxx). I don't recommend this sort of scheme, though, except in unusual circumstances: IMO, it's fragile, leaky, and inefficient.
Thanks for replying, but unless you're willing to explain to the rest of us why so, why bother? I can't make heads or tails of this pissing contest thread.
"Site Delivery" is purged globally within 7 minutes.
"Fast purge" enabled products (site delivery) is purged globally in under 20 seconds.
All assets for websites are site delivery. Only media services family of products are not site delivery. Those purge within a couple of hours. If you are someone who uses media services products, you know how to force Akamai to go back to origin globally instantaneously and how to trigger invalidation. In media services you should never serve stale.
Where it doesn't help is if Cloudflare itself is having issues (all it does is move the weak link one layer up in this case). But that is easy enough to disable (assuming the cloudflare API or dashboard is up).
However, I do agree that it should be possible to turn on without pagerules for these sorts of scenarios.
However, the case of a static website is not really hitting me as something can't afford a few seconds downtime...
> If you're fancy you can even programmatically purge the cache when you do CI deploys using the cloudflare API.
You need to be really careful with this.
I'm not sure how you've got things set up but isn't this going to lead to issues unless you have a cache busting or revalidation strategy? You can purge Cloudflare's cache but that's not going to purge the cache on browsers that already visited your site and cached a page already. You might get cases where an old cached HTML page is asking for page resources that don't exist anymore or pages break because the user is seeing the old HTML with the new CSS/JavaScript.
Also, if you can log in to your site, it can potentially cache one user's logged in page content and share it with others if your Cache-Control settings don't include "private".
You could set things up to make sure the browser and then the Cloudflare cache always asks your server first if a page has been updated recently (revalidation) but your server has to be configured right for this (e.g. etag or modified date usage).
Cloudflare is actually meant to have a feature that keeps a static version of your site to show in the event of outages:
https://www.cloudflare.com/always-online/
Ref: https://support.cloudflare.com/hc/en-us/articles/202238800-W...