All 3 heroku hosted sites which I look after appear to be up and running normally, despite the heroku console being unavailable. Not sure if auto-scaling is impacted yet.
Edit: all my apps are in the EU region, which is eu-west-1 (Ireland)
I created a ticket through Salesforce's "premium" phone support, but the URL of the ticket is hosted at help.heroku.com, which is also down. As of now, Heroku's status page appears green. Our Heroku app is hosted in Amazon East, but appears to be functioning at the moment.
Sorry no offense, but it's obvious you've never worked in support before.... :-) I'm sure they already know that half the infrastructure is down. Every support request brings exactly nothing, except more chaos.
People are anxious, if you have a problem in your infrastructure at least set an automatic reply linking the status page or with a short text describing the situation.
Trello is down as well (at least the checklists aren't loading anymore). GraphQL requests timing out in the Console. Not sure if it is connected to Heroku going down. Just wanted to add this here in case anyone else is also experiencing the same issue.
Could somebody please explain, how come such a sophisticated PaaS provider as Heroku was not able to properly set up multiple AWS availability zones for their own services? I would think that if somebody had expertise to do it flawlessly, it should have been them.
I don't understand much in cloud devops, I just judge from recurring HN comments saying left and right "your startup should have configured up multiple AZ" when some AWS region is down.
I don’t know know how Heroku use AZs but I do know their platform in AWS EU-West-1 (Dublin) if funtioning fine. Dashboard + metrics all ok, haven’t deployed though. So that’s to say they aren’t dependent on US-East-1 for all internal operations if you aren’t in it.
Edit: maybe the dashboard is just back up and I don’t see it down.
Most heroku users are using relational databases like postgres, and relational databases don't failover transparently. If client apps can't failover to other AZs, why does it matter for heroku's service itself to do so?
Getting geographic failover to work seamlessly generally requires application-level knowledge or at the very least, exotic datastores like DynamoDB or Firebase. For better or worse, that's just not the way most people want to develop.
They are. RDS multi-AZ replication is synchronous. Aurora is asynchronous, and multi-region replication is asynchronous, but a multi-AZ deployment of the "traditional" database engines in RDS is synchronous. It's rather the opposite of what you said before: if you stick to the old fashioned databases, this is easy to set up, failover is automatic, and no app-level changes are needed.
I think you may have been thinking of multi-region, since you mentioned geographic failover. Multi-AZ isn't really geographic failover.
It is usually some unexpected secondary service that wasn’t known to be in the critical path. Or a hidden dependency on an aws service that didn’t fail over gracefully. I look forward to reading the post mortem.
Render.com is such a better option these days (and I've been a huge Heroku fan since the beginning). Render has better support and pricing. In addition to great docker support and free static sites, they support distributed Elixir out of the box as well as autoscaling.
Still, they're not immune to these types of outages - I believe they're dependent on Google Could but are moving towards running their own data centers at some point. But so far, uptime has been great for us.
Fly.io also looks promising; keeping an eye on that one.
Fly.io's pricing page seems impenetrable to me... I just checked out Render.com and it looks so easy to understand.
I've been wanting to run a Node backend API thing but can't figure out where to put it... been using Cloudflare workers but it kept timing out (going over 10s). Will definitely check out Render's services and $1/mo crons.
I've been a Netlify customer for a number of years now and I'm considering switching over to Render. My biggest gripe with Netlify recently is don't make it straightforward to upload site without using their build system, and there are a bunch of little workarounds to implement to make everything work right.
How is render for static sites? Do they make you build the site with them or is it a simple "here's an NPM package that github actions can consume".
(I work at Render) We do need a Github/Gitlab repo, but we're not wedded to it. So I'm clear, you want the static site hosted on Render but built via GH Actions?
Haven't used render.com for long, but it looks very promising so far, not just pricing wise.
One of the biggest appeals to me is their simplified, docker-compose like approach to infrastructure as code. Even if I only have a few services, dbs, etc. I still like to have it as code under version control, but things like Pulumi or Terraform are way overkill
https://render.com/docs/blueprint-spec
There are still various quality of life improvements I'd like to see, but to me they are on a very attractive path
Wow, I was just talking to someone about how I'd love to be able to define all of the parts of our Heroku infrastructure in code. I've been going down the Terraform route but I think it's just too complicated for our team. Render's blueprint spec is pretty much exactly what I was looking for.
Hey there, Adam from Render here, I manage our product eng team. Could you say more about the quality of life improvements you'd like to see? Or happy to chat with you or anyone else with suggestions. I'm adam@render.com.
Thanks for the offer! Will write bit more details later on when I have time, so briefly:
Maybe it is less about new/improved features, I strongly prefer a focused product over one that tries to fulfill everyones whims, but more about guidance on for example for a small team what you think is the “best” way for them to integrate render.com into their dev workflows with typical stacks. How to solve the most common needs and what kind of things are out of scope for render to solve and what could be used instead. For an opinionated tool, it could be a bit clearer, what its opinions are. I think lot the information is already there, but it is quite scattered around between docs, feature request pages and forums.
Render had some pretty bad outages (10+ hours) and couple shorter ones just couple months ago, so it's not perfect either. But anyways I have also found it better than Heroku for my projects, it's even easier to setup than Heroku and pricing is slightly better too (especially the free static sites).
(Render founder) We've learned a lot from these outages, most of which stemmed from DDoS attacks. We've since built DDoS protection into the fabric of Render: https://render.com/blog/free-ddos-protection.
Specifically related to this AWS outage, Render can already transparently shift your app between AZs, and we're also working on moving services across underlying cloud providers (GCP/AWS and eventually bare metal).
(Render founder) Thanks for using and recommending Render. Happy to answer questions. Incidentally, we launched a Heroku to Render migration tool just a couple of weeks ago: https://render.com/docs/migrate-from-heroku
We're looking at Digital Ocean App Platform for this right now. Seems to be a very similar implementation and similar pricing. Has anyone does a comparison they can share?
58 comments
[ 3.0 ms ] story [ 133 ms ] threadEdit: all my apps are in the EU region, which is eu-west-1 (Ireland)
All our monitor are green, but we did experience a bit of increased latency for about 30 minutes.
And we will reply with the obvious response, "run your status page on different infrastructure".
Edit: status page updated
I don't understand much in cloud devops, I just judge from recurring HN comments saying left and right "your startup should have configured up multiple AZ" when some AWS region is down.
Edit: maybe the dashboard is just back up and I don’t see it down.
Getting geographic failover to work seamlessly generally requires application-level knowledge or at the very least, exotic datastores like DynamoDB or Firebase. For better or worse, that's just not the way most people want to develop.
If not, you have a major consistency issue during failover. Maybe your workload can accept dropping a window of transactions, maybe not.
I think you may have been thinking of multi-region, since you mentioned geographic failover. Multi-AZ isn't really geographic failover.
Still, they're not immune to these types of outages - I believe they're dependent on Google Could but are moving towards running their own data centers at some point. But so far, uptime has been great for us.
Fly.io also looks promising; keeping an eye on that one.
Fly.io's pricing page seems impenetrable to me... I just checked out Render.com and it looks so easy to understand.
I've been wanting to run a Node backend API thing but can't figure out where to put it... been using Cloudflare workers but it kept timing out (going over 10s). Will definitely check out Render's services and $1/mo crons.
How is render for static sites? Do they make you build the site with them or is it a simple "here's an NPM package that github actions can consume".
One of the biggest appeals to me is their simplified, docker-compose like approach to infrastructure as code. Even if I only have a few services, dbs, etc. I still like to have it as code under version control, but things like Pulumi or Terraform are way overkill https://render.com/docs/blueprint-spec
There are still various quality of life improvements I'd like to see, but to me they are on a very attractive path
Thanks so much for pointing it out.
Maybe it is less about new/improved features, I strongly prefer a focused product over one that tries to fulfill everyones whims, but more about guidance on for example for a small team what you think is the “best” way for them to integrate render.com into their dev workflows with typical stacks. How to solve the most common needs and what kind of things are out of scope for render to solve and what could be used instead. For an opinionated tool, it could be a bit clearer, what its opinions are. I think lot the information is already there, but it is quite scattered around between docs, feature request pages and forums.
Specifically related to this AWS outage, Render can already transparently shift your app between AZs, and we're also working on moving services across underlying cloud providers (GCP/AWS and eventually bare metal).