HN Explorer

Heroku was Down

254 points by ericpauley ↗ HN
Incident: https://status.heroku.com/incidents/2402

Update: our apps appear back up after 23 minutes total downtime. Others are reporting applications still down.

Update: it appears most or all services have been restored.

148 comments

[ 299 ms ] story [ 3091 ms ] thread
[deleted]
(comment deleted)
emilsedgh
Yes it appears to be down for us as well.
ericpauley
Our app is currently 100% down at the router layer, and dashboard requests are mostly failing.
VWWHFSfQ
Routing and logs are down for me as well
btown
Same here, hard down for us
pwned1
It would be nice if they updated their status page. sigh
teagee
HN is truly a market leader in status page technology
[deleted]
(comment deleted)
duxup
Except when it is a false alarm.

I've seen a few get to the front page.

But when we're right we're right.

dec0dedab0de
Even when it's a false alarm it's usually something else is having a problem that is affecting many people and manifesting itself as a particular service being down.
jonpon
I came to HN to make sure it was actually down.
js4ever
HN was also down for a moment because too much traffic
chasd00
A couple of outages that have affected my client i learned about first on HN. We were able to mobilize a team and get on top of it faster than any monitoring team at the client ( a state government ). I feel like HN should invoice us haha
castaway3000
You might benefit from some better monitoring!
odiroot
Such a lightweight tool at that! Barely any JS loaded.
thecleaner
I'll make my employer pay for it if HN starts a paid service. Honestly they should write a blog post about when down times were first reported on Hn.
iamricks
So... what are the chances this is related to a cyber attack from Russia? Also our apps are down
brimble
Now is a good time for everyone to double-check their backups and go over their disaster recovery plans. I know I am.

I mean, yesterday was a good day. But today's the best you've got if you didn't do it then.

driverdan
Practically zero. Why would Russia care about Heroku?
adamnemecek
Maybe not heroku per se but something hosted on heroku?
Trasmatta
Heroku runs on AWS, so it would be an AWS attack of some sort. But AWS in general seems to be up from what I can tell, so I'm feeling like it's not a cyberattack, and just a Heroku specific problem.
adamrezich
what would lead you to this conclusion?
2OEH8eoCRo0
https://www.cisa.gov/shields-up
adamrezich
what does this have to do with Heroku? why would it be targeted? especially given that they're running on AWS?
2OEH8eoCRo0
> Every organization in the United States is at risk from cyber threats

Heroku and AWS are organizations no?

> While there are not currently any specific credible threats to the U.S. homeland, we are mindful of the potential for the Russian government to consider escalating its destabilizing actions in ways that may impact others outside of Ukraine.

adamrezich
this morning at work we had a database connection issue for a few hours. I work for a school district, which is an organization. therefore, it was probably a Russian cyberattack.

like what

2OEH8eoCRo0
> what would lead you to this conclusion?

I'm just posting what would lead somebody to that conclusion. Nobody is definitively saying anything right now.

adamrezich
yes, and I'm just pointing out how ridiculous of a conclusion this is.
seannui
No update to their own status or their Twitter (@herokustatus) after 10+ minutes. Pro.
canadianwriter
Gotta love asking in slack "is uh... literally everything down right now?"
ezekg
I've been with Heroku for 7 years and this is the first time in recent memory that I've seen Heroku completely go down. Nothing at all works.
digerata
Ahh... were you sleeping in 2021? They went down in September, November, and December. Probably some other times earlier in the year as well. I stopped keeping track at how bad their service is because of how bad AWS's service is.
ezekg
I'm well aware of their other outages. But I don't remember another time when they were completely down like this, even down to their status page.
digerata
Well, they aren't completely down now either. Heroku Shield is up. Dashboard is up. EU is reported as up.

As an aside, I spoke with our AE in early January on where they are going in dealing with AWS unreliability. One would think they would have a good answer. They don't.

jrochkind1
My app didn't go down during any of those times.

I recall at least one of them I could not deploy new versions of my app though, which is pretty bad. I don't believe I had any downtime at all due to heroku platform outages in 2021. i believe you if you say you did though!

This particular outage definitely seems to be of a rare level of severity.

heyoni
Their authentication system completely broke I think in January...for hours.
jrochkind1
Oh right, I remember that now.

Didn't bring my deployed app down though! No user-facing outage for me.

This time my app was down for about 35 minutes.

heyoni
I think apps still worked but I'm not sure. I know I couldn't create a privatelink to our AWS environment because the CLI kept failing, so we were locked out of a dev database. Not too bad.
driverdan
You haven't been paying attention. We're very heavy Heroku users. Their dashboard/API has gone down a few times in the past year. Overall they've been reasonably good.

It's much better than it was 5+ years ago. Back then they had almost weekly downtime.

ezekg
I am paying attention. My business runs on Heroku. I never said they don't go down... they go down quite a lot (much to my dismay), but never completely down like this one. Even their status page is down, which is a first for me. Must be bad.
bin_bash
they had their big layoff in 2020 or 2021 though so I wouldn't expect it to improve
jlangenauer
All our apps are down as well. And it seems (and I really have to almost laugh here) that https://status.heroku.com no longer loads.
dspillett
As always, a note to infrastructure providers: HOST YOUR LIVE STATUS DETAILS ON OTHER INFRASTRUCTURE.

Of course, they won't. If they host is on someone else's then that might look bad (tacitly saying that a competitor is reliable and might be up when they are down) and if they hive off an extra copy of some of their infrastructure there will still be single points of failure either accidentally, by human error (someone somehow messing up both segments at once), or by design (possibly through management trying to save pennies when they noticed this extra bit of infrastructure on the balance sheet).

bin_bash
I am pretty sure Heroku has a Linode box or something for their status page. I may be wrong though. They may also have moved it but that would seem like an incredibly dumb idea.

I worked there and I vaguely remember something like this but it's been a long time.

If I had to guess this is probably a DNS issue (it's always DNS).

wuputah
I agree with your lead statement and argued as such, but was overruled. Last I knew and understood, Heroku Status is static pages pushed out to Fastly, with the internal admin site (that does that work) running in a Heroku Private Space. If you look at the DNS, it still appears to be served by Fastly, and Heroku Private Spaces are generally pretty isolated infra, so I would be curious what the failure mode was here. But ultimately this is the fire you play with when you self-host your status site...
reubano
Yep, all 6 of my Heroku apps are down.
reubano
Interestingly enough, my 2 Squarespace sites weren't affected. Anyone know who they host with?
andygcook
Our app is currently down as well and we host on Heroku. We received a down alert from our monitoring service at 11:32AM EST.
perryraskin
My Heroku app is completely down as well
[deleted]
(comment deleted)
reubano
AWS is down too, this is likely the cause since Heroku runs on it. https://downdetector.com/status/aws-amazon-web-services/
jermaustin1
AWS status page is up, but going VERY slow, and says no issues.
XzAeRosho
As usual whenever there's an AWS outage
reubano
I still don't get why they use images

    <td class="bb pad04 top center" style="width: 32px">
      <img src="/images/status0.gif">
    </td>
Instead of unicode:https://www.htmlsymbols.xyz/unicode/U%2b2705
jermaustin1
if the image tag is drawn dynamically then it is actually probably less bandwidth than the unicode character since the image can be cached at multiple locations including the browser.
cure
.... no way.

Cache freshness checks involve a lot of headers, which take up way more bandwidth than one unicode character.

jermaustin1
if you care about cache freshness - you could set the cache header to 10 years and it never be an issue again.
jrochkind1
Less bandwidth than the 1-4 bytes of a codepoint in UTF-8? How do you figure?
jermaustin1
Because if it is cached, then it is 0 bytes transferred. First request could be probably a few hundred bytes, but never needed again. And once it is at a CDN, there is never another request to the server.
cure
It's because they can host the images in S3 buckets, so that their status page goes down when S3 is down.

/s

But, this really happened in 2017.

mattwad
AWS must be only partially down, all of it is running fine for me in us-east-1. Elastic beanstalk ftw! :)
throwthere
If us-east-1 is up I have a heard time believing ANYTHING aws is down
jmartens
What is the proof that AWS is down? Functional monitoring of AWS by metrist.io (I'm a co-founder) shows no AWS problems. Downdetector is not a reliable source.
reubano
What makes Downdetector unreliable? It's showing a huge spike right now.
djbusby
Their (metrist) claim is that DD is human reported and therefore unreliable.

Metrist monitors via bots

jmartens
It can be useful, but you have to take it with a grain of salt. A perfect example is the recent Facebook (Meta) outage. When that happened, Downdetector showed that ATT, Verizon, and T-Mobile all had issues. They didn't, it was just Facebook and users mentioned or otherwise claimed that it was their mobile carrier.
zorpner
Downdetector relies on user reports, so e.g. if a user's ISP is down and they can't get to Facebook, they might report Facebook being down (or vice versa). DD spikes are typically indicative of _something_, but it's not always the actual down service.
jaywalk
It's solely based off social media and user reports. It's the "smoke" in the saying "where there's smoke, there's fire" with the caveat that in some cases there's actually no fire even if there's a decent amount of smoke.
djbusby
Neat! Wish you had a clickable demo rather than just screenshots.
jmartens
Thanks for the feedback, we can do that soon.
nuggien
maybe your service isn't as reliable as you think it is ;)
jrochkind1
AWS status page is all green.... Just kidding! I know the information content of the AWS status page is literally zero, it's always green!
[deleted]
(comment deleted)
mabsboza
reporting from Nicaragua, our apps are down!