Tell HN: Heroku deleted my database with no warning
Before the hate comes out, yes I know Heroku deprecated free tiers. However, I did not understand this would affect my projects on paid dynos. The real issue here is that I never received a single email or notice of any kind to my email about this. From researching, it appears most people received SEVERAL notices about this. I did not think there was an issue with my setup because I received zero communication.
Upon reaching out, Heroku has told me that they cannot recover the database. They also admitted that there was "an issue" sending out notifications to me, and confirmed that none were sent.
So I guess just a warning to all - your database might be nuked at any time. I learned my lesson about not doing an offsite backup regularly. I guess the bigger lesson though is that Heroku should really be a last resort option for projects these days. RIP.
215 comments
[ 2.5 ms ] story [ 280 ms ] thread$5-10 a month is pocket change in some contexts, but for a hobby or a one-off project, it's a huge chunk of change.
I really wish they offered a lower-cost tier.
(Context: I maintain an app for an event that runs about one week a year. The ability to use free dynos the other 51 weeks means we retain the ability to do one-off analytics queries, minor development work, etc. during the off-season, without having to delete and recreate the app or something every year. Eco isn't quite as good as free, but it means we can still have separate staging and production instances during the off-season, without paying extra for staging to be idle, and without having to destroy and recreate staging every year.)
You can also go to any competitors, which have their smallest tier around the same price. Linode has a $5/m tier.
You could also buy a Raspberry Pi for $50 and run it in your closet.
I don't want to pay $5 / month indefinitely to host a weekend project.
I don't want to deal with the complexities of hosting a weekend project myself. I've done plenty of that.
I just don't want to end up spending $100-200 / month to host a lifetime of weekend projects and experiments.
That involves port forwarding, figuring out something like dyndns, and whatever I need to share domains on a single IP.
Sure there are others out there that provide the same easy of use, I am just too lazy to switch ;)
OTOH I've never had them wipe a database on me, so I guess that would be the motivation to move on.
Give it time, apparently ;)
If you're like me and are really into ease of use, I really endorse Digital Ocean. People don't seem to use it in Big Important Companies, but I never had a problem with anything I hosted there when I was my last, medium-sized company, and everything was so easy to manage in the GUI without having to learn proprietary APIs and whole stacks of infrastructure-as-code stuff like in AWS land.
I haven't looked recently but does DO have the heroku click to add resources and git push to deploy offerings?
About a year ago they shut down my personal blog based on a vague "violation of our terms." There was no violation, either. (I suspect they had to shut down phishing and other abusive sites, and cast a little too wide of a net.)
When they claimed they turned it back on, I had to keep pestering them because they goofed something or other. Eventually I couldn't push updates via Github. (I wrote the blog engine as an exercise to learn NodeJS.)
Again, if I could have bought an ultra-cheap tier, I would have been happy to pay.
Nope.
In my view this has caused yet further reputational harm for Heroku, and is going to have a long-term effect on the bottom line from paid projects. The value prop of Heroku has always been being able to sleep at night, but clearly that's gone now.
I also assume Salesforce only bought them as a cash generator and has no interest in investing in it. So if they saved bottom line from this move, that's a win for them.
(Feel free to correct my assumptions if I'm very wrong)
Heroku is still quite a bit of value-added over "AWS with some basic CI integration", people chose it and still choose it because it requires a lot lot lot less in-house expertise and management hours than AWS for most kinds of standard apps. (I'm not totally sure what AWS services/architecutres you are thinking of when you say that; but I'll say: for pretty much all of them.)
(Heroku def has more competition in that space than it did 10 years ago, opinions differ on the relative merits)
AWS of course already existed when Heroku was launched, so if you do consider them just "AWS with some CI integration" then I guess they would have been from the start? What would have made that true now if it wasn't then?
I think Salesforce thought it would somehow be more "synergistic" with their other offerings than it has turned out to be. That they'd get Salesforce customers on heroku when they needed something beyond the "no-code" tools Salesforce already provided, or that they'd do better at converting heroku customers to salesforce customers than they have been. It does seem to be true that salesforce stopped really investing in new heroku features or improvements some years ago, and seem to be looking to minimize costs while continuing to collect revenue, I agree. (Sometimes I'm not even sure how much they care about continuing to collect revenue...)
> it requires a lot lot lot less in-house expertise and management hours than AWS
Yeah, you're not wrong there. I do worry though that it's still too complex to really trust someone with no devops bg to be fully responsible for, and at the same time, it's so outmoded (just meaning, it's not that popular now besides for legacy projects) that you aren't going to find a lot of good candidates who are expert with Heroku compared to expert with AWS.
So to me, Heroku is a gamble that you can get by indefinitely with only what you(r existing eng staff) know how to do in Heroku. Which for some projects, maybe that's a decent gamble for the payoff of easier admin in the short term.
> if you do consider them just "AWS with some CI integration" then I guess they would have been from the start? What would have made that true now if it wasn't then?
Yeah good point -- Let's see if I can try to explain my claim better. I think in 2009 using AWS was completely inelegant in every way, it was pretty barebones and basically just "Rent an EC2 server and do whatever you want, good luck." Heroku had a unique (at that time) approach which made cloud hosting much simpler and more abstract. AWS today at least offers a few ways you can host an application on AWS while doing less server maintenance than you'd have needed in '09. Infrastructure As Code is also something I think has a lot of popularity today that wasn't really a thing in 2009, and Heroku if I remember correctly, isn't really about that. So there isn't as much repeatability on that platform, it's more "click around and build your stuff out in the GUI." Which again, is cool in the short term to bootstrap, but can be painful at a bigger scale.
As an early Heroku employee, this gave me quite the chuckle.
Also re the timeline, the acquisition was announced in 2010. So your own usage was already a few years post.
I just imagined Benioff going "Cool, we can buy this and continue to collect the checks every month, while not developing the UI or integrating it in any way into our business." Basically the same thing they did with Exact Target.
You could easily block all incoming connections to the database. For a free database of 10k rows there were no SLAs, and you would still technically be hosting the database.
Even taking a dump and emailing it to me feels like a safer option here.
There were better answers here for sure. If the honest answer is we just didn't feel the effort was worth it for this class of users at least own that.
Probably not, because it’s ” difficult “
You can pick an arbitrary time frame for retention, but whatever you pick, you have to communicate to users, and you can't just change it on a whim. Normal customers want this clock short. They don't want you to retain their stuff after they cancel.
I highly doubt normal customers want this clock short when the cancellation is not customer initiated.
Thus didn't happen because of the contract, but because the people implementing this transition didn't give a crap.
Silos lead to issues like this. Cross-functional teams cost more, etc.
Onboarding users/partners would be easier without making them read/click all those consent checkboxes, etc.
Ideally, product people defend the end users. Realistically, compliance people defend the company from getting screwed in an audit and ultimately sued by the government.
I had a vague thought for a moment about maybe being able to give customers the option to opt in to an updated set of terms that would've been better designed for the situation ... but since the original Tell HN author didn't get any emails about this at all, presumably they wouldn't've got that one either so even if (and given 'vague thought' I'm not claiming any particular level of 'good idea' to it) that had been done it presumably wouldn't've helped them anyway.
It seems to me the root problem here isn't so much the compliance initiated policies as a complete failure to take 'make sure people have been adequately warned' sufficiently seriously given the potential consequences.
Deeply unfortunate.
(read that last sentence in understated deadpan dry en_UK to get an appropriate read on just how impressed I'm not by the communications cockup)
Why could they not turn it into a read-only database without access from the Heroku apps instead? Then it'd just be a routine change to the service offered, would it not?
I genuinely had to read this twice to get the intended meaning.
i'd probably prefer feces-by-email to surprise database deletion
The company has no commercial interest in doing that, though.
That's hiding behind the T&Cs instead of owning their decision not to even try because there is nothing in it for them.
The T&Cs are an agreement between the parties. That agreement can be changed at any time if both parties agree. So they just need to ask.
Do you really never get e-mails from various companies every few weeks telling you that the terms of service have been updated?
I think I get one from eBay alone every other day.
I can't be the only one who's basically completely blind to emails from major companies, including SaaS providers, because they're so fucking spammy that the SNR is like 1:99. Notifying me by email, for one of these places, is functionally the same as not notifying me at all.
[EDIT] Sorry, didn't mean to imply the parent wasn't paying attention, just that I'd fully expect a very high percentage of their users to miss the warning in all the noise even if they emailed everyone—even if they emailed them a couple times, actually. That's the cost of every company sending out tons of "join our online seminar on [product]!" and "hey, look, it's our newsletter you never read!" and "it's time for our weekly TOS modification!" emails.
This 1000x. I signed up for an SMS gateway service last year. Just for my own hobby use, nothing major. I gave them $10 to start service, and they charge like 2¢ or something per outgoing message.
They have like 180 different prices for 180 countries, territories, provinces, parishes, cantons, prefectures, etc. Every week one of those prices changes, and I get an email notifying me of that. I tried to turn those off in my preferences, but they refused. I can opt out of marketing, weekly digests, and "tips and tricks"; but I can't opt out of pricing notices.
So I added a rule on my end to hide those. I totally understand where they're coming from. They can't NOT give pricing change warnings. But at the same time, in the flood of constant notices, there may be something major I will miss.
It would be nice if they instead gave me the option to never spend more than $0.XX per message, and return an API error if an attempted send fails for price threshold violation. Then the spam wouldn't be needed.
The fact that the failure to send a single warning email to this particular customer wasn't both detected and considered a five alarm fire level bug under the circumstances is the thing that moved the decision to sunset the free tier from sad to disastrous for the user who posted the Tell HN in the first place.
Plus, it would all likely have worked out fine if they'd emailed the customer a warning or three like they intended to do - it was the failure to do so combined with the failure to detect and remediate the initial failure that sent things down such a dark path here.
Whoever fostered that naive interpretation was a nitwit. If they’re an actual lawyer, they promoted an intentional, mutually harmful unilateral reinterpretation of an agreement and should be sacked.
Cowering behind T&Cs like this is intellectual bankruptcy. There’s always another solution. The law is not a programming language.
Contracts with some customers, surely? You could have the default be 90+ days, then those customers whose contracts specify a shorter timeframe get that shorter timeframe configured on their account instead. You could give the customer the choice at signup, and let them change it later using the settings console. If their contract doesn't specify a period, send them a notification that you will be changing it to 90+ days, but telling them they have the right to object if they disagree with that.
Yea, no. You decided to make the decision for contracts to be that way. The fact that you "fought hard" but that decided on the 30 day retention anyways means that clearly the opinions of engineers don't matter and that the company is completely captured by the lawyers and out of touch executives. It hardly inspires confidence.
It also doesn't at all address the fact that you failed to contact an apparently paying customer that their data was about to be nuked, contract or no.
How Heroku missed this is beyond me. They managed to screw over paying customers in their broad attempt to stop freeloaders.
These are good accounts with credit cards on file. Why not just autocovert me to the lowest tier paid database?
FWIW I was able to get them to restore my databases. But I also had free Heroku Redis on one my projects and that, they assured me, is gone forever.
[0] - https://redis.io/docs/management/persistence/
So I take this opportunity to ask, what alternatives have people moved to? I really haven't gone back to look what's out there.
Not looking for free, just alternatives to review.
For small apps, if containerized, I would host on Google Cloud Platform using App Engine Flexible Environment: https://cloud.google.com/appengine/docs/flexible Add on their managed database, of course.
But also, random VPSs are so cheap that if it works fine as is, you could do a lot worse than just running it on Linode, Digital Ocean, Hetzner, etc.
Junk mail wasn't the issue in this case.
Heroku is a shitshow after the Salesforce takeover and not to shit on you because I know it really sucks. BUT please everyone, do offsite backups and test them. Please people. Please. If you have anything that is important, BACK THEM UP on your own outside of the provider.
Heck, we wrote our own script to backup RDS databases offsite as well even though RDS has backups and restore options. I want that database file.
(guess based only on having watched things unfold from the outside without ever being a user/customer so please do add salt to taste)
1. GitHub integration breach, which wasn't handled well but luckily didn't affect me.
2. No more free-tier DBs :(
3. Salesforce logo, which ofc doesn't matter.
Hearing vague negative things here makes me nervous about the future because I really like Heroku and don't want to be stuck using AWS directly.
I would suggest you start poking around at possible alternatives just in case, but not as yet with any great sense of urgency.
If you Ctrl-F for crunchydata in this comment section you'll find an employee of theirs talking about their postgres hosting and listing a bunch of services their customer use for the other parts of the puzzle, and have hired enough heavyweight core contributors and active and clueful community members that I think it's reasonable to say they're -good- at postgres.
I would say that what I expect to be most likely to happen is gradually increasing prices and gradually degrading service (the people still working on it do seem to care but I don't know if there are enough of them left to avoid bitrot setting in and even if there are today, in a year or two there may not be) so having a plan to move in an orderly fashion but not executing it yet seems like the wise approach.
Neither panicking and moving in a fast and risky way now, nor waiting until (if it happens) things go from aggravating to actively intolerable and moving in a fast and risky way then, are likely to be particularly good ideas.
OTOH, being prepared to migrate elsewhere in an olderly fashion if the cost/benefit calculations tell you it's time, re-running those calculations and double checking your plan every so often, and continuing to enjoy the service in the mean time, seems like a reasonable, responsible, and overall relatively pleasant way forwards.
(disclaimer: I am not a Heroku user myself, but I think the general principles almost certainly apply here just fine, and I'd certainly be comfortable giving the same advice to a consultancy client at work, so without being foolish enough to claim I'm definitely right I'd suggest the above analysis is at least worthy of giving serious consideration to)
Then, year by year, those things were taken away. Eventually, circa 2018, investment in the core H product started being reduced and individuals and teams started being reorged to focus on Salesforce priorities. H became a "keeping the lights on" job. Folks who wanted to keep pushing left to work on GitHub actions or on render.com, netlify, other places still striving for great devex.
They took some half-hearted efforts to back up the data, but it was far from ideal.
Back up your data, and do it in different places so a failure in one won't affect any other copy of the data.
Unless you're doing regular restores, you don't have a backup. You have hope. So yes, doing backups in a way that gives you some form of guarantee isn't exactly cheap.
[1] https://imapsync.lamiral.info/
1. Setup IMAP backups locally on my computer and everything is backed up on computer to 3rd party backup tool
2. Use Google Takeout from time to time (it is a bit weird at times though).
(or just maybe we shouldn't have thrown out all the sysadmins with the bathwater)
I would like to do backups but it's already not really easy, even harder when you add the whole "figure out how to do backups" to it.
On Heroku, databases and dynos are completely distinct entities, each with their own payment plans.
When you set up a dyno and “add” a database, as most people do, it’s really easy to think that they are part of the same plan, and conclude that the “pro” or paid dyno is actually the combo of the dyno and database. They are not.
I understand why they did this, databases can be shared between applications, that’s handy, but this is a sharp edge.
This gets people on the phone with support quickly if they don't react to their account manager or mails. Like, we've had a possible security incident coming from a customer system and they didn't react - because they tried to hide their processes internally. Scream-Testing got us on calls with their InfoSec in a hurry.
"Remind HN: Heroku will delete all free dbs and shut down all free dynos Monday" https://news.ycombinator.com/item?id=33755651
I still use Heroku on a daily basis and wouldn't say this has caused me to re-evaluate my decision to stay with them, but then again i'm not using anything free from them.
What pissed me off most is that I WAS paying for the account. I was paying $17/mo for redis and dynos, so there was an active card on file.
Why not just start charging for the postgres db, and only delete if there's no active billing?
Heroku was already a no-go for me with new projects, I just keep old things running in there since it's too much work to migrate off. This just cements that for me.
Probably because the original terms you agreed to were written not anticipating Salesforce perpetrating this and so while I would not at all be surprised if the vast majority of customers in your position would've been entirely happy with it they probably didn't have a legal path to do so.
That makes no sense. Most large companies change both agreements and prices via notification emails all the time. Probably even SalesForce does so.
If you want to do a close read of the terms and conditions/contract language as of a reasonable guess as to when OP signed up and link the section you believe does give them the right to make such a change unilaterally, I'd be happy to read it and discuss further.
My current position in the mean time will, however, remain that while there's always a chance (often a pretty good one) that my hypothesis is wrong, claiming it makes no sense at all is a stronger claim than is justified by the information forming the basis of this discussion.
Not the end of the world for me, because 95% of the data I had in there had already been processed, but I did lose some. Complete joke of a service.
1. probably the notice sending and db deletion is two separate teams or responsibilities. Are they?
2. Did people know there was a bug in notice sending so some notices were not being sent, all notices not being sent? I ask this because generally in places I've worked where notice sending was an important part of things you knew if there was a bug and notices were not being sent. But maybe it wasn't that important for Heroku. Maybe it was not known that notices were not being sent for a while - or was it known immediately but things on other parts of business chugged along anyway.
3. If they knew notices were not being sent and they went ahead and deleted db anyway, seems messed up, but that would probably be ok with people if they had data retention for people who did not get notices sent.
4. The whole thing your stuff can be deleted at any time without telling you is basically probably true almost everywhere in that notice sending can have a bug and deletion of stuff is probably not adequately tied to notice sending so that if notice not sent automatic deletion is stopped. Which I'm thinking is probably everywhere - if you work somewhere with automatic deletion and a notice sending module - what happens? Is this scenario handled?
5. answer to this is probably not, but is there a legal issue if notices not sent and stuff deleted, issue might be if some notices were sent - if account A gets notice about deletion and is thus able to act on it and account B does not get notice and is thus not able to act on it there might be a ground for action. Probably not, but when something seems unfair there might be a law that can be stretched to fit it.
Note that this is not to say the end result wasn't an indefensible disaster, only that disasters seldom have only a single cause and the above is my best uneducated guess at how things came together to cause this one.
My personal account did get a bunch of warning emails but nothing on our business accounts.
Make offsite backups.
For this use case, a raspberry pi 4 with a 1TB SD card hidden somewhere in your home with a cronjob is probably more than enough.
I have no faith that if a rouge employee clicks the "delete app" button (because did you know to give an employee the ability to update the ssl cert on your web app you also need to give them permission to delete the whole damn app?) you'll ever be able to get your database back (although you might have a 30 day window to do so, but I wouldn't trust it.)
I wonder how many databases they deleted that they could have instead started charging for. Seems to be incredibly destructive for all involved. (SF shareholders, Heroku customers, etc.)