Ask HN: So you moved off Heroku, where did you go?
So you moved your apps off Heroku, where did you go to and how has it worked out?
Particularly interested in:
1. How much work it took to move apps (be honest)
2. How much experience you had at the time of the migration - e.g. on one extreme your entire devops experience may consist of just Heroku (that's me) or on the other extreme you may be a k8s guru (this helps others gauge how they'll go)
3. How valuable were your learnings? E.g. replacing Heroku with an IAAS instead of another PAAS might take longer but give more fundamental learnings, and hence be worth it for some
4. Cost comparison
5. Summary/description of your apps (e.g. 20 tiny apps with a few hits per month, 5 medium with ~20k hits per month, 2 large with 1-2m hits per month type thing). Please give language/framework.
6. Anything else you want to add
321 comments
[ 3.3 ms ] story [ 296 ms ] thread1) Took a few hours to have everything setup. Transition was easy 2) Not that much. I chose heroku because i hate doing sys admin stuff 3) Can't say for this one, scalingo is a PAAS 4) Scalingo will probably be more expensive for starter plans but as you grow it won't be like heroku's horrible pricing 5) Mostly app with no traffic to be honest
Thanks to a couple of people suggesting scalingo here on HN, I gave it a try last night and I'm very impressed with the offering. It took me less than an hour to get our rails template running - with no code changes at all.
The prices are comparable or lower than heroku, and unlike heroku your not forced into paying much more when your memory usage starts exceeding 512 mb.
No affiliation, I’m actually preferring the experience. Heroku is showing its age now.
I heard keyob is decent, otherwise fly.io. Render and railways.app have inactive or day limited services.
It's not as easy as Heroku as there's no managed Postgres, but it's well-documented with a few paths.
However, the main reason Fly stands out is their focus on multi-region deployments. Ever wondered why facebook.com is fast no matter where you are? Fly provides app-level tooling to make this superpower available to everyone.
It's especially important if you're making heavy use of Web Sockets.
Everyone wants to be the new Heroku, but that should be table stakes. Fly is innovative in the same way Heroku was when it launched.
They're doing that in the app? GeoDNS has been around for quite some time: https://en.wikipedia.org/wiki/GeoDNS or more thoroughly explained: https://jameshfisher.com/2017/02/08/how-does-geodns-work/
One example of app-level tooling is the fly-replay header, which permits your app to automatically replay requests in different regions. This works really well alongside the FLY-REGION environment variable that is available to each node. https://fly.io/docs/reference/fly-replay/
Another example of app-level tooling is the availability of NATS Jetstream, a key/value store that is less feature-rich than Redis, but supports multi-region active/active replication. This is a big deal, as Redis doesn't offer it at the free tier, and many others (KeyDB) who advertise it seem to have trouble delivering it when there's more than two regions in play.
One solution to this problem that Fly provides app-level support for is to allow you to have a fast local read-only copy of your database on/near each server, so that only inserts/updates need to run remotely. Your regional copies basically fail whenever there's an update, and Fly's platform re-runs the request on the leading webserver (which replicates the update to the regional database servers).
How well does it work? Never used it. But it would save you the trouble of having to notice the problem, come up with a solution and then implement it.
https://github.com/tomwojcik/django-fly.io-example-project
With 20Eur a month and a VPS on Digital Ocean you can get quite far.
I also made a small project to spin up a PaaS like environment with docker swarm, Portainer and Traefik if you're interested: https://github.com/sergioisidoro/honey-swarm
Yes Hetzner is cheap, but I'm scarred from that event.
API & Platform Results:
Rails: @Railway wins. Render wakeup boot extremely slow (~1m). Flyio cannot run Rails for free.
Crystal: @Render wins. 5-10s wakeup boot time. Railway & Fly keep growing memory infinitely.
Node: @flydotio + Railway both win. Very slow boot on Render.
Stuff like this should always be disclosed. Something like "My co-founder / friend built ..." would be totally fine, but this comment looks like it intentionally obscures the connection. Not a good look.
Plus, I doubt they would spend their lives building the project and not recommend it.
Other project meaning actually founding that project together [1]:
> The Makers for Life collective and project were founded by three entrepreneurs in Nantes. Quentin Adam (CEO Clever Cloud), Baptiste Jamin and Valérian Saliou (Co-founders Crisp) [...]
You live in the same city. You founded a project with him. It's not like you have submitted patches to Linux from different parts of the world, without ever seeing each other.
I don't want to sound like I have something personal (I didn't even know your names until yesterday) but your assumption that the readers here on HN have no clue makes me sick.
If you know each other, that should be disclosed. I have no qualms about recommending a friend's startup, given that the friendship is disclosed.
[1] https://makair.life/2020/05/01/makers-for-life-the-making-of...
The truth is, on this scale Heroku worked well too. The reason to move for me was that they are an EU company so there is less questions from my clients about GDPR compliance.
I'd say the UI is quite old-school and I am missing good built-in metrics. There are some paid add-ons providing metrics but I'd really want to have it built-in.
Very reliable, I've never had a problem with it.
https://aws.amazon.com/lambda/
https://aws.amazon.com/fargate/
https://aws.amazon.com/appsync/
https://aws.amazon.com/elasticbeanstalk/
Whereas, Lambda / Lambda@Edge / CloudFront Functions are of course wayyyyy more expensive (even if more capable) than workers.dev.
AWS doesn't push EB much, partly because it does offer great value and doesn't make them nearly as much money as things like FarGate or AppSync. If you need something that just does what really needs doing cheaply and well, though, give it another look.
(I'm being tongue-in-cheek cheeky here.)
I know you're mostly pushing boundaries with your $150k figure, although at this point it wouldn't surprise me, but AWS is a professional tool aimed at professional engineers. They created Lightsail for "personal stuff."
Right tool for the right job, I guess? Although tyou can actually combine the two on a network level (Lightsail and AWS, that is.)
He's talking about the pricing model not the way to operate it. AWS bills for egress data. You can't operate it in any different way to stop that.
[a] which is fine if you're happy with the inflexibility that free Cloudflare offers. And you live in a country where the free tier doesn't have horrible routing (eg use the Sydney AWS region, put Cloudflare in front of it and then watch your traffic to/from Sydney take a round trip via the US or Singapore)
[b] every single AWS service charges egress fees (ie Cloudfront doesn't help at all)
[c] this does nothing for non-static assets
Am I missing something?
No, but you're not the OP above my comment, so my question still stands.
> (eg use the Sydney AWS region, put Cloudflare in front of it and then watch your traffic to/from Sydney take a round trip via the US or Singapore)
I don't understand? I don't have this issue (I'm in Brisbane; I use ap-southeast-2)
> [c] this does nothing for non-static assets
Non-static assets are going to be very tiny in most cases, and the problem then becomes about volume. If you've got volume and your business model doesn't suck, then you can afford the rate (my understanding is AWS' network egress charges are gross compared to other vendors.)
Every service on AWS charges egress fees that's my comment. There are other cloud operators that do not. I can safely run some static compute / storage / network at a fixed cost, you can't do this on AWS.
If too many people come to my website it won't wipe out my credit card. The site might go offline but I'd rather take that than a huge bill.
No provider on the planet gives you truly unlimited, fixed cost networking throughput. None.
AWS provides Lightsail for a fixed cost, static compute, storage, and networking solution. It's a not strawman argument just because you don't understand it.
There was no misconfiguration, just millions of DNS requests but not millions of actual users. I was in contact with AWS support multiple times. The only solution was so use AWS Shield Advanced. They did refund most of the charges but it was too risky for me. Even after I moved DNS provider there was DNS requests to the R53 zones. I can highly recommend https://dnsimple.com though.
I'm guessing AWS refunded close to 100% of fees associated with provable bad DNS requests.
AWS did do a refund but it requires me to monitor usage and do some investigation. I really don't want to spend time monitoring DNS requests.
But more important than that is of course to have a CloudFront cache infront of anything that exposes S3 storage.
I have a linode vps with a wordpress blog up but I want to learn docker and host my apps on there for 5 dollars a month.
Basically these three
- caprover
- dokku
- k8s
https://dev.to/timhub/self-host-heroku-alternative-40l4
Found them fairly easy to use, really excellent support.
Interesting feedback--feel like sharing in detail? You can reach me directly at ed@render.com.
I’ve found render the closest to heroku. I was able to setup a service for my rails app, for Postgres and for sidekick.
Not as polished as heroku or maybe fly but like I said they made up for any gaps from the bend over backwards live chat support.
https://dokku.com/
In contrast, Coolify has a great GUI that abstracts away the most common things about PaaS hosting, like connecting to GitHub automatically for git push deploys, SSL certificates, reverse proxying and custom domain support, and best of all, having support for Heroku style buildpacks as well as Dockerfiles. I've been quite happy with it, the creator has a Discord and responds to issues very quickly.
With regards to non-self-hosted options, I did try out Render, Fly.io and Railway but I found that their free servers were too anemic. I was compiling a Rust backend and it simply could not compile on their free servers. On Hetzner, for 5 bucks I could get a 2 AMD vCPU and 2 GB RAM machine that was sufficient to compile my Rust apps in a way that the non-self-hosted ones were not. I have a JS frontend app that works fine though but I wanted to keep everything under the same VPS, plus I can run other types of self-hosted services on it too, like Plausible analytics and a Ghost blog. I'm not sure if those are allowed on non-self-hosted options.
All in all, it costs me 5 bucks a month, and I never have to worry about sudden upcharges for traffic à la AWS as in the very worst, my VPS goes down for a while. I'm now running about 20 different services on this 5 dollar box including databases and applications as well as other services, works just fine.
In contrast, with Coolify, you simply login with your GitHub account and it installs a (one-time setup) GitHub app onto your account that automatically sets up the webhooks and I didn't need to do any configuration after that, it just worked for all subsequent apps. That level of ease of use is the experience I had with Heroku and I do not understand why more self-hosted PaaS don't seem to replicate that as well.
Also I like the Coolify GUI over the CapRover one, dark mode plus modern non-Bootstrap design is nice.
You can also use the built-in webhook from inside CapRover control panel but it requires you to log in via your GitHub account in Cap (I created a secondary CI account for this).
Installing a GitHub app comes with challenges, it basically gives Coolify access to your account should they want it, since they control the app and not you. This is bad from a privacy perspective.
CapRover has dark mode. :-) But yes the design is more utilitarian in Cap and not as nice to look at.
Depending on the permissions they ask for, this may be limited to modifying webhooks in the repositories you specify.
And UX wise both have done a pretty good job at being plenty sufficient.
Also when I update docker some apps don't restart properly, but that may just be my (basic Ubuntu dedicated) server though.
Tangentially I was looking at Render.com the other day and they also don't support zero downtime deployments if you have a mounted volume, so it's not very uncommon even on cloud platforms.
As for restarts, I didn't have that problem yet, I believe CapRover adds `restart: always` to all the running containers so they should automatically boot. You might want to check out the logs of the containers that don't restart or just always hard restart the server after a Docker update.
If you're treating application servers like cattle, then none of them should have persistent data (except caching that's disposable). There are fire-drill days that require deployments, so downtime incurred when updating your application code is unacceptable.
To solve this, put your persistent data on a separate server and store it in a proper database (Postgres, for example). Postgres never needs emergency updates, so the downtime for those (infrequent) updates can be pushed to non-peak hours.
Which ones are you missing (except Docker Swarm)?
Also I think the way you create different kinds of resources are confusing in Coolify, like what is a "Git Source" or "Destination", all I want is to run my container. It's very different from Herokus/Caps interface and not in a good way.
Also why can't I just deploy from an image? In CapRover I often spin up a container, like `redis-commander` or `wordpress` and then just configure its env vars and volume mounts. In Coolify it seems you have to connect a GitHub repo which is not a desirable workflow for me.
Resources are defined in this way, because you can connect a lot of things to Coolify, GitHub, GitLab, hosted or self-hosted, later on Gitea and other git sources, and also different destinations, local docker engine, remote docker engine and later on Kubernetes. I will improve the onboarding experience to not feel the first use as a burden. :)
Git based deployment is only required, if you would like to deploy a custom thing, that is currently not supported by Coolify. I'm working on a solution where you can use Docker Hub to deploy images (https://feedback.coolify.io/posts/6/deploy-from-docker-hub).
Now if a service has only a docker-compose file I have to pull it apart and start every service individually.
I will add such DX improvement, for sure!
E.g. I use Fly.io for one of my apps. I build a container image as part of CI on Github Actions and then just push that image to Fly as part of a deploy. Fly never deals with a build step (although it can do that as well).
$5 is very cheap for the Hetzner server you describe though.
https://softwarerecs.stackexchange.com/a/84179/7653
Sailor[1] is a tiny PaaS to install on your servers/VPS that uses git push to deploy micro-apps, micro-services, sites with SSL, on your own servers or VPS
[1] https://github.com/mardix/sailor.
But to answer your questions: 1. It took around 1-2 months of experienced DevOps works (1-2 devops), this included testing and the actual switchover to new AWS infra.
2. We actually used outside contractors (actual k8s gurus) to do the work. But the decision was made in-house by myself. I'm no k8s guru, hell, I'm not even devops, but I have enough experience with the "big guys" so the choice was calculated from technical perspective, considering our growth needs and technical needs for now and for the future.
3. I haven't had any reasons to regret the decision, but I knew that before. I knew AWS offered the things we need and how to use them and what it's roughly going to cost, so this wasn't really a surprise. I did my due diligence.
4. Heroku bills were around 5$k/mon... AWS ones are in 5 digit, but it was expected because we started to use so much more services and resources than we did (or could) in Heroku. We also expanded into two new countries (one of the main reasons the switchover was done) so the costs are sadly not directly comparable. I can say that the new costs are in expected ranges and I'm happy with what we get for the money.
5. We run a eCommerce site with monthly GMV of 1m€+ and around 700k monthly unique visitors. Our stack is (per country) 3 nodejs services, PostgreSQL, redis and some frontend services. Main cost factors are the database(s) and CloudFront.
6. These decision have to made per use-case, per project and taking into consideration your actual infrastructure needs and budget limits. I don't think there isn't anything AWS can't do, but is the cheapest? Of course not. Hell, there have been plenty of topics on this very page about moving your infrastructure to your own bare metal and saving hundreds of thousands per month.
Think through what are the issues you are currently having (ie, for us it was the Herokus stack limitations, couldn't get http2, couldn't do custom monitoring / alerting, no access to LBs, no scalable DB hosting, the need to easily roll out new countries without having to do too much of manual work, weirdly high cost of some services, like redis for an example). If your problems are in the wallet, you need to consider this as your first priority and find a provider which will meet your technical needs with best price. All the big guys also have cost calculators available which will allow you to get an estimation on what you would be paying. Take time, this isn't an easy decision and it will affect you a lot in the future as well, so you don't want to get it wrong to be stuck with another set of issues.
DO App Platform is better than last year but still a bit quirky. I haven't found a better alternative that is more 1) mature, 2) reliable, 3) paid/supported, 4) enough features.
Awesome to hear Password Pusher is going very well :)
I was looking for somewhere I could run web services and cron jobs both in the same place.
They have a Heroku importer, however I think you need to ask to have it turned on. I found that it was easier for me to build docker images for my apps and use a 'build service' instead however. YMMV
I found their support to be very responsive and enjoy using their UI. The UI, builds and so on all feel very fast.
Northflank can run databases, however for my databases I've been running them on ElephantSQL for some time (https://www.elephantsql.com) - even when I was on Heroku.
Free for Dev's PaaS list is worth a review: https://free-for.dev/#/?id=paas
I actually don't see a way to delete a project once created either.
You can delete your project navigating to the billing page inside a project. To delete your account you can send a support request and we can process your request (described in the privacy policy). We'd like to automate it more, however we'd like a formal opt-in via email of the account/team owner when deleting backups and stateful workloads right now.
Addon disk pricing is the same as our volume pricing for services.
The disks are SSDs. We've added a margin on-top of GCP, EC2 and Azure SSD pricing so I wouldn't say they are expensive in comparison to other providers.
It's possible to configure HDD storage which is much cheaper, would be happy to enable that feature flag for you. SSD $0.30 per GB, HDD $0.15 per GB. We'll add HDD pricing to the site and start to enable it by default for everyone.
Edit: though I don't want to pay for staticman stuff. Are you running them for free?