I got lucky and hit all of these problems in my first few hours of trying Fly (much to their detriment, unfortunately). I wanted to love it, but I encountered nothing but bugs when I played with it over a weekend a month or so ago. Internal DNS worked for one but not the other of my nodes; Dashboard accepted my credit card but didn't actually upgrade my plan; SSH console would hang, requiring killing the client process with SIGKILL; my builder hung with a Dockerfile that was literally just "FROM ubuntu" and "apt update"; this failure resulted in `flyctl machines list` hanging and `flyctl machines kill` hanging. Maybe it was just a bad weekend at Fly, but their status page was all green, too.
So many problems between me and even giving fly a serious try. Unfortunate, but maybe in a year it will be better.
It'll definitely be better in a year. It sounds like you hit some of our capacity issues. In a year we'll either be ahead of those, or not growing anymore due to ongoing capacity issues. I'm hoping for the former.
The SSH and Builder issues are interesting. Those sound like something related to our wireguard stack.
Yeah, that would certainly have slowed me down too if I'd experienced that. I haven't had any of these issues. Fingers crossed! And hopefully they're working on stability.
My account got marked as "high possibility of fraud", even though I validated my email and added a credit card they even charged. The error message said that I had to contact support@fly.io but that address only accepts email from paid subscriptions.
With no way to evaluate the service, I opted to delete my account.
Exactly what happened to me as well. In my case, I was (fortunately?) messaged by someone from Fly.io after asking around the community forum how to remedy the issue. I think the person I talked to was inferring that the system got tripped because I was using my own domain for my email address.
It's since been corrected, and I still have a handful of services running with them, but getting started with the platform definitely was a rougher experience than I was expecting it to be.
Large company? Literally every company I ever worked for, the smallest one were 5 people, had their own domain for emails. And they were not all in IT.
I don't know what merchant system they are using but I will say it is very common for them to have their own anti fraud detection and rejection on charges. Your specific case could be related to a huge list of possibilities (even many of them a rollup of many other interactions on other sites that happen to use the same merchant or foundational fraud data).
Thinking this is just because of a domain name is silly.
Also Fly.io -- You may want to clearly accept and manage trial issues (or at least a subset) via a support path. It seems silly to effectively bounce users with the impression that you have no support because they are not YET paying customers while in the trial phase.
Sure, but what's edge? The world is big. Can't have servers everywhere (ok, maybe Cloudflare is close). It's possible due to routing decisions that you get sent to a "different edge" at different times.
So what's the point then? 250ms is enough to literally send traffic to the other side of the world. So why bother with edges if you can just have a single server where most of your customers are and get the same latencies?
I thought it could have been because of a shared CPU at first as well, but I upgraded to dedicated CPUs and it didn't make a difference.
I don't know whether this is a platform issue with Fly or something else, but it was kind of annoying to have such a high variation with seemingly no way to fix it.
After I have experienced several fly.io render railway northflank platforms, I think the problem with fly.io is that the web console is not fully functional, too dependent on CLI and the response time of the application is unstable. The problem with render is that it is too slow to build, north is similar to railway, and the experience is good. But although the north console is very comprehensive, it is too complicated and dazzling. By contrast, the interface of railway is very simple and comprehensive, and UX is great. (found by wappalyzer analysis that it is based on nextjs,very nice)
I tried railway and northflank. Really like the UI northflank has; I think anyone familiar with Kubernetes should feel right at home. I also agree with the sentiment that railway ui is simpler.
More context:
The UI shows that the service is available but when I visit the live url, I get a 404.
What I think is happening is that the UI shows the status of the underlying container and there is some fault with the reverse proxy they are using to expose the container to the internet.
Fly.io doesn't have this issue but I am leery of all the complaints here.
Hey there fyzix! Angelo here, Support Engineer from Railway. I was here procrastinating from work when I saw this comment.
The downtime you see is our proxy taking time to cutover- however, we have https://docs.railway.app/deploy/healthchecks that will only cutover once we have a 200 from your API. This way we can keep your old deploy live and serving requests if and only if it's live.
Theres more we can do to make it magical, but this should help in the meantime.
This could be done automatically by pinging the root and searching for a header set by railway's default page. If it doesn't exist then the service is live.
We used to do that by default, but it led to issues with load on our proxy. (Imagine n * m pings against your host for a shit-tonne (official measurement) of builds + deploys a day.)
We are hiring Network Engineers for exactly this reason :')
Yes, but I think the difference between free layer and paid users should be distinguished in terms of capacity and scalability. This crude entry experience may lead to a reduction in user conversion.
There’s an edge case in which you can lose access to your account as well when switching from github to email login with the same email. It’s an edge case but quite a bad one.
Yep. Fly isn't fully baked. Deployed a service earlier this month and got bitten in a few ways:
- It took several hours to get a 1 GB disk provisioned (their API would time out and I'd get left with an allocated but unusable disk that couldn't be deleted without deleting the entire app it was attached to.) The Fly forums have numerous reports of the same issue, going back months, across multiple data centers. All resolved with the moral equivalent of "oops, try again later?"
- The fly.toml reference is incomplete and internally inconsistent, documenting keys which are not valid, while omitting valid keys that appear elsewhere in the docs.
- Volume snapshots are retained for either 5 or 7 days, depending on which of the multiple conflicting docs pages you believe.
- I wanted to let them know about that last one, so I emailed the address listed at https://fly.io/docs/about/support/, only to get an autoresponse saying that they don't read that mailbox and that I should pound sand unless I'm already paying them to read my emails.
Reading between the lines of their recent blog posts, it seems like a lot of the infrastructure pains are pinned on Nomad, with the hope that moving to Fly Machines / Apps V2 will obviate a lot of those issues.
I hope they're right, because what they're trying to deliver is exactly what I want.
Volume provisioning issues have been a game of whackamole, largely as the result of scaling pains. We've grown like 4x in the last 3 months (thanks, Heroku) and it's put a strain on pretty much everything.
Echo this, and I _did_ notice it in my first two weeks.
3 separate issues prevented me deploying. One was my fault, but hard to diagnose because error messages contain no useful information.
The other two were the platform falling over, with the support forum having no resolution but plenty of other people in the same boat.
Lots to love about fly. The latency is noticeably better than anything I’ve used except cloudflare. Prices are good. Hopefully they’ll get the growing pains sorted and we’ll have another contender.
These deploy issues are all kinds of ulcer inducing. They're not strictly Nomad's fault (we're holding it wrong), but the complexity of flyctl -> nomad -> global hosts makes it difficult to bubble errors back up to you.
The TLS cert renewal was a big failure on our part. I know that's, like, a one strike and you're out problem, but we've improved that part of our stack substantially. It should be much more resilient.
Redis/Postgres timeouts sound like they might be a problem we can help with, though. Both those services work through a load balancer with a TCP idle timeout. If you get timeouts after a period of inactivity, it's usually a simple config tweak to have a driver handle that seamlessly.
The timeout issues surprised me, more drivers than we expected can't handle a DB connection timing out when it's idle.
I had (a couple years ago) some weird websocket throttle or debouncing characteristic. I tried to construct an evidence, but it's quite hassle so I just ignored it.
I think they are too engineering culture. Target is devs, sure, but they could use some product guys.
I'd spend some time reading community.fly.io before committing to it. I recently helped a startup that has been on fly.io and there was a lot of frustration about things not working.
it was very frustrating to see un-answered problems reported, fly.io down for what seemed to be many users, the same time seeing a fly.io co-founder arguing about politics all day long here on HN...
In fact, when you enter the fly.io console, you will feel that this is an unfinished construction site, especially when you start to deploy a new app, and the frame text icons in the options are simply truncated, which feels rough.
Maybe I'm missing something, but 500 qps for serving static files seems supremely unimpressive. Fair play about it being more fun, or saving some money from AWS, but I am not convinced that this setup is a good achievement in terms of performance.
Only 1 out of 4 URLs in the test was static. The others are rendered HTML templates (using Go's html/template -- I'll update the article to make that clearer), and one page does two SQL queries.
Note that my load test wasn't to see how much I could squeeze out of this, but to determine that it's "more than good enough" for this use case. The typical qps I get from real customers on the site is about 0.5 qps, so I think I'm good. :-)
I skipped over fly.io because I don't want to run a command to deploy an update to an app. I much prefer Render.com and DigitalOcean's setup where you select a repo and then it takes care of the rest with auto update on git push.
This seems like a trivial thing to add. If I wanted to use a domain specific configuration file then that's always an option, later.
Fly did send an email linking to one of their articles on how to set it up for github maybe a few hours after the initial setup. I think it took me like 5 minutes to setup deploy on git push to main (or your deploy branch)
>I much prefer Render.com and DigitalOcean's setup where you select a repo and then it takes care of the rest with auto update on git push.
Doesn't that force you to use your hosting provider as your CI provider?
I've always avoided the automatic "deploy on git push" solutions because I want to keep my CI provider uncoupled from my hosting provider. That way, I can switch hosts without having to rewrite all my CI code.
To keep CI and hosting decoupled, your CI can push fresh images to a docker registry, and your deployment can poll for updated images. An easy way to achieve this is to add the Watchtower container to your compose files: https://containrrr.dev/watchtower/running-multiple-instances...
I don't think that's a good solution in general unless you're confident that all of your deployments will use Docker Compose.
About 1/3 of my deployments are just static sites, and the rest all fit into a single Docker container, so integrating Compose would be a big increase in complexity.
I'd say Compose simplifies even your single-container deployments, as the compose file is the single place to set your env variables, port forwards etc.
That being said, you can use Watchtower without Compose. It's enough to run
docker run -d -v /var/run/docker.sock:/var/run/docker.sock containrrr/watchtower
I'm really happy with using docker and compose for everything. You really can't beat how well it works on a single host.
Worrying about having to configure systemd? "restart: always" in compose. Caddy updates? Caddy provides an up to date docker image. You can't get around caddy's config, but tbh, you do it once, and then you're done. It's way better than nginx. Deploying a new version of your go application is just "docker compose up -d".
Any issues with persistence, like using Postgres? I heard that can be quite funky.
Just wondering about having a "native" NGINX reverse-proxy + multiple apps ran via docker-compose on my VPN, seems the most hassle free. Currently using it for integration tests and it works great.
Did you maybe consider k8s, maybe as a learning experience?
I'm pretty happy with fly.io, except for one thing: I moved over two apps from Heroku, lured by the free tier. But you do have to enter your credit card details whether or not you intend to pay. And then they do charge me a few bucks per month for an almost completely unused app.
It's probably all in line with the ToS and everything, but I find it a bit shady that I have to investigate this, rather than having a free offering that doesn't suddenly charge me.
CPU - I would assume it's not a mistake, maybe if I dig some more I can figure out how to keep within the allowance, but on Heroku I never had to deal with this kind of thing for free apps.
I agree, as an individual user I much prefer to be locked in to a fixed rate. And not just for the free tier- I don't want surprises even if I'm already paying
As a company with a cushion, hosting a profitable app, it might be worth doing the usage-based pricing to try and save some money. But the risk is just not worth it for smaller projects
I probably have to fiddle with the settings to stay within 3 CPUs, but since it was heavily recommended as an alternative to Heroku's free tier, I'm a bit disappointed I have to. Maybe they never intended to be recommended for that, don't know.
I haven't used it, but from what I know it's not a very good analogue to Heroku (other than being an easy-ish/dev-focused host). Their pricing model is pretty different
True, Heroku allowed as many dynos as you want when their free tier existed, however, they had a limit of 1,000 hours per month. That pretty much meant you couldn't run more than 1 VM (~744 hours/month) for free. Fly.io is more generous as those 3 free VMs can be up and running all the time (but VMs have 256MB of RAM instead of 512).
Right now the most generous free tier is from Oracle (24GB RAM machines feels unreal). But you can find other free tier offerings here: https://github.com/ripienaar/free-for-dev
I tried to look at OCI but was immediately bamboozled with a slow as heck web site (locations table took 15 seconds to load!), a tonne of management BS about "aligning strategic objectives to operational constraints in a dynamic environment" and then was expected to immediately book a call with a salesperson.
"zombiefied" would be better description. Account exists but you don't have permission to do anything and you don't even have permission(lol) to upgrade to paid account
They will yeet your machines without any info tho. I had test (smallest in free tier) VM that just got destroyed one day.
But that is not all
My account is now "inactive", with no option to activate it and with no option to even pay and upgrade to paid tier.
The account is in some broken state where half of the dialogues produce dialogues like "Authorization failed or requested resource not found", like I had account with permissions denied to everything.
Even the "upgrade to paid tier" option gives
> You do not have access to perform this operation. Please contact your cloud account administrator for additional access.
On top of what other people said: capacity for the free tier machines is very limited. I wanted one to fool around with and it took multiple weeks of trying daily at varying times before I was able to actually allocate a VM.
OTOH they don't even charge you if your bill is less than $5/month. I've had an app do nothing for a few months now, and my bills are always just thrown away due them being under $5.
A few months ago, I found out I had an AWS account from college that was racking up $0.50/month in charges. After 40 months, I finally got an email from AWS warning me to pay the $20 or they'd just close my account. I paid it, and closed my account, but I guess I could've just left if I wanted to.
Good to know. I'm paying anyway because I have other accounts there with bigger charges and I actually want the service, including the small personal account (which really shows as a separate transaction on a bank statement).
That's a good point, and you're spot-on about the Route53 zone. I did close my AWS account already. I'm hoping it just kind of doesn't renew that domain name.
Yes. But building an exception into the billing system and possibly accumulating charges below $x over a longer period might just not be worth the effort.
My concern (for non-paying or low-paying side projects) is not so much the difference between $5 and $10 for monthly hosting - that's like the difference between one or two cups of coffee - but what safeguards are in place to prevent me from inadvertently exceeding an expected amount due to a high traffic spike, misconfigured cloud function or whatever.
So I don't really care if one service costs x2 if the difference is between 5 and 10, but I care a lot if I could accidentally end up with a $2000 bill one day (and warnings don't count - not much good if I get a warning if I'm away from my laptop in a log cabin or something).
I guess that's the modus operandi of the Cloud providers - fake your concern by an useless warning and hope that no one will react before they'll rake up the bill.
That's why I stay away from e.g. AWS for side projects, too many histories of a stupid mistake that costed quite a bit of money when all that is needed is some breaker if usage is exceeded beyond a point.
We don't suddenly charge people. We charge per VM you have running. Setting these up is intentionally manual. You can opt in to autoscaling, which will create new VMs for you, but you have to set that up after you've deployed your app.
If you're getting charged you should run `fly apps list` and see if you have more than two apps running.
One thing that trips people up is Postgres. Postgres is just another fly app. So if you moved two apps with DBs, you're probably running four total apps. We prompt you for database info when you do this, though, it's not done in the background.
One problem we've come across is expectations from Heroku users. We are very different (like the Postgres thing, there's no shared tenant postgres). People expect us to be the same. We haven't figured out how to expose those differences to people in a way that sticks.
Perhaps because I'm not used to Heroku and such so I have different expectations, but I found it quite clear. There's a list as soon as you open the dashboard that looks like
> It might be “quite trivial”, but it’s still too much work for lazy developers, let alone non-technical people.
I don't think you have to be "lazy" to take advantage of the value proposition of saving time or complexity. I think Dropbox and similar are successful because they have put a higher emphasis on their target audiences time than other competitors in the past.
The author doesn't seem to explain the before and after cost difference. It used to be $9, and now it's what? Is the 10 cents talking about the AWS S3 pricing or the Fly.io pricing?
Sorry if that wasn't clear. This was meant to answer it:
> Fly.io looked more geeky and command line-oriented, which suited me, and their prices are also ridiculously low: free for up to three small virtual machines (I only need two), and $2/month for small VMs after that.
Their free allowance allows up to 3 small VMs and 3 GB of permanent volume space. I'm under that (2 VMs and 2 x 1GB volumes) so I get both my apps for free, and I could add one more. Even if I had to pay the regular rates they charge after the allowance, it'd be only $4.18 per month.
In any case, I was paying AWS $9/mo, now I'm paying Fly.io $0/mo, so that's what I saved.
Good for them. Looking superficially at the application they're talking about, it's also possible that a more cloudy implementation of said software would have lowered the costs also on AWS side. Like using lambda functions instead of EC2.
What stops me from using such platforms is the risk that one day for any reason my account can be suspended and I will have to migrate my apps and data.
Do you have a strategy for that situation? Your old Ansible branch hopefully is still there and compatible with your recent production code changes. And your sqlite DB is replicated to an external storage and your data is reachable even if you have lost control over “your” PaaS account.
> What stops me from using such platforms is the risk that one day for any reason my account can be suspended and I will have to migrate my apps and data.
You have found a hosting provider where this is not a risk?
If you're on AWS or GCP or Azure, and they terminate your account, and you use only the "standard" services (raw instances, k8s, database, redis, queues, email, etc.) it's generally relatively easy to migrate to one of the other two.
Fly is different enough that moving to or from it takes much more time and effort.
All of which have a counter-party who can suspend your service if they believe you're abusing the service: A&B your cloud / hosting provider, C&D the transit provider - all will have language in their contracts allowing for suspension or termination if they suspect abuse.
The only things that help protect you against this are (1) having named contacts on your account who will vouch internally at a supplier; (2) negotiating these clauses in contracts to prevent instant suspension in all but the most serious cases; and (3) utilising multiple suppliers.
This is also about the number of suppliers. Clouds - a dozen. Hosting providers, collocation - thousands.
I agree that in all cases, you depend on someone, but you can lower the risk to the point where other dangers that threaten your business become more probable.
One nice thing fly has going for it is that you’re just deploying standard OCI containers, so there’s pretty minimal lock-in if you need to get off in a hurry. Due to some edge cases I recently transitioned a workload from Fly to AWS in 2 hours or so.
Why should HTML files be embedded within the Go binary bloating its size ? And then needing to fiddle and hack around with Go's embedded FileServer to support basic HTTP caching is painful.
I assume there is. But that’s not the point I think.
The point of embedding is similar to making a jar from an ops perspective. You “bundle” everything into one thing and you deploy that thing as single binary/archive. Less moving parts, less complexity.
Secondly, providing etags/modified headers for serving static assets is not a hack. It’s just not built-in. But it’s perfectly fine to serve those directly from your app server with the appropriate headers, so proxies can serve them directly.
In fact I would ask the reverse: why use a CDN when you can use canonical features for deployment (Go embedd, Java jar etc) and serving (HTTP cache headers)?
Personally I wouldn’t necessarily use a 3rd party library for this but to each his own.
> why use a CDN when you can use canonical features for deployment (Go embedd, Java jar etc) and serving (HTTP cache headers)?
* different work profiles (streaming large amounts of compressed data vs small JSON requests/replies) means that different software is more optimal for each -- an application thread usually is a lot more powerful and does a lot more (and thus is slower in the general case) than an nginx thread
* you're using your own compute on your own dime to do what your service provider is probably offering to do for cheaper
* bloats the image size -- there is a productivity difference between working with a 150MB image and a 1GB one (debugging, easily diffing changes)
Admittedly, these aren't problems for a one man startup or a very low traffic website, but these things do eventually matter as your system grows in size.
As for the first two: I would assume that you’d put nginx or some other reverse proxy (or a network of proxies) in front of your application server anyway.
With the appropriate headers and configuration you would have the same kind of performance characteristics and resource utilization. Am I missing something here?
As for the last point: you have 1GB of stuff in either case. And all of that stuff needs to be looked at. The difference is really the deployment: do you prefer to move everything in a single piece or would you rather make individual, fine grained deployments? Either has advantages and disadvantages.
I’m not sure how Go let’s you inspect embedded files in a binary though. There might be gotchas. An archive is probably more transparent and more general tools can be used.
At some point the image size does start to matter. Both for local convenience and for the time of deployment.
Also serving only the current version's assets will cause failures during a restart. If you have more than one container, there's a chance the page request will hit the new one, but the resource request will hit the old version - without the needed asset.
And once you hash your assets, you normally only deploy a couple of extra files, not the whole 1gb. I don't get why people would want to keep them embedded (apart from convenience for a third party who will run it)
Fly actually has rudimentary static file hosting: you can use a [[statics]] block in fly.toml and any exact filename match (no index.html etc) will be served directly from the router. I'm not sure this would end up being useful for OP's exact usecase, i'm pretty sure it doesn't set etags.
You can get equivalent hardware for much cheaper elsewhere and save way more than 9$.
Add some load balancing across cheap hosts and you can achieve good reliability.
I run things on Digital Ocean, hetzner, ovh, linode, fly: they all cause downtime from time to time.
I have clients on azure / Aws / Google and it's a much rarer occurrence.
Fly is definitely among the unreliable ones, so you're paying for a simplified deployment / ops experience here - the heroku model.
Even at that, configuration is not the easiest and there are gotchas you need to know. Documentation is also pretty sparse and the example images were stale. Getting a build to work was a huge pain.
The free offering is nice, but I'm not recommending it for production.
I currently rent AMD 16 core server with 128GB RAM and 2x2TB NVME drives for around $100 / month from Hetzner. Price for equivalent hardware on Fly (without mentioning drives) is $1,126 / month.
Well yes of course. But then you have to manage everything yourself. The whole purpose of fly.io, heroku, raleway, etc, is that you don't have to care (much) about ops & sec. Handling it yourself (or hiring someone to do it) can cost way more than 1000$/m. When you grow big enough it might not be worth it anymore, but for smaller project, it can be really useful.
And for the record, at the moment, I host most of my service on my own server.
Not sure why giftyweddings.com needs to serve static files with an ETag in the HTTP response header: a hash of the file content is already included in the URL, which means it will invalidated when the content changes, just based on the path, regardless of the ETag.
Yeah, this is a good point. I've previously just used the "hash in the URL" approach as it seemed more obvious than ETag. I asked Ben Johnson why his hashfs library uses both, and he said "Yeah, that's a good point. I can't remember why I added both. I suppose some proxies may use ETag? I dunno."
The only problem I have with Fly is that it is VC backed. It's receiving a lot of developer love now, but wonder what happens when it eventually gets acquired by a behemoth in a few years and the same fate as Heroku is bestowed upon it.
Appreciate the honesty and transparency but sadly VC isn't chasing the profits from being a cloud- or app-server-provider. Looking at the profiles of the VCs which participated in funding rounds, most of them are exit-oriented. A16z's portfolio proudly opens with "exits". Intel Capital is also very proud of their ~50% IPO or acquisition stat. Hopefully you can work on Fly forever but history has learned that probably won't be the case - and that's not judgemental.
You'll notice that the ones they feature most are IPOs. Which isn't the same as getting acquired. I'm not opposed to an IPO, I am opposed to get subsumed by a larger company.
211 comments
[ 3.8 ms ] story [ 157 ms ] threadA couple of times per mo, redis (and occasionally postgres) times out.
My TLS cert did not renew, resulting in 8 hours of downtime.
You get what you pay for.
So many problems between me and even giving fly a serious try. Unfortunate, but maybe in a year it will be better.
The SSH and Builder issues are interesting. Those sound like something related to our wireguard stack.
With no way to evaluate the service, I opted to delete my account.
It's since been corrected, and I still have a handful of services running with them, but getting started with the platform definitely was a rougher experience than I was expecting it to be.
Thinking this is just because of a domain name is silly.
Also Fly.io -- You may want to clearly accept and manage trial issues (or at least a subset) via a support path. It seems silly to effectively bounce users with the impression that you have no support because they are not YET paying customers while in the trial phase.
Probably that was it, but my point is, there is no process you (as a customer) can follow to remove the fraud flag.
Also the flag was still there long after the preauth.
My server response time seemed to vary from <100ms to >300ms for the same operation.
I don't know whether this is a platform issue with Fly or something else, but it was kind of annoying to have such a high variation with seemingly no way to fix it.
That's just unacceptible imo.
What I think is happening is that the UI shows the status of the underlying container and there is some fault with the reverse proxy they are using to expose the container to the internet.
Fly.io doesn't have this issue but I am leery of all the complaints here.
The downtime you see is our proxy taking time to cutover- however, we have https://docs.railway.app/deploy/healthchecks that will only cutover once we have a 200 from your API. This way we can keep your old deploy live and serving requests if and only if it's live.
Theres more we can do to make it magical, but this should help in the meantime.
This could be done automatically by pinging the root and searching for a header set by railway's default page. If it doesn't exist then the service is live.
We are hiring Network Engineers for exactly this reason :')
This is only true for the free tier, and (hopefully) won't be true for much longer.
- It took several hours to get a 1 GB disk provisioned (their API would time out and I'd get left with an allocated but unusable disk that couldn't be deleted without deleting the entire app it was attached to.) The Fly forums have numerous reports of the same issue, going back months, across multiple data centers. All resolved with the moral equivalent of "oops, try again later?"
- The fly.toml reference is incomplete and internally inconsistent, documenting keys which are not valid, while omitting valid keys that appear elsewhere in the docs.
- Volume snapshots are retained for either 5 or 7 days, depending on which of the multiple conflicting docs pages you believe.
- I wanted to let them know about that last one, so I emailed the address listed at https://fly.io/docs/about/support/, only to get an autoresponse saying that they don't read that mailbox and that I should pound sand unless I'm already paying them to read my emails.
Reading between the lines of their recent blog posts, it seems like a lot of the infrastructure pains are pinned on Nomad, with the hope that moving to Fly Machines / Apps V2 will obviate a lot of those issues.
I hope they're right, because what they're trying to deliver is exactly what I want.
Volume provisioning issues have been a game of whackamole, largely as the result of scaling pains. We've grown like 4x in the last 3 months (thanks, Heroku) and it's put a strain on pretty much everything.
3 separate issues prevented me deploying. One was my fault, but hard to diagnose because error messages contain no useful information.
The other two were the platform falling over, with the support forum having no resolution but plenty of other people in the same boat.
Lots to love about fly. The latency is noticeably better than anything I’ve used except cloudflare. Prices are good. Hopefully they’ll get the growing pains sorted and we’ll have another contender.
If you end up trying again, give the machines based apps a shot. It's much simpler infrastructure: https://community.fly.io/t/fly-apps-on-machine-prerelease/10...
Redis/Postgres timeouts sound like they might be a problem we can help with, though. Both those services work through a load balancer with a TCP idle timeout. If you get timeouts after a period of inactivity, it's usually a simple config tweak to have a driver handle that seamlessly.
The timeout issues surprised me, more drivers than we expected can't handle a DB connection timing out when it's idle.
I think they are too engineering culture. Target is devs, sure, but they could use some product guys.
it was very frustrating to see un-answered problems reported, fly.io down for what seemed to be many users, the same time seeing a fly.io co-founder arguing about politics all day long here on HN...
Note that my load test wasn't to see how much I could squeeze out of this, but to determine that it's "more than good enough" for this use case. The typical qps I get from real customers on the site is about 0.5 qps, so I think I'm good. :-)
This seems like a trivial thing to add. If I wanted to use a domain specific configuration file then that's always an option, later.
Fly devs have confirmed this is in the works, think macaroons.
The only defense here is that Fly isn't the only party being incredibly sloppy with their API keys. Cloudflare is just as bad.
Doesn't that force you to use your hosting provider as your CI provider?
I've always avoided the automatic "deploy on git push" solutions because I want to keep my CI provider uncoupled from my hosting provider. That way, I can switch hosts without having to rewrite all my CI code.
About 1/3 of my deployments are just static sites, and the rest all fit into a single Docker container, so integrating Compose would be a big increase in complexity.
That being said, you can use Watchtower without Compose. It's enough to run
Worrying about having to configure systemd? "restart: always" in compose. Caddy updates? Caddy provides an up to date docker image. You can't get around caddy's config, but tbh, you do it once, and then you're done. It's way better than nginx. Deploying a new version of your go application is just "docker compose up -d".
Just wondering about having a "native" NGINX reverse-proxy + multiple apps ran via docker-compose on my VPN, seems the most hassle free. Currently using it for integration tests and it works great.
Did you maybe consider k8s, maybe as a learning experience?
It's probably all in line with the ToS and everything, but I find it a bit shady that I have to investigate this, rather than having a free offering that doesn't suddenly charge me.
As a company with a cushion, hosting a profitable app, it might be worth doing the usage-based pricing to try and save some money. But the risk is just not worth it for smaller projects
- Up to 3 shared-cpu-1x 256mb VMs
- 3GB persistent volume storage (total)
- 160GB outbound data transfer
If you spin up, say, 4 VMs you will pay for it.
No thanks Oracle even for 24GB free VMs!
But that is not all
My account is now "inactive", with no option to activate it and with no option to even pay and upgrade to paid tier.
The account is in some broken state where half of the dialogues produce dialogues like "Authorization failed or requested resource not found", like I had account with permissions denied to everything.
Even the "upgrade to paid tier" option gives
> You do not have access to perform this operation. Please contact your cloud account administrator for additional access.
Utter fucking disgrace
You may want to check if you also have a domain name purcahed through route53 that is set for automatic renewal every 3 years or whatever
So I don't really care if one service costs x2 if the difference is between 5 and 10, but I care a lot if I could accidentally end up with a $2000 bill one day (and warnings don't count - not much good if I get a warning if I'm away from my laptop in a log cabin or something).
That's why I stay away from e.g. AWS for side projects, too many histories of a stupid mistake that costed quite a bit of money when all that is needed is some breaker if usage is exceeded beyond a point.
If you're getting charged you should run `fly apps list` and see if you have more than two apps running.
One thing that trips people up is Postgres. Postgres is just another fly app. So if you moved two apps with DBs, you're probably running four total apps. We prompt you for database info when you do this, though, it's not done in the background.
One problem we've come across is expectations from Heroku users. We are very different (like the Postgres thing, there's no shared tenant postgres). People expect us to be the same. We haven't figured out how to expose those differences to people in a way that sticks.
Sounds like there may be room for improvement in your documentation.
Second paragraph
I don't think you have to be "lazy" to take advantage of the value proposition of saving time or complexity. I think Dropbox and similar are successful because they have put a higher emphasis on their target audiences time than other competitors in the past.
> Fly.io looked more geeky and command line-oriented, which suited me, and their prices are also ridiculously low: free for up to three small virtual machines (I only need two), and $2/month for small VMs after that.
Their free allowance allows up to 3 small VMs and 3 GB of permanent volume space. I'm under that (2 VMs and 2 x 1GB volumes) so I get both my apps for free, and I could add one more. Even if I had to pay the regular rates they charge after the allowance, it'd be only $4.18 per month.
In any case, I was paying AWS $9/mo, now I'm paying Fly.io $0/mo, so that's what I saved.
for low, intermittent traffic sites, go on lambda might be a better comparison:
https://github.com/nathants/libaws/tree/master/examples/simp...
Edit: apparently you can select the region, and you can possibly get a pre made DPA document template.
Customers with HIPAA compliance needs may need an Business Associate Agreement (BAA). That BBA thing is only included in paid plans.
Do you have a strategy for that situation? Your old Ansible branch hopefully is still there and compatible with your recent production code changes. And your sqlite DB is replicated to an external storage and your data is reachable even if you have lost control over “your” PaaS account.
You have found a hosting provider where this is not a risk?
Fly is different enough that moving to or from it takes much more time and effort.
Your level of abstraction may vary. I stick to VMs after realising how much overhead I had with the cloud as a one-person shop.
Teams can interface with k8s which is also standard nowadays, taking you can spawn a cluster fast.
It is? Fly seems to be only docker containers. You can't get much more standard than that these days!
Have you actually used fly? It's pretty vanilla. Having migrated to and from fly I don't think your assessment is accurate.
It uses Docker files or Heroku style procfiles - not particularly exotic. Your apps live in a private network, which is managed for you.
a) Cloud providers; b) Hosting providers; c) Collocation; d) On-prem;
The only things that help protect you against this are (1) having named contacts on your account who will vouch internally at a supplier; (2) negotiating these clauses in contracts to prevent instant suspension in all but the most serious cases; and (3) utilising multiple suppliers.
I agree that in all cases, you depend on someone, but you can lower the risk to the point where other dangers that threaten your business become more probable.
Is there no facility for a CDN in fly.io ?
The point of embedding is similar to making a jar from an ops perspective. You “bundle” everything into one thing and you deploy that thing as single binary/archive. Less moving parts, less complexity.
Secondly, providing etags/modified headers for serving static assets is not a hack. It’s just not built-in. But it’s perfectly fine to serve those directly from your app server with the appropriate headers, so proxies can serve them directly.
In fact I would ask the reverse: why use a CDN when you can use canonical features for deployment (Go embedd, Java jar etc) and serving (HTTP cache headers)?
Personally I wouldn’t necessarily use a 3rd party library for this but to each his own.
* different work profiles (streaming large amounts of compressed data vs small JSON requests/replies) means that different software is more optimal for each -- an application thread usually is a lot more powerful and does a lot more (and thus is slower in the general case) than an nginx thread
* you're using your own compute on your own dime to do what your service provider is probably offering to do for cheaper
* bloats the image size -- there is a productivity difference between working with a 150MB image and a 1GB one (debugging, easily diffing changes)
Admittedly, these aren't problems for a one man startup or a very low traffic website, but these things do eventually matter as your system grows in size.
With the appropriate headers and configuration you would have the same kind of performance characteristics and resource utilization. Am I missing something here?
As for the last point: you have 1GB of stuff in either case. And all of that stuff needs to be looked at. The difference is really the deployment: do you prefer to move everything in a single piece or would you rather make individual, fine grained deployments? Either has advantages and disadvantages.
I’m not sure how Go let’s you inspect embedded files in a binary though. There might be gotchas. An archive is probably more transparent and more general tools can be used.
Also serving only the current version's assets will cause failures during a restart. If you have more than one container, there's a chance the page request will hit the new one, but the resource request will hit the old version - without the needed asset.
And once you hash your assets, you normally only deploy a couple of extra files, not the whole 1gb. I don't get why people would want to keep them embedded (apart from convenience for a third party who will run it)
[0]: https://news.ycombinator.com/item?id=34953567
https://fly.io/docs/reference/configuration/#the-statics-sec...
You can get equivalent hardware for much cheaper elsewhere and save way more than 9$. Add some load balancing across cheap hosts and you can achieve good reliability.
I run things on Digital Ocean, hetzner, ovh, linode, fly: they all cause downtime from time to time. I have clients on azure / Aws / Google and it's a much rarer occurrence.
Fly is definitely among the unreliable ones, so you're paying for a simplified deployment / ops experience here - the heroku model.
Even at that, configuration is not the easiest and there are gotchas you need to know. Documentation is also pretty sparse and the example images were stale. Getting a build to work was a huge pain.
The free offering is nice, but I'm not recommending it for production.
Not sure why giftyweddings.com needs to serve static files with an ETag in the HTTP response header: a hash of the file content is already included in the URL, which means it will invalidated when the content changes, just based on the path, regardless of the ETag.
But I also don't think you can build a from-scratch cloud without VC money. It's very expensive.
Linode did it 20 years ago (somehow). Dunno if there's a more recent example though. :)