“ The result of this is an Internet where all of the world's CRUD apps are hosted in Loudoun County, VA (motto: "where tradition meets innovation"), at Amazon's us-east-1 in Ashburn, a city with so many Rails apps that one of them was elected to the county Board of Supervisors.”
In Florida you have so many hispanic citizens that they have significant representation in local politics. In this specific county you have more cloud servers serving rails apps. However, they're not citizens and can't get elected to public service.
Honestly the question felt very Llama in the sense that it didn't really pass my Turing test, so I couldn't really answer it in a way that didn't feel robotic.
Maybe also related to the recent news of a small town in Delaware allowing corporations to vote in municipal elections (not a new thing for Delaware apparently).
Oh man, early Slack was great. Back in ~2015 or so I submitted a ticket about something trivial (why I could no longer hide the menubar or something, I forget), but just for fun I did it all in stilted archaic language like you'd see in the 1600s or something. "Behold, in days of yore when the Users of the Land sought refuge from the tiresome typing that had so bedeviled their lives hitherto..." etc. Anyway, their CS team responded in kind. Hands-down the most fun I've ever had with a support ticket.
I've thought a lot about this too. Their blog posts are consistently interesting and well written. I wonder how much of their success is related to the product marketing as well?
We're not. For a company of our age and scale? To claim competitive reliability to the world's most important data center? Risible. We're making fun of the fact that everybody deploys their apps in a random city in Virginia with like 3 restaurants in it. We're not saying they're dumb to do it, or that Ashburn isn't up to the challenge --- it self-evidently is.
(The comment I'm responding to said something about people in glass houses not throwing stones, but was then edited).
You guys may want to rebrand once you get your issues sorted out. Fly is going to have the same problem as btrfs, where folks will have a hard time deciding if it's ready for production yet.
It's comments like this that moved me to claim, in our infra/ops job post, the benefit "The most lovable user base in the whole technology industry, and a company culture that will keep you in touch with them directly."
Fly.io was the darling of the webhosting world amongst techworkers particularly because of the honest transparent communication and blog posts explicitly admitting fuck ups
Raising a bunch of money just isn't a "good" announcement to most of us. When unpopular things have happened with the company, your communication usually directly acknowledges and focuses on the concerns of the userbase. This serves two purposes:
1. It gives you a chance to show concrete steps to prevent specific concerns we have
2. Perhaps more importantly, it shows we're all on the same page and you're actually in touch with how your userbase thinks
Failing on #1 is something you can fix later, but I've never seen a company rebound from losing the trust gained by #2. It loses confidence that future decisions made can also be equally out of touch.
I've never seen a company rebound from losing that confidence
If we're "honest" and "transparent", it's not a conscious strategy to sign up customers. It's just the easiest way for us to write. Meanwhile: we're a hardware-based global hosting company. We've always been externally funded. That's how you build a global hardware-backed hosting company, because servers and colocation and infra engineers are expensive.
There are things that happen that disappoint customers that we care a lot about. Reliability issues several months ago absolutely fit that bill. So does hiring: we paused hiring for most of our engineering roles because we weren't happy with the experience candidates were getting. I don't know how "honest" and "transparent" we are about those things, but I do feel like when we're asked about them, we give straight answers.
There are other things that happen that disappoint people that we don't care so much about. Being unhappy that we took external funding to build out our fleet and improve reliability would count as one of those things. Again with the straight answers: I wouldn't expect us to spend too much time trying to "rebound" from this announcement.
>Raising a bunch of money just isn't a "good" announcement to most of us
Speak for yourself. Announcements of adequate funding inspires confidence fly will have longevity and resources to scale. Other concerns can be tackled in other blog posts, but nothing about the announcement of funding should inspire less confidence, what sense does that make?
The issue is not raising money, precisely, the issue is raising money by selling large amounts of controlling equity (as is usually the case for VC funding and IPOs, with a few notable exceptions). The purchasers will expect their money’s worth, and they will (eventually) not hesitate to exercise the control it bought them in order to get that.
VCs in particular will expect much more than their money’s worth, or (most of the time) for the company to die trying to earn it, and they won’t hesitate to force the latter in pursuit of the former. This increase in variance is literally their investment model.
So if VCs expect their investments to fail most of the time, and I see a company take VC money, I must have very strong reasons if I’m to disbelieve the VCs and expect the company to succeed—and I’m not following a venture investment strategy when I choose a tech supplier.
IPOs are somewhat milder in their intrinsic character, but they still usually lead to professional boards of directors and eventually professional execs, where “professional” usually means “no subject matter expertise”. And hardly anybody runs a public company on customer satisfaction nowadays, unfortunately.
So when I see controlling equity sold, my outlook changes from “will this go to shit?” to “when this inevitably goes to shit, will it be in the distant enough future that I don’t care or can plan for the loss?” (that being maybe a decade or so for infra, or implausibly long—like the rest of my natural life—for anything involving a community).
As usual, the question remains—why does none of this seem to be priced in? I don’t know.
This is neither here nor there, but every time I read a comment like this, all I can think about is that scene in The Office where Dwight Schrute is screaming into his cell phone "Are you saying you invented paper?!"
Yeah this is pretty funny - if you speak to PR before posting something, it's 'PR-riddled bull** written by some executive', or 'corporate talk'. If you don't speak to PR before posting something, it's 'Did you speak to PR before posting this?' There's no winning.
the whole point is deploying closer to your users, unless your customer base is the owners of those restaurants, it would be better to deploy to other (even all!) regions
To be fair (this comment is mostly a joke), Ashburn/Loudoun County is pretty nice for a suburb/strip mall hell. I've had some really good food in that area.
Ashburn does not have "like three restaurants". The datacenter zoned part of the city has "like three restaurants". If you ever drove >5 minutes from those large shed warehouses, you'd know what you're talking about.
You are probably thinking of Sterling, Chantilly, etc.
Ashburn is literally like the Embassy Suites next to the Five Guys, the Aloft next to the taco place, Equinix, Digital Reality, Verizon, NTT, CenturyLink, etc.
So you fuss at your users who have valid complaints about the availability of your product? It's a valid complaint when the uptime of a service, program, or feature running is critical to your business.
I'm really concerned that Fly's employees like yourself think availability and service uptime isn't important. I think I'll be avoiding Fly on this alone.
Honestly though, Fly.io feels like it's off into the future in terms of hosting platforms. Undoubtedly they'll have to work on the reliability problems, not denying that - however, for anyone that can tolerate a bit of reliability issues, which to be honest, is more people than will readily admit it, it might just be worth the trouble for developer experience that legitimately feels like it's from the future.
It's definitely not unusably unstable, at least from my PoV, and in fact, I haven't really been hit by most of the instability personally; or at least, if I have, it has gone mostly unnoticed by me. I had an issue where I couldn't deploy for a while, but the services that were up seemed fine. (Disclaimer: I do not use the Redis/database features.)
That said, the point I'm trying to make is that it is in fact a lot nicer to use than simply YOLOing whatever onto servers. This model of PaaS is basically what App Engine should have been. It's difficult to want to go back once you've used it.
I've worked on the reliability team at a place that made what they call an "edgy app" (long before fly, so we made our own stuff on baremetal) and to be fair, reliability is the actual challenge when you're talking multi-region seamless deployments of scalable web platforms. It makes or breaks that kind of product at least for the original market.
Money doesn't contribute all that much to solving it though. It's usually time to make the early mistakes, continuous improvements, etc. But AWS had several decades to solve this, whereas fly clearly hasn't, so I'm not sure I'd count them out yet.
People forget AWS is 20 years old. When I routinely remind people of that fact they often remark (incredulously) with "WHAT? Really?" When I think about it the entire paradigm of "regions" itself seems completely antiquated in the grand scheme of things. AWS has made a few moves on this but in the end you're largely still tied to this fundamental regional concept in terms of control planes, etc and you incur the ridiculousness that is VPCs, wild billing and availability issues, all kinds of hacks, stacks, and effort all over the place to have multiple regions/AZs, etc.
I suspect that later starters like Fly, Cloudflare, etc with their approaches of (more or less) compute at edge with no concept of region etc will grow more and more to dominate the cloud space with AWS region-based approaches primarily relegated to "legacy applications" that are stuck there. It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends.
When I think about it the region based approaches are more akin to "hosted elastic datacenter" than they are what could be described as a "true cloud" - where it's just everywhere and not even a consideration.
The realities of regions don’t go away. It’s a primitive to save you money. If a new entrant doesn’t have it they are either just operating in one region or you’re paying cross-region prices for everything.
Cloudflare products and pricing (what I'm most familiar with) are in wild opposition to this view.
I've never seen anything even remotely hinting at region or geography in their product line other than geo-routing for load balancing products, headers with geo info for you to do something with, etc. They include the serving "POP" in headers for diagnostic purposes but other than that you have no idea.
Where do my Workers run? Don't know, don't care. They are substantially cheaper and offer better availability and response times than their region-based "cloud" competitors. Same for KV, D1, R2, and anything else they come up with as they move further and further in (out?).
One would expect with them having the ability to allocate supporting hardware dynamically and globally their cost basis is substantially better than having customer facing and controlled region-based resources that still need to be built out for product support (regardless of usage) AND maintain the excess capacity local to each region to be anywhere near "elastic".
It's important to qualify this though. E.g. Amazon was not profitable for a very long time, but only because they furiously reinvested everything. They could've turned profitable at any time.
Assuming Cloudflare is a similar case to Amazon: ie they could turn profitable at any time but choose to reinvest those profits immediately into more business, that is a very good spot to be in.
It really doesn't though. Sure, it would if the difference is only "16 mega-regions (regional-model) versus 300 mini-regions (edge)". But that's not the difference.
The actual difference, at least how Cloudflare pushes it, is in the software; its a higher layer of abstraction; that customers get to stop thinking about regions and locations. Your software is global; its deployed everywhere; it runs everywhere.
This is ENTIRELY antithetical to everything on AWS except CloudFront, Route53, IAM(ish), maybe a couple other minor services. Every Single Thing Every Single AWS Customer configures beyond those global services forces you into a region. Want to deploy a lambda function? Which region would you like? Pricing and available products is different in every region. Fronting it with an API? Where would you like that API Gateway? We'll put CloudFront in front of it so its a bit faster for ya, its global(ish), don't worry.
In a no-true-scottsman true-edge world, this goes out the window. Cloudflare Workers is like this; D1 is kinda-ish-there-ish; R2 isn't. Here's the code; spider it across the world; automatically route clients to your closest PoP. No regions. No locations.
Its not obvious that AWS will ever make meaningful moves to a world like this, for a ton of reasons, but most meaningfully: it requires customer demand; but AWS's existing customers aren't demanding it, because AWS's customers are old CTOs worried about past quarters too much to think about the next one. The younger, smarter, potential customers who recognize how valuable this is aren't AWS's customers; so AWS never hears from them (which, by the way, should more generally concern Amazon's investors; their "relentless customer driven" approach to their products is great for a few decades, but its absolutely true that most of the time customers just want a faster horse, and you're not hearing from the people who aren't your customers because you've never solved their problems before). So, Amazon has dumped billions upon billions into their couple-dozen giga-regions; you cannot pivot that into hundreds upon hundreds of mini-regions like what Cloudflare (and to a lesser but still awesome degree Fly) has.
Vercel is the best case study on this. They are exploding in popularity. They don't run any of their own infra afaik, to be clear, but they whitelabel products from Cloudflare, Neon, and others; all Edge infra providers. No one on Vercel is cross-shopping AWS, and AWS has nothing that feels competitive to what Vercel offers. They can't compete.
The outcome, in maybe 10-20 years, of this landscape feels really obvious right now, but AWS's size and the insane scale of their investment into "how we did things in 2010" will turn them into the next IBM. Some would say they already are. There's nothing they can do about it. They're not making the right investments. Even if they wanted to, they can't at the necessary scale, because what they're doing is working right now and their existing customers keep forcing them to spend more money building more structures, more racks, more blades, all in Ashburn VA.
Stateless compute is relatively simple to be region-less, but even fly’s homepage has a map of regions.
When state is introduced, then CAP/PACELC distributed systems issues arise, and figuring out approaches to deal with them are a must. Read fly’s Postgres docs, and regions come right back in.
Only the most trivial business systems can operate without any state.
If your application has any sort of shared, mutable state, you absolutely need to pick a region, or go really deep into the EC/consensus/clustering rabbit hole. With a dedicated region with 1 big DB, you can achieve latency figures (over the shared state) that would be infeasible with other schemes. Serializing transactions in a distributed manner is a circus compared to what Postgres has to do to achieve the same.
For better or worse, sticking a big SQL database in exactly one region will almost always be the best path for most forms of business. If latency/edge are still a concern at this point, that's when we maybe start talking about more specific geographies and standing up read replicas in those regions. In my experience, if you are peddling B2B webapps, no one will ever complain about east vs west coast latency unless you actually screwed something up in code.
This is Fly - a much earlier stage company and them receiving early funding is the genesis of this entire discussion.
I point to Cloudflare (what I'm most familiar with). Across their entire stack of compute, storage, db, etc product lines you won't see a hint of region anywhere. It just works, everywhere. Same with Fastly/Oracle (although they like Fly are quite a ways behind Cloudflare).
I suspect (hope) that as Fly matures their apparent true goal of edge and region-less applications will mature more. Their compromises on region awareness for DB has always struck me as a more of a resource restraint compromise on their part than end goal. As you say these challenges are hard (expensive).
If I were at Cloudflare I'd suggest calling these "geographies" where the magic still applies (any one of those geographies has at least dozens of component POPs that are invisible).
The fact remains this is somewhat down in the docs because it is automatic and invisible to the point it works well enough where most developers don't know, don't care, and aren't exposed to it in any meaningful way. Contrast this with AWS where the first thing you are presented with is picking a region where everything lives, while not clearly and explicitly explaining that many things are ultimately dependent on us-east. Cue the shock and confusion when us-east has yet another issue and those in all other regions don't understand how and why they are impacted.
You are correct about the free lunch, information theory, and physics at the end. I don't think a reasonable person can think when you write data to a pop in Tokyo that it could possibly show up in Chicago instantly. That is fundamentally impossible and always will be. I think CF does a pretty good job making it "magic" for most devs and use cases while explaining these fundamental realities and limitations to those who need to know (potentially) for whatever their application is.
"Won't see a hint of region anywhere" is an exaggeration.
Here's the Durable Objects documentation [1]. Don't those "location hints" look like regions?
Also, here's the "Data location" page for Cloudflare D1 [2]. See "available hints" at the bottom.
I think it would be more accurate to say that Cloudflare's architecture is designed around having lots of regions, automatic migrations between them, and no guarantee about which one you'll get. But they're still there.
These are offered more as an escape hatch in case the system doesn't do what you want automatically. But the goal is definitely for the system to do what you want automatically, so that no one ever has to provide these hints.
I actually argued against adding these hints at all, since it confuses the messaging and most people don't need them... but they were easy to add and solved an immediate problem for a few customers, so we did.
In other cases we've held firm. For example, some people have wanted to restrict their Workers to run in specific regions because the Worker made lots of requests to a specific back-end in that region. Instead of actually offering explicit hints, we built a system to detect this automatically: https://blog.cloudflare.com/announcing-workers-smart-placeme...
In any case, most people don't have to think about regions on Workers, and for the few that do, our goal is to change that.
Thank you for the context and this is what I was getting at - if you really want to dig in on things like KV, D1 and DO you will see occasional mentions of regions (I'd really call it geography) to add context on things like "KV is eventually consistent, instant in the pop where the write takes place but takes X ms to propagate everywhere". For many (most?) applications it doesn't really matter, I get the sense for these you clearly document these things so people understand the more-or-less inescapable fundamental reasons why things are what they are. As other commenters have noted "there is no free lunch" when it comes to things like the physics of getting data from a pop in Tokyo to Chicago but Cloudflare does a good job and making it happen more-or-less invisible.
> In other cases we've held firm. For example, some people have wanted to restrict their Workers to run in specific regions because the Worker made lots of requests to a specific back-end in that region. Instead of actually offering explicit hints, we built a system to detect this automatically: https://blog.cloudflare.com/announcing-workers-smart-placeme...
That is an excellent way to design a product. Great job.
THIS. No one cares about regions, you just care that the app is fast and globally available. The way we did things is a relic of the past and everyone mimics it because trying to guide new user behaviour is hard. The reality is we just need the app to be globally deployed with a globally distributed database and anycast DNS to route you to the local most DC. There's a lot more to it, have built this architecture more than a couple times, but it all needs to be in the service of something and if you're UX isn't better than the next guy, it won't matter much.
As a counterpoint, anyone working with compliance data needs to know and be certain that data does not cross geo-boundaries. Intelligence data for US gov’t agencies must typically be kept stateside: AWS uses the GovCloud region specifically for satisfying gov compliance requirements.
Passing HIPAA data through other countries when not needed to is generally frowned upon - data should be kept as local as reasonably possible.
I haven’t worked with GDPR requirements but I have heard they are similar.
AWS solved this problem well with regions. I’m curious what other providers that are “regionless” do to solve the compliance problem.
Cloudflare does this although to my knowledge it's not available without an Enterprise subscription which is the only thing they offer with this level of compliance anyway.
All lower plans maybe get PCI DSS and that's it.
It's a valid counter to my point - while they clearly can do it locking it behind an enterprise "talk to us" subscription is lame. That said they are generally customer, market, and product savvy and I suspect they know that any/most orgs that are able to pull off the broader process, etc for actual compliance with these requirements are in fundamentally different positions than some random startup with a bunch of consumer data.
We're also seeing more and more ambiguity with coaching platform startups, etc who don't have a covered entity in sight but that's a different topic for a different day.
I've spent most of my career in HIPAA-relevant startups. Generally, I think it's actually a disservice all around for Amazon to basically rubber stamp these architectures and solutions with a HIPAA BAA so that companies think they can call themselves "HIPAA compliant". That's not even what it's supposed to mean, let alone the virtually endless issues and process that need to happen throughout the org for "compliant" to actually mean anything. I haven't reviewed it but I'm certain the AWS BAA is so specific and exclusionary it's trivial to step outside of what it covers.
Yes these agreements are a piece of the puzzle but it's completely reckless to throw something together in AWS, present the BAA to a customer, and call yourselves "HIPAA compliant". Savvy customers, investors, etc will call you out but otherwise it's mostly just a matter of time before you discover your standalone AWS BAA is actually just one of the things on a 50 point (or whatever) checklist.
> Amazon to basically rubber stamp these architectures and solutions with a HIPAA BAA so that companies think they can call themselves "HIPAA compliant".
N-2 companies ago I was a tech lead in 2015 where we were migrating to AWS. AWS was constantly explaining which underlying services were HIPAA compliant and what you build on top of it was your responsibility. I had to answer to separate non AWS compliance folks about my architecture.
AWS also gave guidance. But made no promises that your implementation was compliant. That’s where the entire “shared responsibility model” comes in.
Now, I work at AWS in ProServe. I am very familiar with our messaging regarding HIPAA compliance. We are very careful not to rubber stamp things as compliant. I know the best practices in and out. But when I’m asked if something is compliant, I say this is the guidance we are given. I’ll do a presentation before an Architecture Review board. But I would never make anything that hints as a guarantee.
How large were your employers? If you have Enterprise support, you definitely get one or more TAMs dedicated to helping your company.
If you are a smaller organization, you may be able to request an SA.
My personal experience is from when companies pay for consulting hours from ProServe.
At the n-2 company I referred to above, the only thing we got from AWS is a BAA and we had an outside consulting company and compliance auditors. I was just learning AWS then.
Compliance is a completely different conversation.
In the Cloudflare Workers platform, we have a number of features to control "jurisdictions" where compute runs or where data is stored. These are meant for compliance, not for performance. Our goal is that no one has to think about regions for performance reasons, but certainly compliance issues are not going to go away -- in fact, they are getting more onerous over time. So features to control jurisdiction are likely to expand.
This is MAY be true consumer applications and even then not all.
But it is rarely true for enterprise customers.
They want the guarantees their data never leaves certain regions even in a request along with other security and compliance guarantees.
The question of whether "the edge" takes off will come down to whether or not the culture at large will swallow the illusion at its heart. You cannot remove the concept of a region any more than you can remove the concept of "the computer" in a cloud environment.
> any more than you can remove the concept of "the computer" in a cloud environment
But that's just not true. Tons of cloud services completely abstract the computer out of the picture. You are paying for capacity or throughput as an abstract unit of cost and the cloud configures as many computers as needed to run your request. You never interact with anything resembling a computer in this situation.
Except, you always interact with computerS no matter what provider or architecture you use. It's inescapable as long as you're doing business in software. The trade-offs in architecture and regionality do NOT go away just because a company abstracts over it. They may be able to set up read replicas and consensus on the fly and be able to detect when various approaches make the most sense, but you are still at the mercy of the accompanying constraints. For most use cases, none of this may matter. When you do run into a use case where it matters, you should be fully aware of what trade-offs you're making and have control over them.
Not to mention, regions matter when you're serious. GDPR/DSA/DMA/India/China -- the list goes on. Certain data must live in certain bubbles.
> It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends.
I'm currently at a startup that started on Aptible a few years ago because there was a lack of expertise in systems development. We're now in the process of migrating to AWS natively because of cost savings (it's why I was hired). Aptible is costing us 3-4x what native AWS will. Yes, we're incurring some operational overhead, but the automation and tooling for managing infra is reasonably mature (Terragrunt & Terraform aren't perfect, but they work well enough).
Ultimately, startups may start with higher-level cloud providers, but they'll eventually migrate to lower-level providers eventually. At the higher end of the growth curve, companies likely wind up getting back to on-prem in some way.
AWS does have local zones and has Lambda at Edge. I will be the first to admit though that the local zones are tied to regional zones and that some global services are tied to us-east-1 like IAM.
These days, "edge" just means "has more than N data centers" (where N depends on the speaker). Fly and Cloudflare have regions too, try `flyctl platform regions` or see https://www.cloudflare.com/network/
No. That’s wrong. We don’t “manage regions”. 300 data centers. Everything running everything. Only time you’d need to think of a region is if you have some compliance situation (“keep all my data in North America” or “only terminate TLS in the EU”) and ask to subdivide our network using Regional Services. But there’s no decision on here you say “I’ll use Cloudflare-EU-North-42” and get stuck with it as no such things exist.
While "where tradition meets innovation" is a set of words that the county puts on their logo, and appears to meet the definition of a motto, the parent is correct that the Official Motto is indeed "I Byde My Time".
Having grown up on Heroku, it's such a breath of fresh air to see companies innovating in this space. Render, Fly are the big ones, but even little one-man shops like Hatchbox are good enough to deploy medium-scale production apps.
Kudos to these companies, the VCs, and the beta testers.
We have a multi-region single-write-leader Postgres offering now, but "databases" means lots of things to lots of people, and the platform strategy is to build durable storage primitives that work for as many different databases as possible. People run things like Cockroach here, and edge deployment also makes SQLite especially interesting.
Any plans to port fly_postgres_elixir to sqlite to enable using this feature easily with with Elixir?
I think your company building out developer tools to help use your platform would aid greatly in adoption...seems a necessity really, particularly addressing the pain-point of accessing data close to users.
a bit off topic, but a big struggle that is growing related to data and databases; managing data residency. I've not found an easy way to handle this and countries seem to be passing more and more laws... it's tough for global applications!
I think fly.io believes that they can allow the bookstore to do so in less than two hours for a price where it's still important enough for the bookstore and profitable - in aggregate - for fly.io.
I personally believe that could be true, but their investors going to find out ;)
I've long suspected this is what you guys would end up focusing on (and applaud it) -almost like an OS for edge compute that abstracts away its details while allowing boring architectures built for single-point to 'just run' on it.
But is there a limit to how boring you can go? Can you truly, fully abstract over distributed systems stuff like eventual consistency and its basically random effects on UX in certain usage edge cases?
Or do you think 'boring' will have to meet you in the middle with frameworks and developer mindset also shifting over time to a more edge-first world?
Self-aware commentary like this only works if used sparingly, but this one is extremely effective - acknowledges the mercenary motivations, implicit hypocrisy, and self-aggrandization of such announcements while nonetheless communicating all the important information and succinctly breaking down a real problem that their platform solves. On top of that, the legitimate uses of funding are clearly identified, answering any questions about both "why" and "how much".
I loved it. They stayed as far away as possible for your traditional grandiose, jargon-filled, corporate speak we normally get in these announcenments. They definitely gained a few points with me :).
so I'm guessing Vercel and Netlify deploy cloud functions only to the "least worst datacenter"?
while Fly.io deploys compute instances to more regions and also has a database?
asking because I've been content with Vercel and Netlify's CDN for frontend assets and simply stopped doing system design around relational storage, when I want to stay on a free tier.
can you elaborate on a comparison to existing offerings?
Heroku has been handicapped for nearly half a decade, there are a lot of people that already know of alternatives and have changed their system design to accommodate the alternatives. How does Fly.io fit in with Vercel, or Netlify? As opposed to people keeping their Heroku apps on life support
Vercel (for instance) does pure serverless --- you configure "functions" that are invoked by their orchestrator --- that are all Javascript, or compiled to JS/WASM.
Fly.io takes Docker containers, turns them into VMs, and lets you decide where and when to have them running.
One isn't better than the other; they're just totally different models for hosting applications. The Fly.io model is much closer to what Heroku did, just modernized (with Docker, edge deployments, the `fly-proxy`, etc).
No, with Vercel at least you can choose other “primary” data centers than northern Virginia. I haven’t used Netlify for a few years so I can’t speak to that.
This is the biggest draw for me. I don't have a need for Fly at the moment, like I have not needed Heroku most of the time, but its got strong mindshare for me. Heroku was basically great, and apart from not quite keeping up with modern practices and having recent outages, it's still otherwise a great UX. Fly feels like that but updated for the modern age.
I'm looking to deploy a $5/m DigitalOcean server (single binary) to run a simple HTTP[s] server, with persistent sqlite/fs and backed up via braindead rsync to my local system.
Is fly competitive with this? These sorts of calculations are always awkward for me with PaaS providers. I know what i'm getting with $5/m with DO, but with PaaS it often feels abstracted and sneaky. Heroku in my very, very old memory was a continual set of asterisks.. but i'm probably being unfair and PTSD'y.
Think i should give Fly a look? I at least find their SQLite features super interesting
A 1GB Fly machine is ~$5.70/mo (we charge by the second, some months are better than others).
Persistent volumes are $0.15/gb each.
Not too far off, depending on how much data you have. However the platform has different tradeoffs than a single DigitalOcean VM. You'll get better uptime percentages on a single DO VM than you will on a single VM Fly app. You may not notice, but it's statistically true.
Does Fly handle price caps? Ie to setup pricing to replicate ~$5/m behavior. Helping to ensure i can't use up all my ~$5/m budget in a few days of traffic, but i also can't go over some specified cap.
If so i'll probably give it a try!
edit: Regardless, going to give it a try. I was totally set on just using DO because i wanted something dumb and simple. This sounds like it might offer me that, plus not having to manage the OS. I really, really like that if i can keep pricing similar.
Bandwidth costs might become a concern, but i can always migrate to DO/etc if needed. Hopefully not, hah.
edit2: Looks like Fly has pre-paid billing[1], though i'm unclear if i can replicate the functionality i'm asking about. In this case i'd need a prepaid limit and a subscription to fill pre-paid with $N once a month.
Prepaid by itself is at least something, if nothing else exists. Just super annoying if you forget to add money.
They claimed they did support price caps the HN launch thread referenced in this new press release. They got a lot of praise for it in the comments back then.
> Max monthly spend: unexpected traffic spikes happen, and the thought of spending an unbounded amount of money in a month is really uncomfortable. You can configure fly.io apps with a max monthly budget, we'll suspend them when they hit that budget, and then re-enable them at the beginning of the next month.
- Volumes (attached SSD, one per compute instance) - $0.15/GB/month.
- Bandwidth - if you go over 160GB you have to pay per GB.
Gut feel, ~$5/month should be fine for a small project as long as you’re not storing loads of data or doing something that requires lots of bandwidth.
If your project is used irregularly and HTTP based, you can also scale compute down to 0 machines. It’ll boot up again when you get a HTTP request. In my experience it took ~3s or so to boot up a NestJS app on a small VM, totally acceptable for a dev environment.
Any idea if there are price caps? That's one thing i like about "dumb simple" machines is i'm paying for the machine, if it gets traffic and dies i still only pay $5/m. I'm not trying to build facebook, it's okay if it dies.
Huh, i'll have to look into that. I'm not aware of unattended upgrades. Though i still value not risking a messed up config leaking data by way of ssh access, or something.
If you install a current version of Ubuntu it uses perfectly sane defaults for SSH.
If you go the step further and disable password authentication entirely and only use keys you’ll be very secure.
I fully understand the preference of a serverless platform of course, especially if you’re more of a programmer and less sysadmin.
> There are two kinds of platform companies : the kind where you can get your Python or Rust or Julia code running nicely, and the kind where you find a way to recompile it to Javascript.
Playful jab on Cloudflare and Deno. Would be interesting if Fly took up hosting "WASM containers" in addition to Docker containers.
I'm not doubting this at all. But let me be more specific what I don't like about their communication.
"$70MM led by EQT Ventures." That's awesome, but what are you giving away for this? Are founders selling their shares? Where is the beef? $70M can be everything and nothing at the same time.
"There are customers who are comfortable engaging with tiny Fly.io, and others who are comfortable engaging with the Fly.io that raised an additional $70M led by EQT ventures." I'm pretty sure that your enterprise customers are already wowed by a 25M infusion by AtoZ. So again, where is the beef, especially after stating: "Why do startups write announcements like these? We went back and forth on it. There are lots of reasons, most of them dumb."
I can keep going with more examples. But the headlines says it all. The 70M clickbait 'we raised a bunch of money" - wow, so chill, bruv.
yes and I reread it a few times in case I was missing something. I even turned my adblocker off in case it was removing the section that answers OP's questions but it didn't appear.
It's obviously perfectly fine if you don't want to or can't answer those questions and nobody is owed answers to them.
Just, you're known for a straight talking style so the omission is conspicuous.
I honestly don't know what it is you're looking to hear. We said what we're using the money for: hardware, rolling out more regions, and a bunch more support and infra engineers.
yes very common. therefore it's highly unlikely to be what OP is asking about.
by this point, it's hard not to conclude that OriginalMrPink and I are being wilfully misunderstood here.
OP is interested in more details about the deal, particularly what power the investors might have, to ascertain for himself to what extent the investment might change the business.
"Why We Raised A Bunch Of Money" doesn't answer any of the questions above.
I've seen private equity rounds disguised as traditional founding rounds, which is neither very transparent, nor very honest. Raising 70M just shortly after a 25M round makes me feel like there was more at play then just 'raising' money tbh. But I might be totally off here - but therefore disclosing a bit more than you currently do might be really helpful.
None of this makes any sense. No PE firm buys a company like Fly.io. I honestly don't understand what either of you are getting at, but I'm confident when I say "you're way off".
We'll be hiring for many of these positions again shortly. We weren't funding constrained, just people constrained; there are whole sprawling threads on HN about how annoyed we managed to make some people (a small minority of applicants, but that doesn't make it any easier for us to metabolize) with our responsiveness in our hiring process. We halted most hiring to retool and nail down hiring processes, and also to onboard all the people we hired (so they can help us do more hiring stuff).
Raising lots of money to hire 2 people is a funny flex, so I sort of wanted to leave this unanswered, but I also don't want to miss an opportunity to put on a hairshirt and talk about things we fucked up.
A but if a side note but … their jobs page is pretty great. Nothing available but they already have very specific and interesting descriptions written for all the positions they have or expect to have. Feels like they are being very mindful about the company they are building.
Running (diskless) production workloads on Fly.io since 2021 (300+ req/sec across 30+ regions). Sure you ran into problems almost every month back then, but not anymore. Not in my experience.
I've been at a bunch of companies with a bunch of raises. 100% of the time, the announcement was an excuse for press. If you can come up with any excuse to get an article published in a bunch of tech press (other than "CEO arrested for embezzlement + harassment at the same time"), you get a bunch of free advertising. Bonus points if your target customer tends to read tech press.
There's nothing wrong with that, to be clear! I'm just surprised they acknowledged mercenary reasons without mentioning what I've seen as the most common one.
Saying it's "for press" just kicks the can down the road, raising the question "why do startups want press on announcements like these?" They do it to get out a certain message and leave a certain impression, usually of the mercenary sort the post outlines.
Although I see your point about it just being to get the name of the company out there, that's definitely one mercenary motivation that the post maybe did not mention.
At the very least, it's marketing. I have no idea what they do, as I'm not in that particular web development niche and don't know what an "edgy app" is, but it made it to the top of HN, so people like me now know at least that there exists a company called fly.io.
You're right - I also think (personal opinion) the trust gained in a company by reading their blog post is greater than the trust gained by seeing an ad. Because I saw this blogpost, I trust fly.io more than I would have had I seen a YouTube ad of them or something of the sorts.
Startups want press in general, and closing a funding round is reliably something that the press will cover. (Unlike, for example, product or feature releases.)
Linear waited nearly a year to announce a round, presumably because they did two in quick succession and didn't want to waste the PR opportunity of the second one.
> I've been at a bunch of companies with a bunch of raises. 100% of the time, the announcement was an excuse for press. If you can come up with any excuse to get an article published in a bunch of tech press (other than "CEO arrested for embezzlement + harassment at the same time"), you get a bunch of free advertising. Bonus points if your target customer tends to read tech press.
It's less about "excuse for press" and more "controlling the narrative". Raising this amount will mean that other news outlets will pick up on it. Technically, as I recall, it's public information (I founded a company and this was all explained to me - but it was years ago).
So if you don't publish they will. You're much better off publishing first so that people read your post, not theirs.
They aren't wrong that there are some companies that feel more comfortable if you have more money in the bank BUT most companies wildly overestimate how much prospects care. Case studies in their industry & Gartner quadrant > $100MM in the bank.
TL;DR: It's an appendix slide in your first meeting deck, don't put it up front :)
Yup. When I was deep in the VC-fueled startup game, tech press did not want to write about us atall, unless there was also some kind of funding news included, too.. (I interpreted this as the writers knowing their readers... and only stories with funding announcements got the clicks...) So guess what happened? We would raise money, then not talk about it for up to 6 months until we also had some new feature or product to launch.
If people started clicking more on tech stories that didn't have funding announcements, maybe the cycle would change... But it feels like we're in some kind of Prisoner's Dilemma game now. Those stories don't written, so they don't get the clicks, so they don't get written...
Yeah, "hold off funding announcements until we've got something else we want to push into the news" was a pretty common thing in my experience as well.
I'm somewhat sympathetic to both the press and the readers here - they want to report/read news. Events. "Company X continues to exist" isn't really news. And "Company X adds new feature Y" is rarely noteworthy news, no matter how big a deal it is in Company X. Sometimes new features are news if either the company is huge ("Amazon adds support for Foo to AWSBar") or it's strategically significant ("Company X adds feature Y, now putting it in direct competition with Company Z"), but I think those are the exceptions.
> There are fun, technical, “control your own destiny” reasons to rack hardware instead of layering on top of commodity clouds. But it's really just economics. ... Hardware is what makes the margins work.
Amen to that.
Nice to see somebody who has raised a ton of money, and yet is still being smart with it.
But I suppose the difficult question is, "Why shouldn't Fly's potential customers also run their own hardware?"
I don't think there's enough expertise in running hardware out there, frankly. For a company where running hardware isn't their bread and butter finding and hiring people who can do it, and do it well, is existentially risky. It is hard enough to find people who can properly run on a PaaS.
Okay so they raised 100m which means they need to work towards a 1bn exit. And if they want to keep their momentum going their r&d cost will go through the roof. Because they have to build the whole stack and it all has to be rock solid (eventually). It’s an immense engineering effort.
I get how this looks profitable on paper, because Fly charges for compute, and hardware and bandwidth are cheap. But the real expense is r&d, marketing and SBC and those will rule out profitability at any scale. These are not one-time expenses. You have to keep spending forever or your customers will leave. We’ve seen this time and time again. Heroku isn’t the only example. It’s the tragedy of platform/infra startups.
That's backwards. Profitability at scale is about the _only_ way for a cloud provider to be profitable. There's a reason you don't see a bunch of small cloud providers, the economics only work at scale.
Platform companies can't scale by just getting more of the type of customers they already have. They scale by getting bigger and more demanding customers. Those customers will expect the things AWS and the other big cloud platforms already offer. And then R&D expenses explode. The bigger they get the higher R&D will be. In absolute numbers and relative to revenue.
Yes, it's true that if you become as big as AWS then economy of scale starts to work in your favor again. But that outcome is so unlikely it's not worth thinking about. The realistic best case outcome is that they become a mid-sized cloud provider, like Digital Ocean. And Digital Ocean, in business for 11 years and having raised 500m (pre IPO) is still losing money. Linode, after 20 years of struggle, ultimately gave up and sold for $900m last year. Even the best case outcome looks pretty bleak.
Will you be publishing a follow up blog post about how you’re increasing API user fees once you’ve monopolized your particular market? Or how about how you’ll be increasing margins for investors in 5 years as you prepare for your IPO?
Why won’t you suffer the fate of every single other tech company that raises a shit load of money which is completely and irrevocably selling out any pretense of being beneficial for customers and employees (primarily) in the extreme long term?
My new heuristic is that I avoid every single company that raises venture funding. Hopefully others adopt this heuristic because by raising tons of money, so you are explicitly creating an adversarial relationship between the customers/users and your investors so everyone but your founding team and investors in the long term is worse off.
Edit: I’ve been a HN power user since 2012 - don’t ask me why I’m here.
The simple answer is: we sell something people want to pay for (VM time, network services, etc). We'll obviously want to improve our margins over time, but there's a market price for this stuff and we don't have pricing power.
I don't think you can build an interesting public cloud without raising money, unfortunately. At least, not without jumping back in time 25 years and starting then.
Yeah no I have to agree with you on this one. Every company is going to raise prices at one point or another - it might be inflation, it might be profit-chasing, padding for an IPO - whatever.
Just because a startup might raise prices in the future shouldn't stop you from using their products - with a caveat, that is, how easy it is to shift to another operator. I haven't used fly.io personally, so I don't know about that - could you elaborate on shifting to another provider just in case?
I am skeptical that Hetzner or OVH or similar could get off the ground starting in 2023.
There are a couple of things working against a boostrapped public cloud:
First, you gotta buy expensive kit. And then hope you can make your money back over the next 18 months.
Funding this is hard. You could try to borrow money, but that (a) increases your underlying cost and (b) dictates how you sell. You can't borrow money against developer usage / traction because banks don't know how to value that. So you have to do top down sales and land some big, committed customers before you can use debt.
The sales model constrains the product. There is no big, committed customer on the planet that will buy global infrastructure from a company who hasn't built it out yet. So you won't be building a global cloud, you'll end up in one region.
And, no big committed customer is going to buy something novel. They do not care that "fly launch" makes it easy for a dev to launch a new project. They care that they got the best possible pricing when they were shopping for their hardware. They might care that they can run k8s on it.
This is all fine, though. I'm not opposed to it. But I think it leaves you with something that's not as good as AWS, even though it's cheaper.
> There are customers who are comfortable engaging with tiny Fly.io, and others who are comfortable engaging with the Fly.io that raised an additional $70MM led by EQT ventures
There's the other kind of customer who sees that "$70MM led by EQT ventures" is a liability and will inevitably screw over their customers at a later date.
It just all seems irrelevant to fly.io. I can host stuff on there for money, which is all I want from a service provider. Electricity and storage and network and compute and infrastructure engineers cost money, and I'm happy to give them money. How does web3 get these things paid for?
It’s not irrelevant to the parent and grandparent comment, which (having experienced the recent buyouts at Reddit and others) are rightly wary of a similar outcome, and regard the mere fact of being funded by VCs as a ticking time bomb. So what is the alternative?
Don’t build your app on someone else’s platform. Whether it’s Twitter and Periscope, or Apple vs mamy developers (https://www.washingtonpost.com/technology/2019/09/05/how-app...) … build on a permissionless platform that isn’t owned by any gatekeepers. Like the Web. Or Ethereum. Or Polygon.
Utility tokens and micropayments via trustlines are exactly done for that purpose.
I’m not sure I follow. It seems like a strawman question of some kind.
Well, fly.io is not the Web, but also it’s not the point
The closest I can get to what your questions is to say, it’s under the control of a for-profit company financing itself with shareholders who want to see profits? That’s true of many companies, the sentiment in the GP comments applies to the general pattern of becoming enshittified after a large investment / sale, not specifically fly.io
But it exposes things on the Web. And if you don't like them you can move somewhere else?
And more generally: people do things for money. Companies wanting a profit is not a bad thing. That urge can go wrong, but that risk is mitigated, in fly.io's case better than most, by the lack of lock-in technology choices.
For example, if you want to, for some insane reason, build an application that uses Microsoft's CosmosDB, then your data is stuck in Azure forever. Fly.io doesn't have that problem, as they use things you can run elsewhere.
So: what is the problem, as pertaining to the topic of this comment page: fly.io?
I think you could just say: "Instead of relying on the concentrated capital of VC firms, we should rely more on individuals investing smaller amounts in companies they want to use. The individuals collective stake in the company can help protect them from the company going against their interests."
That's just the main point right? Everything becomes essentially Kickstarter/Patreon subscriptions, and maybe you pay a little more fees to exchange your money to tokens?
Its surely an appropriate argument to make I guess, just falls a little flat to me personally. Like I just want to use a service and not get ripped off, I don't want to enter into and have stake in some distinct governance structure for every service I want use.
Well yes, and now you can do it using open protocols, without relying on centralized platforms for the money part.
Utility tokens need a decentralized network so there is no single point of failure. Just like the Web itself (1.0) disrupted gatekeepers at newspapers, TV and Radio stations, etc. that you used to have to pay (“payola”) to get the word out. VOIP lowered the cost of long distance phonecalls to pretty much zero by eliminating middlemen, too. The infrastructure providers shouldn’t ALSO control your app layer, that’s the point!
Why is it that any time VC financing comes up in any fashion we end up with some unrelated pseudo-meme walloftext nonsense bouncing back and forth between a treatise on the structural inefficiencies of capitalism and tech fundraising from 30 years ago? All with some tenuous barely coherent tie back to blockchain, or NFTs, or web3, or whatever The Next Bitcoin™ du jour is.
Because open source software is the only alternative we know to the problem that was described, that has consistently liberated people everywhere it was introduced. And utility tokens are a great way that was invented several years ago to monetize open source and turn it into a free market instead of only a gift economy.
Blockchain is the most widespread and proven way thus far to eliminate the middleman, and it’s there right now, you can deploy a token in a few minutes if you wanted. Many people here seem to keep repeating the mantra that “Web3 has no good applications” but here is one that literally solves the societal problem you are complaining about and has done so with IPFS, Ethereum, Polygon and many other permissionless networks.
It’s a wall of text because there are literally so many detailed examples, that I chose to engage in a few. I like to back up what I’m saying so people don’t think I’m just talking out of my ass. Now the ball is in your court to explain why you disagree anyway.
I don't even know that I do disagree with you. I just don't think Venture Capital is some sort of existential threat, or by definition a problem (some specific applications of it, sure), so I don't think the blockchain solves any real problem out of the box. It's an interesting thing from a technical standpoint, and some interesting things have been built on top of it, and I made some money on BTC back before it was 5 figures.
But you're not going to convince anyone to swap out VC funding for utility tokens by talking about Napster and FogBugz.
You really ought to do some standup comedy, Thomas (do you?); I regularly LOL at some of your comments — even the subtle put-downs, like the parent — and yet you're invariably courteous and non-snarky.
If that is the case, then you shouldn't be using Apple, Google, Intel, Nvidia, Stripe, Shopify, Meta, Git, etc etc.
Companies raise money now to generate cash flows in the future from a wildly risky innovation. Assuming the thesis is correct, the company will then to pay it back to investors (VCs and their investors including Endowment Funds), but also employee and taxes.
As companies grow, they pay more taxes, and this will let government spend in things like education, infrastructure, social security.
It's either this or socialism.
"Capitalism: The worst economic system, except for all the others"
> As companies grow, they pay more taxes, and this will let government spend in things like education, infrastructure, social security.
As companies grow so does their ability to dodge taxes. That's why they're headquartered in Ireland, Bermuda or whatever the latest low corporate tax state is. Proportionally they're likely paying less than a corner store. I certainly pay more in income tax.
I don't understand why someone comes to a startup incubator and VC forum when they hate venture funded startups. You know what YC does, right?
There are so many forums you could go to, but you specifically select this one when you fundamentally dislike the premise? Why? It's just baffling behavior.
It may be a VC forum, but it functions as a broad technical forum. How many posts here are explicitly about startup economics?
It's important to remind people in the broader technical community that financial success should not the only goal, nor should it be required to maximize profit at the expense of customers.
Not OP, but I think the answer is that a lot of people who quit Reddit a couple weeks ago have spread out to other news/discussion forums to find something to replace it. They're still in the exploration phase trying to find communities that resemble subreddits they used. The content on this forum sort of resembles a few tech related subreddits, even if the discussions and intent are quite different.
I suppose the community makes the purpose of the community. And this sort of thing makes me worse, so I should seek something else, or modify my interaction with this community.
It's an open forum and presumably those running it prefer this outcome.
This is the cold truth. The entire process of raising money in the modern software world has nothing to do with creating a better product and business specifically. It's simply the de facto step-by-step playbook process one follows if one is an entrepreneur with the eventual become very rich. There is nothing novel, innovative, or remotely surprising about this process. It is well defined, well established, and simply how business is done. Every single person looking to build a company for the purposes of becoming rich should adhere religiously to this playbook. It's a cold, hard, transactional fact of life. Whether or not the product is good and fair and honest is largely an independent variable to this basic playbook process.
Sometimes funding is necessary to grow income enough to sustain the business or hire more engineers or whatever you want to do.
If it costs $40 to acquire a new customer, and you expect a customer to stick around for long enough to spend $90 then it's totally worth doing that, but you need $40 now to make 90$ over some period of time.
That’s what's so great about five year projections! Somehow it’s always worth it until reality comes knocking after the money’s been raised. Then it’s a mad scramble for ROI that debases everything that built the business.
There is an entirely possible and alternative funding model that perpetually allocates a responsible amount of incremental funds for the purposes of novel R&D. It's trivial to imagine how a responsible incremental funding approach could create a better, more transparent, more estimable, more reportable, more mappable, more rigorously trackable innovation process.
Raising massive lump sums of money is about VALUATION. You have been led to believe this is the only way to go about building a software business like this.
It's all a major silkscreen for the colder simpler goal of generating a lot of eventual wealth for a specific cast of people.
Again, no shame or emotion here. It's just how business is done in this world.
> There is an entirely possible and alternative funding model that perpetually allocates a responsible amount of incremental funds for the purposes of novel R&D.
This works for some but not all business ventures.
There's a reason corporations were invented in the Age of Sail. If you build 5% of a ship, you can't sail to the New World and bring back 5% of the resources. You just sink in the harbor.
(The moral implications of relating modern VC-invested corporations to rapacious European conquests of the already-inhabited-thank-you-very-much New World is not lost on me.)
I believe there are healthy ways to start up a business that has startup costs too large to bootstrap. I agree with you that VCs are often not it.
And by the time you've done that 97.3% of all shipping from the new world is being handled by a small handful of companies with gigantic cargo ships.
But congratulations, you've successfully created a small "local" business with a decent number of customers who are reasonably loyal. You should be happy with what you have and possibly invest in future generations who want to colonize the moon or something (assuming you want to expand your wealth)
Or maybe 20 customers who believe that having a shared ship might form a join stock company to take care of their respective 5%... and when the venture proves itself, they might convert it to a publicly traded company and let it scale up with demand.
That alternative only works in the absence of competitors who raise capital to pursue more rapid growth. It is an arms race to an extent, but if you don't play the game then you may get locked out.
Particularly when that rapid growth involves price dumping. I've bootstrapped businesses before. It's really difficult to compete with free offerings when you have to pay the bills. Your competitors can burn a pile of cash to flush out the competition. They don't even need a better product... people love getting stuff for free. Invariably, those services either go bust or don't remain free since it's often unsustainable. But, it means in the short-term you often need to raise money if you're competing with a VC-backed business.
I'm curious have you ever tried to raise money for a tech business before?
There's a big reason VC and angels were and largely continue to be only game in town. No one understands or would risk loaning millions of dollars on a high risk business aiming for marginal incremental growth.
Real world example: was DropBox a better or worse product when they were a simple syncing app before they started “innovating”.
The only consumer company that I can think of that went from “initial value proposition” to “not sucking after it went public” is BackBlaze. None of their new offerings has taken away from their initial “unlimited cheap backup” offering.
> There is an entirely possible and alternative funding model that perpetually allocates a responsible amount of incremental funds for the purposes of novel R&D.
You are completely missing the point of venture capital here.
Of course, bootstrapping exists, but I can't point to any similar companies to Fly which have achieved standout success in an area like this with a model like that.
> Raising massive lump sums of money is about VALUATION.
Valuation is merely a side effect in pursuit of market dominance or hopefully establishing a niche monopoly.
You raise money to pay for costs of running the business. If you couldn't raise money, then only people who are already extremely rich would be able to start a business of any size, let alone one that requires deploying physical hardware in multiple regions. Good luck getting even four regions for less than $1M. The world would be far worse off if all tech businesses had to be bootstrapped.
That being said, there's certainly many businesses that raised too much money based on bullshit, had the founders take money off the top, and then have to drastically change to become sustainable, especially in the ZIRP era. But I don't see that happening here.
To add to it, without external funding _any_ innovation would be stifled by existing players. If you can't even get low millions in funding, existing corporations with tens of billions of free money will eat you alive.
Organically growing a business; bootstrapping sounds fine, until you try to do something that potentially has any global impact.
It's funny to hear this in this context. Fly is treated as one of the few potential successors to Heroku, and Heroku raised very little money while remaining the number one choice for most devs here (based on sentiments). And then an existing corporation with tens of billions of free money came along, bought Heroku ... and ran it into the ground. So I don't buy that you need to fundraise or else be outpaced, at least not by big corporation.
To be fair, they do call out in the article that they're hosting on their own hardware. Afaik Heroku has always been hosted on AWS which is a lot less capital-intensive than buying hardware.
Also, Heroku was acquired in 2010. I know that the prevailing sentiment around here is that the acquisition was a mistake, but Heroku has been owned by Salesforce for 13 of it's 16 years of life...
You can offset burn rate with more skills. Putting a few machines in multiple regions is fairly cheap. If you’re ok doing the ops yourself, you can get buy on a shoe string budget. I know. A startup I was part of did this ten years ago and the tooling to do this has only gotten easier and better.
To be clear, I’m not saying that venture is intrinsically evil, but also companies are explicitly trading off higher burn rates and higher raise rounds to reduce the need to build all of this in house. That’s not a bad idea because it’s usually better for the business to scale as quickly as the sales channel can fill it rather than be stuck on difficult engineering roadblocks.
> If you couldn't raise money, then only people who are already extremely rich would be able to start a business of any size, let alone one that requires deploying physical hardware in multiple regions
You’re describing the current state of the world
We’re there already. The entire system is built to exploit everybody who does not have significant capital to fight back against it
I am so curious why you repeatedly spit in the collective face of the millions of small business owners in the US and around the world.
If what you say were true, the world would exist as a series of 3-5 megacorps, and small businesses would not exist. Since they do exist, how do you square that with your claim that only the "extremely rich would be able to start a business of any size" is currently true?
Starting a business requires little more than filing incorporation documents with your state.
Growing a business to dominate a market and get crazy rich may require outside funding, but growing a business to support and maintain your individual lifestyle does not.
I would also add to your point that starting a business is easier than it has ever been. While far from perfect, the internet has been an amazing equalizer.
On the one hand, it makes it easy for you to reach customers.
On the other hand, it makes it easy for your competitors to reach customers.
I'd argue that, on the whole, the internet has made starting "just an idea and a garage" businesses harder, because they now face immediate, maximally-funded competition. Whereas pre-Internet they would have been geographically/physically protected for awhile.
True, net win for customers, efficiency, etc. (maybe). But you couldn't start a Starbucks these days.
> spit in the collective face of the millions of small business owners in the US and around the world.
Their income is similar to that of most working professionals and their assets are similar to the savings of most working professionals. They entrepreneured themselves into having a job, they're not the investor/capitalist multimillionaires we picture when we think of rich people.
>the world would exist as a series of 3-5 megacorps, and small businesses would not exist
The economy exists as the 500 megacorps that make up the s&p 500. Small businesses just fill in the gaps and are for the most part beholden to the megacorps for their business, their supplies and/or their financing.
It's just not though. Surely you know (or know of) people who've built successful businesses with the help of funding, and couldn't have done so without it.
Just because someone takes funding, doesn't make them any less or a part of some "built" system. Even my favourite bakery could never have got started without funding - with debt not equity. It doesn't make him any less brilliant. He would've had to be extremely rich so start the business otherwise
Not all funding is creating equal or share the same expectations. Borrowing money from a bank to open a bakery or taking an investment from friends and family to cover the initial startup costs is very different from raising venture capital money. The growth and return on investment expectations are wildly different.
Which is sort of natural, isn't it? The natural world is set up to exploit everything that does not have a defense against it. You adapt or you get eaten or out-competed. The fact that "the non-rich" have not yet realized that they can, in fact, band together to force more powerful forces to behave better, doesn't mean they can't still do it.
Our biggest advantage is working together as a community, and our biggest flaw is not helping each other when we easily could.
No, it's not. It's emotionally manipulative propaganda that is thinly-veiled political ranting, and so is your comment, and neither belong on HN. Stick to interesting, intellectually stimulating discussion, please.
Curious people are allowed some latitude when historical performance predicts future outcomes [1]. This is "not the first rodeo" as it were. Agree being antagonistic is not welcome, but it should not be unwelcome to ask hard questions about how something won't end up the same.
I like the Fly.io folks! But asking questions should never be off the table; how else would you be curious?
(it is unfortunate that something with so much value and enjoyment like HN is a product of an investment firm, but a component of life is compromise; enjoy it like you would a public good while it lasts)
> why are you on a site whose community is literally created and run by a startup accelerator?
Why do missionaries go where no one believes their religion already? The idea that you must already be a believer in some community to participate in the community makes little sense and certainly isn't a good way to make progress.
> I guess my only other question is why seek attention like this?
Would you also characterize your own reply as mere attention seeking?
Because they have a political agenda to push and have decided that Hacker News is a good place to push it (even though that's a blatant violation of the HN guidelines).
I joined in '09 and I think I've changed pretty substantially, so the idea that this or any website would lock-in to a particular viewpoint isn't realistic IMO.
And you're being awfully reasonable now, once we've gotten past the bombastic opening claim. I think part of what makes HN great is that at least sometimes we can skip that opening wild claim.
I think this is just due to the maturation of the community; it has grown and diversified, so enthusiast hackers represent a smaller proportion of it. The same dynamic is at play in the tech community at large.
But that's a fundamentally different view than "All VC funding is bad"; there are tons of risks involved and many actors aren't good in every industry, why specifically does venture capital draw your ire?
>My new heuristic is that I avoid every single company that raises venture funding
you're basically going to be prevented from using anything that isn't an established enterprise(which are also beholden to shareholders) or consumer apps. You can't bootstrap anything that requires significant R&D or infrastructure unless the founder is already rich
This post seems to be referencing the Reddit API situation. The solution is to not make your company reliant on a single API , pretty sure with Fly you can deploy Docker containers so you could always switch easily if they tried to price gouge. If something is a critical component of your business you should always plan to be able to switch providers in a hurry for any number of reasons
I don't think they they'll need to increase, fly.io is already rather expensive as far as cloud offerings go today: shared 8 vCPU / 16GB VPS is $85/mo at fly, Hetzner is $16/mo for the same.
It's worse. For a 16 vCPU + 32GB RAM + 20TB traffic (fly.io has max 8 vCPU) I would pay around $750 on fly.io instead of €30 on Hetzner. People are willing to pay a lot for I guess an easy deploy story ?
Developer time is pretty costy and if you can save a lot of dev time a noticeable higher running cost can be well worth it for some companies (like smaller companies, startups which have still a fluent changing business model etc.).
Through also it's not a fair compression because for the kind of services where being close to the edge and redundant over many places matters Hetzner isn't even an option. AWS maybe, but AWS is also pain in my experience.
How isn’t? You’re paying for a VM with access to a slice of resources. Yes fly.io offers some clever deployment and routing, but it’s still your code running on a VM.
You could ask the same question about why Hetzner is so much cheaper than the big cloud providers. If there weren’t any tradeoffs, presumably every company running containers or VMs on a big cloud would move to Hetzner and save 80%. Why isn’t that happening?
Whatever the answer is, it’s probably the same answer for why many companies would likely be willing to pay more for fly.io than Hetzner as well.
A lot of companies are bootstrapped or "seed-strapped".
My company raised a $1.2m seed after doing a startup accelerator in 2015, used it to get to profitability, and we've been growing 40% YoY for a very long time without any additional outside capital.
It's also a huge selling point to prospective employees while interviewing candidates: "Unlike many other startups, we haven't raised lots of money, we're profitable, we've never done lay offs. That's all possible because we have product market fit and we have a sustainable business model. Our customers (rather than VCs) fund our growth."
Personally, I really like the "seed strapped" model... raise 1-2 million in order to find PMF and begin an early sales/marketing function with the goal of reaching profitability before the $1-2 million is burned through. It also creates a short window to fail fast... lots of companies raised way too much money and will end up failing very slowly.
"Failing very slowly" is what should be avoided.
Raising huge amounts of VC is a signal the company couldn't sustain their growth with their current revenue and operating model. It's also a signal that the company is likely using inorganic / unscalable tactics to grow, which might work now but won't work forever. That means the company will likely pivot their model at some point (from an employee perspective, that means higher risk of lay offs, and from a customer perspective, that means higher risk of price hikes in the future, etc)
This is an entirely valid model, and many startups would be better served by following it. Still, the whole point of (larger) VC funding rounds is facilitate rapid growth, far more than 40% year over year. There's no contradiction between embracing bootstrapping/seedstrapping where it works, and VC funding for the cases with hyper-growth potential.
I chose those because they are ubiquitous and downright hard to avoid. What do you want to bet OP made that post on a Chrome browser? Or an Android/iPhone/Macbook? Is there even a way to purchase a computer of any sort that doesn't involve supporting a venture backed company? The list of venture backed companies is so long, it would be exhaustive if not impossible to avoid using them without going off grid.
How many venture backed companies do you think are involved just in me pressing the "reply" button to send this post?
I think It's actually quite a beneficial pattern for the likes of the consumer and the founders.
Say Fly will eventually get acquired or IPO, their founders will justify it that they could use that money to work on Fly's mission statement or if they've had their fill of servers go on to build further startups in Fusion, AI, or Space Technologies or even give to charity (take your pick).
When they do get acquired/IPO the natural bureaucracy of large organisations and shift in incentives will set in (we've been here before with Heroku/Salesforce and SendGrid/Twilio...) and they'll become slow, more risk-averse, and ultimately less innovative catering for enterprise and other large businesses (where the easy corporate money is at) instead of scrappy startups and curious hackers.
This is where Fly 2.0 comes in 5 years down the line reaching No. 1 on Hacker New, where utilising the latest technology they'll create a completely new and innovative solution that will solve the current problem even better than now and they'll start by catering for startups and hackers until they themselves get acquired/IPO.
This doesn't mean it's necessarily a bad thing - founders get a chance to cash out, consumers get cycles of new innovation.
Except when you get something like YouTube or the American telcos that have a monopoly/oligopoly and suck all potential for competition out of the space.
While I tend to agree with your overall thesis, and I get particularly baffled when some SaaS company, whose nearly sole expense is payroll, feels the need to raise hundreds of millions of dollars, fly.io is a cloud infrastructure company.
They literally run physical servers all over the world (at least, that's my understanding from their website). I've got to imagine then that this business has huge capital costs, and it's nearly impossible to grow a very capital intensive business without outside capital.
I don't disagree with your main point. Everyone has seen "enshittification" eventually take over all tech startups as they switch the focus from satisfying users to satisfying investors. But I just don't see how you build a business that requires running servers all over the world without a lot of capital.
> That's the promise of pay-per-minute (or even second) cloud computing, right?
Maybe I misunderstood your point, but that's the promise for cloud computing customers. Fly.io is a cloud computing provider, and those are all hugely capital intensive businesses.
> They literally run physical servers all over the world ... I've got to imagine then that this business has huge capital costs, and it's nearly impossible to grow a very capital intensive business without outside capital.
I genuinely struggle to understand how Fly.io managed (manages) to have this much physical server presence to date, having raised only ~$16M prior to this round. Hire a dozen engineers and that money starts to go fast. The regions page [1] shows 34 cities largely across 4 continents, but technically across 6 if you count the 1 region each in Africa and Australia.
Maybe they have managed to get inside other existing cloud's data centers and it's not literally their own physical hardware, or behind-the-scenes they are leasing collocated servers, etc.
I would love some insight here if one of the founders in the thread sees this.
The short answer is that none of our regions are so big that we have to own our own top-of-rack switching yet (though some are getting close), so we can take a random spot in (say) an Equinix rack, rather than having to engineer a network (we run our own BGP for Anycast, but our uplinks are just DAC connections to our upstream providers switches.
For what it's worth: we've been on our own hardware ever since we launched, long before the 14MM Intel/Dell round. It's really the only way we can see to make the margins make sense. A big part of the premise of AWS and GCP is that they're allocating hyperscaler-grade resources to the task of making sure they claim most of the margins in hosting things on their own platform, not middlemen --- though that may be more true for some services (like EC2) than others (like S3).
I think it's a good question, but the linked article says unambiguously near the bottom that they run their own hardware:
> Why We Raised A Bunch Of Money
> Here's what we think it takes to build this kind of platform:
> A hardware fleet. Fly.io has always run on its own hardware. There are fun, technical, “control your own destiny” reasons to rack hardware instead of layering on top of commodity clouds. But it's really just economics. If you want to get people to build apps on your platform, you need a shot at being around 10 years from now. Hardware is what makes the margins work.
you can finance new hardware and rack in a managed colo for extremely cheap. like really cheap.
i was in this business until a couple of years ago.
we'd quote prices for potential customers currently in public cloud and they literally wouldn't believe us. it actually became an impediment in the sales cycle and part of the reason we sold. people are dumb, but i'd be even dumber if i expected them to change for my benefit.
>selling out any pretense of being beneficial for customers and employees (primarily) in the extreme long term?
I mean... raising prices ≠ "selling out any pretense..."
You're upset that a growing business lowers prices to grow faster? Ok, but maybe relax a little. You're upset that advertising-based businesses eventually shut down the ad-free clients that use the API? Ok, but maybe relax a little.
A business grows with low prices and loses some customers not willing to pay when they put up prices that reflect longer term sustainability. That happens, why does it have to be a moral outrage?
Yeah, if you're using a VC-backed startup expect the cheap services to get pulled back on when they hit IPO time.
I’m not against this, by that point there will be another hosting provider with even cooler features for small companies to make their next startup in.
Fly.io are doing great I think and that they might one day charge more I’m not that upset by it at all. If fly.io can make money at a higher price that means they are adding more value.
I hasten to add you appear to be hanging out on a forum focused on VC funded startups.
Complaining about companies raising venture capital on hacker news! What!? No way!?
This is going to sound wild, but believe it or not it takes quite a bit of capital to reserve capacity in data centers ALL OVER THE WORLD in order to, like, deliver on your core value prop of enabling deploying normal apps AT THE EDGE. This used to just be called making a capital investment (because it takes a lot of upfront capital), but then it became en vogue to whine about venture capital.
You may want your hosting provider to be capital poor and running a razor thin balance sheet on the brink of insolvency (aka bootstrapped) but I don't. Or you might want to lease your own space in a colo and rack your own servers and hire your own remote hands and your own sysadmins and dev ops with 24/7 coverage so if you have a hardware failure you can deal with it asap but I don't. Not having to do all that shit takes...capital.
Where would you have them fetch said necessary capital if not from venture capital firms?
Honestly, do you even know what you're bitching about? A theoretical monopolistic reality that a. does not exist and b. would not exist if fly did not theoretically create a product so good it made all their competitors irrelevant (note: this is not a monopoly, it's market dominance; they are very different)?
And honestly, if they're still around in 5 years, they probably should raise prices so they can continue to be around. Fly is literally orders of magnitude cheaper than running on AWS, and orders of magnitude easier.
Further, it seems like these highly integrated “app platforms as a service” have been the primary type of dev tool that VCs are drawn too given the complete vendor lock in they provide/demand. It worked for squarespace, wix, Shopify, et al. Vercel has a softer sell on this, and firebase was one of the first to offer a fully managed database in this space.
My hot take is that these platforms are relying on an influx of new developers who need a friction free way to build and deploy applications to learn and showcase while selling to companies that don’t have time or budget to create a full dev team and CI/CD environment, creating both sides of supply and demand. Pricing is engineered in a way such that it gets very expensive the moment before companies notice how much cost it incurs but the cost to switch aways is much higher.
I agree with the heuristic of avoiding VC-backed products, it fosters incentives that often leave otherwise loyal customers holding the bag for a product who price does not match the service provided. It is for this reason I consider high vendor-lock-in products rather insidious.
How can you monopolize this market? There is no moat here at all. There are already trillion dollar companies competing here. They are selling a commodity.
Sure, but if you are a startup and your bill went from $10,000/m (say this is comparable to AWS) to $100,000/m because the fly investors said so, you would probably move? Or go bust? Either way it is not good game theoretically for the vendor to do that.
And if they did they also wont get the enterprise customers who are considering conservative alternatives.
I would partially disagree. I think your statements are true, but only for companies that raise investments on the promise of scaling to greater profitability, but ultimately fail to deliver. If the founders delude themselves or are overly optimistic, the only recourse they have to investors is to gouge the customer.
Raising the 'right' amount should be the goal. Not 'as much as possible'
I can understand how you might think investors mis-align the business with customers, but this isn't necessarily true.
First, businesses aren't in the business of leaving money on the table. That is fine for a non-profit, but ultimately they need to weigh money making with customer satisfaction and growth. It's crazy to think a business should not optimize here.
Second, you're conflating venture-funded consumer businesses with B2B businesses. Consumer businesses with venture funding are typically going to fuel growth/momentum by doing things which don't scale. But eventually push comes to shove. B2B business models are usually more transparent about what is a promotion and what is not. Consumer businesses don't always know how the business model is going to play out to even offer that transparency.
Third, if your business model is advertising and data is your moat, then when you give away your data via API to promote distribution, yes you run into an issue later on when you need to advertise or you have underinvested somewhere (in Reddit's case it was mobile). For Reddit or Twitter, these mobile apps were profiting off their users/data while taking on none of the costs (infrastructure, moderation, etc.), and limiting their means of monetizing themselves. You don't need an investor to tell you that isn't in your long term interest.
> Why won’t you suffer the fate of every single other tech company that raises a shit load of money which is completely and irrevocably selling out any pretense of being beneficial for customers and employees (primarily) in the extreme long term?
To be fair, the fate of most of them is actually to fail. Hence this amplified effect of why VCs need such a standout massive return to make the fund model work.
That said, while it seems like it to us normal folk, in the grand scheme of VC, $25M is not really "a bunch of money". In 2021, a16z led or co-lead $3.2B [1] in funding rounds. I can't find a publicly available stat for how many rounds that entailed, but napkin math says 100–300. Of course the distribution is not linear, but if we take the 200 investments midpoint, that's a $16M check on average.
The additional $70M here follow-on is the real story. It looks like they are not giving it a label, though Crunchbase lists the $25M from a16z a year ago as Series B, so I'm inclined to call this new round the Series C.
> My new heuristic is that I avoid every single company that raises venture funding.
IMO tech, startups at least, are not long-minded like this in general. Companies regularly come and go in 6–24 months. Good luck convincing the current and next waves of CTOs of not using venture-backed tech [that saves them tons of time for great prices right now]. And then portcos also often make deals with other portcos... the cycle feeds itslef in more ways than one.
It's like trying to change the color of the ocean with one single cup of red dye. At the end of the day, this is just a rounding error.
But also, Big Tech will end up acquiring many of the standouts a la Firebase or Heroku. Resisting the model won't upend or stop it.
Good luck avoiding every single venture-funded company. Can you give an example of such a "monopoly"? Vercel? They have plenty of competitors, all they are doing is charging a premium for convenience. There will always be alternatives to any given tool if you don't like the costs. Seems your problem is with capitalism and not this particular startup.
Consider this perspective: Users of Fly.io are essentially getting a loan from venture capitalists. However, just like any loan, it has to be repaid eventually. Thus, it's prudent not to take on an excessively large commitment.
Yeah, I follow the same view, I've stopped paying attention to companies that raise venture capital or are publicly traded (aka, they don't truly own the direction of their own business). Whatever good is there is _always_ subverted by the persistent need for "growth" and "increasing engagement" and hitting metrics that are pleasing to investors. It'll start out pretty minor, but the trajectory is persistently altered at nearly every single point of decision. The sway is there, no matter what, and sometimes major decisions are driven solely by "it's what our stakeholders want/need", and the thought of the end-user of the software is practically an afterthought or side-effect.
other things being equal, organic growth via paying customers is the time-honored way of avoiding the pathologies you mention.
in this model speculative capital is only required in the very early stages. a company either reaches a symbiotic relation with its clients or not. the burden is primarily on intrinsic aspects of what the startup venture delivers and how much this resonates in its sector.
the VC model is basically turbo-charging this process. though it is risk capital and not lending, it creates an implicit, arbitrarily sized liability that need not have much to do with the underlying value proposition. it removes the organic cashlow constraint via a faustian bargain.
the "beauty" of it is that you can't have both models in the same economy. the set of ideas that are ripe for exploration at an given era are what they are. if some people pursue them while being on steroids this means there is no room for people to explore them in a less toxic way
Here is an alternative, unpopular (for the narrative like in the above post) hot take.
Nothing is forever. Five years is a long time. People need motivation. Enjoy and make use of things while they last. Nobody owes you anything.
But you have options.
You can have a business genius build a company that delivers great product forever without grinding employees into the dust, burning up founders and still outmaneuvering competitors.
You can have no product or a company going under because founders have stretched them thin, burned out and went broke ruining their lives pursuing a good idea that morphed into a sunken cost.
Or you can crazy pricing models justified by the desire to live like a VC LP while shipping a 3rd party API-dependent cute app. But without VC!
Or you can have a VC-subsidized business that ships a product at ridiculously low prices, locks you in for some 3-5 years and then goes ballistic and becomes unusable. But it’s been five years of a very good run, FIVE YEARS. And then the next will come to take their place.
Reading how five years is a short timeline for a low-cost-high-value product is wild. Now that’s some entitlement.
This kind of comment is blatantly violating the HN guidelines in both letter and spirit - it's a generic political tangent meant to spark a flamewar that has no informational content or value whatsoever.
> There are two kinds of platform companies : the kind where you can get your Python or Rust or Julia code running nicely, and the kind where you find a way to recompile it to Javascript.
My impression is that when fly.io has been discussed in the recent past on HN, the conversation tends to be less about databases and more about Heroku alternatives. (?)
Out of curiosity, how _does_ fly.io see itself in relation to Heroku?
I'd never used it before I joined Fly.io, but everybody else in this company absolutely adored it; they're like the Rolling Stones to our Black Crowes, the key influence.
Most of Heroku's revenue is from Postgres. They're arguably a managed Postgres provider with some app hosting tooling (this is not fair, but an interesting way to think about Heroku).
We're not Heroku. We _like_ how amazingly easy it is to get an app up and running on Heroku. But most of us were interested in flexing the underlying infrastructure to build new types of apps. So Heroku felt very constrained.
We see ourselves as, like, a good RPG. Quick to get going, rewarding and replayable when you go deep.
I'd dispute that MOST of the revenue is from Postgres, though it is likely a very large double digit percentage of revenue.
Their managed Postgres got an unfair advantage early on by being the default. With a new rails app you were just given a Heroku Postgres database. Heroku's Postgres offering was over 5 years ahead of RDS Postgres and a large part of the reason Amazon added support for Postgres for years they fought it, but eventually had to give into the constant customer request. Unfortunately that innovation of Postgres has stalled out a bit over time due to Salesforce starvation.
I do admit for what people think of as PaaS the Postgres revenue is significant, the part people would probably be surprised about is how much revenue the add-ons marketplace constitutes.
Interestingly New Relic got the same privileged experience in the early days and it resulted in the same type of revenue and mass adoption for New Relic.
My impression is that when fly.io has been discussed in the recent past on HN, its founders/employees/whatever swarm the comments to try and control the conversation, heh.
Have you seen other attempts by other companies to do this here, though? They often end badly. It only works for fly.io because they know how to read the room and aren’t trying to do something horrible.
EDIT: Oh, and they often add valuable info to what they posted, too.
Congrats! Please focus on getting your documentation cleaned up, your cli consistent, and bugs excised. Your platform is good, and doesn't need too many additional features. Make it all cleaner, rinse and repeat.
I'm a pretty happy Fly.io customer and glad to see the direction things have been going overall. While it's great to see Fly is getting more resources to continue improving and building the business I worry about the inevitable VC Countdown Clock to Exit.
What does this new round mean for Fly's long term independence?
I don't think we can build a company like this without a lot of money.
We're to the point where we're making $10-25mm capital expenditures in one go. And we have to to optimize our own costs. Optimizing our own costs means long term independence.
What I think I've learned about VCs is: the trick to long term independence is to find investors who already value the sales mechanism. Bottoms up, dev focused "sales" is something investors like, now (this was not true in 2012).
483 comments
[ 3.4 ms ] story [ 156 ms ] threadSo true it hurts
can you explain this part of the joke?
It's not intended to be taken literally (I assume).
Must be gerrymandering or something, maybe aided by voter suppression by English speakers against those in the county whose native language is x86-64.
Their brand marketing is impeccable as a result. Every post on their blog comes through with this very clear, slightly snarky voice.
If you want to be a famous hacker, you must also be a slightly (industry) famous author.
Their tone of voice reminds me of early Slack before that became rote and boring.
Snarky is passable in open source, a genius whose abrasive manner is the cost of the amazing product.
In a corpo it's sleazy
Doing stuff isn't enough - you also need to communicate to the Earth that you have done stuff.
Was it just articles or more?
(The comment I'm responding to said something about people in glass houses not throwing stones, but was then edited).
Raising a bunch of money just isn't a "good" announcement to most of us. When unpopular things have happened with the company, your communication usually directly acknowledges and focuses on the concerns of the userbase. This serves two purposes:
1. It gives you a chance to show concrete steps to prevent specific concerns we have
2. Perhaps more importantly, it shows we're all on the same page and you're actually in touch with how your userbase thinks
Failing on #1 is something you can fix later, but I've never seen a company rebound from losing the trust gained by #2. It loses confidence that future decisions made can also be equally out of touch.
I've never seen a company rebound from losing that confidence
There are things that happen that disappoint customers that we care a lot about. Reliability issues several months ago absolutely fit that bill. So does hiring: we paused hiring for most of our engineering roles because we weren't happy with the experience candidates were getting. I don't know how "honest" and "transparent" we are about those things, but I do feel like when we're asked about them, we give straight answers.
There are other things that happen that disappoint people that we don't care so much about. Being unhappy that we took external funding to build out our fleet and improve reliability would count as one of those things. Again with the straight answers: I wouldn't expect us to spend too much time trying to "rebound" from this announcement.
Speak for yourself. Announcements of adequate funding inspires confidence fly will have longevity and resources to scale. Other concerns can be tackled in other blog posts, but nothing about the announcement of funding should inspire less confidence, what sense does that make?
VCs in particular will expect much more than their money’s worth, or (most of the time) for the company to die trying to earn it, and they won’t hesitate to force the latter in pursuit of the former. This increase in variance is literally their investment model.
So if VCs expect their investments to fail most of the time, and I see a company take VC money, I must have very strong reasons if I’m to disbelieve the VCs and expect the company to succeed—and I’m not following a venture investment strategy when I choose a tech supplier.
IPOs are somewhat milder in their intrinsic character, but they still usually lead to professional boards of directors and eventually professional execs, where “professional” usually means “no subject matter expertise”. And hardly anybody runs a public company on customer satisfaction nowadays, unfortunately.
So when I see controlling equity sold, my outlook changes from “will this go to shit?” to “when this inevitably goes to shit, will it be in the distant enough future that I don’t care or can plan for the loss?” (that being maybe a decade or so for infra, or implausibly long—like the rest of my natural life—for anything involving a community).
As usual, the question remains—why does none of this seem to be priced in? I don’t know.
ARE YOU SAYING YOU INVENTED THE OFFICE?
Ashburn is literally like the Embassy Suites next to the Five Guys, the Aloft next to the taco place, Equinix, Digital Reality, Verizon, NTT, CenturyLink, etc.
Take a look at this map here: https://www.loudoun.gov/DocumentCenter/View/171141/2022Elect...
Ashburn is not literally just one shopping plaza. It's a giant area zoned into different parts of datacenter, suburbs, and commercial sections.
Even forgetting all of this, just type "restaurants in Ashburn" into Google Maps see the plethora of results.
Just because someone has some imaginary boundary of what constitutes as Ashburn doesn't make it definitively small.
I'm really concerned that Fly's employees like yourself think availability and service uptime isn't important. I think I'll be avoiding Fly on this alone.
I'm confused, when did any of Fly's employees say as much?
That said, the point I'm trying to make is that it is in fact a lot nicer to use than simply YOLOing whatever onto servers. This model of PaaS is basically what App Engine should have been. It's difficult to want to go back once you've used it.
Money doesn't contribute all that much to solving it though. It's usually time to make the early mistakes, continuous improvements, etc. But AWS had several decades to solve this, whereas fly clearly hasn't, so I'm not sure I'd count them out yet.
I suspect that later starters like Fly, Cloudflare, etc with their approaches of (more or less) compute at edge with no concept of region etc will grow more and more to dominate the cloud space with AWS region-based approaches primarily relegated to "legacy applications" that are stuck there. It may take a generation or so to get there but more and more the next crop of startups, etc are building on things like Fly instead of AWS and friends.
When I think about it the region based approaches are more akin to "hosted elastic datacenter" than they are what could be described as a "true cloud" - where it's just everywhere and not even a consideration.
I've never seen anything even remotely hinting at region or geography in their product line other than geo-routing for load balancing products, headers with geo info for you to do something with, etc. They include the serving "POP" in headers for diagnostic purposes but other than that you have no idea.
Where do my Workers run? Don't know, don't care. They are substantially cheaper and offer better availability and response times than their region-based "cloud" competitors. Same for KV, D1, R2, and anything else they come up with as they move further and further in (out?).
One would expect with them having the ability to allocate supporting hardware dynamically and globally their cost basis is substantially better than having customer facing and controlled region-based resources that still need to be built out for product support (regardless of usage) AND maintain the excess capacity local to each region to be anywhere near "elastic".
Glad that works for you but for my game servers edge computing is not ideal and generally currently not even applicable.
At some point in time they'll have to start thinking about profitability.
Assuming Cloudflare is a similar case to Amazon: ie they could turn profitable at any time but choose to reinvest those profits immediately into more business, that is a very good spot to be in.
The actual difference, at least how Cloudflare pushes it, is in the software; its a higher layer of abstraction; that customers get to stop thinking about regions and locations. Your software is global; its deployed everywhere; it runs everywhere.
This is ENTIRELY antithetical to everything on AWS except CloudFront, Route53, IAM(ish), maybe a couple other minor services. Every Single Thing Every Single AWS Customer configures beyond those global services forces you into a region. Want to deploy a lambda function? Which region would you like? Pricing and available products is different in every region. Fronting it with an API? Where would you like that API Gateway? We'll put CloudFront in front of it so its a bit faster for ya, its global(ish), don't worry.
In a no-true-scottsman true-edge world, this goes out the window. Cloudflare Workers is like this; D1 is kinda-ish-there-ish; R2 isn't. Here's the code; spider it across the world; automatically route clients to your closest PoP. No regions. No locations.
Its not obvious that AWS will ever make meaningful moves to a world like this, for a ton of reasons, but most meaningfully: it requires customer demand; but AWS's existing customers aren't demanding it, because AWS's customers are old CTOs worried about past quarters too much to think about the next one. The younger, smarter, potential customers who recognize how valuable this is aren't AWS's customers; so AWS never hears from them (which, by the way, should more generally concern Amazon's investors; their "relentless customer driven" approach to their products is great for a few decades, but its absolutely true that most of the time customers just want a faster horse, and you're not hearing from the people who aren't your customers because you've never solved their problems before). So, Amazon has dumped billions upon billions into their couple-dozen giga-regions; you cannot pivot that into hundreds upon hundreds of mini-regions like what Cloudflare (and to a lesser but still awesome degree Fly) has.
Vercel is the best case study on this. They are exploding in popularity. They don't run any of their own infra afaik, to be clear, but they whitelabel products from Cloudflare, Neon, and others; all Edge infra providers. No one on Vercel is cross-shopping AWS, and AWS has nothing that feels competitive to what Vercel offers. They can't compete.
The outcome, in maybe 10-20 years, of this landscape feels really obvious right now, but AWS's size and the insane scale of their investment into "how we did things in 2010" will turn them into the next IBM. Some would say they already are. There's nothing they can do about it. They're not making the right investments. Even if they wanted to, they can't at the necessary scale, because what they're doing is working right now and their existing customers keep forcing them to spend more money building more structures, more racks, more blades, all in Ashburn VA.
When state is introduced, then CAP/PACELC distributed systems issues arise, and figuring out approaches to deal with them are a must. Read fly’s Postgres docs, and regions come right back in.
Only the most trivial business systems can operate without any state.
If your application has any sort of shared, mutable state, you absolutely need to pick a region, or go really deep into the EC/consensus/clustering rabbit hole. With a dedicated region with 1 big DB, you can achieve latency figures (over the shared state) that would be infeasible with other schemes. Serializing transactions in a distributed manner is a circus compared to what Postgres has to do to achieve the same.
For better or worse, sticking a big SQL database in exactly one region will almost always be the best path for most forms of business. If latency/edge are still a concern at this point, that's when we maybe start talking about more specific geographies and standing up read replicas in those regions. In my experience, if you are peddling B2B webapps, no one will ever complain about east vs west coast latency unless you actually screwed something up in code.
I point to Cloudflare (what I'm most familiar with). Across their entire stack of compute, storage, db, etc product lines you won't see a hint of region anywhere. It just works, everywhere. Same with Fastly/Oracle (although they like Fly are quite a ways behind Cloudflare).
I suspect (hope) that as Fly matures their apparent true goal of edge and region-less applications will mature more. Their compromises on region awareness for DB has always struck me as a more of a resource restraint compromise on their part than end goal. As you say these challenges are hard (expensive).
https://developers.cloudflare.com/d1/learning/data-location/
There is no free lunch here. This is always a game of information theory and physics at the end.The fact remains this is somewhat down in the docs because it is automatic and invisible to the point it works well enough where most developers don't know, don't care, and aren't exposed to it in any meaningful way. Contrast this with AWS where the first thing you are presented with is picking a region where everything lives, while not clearly and explicitly explaining that many things are ultimately dependent on us-east. Cue the shock and confusion when us-east has yet another issue and those in all other regions don't understand how and why they are impacted.
You are correct about the free lunch, information theory, and physics at the end. I don't think a reasonable person can think when you write data to a pop in Tokyo that it could possibly show up in Chicago instantly. That is fundamentally impossible and always will be. I think CF does a pretty good job making it "magic" for most devs and use cases while explaining these fundamental realities and limitations to those who need to know (potentially) for whatever their application is.
Here's the Durable Objects documentation [1]. Don't those "location hints" look like regions?
Also, here's the "Data location" page for Cloudflare D1 [2]. See "available hints" at the bottom.
I think it would be more accurate to say that Cloudflare's architecture is designed around having lots of regions, automatic migrations between them, and no guarantee about which one you'll get. But they're still there.
[1] https://developers.cloudflare.com/workers/runtime-apis/durab...
[2] https://developers.cloudflare.com/d1/learning/data-location/
I actually argued against adding these hints at all, since it confuses the messaging and most people don't need them... but they were easy to add and solved an immediate problem for a few customers, so we did.
In other cases we've held firm. For example, some people have wanted to restrict their Workers to run in specific regions because the Worker made lots of requests to a specific back-end in that region. Instead of actually offering explicit hints, we built a system to detect this automatically: https://blog.cloudflare.com/announcing-workers-smart-placeme...
In any case, most people don't have to think about regions on Workers, and for the few that do, our goal is to change that.
(I'm the tech lead for Cloudflare Workers.)
That is an excellent way to design a product. Great job.
Passing HIPAA data through other countries when not needed to is generally frowned upon - data should be kept as local as reasonably possible. I haven’t worked with GDPR requirements but I have heard they are similar. AWS solved this problem well with regions. I’m curious what other providers that are “regionless” do to solve the compliance problem.
All lower plans maybe get PCI DSS and that's it.
It's a valid counter to my point - while they clearly can do it locking it behind an enterprise "talk to us" subscription is lame. That said they are generally customer, market, and product savvy and I suspect they know that any/most orgs that are able to pull off the broader process, etc for actual compliance with these requirements are in fundamentally different positions than some random startup with a bunch of consumer data.
We're also seeing more and more ambiguity with coaching platform startups, etc who don't have a covered entity in sight but that's a different topic for a different day.
I've spent most of my career in HIPAA-relevant startups. Generally, I think it's actually a disservice all around for Amazon to basically rubber stamp these architectures and solutions with a HIPAA BAA so that companies think they can call themselves "HIPAA compliant". That's not even what it's supposed to mean, let alone the virtually endless issues and process that need to happen throughout the org for "compliant" to actually mean anything. I haven't reviewed it but I'm certain the AWS BAA is so specific and exclusionary it's trivial to step outside of what it covers.
Yes these agreements are a piece of the puzzle but it's completely reckless to throw something together in AWS, present the BAA to a customer, and call yourselves "HIPAA compliant". Savvy customers, investors, etc will call you out but otherwise it's mostly just a matter of time before you discover your standalone AWS BAA is actually just one of the things on a 50 point (or whatever) checklist.
N-2 companies ago I was a tech lead in 2015 where we were migrating to AWS. AWS was constantly explaining which underlying services were HIPAA compliant and what you build on top of it was your responsibility. I had to answer to separate non AWS compliance folks about my architecture.
AWS also gave guidance. But made no promises that your implementation was compliant. That’s where the entire “shared responsibility model” comes in.
Now, I work at AWS in ProServe. I am very familiar with our messaging regarding HIPAA compliance. We are very careful not to rubber stamp things as compliant. I know the best practices in and out. But when I’m asked if something is compliant, I say this is the guidance we are given. I’ll do a presentation before an Architecture Review board. But I would never make anything that hints as a guarantee.
The messaging/communication was more like what I described - "Oh yeah here's BAA you're good to go".
If you are a smaller organization, you may be able to request an SA.
My personal experience is from when companies pay for consulting hours from ProServe.
At the n-2 company I referred to above, the only thing we got from AWS is a BAA and we had an outside consulting company and compliance auditors. I was just learning AWS then.
In the Cloudflare Workers platform, we have a number of features to control "jurisdictions" where compute runs or where data is stored. These are meant for compliance, not for performance. Our goal is that no one has to think about regions for performance reasons, but certainly compliance issues are not going to go away -- in fact, they are getting more onerous over time. So features to control jurisdiction are likely to expand.
(I'm the tech lead for Workers.)
But it is rarely true for enterprise customers. They want the guarantees their data never leaves certain regions even in a request along with other security and compliance guarantees.
But that's just not true. Tons of cloud services completely abstract the computer out of the picture. You are paying for capacity or throughput as an abstract unit of cost and the cloud configures as many computers as needed to run your request. You never interact with anything resembling a computer in this situation.
Not to mention, regions matter when you're serious. GDPR/DSA/DMA/India/China -- the list goes on. Certain data must live in certain bubbles.
I'm currently at a startup that started on Aptible a few years ago because there was a lack of expertise in systems development. We're now in the process of migrating to AWS natively because of cost savings (it's why I was hired). Aptible is costing us 3-4x what native AWS will. Yes, we're incurring some operational overhead, but the automation and tooling for managing infra is reasonably mature (Terragrunt & Terraform aren't perfect, but they work well enough).
Ultimately, startups may start with higher-level cloud providers, but they'll eventually migrate to lower-level providers eventually. At the higher end of the growth curve, companies likely wind up getting back to on-prem in some way.
AWS does have local zones and has Lambda at Edge. I will be the first to admit though that the local zones are tied to regional zones and that some global services are tied to us-east-1 like IAM.
For example, make your app talk to two backends in different EC2 regions, and now none of the smart placement helps you.
The poster we're replying to seems to be gleefully ignorant of speed of light limits and fundamental distributed systems challenges.
https://www.loudoun.gov/177/Coat-of-Arms
"Where Tradition Meets Innovation" appears to be a branding push by the county dating to 2019 rather than an official motto. https://www.loudoun.gov/ArchiveCenter/ViewFile/Item/8563
Kudos to these companies, the VCs, and the beta testers.
And they completely avoid discussing it again in this article.
It would be nice if I could know what their product is for because it doesn't seem to be the apps I work on.
And then some load balancer cleverness that reroutes writes to a specific VM: https://fly.io/blog/globally-distributed-postgres/
I think your company building out developer tools to help use your platform would aid greatly in adoption...seems a necessity really, particularly addressing the pain-point of accessing data close to users.
I personally believe that could be true, but their investors going to find out ;)
But is there a limit to how boring you can go? Can you truly, fully abstract over distributed systems stuff like eventual consistency and its basically random effects on UX in certain usage edge cases?
Or do you think 'boring' will have to meet you in the middle with frameworks and developer mindset also shifting over time to a more edge-first world?
I love Fly.io, used it for side projects and I was very pleased with the UX and the free tier.
The blog post is funny but the style is a bit excruciating to read.
Cheers, guys.
while Fly.io deploys compute instances to more regions and also has a database?
asking because I've been content with Vercel and Netlify's CDN for frontend assets and simply stopped doing system design around relational storage, when I want to stay on a free tier.
Heroku has been handicapped for nearly half a decade, there are a lot of people that already know of alternatives and have changed their system design to accommodate the alternatives. How does Fly.io fit in with Vercel, or Netlify? As opposed to people keeping their Heroku apps on life support
Fly.io takes Docker containers, turns them into VMs, and lets you decide where and when to have them running.
One isn't better than the other; they're just totally different models for hosting applications. The Fly.io model is much closer to what Heroku did, just modernized (with Docker, edge deployments, the `fly-proxy`, etc).
I also liked Heroku, so it's sad to see their status now.
Is fly competitive with this? These sorts of calculations are always awkward for me with PaaS providers. I know what i'm getting with $5/m with DO, but with PaaS it often feels abstracted and sneaky. Heroku in my very, very old memory was a continual set of asterisks.. but i'm probably being unfair and PTSD'y.
Think i should give Fly a look? I at least find their SQLite features super interesting
Persistent volumes are $0.15/gb each.
Not too far off, depending on how much data you have. However the platform has different tradeoffs than a single DigitalOcean VM. You'll get better uptime percentages on a single DO VM than you will on a single VM Fly app. You may not notice, but it's statistically true.
If so i'll probably give it a try!
edit: Regardless, going to give it a try. I was totally set on just using DO because i wanted something dumb and simple. This sounds like it might offer me that, plus not having to manage the OS. I really, really like that if i can keep pricing similar.
Bandwidth costs might become a concern, but i can always migrate to DO/etc if needed. Hopefully not, hah.
edit2: Looks like Fly has pre-paid billing[1], though i'm unclear if i can replicate the functionality i'm asking about. In this case i'd need a prepaid limit and a subscription to fill pre-paid with $N once a month.
Prepaid by itself is at least something, if nothing else exists. Just super annoying if you forget to add money.
[1]: https://community.fly.io/t/can-i-set-a-billing-limit-per-mon...
> Max monthly spend: unexpected traffic spikes happen, and the thought of spending an unbounded amount of money in a month is really uncomfortable. You can configure fly.io apps with a max monthly budget, we'll suspend them when they hit that budget, and then re-enable them at the beginning of the next month.
https://news.ycombinator.com/item?id=22616857
Under $5/month you pay nothing at the moment.
https://fly.io/docs/about/pricing/
You need to look at the following for pricing:
- Compute
- Volumes (attached SSD, one per compute instance) - $0.15/GB/month.
- Bandwidth - if you go over 160GB you have to pay per GB.
Gut feel, ~$5/month should be fine for a small project as long as you’re not storing loads of data or doing something that requires lots of bandwidth.
If your project is used irregularly and HTTP based, you can also scale compute down to 0 machines. It’ll boot up again when you get a HTTP request. In my experience it took ~3s or so to boot up a NestJS app on a small VM, totally acceptable for a dev environment.
I’d say if static pricing is important to you stick with DO.
Security wise, Ubuntu LTS with unattended upgrades and livepatch is likely better than 99% of public facing servers.
I fully understand the preference of a serverless platform of course, especially if you’re more of a programmer and less sysadmin.
Playful jab on Cloudflare and Deno. Would be interesting if Fly took up hosting "WASM containers" in addition to Docker containers.
I think, the jab on Cloudflare was the "get a salesperson to call" part.
That's a lot of cash to go toward 14 jobs https://fly.io/jobs/.
"$70MM led by EQT Ventures." That's awesome, but what are you giving away for this? Are founders selling their shares? Where is the beef? $70M can be everything and nothing at the same time.
"There are customers who are comfortable engaging with tiny Fly.io, and others who are comfortable engaging with the Fly.io that raised an additional $70M led by EQT ventures." I'm pretty sure that your enterprise customers are already wowed by a 25M infusion by AtoZ. So again, where is the beef, especially after stating: "Why do startups write announcements like these? We went back and forth on it. There are lots of reasons, most of them dumb."
I can keep going with more examples. But the headlines says it all. The 70M clickbait 'we raised a bunch of money" - wow, so chill, bruv.
proceeds to not really talk about it
> but what are you giving away for this? Are founders selling their shares? Where is the beef? $70M can be everything and nothing at the same time.
It's obviously perfectly fine if you don't want to or can't answer those questions and nobody is owed answers to them.
Just, you're known for a straight talking style so the omission is conspicuous.
what OP is asking is, what did the investors receive in exchange for giving fly the money.
by this point, it's hard not to conclude that OriginalMrPink and I are being wilfully misunderstood here.
OP is interested in more details about the deal, particularly what power the investors might have, to ascertain for himself to what extent the investment might change the business.
You might have found your audience. But for me the tonality of the post felt way off.
Raising lots of money to hire 2 people is a funny flex, so I sort of wanted to leave this unanswered, but I also don't want to miss an opportunity to put on a hairshirt and talk about things we fucked up.
Spinning up VMs. It's not difficult to do this on AWS or GCP.
Nevertheless, I'm excited for them and continue to cheer them on until they mature.
I've been at a bunch of companies with a bunch of raises. 100% of the time, the announcement was an excuse for press. If you can come up with any excuse to get an article published in a bunch of tech press (other than "CEO arrested for embezzlement + harassment at the same time"), you get a bunch of free advertising. Bonus points if your target customer tends to read tech press.
There's nothing wrong with that, to be clear! I'm just surprised they acknowledged mercenary reasons without mentioning what I've seen as the most common one.
Although I see your point about it just being to get the name of the company out there, that's definitely one mercenary motivation that the post maybe did not mention.
Cost to make article = $2000
Article gets 100,000 views
Fly.io converts 1% of views to paying customers
1,000 new customers.
Revenue per customer = $10 monthly
MRR goes up by $10,000
Obviously I am pulling these numbers out of my ass, but there a pretty direct path to value.
This is way more efficient than Google ads in a competitive space, I can only imagine hosting related ads are $3-5 per click...
Advertising a raise specifically is to get attention from customers and the industry as a legitimized successful operation.
It's less about "excuse for press" and more "controlling the narrative". Raising this amount will mean that other news outlets will pick up on it. Technically, as I recall, it's public information (I founded a company and this was all explained to me - but it was years ago).
So if you don't publish they will. You're much better off publishing first so that people read your post, not theirs.
TL;DR: It's an appendix slide in your first meeting deck, don't put it up front :)
So I guess 400 bank accounts would be enough?
If people started clicking more on tech stories that didn't have funding announcements, maybe the cycle would change... But it feels like we're in some kind of Prisoner's Dilemma game now. Those stories don't written, so they don't get the clicks, so they don't get written...
I'm somewhat sympathetic to both the press and the readers here - they want to report/read news. Events. "Company X continues to exist" isn't really news. And "Company X adds new feature Y" is rarely noteworthy news, no matter how big a deal it is in Company X. Sometimes new features are news if either the company is huge ("Amazon adds support for Foo to AWSBar") or it's strategically significant ("Company X adds feature Y, now putting it in direct competition with Company Z"), but I think those are the exceptions.
Just like press coverage is 3rd-party validation to potential customers that a company is actually worth considering.
even though there’s a reason (or three) behind the style of the post, including the use of transparency, it is refreshing.
Amen to that.
Nice to see somebody who has raised a ton of money, and yet is still being smart with it.
But I suppose the difficult question is, "Why shouldn't Fly's potential customers also run their own hardware?"
You can only build so many company margins on top of capital expenditures.
I get how this looks profitable on paper, because Fly charges for compute, and hardware and bandwidth are cheap. But the real expense is r&d, marketing and SBC and those will rule out profitability at any scale. These are not one-time expenses. You have to keep spending forever or your customers will leave. We’ve seen this time and time again. Heroku isn’t the only example. It’s the tragedy of platform/infra startups.
Yes, it's true that if you become as big as AWS then economy of scale starts to work in your favor again. But that outcome is so unlikely it's not worth thinking about. The realistic best case outcome is that they become a mid-sized cloud provider, like Digital Ocean. And Digital Ocean, in business for 11 years and having raised 500m (pre IPO) is still losing money. Linode, after 20 years of struggle, ultimately gave up and sold for $900m last year. Even the best case outcome looks pretty bleak.
Why won’t you suffer the fate of every single other tech company that raises a shit load of money which is completely and irrevocably selling out any pretense of being beneficial for customers and employees (primarily) in the extreme long term?
My new heuristic is that I avoid every single company that raises venture funding. Hopefully others adopt this heuristic because by raising tons of money, so you are explicitly creating an adversarial relationship between the customers/users and your investors so everyone but your founding team and investors in the long term is worse off.
Edit: I’ve been a HN power user since 2012 - don’t ask me why I’m here.
I don't think you can build an interesting public cloud without raising money, unfortunately. At least, not without jumping back in time 25 years and starting then.
Do you provide a seamless offboarding experience?
Or do you lock in customers?
It's pretty easy to move off, though. `fly launch` generates a Dockerfile you can run pretty much anywhere.
Just because a startup might raise prices in the future shouldn't stop you from using their products - with a caveat, that is, how easy it is to shift to another operator. I haven't used fly.io personally, so I don't know about that - could you elaborate on shifting to another provider just in case?
Only if your condition is that you want to do it in the next 3 years.
Given 15, I imagine quite a lot is possible.
There are a couple of things working against a boostrapped public cloud:
First, you gotta buy expensive kit. And then hope you can make your money back over the next 18 months.
Funding this is hard. You could try to borrow money, but that (a) increases your underlying cost and (b) dictates how you sell. You can't borrow money against developer usage / traction because banks don't know how to value that. So you have to do top down sales and land some big, committed customers before you can use debt.
The sales model constrains the product. There is no big, committed customer on the planet that will buy global infrastructure from a company who hasn't built it out yet. So you won't be building a global cloud, you'll end up in one region.
And, no big committed customer is going to buy something novel. They do not care that "fly launch" makes it easy for a dev to launch a new project. They care that they got the best possible pricing when they were shopping for their hardware. They might care that they can run k8s on it.
This is all fine, though. I'm not opposed to it. But I think it leaves you with something that's not as good as AWS, even though it's cheaper.
> There are customers who are comfortable engaging with tiny Fly.io, and others who are comfortable engaging with the Fly.io that raised an additional $70MM led by EQT ventures
There's the other kind of customer who sees that "$70MM led by EQT ventures" is a liability and will inevitably screw over their customers at a later date.
Don’t build your app on someone else’s platform. Whether it’s Twitter and Periscope, or Apple vs mamy developers (https://www.washingtonpost.com/technology/2019/09/05/how-app...) … build on a permissionless platform that isn’t owned by any gatekeepers. Like the Web. Or Ethereum. Or Polygon.
Utility tokens and micropayments via trustlines are exactly done for that purpose.
But okay, I'll bite. How is fly.io not the Web?
Well, fly.io is not the Web, but also it’s not the point
The closest I can get to what your questions is to say, it’s under the control of a for-profit company financing itself with shareholders who want to see profits? That’s true of many companies, the sentiment in the GP comments applies to the general pattern of becoming enshittified after a large investment / sale, not specifically fly.io
And more generally: people do things for money. Companies wanting a profit is not a bad thing. That urge can go wrong, but that risk is mitigated, in fly.io's case better than most, by the lack of lock-in technology choices.
For example, if you want to, for some insane reason, build an application that uses Microsoft's CosmosDB, then your data is stuck in Azure forever. Fly.io doesn't have that problem, as they use things you can run elsewhere.
So: what is the problem, as pertaining to the topic of this comment page: fly.io?
That's just the main point right? Everything becomes essentially Kickstarter/Patreon subscriptions, and maybe you pay a little more fees to exchange your money to tokens?
Its surely an appropriate argument to make I guess, just falls a little flat to me personally. Like I just want to use a service and not get ripped off, I don't want to enter into and have stake in some distinct governance structure for every service I want use.
Utility tokens need a decentralized network so there is no single point of failure. Just like the Web itself (1.0) disrupted gatekeepers at newspapers, TV and Radio stations, etc. that you used to have to pay (“payola”) to get the word out. VOIP lowered the cost of long distance phonecalls to pretty much zero by eliminating middlemen, too. The infrastructure providers shouldn’t ALSO control your app layer, that’s the point!
Blockchain is the most widespread and proven way thus far to eliminate the middleman, and it’s there right now, you can deploy a token in a few minutes if you wanted. Many people here seem to keep repeating the mantra that “Web3 has no good applications” but here is one that literally solves the societal problem you are complaining about and has done so with IPFS, Ethereum, Polygon and many other permissionless networks.
It’s a wall of text because there are literally so many detailed examples, that I chose to engage in a few. I like to back up what I’m saying so people don’t think I’m just talking out of my ass. Now the ball is in your court to explain why you disagree anyway.
But you're not going to convince anyone to swap out VC funding for utility tokens by talking about Napster and FogBugz.
Companies raise money now to generate cash flows in the future from a wildly risky innovation. Assuming the thesis is correct, the company will then to pay it back to investors (VCs and their investors including Endowment Funds), but also employee and taxes.
As companies grow, they pay more taxes, and this will let government spend in things like education, infrastructure, social security.
It's either this or socialism.
"Capitalism: The worst economic system, except for all the others"
As companies grow so does their ability to dodge taxes. That's why they're headquartered in Ireland, Bermuda or whatever the latest low corporate tax state is. Proportionally they're likely paying less than a corner store. I certainly pay more in income tax.
There are so many forums you could go to, but you specifically select this one when you fundamentally dislike the premise? Why? It's just baffling behavior.
It's important to remind people in the broader technical community that financial success should not the only goal, nor should it be required to maximize profit at the expense of customers.
It's an open forum and presumably those running it prefer this outcome.
Fair enough.
If it costs $40 to acquire a new customer, and you expect a customer to stick around for long enough to spend $90 then it's totally worth doing that, but you need $40 now to make 90$ over some period of time.
Raising massive lump sums of money is about VALUATION. You have been led to believe this is the only way to go about building a software business like this.
It's all a major silkscreen for the colder simpler goal of generating a lot of eventual wealth for a specific cast of people.
Again, no shame or emotion here. It's just how business is done in this world.
This works for some but not all business ventures.
There's a reason corporations were invented in the Age of Sail. If you build 5% of a ship, you can't sail to the New World and bring back 5% of the resources. You just sink in the harbor.
(The moral implications of relating modern VC-invested corporations to rapacious European conquests of the already-inhabited-thank-you-very-much New World is not lost on me.)
I believe there are healthy ways to start up a business that has startup costs too large to bootstrap. I agree with you that VCs are often not it.
But congratulations, you've successfully created a small "local" business with a decent number of customers who are reasonably loyal. You should be happy with what you have and possibly invest in future generations who want to colonize the moon or something (assuming you want to expand your wealth)
At some point, you need the capital investment to build a big ship, and the return on that investment is zero unless you build the whole ship.
You might describe these people as "capitalizing a venture", in fact, or "venture capitalists".
There's a big reason VC and angels were and largely continue to be only game in town. No one understands or would risk loaning millions of dollars on a high risk business aiming for marginal incremental growth.
VC's and angels wouldn't be interested in that, either.
The only consumer company that I can think of that went from “initial value proposition” to “not sucking after it went public” is BackBlaze. None of their new offerings has taken away from their initial “unlimited cheap backup” offering.
You are completely missing the point of venture capital here.
Of course, bootstrapping exists, but I can't point to any similar companies to Fly which have achieved standout success in an area like this with a model like that.
> Raising massive lump sums of money is about VALUATION.
Valuation is merely a side effect in pursuit of market dominance or hopefully establishing a niche monopoly.
Who does this allocation?
That being said, there's certainly many businesses that raised too much money based on bullshit, had the founders take money off the top, and then have to drastically change to become sustainable, especially in the ZIRP era. But I don't see that happening here.
Organically growing a business; bootstrapping sounds fine, until you try to do something that potentially has any global impact.
Also, Heroku was acquired in 2010. I know that the prevailing sentiment around here is that the acquisition was a mistake, but Heroku has been owned by Salesforce for 13 of it's 16 years of life...
To be clear, I’m not saying that venture is intrinsically evil, but also companies are explicitly trading off higher burn rates and higher raise rounds to reduce the need to build all of this in house. That’s not a bad idea because it’s usually better for the business to scale as quickly as the sales channel can fill it rather than be stuck on difficult engineering roadblocks.
You’re describing the current state of the world
We’re there already. The entire system is built to exploit everybody who does not have significant capital to fight back against it
If what you say were true, the world would exist as a series of 3-5 megacorps, and small businesses would not exist. Since they do exist, how do you square that with your claim that only the "extremely rich would be able to start a business of any size" is currently true?
4.7M of the 8M US businesses have <5 employees.
https://www.census.gov/newsroom/stories/small-business-week....
Starting a business requires little more than filing incorporation documents with your state.
Growing a business to dominate a market and get crazy rich may require outside funding, but growing a business to support and maintain your individual lifestyle does not.
On the other hand, it makes it easy for your competitors to reach customers.
I'd argue that, on the whole, the internet has made starting "just an idea and a garage" businesses harder, because they now face immediate, maximally-funded competition. Whereas pre-Internet they would have been geographically/physically protected for awhile.
True, net win for customers, efficiency, etc. (maybe). But you couldn't start a Starbucks these days.
Their income is similar to that of most working professionals and their assets are similar to the savings of most working professionals. They entrepreneured themselves into having a job, they're not the investor/capitalist multimillionaires we picture when we think of rich people.
>the world would exist as a series of 3-5 megacorps, and small businesses would not exist
The economy exists as the 500 megacorps that make up the s&p 500. Small businesses just fill in the gaps and are for the most part beholden to the megacorps for their business, their supplies and/or their financing.
Just because someone takes funding, doesn't make them any less or a part of some "built" system. Even my favourite bakery could never have got started without funding - with debt not equity. It doesn't make him any less brilliant. He would've had to be extremely rich so start the business otherwise
Our biggest advantage is working together as a community, and our biggest flaw is not helping each other when we easily could.
No, it's not. It's emotionally manipulative propaganda that is thinly-veiled political ranting, and so is your comment, and neither belong on HN. Stick to interesting, intellectually stimulating discussion, please.
It's a given you realize how antagonistic you're being here, so I guess my only other question is why seek attention like this?
I like the Fly.io folks! But asking questions should never be off the table; how else would you be curious?
[1] https://ourincrediblejourney.tumblr.com/
(it is unfortunate that something with so much value and enjoyment like HN is a product of an investment firm, but a component of life is compromise; enjoy it like you would a public good while it lasts)
This is not what curiosity looks like, is my point.
I don’t think anyone wants HN to become an echo chamber that believes VC funding is the solution to all problems.
Why do missionaries go where no one believes their religion already? The idea that you must already be a believer in some community to participate in the community makes little sense and certainly isn't a good way to make progress.
> I guess my only other question is why seek attention like this?
Would you also characterize your own reply as mere attention seeking?
And further, why go on a (holy) mission to convince people funding companies is bad? That's a weird windmill to tilt at...
Svelte, SolidJS, Remix, and NextJS are all tied to San Francisco venture capitalism and that's got me worried about their future.
https://news.ycombinator.com/newsguidelines.html
I stand by my point though that the culture has significantly changed since 2012 - you of all people know that.
Just go back and look a decade ago at when people would post “hey we raise all this money” posts.
A full half of the comments would be like “nobody cares that you raise money are you actually doing something important?” or similar.
Tbh I think it was a healthier perspective, and we need to bring more “hacker” back.
And you're being awfully reasonable now, once we've gotten past the bombastic opening claim. I think part of what makes HN great is that at least sometimes we can skip that opening wild claim.
you're basically going to be prevented from using anything that isn't an established enterprise(which are also beholden to shareholders) or consumer apps. You can't bootstrap anything that requires significant R&D or infrastructure unless the founder is already rich
This post seems to be referencing the Reddit API situation. The solution is to not make your company reliant on a single API , pretty sure with Fly you can deploy Docker containers so you could always switch easily if they tried to price gouge. If something is a critical component of your business you should always plan to be able to switch providers in a hurry for any number of reasons
Through also it's not a fair compression because for the kind of services where being close to the edge and redundant over many places matters Hetzner isn't even an option. AWS maybe, but AWS is also pain in my experience.
Whatever the answer is, it’s probably the same answer for why many companies would likely be willing to pay more for fly.io than Hetzner as well.
My company raised a $1.2m seed after doing a startup accelerator in 2015, used it to get to profitability, and we've been growing 40% YoY for a very long time without any additional outside capital.
It's also a huge selling point to prospective employees while interviewing candidates: "Unlike many other startups, we haven't raised lots of money, we're profitable, we've never done lay offs. That's all possible because we have product market fit and we have a sustainable business model. Our customers (rather than VCs) fund our growth."
Personally, I really like the "seed strapped" model... raise 1-2 million in order to find PMF and begin an early sales/marketing function with the goal of reaching profitability before the $1-2 million is burned through. It also creates a short window to fail fast... lots of companies raised way too much money and will end up failing very slowly.
"Failing very slowly" is what should be avoided.
Raising huge amounts of VC is a signal the company couldn't sustain their growth with their current revenue and operating model. It's also a signal that the company is likely using inorganic / unscalable tactics to grow, which might work now but won't work forever. That means the company will likely pivot their model at some point (from an employee perspective, that means higher risk of lay offs, and from a customer perspective, that means higher risk of price hikes in the future, etc)
What about posting on a forum owned and operated by a venture firm?
Do you use Google? Slack? Facebook? The list of venture backed tech firms is endless.
I don't even disagree with the general sentiment...but your resolution seems particularly short-sighted.
Not a convincing list of companies there...
How many venture backed companies do you think are involved just in me pressing the "reply" button to send this post?
Say Fly will eventually get acquired or IPO, their founders will justify it that they could use that money to work on Fly's mission statement or if they've had their fill of servers go on to build further startups in Fusion, AI, or Space Technologies or even give to charity (take your pick).
When they do get acquired/IPO the natural bureaucracy of large organisations and shift in incentives will set in (we've been here before with Heroku/Salesforce and SendGrid/Twilio...) and they'll become slow, more risk-averse, and ultimately less innovative catering for enterprise and other large businesses (where the easy corporate money is at) instead of scrappy startups and curious hackers.
This is where Fly 2.0 comes in 5 years down the line reaching No. 1 on Hacker New, where utilising the latest technology they'll create a completely new and innovative solution that will solve the current problem even better than now and they'll start by catering for startups and hackers until they themselves get acquired/IPO.
This doesn't mean it's necessarily a bad thing - founders get a chance to cash out, consumers get cycles of new innovation.
They literally run physical servers all over the world (at least, that's my understanding from their website). I've got to imagine then that this business has huge capital costs, and it's nearly impossible to grow a very capital intensive business without outside capital.
I don't disagree with your main point. Everyone has seen "enshittification" eventually take over all tech startups as they switch the focus from satisfying users to satisfying investors. But I just don't see how you build a business that requires running servers all over the world without a lot of capital.
That's the promise of pay-per-minute (or even second) cloud computing, right? You don't spend what you don't need _right now_.
But Fly.io makes a good point about reaching critical mass to discuss better prices from their vendors. I can totally understand that.
Maybe I misunderstood your point, but that's the promise for cloud computing customers. Fly.io is a cloud computing provider, and those are all hugely capital intensive businesses.
I genuinely struggle to understand how Fly.io managed (manages) to have this much physical server presence to date, having raised only ~$16M prior to this round. Hire a dozen engineers and that money starts to go fast. The regions page [1] shows 34 cities largely across 4 continents, but technically across 6 if you count the 1 region each in Africa and Australia.
Maybe they have managed to get inside other existing cloud's data centers and it's not literally their own physical hardware, or behind-the-scenes they are leasing collocated servers, etc.
I would love some insight here if one of the founders in the thread sees this.
[1]: https://fly.io/docs/reference/regions/
For what it's worth: we've been on our own hardware ever since we launched, long before the 14MM Intel/Dell round. It's really the only way we can see to make the margins make sense. A big part of the premise of AWS and GCP is that they're allocating hyperscaler-grade resources to the task of making sure they claim most of the margins in hosting things on their own platform, not middlemen --- though that may be more true for some services (like EC2) than others (like S3).
> Why We Raised A Bunch Of Money
> Here's what we think it takes to build this kind of platform:
> A hardware fleet. Fly.io has always run on its own hardware. There are fun, technical, “control your own destiny” reasons to rack hardware instead of layering on top of commodity clouds. But it's really just economics. If you want to get people to build apps on your platform, you need a shot at being around 10 years from now. Hardware is what makes the margins work.
i was in this business until a couple of years ago.
we'd quote prices for potential customers currently in public cloud and they literally wouldn't believe us. it actually became an impediment in the sales cycle and part of the reason we sold. people are dumb, but i'd be even dumber if i expected them to change for my benefit.
I mean... raising prices ≠ "selling out any pretense..."
You're upset that a growing business lowers prices to grow faster? Ok, but maybe relax a little. You're upset that advertising-based businesses eventually shut down the ad-free clients that use the API? Ok, but maybe relax a little.
A business grows with low prices and loses some customers not willing to pay when they put up prices that reflect longer term sustainability. That happens, why does it have to be a moral outrage?
Yeah, if you're using a VC-backed startup expect the cheap services to get pulled back on when they hit IPO time.
Fly.io are doing great I think and that they might one day charge more I’m not that upset by it at all. If fly.io can make money at a higher price that means they are adding more value.
I hasten to add you appear to be hanging out on a forum focused on VC funded startups.
"Growth" is the focus now, until the business dies (by acquisition, IPO or bankrupcy, all of which are bad for users).
This is going to sound wild, but believe it or not it takes quite a bit of capital to reserve capacity in data centers ALL OVER THE WORLD in order to, like, deliver on your core value prop of enabling deploying normal apps AT THE EDGE. This used to just be called making a capital investment (because it takes a lot of upfront capital), but then it became en vogue to whine about venture capital.
You may want your hosting provider to be capital poor and running a razor thin balance sheet on the brink of insolvency (aka bootstrapped) but I don't. Or you might want to lease your own space in a colo and rack your own servers and hire your own remote hands and your own sysadmins and dev ops with 24/7 coverage so if you have a hardware failure you can deal with it asap but I don't. Not having to do all that shit takes...capital.
Where would you have them fetch said necessary capital if not from venture capital firms?
Honestly, do you even know what you're bitching about? A theoretical monopolistic reality that a. does not exist and b. would not exist if fly did not theoretically create a product so good it made all their competitors irrelevant (note: this is not a monopoly, it's market dominance; they are very different)?
And honestly, if they're still around in 5 years, they probably should raise prices so they can continue to be around. Fly is literally orders of magnitude cheaper than running on AWS, and orders of magnitude easier.
Take your aimless cope elsewhere.
My hot take is that these platforms are relying on an influx of new developers who need a friction free way to build and deploy applications to learn and showcase while selling to companies that don’t have time or budget to create a full dev team and CI/CD environment, creating both sides of supply and demand. Pricing is engineered in a way such that it gets very expensive the moment before companies notice how much cost it incurs but the cost to switch aways is much higher.
I agree with the heuristic of avoiding VC-backed products, it fosters incentives that often leave otherwise loyal customers holding the bag for a product who price does not match the service provided. It is for this reason I consider high vendor-lock-in products rather insidious.
How can you monopolize this market? There is no moat here at all. There are already trillion dollar companies competing here. They are selling a commodity.
The "etc." is basically thousands of other companies.
And if they did they also wont get the enterprise customers who are considering conservative alternatives.
Raising the 'right' amount should be the goal. Not 'as much as possible'
Money is like sex - only too much is enough.
First, businesses aren't in the business of leaving money on the table. That is fine for a non-profit, but ultimately they need to weigh money making with customer satisfaction and growth. It's crazy to think a business should not optimize here.
Second, you're conflating venture-funded consumer businesses with B2B businesses. Consumer businesses with venture funding are typically going to fuel growth/momentum by doing things which don't scale. But eventually push comes to shove. B2B business models are usually more transparent about what is a promotion and what is not. Consumer businesses don't always know how the business model is going to play out to even offer that transparency.
Third, if your business model is advertising and data is your moat, then when you give away your data via API to promote distribution, yes you run into an issue later on when you need to advertise or you have underinvested somewhere (in Reddit's case it was mobile). For Reddit or Twitter, these mobile apps were profiting off their users/data while taking on none of the costs (infrastructure, moderation, etc.), and limiting their means of monetizing themselves. You don't need an investor to tell you that isn't in your long term interest.
To be fair, the fate of most of them is actually to fail. Hence this amplified effect of why VCs need such a standout massive return to make the fund model work.
That said, while it seems like it to us normal folk, in the grand scheme of VC, $25M is not really "a bunch of money". In 2021, a16z led or co-lead $3.2B [1] in funding rounds. I can't find a publicly available stat for how many rounds that entailed, but napkin math says 100–300. Of course the distribution is not linear, but if we take the 200 investments midpoint, that's a $16M check on average.
The additional $70M here follow-on is the real story. It looks like they are not giving it a label, though Crunchbase lists the $25M from a16z a year ago as Series B, so I'm inclined to call this new round the Series C.
> My new heuristic is that I avoid every single company that raises venture funding.
IMO tech, startups at least, are not long-minded like this in general. Companies regularly come and go in 6–24 months. Good luck convincing the current and next waves of CTOs of not using venture-backed tech [that saves them tons of time for great prices right now]. And then portcos also often make deals with other portcos... the cycle feeds itslef in more ways than one.
It's like trying to change the color of the ocean with one single cup of red dye. At the end of the day, this is just a rounding error.
But also, Big Tech will end up acquiring many of the standouts a la Firebase or Heroku. Resisting the model won't upend or stop it.
[1]: https://news.crunchbase.com/liquidity/under-the-hood-a-decad...
Pretty safe bet that they'll stay cheap for a long time because of that.
Because outside of tech innovation, cost is still a mayor selling point.
Consider this perspective: Users of Fly.io are essentially getting a loan from venture capitalists. However, just like any loan, it has to be repaid eventually. Thus, it's prudent not to take on an excessively large commitment.
in this model speculative capital is only required in the very early stages. a company either reaches a symbiotic relation with its clients or not. the burden is primarily on intrinsic aspects of what the startup venture delivers and how much this resonates in its sector.
the VC model is basically turbo-charging this process. though it is risk capital and not lending, it creates an implicit, arbitrarily sized liability that need not have much to do with the underlying value proposition. it removes the organic cashlow constraint via a faustian bargain.
the "beauty" of it is that you can't have both models in the same economy. the set of ideas that are ripe for exploration at an given era are what they are. if some people pursue them while being on steroids this means there is no room for people to explore them in a less toxic way
Nothing is forever. Five years is a long time. People need motivation. Enjoy and make use of things while they last. Nobody owes you anything.
But you have options.
You can have a business genius build a company that delivers great product forever without grinding employees into the dust, burning up founders and still outmaneuvering competitors.
You can have no product or a company going under because founders have stretched them thin, burned out and went broke ruining their lives pursuing a good idea that morphed into a sunken cost.
Or you can crazy pricing models justified by the desire to live like a VC LP while shipping a 3rd party API-dependent cute app. But without VC!
Or you can have a VC-subsidized business that ships a product at ridiculously low prices, locks you in for some 3-5 years and then goes ballistic and becomes unusable. But it’s been five years of a very good run, FIVE YEARS. And then the next will come to take their place.
Reading how five years is a short timeline for a low-cost-high-value product is wild. Now that’s some entitlement.
Throwing shade at cloudflare edge workers?
Out of curiosity, how _does_ fly.io see itself in relation to Heroku?
We're not Heroku. We _like_ how amazingly easy it is to get an app up and running on Heroku. But most of us were interested in flexing the underlying infrastructure to build new types of apps. So Heroku felt very constrained.
We see ourselves as, like, a good RPG. Quick to get going, rewarding and replayable when you go deep.
Their managed Postgres got an unfair advantage early on by being the default. With a new rails app you were just given a Heroku Postgres database. Heroku's Postgres offering was over 5 years ahead of RDS Postgres and a large part of the reason Amazon added support for Postgres for years they fought it, but eventually had to give into the constant customer request. Unfortunately that innovation of Postgres has stalled out a bit over time due to Salesforce starvation.
I do admit for what people think of as PaaS the Postgres revenue is significant, the part people would probably be surprised about is how much revenue the add-ons marketplace constitutes.
Interestingly New Relic got the same privileged experience in the early days and it resulted in the same type of revenue and mass adoption for New Relic.
EDIT: Oh, and they often add valuable info to what they posted, too.
The responses here do feel swarmed, but also informative, "honest"[1], and often interesting about tech/mindset/etc.
This post swayed my upcoming DO purchase, so.. it works a bit at least, hah.
[1]: As much as we can assume, of course.
What does this new round mean for Fly's long term independence?
We're to the point where we're making $10-25mm capital expenditures in one go. And we have to to optimize our own costs. Optimizing our own costs means long term independence.
What I think I've learned about VCs is: the trick to long term independence is to find investors who already value the sales mechanism. Bottoms up, dev focused "sales" is something investors like, now (this was not true in 2012).