Hard to confirm the article and the tweet. The author should provide the exact details and operations for someone to replicate and verify, or at least do the math.
Racking up a $2500/day bill is several of magnitudes off - there’s nothing in the article to help put the workload in context.
I have zero interest in mongodb, as a stock or a technology, but this article doesn’t help to actually determine anything about cloud costs. (Lack of data points, irreproducible method and conclusion)
Depending on exact specifications that is like $10k of hardware, so 4 days of rent pays the machine in full.
So in this service the actual cost of the hardware is a rounding error. It also needs power, cooling, a roof and some maintenance crew, but that still doesn't come close to the rental price. So what are customers actually paying for?
Was your code not written with cost control in mind because there was a finite hardware limit you were running yourselves up against? Otherwise, I am baffled at how the difference could be so great between the three.
That makes it even more sus. Can you describe your workload? Unless you were doing something extremely heavy-duty for a long time, I just don't see this being at all factual.
+1 - the post speaks about 100 times difference between VM and Atlas setup.
At a glance, Atlas cloud has typical ~ 20% premium if we compare their pricing to AWS/Azure VM setup. 20% is not x100.
So IMHO and AFAIK we're speaking here about serious misconfiguration.
All of these posts never seem to talk about the operational overheard they're saving by using the cloud. 500k/year is a couple of engineers. And IME, at scale, Elasticsearch is a huge pain to self manage.
Self hosting has become very easy these days. Elasticsearch, specifically was a pain to self manage years ago. With modern versions, it has become incredibly stable, defaults have made more sense, and you now have modern tools like Kubernetes to make deploying/scaling easier too. The work required to self-host has improved in the past decade, and ppl who "default to the cloud" ignore this.
I'm saying this as someone who manages a 50 TB cluster for a startup (which I know isn't a HUGE amount, but definitely not small either)
cannot upvote this enough. DHH perfectly explained what is wrong with cloud. Constant gaslighting from cloud marketing about: capital costs, human cost, hardware cost, etc., while ignoring all negatives of cloud.
Cloud computing is pure scam. you rent 1 vCPU and people often think this is as good as a real 1 hardware core, while in reality that physical core is being sold twice/three times to different customer. Your 1 vCPU is maybe 0.5-0.75 of a real hardware CPU, perhaps even less, depending on how greedy cloud provider is.
It is almost like instead of driving your own car and making stable car payment - you decide to exclusively use Uber/Lyft to go around. Sounds good if you are in NYC/SF, but not so much outside of these perfect use cases. Also doesn't make sense if your primary job is pizza delivery, all your margins from delivering pizzas will just transfer to ride hailing company.
The most valuable information for the average HN reader, especially younger colleagues who have only been in the workforce for the last decade, is that the ~70% gross margins of the hyperscalers, and by extension the $300k+ salaries for early career software developers, has been fueled by a temporary era of low central bank interest rates and an excess of VC capital. This era is almost certainly coming to a close. We should all be planning accordingly with respect to personal finance and especially be smart about getting locked into high mortgage payments.
Always good advice to budget and prepare for the worst, but wow this is a big stretch from the article.
Of the large salary companies - Amazon, Google, and Microsoft are in the cloud space. Google doesn't make most of its profits from cloud. Netflix and Facebook don't sell these services at all. Dropbox and Digital Ocean (and sort of Cloudflare) are also in the cloud space and aren't known for such salaries.
It's possible everything is in a bubble from the low interest rates. My prediction is Adtech is far more likely to be in a bubble than cloud computing. The productivity gains and small number of people who can fill the role for cloud are going keep it dominate probably for a long time even in a recession.
> My prediction is Adtech is far more likely to be in a bubble than cloud computing.
It's always so hard to nail down AdTech, to the outsider it seems like a world filled with bots and bad data and fueled by FOMO. Probably a good chunk of Google's revenue comes from companies that don't really need to advertise, but do so anyways so they won't loose their "competitive edge"
I only ask because it is has been around for a few decades now, has steadily been on an incline while old advertising methods has slowly been on a decline. Those trends are more likely here to stay…
The pandemic fueled weird economics everywhere but I don’t think we will see falls below pre-COVID
Startups have to show growth and/or revenue. They dump most of their funding into advertising. Many are spending more than the lifetime value of their customers (if that can even be calculated at their stage at all).
Interest rates go up -> VC Money goes down -> Advertising budgets go down
Online advertising is "theoretically" better than other advertising. You can track exactly how it converts to sales or not. It is still the best way to advertise in my opinion so will beat out any other method for sure.
Interest rate increases are causing money to move around for sure. I’m just not convinced startups are the primary buyer of these online ads. Hell I would barely even consider them a drop in the bucket. Im sure basically every Fortune 500 and brick and mortar store have hefty online ad spends now a days. I suppose we will see soon enough
Yeah, this isn't the dotcom era when internet ads was mainly startups buying ads on Yahoo thereby inflating their own value and Yahoo's value. The standard online advertisers are the standard advertisers these days (apart from a few smaller islands like podcasts or integrated sponsors on youtube), so it'd be a bigger recession to impact adtech, and the impact there is more likely to be in line with the impact to the rest of the economy than disproportionate popping.
Last year I worked as a dev at an ad tech firm that apparently had a money printer. The amount of waste, particularly in their cloud architecture, was staggering. Prestigious office in a very expensive city. I noticed that they hired from big-name companies known for being staggeringly wasteful like Uber.
Just one company, sure, but it did make me wonder about adtech in general.
This is just incredibly frustrating and disheartening to read. How is one supposed to plan for a future in this career when your lifestyle can just disappear overnight?
> This is just incredibly frustrating and disheartening to read. How is one supposed to plan for a future in this career when your lifestyle can just disappear overnight?
Plan like everyone else in your country who has to deal with this uncertainty on significantly lower salaries.
Budget. Save. Trim expenditure.
And enjoy your career because you chose it as a career, not because it allowed you a lifestyle.
I want to highlight this. I do probably a lot more "prep" than the average HNer is, but it's not out of panic. It's just a cold engineering assessment of costs & benefits. A lot of prep is just moving forward purchases and storing easily stored things, so the real cost is minimal if you have the space anyhow. And not spending all your money is a good plan anyhow; it has been for a long time and will continue to be. Even if you knew 100% the economy is fine and will be fine for the rest of your life, there's still good reason to prepare because you may have problems, health problems, natural disasters, maybe just the desire to take three months off work to prevent burnout, any number of reasons you should try very hard not to live paycheck to paycheck no matter how little you make.
And then, when you are prepared, you can be more calm about a lot more things. Yes, it's possible for the Zombie Apocalypse to hit and make a mockery of all my prep... but because I'm more-or-less ready for more realistic scenarios, I'm also more chill about the possibility of them happening.
And they may not, in which case frankly I won't have wasted many resources anyhow, any more than it's a waste to buy an insurance policy that you never get to collect on.
What we do know is that the market thinks the career is risky, charging a substantial risk premium. The market isn't infallible, but tends to be right more often than not.
But that also means that you need not be too alarmed. Even just one year of making the aforementioned $300,000 allows you several years of not working at all before you are making less, on average, than a typical low risk job. As you've already priced in the risk, you'll be fine.
And if the market is wrong then you've gained a nice reward.
What makes you interpret it as a risk premium? My sense is it's mostly driven by demand growing ahead of supply, and barriers existing to the supply expanding (which is partly because of the value of experience, which takes time to build). I.e. I'd say it's a scarcity thing more than a risk thing.
> My sense is it's mostly driven by demand growing ahead of supply
That, from an employer point of view, is certainly what makes the difference between offering a high risk premium and deciding that the work isn't worth doing.
> which is partly because of the value of experience, which takes time to build
All jobs value from experience. The question the worker has to ask himself is: Where am I going to allocate my time to gain experience? One can't do everything. They have to make a choice. Tech is a risky place to allocate your time, but compensates for that by offering a premium to compel you in that direction.
Tech is risky because it is all about exploring unknowns. You get periods of fruit, but you are also likely to get periods of dead ends, and when those dead ends start to accumulate people start to back away. That is quite unlike, say, road maintenance where there is a strong belief that roads will be around for the lifetime of one's career. That isn't a guarantee, but it is lower risk. Tech is known to disappear.
Tech has crashed many, many times before after too many dead ends. It feels to me like we're in a dead end period. When was the last time you were excited about new tech? The WFH period during the pandemic brought some small excitement in tech emergence, but it seems we're trying as hard as possible to back away from that.
But who knows? This time might be different. And if that's the case you've got your risk premium as a reward for taking the risk.
I mean, I love software development, spend a lot of my own time on it outside of work. In fact I've been writing code since I was 11 or 12, close to 20 years now, professionally for 8 years, but I'm also very intelligent and like my lifestyle, so if that evaporated the chance that I'd continue to do professional software development, especially since many of the things we deal with day to day for pay are arguably the worst parts of software development, would be very low.
This whole "its your calling" thing is how the abuse and low pay in adjacent industries, like game development, are maintained, and frankly it's BS.
I'd go back to school and into something that will maintain my lifestyle.
One plans by charging a high rate during the good times, to cushion their lifestyle during the bad times. That is why we see those $300k salaries. Given the risk, nobody wants to do the job for less.
The risk of what? Having to make ends meet on a $150,000 one?
You're not exactly risking your life and limb by taking a job at a cloud firm. Demand for software isn't going anywhere, and no hiring manager at any future job will give two cares as to whether or not your previous employer's revenues were inflated or not.
Let's say your options are working as a developer for a random Fart App startup or a government job sweeping floors that has been around for centuries. Which are you going to choose? If they both offer $50,000 per year, the choice is clear: The government job. It is the one that is almost certain to have more longevity.
But if the Fart App startup increases their offer to $300,000, well, now it's a harder choice. The startup still isn't likely to last, but with $300,000 in your pocket you're not so concerned about it disappearing into the void on a whim. And if on the unheard of chance they do make it as a lasting company that spans centuries then you're in a really great position.
> Demand for software isn't going anywhere
Thing is, it does crater approximately every 10 years, as we've observed since software first became a thing. It is more entrenched now than it ever was, so there is some reason to think this time is different, but the market is pricing in the risk regardless. The market can be wrong, but the market likes to be prepared.
> Thing is, it does disappear approximately every 10 years, as we've observed since software first became a thing.
It slows down, along with the rest of the economy during recessions, because it, like everything else, is driven by the same market forces.
My point is that in the average case, there is ~zero opportunity cost or damage to your career, because you worked at MS Azure, instead of... Somewhere else.
You cite fartapp startups that run on rainbows and free VC money, but cloud providers are about as dissimilar from the fartapp as you can get, in terms of their economics and prospects.
You may be bearish on them, that's fine, I can't predict the future, but I see no reason to believe that you're going to have a better life working at literally any other random software firm, during an economic contraction. There's always a few workplaces that tend to be counter-cyclical, but they are, by definition, a minority.
... Also, many government jobs are a political football, that are only as secure as the next election. The best part is that after you worked one, and had half your compensation deferred to retirement, some politician will come along, and campaign on reneging on your pay, after you've given your thirty years of service.
> It slows down, along with the rest of the economy during recessions, because it, like everything else, is driven by the same market forces.
Tech has been the darling of late exactly because it hasn't followed the general economy. In 2008, when virtually every other industry was about to sound death knells, developers were making bank producing apps. Similar story through the pandemic, when people were locked out of work during lockdowns tech was making a killing selling them things to do with their free time. Not having a good reckoning like everyone else only makes it appear more risky as time marches forward as it is assumed its day will come.
> but cloud providers are about as dissimilar from the fartapp as you can get.
I can't imagine Firebase, which is supposedly the fourth largest cloud provider according to a recent article here, is used for anything much beyond fart apps. As the fart apps die, where does that leave their business model?
Providing tooling to the gold prospectors is a good business to be in... Until people stop prospecting for gold. Firebase is Google at the end of the day, but if that division dies, that's still significant.
[I work at Firebase, but opinions are my own]
I am happy to report that Firebase is actually used by companies of all sizes and types, including Fortune 1000.
Fortune 1000 and fart apps aren't mutually exclusive, to be fair. Look at how many fart apps Google themselves create and then throw away soon after[1]. It is one thing to toss anything you can come up with into the wind to see what sticks during the gold rush, but that doesn't last forever.
The software industry isn't going to disappear overnight. It didn't disappear in 2001 or in 2008, despite all the concerned emails I read from my relatives.
That said it's always a good idea in general to live below your means, and to be specific, not get the biggest mortgage your lender thinks you can afford.
Another unhinged HN comment ripped straight from ZeroHedge. Loss making venture funded startups are a small part of cloud revenues. The overwhelming majority of cloud revenue and gross profit comes from stable and profitable enterprise customers who have a 15 year plan to exit the majority of their on prem footprint, and interest rates have little bearing on these plans.
As someone who doesn't understand finance or economics very well, both the parent comment and yours were interesting, but I don't know why you have to use such an adversarial tone. Your point seems like it stands on its own, no need to go into attack mode.
There's no charitable way to interpret "unhinged" and "ripped straight from" as anything other than aggressive. Note that I upvoted your original comment despite the hostility because I thought the substance was interesting.
Bubbles by definition don’t exit unless the majority of market players temporarily believe that the current trajectory is the new normal and not an anomaly.
Buying excessive housing is a huge waste, but locking in a mortgage can be a major win. Moderate inflation quickly makes high mortgage payments irrelevant, 6% inflation is a 44% discount over a decade. On top of this if you lock in a fixed mortgage today your rates can only drop after refinancing.
In the end major financial decisions have many tradeoffs, but the micro optimizations are less important than having a high savings rate. If you make 150+k and make reasonable choices more money hits serious diminishing returns.
If you believe we're in a high-inflation economy, your recommendation is incorrect, as the real cost of mortgage payments very quickly gets eaten by inflation.
If you believe we're in a low-inflation economy, you may want to have an argument on that subject with folks that don't. :)
Central bank interest rates don't really matter much for investors or borrowers. Typically investors would use the 10y treasury yield as an approximate risk free rate and apply a risk premium to that calculate their required ROI. Central bank interest rates only matter to banks.
For the last decade 10y treasury yields have ranged from around 1.5% - 3%. And as recently as 2018 they were around 2.8% - 3.1%. Today the 10y treasury yields around 4.2% which is meaningfully higher, but I think people are vastly over stating the impact of this. It's also quite possible the 10y yield is currently peaking and will move down over the next year. The long-end of the yield curve typical moves with inflation + economic growth expectations. I'm not committed to any side of this argument but I will say there is good reason to believe inflation pressures are likely to trend lower globally over the next several decades because of global demographic trends and slower rates of productivity growth. This is why interest rates probably haven't been as recklessly low over the last decade as many people in 2022 like to suggest - they've needed to be as low as they have been to prevent deflation and economic contraction.
Similarly to investors, companies don't borrow at the Fed funds rate, they borrow at a premium to what the treasury market yields. So while corporates will need to borrow at a slightly higher rates to fund growth going forward, I again think people are vastly over stating the impact of 1-2% higher borrowing costs.
That said, it's totally reasonable to expect margins to fall as industries mature - especially when there is a lot of competition in that industry. I do believe cloud margins will contract slightly over the next decade, but I'm not convinced pricing is in a bubble either. Firstly, investors (collectively) are far from stupid. If you think the market is wrong and that you're the only one to see these risks then you probably haven't been humbled by the wisdom of markets enough. Cloud company will be growing for many years to come and valuations today imo do already allow for some margin contraction. But we've seen this all before... As OP mentioned SSL certs used to be ridiculously expensive, now you can get them for free. I personally remember having to pay several hundred dollars a year for a VPS, dedicated IP and SSL cert back in the mid 2000s. Hell, it cost about a $100 a year just for a domain + decent webhosting back then. I mean if pricing is in a bubble today then you should have seen it a decade ago...
Finally, when it comes to mortgages obviously it's important to ensure you can afford your repayments even if the unexpected happens, but as a long-term borrower inflation is your friend. Even if real wage growth for software engineers decline the likelihood of significant nominal wages declines for software engineers is practically zero. As an example while the real wages of cleaners and factory workers may have declined in real terms over the last few decades, in nominal terms they're making more than ever, and that's really all that matters in terms of servicing debts.
Having used both, I personally would never choose mongo over Postgres, there are just too many conveniences Postgres offers that are a pain with Mongo.
Oracle and SQL Server are also embedded databases that happen to have a networking layer on top. And, given the context, a networking layer is likely what you're building anyway.
Mongo will scale out horizontally better than Postgres. But most people don’t need that, and there are other alternatives to Mongo that will also scale in this way.
In the vast majority of cases these days I'd rather have a dumb document database to shove my data into rather than fretting over relationships between tables with a RMDBS where complexity grows at best, quadratically.
I don't use mongodb in particular but I never want to go back to worrying about what a SQL application instance is up to, either.
I found having big doc store tables in Postgres became harder to manage as time went by from a people perspective.
E.g. around the 300GB mark I had to constantly remind myself and others to be careful of it because it might have normal relational columns you think are fine to query, but it also has a fat json payload on each row and if you're careless the query planner would basically do a table scan for things you wouldn't expect.
This isn't so much a problem with json in an rdbms but more the false sense of query performance on non-json columns in the same table...after moving it to mongo never had the same problems because people knew not to query it in an unsupported manner.
I have always kind of disliked both cloud and containers. Primarily, they just add more complexity and overhead into the computing landscape. I've thought this to be bad both in material resources and energy consumption, but humans have a high time preference, and therefore they choose to spend both materials and resources to get a little more time back. This is much the same reason that hipster languages and frameworks are in a constant state of churn. I myself am guilty of perpetuating these things that I dislike from an analytical perspective where I say they are wasteful, because I often just want to get something done so I spin up a cloudy service of some kind, deploy some code, close a ticket and walk away knowing I didn't go about things the "correct" way.
For all of this to stop, not only does the bubble need to burst but people would need to be far more intentional about what resources they use where and when. People would also need to have longer software lifecycles, more QA, and so on. This would only happen in an environment with far higher interest rates and far lower monetary velocity. In essence, there'd have to be a lower tolerance of risk to stop the constant churn of hardware, software, and development methodologies on both. There are tradeoffs to such changes.
> Primarily, they just add more complexity and overhead into the computing landscape.
No, they don't 'just add more complexity and overhead'. They are tools. Properly utilized, they are great.
There's a reason why you are (probably!) not running a power station and home and have, instead, hooked to a power utility company. They have the expertise and can amortize the costs among many customers.
Similarly, containers actually brought efficiency – they replaced many(obviously, not all) of the use-cases that usually required virtual machines. They also simplified some very difficult deployments.
The problem arises when you are doing things for the wrong reasons. For example, using 'microservices' just so you can try to avoid teams from having to talk to one another and end up shipping your org chart.
I do agree with a more generalized statement: more often than not, monoliths are what you need.
I worked in a small team that brought containers into a co-lo / bare metal / centos / rpm / puppet shop. There was extra shit for developers to learn but I don't think it was waste. Not just because "everything is containers" nowadays, but because until they tried to build a container with their code actually working in it, most of them had little idea about how their shit worked. One example from the field: a VM got rebuilt and the app installed from scratch, but it wouldn't work. I hopped into the box and farted around with logs, strace, ldd, rpm commands and eventually realised that they introduced a dependence on a package that used to be installed on the system but was no longer there. It kept working on all existing machines but failed in a "build VM from scratch and run Puppet". Not only did that mistake make it out to production, but at that time few of the developers would have had a clue how to fix it. After the containerization of our services, this was absolutely not the case. (And honestly, we just conceptually replaced RPMs with containers, standardized all of our servers to the same build and forced developers to be aware of how their code actually worked).
Be it VC money, or just general apathy, it seems cloud costs are rarely a priority. My biggest gripe is when see that justification that goes like, "developer's time is $100/hr and better spent elsewhere." Okay, I agree, doing something like self-hosting everything to reduce costs would be a terrible idea, but there's a fine balance.
In 2019, Lyft spent $300M on AWS [1]. Sure, it's a big service, but are you seriously telling my that it's necessary to spend $300M in a single year?
I think the meme that developer's time spent on cost control is useless needs to die. Engineers should be encouraged to control costs so that they can take better decisions early on. They should be nudged to pick more cost-efficient choices. This won't only reduce costs, the chances are companies will also end up having dramatically simpler architecture.
It's absolutely a balance, but I've seen it the other way too. Devs spending weeks to make something more efficient when compute time costs cents.
That has a minimum cost of their hourly rate. The maximum cost is the cost of not doing what's most valuable to the company.
> In 2019, Lyft spent $300M on AWS [1]. Sure, it's a big service, but are you seriously telling my that it's necessary to spend $300M in a single year?
They didn't spend that in a single year.
It also depends what they were doing, I was recently very shocked by the data costs of a major hardware store in the US but after breaking it down it made more sense. It all depends what they get for that $300M over 3 years.
For better or worse a lot of folks are allergic to this. A framing that I’ve seen is to just understand the cost drivers from a technical perspective, for example large materialized views in BigQuery consume a lot of CPU. That’s technically interesting and it’s the reason they’re expensive.
Literally from the first line of the article you linked:
> Lyft has signed up to pay cloud market leader Amazon Web Services at least $80 million per year for the next three years, totaling at least $300 million.
You can launch a massive project, hire dozens of people, buy server and colo space across the globe, rebuild their infra from scratch, and get costs down to – what – $60 million a year? Lyft is closing in on $4 billion in annual revenue. A few million in savings is meaningless, especially when balanced with the massive risks of such a project.
I think that an uber are more like a public service now, lol. So whoever invests in them long term doing the public service and not expecting real good returns.
I've managed down a cloud budget in that range. Trust me you can get there surprisingly easily if you want resiliency, security, and uptime on thousands of services.
The weird thing is, the nature of their business does not imply any significant computing needs. As a rough estimate they have served on the order of 1 billion customers, so that suggest they need 1 billion, multiplied by some factor of cruft, database rows. Depending on the cruft factor that should be somewhere between easily fitting on a single disk, up to a few of them. Add in redundancy and backup and we have still not filled a rack cabinet.
Maybe all those computers are doing something that someone thought was important, but it is not serving the core business.
When you have big contracts like that, you get significantly discounted rates. And when you need large amounts of compute like that, you do need a whole bunch of staff to manage it and make it efficient and start reproducing / maintaining similar software created by the cloud providers. Let's say AWS makes a %50 profit margin over the raw cost of the hardware than you doing it yourself in DC, and each engineer is $400k/yr. You need to keep under 100 engineers to just start beating AWS, and your not in the business of running cloud infra.
Also when you run the DCs yourself, usually it's much slower for other engineers using your DC infra than using AWS or similar. It gets complicated!
Execs will either turn a blind eye or simply won't understand that it's a problem, until a massive purchase order lands in accounting and gets escalated to the highest levels.
Now it's suddenly a problem and it is up to the operations/devops/SRE/name of the month/SecFinDeVOps to figure out how to reduce costs. Which is completely backwards as they are probably not the group who requested the resources in the first place.
I find it interesting that there are two MongoDb Directors of Developer Relations in this thread - one former, one current - either attempting to replicate the suggested costs to see if the author's faking it, or outright blaming him he is.
Dude's got a screenshot. I doubt he faked it for the attention - he probably actually got that bill.
He feels it's expensive and he should pay less for the value he's getting considering his alternatives (Azure VMs in a 1/3 of the price).
Perhaps refer to the actual price of a billed hour there? That feels like the core theme here.
Editing a screenshot to push a narrative is pretty easy. I'm no longer with MongoDB, although still a big fan of the product, but I'm more curious as to how you could rack up a $2,500 bill just trying the product out, especially the serverless offering, coupled that with the guy being a "twitter investor", so that kicked off my BS detector.
MongoDB Atlas runs on the big 3 clouds, and of course you're going to pay more for a fully managed service, as opposed to hosting it yourself directly on the big 3, but racking up $2,500 in serverless compute charges in 24 hours seems very very hard to do, and especially by accident.
So I read your second sentence first for some reason, which seemed plausible and I wanted to respond nicely.
And then I read your first sentence - why is it always the assumption that people are going the hyperbole route? This is a well-upvoted link, with enough comments to merit a good discussion, and what appears to be a nicely-thought out post after the link.
Also, I've been following Snir for quite a while - dude wrote a massive Data Engineering roadmap which is heavily used by people learning how to ETL and has a very interesting investing blog. He's an engineer turned investor, and looks at the market from this perspective.
> racking up $2,500 in serverless compute charges in 24 hours seems very very hard to do
TBH I know how to spend that much and more with a single query (and ~30 minutes of compute time) in BigQuery. Serverless BI analytics workloads over large public datasets can do amazing things... and are also exactly the sorts of things a naive user would use to test + benchmark the capabilities of serverless data-warehousing systems.
Everyone always lauds AWS for not raising prices but fails to realize that hardware has continued to get cheaper, faster, and easier. Storage especially has taken such huge leaps that I imagine all of the big cloud companies must be making money hand over fist at this point.
I think this bubble represents a huge amount of opportunities. Sure the crazy (70%?) gross margins for cloud is insane, but this means there are more cloud companies to be made -- the market isn't saturated yet.
With the right offering and the right software you can build something just 10% as good as some of the hyperscalers and have an amazing company.
Competing on price is not a sin, and it's what I'm working on/betting on.
That said, I think the real big threat to the hyperscalers is cheap chips and more people being able to treat datacenters like parking lots (nice ways to use capital to generate cashflow) -- the commoditization of data center operations.
This also assumes that the open source model of self-hosting is sustainable. We used to have extraordinarily expensive enterprise options for software, then there was an open source boom which eliminated marginal cost with the hope of building the software collaboratively. In practice, many of these open-source projects such as MongoDB simply spent VC money to make the software they were giving away.
I work for Vantage and I do think that startups are starting to care about cloud costs, but not necessarily for the cost itself.
The thing to understand early these days is COGS and whether the business can scale its service with good margins. That is a significant change of concerns for investors from the past years where growth was the only thing that mattered.
The other thing that I think is a little off in this post is the idea that startups not caring about costs are a big driver of hyperscaler revenue. I suppose that may have been true a few years ago but AWS, Azure, and to some extent GCP are having a lot of success moving legacy on-prem deployments to the cloud. It's not uncommon to encounter 50 year old enterprises with a billion dollars of cloud spend. One example of this is the dedicated SAP instances that AWS has, some of which cost $100K per month.
This is more SaaS costs are in a bubble, not core-infra (Compute & Storage) offered by Azure/Amazon/GCP.
Some of the cloud provider compute (VMs) and storage (disk/object store) has very low margin. With it highly unlikely a company could provide equivalent offering in-house.
Many truths are admitted in a downturn. The cost of the cloud is that it clogged young engineers' brains with knowledge of multiple proprietary systems that are incompatible with each other, not open source, and bound to become useless knowledge as soon as the cloud companies ditch their offerings to chase the next profitable thing. That's why open systems and standards, despite their imperfection, are so much more valuable
I've always thought cloud costs have been excessive profit centers, and this goes back to before it was called cloud with "ASPs" in regulated industries product offerings.
Early days I remember looking at Cloud vendors that were offering FISMA virtualization capabilities and I did an analysis of alternatives around running a set amount of CPUs, RAM, i/o and disk to support VMs. Terremark was the first offering that could meet our reqs, at around 10k a month per for a server's equivalent capability.
This put in my mind from day one that I was trading off High Capex with low Opex, for No Capex and High Opex. Given our trades, it was clear that high Capex was preferable to high Opex, as our high capex would have been equal to one month's opex.
When we later on saw more mainstream vendors like Amazon move into the space, their offerings were also coming with insane overhead, $2000 a month automatic surcharges for a FISMA regulated environment.
This is already happening, check out https://text-generator.io around 10x cheaper than OpenAI for text-generation also has speech-to-text 8x cheaper than Google cloud.
New era of cost-cutting, running systems yourself and distributed computing will bring far cheaper prices soon. If we do well energy and computation/intelligence will be trending toward 0 generally speaking
I read both Snir's port and the HN thread, but I'd like to see more details about his workload and setup. $2500/day is a lot. Of course, it's possible to reach such numbers, but IMHO and AFAIK:
1) it's not the same setup as " $800 Azure VMs deployment " - I promise you
2) the post is useless w/o detailed bill
At a glance, Atlas pricing includes the typical ~ 20% premium if we compare it with AWS (Azure/GCP) VM setup. 20% is very far from *100 price difference as per article.
102 comments
[ 3.0 ms ] story [ 190 ms ] threadThe mongodb pricing page says one can setup a M10 instance for $57/month. [https://www.mongodb.com/pricing]
Racking up a $2500/day bill is several of magnitudes off - there’s nothing in the article to help put the workload in context.
I have zero interest in mongodb, as a stock or a technology, but this article doesn’t help to actually determine anything about cloud costs. (Lack of data points, irreproducible method and conclusion)
That is priced quite clearly on the bottom page at 102.49 USD an hour.
So in this service the actual cost of the hardware is a rounding error. It also needs power, cooling, a roof and some maintenance crew, but that still doesn't come close to the rental price. So what are customers actually paying for?
I used the "Serverless" product, not the "Dedicated" product. That's the gap you see here.
But your point is important, I'll add it to the post. Thank you.
see also https://news.ycombinator.com/item?id=33257204
I'm saying this as someone who manages a 50 TB cluster for a startup (which I know isn't a HUGE amount, but definitely not small either)
Cloud computing is pure scam. you rent 1 vCPU and people often think this is as good as a real 1 hardware core, while in reality that physical core is being sold twice/three times to different customer. Your 1 vCPU is maybe 0.5-0.75 of a real hardware CPU, perhaps even less, depending on how greedy cloud provider is.
It is almost like instead of driving your own car and making stable car payment - you decide to exclusively use Uber/Lyft to go around. Sounds good if you are in NYC/SF, but not so much outside of these perfect use cases. Also doesn't make sense if your primary job is pizza delivery, all your margins from delivering pizzas will just transfer to ride hailing company.
Of the large salary companies - Amazon, Google, and Microsoft are in the cloud space. Google doesn't make most of its profits from cloud. Netflix and Facebook don't sell these services at all. Dropbox and Digital Ocean (and sort of Cloudflare) are also in the cloud space and aren't known for such salaries.
It's possible everything is in a bubble from the low interest rates. My prediction is Adtech is far more likely to be in a bubble than cloud computing. The productivity gains and small number of people who can fill the role for cloud are going keep it dominate probably for a long time even in a recession.
It's always so hard to nail down AdTech, to the outsider it seems like a world filled with bots and bad data and fueled by FOMO. Probably a good chunk of Google's revenue comes from companies that don't really need to advertise, but do so anyways so they won't loose their "competitive edge"
I only ask because it is has been around for a few decades now, has steadily been on an incline while old advertising methods has slowly been on a decline. Those trends are more likely here to stay…
The pandemic fueled weird economics everywhere but I don’t think we will see falls below pre-COVID
Interest rates go up -> VC Money goes down -> Advertising budgets go down
Online advertising is "theoretically" better than other advertising. You can track exactly how it converts to sales or not. It is still the best way to advertise in my opinion so will beat out any other method for sure.
Just one company, sure, but it did make me wonder about adtech in general.
The margins are so insane that it just doesn’t matter. They are trading engineering time for time to market.
Plan like everyone else in your country who has to deal with this uncertainty on significantly lower salaries.
Budget. Save. Trim expenditure. And enjoy your career because you chose it as a career, not because it allowed you a lifestyle.
And then, when you are prepared, you can be more calm about a lot more things. Yes, it's possible for the Zombie Apocalypse to hit and make a mockery of all my prep... but because I'm more-or-less ready for more realistic scenarios, I'm also more chill about the possibility of them happening.
And they may not, in which case frankly I won't have wasted many resources anyhow, any more than it's a waste to buy an insurance policy that you never get to collect on.
But that also means that you need not be too alarmed. Even just one year of making the aforementioned $300,000 allows you several years of not working at all before you are making less, on average, than a typical low risk job. As you've already priced in the risk, you'll be fine.
And if the market is wrong then you've gained a nice reward.
That, from an employer point of view, is certainly what makes the difference between offering a high risk premium and deciding that the work isn't worth doing.
> which is partly because of the value of experience, which takes time to build
All jobs value from experience. The question the worker has to ask himself is: Where am I going to allocate my time to gain experience? One can't do everything. They have to make a choice. Tech is a risky place to allocate your time, but compensates for that by offering a premium to compel you in that direction.
Tech is risky because it is all about exploring unknowns. You get periods of fruit, but you are also likely to get periods of dead ends, and when those dead ends start to accumulate people start to back away. That is quite unlike, say, road maintenance where there is a strong belief that roads will be around for the lifetime of one's career. That isn't a guarantee, but it is lower risk. Tech is known to disappear.
Tech has crashed many, many times before after too many dead ends. It feels to me like we're in a dead end period. When was the last time you were excited about new tech? The WFH period during the pandemic brought some small excitement in tech emergence, but it seems we're trying as hard as possible to back away from that.
But who knows? This time might be different. And if that's the case you've got your risk premium as a reward for taking the risk.
This whole "its your calling" thing is how the abuse and low pay in adjacent industries, like game development, are maintained, and frankly it's BS.
I'd go back to school and into something that will maintain my lifestyle.
You're not exactly risking your life and limb by taking a job at a cloud firm. Demand for software isn't going anywhere, and no hiring manager at any future job will give two cares as to whether or not your previous employer's revenues were inflated or not.
You are risking opportunity elsewhere.
Let's say your options are working as a developer for a random Fart App startup or a government job sweeping floors that has been around for centuries. Which are you going to choose? If they both offer $50,000 per year, the choice is clear: The government job. It is the one that is almost certain to have more longevity.
But if the Fart App startup increases their offer to $300,000, well, now it's a harder choice. The startup still isn't likely to last, but with $300,000 in your pocket you're not so concerned about it disappearing into the void on a whim. And if on the unheard of chance they do make it as a lasting company that spans centuries then you're in a really great position.
> Demand for software isn't going anywhere
Thing is, it does crater approximately every 10 years, as we've observed since software first became a thing. It is more entrenched now than it ever was, so there is some reason to think this time is different, but the market is pricing in the risk regardless. The market can be wrong, but the market likes to be prepared.
It slows down, along with the rest of the economy during recessions, because it, like everything else, is driven by the same market forces.
My point is that in the average case, there is ~zero opportunity cost or damage to your career, because you worked at MS Azure, instead of... Somewhere else.
You cite fartapp startups that run on rainbows and free VC money, but cloud providers are about as dissimilar from the fartapp as you can get, in terms of their economics and prospects.
You may be bearish on them, that's fine, I can't predict the future, but I see no reason to believe that you're going to have a better life working at literally any other random software firm, during an economic contraction. There's always a few workplaces that tend to be counter-cyclical, but they are, by definition, a minority.
... Also, many government jobs are a political football, that are only as secure as the next election. The best part is that after you worked one, and had half your compensation deferred to retirement, some politician will come along, and campaign on reneging on your pay, after you've given your thirty years of service.
Tech has been the darling of late exactly because it hasn't followed the general economy. In 2008, when virtually every other industry was about to sound death knells, developers were making bank producing apps. Similar story through the pandemic, when people were locked out of work during lockdowns tech was making a killing selling them things to do with their free time. Not having a good reckoning like everyone else only makes it appear more risky as time marches forward as it is assumed its day will come.
> but cloud providers are about as dissimilar from the fartapp as you can get.
I can't imagine Firebase, which is supposedly the fourth largest cloud provider according to a recent article here, is used for anything much beyond fart apps. As the fart apps die, where does that leave their business model?
Providing tooling to the gold prospectors is a good business to be in... Until people stop prospecting for gold. Firebase is Google at the end of the day, but if that division dies, that's still significant.
I appreciate the concern though :).
[1] https://killedbygoogle.com
That said it's always a good idea in general to live below your means, and to be specific, not get the biggest mortgage your lender thinks you can afford.
In the end major financial decisions have many tradeoffs, but the micro optimizations are less important than having a high savings rate. If you make 150+k and make reasonable choices more money hits serious diminishing returns.
If you believe we're in a low-inflation economy, you may want to have an argument on that subject with folks that don't. :)
Central bank interest rates don't really matter much for investors or borrowers. Typically investors would use the 10y treasury yield as an approximate risk free rate and apply a risk premium to that calculate their required ROI. Central bank interest rates only matter to banks.
For the last decade 10y treasury yields have ranged from around 1.5% - 3%. And as recently as 2018 they were around 2.8% - 3.1%. Today the 10y treasury yields around 4.2% which is meaningfully higher, but I think people are vastly over stating the impact of this. It's also quite possible the 10y yield is currently peaking and will move down over the next year. The long-end of the yield curve typical moves with inflation + economic growth expectations. I'm not committed to any side of this argument but I will say there is good reason to believe inflation pressures are likely to trend lower globally over the next several decades because of global demographic trends and slower rates of productivity growth. This is why interest rates probably haven't been as recklessly low over the last decade as many people in 2022 like to suggest - they've needed to be as low as they have been to prevent deflation and economic contraction.
Similarly to investors, companies don't borrow at the Fed funds rate, they borrow at a premium to what the treasury market yields. So while corporates will need to borrow at a slightly higher rates to fund growth going forward, I again think people are vastly over stating the impact of 1-2% higher borrowing costs.
That said, it's totally reasonable to expect margins to fall as industries mature - especially when there is a lot of competition in that industry. I do believe cloud margins will contract slightly over the next decade, but I'm not convinced pricing is in a bubble either. Firstly, investors (collectively) are far from stupid. If you think the market is wrong and that you're the only one to see these risks then you probably haven't been humbled by the wisdom of markets enough. Cloud company will be growing for many years to come and valuations today imo do already allow for some margin contraction. But we've seen this all before... As OP mentioned SSL certs used to be ridiculously expensive, now you can get them for free. I personally remember having to pay several hundred dollars a year for a VPS, dedicated IP and SSL cert back in the mid 2000s. Hell, it cost about a $100 a year just for a domain + decent webhosting back then. I mean if pricing is in a bubble today then you should have seen it a decade ago...
Finally, when it comes to mortgages obviously it's important to ensure you can afford your repayments even if the unexpected happens, but as a long-term borrower inflation is your friend. Even if real wage growth for software engineers decline the likelihood of significant nominal wages declines for software engineers is practically zero. As an example while the real wages of cleaners and factory workers may have declined in real terms over the last few decades, in nominal terms they're making more than ever, and that's really all that matters in terms of servicing debts.
Is this just legacy applications that are stuck with Mongo? Certainly nobody is starting new projects on Mongo?
Serious question.
Having used both, I personally would never choose mongo over Postgres, there are just too many conveniences Postgres offers that are a pain with Mongo.
Serious question.
because that is how they compare feature wise.
postgres is not easily comparable to mongo, these are totally different db engines with different tradeoffs made for different purposes.
They very obviously have a significant degree of feature parity.
I don't use mongodb in particular but I never want to go back to worrying about what a SQL application instance is up to, either.
E.g. around the 300GB mark I had to constantly remind myself and others to be careful of it because it might have normal relational columns you think are fine to query, but it also has a fat json payload on each row and if you're careless the query planner would basically do a table scan for things you wouldn't expect.
This isn't so much a problem with json in an rdbms but more the false sense of query performance on non-json columns in the same table...after moving it to mongo never had the same problems because people knew not to query it in an unsupported manner.
For all of this to stop, not only does the bubble need to burst but people would need to be far more intentional about what resources they use where and when. People would also need to have longer software lifecycles, more QA, and so on. This would only happen in an environment with far higher interest rates and far lower monetary velocity. In essence, there'd have to be a lower tolerance of risk to stop the constant churn of hardware, software, and development methodologies on both. There are tradeoffs to such changes.
For the same reasons you outlined above: complexity.
No, they don't 'just add more complexity and overhead'. They are tools. Properly utilized, they are great.
There's a reason why you are (probably!) not running a power station and home and have, instead, hooked to a power utility company. They have the expertise and can amortize the costs among many customers.
Similarly, containers actually brought efficiency – they replaced many(obviously, not all) of the use-cases that usually required virtual machines. They also simplified some very difficult deployments.
The problem arises when you are doing things for the wrong reasons. For example, using 'microservices' just so you can try to avoid teams from having to talk to one another and end up shipping your org chart.
I do agree with a more generalized statement: more often than not, monoliths are what you need.
In 2019, Lyft spent $300M on AWS [1]. Sure, it's a big service, but are you seriously telling my that it's necessary to spend $300M in a single year?
I think the meme that developer's time spent on cost control is useless needs to die. Engineers should be encouraged to control costs so that they can take better decisions early on. They should be nudged to pick more cost-efficient choices. This won't only reduce costs, the chances are companies will also end up having dramatically simpler architecture.
[1]: https://www.cnbc.com/2019/03/01/lyft-plans-to-spend-300-mill...
That has a minimum cost of their hourly rate. The maximum cost is the cost of not doing what's most valuable to the company.
> In 2019, Lyft spent $300M on AWS [1]. Sure, it's a big service, but are you seriously telling my that it's necessary to spend $300M in a single year?
They didn't spend that in a single year.
It also depends what they were doing, I was recently very shocked by the data costs of a major hardware store in the US but after breaking it down it made more sense. It all depends what they get for that $300M over 3 years.
For better or worse a lot of folks are allergic to this. A framing that I’ve seen is to just understand the cost drivers from a technical perspective, for example large materialized views in BigQuery consume a lot of CPU. That’s technically interesting and it’s the reason they’re expensive.
> Lyft has signed up to pay cloud market leader Amazon Web Services at least $80 million per year for the next three years, totaling at least $300 million.
You can launch a massive project, hire dozens of people, buy server and colo space across the globe, rebuild their infra from scratch, and get costs down to – what – $60 million a year? Lyft is closing in on $4 billion in annual revenue. A few million in savings is meaningless, especially when balanced with the massive risks of such a project.
Maybe all those computers are doing something that someone thought was important, but it is not serving the core business.
Also when you run the DCs yourself, usually it's much slower for other engineers using your DC infra than using AWS or similar. It gets complicated!
Because they aren't a priority until they are.
Execs will either turn a blind eye or simply won't understand that it's a problem, until a massive purchase order lands in accounting and gets escalated to the highest levels.
Now it's suddenly a problem and it is up to the operations/devops/SRE/name of the month/SecFinDeVOps to figure out how to reduce costs. Which is completely backwards as they are probably not the group who requested the resources in the first place.
Dude's got a screenshot. I doubt he faked it for the attention - he probably actually got that bill.
He feels it's expensive and he should pay less for the value he's getting considering his alternatives (Azure VMs in a 1/3 of the price).
Perhaps refer to the actual price of a billed hour there? That feels like the core theme here.
MongoDB Atlas runs on the big 3 clouds, and of course you're going to pay more for a fully managed service, as opposed to hosting it yourself directly on the big 3, but racking up $2,500 in serverless compute charges in 24 hours seems very very hard to do, and especially by accident.
And then I read your first sentence - why is it always the assumption that people are going the hyperbole route? This is a well-upvoted link, with enough comments to merit a good discussion, and what appears to be a nicely-thought out post after the link.
Also, I've been following Snir for quite a while - dude wrote a massive Data Engineering roadmap which is heavily used by people learning how to ETL and has a very interesting investing blog. He's an engineer turned investor, and looks at the market from this perspective.
The derogatory quotes are unnecessary.
TBH I know how to spend that much and more with a single query (and ~30 minutes of compute time) in BigQuery. Serverless BI analytics workloads over large public datasets can do amazing things... and are also exactly the sorts of things a naive user would use to test + benchmark the capabilities of serverless data-warehousing systems.
With the right offering and the right software you can build something just 10% as good as some of the hyperscalers and have an amazing company.
Competing on price is not a sin, and it's what I'm working on/betting on.
That said, I think the real big threat to the hyperscalers is cheap chips and more people being able to treat datacenters like parking lots (nice ways to use capital to generate cashflow) -- the commoditization of data center operations.
The thing to understand early these days is COGS and whether the business can scale its service with good margins. That is a significant change of concerns for investors from the past years where growth was the only thing that mattered.
The other thing that I think is a little off in this post is the idea that startups not caring about costs are a big driver of hyperscaler revenue. I suppose that may have been true a few years ago but AWS, Azure, and to some extent GCP are having a lot of success moving legacy on-prem deployments to the cloud. It's not uncommon to encounter 50 year old enterprises with a billion dollars of cloud spend. One example of this is the dedicated SAP instances that AWS has, some of which cost $100K per month.
Some of the cloud provider compute (VMs) and storage (disk/object store) has very low margin. With it highly unlikely a company could provide equivalent offering in-house.
Early days I remember looking at Cloud vendors that were offering FISMA virtualization capabilities and I did an analysis of alternatives around running a set amount of CPUs, RAM, i/o and disk to support VMs. Terremark was the first offering that could meet our reqs, at around 10k a month per for a server's equivalent capability.
This put in my mind from day one that I was trading off High Capex with low Opex, for No Capex and High Opex. Given our trades, it was clear that high Capex was preferable to high Opex, as our high capex would have been equal to one month's opex.
When we later on saw more mainstream vendors like Amazon move into the space, their offerings were also coming with insane overhead, $2000 a month automatic surcharges for a FISMA regulated environment.
New era of cost-cutting, running systems yourself and distributed computing will bring far cheaper prices soon. If we do well energy and computation/intelligence will be trending toward 0 generally speaking
Most of the time is more expensive than paying for the infrastructure and personnel to manage it.
Also, while apparently cloud reduces complexity, it will end up adding lots of other complexities. And dealing with complexity means more money.
Also, the more you invest in a particular cloud technology, the more locked in you are, the less alternatives you have and that means paying up more.
If Stack Overflow does well by managing few dedicated servers, I don't get why smaller operations really "need" cloud.
And to add another perspective, the more layers of abstraction and indirection you add, the more complicated and ineficient things will be.
At a glance, Atlas pricing includes the typical ~ 20% premium if we compare it with AWS (Azure/GCP) VM setup. 20% is very far from *100 price difference as per article.
Vitaly https://linkedin.com/in/vkarasik