> When I look around, I see hundreds of billions of dollars being spent on hardware – GPUs, data centers, and power stations. What I don’t see are people waving large checks at ML infrastructure engineers like me and my team.
That doesn't seem to be the case to me. I guess the author wants to do everything on his own terms and maybe companies aren't interested in that.
If they optimize though - and this is coming at some point - local AI becomes possible, and their entire business case as a cloud monopoly evaporates. I think they know they're in a race between centralized control, and widespread use and control, and that is what is really driving this.
While I agree there’s a lack of attention for the impact of software engineers on near-term industry growth — rather the opposite with layoffs and agentic automation (attempts) et cetera; the mentioned Scott Gray is working at OpenAI now, so the human capital angle is I guess just flying under the mainstream radar.
What also cannot be ignored, is that transformer models are a great unifying force. It's basically one architecture that can be used for many purposes.
This eliminates the need for more specialized models and the associated engineering and optimizations for their infrastructure needs.
Spending a lot (on capex or opex) certainly is not providing any kind of signaling benefit at this level. It's the opposite, because obviously every single financial analyst in the market is worried about the rapid increase in capex. The companies involved are cutting everything else to the bone to make sure they can still make those (necessary) investments without degrading their top-line numbers too much. Or in some cases actively working to hide the debt they're financing this with from their books.
Even if we imagined that the author's conspiracy theory were true, there would still be massive incentives for optimization because everyone is bottlenecked on compute despite expanding it as fast as is physically possible. Like, are we supposed to believe that nobody would run larger training runs if the compute was there? That they're intentionally choosing to be inefficient, and as a result having to rate-limit their paying customers? Of course not.
The reality is that any serious ML operation will have teams trying to make it more efficient, at all levels. If the author's services are not wanted, there are a few more obvious options than the outright moronic theory of intentional inefficiency. In this case most likely that their product is an on-edge speech to text model, which is not at all relevant to what is driving the capex.
Sounds like someone that got lucky in big picture (in ML during Alexnet era), but then unlucky in picking the sub-genre.
>I see hundreds of billions of dollars being spent on hardware
>I don’t see are people waving large checks at ML infrastructure engineers like me
Which seemed like a valid question mark until you look at the github. <1B Raspberry pi class edge speech models. That's not the game the hyperscalers are playing
I don't think we can conclude much of anything about the datacenter build out from that
I see a lot of comments here criticizing the author, and I think both teams have a point. There's definitely a bubble, because companies are buying up infrastructure which doesn't need to be used right now.
But also, the companies are buying up this infrastructure because whoever controls the infrastructure also controls the industry in around 5 years time.
"OpenAI rejected me so the entire industry is going to collapse" is certainly a take. They are still probably one of the less arrogant engineers in silicon valley.
Author is definitely correct in pointing out the incentives for companies to buy hardware. What the article misses is that there is in fact a reasonable economic incentive to not invest in software even if LLMs were not an economic bubble. It is that every single company is developing the same thing, there are many of those who even develop them as open source, and the ones that are closed as well as any company who would hire this guy, have a bunch of industrial spies inside anyway. Buying hardware may increase your moat, but developing software just rises the sea level.
This is very insightful. I remember the epoch of clueless startups wasting venture capital on Sun servers. I worked at one of those startups. Warden is clearly correct that if you want to train your AI faster then the optimal amount to spend on software optimization is at least a substantial fraction of your hardware budget.
However, clueless people who don't know how to optimize probably don't know where to spend money on optimization, either. So maybe it's just not a great fit for outsourcing, especially in a realm where there's no standard of correctness to measure the results of the supposedly "optimized" training against. And Warden seems to be pitching outsourcing rather than trying to get acquihired.
He lost me a bit at the end talking about running chat bots on CPUs. I know it's possible, but it's inherently parallel computing isn't it? Would that ever really make sense? I expected to hear something more like low end consumer gpus.
Recent generation llms do seem to have some significant efficiency gains. And routers to decide if you really need all of their power on a given question. And Google is building their custom tpus. So I'm not sure if I buy the idea that everyone ignores efficiency.
1. Token prices keep plummeting even as models are getting stronger.
2. Most models are being offered for free at a significant loss, so reducing costs would be critical to maintain some path to sustainability.
3. Every hyperscaler has been consistently saying for the past several quarters that they are severely constrained on capacity and in fact have billions in booked backlogs. That is, if they had more capacity they would actually be making even more billions.
I can totally imagine the smaller players renting these cloud resources for their private model uses to be rather inefficient (which is where the 50% utilization number comes from), probably because they are prioritizing time-to-market over other aspects. But I would wager that resource efficiency, at least for inference, is absolutely a top priority for all the big players.
17 comments
[ 4.9 ms ] story [ 34.6 ms ] threadThat doesn't seem to be the case to me. I guess the author wants to do everything on his own terms and maybe companies aren't interested in that.
OTOH garage-startup acquisitions are acquihires.
This eliminates the need for more specialized models and the associated engineering and optimizations for their infrastructure needs.
Neither the hyperscalers nor NVDA are safe from uncertainty.
Spending a lot (on capex or opex) certainly is not providing any kind of signaling benefit at this level. It's the opposite, because obviously every single financial analyst in the market is worried about the rapid increase in capex. The companies involved are cutting everything else to the bone to make sure they can still make those (necessary) investments without degrading their top-line numbers too much. Or in some cases actively working to hide the debt they're financing this with from their books.
Even if we imagined that the author's conspiracy theory were true, there would still be massive incentives for optimization because everyone is bottlenecked on compute despite expanding it as fast as is physically possible. Like, are we supposed to believe that nobody would run larger training runs if the compute was there? That they're intentionally choosing to be inefficient, and as a result having to rate-limit their paying customers? Of course not.
The reality is that any serious ML operation will have teams trying to make it more efficient, at all levels. If the author's services are not wanted, there are a few more obvious options than the outright moronic theory of intentional inefficiency. In this case most likely that their product is an on-edge speech to text model, which is not at all relevant to what is driving the capex.
>I see hundreds of billions of dollars being spent on hardware
>I don’t see are people waving large checks at ML infrastructure engineers like me
Which seemed like a valid question mark until you look at the github. <1B Raspberry pi class edge speech models. That's not the game the hyperscalers are playing
I don't think we can conclude much of anything about the datacenter build out from that
But also, the companies are buying up this infrastructure because whoever controls the infrastructure also controls the industry in around 5 years time.
However, clueless people who don't know how to optimize probably don't know where to spend money on optimization, either. So maybe it's just not a great fit for outsourcing, especially in a realm where there's no standard of correctness to measure the results of the supposedly "optimized" training against. And Warden seems to be pitching outsourcing rather than trying to get acquihired.
Recent generation llms do seem to have some significant efficiency gains. And routers to decide if you really need all of their power on a given question. And Google is building their custom tpus. So I'm not sure if I buy the idea that everyone ignores efficiency.
1. Token prices keep plummeting even as models are getting stronger.
2. Most models are being offered for free at a significant loss, so reducing costs would be critical to maintain some path to sustainability.
3. Every hyperscaler has been consistently saying for the past several quarters that they are severely constrained on capacity and in fact have billions in booked backlogs. That is, if they had more capacity they would actually be making even more billions.
I can totally imagine the smaller players renting these cloud resources for their private model uses to be rather inefficient (which is where the 50% utilization number comes from), probably because they are prioritizing time-to-market over other aspects. But I would wager that resource efficiency, at least for inference, is absolutely a top priority for all the big players.