TPUs are like the NPU of the training world. You take a bunch of extra time, money and dedicated silicon and end up with an ASIC that barely competes on equal terms with a similarly priced GPU. Unless you've got access to Nvidia's TSMC supply, you're probably not going to make a dent on their demand.
Additionally - TPUs are completely useless if AI goes out of style, unlike CUDA GPUs. The great thing about Nvidia's hardware right now is that you can truly use the GPU for whatever you want. Maybe AI falls through in 2026, and now those GPUs can be used for protein folding or crypto mining. Maybe crypto mining and protein folding falls through - you can still use most of those GPUs for raster renders and gaming too! TPUs are just TPUs - if AI demand goes away, your dedicated tensor hardware is dead weight.
Also TPU v1,v2 and v3 were ASICs, but since v4 they have added some new features so they have a lower performance/watt which is quite near Nvidia's power draw. I think Hopper is at 700W and TPU are around 600W.
Is there a possible way to unblow the fuse? I imagine it depends on the type of e-fuse used. The Athlon XP pencil trick probably won't work, haha. Curious if anyone has more information.
Fascinating, I had no idea this was even a thing. Simultaneously badass team green is far ahead and also a bummer to be artificially limited / segmented.
I wonder if the upcoming 5090 core will mostly be a fuse-intact 4090. I imagine nearly all of NVs current focus is on H200 and Blackwell and whatever else is in the pipeline rather than these "silly" little gamer cards which bring in comparably trivial financial resources.
From my research it seems that restoring full GPU capabilities by repairing or circumventing a deliberately blown eFuse on an NVIDIA AD102 die is, for all practical purposes, impossible.
Maybe, but it's also possible that it wouldn't do anything. Other Nvidia boards (like the Tegra in the Switch) also come with arbitrarily disabled "dark silicon", but enabling the extra SOC hardware only causes the board to crash when using everything at once. It wouldn't surprise me if this was a binning measure, even though I also wouldn't be surprised if it was an arbitrary limit.
In a way... I can actually see this as fair. What's the difference between the 4090 rtx and the 6000 ada? 5x the price for 2x the memory? Ridiculous. But then you have to factor in all the R&D dollars Nvidia poured into their compute/non-graphics ecosystem which now easily eclipses the gaming one, probably by a factor of 10 or more, and suddenly it doesn't seem so ridiculous. You either a) don't get 4090 level of a graphics card anymore... or b) you do get it, but only if it's nerfed for non-graphics uses... Nvidia wants its big R&D bucks back (and then some) and its gonna get em
I think this is a vast understatement. Google has been using their own TPUs for a very long time now. I think they still have some GPUs from Nvidia, but it's marginal compared to their own silicon. Other big players are behind the curve on this front, but very much working to close the gap. These are companies with nearly infinite pockets, Google has shown that you can make it work without Nvidia, it's only a matter of time before others do it too.
See I absolutely dislike this thought that hyperscalars can easily beat nvidia. It is not their domain of expertise. Tpu are not where near GPU in performance. People really underestimate Nvidia's expertise and strengths.
They don't need to beat them on performance though. If you get half the performance at third the price you can just make more chips and be fine. It's not like Google is gonna run out of datacenter space.
All of this may very well be true — but it doesn't matter. Google is getting results very similar to the rest of the pack, probably at a fraction of the cost. The implementation doesn't matter so long as you get the results.
It won’t happen by other company coming up with faster chips. It will happen by other company coming with cheaper chips and less energy demanding, dominating the low end and then reaching higher.
While Nvidia might have won the training market, it’s inference where the real money is.
Genuine question: is that true? It seems bonkers that they'd be cranking out more proprietary processors than they could acquire from an established GPU manufacturer.
It is wild that the number of GPUs purchased by a company has become, like, an infrastructure investment or something. Like the count itself is worth reporting.
What will they accomplish with the things? Why even think about that part? Probably AI. Selling premium GEMMs, what a trick. Bah. Hopefully TSMC got a really good cut, they are at least doing some interesting engineering.
>It is wild that the number of GPUs purchased by a company has become, like, an infrastructure investment or something. Like the count itself is worth reporting.
An estimated count is newsworthy for the journalists and the readers because it's an indirect proxy for outsiders -- who are not privy to the internal plans of FAANG companies -- to try and figure out what's happening. Basically trying to "read the tea leaves" of the AI industry.
Demand exceeds supply. NVIDIA has limited number chips to sell and TSMC factory time is overbooked. In the current zero-sum situation, NVIDIA picking and choosing who to sell to may be a signal of something. And/or Microsoft/OpenAI's willingness to spend billions on 2x the NVIDIA chips is a signal of something.
Aren't Amazon, Google, and Meta running their own silicon for some training and inference? Does MS have an equivalent?
That could explain a large part of the gap.
Edit: it looks like Microsoft announced their own last year, but I can imagine they may be behind the curve in capability and scale out compared to the others
My outsider's understanding is Google is the only one whose custom silicon is the primary compute for their flagship foundation models. I didn't see any messaging about the Nova models being trained on Traininium (AWS), and Meta still talks about the number of H110's training their Llama models.
I was writing a comment saying the same thing when your comment appeared. Yest, Meta, Google, and Amazon all have custom silicon, and it seems Microsoft's similar efforts came later. None of these companies want to give Nvidia all of the money, so going forward, I think Nvidia isn't going to see more competition from these efforts. The big players aren't going to sell their chips to others (I don't think), but they'll make them available to cloud customers.
Maybe some other company will catch up. But it is hard.
Intel is better at chip design than any of those companies. They spent a lot of effort coming up with a very clever chip that competed well against the current generation of Nvidia chips, while still running your old x86 codes.
Nvidia continued increasing memory bandwidth, and nobody cared about Knights Whatever.
Amazon and Meta is at an early stage for their TPU equivalent and I don't think they're ready for production loads. Only Google has comparable silicons but I suspect even Google TPU are mostly for internal products rather than consumers.
This is what they agreed to, in order to win the OpenAI partnership. In exchange, OpenAI doesn't have to build or support their own infra. In theory, a win-win, but only if MSFT can effectively sell OpenAI-on-Azure.
54 comments
[ 3.1 ms ] story [ 115 ms ] threadAdditionally - TPUs are completely useless if AI goes out of style, unlike CUDA GPUs. The great thing about Nvidia's hardware right now is that you can truly use the GPU for whatever you want. Maybe AI falls through in 2026, and now those GPUs can be used for protein folding or crypto mining. Maybe crypto mining and protein folding falls through - you can still use most of those GPUs for raster renders and gaming too! TPUs are just TPUs - if AI demand goes away, your dedicated tensor hardware is dead weight.
https://x.com/realGeorgeHotz/status/1868356459542770087
Fascinating, I had no idea this was even a thing. Simultaneously badass team green is far ahead and also a bummer to be artificially limited / segmented.
I wonder if the upcoming 5090 core will mostly be a fuse-intact 4090. I imagine nearly all of NVs current focus is on H200 and Blackwell and whatever else is in the pipeline rather than these "silly" little gamer cards which bring in comparably trivial financial resources.
/me *cries a tear*
https://x.com/cognitivecompai/status/1868399108924592391
https://x.com/cognitivecompai/status/1868401738706993301
From my research it seems that restoring full GPU capabilities by repairing or circumventing a deliberately blown eFuse on an NVIDIA AD102 die is, for all practical purposes, impossible.
Maybe, but it's also possible that it wouldn't do anything. Other Nvidia boards (like the Tegra in the Switch) also come with arbitrarily disabled "dark silicon", but enabling the extra SOC hardware only causes the board to crash when using everything at once. It wouldn't surprise me if this was a binning measure, even though I also wouldn't be surprised if it was an arbitrary limit.
While Nvidia might have won the training market, it’s inference where the real money is.
What will they accomplish with the things? Why even think about that part? Probably AI. Selling premium GEMMs, what a trick. Bah. Hopefully TSMC got a really good cut, they are at least doing some interesting engineering.
An estimated count is newsworthy for the journalists and the readers because it's an indirect proxy for outsiders -- who are not privy to the internal plans of FAANG companies -- to try and figure out what's happening. Basically trying to "read the tea leaves" of the AI industry.
Demand exceeds supply. NVIDIA has limited number chips to sell and TSMC factory time is overbooked. In the current zero-sum situation, NVIDIA picking and choosing who to sell to may be a signal of something. And/or Microsoft/OpenAI's willingness to spend billions on 2x the NVIDIA chips is a signal of something.
https://fortune.com/2024/02/21/nvidia-earnings-ceo-jensen-hu...
https://fortune.com/2024/09/12/nvidia-jensen-huang-ai-traini...
That could explain a large part of the gap.
Edit: it looks like Microsoft announced their own last year, but I can imagine they may be behind the curve in capability and scale out compared to the others
Intel is better at chip design than any of those companies. They spent a lot of effort coming up with a very clever chip that competed well against the current generation of Nvidia chips, while still running your old x86 codes.
Nvidia continued increasing memory bandwidth, and nobody cared about Knights Whatever.
"Omdia analyses companies’ publicly disclosed capital spending, server shipments and supply chain intelligence to calculate its estimates."
https://finance.yahoo.com/news/microsoft-stock-receives-rare...