Guess MSFT needs somewhere else AI adjacent to funnel money into to produce the illusion of growth and future cash flow in this bubblified environment.
So similar to Apple Silicon. If this means they'll be on par with Apple Silicon I'm okay with this, I'm surprised they didn't do this sooner for their Surface devices.
Oh right, for their data centers. I could see this being useful there too, brings costs down lower.
Not suprising that the hyperscalers will make this decision for inference and maybe even a large chunk of training. I wonder if it will spur nvidia to work on an inference only accelerator.
Even just saying this applies downward pressure on pricing: NVIDIA has an enormous amount of market power (~"excess" profit) right now and there aren't enough near competitors to drive that down. The only thing that will work is their biggest _consumers_ investing, or threatening to invest, if their prices are too high.
Long term, I wonder if we're exiting the "platform compute" era, for want of a better term. By that I mean compute which can run more or less any operating system, software, etc. If everyone is siloed into their own vertically integrated hardware+operating system stack, the results will be awful for free software.
Google has been using its own TPU silicon for machine learning since 2015.
I think they do all deep learning for Gemini on ther own silicon.
But they also invented AI as we know it when they introduced transformer architecture and they’ve been more invested in machine learning than most companies for a very long time.
> The software titan is rather late to the custom silicon party. While Amazon and Google have been building custom CPUs and AI accelerators for years, Microsoft only revealed its Maia AI accelerators in late 2023.
They are too late for now, they realistically hardware takes a couple generations to become a serious contender and by the time Microsoft has a chance to learn from their hardware mistakes the “AI” bubble will have popped.
But, there will probably be some little LLM tools that do end up having practical value; maybe there will be a happy line-crossing point for MS and they’ll have cheap in-house compute when the models actually need to be able to turn a profit.
I guess Microsoft’s investment into Graphcore didn’t pay off. Not sure what they’re planning but more of that isn’t going to cut it. At the time (late 2019) I was arguing for either a GPU approach or specialized architecture targeting transformers.
There was a split at MS where the ‘Next Gen’ bayesian was being done in the US and the frequentist work was being shipped off to China. Chris Bishop was promoted to head of MSR Cambridge which didn’t help.
Microsoft really is an institutionally stupid organization so I have no idea on which direction they actually go. My best guess is that it’s all talk.
I used to work for Graphcore. Microsoft gave up on Graphcore fairly early on actually and I think it was a wise decision. There were a number of issues with Graphcore's GC1 and GC2 chips:
* a bet on storing the entire model (and code) in 900MB of SRAM. Hell of a lot of SRAM but it only really works for small models and the world wants enormous models.
* Blew it's weirdness budget by a lot. Everything is quite different so it's a significant effort to port software to it. Often you did get a decent speedup (like 2-10x) but I doubt many people thought that was worth the software pain.
* The price was POA so normal people couldn't buy one. (And it would have been too expensive for individuals anyway.) So there was little grass roots community support and research. Nvidia gets that because it's very easy to buy a consumer 4090 or whatever and run AI on it.
* Nvidia were killing it with Grace Hopper.
GC3 was way more ambitious and supports a mountain of DRAM with crazy memory bandwidth so if they ever finish it maybe they'll make a comeback.
It always falls back on the software. AMD is behind, not because the hardware is bad, but because their software historically has played second fiddle to their hardware. The CUDA moat is real.
So, unless they also solve that issue with their own hardware, then it will be like the TPU, which is limited to usage primarily at Google, or within very specific use cases.
There are only so many super talented software engineers to go around. If you're going to become an expert in something, you're going to pick what everyone else is using first.
The name of the game has been custom SoCs and ASICs for a couple years now, because inference and model training is an "embarrassingly parallel" problem, and models that are optimized for older hardware can provide similar gains to models that are run on unoptimized but more performant hardware.
Same reason H100s remain a mainstay in the industry today, as their performance profile is well understood now.
On a slightly different tangent, is anyone working on analog machine learning ASICs? Sub-threshold CMOS or something? I mean even at the research level? Using a handful of transistor for an analog multiplier. And get all of the crazy fascinating translinear stuff of Barrie Gilbert fame.
33 comments
[ 4.4 ms ] story [ 51.3 ms ] threadOh right, for their data centers. I could see this being useful there too, brings costs down lower.
Long term, I wonder if we're exiting the "platform compute" era, for want of a better term. By that I mean compute which can run more or less any operating system, software, etc. If everyone is siloed into their own vertically integrated hardware+operating system stack, the results will be awful for free software.
I think they do all deep learning for Gemini on ther own silicon.
But they also invented AI as we know it when they introduced transformer architecture and they’ve been more invested in machine learning than most companies for a very long time.
> The software titan is rather late to the custom silicon party. While Amazon and Google have been building custom CPUs and AI accelerators for years, Microsoft only revealed its Maia AI accelerators in late 2023.
They are too late for now, they realistically hardware takes a couple generations to become a serious contender and by the time Microsoft has a chance to learn from their hardware mistakes the “AI” bubble will have popped.
But, there will probably be some little LLM tools that do end up having practical value; maybe there will be a happy line-crossing point for MS and they’ll have cheap in-house compute when the models actually need to be able to turn a profit.
There was a split at MS where the ‘Next Gen’ bayesian was being done in the US and the frequentist work was being shipped off to China. Chris Bishop was promoted to head of MSR Cambridge which didn’t help.
Microsoft really is an institutionally stupid organization so I have no idea on which direction they actually go. My best guess is that it’s all talk.
* a bet on storing the entire model (and code) in 900MB of SRAM. Hell of a lot of SRAM but it only really works for small models and the world wants enormous models.
* Blew it's weirdness budget by a lot. Everything is quite different so it's a significant effort to port software to it. Often you did get a decent speedup (like 2-10x) but I doubt many people thought that was worth the software pain.
* The price was POA so normal people couldn't buy one. (And it would have been too expensive for individuals anyway.) So there was little grass roots community support and research. Nvidia gets that because it's very easy to buy a consumer 4090 or whatever and run AI on it.
* Nvidia were killing it with Grace Hopper.
GC3 was way more ambitious and supports a mountain of DRAM with crazy memory bandwidth so if they ever finish it maybe they'll make a comeback.
So, unless they also solve that issue with their own hardware, then it will be like the TPU, which is limited to usage primarily at Google, or within very specific use cases.
There are only so many super talented software engineers to go around. If you're going to become an expert in something, you're going to pick what everyone else is using first.
The name of the game has been custom SoCs and ASICs for a couple years now, because inference and model training is an "embarrassingly parallel" problem, and models that are optimized for older hardware can provide similar gains to models that are run on unoptimized but more performant hardware.
Same reason H100s remain a mainstay in the industry today, as their performance profile is well understood now.
[0] - https://news.ycombinator.com/item?id=45275413
[1] - https://news.ycombinator.com/item?id=43383418
https://www.electronicdesign.com/technologies/analog/article...
https://www.analog.com/en/resources/analog-dialogue/articles...
http://madvlsi.olin.edu/bminch/talks/090402_atact.pdf
Is it practical for them to buy an existing chip maker? Or would they just go home-grown?
- As of today, Nvidia's market cap is a whopping 4.51 trillion USD compared to Microsoft's 3.85 trillion USD, so that might not work.
- AMD's market cap is 266.49 billion USD, which is more in reach.