I dont think big companies like Google react to news like that. Very likely this was meant to go out on Tuesday for weeks now. And it just happened to have coincided with another news.
They do different things and sit in different parts of the stack (training vs inference). TorchTPU seems more like a play to get more people familiar with Googles TPU chips as they want to sell them to people outside of google.
Yeah they read the news this morning and whipped up a quick Torch->XLA translation layer and coordinated with four different inference engines (who also all wrote their own blog posts) and shoved this out the door quick to stay on top of the news cycle
We might never know, but they probably didn't read the news and started working to get this out, they were probably working on it for a while now.
Personally the question is if they are "saving" other projects or news so they can be released in case a competitor makes a move? Who knows if this was ready days or weeks ago and were waiting on someone to drop some news, so they can shadow those news.
Yeah a lot of people bought into the idea of CUDA being a moat but we’re well past the point where it’s pretty trivial to jump to another equivalent stack on the software side.
Those are the two main interfaces: the inference hardware and the endpoints. Endpoints are pretty much standardized to the point that you can switch them out in seconds. CUDA still has an edge but every release like this chips away at that. Even so, TPUs are unobtanium so while I appreciate Google's effort here I also recognize that it is not something that will be useful to me.
In that sense AMD is still better positioned to give NVidia a run for their money but given the family connection and AMD's underdog status I am not very hopeful that we'll see meaningful competition in inference hardware any time soon.
NVidia is a much harder nut to crack than OpenAI or Anthropic. They are easily a decade ahead of viable competition. Check out the GB300 if you don't believe that.
While I have the same sentiment, I am also curious why a competitor can't accelerate their developments with AI. In the end, hardware development is to a large extent also software development. Is it the patents that NVIDIA has that are preventing this? Or maybe some trade secrets? AFAIK the NVIDIA hardware is manufactured entirely by TSMC.
Google does provide TPUs to big players like Anthropic and other large enterprise customers for on prem deploymens. That's their target just like those the GB300 is aimed at.
Even if it's allegedly possible, the devil has always been in the details. It always sucks to find out that there's some quirk of your model architecture that breaks an implicit assumption that underpins the export process.
For people interested in this: I've been working on an open-source libtpu implementation for https://tenstorrent.com/ accelerators as a side project, it can already run most of torchtpu, see https://github.com/pcmoritz/libtt (based on the tenstorrent software stack). It shows all the things involved in lowering torch to tile based accelerators.
Pardon my ignorance, but who else besides Google has TPUs? Are they commercially available somewhere? What’s their main application right now outside of Google?
You can rent them on cloud + they sell directly to Anthropic. Its pretty well known that Anthropic is a heavy user of TPU's (along with Trainium & GPU's)
Google is selling TPUs to enterprise clients for on prem deployments. We’ve evaluated them but decided to stick with Nvidia, at least for one more generation. We’re a relatively large user of GCP.
Google and Apple are going to move consumers and ordinary business users to edge inference. This is more disruptive to the upcoming IPOs than most investors understand.
They are just wildly outdated for most tasks. Even projects like frigate suggest buying different hardware. Coral was released in 2019 and hasn't been updated once. With a jetson nano or even a raspi ai hat you get way more tops
I've been really hoping to get Whisper running on a coral, but it can't even do that. There was recently some progress getting it to export to tflite (and even an official example added) but the coral is missing some instructions.
Yeah right, try generating an image on Apple Silicon or an Android phone with any of the Qwen or Flux family of models, let alone videos or prefill heavy LLMs.
It makes PyTorch work nicely on TPU, which is impressive mostly because of how unstructured PyTorch is with its conventions. From a compiler standpoint JAX is much easier to work with since its all pure functions. This means that JAX also has access to fairly cutting edge compiler tech like Enzyme which is quite relevant for scientific machine learning work.
64 comments
[ 10.4 ms ] story [ 76.4 ms ] threadhttps://github.com/aws-neuron/torch-neuronx
Is this somehow Google's answer to today's best DeepSeek open-sourcing inferencing engine?
Personally the question is if they are "saving" other projects or news so they can be released in case a competitor makes a move? Who knows if this was ready days or weeks ago and were waiting on someone to drop some news, so they can shadow those news.
There will be an abundance of shovels. Nebius already crashed by almost half from its hype peak.
In that sense AMD is still better positioned to give NVidia a run for their money but given the family connection and AMD's underdog status I am not very hopeful that we'll see meaningful competition in inference hardware any time soon.
NVidia is a much harder nut to crack than OpenAI or Anthropic. They are easily a decade ahead of viable competition. Check out the GB300 if you don't believe that.
High bandwidth flash will be here soon, I bet someone makes an economical competitor thats not quite as fast but much cheaper.
Also I guess this is the final nail in the coffin of Tensorflow.
Since HW specifics live mostly in the kernel layer (Pallas or CUDA), I imagine the Jax vs PyTorch question is more of a style preference now?