> (For context, Hotz raised $5M to improve RX 7900 XTX support and sell a $15K prebuilt consumer computer that runs 65B-parameter LLMs. A plethora of driver crashes later, he almost gave up on AMD.)
Again, I wish Hotz and TinyGrad the best, especially for training/experimentation on AMD, but I feel like Apache TVM and the Various MLIR efforts (like Pytorch MLIR, SHARK, Mojo) are much more promising for ML inference. Even triton in PyTorch is very promising, with an endorsement from AMD.
i guess it's that time of the week for my weekly response that de-hypes the buzzword hype:
>but I feel like Apache TVM and the Various MLIR efforts (like Pytorch MLIR, SHARK, Mojo) are much more promising for ML inference
1. There is no PyTorch MLIR. There is torch-mlir but that is not a PyTorch sponsored/affiliated project.
2. MLIR has nothing to do with any of the issues that AMD is having with drivers because MLIR is about IR not drivers. Nor does it have anything to do with tinygrad because there is no mature autograd implementation in terms of MLIR (Polygeist, with all due respect, is not mature). So each of the projects you name-drop depends on some existing frontend that includes an autodiff implementation. I don't know how tinygrad works but if at some point they decide to lower their graph to MLIR (instead of their own kernels), there is no barrier/impediment (other than actually implementing the lowering).
3. Mojo is just a complete non-sequitur here; besides the fact that you might as well consider it vaporware until there's some release outside of their walled garden (sometime this month supposedly for the SDK), they expressly aim to target CPU rather than GPU.
I work on MLIR. I like MLIR. I hope MLIR succeeds bigly but it doesn't need to be shoe-horned into absolutely every single conversation about AMD/GPU/ML/LLM whatever.
[1] The compilers don’t produce great instructions;
[2] The drivers crash frequently: ML workloads feel experimental;
[3] Software adoption is getting there, but kernels are less optimized within frameworks, in particular because of the fracture between ROCm and CUDA. When you are a developer and you need to write code twice, one version won’t be as good, and it is the one with less adoption;
[4] StackOverflow mindshare is lesser. Debugging problems is thus harder, as fewer people have encountered them.
4 comments
[ 4.4 ms ] story [ 24.5 ms ] threadAgain, I wish Hotz and TinyGrad the best, especially for training/experimentation on AMD, but I feel like Apache TVM and the Various MLIR efforts (like Pytorch MLIR, SHARK, Mojo) are much more promising for ML inference. Even triton in PyTorch is very promising, with an endorsement from AMD.
>but I feel like Apache TVM and the Various MLIR efforts (like Pytorch MLIR, SHARK, Mojo) are much more promising for ML inference
1. There is no PyTorch MLIR. There is torch-mlir but that is not a PyTorch sponsored/affiliated project.
2. MLIR has nothing to do with any of the issues that AMD is having with drivers because MLIR is about IR not drivers. Nor does it have anything to do with tinygrad because there is no mature autograd implementation in terms of MLIR (Polygeist, with all due respect, is not mature). So each of the projects you name-drop depends on some existing frontend that includes an autodiff implementation. I don't know how tinygrad works but if at some point they decide to lower their graph to MLIR (instead of their own kernels), there is no barrier/impediment (other than actually implementing the lowering).
3. Mojo is just a complete non-sequitur here; besides the fact that you might as well consider it vaporware until there's some release outside of their walled garden (sometime this month supposedly for the SDK), they expressly aim to target CPU rather than GPU.
I work on MLIR. I like MLIR. I hope MLIR succeeds bigly but it doesn't need to be shoe-horned into absolutely every single conversation about AMD/GPU/ML/LLM whatever.
I need to stop posting so late...
[1] The compilers don’t produce great instructions;
[2] The drivers crash frequently: ML workloads feel experimental;
[3] Software adoption is getting there, but kernels are less optimized within frameworks, in particular because of the fracture between ROCm and CUDA. When you are a developer and you need to write code twice, one version won’t be as good, and it is the one with less adoption;
[4] StackOverflow mindshare is lesser. Debugging problems is thus harder, as fewer people have encountered them.
---
were crucial when we had enough supply of NVidia GPUs, but if demand described in https://gpus.llm-utils.org/nvidia-h100-gpus-supply-and-deman...
is real (450,000+ H100)
Software bottlenecks most likely will be addressed sometime soon