Sounds more like SPIR-V and intermediary optimization for custom hardware or their current setup. I'm guessing sama wants to copy Deepseek's strategy, not build a language.
One thing I've thought about for coding with LLMs, is that passing in source code to be tokenized by the clip/whatever English text parser seems like it would be suboptimal compared to training on the AST that gets generated by the compiler after parsing the source.
8 comments
[ 5.3 ms ] story [ 26.2 ms ] threadSounds more like SPIR-V and intermediary optimization for custom hardware or their current setup. I'm guessing sama wants to copy Deepseek's strategy, not build a language.
So there would need to be compiler support in Triton for their new custom inference chips that will be used run on them.