Python and PyTorch all call out to C libraries… I don’t get what he means by “proving LLMs can run without Python and PyTorch” at all. Seems like they don’t understand basic fundamentals about things here…
My first implementation of gemma.cpp was kind of like this.
There's such a massive performance differential vs. SIMD though that I learned to appreciate SIMD (via highway) as one sweet spot of low-dependency portability that sits between C loops and the messy world of GPUs + their fat tree of dependencies.
If anyone want to learn the basics - whip out your favorite LLM pair programmer and ask it to help you study the kernels in the ops/ library of gemma.cpp:
I'm curious how many more functions you'd need to add to have full coverage of every publically available LLM innovation (e.g. QK-Norm from Qwen3, SwiGLU etc.).
Obviously llama.cpp has a much bigger library but it's lovely to see everything in one clean file.
8 comments
[ 2.9 ms ] story [ 24.9 ms ] threadDid we need any proof of that ?
There's such a massive performance differential vs. SIMD though that I learned to appreciate SIMD (via highway) as one sweet spot of low-dependency portability that sits between C loops and the messy world of GPUs + their fat tree of dependencies.
If anyone want to learn the basics - whip out your favorite LLM pair programmer and ask it to help you study the kernels in the ops/ library of gemma.cpp:
https://github.com/google/gemma.cpp/tree/main/ops
In particular it's got a single ~600 line file (https://github.com/robitec97/gemma3.c/blob/main/gemma3_kerne...) with a clear straightforward implementation of every major function used in inferencing (google's models) from gelu to rope.
I'm curious how many more functions you'd need to add to have full coverage of every publically available LLM innovation (e.g. QK-Norm from Qwen3, SwiGLU etc.).
Obviously llama.cpp has a much bigger library but it's lovely to see everything in one clean file.