Transformers scale poorly vs. context window size and parameter count.
Which means really impressive when those N’s are small!
I’m but a pundit in this area so don’t know much. But one wonders if there’s a future in burning larger models to FPGAs - whether big enough FPGAs exist (or can be built), and whether locating specialized compute right with the memory it needs can speed things up.
Likely would need a lot of algorithm parallelism work that’d translate back to CPUs/GPUs.
Huge FPGAs don’t really exist, but you can couple many together with high-speed interconnects.
They will never be as fast as an NPU designed to run large models, though. GPUs are extremely general purpose in comparison, and FPGAs are about as general purpose as one can get.
6 comments
[ 4.5 ms ] story [ 25.3 ms ] threadhttps://rits.shanghai.nyu.edu/ai/karpathys-microgpt-on-fpga-...
TL;DR: The CPU implementation was 71x faster than the FPGA.
Note: model has only 4192 parameters.
Which means really impressive when those N’s are small!
I’m but a pundit in this area so don’t know much. But one wonders if there’s a future in burning larger models to FPGAs - whether big enough FPGAs exist (or can be built), and whether locating specialized compute right with the memory it needs can speed things up.
Likely would need a lot of algorithm parallelism work that’d translate back to CPUs/GPUs.
They will never be as fast as an NPU designed to run large models, though. GPUs are extremely general purpose in comparison, and FPGAs are about as general purpose as one can get.