Toronto-based Taalas just emerged from stealth with a claim that’s shaking the hardware world: 17,000 tokens per second on Llama 3.1 8B.
How? By physically etching the AI model directly into the silicon transistors. No HBM. No liquid cooling. Just raw, hardwired performance that is 10x faster and 20x cheaper than traditional GPU inference.
An interesting direction, beyond optimizing the KV cache for long-context inference, is to rethink where inference actually runs. If LLMs can be optimized to be efficiently deployed at the edge — for example on AI PCs — the burden on centralized data centers could be significantly reduced. In that case, inference demand may shift away from hyperscale compute clusters, easing both capacity and power pressures.
1 comment
[ 2.8 ms ] story [ 12.0 ms ] threadHow? By physically etching the AI model directly into the silicon transistors. No HBM. No liquid cooling. Just raw, hardwired performance that is 10x faster and 20x cheaper than traditional GPU inference.
An interesting direction, beyond optimizing the KV cache for long-context inference, is to rethink where inference actually runs. If LLMs can be optimized to be efficiently deployed at the edge — for example on AI PCs — the burden on centralized data centers could be significantly reduced. In that case, inference demand may shift away from hyperscale compute clusters, easing both capacity and power pressures.