TTFT is around 180ms, prefill speed is ~2.7k tok/sec. Yes, both prefill and decode slows down as concurrency increases (it's not linear). This 255 tok/s is at conc = 1.
It's faster and more accurate. Bonsai was a way to demonstrate how effective ternary quant is, and how much it accelerates throughput. If you try coding with it you'll find that it often makes mistakes with tool calls…
That is true, the lack of good benchmarks will be with us for a while I think. Ternary is not a good quant algo, but because it's so drastic, it allows people to use enormous models on worse hardware, or just increase…
Lilith-zero sits between the agent and the tool server as a transparent proxy, meaning that the agent speaks to the middleware over stdio, and the middleware manages the upstream server process directly (spawning,…
TTFT is around 180ms, prefill speed is ~2.7k tok/sec. Yes, both prefill and decode slows down as concurrency increases (it's not linear). This 255 tok/s is at conc = 1.
It's faster and more accurate. Bonsai was a way to demonstrate how effective ternary quant is, and how much it accelerates throughput. If you try coding with it you'll find that it often makes mistakes with tool calls…
That is true, the lack of good benchmarks will be with us for a while I think. Ternary is not a good quant algo, but because it's so drastic, it allows people to use enormous models on worse hardware, or just increase…
Lilith-zero sits between the agent and the tool server as a transparent proxy, meaning that the agent speaks to the middleware over stdio, and the middleware manages the upstream server process directly (spawning,…