[–] pvtmert 2d ago ↗ Well, also GH Release:- https://github.com/ggml-org/llama.cpp/releases/tag/v1.0.0- https://news.ycombinator.com/item?id=50481715
[–] SlavikCA 2d ago ↗ Recently I see quite a few changes, related to KV caches.Sounds like earlier the advice was: for coding, quantizing KV cache is bad, keep it at full F16/BF16.Sounds like now, 8 bit KV cache is ok for coding, but the question is which one? I see f8_e4m3, q8_0. Would be great to get more info about that.
2 comments
[ 0.25 ms ] story [ 14.5 ms ] thread- https://github.com/ggml-org/llama.cpp/releases/tag/v1.0.0
- https://news.ycombinator.com/item?id=50481715
Sounds like earlier the advice was: for coding, quantizing KV cache is bad, keep it at full F16/BF16.
Sounds like now, 8 bit KV cache is ok for coding, but the question is which one? I see f8_e4m3, q8_0. Would be great to get more info about that.