parched99
No user record in our sample, but parched99 has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
No user record in our sample, but parched99 has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
I answered the question directly. IQ4_X_S is smaller, but slower and less accurate than Q4_0. The parent comment specifically asked about the QAT version. That's literally what this thread is about. The context-length…
Resolving that issue, would help reduce (not eliminate) the size of the context. The model will still only just barely fit in 16 GB, which is what the parent comment asked. Best to have two or more low-end, 16GB GPUs…
I'm aware. I was addressing the question being asked.
I think Powershell is a bad test. I've noticed all local models have trouble providing accurate responses to Powershell-related prompts. Strangely, even Microsoft's model, Phi 4, is bad at answering these questions…
I am only able to get the Gemma-3-27b-it-qat-Q4_0.gguf (15.6GB) to run with a 100 token context size on a 5070 ti (16GB) using llamacpp. Prompt Tokens: 10 Time: 229.089 ms Speed: 43.7 t/s Generation Tokens: 41 Time:…