I don't want to sound overly dismissive, but for quite a few practical cases pipeline parallelism with micro-batches will be a likely win over tensor parallelism. Of course this inherently comes with a requirement to…
> speculative decoding is a great boon, but mostly for large labs hosting and serving many requests at once, and maybe not so effective for local inference This is somewhat of a nitpicking point, but AIUI speculative…
It's a very nice summary of KV cache storage requirements for the leading open weights models, but ultimately it still shows the DeepSeek V4 series winning by a huge margin on that front. And DeepSeek actually does…
0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a…
Kimi uses INT4 as its native format, there's no such thing as "better than 4-bit precision" for that model. This is in contrast with GLM for which 16-bit precision is native and 8-bit is in common use.
> The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those…
10k context is not a whole lot, this model theoretically supports up to 1M. But the KV cache storage takes up a whole lot more memory capacity at full context than DeepSeek V4 Pro, let alone Flash. (About ~96GB…
AIUI the llama.cpp implementation for this model is still quite half-baked due to missing the support for DSA sparse attention mechanism. This leads to running the model with a different mechanism that it has not been…
Lousy benchmark, they explicitly focus on the easiest tasks to automate for AI (i.e. heavily cherry picked outcomes) and it seems that they don't bother to test anything except just-released proprietary models.
What's "exponential" about AI development? Model parameter counts? Anthropic doesn't publish those for their own models, last I checked. Datacenter buildouts? Water consumption per request? There just isn't enough…
That's only really true if one ignores the possibility of SSD offloading, which effectively opens up inference with far larger models. It's possible that the combination of batched inference and SSD streaming may be…
DeepSeek V4 Pro seems to have significantly lower overhead than GLM 5.2 for the same context size. If the two are about equally smart, that's not a very good look for GLM. E.g. the KV-cache storage for GLM at full…
The Chicago95 theme is great, but https://github.com/B00merang-Project/Windows-Classic (and many other repositories under the same GitHub user) is a nice addition: it has styling for other DEs and a theme for GTK 4. The…
The switch widget is for settings where flipping the switch takes immediate effect, unlike a checkbox where you have to press some "OK", "Submit" or "Apply" button. In Windows 2000/Office UIs, this would've been shown…
> Unfortunately that is nowhere close to being able to run a 750B param model. For something like that, we're getting closer to 1TB VRAM You don't have to run a model from VRAM, or even from a sizeable amount of RAM.…
You don't need that much VRAM unless you're targeting a high-performance deployment that's intended to scale far beyond local use. For a lower-throughput case, you can keep the model weights on SSD at very low cost and…
The problem with batching local LLMs is not any inherent lack of multiple parallel sessions, but rather that local dGPUs lack the VRAM capacity to host KV-cache for several of those at once, whereas unified memory…
> I would not call him a crank > And he writes about his work and himself in grandiose ways, usually comparing himself to Newton and Einstein ... Nothing major has come out of his research okay
> The cost of hardware still needs to dramatically drop for open-weight models to be viable for local usage. Even with the release of things like Nvidia DGX Spark and Ryzen AI Halo, you'd likely want a few of them to…
So, just like Fable? You can shorten the thinking effort to tweak the "slow and expensive" part a little bit, but at the higher end being more meticulous than even Fable is actually a benefit.
Opus 4.8 is quite weak. And GPT-Pro is very much available unlike Fable, it's just not hooked up to the Codex harness yet.
From a strict PL perspective, the Wolfram/Mathematica language is rather based on a term rewriting paradigm. The languages Maude, Pure and TXL would be examples of something that's broadly comparable but more generic.…
Fine tuning happens on top of pretraining, so of course it can "forget" pretrained defaults when warranted by the new data it's being fine tuned on.
You keep building on the last available version? Fine tuning is a whole lot cheaper, easier and more useful than pretraining a model from scratch. It's a complete no brainer.
I'll definitely believe that for video generation models, but those are also very compute-intensive for rather middling results.
I don't want to sound overly dismissive, but for quite a few practical cases pipeline parallelism with micro-batches will be a likely win over tensor parallelism. Of course this inherently comes with a requirement to…
> speculative decoding is a great boon, but mostly for large labs hosting and serving many requests at once, and maybe not so effective for local inference This is somewhat of a nitpicking point, but AIUI speculative…
It's a very nice summary of KV cache storage requirements for the leading open weights models, but ultimately it still shows the DeepSeek V4 series winning by a huge margin on that front. And DeepSeek actually does…
0.05 to 0.1 per sec could still be quite useful if it was the speed for inferring a whole batch of tokens concurrently. Of course this actually requires fairly good SSD read performance (since you need to read a…
Kimi uses INT4 as its native format, there's no such thing as "better than 4-bit precision" for that model. This is in contrast with GLM for which 16-bit precision is native and 8-bit is in common use.
> The warning I would have for everyone is to temper your expectations and read the fine print carefully. The big build in article starts off with a $40K budget and then includes 4 GPUs that are $12K each. For those…
10k context is not a whole lot, this model theoretically supports up to 1M. But the KV cache storage takes up a whole lot more memory capacity at full context than DeepSeek V4 Pro, let alone Flash. (About ~96GB…
AIUI the llama.cpp implementation for this model is still quite half-baked due to missing the support for DSA sparse attention mechanism. This leads to running the model with a different mechanism that it has not been…
Lousy benchmark, they explicitly focus on the easiest tasks to automate for AI (i.e. heavily cherry picked outcomes) and it seems that they don't bother to test anything except just-released proprietary models.
What's "exponential" about AI development? Model parameter counts? Anthropic doesn't publish those for their own models, last I checked. Datacenter buildouts? Water consumption per request? There just isn't enough…
That's only really true if one ignores the possibility of SSD offloading, which effectively opens up inference with far larger models. It's possible that the combination of batched inference and SSD streaming may be…
DeepSeek V4 Pro seems to have significantly lower overhead than GLM 5.2 for the same context size. If the two are about equally smart, that's not a very good look for GLM. E.g. the KV-cache storage for GLM at full…
The Chicago95 theme is great, but https://github.com/B00merang-Project/Windows-Classic (and many other repositories under the same GitHub user) is a nice addition: it has styling for other DEs and a theme for GTK 4. The…
The switch widget is for settings where flipping the switch takes immediate effect, unlike a checkbox where you have to press some "OK", "Submit" or "Apply" button. In Windows 2000/Office UIs, this would've been shown…
> Unfortunately that is nowhere close to being able to run a 750B param model. For something like that, we're getting closer to 1TB VRAM You don't have to run a model from VRAM, or even from a sizeable amount of RAM.…
You don't need that much VRAM unless you're targeting a high-performance deployment that's intended to scale far beyond local use. For a lower-throughput case, you can keep the model weights on SSD at very low cost and…
The problem with batching local LLMs is not any inherent lack of multiple parallel sessions, but rather that local dGPUs lack the VRAM capacity to host KV-cache for several of those at once, whereas unified memory…
> I would not call him a crank > And he writes about his work and himself in grandiose ways, usually comparing himself to Newton and Einstein ... Nothing major has come out of his research okay
> The cost of hardware still needs to dramatically drop for open-weight models to be viable for local usage. Even with the release of things like Nvidia DGX Spark and Ryzen AI Halo, you'd likely want a few of them to…
So, just like Fable? You can shorten the thinking effort to tweak the "slow and expensive" part a little bit, but at the higher end being more meticulous than even Fable is actually a benefit.
Opus 4.8 is quite weak. And GPT-Pro is very much available unlike Fable, it's just not hooked up to the Codex harness yet.
From a strict PL perspective, the Wolfram/Mathematica language is rather based on a term rewriting paradigm. The languages Maude, Pure and TXL would be examples of something that's broadly comparable but more generic.…
Fine tuning happens on top of pretraining, so of course it can "forget" pretrained defaults when warranted by the new data it's being fine tuned on.
You keep building on the last available version? Fine tuning is a whole lot cheaper, easier and more useful than pretraining a model from scratch. It's a complete no brainer.
I'll definitely believe that for video generation models, but those are also very compute-intensive for rather middling results.