This is a great idea, but the models seem pretty outdated - it's recommending things like qwen 2.5 and starcoder 2 as perfect matches for my m4 macbook pro with 128gb of memory.
Slightly tangential, I‘m testdriving an MLX Q4 variant of Qwen3.5 32B (MoE 3B), and it’s surprisingly capable. It’s not Opus ofc. I‘m using it for image labeling (food ingredients) and I‘m continuously blown away how well it does. Quite fast, too, and parallelizable with vLLM.
That’s on an M2 Max Studio with just 32GB. I got this machine refurbed (though it turned out totally new) for €1k.
This pretty cool, and useful but I only wish this was a website. I don’t like the idea of running an executable for something that can perfectly be done as a website. (Other than some minor features, tbh even you can enable Corsair and still check the installed models from a web browser).
i wouldn't mind a set of well-known unix commands that produce a text output of your machine stats to paste into this hypothetical website of yours (think: neofetch?)
What I do is i ask claude or codex to run models on ollama and test them sequentially on a bunch of tasks and rate the outputs. 30 minutes later I have a fit. It even tested the abliterated models.
as someone who's very uneducated when it comes to LLMs I am excited about this. I am still struggling to understand correlation between system resources and context, e.g how much memory i need for N amount of context.
Been recently into using local models for coding agents, mostly due to being tired of waiting for gemini to free up and it constantly retrying to get some compute time on the servers for my prompt to process like you are in the 90s being a university student and have to wait for your turn to compile your program on the university computer. Tried mistral's vibe and it would run out of context easily on a small project (not even 1k lines but multiple files and headers) at 16k or so, so I slammed it at the maximum supported in LM studio, but I wasn't sure if I was slowing it down to a halt with that or not (it did take like 10 minutes for my prompt to finish, which was 'rewrite this C codebase into C++')
this is visually fantastic, but while trying this out, it says I can't run Qwen 3.5 on my machine, while it is running in the background currently, coding. So, not sure what the true value of a tool like this is other than getting a first glimpse, perhaps. Also, with unsloth providing custom adjustments, some models that are listed as undoable become doable, and they're not in the tool. Again, not trying to be harsh, it's just a really hard thing to do properly. And like many other similar tools, the maintainer here will also eventually struggle with the fact that models are popping up left and right faster than they can keep up with it.
Read the headline and thought it rescaled LLMs down for your hardware. That would be fascinating but would degrade performance.
Any work on that? Like let’s say I have 64GB memory and I want to run a 256 parameter model. At 4 bit quantized that’s 128 gigs and usually works well. 2 bits usually degrades it too much. But if you could lose data instead of precision? Would probably imply a fine tuning run afterword, so very compute intensive.
LM Studio has an option on model load that I believe does what you describing here: "K Cache Quantization Type" (and similar for "V"). It's marked as experimental and says the effect is basically hard to predict. Never tried myself, though.
This is probably catching ~85% of cases and you can possibly do better. For example, some AMD iGPUs are not covered by ROCm, so instead you rely on Vulkan support. In that case you can sometimes pass driver arguments to allow the driver to use system RAM to expand VRAM, or to specify the "correct" VRAM amount. (on iGPUs the system RAM and VRAM are physically the same thing) In this case you carefully choose how much system RAM to give up, and balance the two carefully (to avoid either OOM on one hand, or too little VRAM on the other). But do this and you can pick models that wouldn't otherwise load. Especially useful with layer offload and quantized MoE weights.
As a few others have noted already - this should just be a website, not a CLI tool. We can easily enter our CPU, RAM, GPU specs into a form to get this info.
The "biggest model that fits" instinct is just wrong now. Compact models routinely beat massive predecessors from 12 months ago. Scaling laws only reliably predict pre-training loss anyway, not how the model actually performs on your task. Dug into the research behind this: https://philippdubach.com/posts/the-most-expensive-assumptio...
28 comments
[ 2.6 ms ] story [ 45.5 ms ] threadCan I just submit my gear spec in some dropdowns to find out?
That’s on an M2 Max Studio with just 32GB. I got this machine refurbed (though it turned out totally new) for €1k.
FTFY.
Sounds like a fun personal project though.
Been recently into using local models for coding agents, mostly due to being tired of waiting for gemini to free up and it constantly retrying to get some compute time on the servers for my prompt to process like you are in the 90s being a university student and have to wait for your turn to compile your program on the university computer. Tried mistral's vibe and it would run out of context easily on a small project (not even 1k lines but multiple files and headers) at 16k or so, so I slammed it at the maximum supported in LM studio, but I wasn't sure if I was slowing it down to a halt with that or not (it did take like 10 minutes for my prompt to finish, which was 'rewrite this C codebase into C++')
It’s a simple formula:
llm_size = number of params * size_of_param
So a 32B model in 4bit needs a minimum of 16GB ram to load.
Then you calculate
tok_per_s = memory_bandwidth / llm_size
An RTX3090 has 960GB/s, so a 32B model (16GB vram) will produce 960/16 = 60 tok/s
For an MoE the speed is mostly determined by the amount of active params not the total LLM size.
Add a 10% margin to those figures to account for a number of details, but that’s roughly it. RAM use also increases with context window size.
How do you benchmark them? This would be awesome to implement at the page as well. I will link to this project at https://mlemarena.top/
Any work on that? Like let’s say I have 64GB memory and I want to run a 256 parameter model. At 4 bit quantized that’s 128 gigs and usually works well. 2 bits usually degrades it too much. But if you could lose data instead of precision? Would probably imply a fine tuning run afterword, so very compute intensive.