Getting GLM 5.2 running on my slow computer
The dense part (attention, shared experts, embeddings—~17B params) stays resident in RAM at int4 (~9.9 GB); The 21,504 routed experts (75 MoE layers × 256 experts + the MTP head, ~19 MB each at int4) live on disk (~370 GB) and are streamed on demand, with a per-layer LRU cache, an optional pinned hot-store, and the OS page cache as a free L2.
The engine is a single C file (c/glm.c, ~1,300 lines) plus small headers. No BLAS, no Python at runtime, no GPU.
No GPU or serious hardware because I don't have that hardware so I can't test it on hardware that is more powerful than my computer.
Colibrì is a one-person project, written and tested entirely on a 12-core laptop with 25 GB of RAM — the numbers above are the ceiling of what I can measure at home. If this project is useful or interesting to you and you'd like to support its development (better hardware test translates directly into a faster engine for everyone: real NVMe scaling data, bigger pinned caches, int2/int3 quality sweeps on real benchmarks), you can:
star the repo and share it; open issues with benchmark numbers from your hardware; reach out via GitHub issues if you'd like to sponsor development or donate hardware.
Every contribution, from a datapoint to a disk, moves the ceiling.
Any feedback are welcome!
Repo: https://github.com/JustVugg/colibri
215 comments
[ 3.0 ms ] story [ 108 ms ] threadAmazing job!
https://github.com/JustVugg/colibri#ssd-wear-warning
0.05 to 0.1 tok/s on the other hand, as reported in the URL for the lowest class of hardware, isn't really usable for much.
These days, can "ordinary people" afford 24GB of ram and half a TB of NVME ssd?
sigh
32G RAM, nvme 1TB, core ultra 258V.
Looking at the prices now... Wow, was I lucky.
Tried some of the 7b models locally, more than usable, around 30token/sec, not with the NPU, but using the ARC integrated GPU.
I am a noob for this, but I guess it's time to experiment more with this local setup
You can, right now, buy a brand new Mini-PC at or above this spec for $600 at retail [1]
Of course, if you want it in a desktop format with a much faster CPU, its going to cost you more.
[1]: https://www.amazon.com/GMKtec-M6-Ultra-Upgraded-Computers/dp...
The hard thing is always keeping complexity low and being ZeroOps.
https://github.com/skorokithakis/symphony
I have a 3 Mac Studio set up and built an IDE / harness (propelcode.app) and would be interested in contributing if you’re open to collaboration
Docs:
https://mininote.ink/docs/mcp-docs
https://tangled.org/clee.sh/posthorn
This is based on the observation that the medium-sized open weight models (~20-35b) are very able to one-shot smaller discrete tasks but seem to lose their way project managing themselves through larger tasks that have multiple steps.
(I want to spend no more than $10k. And I want to run a model comparable to today’s SOTA.)
But if we can believe you that it's doing what a Claude model was doing a year ago then I'd say: OMG no I really never want to go back to that level of frustration getting an agent to do what I want it to do.
While it probably won't matter enough to change your mind, remember that you've gotten better at extracting value from all models than you were a year ago - plus the harnesses and other tools have gotten a lot better too.
What could be more valuable than outputting the exact thing you asked for?
And if it gets it wrong, at least you have 80% scaffolding that you can ask to iterate on.
Hoping you’ll get 100% you asked for is going to fail. Hell… you can even achieve this with humans!
Expectations seem to be rising at a faster rate than models can improve.
Or buy one on eBay with 512GB that has half its slots populated and then buy the matching 512GB kit to add.
In my experience with rig half that cost, entire exercise of running coding models locally has been a huge disappointment.
Cost/Value when compared to cloud services is just not there, but I see the merit for those who value privacy over quality of output and want a backup of huge condensed corpus of data within their control.
Kudos to OP though, They had clear goals and they achieved it.
Or people who want or need to run an uncensored (abliterated) gguf file to deal with controversial topics that a paid LLM service will refuse to work with or ban you for.
The largest high performance compute ec2 offering, the c9g.metal-48xl , maxes out at 384GB RAM and already costs a shitload.
The m9gd.48xlarge and m9gd.metal-48xl both have 768GB RAM and I cringe to think what they cost monthly.
Yes, it's a boatload of cash, but that's a €13,000 GPU and €20,000 of RAM at present prices. There is a segment of businesses where a fixed €28k/year bill is going to be preferred over plonking down €40k for a (theoretically) depreciating asset and ongoing colocation costs.
[1]: https://www.hetzner.com/dedicated-rootserver/gex131/
And yet basically all AWS customers are doing exactly that. Turns out that making CAPEX "someone else's problem" is worth quite a lot to many businesses
can we trust any US based service to guarantee privacy and confidentiality? especially to us european frienemies?
Insert your dedicated hosting provider of choice for 'AWS' (somewhere like Hetzner will be cheaper anyway).
But in general, AWS hosts are yours, running your code, with your security policies enforced. Sure, the US government can silently subpoena the contents thereof, but aside from that fairly extreme case, it's not like AWS is handing your data over to 3rd parties.
It won't be fast at all, for certain, but it'll have enough memory to prove a configuration and be able to really use gargantuan GGUF format LLMs in the latest compiled llama-server. Re: electricity, I pay the equivalent of $0.07 ro $0.09 USD per kWh so it's not an extreme burden to have a theoretical 500W server running. Something like $35 to $50 of electricity a month if it's 500W 24x7.
So for optimal speed the models must be quantized in this format.
It is very likely that with INT8 models those CPUs are fast enough so that the inference throughput is limited by the memory bandwidth (384-bit interface to DDR4-2933 per socket, i.e. 282 GB/s for both sockets).
The memory throughput for such an old server is very similar to an AMD Ryzen Halo, NVIDIA DGX Spark or Apple M5 Pro, but it has much more memory.
The inference speed should be very similar to those, but with bigger LLMs.
The question is, will you want to run a model comparable to today's (meaning 2026) SOTA in 2028? Humans always want the latest shiny LLM model.
It says so in the quoted text, yes.
With only 1 PCIe 5.0 SSD, the reading throughput is still significantly more than 10 times faster than on author's system.
So it is likely that inference speeds around 1 token/s are achievable on something like a NUC mini-PC.
The benchmarks I'm seeing for many of them don't really make me think that a pair of consumer grade NVME SSD you could fit in a mini-PC or mini-itx size desktop would, added together, be capable of 20GB/s reads.
If there were such a slow SSD, it would not make sense to buy it instead of a cheaper PCIe 4.0 SSD.
For PCIe 4.0 SSDs, I have seen a very large number of benchmarks where the SSDs achieved read speeds close to the theoretical limit, i.e. around 7 GB/s.
Searching now randomly for recent SSD reviews, I find many reviews for "SanDisk WD_BLACK SN8100", which achieve between 13 GB/s and 15 GB/s reading speed, which is better than most other consumer PCIe 5.0 SSDs.
Of course, if you write a very simple program that invokes something like "fread" or "read" in a loop, you will not reach such speeds. Achieving a SSD throughput close to the limit requires a more complex program that can ensure that the SSD controller is permanently busy with pipelined read commands.
If you want a CPU-only machine with 512GB to 1024GB of RAM, despite extreme cost rises, there are still some great options out there from companies selling ex-lease stuff that's 3, 4, 5 years old. It'll be loud as hell under full CPU load when running inference, so if you plan to use it at home, put it in your garage or basement or laundry room or somewhere similar on the far end of a network cable.
The software that OP has published appears to be specifically designed to hold only the active parameters in RAM (<100GB) and read content off local NVME SSD as needed on the fly. All that NVME SSD read wouldn't be necessary if you can hold the model in RAM, even in the absence of any GPUs.
Nice work!
One further step is predicting which experts will be needed next token / next layer. LRU does this okish. But a learned projection from the hidden state can do better. Or even a simple correlation from past activated experts. Expert usage is heavily skewed.
I also know I did some things that would actually make the perf worse to, like I believe I also had AI mmap the KV Cache to make sure to runs under any circumstance. For actual optimizations based on what I currently know, I'm probably going to try and get the llm running under my igpu on my laptop with persistent shader that has some kind of inbuilt request mechanism. That way the weights that are loaded can be used as fast as possible.
For the expert prediction, I assume I could use the medusa paper as kind of a kick off point for that since I'm already using it to try and predict the next 4 tokens. Doing verification on those 4 tokens is about as much as I can do though since it started to thrash on loading the experts. So some method of predicting even more tokens, but then batching together those with the same experts would probably yield slightly better results in this weird case.
Note: All of my tests have been around programming since that's the use case I'm interested in. I don't actually know if this would preform well in other cases (and anything more broad than that I assume would be slower.)
One option is locking the pages. But for that size you need extra privileges.
I experimented with some options. For example: one problem with io_uring is that it still reads to page cache so your reads gets copied in memory after they landed. Now if you pass O_DIRECT that does not happen, but it has its own can of worms.
For full transparency: I had opus write the io_uring layer into llama.cpp for me. And it yielded something slightly short of a 2x tok/s speedup vs simple mmaping. Also I noticed that disabling the warmup and initial test dramatically increases startup time.
The medusa paper looks interesting. My work was a few months ago, multi-token decoding was not a thing then.
Nevertheless, I do not think that "this is not really possible with mmap, even with madvise".
If the kernel is not eager enough to prefetch pages from the SSD when you use madvise with MADV_WILLNEED, then you can use madvise with MADV_POPULATE_READ, which should force the reading of the pages that you request.
Using madvise with MADV_POPULATE_READ for appropriate page ranges at the right moments should be able to provide a performance not much lower than when using explicit asynchronous reads.
Posting on the Hacker News forum is not worth risking your, or anybody else's, life.
Basically I kept needing an inference engine that could stream weights in and out as needed in an LRU manner. So I ended up vibe coding this thing that accepts a `--vram-budget` and stays under it (mostly). It turns out moving mmap'd bytes in and out of VRAM is way cheap compared to compute. Coupled with some pipelining/double-buffering, I almost always end up compute bound not memory bound. Granted I use way smaller models heh.
I'm also curious if you can speed this up by using many disks in parallel to increase bandwidth.
>SSD Wear Warning
> Cold starts are heavy on random reads (~11 GB/token). Reads themselves are safe, but the OS page cache can generate writes. Heavy use may accelerate wear on cheaper SSDs. Use with caution and monitor your drive health.
Hmm, maybe a safe way to do this would be to make a separate partition for the model weights, and set them to read-only? Not sure how the page cache works, if it's like per partition or per disk. If it's per disk, maybe you could have a read-only data.iso formatted as a partition and mount it as a disk?
[0] https://github.com/JustVugg/colibri#ssd-wear-warning
You don't need to be superstitious here: disk activity, including writes in particular, can be measured. E.g. `iostat` or `vmstat` on Linux.
Not hijacking anything as this project is amazing.