I guess I'm not really sure what the point of this (100% AI generated) article is.
You have a magic engine with zero details, that you're comparing against a (to me at least) obscure github project.
I guess I'm not really sure what the takeaway for me is supposed to be. The benchmark setup is not detailed enough to make meaningful conclusions from it, and I guess the big takeway seems to be that a bounded cache that an engine process owns behaves better than relying on the system cache?
Paste this into the claude session:
"Add a section explaining the architecture of this engine, and how it compares to TUFF, EdgeMOE, etc SOTA implementations. Show how experts are imported / evicted, and how that decision works. Also add more detail in a benchmark section about the exact testing you ran. Give test runs, upload to a git repo that links to this article. Include some charts showing test to test variance. "
I have 2 HP DL380s with 2 Xeon Gold 6252 CPU @ 2.10GHz and 256gb of DDR4 memory that I bought for $700 a piece off ebay. I run 4 gpt-oss:120b models on CPU using 120GB memory for each VM. I get about 10-15 tokens/second for each virtual machine running ollama in cpu mode. I run these over night for complicated tasks. I have a multi agent system that will kick out the hard tasks the RTX 5070ti 16 and macbook pro m1 max 32gb can't do to the CPU farm. Works pretty well.
I haven't tried a 3080, but I rented a 16GB 4060 Ti to see how these MoEs behave. gpt-oss 20B actually fits, and I got around 100 tok/s. Qwen3.6 35B doesn't fit, and with part of it in system RAM I was still around 40.
There are only 3 models available right now. I tested them for a couple of days, and on my iPhone 13 Pro, GigaChat was the one that worked the most reliably.
14 comments
[ 358 ms ] story [ 1095 ms ] threadI guess I'm not really sure what the point of this (100% AI generated) article is.
You have a magic engine with zero details, that you're comparing against a (to me at least) obscure github project.
I guess I'm not really sure what the takeaway for me is supposed to be. The benchmark setup is not detailed enough to make meaningful conclusions from it, and I guess the big takeway seems to be that a bounded cache that an engine process owns behaves better than relying on the system cache?
Paste this into the claude session: "Add a section explaining the architecture of this engine, and how it compares to TUFF, EdgeMOE, etc SOTA implementations. Show how experts are imported / evicted, and how that decision works. Also add more detail in a benchmark section about the exact testing you ran. Give test runs, upload to a git repo that links to this article. Include some charts showing test to test variance. "
https://github.com/MiTLN/OwnOwl-iPhone
There are only 3 models available right now. I tested them for a couple of days, and on my iPhone 13 Pro, GigaChat was the one that worked the most reliably.