I note the lack of performance information. I can only imagine it's much, much, slower than any other way to run a larger model (including, e.g. using system RAM and streaming some stuff from disk). Consumer networks, even 10gbit ethernet, are slow as hell compared to local RAM and even disks.
Are we talking 1 token per second for a split model? Less?
Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
I’m one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes.
This is surprisingly similar to what I'm trying to build (at least at the network level), down to the use of iroh. I have not attempted model splitting though, and the angle is slightly different. Major props for this, it seems like a great project. The more we can push local inference, the better for everyone. Interested to see if there's collaboration opportunities. https://github.com/dsegovia90/cocompute
I’ve been curious what a polymorphic botnet that runs one (or multiple) distributed LLMs would be capable of doing. The idea would be to evolve the botnet delivery and payload using the clustered compute of all hosts in the botnet to run LLMs that guides the evolution of various botnet clusters. Bad cluster morphs get caught and cleaned off and bad delivery methods never spread, but the best versions survive to continue to grow.
What I envisioned for how it works is fairly similar to this, QUIC can actually be more difficult to detect than it seems since it’s very dynamic.
Does this have intelligent expert handling for high parallelism MOE? You can get very high throughput for highly parallel MOE if you can mix different queries at each expert stage, but if the batch has to run together for the whole pipeline you get a parallelism loss instead of gain.
@null not as yet, but did a tiny bit of research before. If i understand it this would help mainly with batching (ie concurrent sessions/users/turns etc) to keep the pipeline more busy (vs a batch of a fixed size going together), which is interesting, but most work so far has been on latency hiding for straight line performance to start.
We did have a version of things which used expert islands for MoE parallel alternative (I think that has been mostly scrubbed from the code). It showed early promise by having trunk+hot experts together, but as models got larger, it made both that very large on its own but also didn't seem to work as well (or we just weren't good at grouping experts), the idea being to really side step latency and route sessions to those islands. It felt like as models scaled it didn't stay smart (as diverse experts activated more than I thought they would). Could be our mistake (was an exciting possibility though - if you don't mind accepting some loss).
tldr; yes this seems a very nice enhancement for smarter batching/keeping things busy and seems like most larger models we look at are MoE!
The first picture "gpu rig", "laptop", "server", "cloud node, etc made me realize how little compute I have. I don't have a laptop with 24GB VRAM or a workstation with 96GB. I think if I convinced all of my friends to run LLMs on their gaming PCs, I don't I would have the total VRAM in the picture.
As an aside, I saw this post mentions a public mesh, but I couldn't find any more information.
I've been looking for similar distributed computing style LLM, and I found AI Horde and a few other smaller efforts like one from Aphrodite people and distributed training from Nous Research.
AI Horde seems to be the biggest of them all. Their API speaks KoboldCPP text completion (not even chat completion). It seems that the community (or at least the active people) strongly prefer it this way because the API exposes more tunables than chat completions, which for roleplay use seems to result in better result. I don't know what else you can use AI Horde for anyway since all other use cases likely will require tool use. Just this week I was set out to improve their OpenAI bridge to support chat templates and response parsing. We'll see if I could get it deployed officially then you might be able to use it to code, although you'll have to use RP models.
I think Horde do have a lot more abuse prevention. Workers needs to have 1 week of cumulative uptime to be considered trusted to prevent brigading - users can opt into trusted workers only. Running a worker give you kudos which is required for >512 max tokens generations and also free requests get bumped to last.
I spent a while trying to get mesh-llm running, but none of the installable llama.cpp builds worked with my older gpu. It looks like it should be able to be used to proxy an external llama.cpp service, but I had no luck setting that up either. Seems very cool, but definitely some rough edges.
the https://query.mt/ project has been using iroh based mesh for a while. maybe give it a go, especially if you wanna use your mesh models on your mobile phone as well.
They might want to make their docs less confusing. https://query.mt/showcase/gpu-sharing/ says "no API key needed", and talks about running llama.cpp on a friend's GPU. No API keys are needed for this. But then the page goes on about who "holds" API keys, and where API keys "stay", where API keys are "locked", etc.
44 comments
[ 4.1 ms ] story [ 67.7 ms ] threadAre we talking 1 token per second for a split model? Less?
Edit: Found a number. On the models list, Qwen 235B A22B says "MoE 235B/22B, proven at 16 tok/s across 2 nodes". They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close.
Numbers in this example are arbitrary. How does it actually work? What if the model’s number of layers is 33, or 34?
Is there a document explaining all constraints of this implementation?
What I envisioned for how it works is fairly similar to this, QUIC can actually be more difficult to detect than it seems since it’s very dynamic.
We did have a version of things which used expert islands for MoE parallel alternative (I think that has been mostly scrubbed from the code). It showed early promise by having trunk+hot experts together, but as models got larger, it made both that very large on its own but also didn't seem to work as well (or we just weren't good at grouping experts), the idea being to really side step latency and route sessions to those islands. It felt like as models scaled it didn't stay smart (as diverse experts activated more than I thought they would). Could be our mistake (was an exciting possibility though - if you don't mind accepting some loss).
tldr; yes this seems a very nice enhancement for smarter batching/keeping things busy and seems like most larger models we look at are MoE!
Say a distributed inference for image processing, SDR, local weather monitoring etc. These will run on mediocre specs and produce dependable output.
Nicely done OP.
As an aside, I saw this post mentions a public mesh, but I couldn't find any more information.
AI Horde seems to be the biggest of them all. Their API speaks KoboldCPP text completion (not even chat completion). It seems that the community (or at least the active people) strongly prefer it this way because the API exposes more tunables than chat completions, which for roleplay use seems to result in better result. I don't know what else you can use AI Horde for anyway since all other use cases likely will require tool use. Just this week I was set out to improve their OpenAI bridge to support chat templates and response parsing. We'll see if I could get it deployed officially then you might be able to use it to code, although you'll have to use RP models.
I think Horde do have a lot more abuse prevention. Workers needs to have 1 week of cumulative uptime to be considered trusted to prevent brigading - users can opt into trusted workers only. Running a worker give you kudos which is required for >512 max tokens generations and also free requests get bumped to last.