HN Explorer

Show HN: Lumabri – What if LLMs worked like Napster? (github.com)

9 points by vforno ↗ HN
A while ago I started working on Colibrì to see if it was possible to run huge LLMs on a normal computer. The project grew far beyond what I expected, thanks in large part to the HackerNews community.

That led me to a new question:

What if we stopped thinking about one computer?

This is the idea behind Lumabri.

Instead of requiring a single machine to store and run an entire huge model, Lumabri treats a network of normal computers as a shared pool of resources.

One machine might provide disk space, another compute, another a different part of the model. If a required block or expert isn’t available locally, the system can retrieve or execute it on a peer.

This is particularly interesting for Mixture-of-Experts models. A model can have hundreds of billions of parameters, while only a fraction are activated for each token. Rather than moving huge expert weights over the network, Lumabri can send the small activation to a peer that already has the expert and let it execute it.

The goal is for machines to contribute whatever resources they can afford while using the swarm for the rest.

The idea is very much inspired by peer-to-peer systems: users are the infrastructure.

There are obviously major challenges, especially network latency and security. I’m experimenting with peer verification, SHA-256 verification, signed model state, replica selection, failover, and deterministic execution.

Lumabri is still an early experiment. I don’t have a datacenter or a huge GPU cluster, so I’m building it with the hardware I have and trying to find out whether the idea actually makes sense.

With Colibrì I asked:

Can one normal computer run a huge LLM?

With Lumabri I’m asking:

What if many normal computers could become one huge computer?

Feedback welcome.

Repo: https://github.com/JustVugg/lumabri

4 comments of 10

[ 0.37 ms ] story [ 17.3 ms ] thread
nathan_compton
Here is some feedback: don't put slop in your readme. It's unreadable.
drivingmenuts
There needs to be some sort of safeguard to prevent people from abusing this and running queries to do something an owner objects to. I'd feel horrible if I discovered that someone used my AI to figure out how to kill someone. Another problem would be privacy. My highly personal queries would be available to whomever ran that query for me thru logging.

This is not to say it's a bad idea, but it needs some thought toward protecting users or at least acknowledgment and warnings that their information might be available to unintended eyes.

esafak
LLMs are already slow on one machine. Now you are going to add network calls to it?
hazard
I recommend reading some foundational materials on federated learning and distributed inference. The fundamental issues are (1) compute (2) latency and (3) security.

On the compute side, you can't load most individual experts into even high-end consumer-grade cards (e.g. RTX 5090) because the weights are simply too big.

On the latency side, you need to ship the activations and wait for the somewhat underpowered cards to actually do the matrix multiplications, then send the results back. This means each token takes hundreds of milliseconds or even more, which is borderline unusable.

Finally if you're running a P2P network, the only way to verify that peers aren't lying to you is to run duplicate calculations - I don't see how SHA-256 or signed model state help here unless I'm missing something (What are you calculating the hash of? What exactly is signing attesting to?).

This is one of the few places where borrowing ideas from cryptocurrencies actually makes sense - you could imagine a system where peers place bonds and forfeit them if they lie about calculations. You can look at projects like Bittensor and others for how these kind of things are currently implemented.