I'm not planning to :)). Singulatron is not about scheduling, etc.; it operates at a much higher level of abstraction. To put it in less fancy terms: it's an app, albeit one designed to run on many servers simultaneously.
Are you distributing the LLM generation tasks at a low-level (server 1,2,3,4 each have a GPU and each get 25% of the model + work somehow)? Or is this just a way of interfacing with an existing LLM using a standardized interface on front and back ends?
At the moment the distributed features are entirely missing from the repo, sorry to disappoint on that front despite being part of the readme. It's hectic!
But to answer your question:
I plan to sidestep most difficulties of the distributed story by offloading into the usual distributed tools - horizontally scalable DBs and distributed locks, queues etc.
This will have the added upside of being able to use your favourite tools for which you already have dashboards, alerting etc.
There will be no fancy resource management (an other comment touched in "reinventing kubernetes" which is definitely not a goal at all), but there will be some simple optimizations in terms of where to run which prompt depending on if the model is already running on a node or not etc.
So there will be some knowledge of the nodes' capacity (at least available VRAM etc), but the picking of the prompts from the queue will be a simple unit testable algorithm.
At least this is the plan but it's only half implemented in my mind.
Great job! Can't imagine amount of hours invested into it. How does the offline version perform in different areas comparing with known LLMs (e.g. ChatGPT 4o, LLama, etc)?
10 comments
[ 340 ms ] story [ 2594 ms ] threadOp here. Trying to build a distributed AI platform - lots of work, fairly early phase but I think it's already useful. Hope you'll like it.
Thanks!
Most of them only reinvent 1/10th-1/100th of Kubernetes.
At the moment the distributed features are entirely missing from the repo, sorry to disappoint on that front despite being part of the readme. It's hectic!
But to answer your question: I plan to sidestep most difficulties of the distributed story by offloading into the usual distributed tools - horizontally scalable DBs and distributed locks, queues etc. This will have the added upside of being able to use your favourite tools for which you already have dashboards, alerting etc.
There will be no fancy resource management (an other comment touched in "reinventing kubernetes" which is definitely not a goal at all), but there will be some simple optimizations in terms of where to run which prompt depending on if the model is already running on a node or not etc.
So there will be some knowledge of the nodes' capacity (at least available VRAM etc), but the picking of the prompts from the queue will be a simple unit testable algorithm.
At least this is the plan but it's only half implemented in my mind.
We are running known models like LLama, StableDiffusion etc. So pretty much the same :)
We are not planning on building our own models yet.