That seems to be the point: you leverage existing workstations on your corporate network and run compute batches at night while they're idle. You don't have to buy any extra servers for your low-priority AI workloads.
We were doing this decades ago with engineer's HP C3600s under their desks. It wasn't AI, just low-priority HPC / fluid stuff batched out to hundreds of workstations.
That was the original pitch for Ian Murdock's Progeny startup in the late 90s: a Linux-based distributed filesystem for corporate desktops to boot into overnight for distributed computing.
And on IRIX, SunOS, and DG-UX boxes too. Once Linux existed for a bit, they were added to the fold. The software to do this - Condor - is still maintained.
Minions is an internal enterprise system already in use at Apple. It pools idle Mac GPUs into a shared compute cluster for ML inference.
There is no distributed execution of a single LLM across multiple GPUs using tensor parallelism here. Instead, it relies on task-level parallelism-breaking down a large workload into independent subtasks that run across separate machines. For example, generating embeddings for a million documents.
This pattern has been implemented across hundreds if not thousands of different organizations (and between organizations). I have a patent on data distribution strategies from my time at Google running a large (>1M cores 24/7 for 3 years) idle cycle harvesting project.
it's a good idea but you often end up with huge tail latencies, slight incompatibilities due to fleet heterogeneity, and a database/scheduler management problem.
I guess for companies this can be nice? Run some inference in the off-hours on the machines that you already have. Could be useful for analytical tasks that don't require to be instant. On the technical side this doesn't seem that groundbreaking for me though.
My warm take is that Apple is leaving a lot of money on the table by not investing in creating an independent supply chain for their chips and building their own cloud. They arguably had the first mover advantage when it came to general purpose arm compute; all they had to do was flood the market and make it easy for anyone to rent it. Not only would they corner a part of the market but perhaps ease demand pressure on other chip manufacturers too.
Apart from straight up buying TSMC's or Samsung's fabs I don't see a way of building a vertically integrated high-end chip supply chain for Apple. At least not on timeframes measured below a decade. And on those time frames the current supply issues will probably be resolved by companies who put all their effort into it instead of just tweaking their bottom line.
I actually see the logic there: Apple as a “harmless consumer device company” is probably one will remain very geographically diversified in the long run, between the China and America markets.
(Interestingly enough, I think PRC and ROC are combined in public financials.)
They don't get enterprise, A/UX, mkLinux, OS X Server, Mac Pro tower, all gone now.
Swift's main reason to go open source and have good Linux support was naturally for app developers to be able to use it on server side, which naturally wasn't going to be macOS.
I macOS wasn't required for developing iDevices apps, they would surely sell even less desktops.
My guess is that they have challenges scaling the production anyway (6-16 weeks wait time for Studios), and also they would leave money in the table if they just released chips. I would assume that once they scale the production they will get back to the server game - MacOS server racks, or something more inventive like MacOS supercomputers running on a ton of prev gen chips ;)
At this point, I’m sure bankrolls a significant amount of R&D in new MSes and build-out capital, in return for guaranteed capacity on the first production lines. I’m not sure the extra juice of outright ownership is worth the squeeze.
Reading the “paper”, this is just a job runner built on Ollama.
Exo, and the teams at Apple that built the foundations Exo uses, actually use modern distributed compute.
I don't get it. In the paper it sounds like it's "just" a CRUD DB, then you have some fleet of Macs, and they store inputs and outputs in S3. What's the innovation here? Not trying to poo on this, but it looked like the most basic and obvious approach.
Apple has released some small models previously that seemed genuinely great, along with some quality research/efforts. I’m just not seeing the same level of depth here, am I missing something?
until you get a teen obsessed with 3d rendering then they're up all night finding a way to convert manga into 3d scenes and your old laptop is running 100% cpu trying to keep up. xDD
There is a startup working on this called Distributive I used to work for - their compute nodes are based on js/wasm runtimes so you just need a web browser to turn a device into a compute node...
> if we finally had an accessible consumer-accessible way to turn a network of computers on a network into a supercomputer
There's plenty of devices operating in giant supercomputer networks accessible for cheap to consumers! Unfortunately the consumers in question are spam farms, bot networks, ddos rings, and other mostly criminal types.
It would be cool if similar networks could be used for good instead. Earning credits providing compute that could redeem for spurts of fast parallel compute.
It's not a novel idea, but its provable that someone can take this same concept and do something truly groundbreaking. I wonder if there's value in the idea that you can join a supercomputer network of MAC/Nvidia GPUs as a way of getting off cloud and into "our cloud". As long as you offer compute, you have credit into the system to be able to support your own workloads that can't be done on just your hardware. Need to address concerns about privacy, abuse, etc, but Apple's idea here has some merit if applied right.
Would love to build something like this but for classification models that run on CPU. Like a “Classify @ Home” p2p compute network, to which people could contribute their machines and see what work is being done
There are a lot of parallels between the DGX (and clones) series devices and these Apple devices, maybe it would be possible to rope in the DGX boxes as well. There's lots of them out there and I'm pretty sure not all of them have high utilization.
The DGX are certainly making a case for a new workstation platform to compete with Apple. The user experience is on par, and the local model capabilities are very appealing in many use cases. I've been happy to have had a pair to play with anyway .. I sure hope the price comes down and Nvidia expand the concept ..
For now the prices seem to be only going up. It is complete madness. At the same time: I'm kind of happy about that. It means that lots of people and small to mid sized organization have a very nice incentive to make low end hardware perform better. If you look at how much effort is being poured into model compression and clever VRAM management schemes on the low end it is quite impressive. And plenty of those are compounding.
As with all similar SETI@home approaches, this paper does not address the energy cost.
Yes, we can distribute tasks to idle machines in homes and on office desktops, but it is often much more expensive than sending it to a data center where power is cheaper and the hardware is more efficient.
104 comments
[ 0.26 ms ] story [ 10.2 ms ] threadhttps://pages.cs.wisc.edu/~danb/ref-v7.7.5/ref.html
Basically SETI for AI workloads? At least, this is the impression the landingpage left me with.
I am a bit lost, tbh.
There is no distributed execution of a single LLM across multiple GPUs using tensor parallelism here. Instead, it relies on task-level parallelism-breaking down a large workload into independent subtasks that run across separate machines. For example, generating embeddings for a million documents.
https://developer.nvidia.com/gpudirect
And this is because I can't be bothered to dive into older papers, so recent NVIDIA stuff will do.
*: hell my own compiler is moving to multi-GPU on a single computer and networked devices are the obvious next step.
it's a good idea but you often end up with huge tail latencies, slight incompatibilities due to fleet heterogeneity, and a database/scheduler management problem.
What is it actually like?
(Interestingly enough, I think PRC and ROC are combined in public financials.)
Also they have the ability to put in sufficiently large amount purchase orders to buy up whatever capacity they want.
Even at ludicrous markup prices from TSMC, ifs going to be orders of magnitude cheaper than in house a fab.
Swift's main reason to go open source and have good Linux support was naturally for app developers to be able to use it on server side, which naturally wasn't going to be macOS.
I macOS wasn't required for developing iDevices apps, they would surely sell even less desktops.
Maybe their chip team is already on the work, or already made, a server chip for AI purposes used internally, and doesn't have to talk about it yet.
(As much as conglomerates are fun…)
Apple has released some small models previously that seemed genuinely great, along with some quality research/efforts. I’m just not seeing the same level of depth here, am I missing something?
nothing new here.
https://arxiv.org/pdf/2610.12970
Shortest promoted paper I've seen from a FAANG.
I would definitely opt out.
Is Apple going to compensate the owners?
There's plenty of devices operating in giant supercomputer networks accessible for cheap to consumers! Unfortunately the consumers in question are spam farms, bot networks, ddos rings, and other mostly criminal types.
It would be cool if similar networks could be used for good instead. Earning credits providing compute that could redeem for spurts of fast parallel compute.
Building the classification part now :) https://jeffyclassify.com/
Yes, we can distribute tasks to idle machines in homes and on office desktops, but it is often much more expensive than sending it to a data center where power is cheaper and the hardware is more efficient.