> We burned months trying (and ultimately failing) to get Nvidia’s host drivers working to map virtualized GPUs into Intel Cloud Hypervisor... We think there’s probably a market for users doing lightweight ML work getting tiny GPUs. This is what Nvidia MIG does, slicing a big GPU into arbitrarily small virtual GPUs. But for fully-virtualized workloads, it’s not baked; we can’t use it. Near as we can tell, MIG gives you a UUID to talk to the host driver, not a PCI device.
Apparently this is technically possible, if you can find the right person at Nvidia to talk about vGPU licensing and magic incantations. Hopefully someone reading this HN front page story can make the introduction.
We (I) spent a lot of time talking to several different teams at Nvidia about this. We were able to get VFIO vGPUs to the point where guest libraries would recognize them, but the process fell apart in the guest/host licensing dance, and we weren't really OK with the idea that there'd be a phone-home licensing dance every time a Fly Machine started. Unlike GPU enablement at GCP or AWS, the core DX of a Fly Machine is that it stop/start very quickly; think of it as a midpoint in the design space between Lambda and Fargate. This is what we're talking about when we say it's hard to fit GPUs into our DX.
IIRC, (a) and (c), which I think sort of implies (b)?
Really what we'd have wanted to do would have been to give Fly Machines MIG slices. But to the best of my understanding, MIG is paravirtualized; it doesn't give you SR-IOV-style PCI addresses for the slices, but rather a reference Nvidia's userland libraries pass to the kernel driver, which is a dance you can't do across VM boundaries unless your hypervisor does it deliberately.
Hypothetical scenario for Nvidia licensing of fast-start microVMs:
1. Instead of blocking VM start for license validation, convert that step into non-blocking async submission of usage telemetry, allowing every VM to start instantly. For PoC purposes, Nvidia's existing stack could be binary patched to proxy the license request to a script that isn't blocking VM start, pending step 2 negotiation.
2. Reconcile aggregate vGPU usage telemetry from Nvidia Fly-wide license server (Step 1) with aggregate vGPU usage reports from Fly's orchestration/control plane, which already has that data for VM usage accounting. In theory, Fly orchestration has more awareness of vGPU guest workload context than Nvidia's VM-start gatekeeping license agent, so there might be mutual interest in trading instant VM start for async workload analytics.
Sure. We also could have virtualized CUDA ourselves, used MIG on the host-side, and done a proxy PCI passthrough driver thing in Cloud Hypervisor. I think we could have gotten it to work. But it would have been a huge lift. I'm glad we didn't try.
> Alternatively, we could have used a conventional hypervisor. Nvidia suggested VMware (heh). But they could have gotten things working had we used QEMU. We like QEMU fine, and could have talked ourselves into a security story for it, but the whole point of Fly Machines is that they take milliseconds to start.
Someone could implement virtio-cuda (there are PoCs on github [1] [2]), but it would be a huge maintenance burden. It should really be done by Nvidia, in lockstep with CUDA extensions.
Nvidia vCS makes use of licensed GPGPU emulation code in the VM device model, which is QEMU in the case of KVM and Xen. Cloud Hypervisor doesn't use QEMU, it has its own (Rust?) device model, https://github.com/cloud-hypervisor/cloud-hypervisor/blob/ma...
So the question is, how to reuse Nvidia's proprietary GPGPU emulation code from QEMU, with Cloud Hypervisor? C and Rust are not friends. Can a Firecracker or Cloud Hypervisor VM use QEMU only for GPGPU emulation, alongside the existing device model, without impacting millisecond launch speed? Could an emulated vGPGPU be hotplugged after VM launch?
There has been some design/PoC work for QEMU disaggregation [3][4] of emulation functions into separate processes. It might be possible to apply similar techniques so that Cloud Hypervisor's device model (in Rust) process could run alongside a QEMU GPGPU emulator (in C) process, with some coordination by KVM.
If this approach is feasible, the architecture and code changes should be broadly useful to upstream for long-term support and maintenance, rather than custom to Fly. The custom code would be the GPGPU emulator, which is already maintained by Nvidia and running within QEMU on RedHat, Nutanix, etc.
It would also advance the state of the art in security isolation and access control of emulated devices used by VMs.
> Someone could implement virtio-cuda (there are PoCs on github [1][2]
Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C?
> how to reuse Nvidia's proprietary GPGPU emulation code from QEMU
If it has been contributed to QEMU, it isn't GPL/LGPL?
> Could an emulated vGPGPU be hotplugged after VM launch
gVisor instead bounces ioctls back and forth between "guest" and host. Sounds like a nice, lightweight (even if limited & sandbox-busting) approach, too. Unsure if it mitigates the need for the "licensing dance" tptacek mentioned above, but I reckon the security posture of such a setup is unacceptable for Fly.
NVIDIA support is not special as far as QEMU is concerned—the special parts are all in their proprietary device driver, and they talk to QEMU via the VFIO infrastructure for userspace drivers. They just reimplemented the same thing in Cloud Hypervisor.
Red Hat for one doesn't ship any functionality that isn't available upstream, much less proprietary, and they have large customers using virtual GPU.
Not NVIDIA; fly.io reimplemented the parts that CH didn't already have. I know that CH is developed on GitHub but I don't know whether the changes are public or in-house.
That said, the slowness of QEMU's startup is always exaggerated. Whatever they did with CH they could have done with QEMU.
It is quite possible, bit surprised a minor systems detour would burn months but alas. This inspires me to figure out how to sell it, sounds expensive...
FWIW, I'd consider this good publicity. I have no use for GPUs in the cloud (at least not at the prices that they're available at (in general, not just on fly)). So if fly is moving there effort towards things I actually need (like managed databases) then that's going to increase my confidence in them as a platform quite a bit.
I spent a month setting up serverless endpoint for a custom model last year with Runpod. It was expensive and unreliable, in addition to long cold boot times. The product was unusable even as a prototype, to cover the costs, I'd have to raise money first.
In a different product, I was given some Google Cloud credits, which unlocked me to put the product in front of customer. This one also needed GPU but not as expensive as the previous. It works reliably and it's fast.
Personally, I had two use cases for GPU providers in past 3 months.
I think there's definitely demand for reliability and better pricing. Not sure Fly will be able to touch that market though as it's not known for both (stability & developer friendly pricing).
P.S If anyone is working on a serverless provider and want me to test their product, reach me out :)
Ironically GCP and AWS GPUs are so overpriced that getting even half the number of credits from Runpod is like a 4x increase in "GPU runway", especially with .44/hr A40s.
Yeah but the quality on Runpod is not reliable enough for productionizing it. Do you know a product that works reliably and it's built with Runpod serverless?
I feel like these guys are missing a pretty important point in their own analysis. I tried setting up a ollama LLM on a fly.io GPU machine and it was near impossible because of fly.io limitations such as:
1. Their infrastructure doesnt support streaming responses well at all (which is important part of the LLM experience in my view)
2. The LLM itself is massive, and cant be part of the docker image I was building and uploading. Fly doesnt have a nice way around this, so I had to setup a whole heap of code to pull it in on the fly machines first invocation, which doesnt work well if you start to run multiple machines. It was messy and ended up with a long support ticket with them that didnt get it working any better so I gave up.
I mean, yes? Managing giant model weight files is a big problem with getting people on-demand access to Docker-based micro-VMs. I don't think we missed that point so much as that we acknowledged it, and found some clarity in the idea that we weren't going to break up our existing DX just to fix it. If there were lots and lots and lots of people trying to self-host LLMs running into this problem, it would have been a harder call.
What kind of issues did you have with streaming? I also set up ollama on fly.io, and had no issues getting streaming to work.
For the LLM itself, I just used a custom startup script that downloaded the model once ollama was up. It's the same thing I'd do on a local cluster though. I'm not sure how fly could make it better unless they offered direct integration with ollama or some other inference server?
In the current days of AI I think spelling and grammar mistakes is perhaps a great way to tell it is still written by human...... ( Until AI copy this )
I'm sure AI is already capable of linking anonymous forum handles by writing style. Best to reinvent ourselves every month like one would with monthly password resets
For what it’s worth I don’t think we entirely disagree: it has at times felt absurd that it didn’t make as much money as it maybe otherwise could. We made a bet that the type of cloud platform we wanted to build could be well served by GPUs. It wasn’t as good a bet as we thought. There is probably a different type of cloud product we could build that would be better set up to sell gpus but we are still committed to the primitives our machine product has to offer.
> like with our portfolio of IPv4 addresses, I’m even more comfortable making bets backed by tradable assets with durable value.
Is that referencing the gpus, the hardware? If yes, why should they have a durable value? Historically hardware like that deprecated fast and reaches a value of 0, energy efficiency alone kills e.g. old server hardware. Something different here?
I think it's referencing only the IPv4 block, but it is a bit confusing. It doesn't make sense to be ref'ing the GPUs because their value is definitely not durable.
Yeah fair enough. I think it's just the subjectivity of "durable" at play here. The value of the GPUs may halve in a single year (e.g. H100s), but they'll never* drop to zero in a month. That's at least some kind of durability, because you can get a transaction done in a month.
Absolutely - GPUs are definitely not a very liquid asset. As someone who works at a GPU neocloud provider (Voltage Park), server assets at scale definitely face a huge slippage, you can buy for $1 and get quotes for $1.50 but only be able to sell for $0.60
Which we also see with e.g. A100 80GiB is approaching 5 years of age, but still sold and used widely and still cost ~20k$USD (and I remember a noticable higher price before deep seek...).
The thing is sure a A100 80GiB is a much older arch then successors, but the main bottleneck is the memory of which it has 80GiB.
With that said, this seems quite obvious - the type of customer that chooses Fly, seems like the last person to be spinning up dedicated GPU servers for extended periods of time. Seems much more likely they'll use something serverless which requires a ton of DX work to get right (personally I think Modal is killing it here). To compete, they would have needed to bet the company on it. It's way too competitive otherwise.
As someone who deploys a lot of models on rented GPU hardware, their pricing is not realistic for continous usage.
They're charging hyperscaler rates, and anyone willing to pay that much won't go with Fly.
For serverless usage they're only mildly overpriced compared to say Runpod, but I don't think of serverless as anything more than an onramp to renting dedicated machine, so it's not surprising to hear it's not taking off.
GPU workloads tend to have terrible cold-start performance by their nature, and without a lot of application specific optimizations it rarely ends up making financial sense to not take a cheaper continous option if you have an even mildly consistent workload. (and if you don't then you're not generating that much money for them)
My thing here is just: people self-hosting LLMs think about performance in tokens/sec, and we think about performance in terms of ms/rtt; they're just completely different scales. We don't really have a comparative advantage for developers who are comfortable with multisecond response times. And that's fine!
That reminds me when cloudflare launched their workers gpu product, it was specifically aimed at running models and the pricing was abstracted and based on model output. Did you look what they were doing when building gpu machines?
As the sibling comment points out, usually cold starts are optimized on the order of milliseconds, so 20 seconds is a while for a user to be sitting around with nothing streamed.
And with the premium for per-second GPUs hovering around 2x that for hourly/monthly rentals, it gets even harder for products with scale to justify.
You'd want to have a lot of time where you're scaled to 0, but that in turn maps to a lot of cold starts.
I really liked playing around with fly gpus, but it's just too expensive for hobby-use. Same goes for the rest of fly.io honestly. The DX is great and I wish I could move all of my homelab stuff and public websites to it, but it'd be way too expensive :(
Hmm, having a machine for every process is part of it but I actually like that kind of isolation. Storage and bandwidth also add up fast.
For example, I could get a digitalocean vm with 2gb ram, 1vcpu, 50gb storage, 2tb bandwidth for $12/mo.
For the same specs at fly.io, it'd be ~$22/mo not including any bandwidth. It could be less if it scales to zero/auto stops.
I recently tried experimenting with two different projects at fly. One was an attic server to cache packages for NixOS. Only used by me and my own vms. Even with auto scaling to zero, I think it was still around $15-20/mo.
The other was a fly gpu machine with Ollama on it. The cold start time + downloading a model each time was kind of painful, so I opted for just adding a 100gb volume. I don't actually remember what I was paying for that, but probably another 20/mo? I used it heavily for a few days to play around and then not so much later. I do remember doing the math and thinking it wouldn't be sustainable if I wanted to use it for stuff like home-assistant voice assistant or going through pdfs/etc with paperless.
On their own, neither of these are super expensive. But if I want to run multiple home services, the cost is just going to skyrocket with every new app I run. If I can rent a decent dedicated server for $100-$200/mo, then I at least don't have to worry about the cost increasing on me if a machine never scales to zero due to a healthcheck I forgot about or something like that.
Sorry if it's a bit rambly, happy to answer questions!
I would be curious how the Attic server would have gone with a Tigris bucket and local caching. Not sure how hard that is to pull off, but Tigris should be substantially cheaper than our NVMes and if you don't really NEED the io performance you're not getting anything for that money. Which is a long winded way of saying "we aren't great at block storage for anything but OLTP workloads and caches".
One thing we've struggled to communicate is how _cheap_ autosuspend/autostop make things. If that Machine is alive for 8 hours per day you're probably below $8/mo for that config. And it's so fast that it's viable for it to start/stop 45 times per day.
It's kind of hard to make the thing stay alive with health checks, unless you're meaning external ones?
We are suboptimal for things that make more sense as a bunch of containers on one host.
Tbh I haven't looked at Tigris at all. I still have my attic server deployed (just disabled/not in use) so I might give it a shot just to compare pricing. I do remember a decent portion of the cost being storage-related, so it's a good idea.
I'll have to look at autosuspend again too. I remember having autostop configured, but not autosuspend. I could see that helping with start times a lot for some stuff. It's not supported on GPU machines though, right? I thought I read that but don't see it in the docs at a quick glance.
> It's kind of hard to make the thing stay alive with health checks, unless you're meaning external ones?
Sorry, I did mean external healthchecks. Something like zabbix/uptimekuma. For something public facing, I'd want a health check just to make sure it's alive. With any type of serverless/functions, I'd probably want to reduce the healthcheck frequency to avoid the machine constantly running if it is normally low-traffic.
> We are suboptimal for things that make more sense as a bunch of containers on one host.
I think my ideal offering would be something where I could install a fly.io management/control plane on my own hardware for a small monthly fee and use that until it runs out of resources. I imagine it's a pretty niche case for enterprise unless you can get a bunch of customers with on-prem hardware, but homelabbers would probably be happy.
For instance this is a python flask app, it uses an sqlite DB, but I wouldn't really call it dynamic nor needing state because the DB is read only (gets updated daily from an authoritative source, but a website user can only search the db). It costs about $3.50 a month, but could be less because I have two instances at the ready (although only one is ever active/up).
But if you start needing state or persistence on an instance down/up I'd probably not go with Fly myself personally..
But I think it fits the bill perfectly for like that sort of in-between static and true dynamic. Eg not exactly static (for truly static I'd just use shared hosting or CF Pages, etc, et al), but not really truly dynamic either or needing state. That's sort of the sweet spot in my view. Package up your flask or fastapi or FastHTML into a Dockerfile, send to fly, easy peasy, love it.
I can't wait to play around with FastHTML and fly btw, haven't done so yet.. also would like to play around with a Datasette DB instance on fly..
Not them, but Runpod and Vast are my gotos. Runpod cost slightly more, but is in turn more reliable, so for "hobby pro" I'd go with them, otherwise Vast.
Salad Cloud is also very interesting if your models can fit on a consumer GPU, but it's a different model than typical GPU providers.
I have a decent sized homelab in my basement that I use for most stuff, and then a couple cheap-ish dedicated servers for public-facing things. Nothing has GPUs though, so I don't have a good solution for llm/ai projects yet.
I used to use cheap vms/vps from lowendtalk deals, but usually they're on over-subscribed hosts and can't do anything heavy.
Actual host recommendations: I like Racknerd and Hivelocity currently. OVH too, but I've read a lot of horror stories so I guess ymmv.
That sounds kinda weird to me. I use fly.io exactly because the pricing works out for hobby use. I can enable my machine for a few hours, run a bunch of inference, and turn it off again. The whole auto start/stop thing makes it seamless too.
Either you're familiar with GPU pricing and being willfully ignorant, or you're not familiar with the pricing in which case let someone who is point out:
- "Datacenter" means it's comparable to Runpod's secure cloud pricing.
- A spot instance of an H200 under someone's living room media console wouldn't go for A100 rates.
$3.50 will also get you an H100 at a laundry list of providers people build real businesses on.
Certainly all better track records than fly.io, especially on a post where they explain it's not working out for them as an offering and then promise they'll keep it shambling along.
You seem like you're familiar with vast. Have you used their autoscaler/serverless offering before? I haven't tried it yet, but it wasn't immediately obvious if I could have something like ollama running and scaled to zero instances when not in use.
Hah. We're doing AI, but we're doing vision-based stuff and not LLMs. For us, the problem has been deploying models.
Google and AWS helpfully offered their managed LLM AI services, but they don't really have anything terribly more useful than just machines with GPUs. Which are expensive.
I shelled out for a 4090 when they came out thinking it would be the key factor for running local llms. It turns out that anything worth running takes way more than 24GB VRAM. I would have been better off with 2+ 3090s and a custom power supply. It’s a pity because I thought it would be a great solution for coding and a home assistant, but performance and quality isn’t there yet for small models (afaik). Perhaps DIGITS will scratch the itch for local LLM developers, but performant models really want big metal for now, not something I can afford to own or rent at my scale.
There was a post on r/localLlama the other day about a presentation by the company building Digits hardware for Nvidia. The gist was that Digits is going to be aimed at academic AI research folks and as such don't expect them to be available in large numbers (at least not for this first version). It was disappointing. Now I'm awaiting the AMD Strix Halo based systems.
I haven't tested programming tasks with a local LLM vs. say, Claude 3.5. But it is nice to be able to run 14-32B LLMs locally and get an instant response. I have a single 3090.
Gosh. Good thing I haven't bought a GPU in almost a decade. With a little luck I'll catch this wave on the back end. I haven't had to learn web or mobile development thoroughly either
$6500 depending on VAT. But 10-12 times M4 Mac mini's with 100 Gbps networking gives you triple the cores and 160 GB with 2.5 times the memeory bandwith if the sharding of the NN layers is done right.
$6500!! You may as well buy 5x3090s for $1000 each for 120GB ram, spend the extra $1500 on the sundries.
Like, I'm sure Nvidia is aware of Apple's "unified memory" as an alternative to their cards and yet...they aren't offering >24GB consumer cards yet, so clearly they don't feel threatened.
Don't get me wrong, I've always disliked Apple as a company, but the M series chips are brilliant, I'm writing this on one right now. But people seem to think that Apple will be able to get the same perf increases yoy when they're really stretching process limits by dumping everything onto the same die like that - where do they go from here?
That said Nvidia is using HBM so it does make me wonder why they aren't also doing memory on package with HBM, I think SK Hynix et al were looking at making this possible.
I'm glad we're headed in the direction of 3d silicon though, always seemed like we may as well scale in z, I imagine they can stack silicon/cooling/silicon/cooling etc. I'm sure they can use lithography to create cooling dies to sandwich between everything else. Then just pass connections/coolant through those.
What will you pay me for the benchmarks, for the professional knowledge and analysis?
I can post benchmarks for these Mac machines and clusters of Studios and M4 Mac Minis (see my other HN posts last month, the largest Mac cluster I can benchmark for you has 4 TB of ultrafast unified memory and around 9216 M4 cores).
I mean, I can't pay you anything, but that sounds interesting as hell. Are there any interesting use cases to massive amounts of memory outside of training?
I meant to explain why no one ever posts a benchmark, it's expensive as hell to do a professional benchmark against accepted standards. Its several days work, very expensive rental of several pieces of $10K hardware, etc. You don't often hand that over for free. With my benchmark results some companies can save millions if they take my advice.
>any interesting use cases to massive amounts of memory outside of training?
Dozens, hundreds. Almost anything you use databases, CPUs, GPUs or TPUs for. 90% of computing is done on the wrong hardware, not just datacenter hardware.
The interesting use case we discussed here on HN last week was running full DeepSeek-R1 LLms on 778 GB fast DRAM computers locally. I benchmarked getting hundreds of tokens per second on a cluster of M4 Mac minis or a cluster of M2 Mac Studio Ultras where others reported 0.015 or 6 tokens per second on single machines.
I just heard of a Brazilian man who build a 256 Mac Mini cluster at double the cost that I would. He leaves $600K value on the table because he won't reverse engineer the instruction set, rewrite his software or even call Apple to negotiated a low price.
HN votes me down for commenting that I, a supercomputer builder for 43 years, can build better cheaper faster low power supercomputers from Mac Mini's and FPGA's than from any Nvidia, AMD or Intel state of the art hardware, it even beats the fastest supercomputer of the moment or the Cerebras wafer engine V3 (on energy. coding cost and performance per watt per dollar).
I design and build wafer scale 2 million core reconfigurable supercomputers for $30K a piece that cost $150-$300 million to mass produce. That's why I know how to benchmark M2 Ultra and M4 Macs, as they are the second best chip a.t.m. that we need to compete against.
As a consulting job I do benchmarks or build your on-prem hardware or datacenter. This job consists mainly teaching the customer's programming staf how to program massively parallel software or convincing the CEO not to rent cloud hardware but buy on-prem hardware. OP at Fly.io should have hired me, then he wouldn't have needed to write his blog post.
I replied to your comment in hope of someone hiring me when they read this.
Interesting! Fingers crossed someone who's looking for your skillset finds your post.
What is your process to turn Mac minis into a cluster? Is there any special hardware involved? And you can get 100x tok/s vs others on comparable hardware, what do you do differently - hardware, software, something else?
I discussed all the points you ask about in my HN postings last month, but never in enough detail so you must ask me to specify and that's when people hire me.
As you can see from this comments thread, most people, especially programmers, lack the knowledge we computer scientist, parallel programmers and chip or hardware designers have.
>What is your process
Science. To measure is to know, my prof always said.
To answer your questions in detail, email me.
You first need to be specific. The problem is not how to turn Mac minis into a cluster, with or without custom hardware ( I do both) on code X or Y. Or how to optimize software or rewrite it from scratch (which its often cheaper).
First find the problem. In this case the problem is find the lowest OPEX and Capex to do the stated compute load versus changing the compute load. Turns out in a simulation or a cruder spreadsheet calculation it becomes clear that the energy cost dominates of hardware choice, it trumps the cost of programming, the cost of off the shelf hardware and the difference if you add custom hardware. M4's are lower power, lower OPEX and lower CAPEX especially if you rewrite your (Nvidia GPU) software. The problem is the ignorance of the managers and their employee programmers.
You can repurpose the 2 x 10 Gbps USB-C, the 10 Gbps Ethernet and the three 32 Gbps PCIe ports or Thunderbolts but you have to use better drivers. You need to weigh if double the 960 Gbps 16 GB unified memory for 2 x $400 is faster than 2 Tbps memory at 1.23 times the cost versus 3 x 4 x 32 Gbps PCIe 4.0 versus 3 x 120 Gbps unidirectionally is better for this particular algorithm and wheat changes if you uses both the 10 CPU cores, 10 x 400 GPU corses and 16 Neural Engine cores (at 38 trillion 16 bit OPS) will work batter than just the CUDA cores. Ususally the answers is: rewrite the alogoritm and use an adaptive compiler and then a cluster of smaller 'sweet spot' off the shelf hardware will outperform the most fancy high end hardware if the network is balanced. This varies at runtime so you'll only know if you now how to code. As Akan Kay said and Steve Jobs quoted: if your serious about software you should do your own hardware. If you can't, then you can approach the hardware with commodity components if that turns out to be cheaper. I estimate for $42K labour I can save you a few hundred $k.
>What is your process to turn Mac minis into a cluster
1) Apply science. Benchmark everyting until you understand if its memory bound, i/o bound or compute bound [1].
2) Rewrite software from scratch in a parallel form with message passing.
3) Reverse engineer native instruction sets of CPU, GPU and ANE or TPU. Same for NVIDIA (don't use CUDA).
No special hardware needed but adding FPGA's for optimizing the network between machines might help.
So you analyse the software and hardware, then restructure it by reprogramming and rewireing and adaptive compilers.
Then you benchmark again and you find what hardware runs the algorithm fastest for less $ using less energy and weigh that against the extra cost for reprogramming.
> it's expensive as hell to do a professional benchmark against accepted standards. Its several days work, very expensive rental of several pieces of $10K hardware, etc
When people casually ask for benchmarks in comments, they’re not looking for in-depth comparisons across all of the alternatives.
They just want to see “Running Model X with quantization Y I get Z tokens per second”.
> That's why I know how to benchmark M2 Ultra and M4 Macs, as they are the second best chip a.t.m. that we need to compete against.
Macs are great for being able to fit models into RAM within a budget and run them locally, but I don’t understand how you’re concluding that a Mac is the “second best option” to your $30K machine unless you’re deliberately excluding all of the systems that hobbyist commonly build under $30K which greatly outperform Mac hardware.
>They just want to see “Running Model X with quantization Y I get Z tokens per second”.
Influencers on Youtube will give them that [1] but its meaningless. If a benchmark is not part of an in-depth comparison than it doesn't mean anything and can't inform you on what hardware will run this software best.
These shallow benchmarks influencers post on youtube and twitter are not just meaningless but also take days to browse through. And they are influencers, they are meant to influence you and are therefore not honest or reliable.
>but I don’t understand how you’re concluding that a Mac is the “second best option” to your $30K machine
I conclude that if you can't afford to develop custom chips than in certain cases a cluster of M4 Mac Mini's will be the fastest cheapest option. Cerebras Wafers or NVDIA GPUs have always been too expensive compared to custom chips or Mac Mini clusters, independent of the specific software workload.
I also meant to say that a cluster of $599 Mac Minis will outperform a $6500 M2 Ultra Mac Studio with 192GB and be half the price for higher performance and DRAM but only if you utilize the M4 Mac Mini aggregated 100 Gbps networking.
I'm hoping to see some smaller MoE models released this year, trained with more recent recipes (higher quality data, much longer pretraining). Mixtral 8x7B was impressive when it came out, but the exact same architecture could be a lot more powerful today, and would run quite fast on Apple Silicon.
I have a Mac with a lot of RAM for running models. I haven’t done it in a month because I can tell that it’s not only slow, but the output also doesn’t come close to what I can get from the latest from Claude or ChatGPT.
It’s actually amazing that I can run LLMs locally and get the quality of output that they give me, but it’s just a different level of experience than the state of the art.
I’m becoming convinced that the people who sing the praises of running locally are just operating differently. For them, slow and lower quality output aren’t a problem because they’re having fun doing it themselves. When I want to get work done, the hosted frontier models are barely fast enough and have hit or miss quality for me, so stepping down to the locally hosted options is even more frustrating.
I started buying Macs with more memory, no regrets. An M4 Max with 64GB (in a laptop, no less!) runs most small models comfortably (but get 96GB or more if you really intend to use 70B models regularly). And when I'm not running LLMs, the memory is useful for other stuff.
Not the person you replied to but auto discard only helps so much. Even with it on I often use 80-90% memory (albeit on Windows, not sure if macOS is any different in this regard).
Firefox here too, no such problems. I do however run Google services in a separate browser (Brave or Chromium) because most of them hog the Firefox browser. For example running earth.google.com on FF is... a very special experience. :-/
I'm admittedly a complete LLM noob, so my question might not even make sense. Or it might exist and I haven't found it quite yet.
But have they considered pivoting some of said compute to some 'private, secure LLM in a box' solution?
I've lately been toying with the idea of training from extensive docs and code, some open, some not, for both code generation and insights.
I went down the RAG rabbit hole, and frankly, the amount of competing ideas of 'this is how you should do it', from personal blogs to PaaS companies, overwhelmed me. Vector dbs, ollama, models, langchain, and various one off tools linking to git repos.
I feel there has to be substantial market for whoever can completely simplify that flow for dummies like me, and not charge a fortune for the privilege.
The problem is currently all the "competing" ideas have a ton of tradeoffs and is rarely one size fits all. Furthermore, it's not clear if the idea you choose will be become obsolete by the underlying model architecture getting better. On top of all that, you are essentially competing with Anthropic/OpenAI/Google, where your only advantage is "privacy". Anyone who deeply cares about privacy, and is willing to pay for it, may just likely do it on their own (especially if your CTO is pouting money into "investing" in AI). Anyone who doesn't will likely not want to 6-7 months behind what you can get at OpenAI or Google.
Kudos to owning up to your failed bet on GPUs even if you are putting a lot of blame on Nvidia for it. And to be fair, you're not wrong. Nvidia's artificial market segmentation is terrible and their drivers aren't that great either.
The real problem is the lack of security-isolated slicing one or more GPUs for virtual machines. I want my consumer-grade GPU to be split up into the host machine and also into virtual machines, without worrying about resident neighbor cross-talk! Gosh that sounds like why I moved out of my apartment complex, actually.
The idea of having to assign a whole GPU via PCI passthrough is just asinine. I don't need to do that for my CPU, RAM, network, or storage. Why should I need to do it for my GPU?
Yes devs want LLMs, but also the price of inference compute plummeted 90% over the last 18 months, which is primarily in gpus
So it’s not just that openai and anthropic apis are good enough, they are also cheap enough, and still overpriced compared to the industry
Your GPU investment wont do as well as you thought, but also you are wasting time on security. If the end user and market doesnt care then you can consider not caring as well. Worst case you can pay for any settlement with …. more gpu credits.
It's really a shame GPU slices aren't a thing -- a monthly cost of $1k for "a GPU" is just so far outside of what I could justify. I guess it's not terrible if I can batch-schedule a mega-gpu for an hour a day to catch up on tasks, but then I'm basically still looking at nearly $50/month.
I don't know exactly what type of cloud offering would satisfy my needs, but what's funny is that attaching an AMD consumer GPU to a Raspberry Pi is probably the most economical approach for a lot of problems.
Maybe something like a system where I could hotplug a full GPU into a system for a reservation of a few minutes at a time and then unplug it and let it go back into a pool?
FWIW it's that there's a large number of ML-based workflows that I'd like to plug into progscrape.com, but it's been very difficult to find a model that works without breaking the hobby-project bank.
Do you think that you can use those machines for confidential workflows for enterprise use? I'm currently struggling to balance running inference workloads on expensive AWS instances where I can trust that data remains private vs using more inexpensive platforms.
Just currently exploring how custom AI workflows (e.g. text to sql, custom report generation using private data) can help given the current SOTA. Looking to develop tooling over the next 3-6 months. I'd like to see what we can come up with before dropping $50-100k on hardware.
I threw together a toy project to see if it would help me understand the basic concepts and my takeaway was that, if you can shape your input into something a dedicated classification model (e.g. YOLO for document layout analysis) can work with, you can farm each class out to the most appropriate model.
It turns out that I can run most of the appropriate models on my ancient laptop if I don't mind waiting for the complicated ones to finish. If I do mind, I can just send that part to OpenAI or similar. If your workflow can scale horizontally like my OCR pipeline crap, every box in your shop with RAM >= 16GB might be useful.
Apologies if this is all stuff you're familiar with.
Of course you cannot use these machines "for confidential workflows for enterprise use", at least with AWS you know whose computer you're working with, but also keep in mind that it's really hard to steal your data as long as your data stays in memory, and you use something like mTLS to actually get it in and out of memory via E2EE. You can figure out the rest of your security model along the way, but anything sensitive (i.e. confidential) would surely fall way out of this model.
i use them a lot and constantly forget to turn mine off and it just drains my credits. i really need to write a job to turn them off when it's idle for longer than 20minutes
Which you are! What ever happened to the MIG implementation work that y’all were working on? Last I heard it was “cursed” and nearly made someone go insane, which is very normal for NVIDIA hardware :)
you guys have all this juicy GPU and infrastructure. why not offer models as apis?
i would pay to have apis for:
sam2, florence, blip, flux 1.1, etc.
whatever use case I would have reached Fly for on GPU, i can't justify _not_ using Replicate. maybe Fly can do better offer premium queues for that with their juicy infra?
you're right! as a software dev I see dockerization and foisting these models as a burden, not a necessity.
There is no market for MIG in the cloud. People talk about it a lot, but in reality, nobody wants a partial GPU (at least not paying for it).
One interesting thing about all this is that 1 GPU / 1 VM doesn't work today with AMD GPUs like MI300x. You can't do pcie passthrough, but AMD is working on adding it to ROCm. We plan to be one of the first to offer this.
> A whole enterprise A100 is a compromise position for them; they want an SXM cluster of H100s.
For a lot of use-cases you need at lest two A100s with a very fast interconnect, potentially many more. This isn't even about scaling with requests but about running one single LLM instance.
Sure you will find all of ways how people managed to runt his or that on smaller platforms, problem is that quite often doesn't scale to what is needed in production for a lot of subtle and less subtle reasons.
603 comments
[ 3.8 ms ] story [ 423 ms ] threadWho even knows what the customer is ever going to want? Pivot. Pivot. Pivot.
PS: And pouring one out for the engineering hours that went into shipping GPUs. Sometimes it's a fine product, but just doesn't fit.
Apparently this is technically possible, if you can find the right person at Nvidia to talk about vGPU licensing and magic incantations. Hopefully someone reading this HN front page story can make the introduction.
To userspace Nvidia license server (a) in each host, (b) for entire Fly cloud, or (c) over WAN to Nvidia cloud?
Really what we'd have wanted to do would have been to give Fly Machines MIG slices. But to the best of my understanding, MIG is paravirtualized; it doesn't give you SR-IOV-style PCI addresses for the slices, but rather a reference Nvidia's userland libraries pass to the kernel driver, which is a dance you can't do across VM boundaries unless your hypervisor does it deliberately.
1. Instead of blocking VM start for license validation, convert that step into non-blocking async submission of usage telemetry, allowing every VM to start instantly. For PoC purposes, Nvidia's existing stack could be binary patched to proxy the license request to a script that isn't blocking VM start, pending step 2 negotiation.
2. Reconcile aggregate vGPU usage telemetry from Nvidia Fly-wide license server (Step 1) with aggregate vGPU usage reports from Fly's orchestration/control plane, which already has that data for VM usage accounting. In theory, Fly orchestration has more awareness of vGPU guest workload context than Nvidia's VM-start gatekeeping license agent, so there might be mutual interest in trading instant VM start for async workload analytics.
Do you mean vCS [1], which is already integrated and licensed by KVM/RedHat/Nutanix, Xen/Citrix and VMware?
It's distinct from SR-IOV, PCI passthrough, vGPU-for-VDI, and MIG.
[1] https://blogs.nvidia.com/blog/virtualcomputeserver-expands-v...
> Alternatively, we could have used a conventional hypervisor. Nvidia suggested VMware (heh). But they could have gotten things working had we used QEMU. We like QEMU fine, and could have talked ourselves into a security story for it, but the whole point of Fly Machines is that they take milliseconds to start.
Someone could implement virtio-cuda (there are PoCs on github [1] [2]), but it would be a huge maintenance burden. It should really be done by Nvidia, in lockstep with CUDA extensions.
Nvidia vCS makes use of licensed GPGPU emulation code in the VM device model, which is QEMU in the case of KVM and Xen. Cloud Hypervisor doesn't use QEMU, it has its own (Rust?) device model, https://github.com/cloud-hypervisor/cloud-hypervisor/blob/ma...
So the question is, how to reuse Nvidia's proprietary GPGPU emulation code from QEMU, with Cloud Hypervisor? C and Rust are not friends. Can a Firecracker or Cloud Hypervisor VM use QEMU only for GPGPU emulation, alongside the existing device model, without impacting millisecond launch speed? Could an emulated vGPGPU be hotplugged after VM launch?
There has been some design/PoC work for QEMU disaggregation [3][4] of emulation functions into separate processes. It might be possible to apply similar techniques so that Cloud Hypervisor's device model (in Rust) process could run alongside a QEMU GPGPU emulator (in C) process, with some coordination by KVM.
If this approach is feasible, the architecture and code changes should be broadly useful to upstream for long-term support and maintenance, rather than custom to Fly. The custom code would be the GPGPU emulator, which is already maintained by Nvidia and running within QEMU on RedHat, Nutanix, etc.
It would also advance the state of the art in security isolation and access control of emulated devices used by VMs.
[1] https://github.com/coldfunction/qCUDA
[2] https://github.com/juniorprincewang/virtio-cuda-module
[3] https://www.qemu.org/docs/master/devel/multi-process.html
[4] https://wiki.qemu.org/Features/MultiProcessQEMU
Any company (let alone Fly) doing this won't go against Nvidia Enterprise T&C?
> how to reuse Nvidia's proprietary GPGPU emulation code from QEMU
If it has been contributed to QEMU, it isn't GPL/LGPL?
> Could an emulated vGPGPU be hotplugged after VM launch
gVisor instead bounces ioctls back and forth between "guest" and host. Sounds like a nice, lightweight (even if limited & sandbox-busting) approach, too. Unsure if it mitigates the need for the "licensing dance" tptacek mentioned above, but I reckon the security posture of such a setup is unacceptable for Fly.
https://gvisor.dev/docs/user_guide/gpu/
> would also advance the state of the art in security isolation and access control of emulated devices used by VMs
I hope I'm not talking to DeepSeek / DeepResearch (:
Good question for a lawyer. Even more reason (beyond maintenance cost) that it would be best done by Nvidia. qCUDA paper has a couple dozen references on API remoting research, https://www.cs.nthu.edu.tw/~ychung/conference/2019-CloudCom....
> If it has been contributed to QEMU, it isn't GPL/LGPL?
Not contributed, but integrated with QEMU by commercial licensees. Since the GPGPU emulation code isn't public, presumably it's a binary blob.
> I hope I'm not talking to DeepSeek / DeepResearch (:
Will take that as a compliment :) Not yet tried DS/DR.
Red Hat for one doesn't ship any functionality that isn't available upstream, much less proprietary, and they have large customers using virtual GPU.
Was that recent, with MIG support for GPGPU partitioning? Is there a public mailing list thread or patch series for that work?
Nvidia has a 90-page deployment doc on vCS ("virtual compute server") for RedHat KVM, https://images.nvidia.com/content/Solutions/data-center/depl...
That said, the slowness of QEMU's startup is always exaggerated. Whatever they did with CH they could have done with QEMU.
https://fly.io/blog/a-blog-if-kept/
In a different product, I was given some Google Cloud credits, which unlocked me to put the product in front of customer. This one also needed GPU but not as expensive as the previous. It works reliably and it's fast.
Personally, I had two use cases for GPU providers in past 3 months.
I think there's definitely demand for reliability and better pricing. Not sure Fly will be able to touch that market though as it's not known for both (stability & developer friendly pricing).
P.S If anyone is working on a serverless provider and want me to test their product, reach me out :)
Ironically GCP and AWS GPUs are so overpriced that getting even half the number of credits from Runpod is like a 4x increase in "GPU runway", especially with .44/hr A40s.
A bit like "stop doing this..." and we think: omg, am I doing the same deadly mistake?
Closely followed by, “I was right.” :)
I identify as a consipracy theorist, my pronouns are: Told/You/So.
There are people doing GPU-enabled inference stuff on Fly.io. That particular slice of the market seems fine?
For the LLM itself, I just used a custom startup script that downloaded the model once ollama was up. It's the same thing I'd do on a local cluster though. I'm not sure how fly could make it better unless they offered direct integration with ollama or some other inference server?
The idea that a cloud compute provider can’t make GPU compute into an profitable business is pretty laughable.
for our p5 quota I had to talk to our TAM team on AWS, while most of our quota requests are instant usually.
The whole thing is sorta antithetical.
Ha ha, it didn't terrify Modal. It ships with all those security problems, and pretends it doesn't have them. Sorry Eric.
> like with our portfolio of IPv4 addresses, I’m even more comfortable making bets backed by tradable assets with durable value.
Is that referencing the gpus, the hardware? If yes, why should they have a durable value? Historically hardware like that deprecated fast and reaches a value of 0, energy efficiency alone kills e.g. old server hardware. Something different here?
* never say never
But server GPUs tend to deprecate slower.
Which we also see with e.g. A100 80GiB is approaching 5 years of age, but still sold and used widely and still cost ~20k$USD (and I remember a noticable higher price before deep seek...).
The thing is sure a A100 80GiB is a much older arch then successors, but the main bottleneck is the memory of which it has 80GiB.
What was the price at launch?
With that said, this seems quite obvious - the type of customer that chooses Fly, seems like the last person to be spinning up dedicated GPU servers for extended periods of time. Seems much more likely they'll use something serverless which requires a ton of DX work to get right (personally I think Modal is killing it here). To compete, they would have needed to bet the company on it. It's way too competitive otherwise.
They're charging hyperscaler rates, and anyone willing to pay that much won't go with Fly.
For serverless usage they're only mildly overpriced compared to say Runpod, but I don't think of serverless as anything more than an onramp to renting dedicated machine, so it's not surprising to hear it's not taking off.
GPU workloads tend to have terrible cold-start performance by their nature, and without a lot of application specific optimizations it rarely ends up making financial sense to not take a cheaper continous option if you have an even mildly consistent workload. (and if you don't then you're not generating that much money for them)
https://blog.cloudflare.com/workers-ai/
My Fly machine loads from turned off to first inference complete in about 35 seconds.
If it’s already running, it’s 15 seconds to complete. I think that’s pretty decent.
And with the premium for per-second GPUs hovering around 2x that for hourly/monthly rentals, it gets even harder for products with scale to justify.
You'd want to have a lot of time where you're scaled to 0, but that in turn maps to a lot of cold starts.
What part of the cost gets out of hand? Having to have a Machine for every process? Do you remember what napkin math pricing you were working with?
For example, I could get a digitalocean vm with 2gb ram, 1vcpu, 50gb storage, 2tb bandwidth for $12/mo.
For the same specs at fly.io, it'd be ~$22/mo not including any bandwidth. It could be less if it scales to zero/auto stops.
I recently tried experimenting with two different projects at fly. One was an attic server to cache packages for NixOS. Only used by me and my own vms. Even with auto scaling to zero, I think it was still around $15-20/mo.
The other was a fly gpu machine with Ollama on it. The cold start time + downloading a model each time was kind of painful, so I opted for just adding a 100gb volume. I don't actually remember what I was paying for that, but probably another 20/mo? I used it heavily for a few days to play around and then not so much later. I do remember doing the math and thinking it wouldn't be sustainable if I wanted to use it for stuff like home-assistant voice assistant or going through pdfs/etc with paperless.
On their own, neither of these are super expensive. But if I want to run multiple home services, the cost is just going to skyrocket with every new app I run. If I can rent a decent dedicated server for $100-$200/mo, then I at least don't have to worry about the cost increasing on me if a machine never scales to zero due to a healthcheck I forgot about or something like that.
Sorry if it's a bit rambly, happy to answer questions!
I would be curious how the Attic server would have gone with a Tigris bucket and local caching. Not sure how hard that is to pull off, but Tigris should be substantially cheaper than our NVMes and if you don't really NEED the io performance you're not getting anything for that money. Which is a long winded way of saying "we aren't great at block storage for anything but OLTP workloads and caches".
One thing we've struggled to communicate is how _cheap_ autosuspend/autostop make things. If that Machine is alive for 8 hours per day you're probably below $8/mo for that config. And it's so fast that it's viable for it to start/stop 45 times per day.
It's kind of hard to make the thing stay alive with health checks, unless you're meaning external ones?
We are suboptimal for things that make more sense as a bunch of containers on one host.
I'll have to look at autosuspend again too. I remember having autostop configured, but not autosuspend. I could see that helping with start times a lot for some stuff. It's not supported on GPU machines though, right? I thought I read that but don't see it in the docs at a quick glance.
> It's kind of hard to make the thing stay alive with health checks, unless you're meaning external ones?
Sorry, I did mean external healthchecks. Something like zabbix/uptimekuma. For something public facing, I'd want a health check just to make sure it's alive. With any type of serverless/functions, I'd probably want to reduce the healthcheck frequency to avoid the machine constantly running if it is normally low-traffic.
> We are suboptimal for things that make more sense as a bunch of containers on one host.
I think my ideal offering would be something where I could install a fly.io management/control plane on my own hardware for a small monthly fee and use that until it runs out of resources. I imagine it's a pretty niche case for enterprise unless you can get a bunch of customers with on-prem hardware, but homelabbers would probably be happy.
I love fly. It's perfect for certain kinds of apps/sites.
https://my-upc.com
https://github.com/jgbrwn/my-upc
For instance this is a python flask app, it uses an sqlite DB, but I wouldn't really call it dynamic nor needing state because the DB is read only (gets updated daily from an authoritative source, but a website user can only search the db). It costs about $3.50 a month, but could be less because I have two instances at the ready (although only one is ever active/up).
But if you start needing state or persistence on an instance down/up I'd probably not go with Fly myself personally..
But I think it fits the bill perfectly for like that sort of in-between static and true dynamic. Eg not exactly static (for truly static I'd just use shared hosting or CF Pages, etc, et al), but not really truly dynamic either or needing state. That's sort of the sweet spot in my view. Package up your flask or fastapi or FastHTML into a Dockerfile, send to fly, easy peasy, love it.
I can't wait to play around with FastHTML and fly btw, haven't done so yet.. also would like to play around with a Datasette DB instance on fly..
Salad Cloud is also very interesting if your models can fit on a consumer GPU, but it's a different model than typical GPU providers.
I used to use cheap vms/vps from lowendtalk deals, but usually they're on over-subscribed hosts and can't do anything heavy.
Actual host recommendations: I like Racknerd and Hivelocity currently. OVH too, but I've read a lot of horror stories so I guess ymmv.
- "Datacenter" means it's comparable to Runpod's secure cloud pricing.
- A spot instance of an H200 under someone's living room media console wouldn't go for A100 rates.
$3.50 will also get you an H100 at a laundry list of providers people build real businesses on.
Certainly all better track records than fly.io, especially on a post where they explain it's not working out for them as an offering and then promise they'll keep it shambling along.
fly.io was the first provider I tried any gpu offerings at, I probably should give it another shot now that I've used a few others.
For simple inference, its too expensive for a project that makes no money. Which is most projects.
Google and AWS helpfully offered their managed LLM AI services, but they don't really have anything terribly more useful than just machines with GPUs. Which are expensive.
I'm going to check fly.io...
Like, I'm sure Nvidia is aware of Apple's "unified memory" as an alternative to their cards and yet...they aren't offering >24GB consumer cards yet, so clearly they don't feel threatened.
Don't get me wrong, I've always disliked Apple as a company, but the M series chips are brilliant, I'm writing this on one right now. But people seem to think that Apple will be able to get the same perf increases yoy when they're really stretching process limits by dumping everything onto the same die like that - where do they go from here?
That said Nvidia is using HBM so it does make me wonder why they aren't also doing memory on package with HBM, I think SK Hynix et al were looking at making this possible.
I'm glad we're headed in the direction of 3d silicon though, always seemed like we may as well scale in z, I imagine they can stack silicon/cooling/silicon/cooling etc. I'm sure they can use lithography to create cooling dies to sandwich between everything else. Then just pass connections/coolant through those.
Prompt eval is slow, inference for large models at high context is slow, training is limited and slow.
It's better than not having anything, but we got rid of our M1 Max 192GBs after about a year.
I can post benchmarks for these Mac machines and clusters of Studios and M4 Mac Minis (see my other HN posts last month, the largest Mac cluster I can benchmark for you has 4 TB of ultrafast unified memory and around 9216 M4 cores).
I meant to explain why no one ever posts a benchmark, it's expensive as hell to do a professional benchmark against accepted standards. Its several days work, very expensive rental of several pieces of $10K hardware, etc. You don't often hand that over for free. With my benchmark results some companies can save millions if they take my advice.
>any interesting use cases to massive amounts of memory outside of training?
Dozens, hundreds. Almost anything you use databases, CPUs, GPUs or TPUs for. 90% of computing is done on the wrong hardware, not just datacenter hardware.
The interesting use case we discussed here on HN last week was running full DeepSeek-R1 LLms on 778 GB fast DRAM computers locally. I benchmarked getting hundreds of tokens per second on a cluster of M4 Mac minis or a cluster of M2 Mac Studio Ultras where others reported 0.015 or 6 tokens per second on single machines.
I just heard of a Brazilian man who build a 256 Mac Mini cluster at double the cost that I would. He leaves $600K value on the table because he won't reverse engineer the instruction set, rewrite his software or even call Apple to negotiated a low price.
HN votes me down for commenting that I, a supercomputer builder for 43 years, can build better cheaper faster low power supercomputers from Mac Mini's and FPGA's than from any Nvidia, AMD or Intel state of the art hardware, it even beats the fastest supercomputer of the moment or the Cerebras wafer engine V3 (on energy. coding cost and performance per watt per dollar).
I design and build wafer scale 2 million core reconfigurable supercomputers for $30K a piece that cost $150-$300 million to mass produce. That's why I know how to benchmark M2 Ultra and M4 Macs, as they are the second best chip a.t.m. that we need to compete against.
As a consulting job I do benchmarks or build your on-prem hardware or datacenter. This job consists mainly teaching the customer's programming staf how to program massively parallel software or convincing the CEO not to rent cloud hardware but buy on-prem hardware. OP at Fly.io should have hired me, then he wouldn't have needed to write his blog post.
I replied to your comment in hope of someone hiring me when they read this.
What is your process to turn Mac minis into a cluster? Is there any special hardware involved? And you can get 100x tok/s vs others on comparable hardware, what do you do differently - hardware, software, something else?
As you can see from this comments thread, most people, especially programmers, lack the knowledge we computer scientist, parallel programmers and chip or hardware designers have.
>What is your process
Science. To measure is to know, my prof always said.
To answer your questions in detail, email me.
You first need to be specific. The problem is not how to turn Mac minis into a cluster, with or without custom hardware ( I do both) on code X or Y. Or how to optimize software or rewrite it from scratch (which its often cheaper).
First find the problem. In this case the problem is find the lowest OPEX and Capex to do the stated compute load versus changing the compute load. Turns out in a simulation or a cruder spreadsheet calculation it becomes clear that the energy cost dominates of hardware choice, it trumps the cost of programming, the cost of off the shelf hardware and the difference if you add custom hardware. M4's are lower power, lower OPEX and lower CAPEX especially if you rewrite your (Nvidia GPU) software. The problem is the ignorance of the managers and their employee programmers.
You can repurpose the 2 x 10 Gbps USB-C, the 10 Gbps Ethernet and the three 32 Gbps PCIe ports or Thunderbolts but you have to use better drivers. You need to weigh if double the 960 Gbps 16 GB unified memory for 2 x $400 is faster than 2 Tbps memory at 1.23 times the cost versus 3 x 4 x 32 Gbps PCIe 4.0 versus 3 x 120 Gbps unidirectionally is better for this particular algorithm and wheat changes if you uses both the 10 CPU cores, 10 x 400 GPU corses and 16 Neural Engine cores (at 38 trillion 16 bit OPS) will work batter than just the CUDA cores. Ususally the answers is: rewrite the alogoritm and use an adaptive compiler and then a cluster of smaller 'sweet spot' off the shelf hardware will outperform the most fancy high end hardware if the network is balanced. This varies at runtime so you'll only know if you now how to code. As Akan Kay said and Steve Jobs quoted: if your serious about software you should do your own hardware. If you can't, then you can approach the hardware with commodity components if that turns out to be cheaper. I estimate for $42K labour I can save you a few hundred $k.
Yes. Several pages of comments about M4 clusters, wafer scale integrations and a few about DeepSeek.
https://news.ycombinator.com/threads?id=morphle (a few pages- press more).
https://news.ycombinator.com/item?id=42799072
1) Apply science. Benchmark everyting until you understand if its memory bound, i/o bound or compute bound [1].
2) Rewrite software from scratch in a parallel form with message passing.
3) Reverse engineer native instruction sets of CPU, GPU and ANE or TPU. Same for NVIDIA (don't use CUDA).
No special hardware needed but adding FPGA's for optimizing the network between machines might help.
So you analyse the software and hardware, then restructure it by reprogramming and rewireing and adaptive compilers. Then you benchmark again and you find what hardware runs the algorithm fastest for less $ using less energy and weigh that against the extra cost for reprogramming.
[1] https://en.wikipedia.org/wiki/Roofline_model
When people casually ask for benchmarks in comments, they’re not looking for in-depth comparisons across all of the alternatives.
They just want to see “Running Model X with quantization Y I get Z tokens per second”.
> That's why I know how to benchmark M2 Ultra and M4 Macs, as they are the second best chip a.t.m. that we need to compete against.
Macs are great for being able to fit models into RAM within a budget and run them locally, but I don’t understand how you’re concluding that a Mac is the “second best option” to your $30K machine unless you’re deliberately excluding all of the systems that hobbyist commonly build under $30K which greatly outperform Mac hardware.
Influencers on Youtube will give them that [1] but its meaningless. If a benchmark is not part of an in-depth comparison than it doesn't mean anything and can't inform you on what hardware will run this software best.
These shallow benchmarks influencers post on youtube and twitter are not just meaningless but also take days to browse through. And they are influencers, they are meant to influence you and are therefore not honest or reliable.
[1] https://www.youtube.com/watch?v=GBR6pHZ68Ho
>but I don’t understand how you’re concluding that a Mac is the “second best option” to your $30K machine
I conclude that if you can't afford to develop custom chips than in certain cases a cluster of M4 Mac Mini's will be the fastest cheapest option. Cerebras Wafers or NVDIA GPUs have always been too expensive compared to custom chips or Mac Mini clusters, independent of the specific software workload.
I also meant to say that a cluster of $599 Mac Minis will outperform a $6500 M2 Ultra Mac Studio with 192GB and be half the price for higher performance and DRAM but only if you utilize the M4 Mac Mini aggregated 100 Gbps networking.
I have a Mac with a lot of RAM for running models. I haven’t done it in a month because I can tell that it’s not only slow, but the output also doesn’t come close to what I can get from the latest from Claude or ChatGPT.
It’s actually amazing that I can run LLMs locally and get the quality of output that they give me, but it’s just a different level of experience than the state of the art.
I’m becoming convinced that the people who sing the praises of running locally are just operating differently. For them, slow and lower quality output aren’t a problem because they’re having fun doing it themselves. When I want to get work done, the hosted frontier models are barely fast enough and have hit or miss quality for me, so stepping down to the locally hosted options is even more frustrating.
Let's be honest, the other stuff is just Chrome: Tell me 96gb is enough?
But have they considered pivoting some of said compute to some 'private, secure LLM in a box' solution?
I've lately been toying with the idea of training from extensive docs and code, some open, some not, for both code generation and insights.
I went down the RAG rabbit hole, and frankly, the amount of competing ideas of 'this is how you should do it', from personal blogs to PaaS companies, overwhelmed me. Vector dbs, ollama, models, langchain, and various one off tools linking to git repos.
I feel there has to be substantial market for whoever can completely simplify that flow for dummies like me, and not charge a fortune for the privilege.
The real problem is the lack of security-isolated slicing one or more GPUs for virtual machines. I want my consumer-grade GPU to be split up into the host machine and also into virtual machines, without worrying about resident neighbor cross-talk! Gosh that sounds like why I moved out of my apartment complex, actually.
The idea of having to assign a whole GPU via PCI passthrough is just asinine. I don't need to do that for my CPU, RAM, network, or storage. Why should I need to do it for my GPU?
So it’s not just that openai and anthropic apis are good enough, they are also cheap enough, and still overpriced compared to the industry
Your GPU investment wont do as well as you thought, but also you are wasting time on security. If the end user and market doesnt care then you can consider not caring as well. Worst case you can pay for any settlement with …. more gpu credits.
I don't know exactly what type of cloud offering would satisfy my needs, but what's funny is that attaching an AMD consumer GPU to a Raspberry Pi is probably the most economical approach for a lot of problems.
Maybe something like a system where I could hotplug a full GPU into a system for a reservation of a few minutes at a time and then unplug it and let it go back into a pool?
FWIW it's that there's a large number of ML-based workflows that I'd like to plug into progscrape.com, but it's been very difficult to find a model that works without breaking the hobby-project bank.
I wouldn't put anything confidential through it.
It turns out that I can run most of the appropriate models on my ancient laptop if I don't mind waiting for the complicated ones to finish. If I do mind, I can just send that part to OpenAI or similar. If your workflow can scale horizontally like my OCR pipeline crap, every box in your shop with RAM >= 16GB might be useful.
Apologies if this is all stuff you're familiar with.
If you could checkpoint a GPU quickly enough it would be possible to run multiple isolated workloads on the same GPUs without any issues.
i would pay to have apis for:
sam2, florence, blip, flux 1.1, etc.
whatever use case I would have reached Fly for on GPU, i can't justify _not_ using Replicate. maybe Fly can do better offer premium queues for that with their juicy infra?
you're right! as a software dev I see dockerization and foisting these models as a burden, not a necessity.
One interesting thing about all this is that 1 GPU / 1 VM doesn't work today with AMD GPUs like MI300x. You can't do pcie passthrough, but AMD is working on adding it to ROCm. We plan to be one of the first to offer this.
For a lot of use-cases you need at lest two A100s with a very fast interconnect, potentially many more. This isn't even about scaling with requests but about running one single LLM instance.
Sure you will find all of ways how people managed to runt his or that on smaller platforms, problem is that quite often doesn't scale to what is needed in production for a lot of subtle and less subtle reasons.