This response is so dense with numbers that it's probably about 1000 tokens. So at 1 token/s, it took you almost 17 minutes to generate this on the MacBook Pro.
And maybe author sent ~1K tokens as the starting prompt, and possibly some more stuff in the system prompt, and add on top of that that Apple hardware is famously bad at prompt processing.
If it's (nearly) all streamed from disk can you use an intel/amd machine and a big fat gpu and use the SSDs (nvme for faster throughput) as your virtual vram? Or something like that?
I hate seeing comments like this on HN. We used to upvote “look at this crazy thing I did” work. Not everything has to make sense or be ground breaking.
I've never understood why "Hacker" News so frequently gets "But why though?" comments at the top.
The entire history of innovation is filled with people doing something just to see they can get it to work, even if badly, and then people continue to iterate on that until it works better, then works well, and then is so obvious people would never even question it. But it all starts with someone doing it to scratch an itch.
Neural networks, the foundation of our current AI revolution, used to fit well into the "neat, but practically useless" category.
Sure there are countless "but why though?" experiments that don't pan out, but that's just the cost of exploration. There can be no step-function innovation in a world where people only do things that make immediate practical sense.
Gen X here. Having grown up in the 80's I remember multiple occasions where someone would ask "what are you going to do with a computer?" as in what would a computer possibly be useful for. Just imagine someone asking this question today. Though it would probably be more shaped like the comment you are responding to.
What are you going to do with a computer? I've always hated this attitude. We do these things because they are interesting to us, for the fun of exploration, because we enjoy learning, because we want to iterate and improve, to make the world better, or any plethora of reasons that involve intellectual curiosity of some sort.
I've had the thought that an underexplored use case for llms is treating them as classification machines. Put in a lot of input then ask for a single token classifying the input. Like "does this code have any bugs? reply "yes" or "no""
Doing something like that at 1 token per second may be useful.
That's the one workload this setup is worst at today, unfortunately: output tokens are cheap at 1/s but input isn't — a 512-token prompt takes ~6 minutes before the first token,
because prefill currently reads each layer's experts once per 64-row pass (~9 TB of reads for a 1.4 TB model).
Fix is scheduling and it's the next thing being built; once prefill reads each expert once per layer, the one-token-out classifier pattern becomes the sweet spot rather than the worst case.
It's kind of insane how having this tech at this speed 5 years ago would have probably still been seen as insanely useful and revolutionary. If LLMs were more capable but dramatically slower, I wonder how it would impact how we use it? Dramatically more thought being put into prompts, much more preparation probably
Those machines were also manually scheduled. So you booked in your time slot to run your program in advance. That syntax error meant not being able to run the fixed version until your next scheduled session, maybe days later.
Write the program, punch the cards, send the cards to another building to be loaded, program runs, printout comes out in another building, somehow this takes 2-3 days
At 1t/s it's still faster than humans for a lot of tasks, basically doing overnight what could take humans half a week. Plus you can always parallelize.
This is what people forget when they see slow performance: at 1 t/s it's still roughly the equivalent of having another person work for you at no extra cost besides the initial purchase/sign-on-bonus.
Frontier models are amazing, but what will really be useful for us is having models and hardware so efficient that you can run useful LLMs locally. One of my favourite LLMs to this day is still my jail-broken gemma4 12b because it's small enough to run on my computer, but also 100% local and free as in liberty.
The future of inference is likely in ASICs, so we'll get the inverse, a bit less capable than frontier but super fast models. Like this 14k tok/s beast https://chatjimmy.ai/ from Taalas (who got acquired by AMD recently).
GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a challenge for older systems.
To the extent that cryptocurrency moved off ASICs, it was because of interest shifting to different cryptocurrencies that were specifically designed to be harder to mine on an ASIC than Bitcoin's compute-heavy, memory-light hashing.
I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to design an LLM architecture with an overt intention to make it better suited to GPUs and hard to target with ASICs.
Yet it doesn't seem that ASICs will have any particular advantage over consumer hardware since AI is very memory heavy, which is (right now) expensive no matter how you package it. And the compute is just simple matrix multiplication, which is almost entirely what GPUs were meant to do anyway.
Go back and correct your idea that consumer hardware made asics obsolete. Then we can figure out if asic or asic like devices for inference will have no advantage.
Yeah! Nobody needs chatjimmy.ai. Nobody needs their results to come back instantly instead of at 10 tokens per second. 66MHz Pentiums ought to be enough for everybody.
ASIC vs GPU doesn't make a ton of difference when both are relying on commodity DRAM; in that sense, LLMs are more like the anti-ASIC cryptocurrencies. But the actually interesting ASICs are the ones that ditch the commodity discrete DRAM chips. They lose out on the memory density and thus struggle to scale up to the largest models, but for what does fit onto a Cerebras wafer or a Taalas chip, the speed is phenomenal. They have a real shot at securing the "smart enough, and really fast" segment of the market.
And it seems more plausible to me that an ASIC architecture rather than GPUs would be able to best make use of something like wafer-bonded custom memory to approach the density of discrete DRAM while retaining the extremely high bandwidth that comes with arbitrarily wide interfaces and minimal PHYs.
My thought was that ASICs turned out not to be worth it for crypto mining because consumer hardware evolved fast enough to do it, while also being cheaper and having some resale value, while ASICs are useless besides mining and have no resale value.
So I'm extrapolating this same idea to LLM inference.
For the problems ASICs exist they vastly outperform general hardware. Typically both in absolute speed and efficiency.
But it's only possible to make custom ASICs when you have a specific problem to solve. For newer crypto systems they can vary enough parameters that building a flexible enough ASIC to recoup the investment before the algorithm changes and makes your hardware useless.
For problems where the problem to solve remain in the problem space the ASIC can solve there is no point to use a thing else.
running my own locally. I just set the tasks to start when systems go idle over x. Read and copy only to external drive projects, codes, ect for review. I review the reports the changes and apply them myself or correct them. Is it slower than say throwing it into fable yes. But I don't have to be monitoring it 24/7
That's because 5 years ago it was still brand new for a computer to be able to speak English. 5 years later, we have accepted that LLMs can speak English and we expect them to do useful things.
OWC Express 1M2 (Thunderbolt 5, single M.2 NVMe) — three of them, two on the Mac's own ports and one behind an OWC Thunderbolt 5 hub since the machine has three ports.
Each enclosure tops out at about 7.1 GB/s on whole-file reads regardless of the drive inside (a 2 TB SN8100 measures the same as the 1 TB);
the drive behind the hub reads 5.7 GB/s and falls with queue depth.
No matter how many external drives you gather, the data coming from them must be squeezed through the peripheral interfaces of the Apple SoC.
So your CPU, made by Apple, Intel, AMD etc., has a number of PCIe lanes and a number of USB/Thunderbolt ports for connecting peripherals.
Those have an aggregated throughput, which sets an upper limit for the amount of data that can be read per second from all the peripheral devices.
In a given computer, usually not all the lanes and ports of the CPU are actually connected, so the limit may be even lower.
In desktop PCs and mini-PCs, usually only 4 + 4 = 8 PCIe lanes are available for SSDs, and when there are more SSD sockets they share some of those lanes.
A much higher SSD throughput could be achieved in a desktop PC by using the GPU connector with an SSD adapter for M.2 SSDs, which has 16 PCIe 5.0 lanes, with a 64 GByte/s throughput.
Taking out the GPU might actually be OK for doing AI inference, because a beefy CPU like a Ryzen 9950X should be able to keep up with a reading throughput of 88 GB/s from 6 SSDs (2 on the motherboard and 4 on the add-on PCIe card), while computing inference in the INT8 or BF16 formats, so the absence of the GPU would not reduce the inference speed when it is limited by the speed of reading the weights.
In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)
When you have heterogeneous SSDs, e.g. you mix PCIe 5.0, PCIe 4.0 and Thunderbolt interfaces, you can obtain a greater throughput by managing in software the distribution of data, than by using RAID0.
RAID0 works fine only when all the interfaces have the same speed.
If one SSD is twice faster than the other, in order to achieve maximal throughput, you must take care to place the data in such a way so that you will need to read twice more data from the twice faster SSD.
In general, you must distribute the data so that the amounts read from each SSD are proportional with the throughputs of the SSDs.
One could write a modified RAID0 device driver, which would use unequal stripes, with widths proportional with the SSD throughputs, but I am not aware of any such already existing RAID0 driver.
This would not work with HDDs, because the time to seek between 2 partitions of the same device would cause abysmal performance, but on SSDs this should work fine, even if some SSD controllers might still have a lower throughput when reading non-sequential pages.
LLMs tend to care for sequential bandwidth more than random access. From that perspective, Optane probably doesn't offer much of an advantage vs other NVMe drives even from the same era while losing out greatly to newer drives with faster busses to fill. There could be something special to this particular instance but https://github.com/argonautlabsai/deltafin/blob/main/k3-publ... and the overall tok/s numbers seem to suggest it's just the sequential that's playing in.
Most desktops and mini-PCs can read simultaneously from only 2 internal M.2 SSDs, 1 of which is PCIe 5.0 and the other is PCIe 4.0.
They may have more M.2 sockets but usually except for the first all the others share the same PCIe 4.0 path.
Those 2 M.2 SSDs have a theoretical maximum aggregated throughput of 24 GByte/s, but what is achievable in practice is typically only slightly above 20 GB/s.
If you have 1 or 2 USB 4 / Thunderbolt 4 ports, the aggregated throughput can be increased with external SSDs, like in TFA.
With 2 internal SSDs and 2 fast external SSDs, a higher throughput than in TFA is possible.
What was the decision to leave the LLM to write the detail section rather than author it yourself? There's a growing resentment about asking people to read LLM-produced words, especially if it's a large amount to read. Not sure if you were aware of that or not (I think there's been links to surveys / polls just recently on H.N)
We’re reaching quadratic slop. Slop projects that don’t understand what they’re shipping built on top of slop projects that also don’t understand what they’re shipping. Magnificent.
> In an April 1985 InfoWorld editorial, James Fawcette wrote that Gates had said something like: “When we set the upper limit of PC-DOS at 640K, we thought nobody would ever need that much memory.”
So yes, Bill Gates denies that story. So it just depends on who you believe.
But to flat out say he never said it is too confident.
To be fair, nobody has upgradable memory in any system that has enough memory bandwidth and compute power to run LLMs with decent performance. It might be interesting to compare against some decade-old x86 server or workstation stuffed full of LRDIMMs to reach 1.5–2TB of RAM, but the bandwidth would be only slightly faster than a desktop today with high-end DDR5: nowhere close to GPU bandwidth. So performance would still suck.
Designing for extreme expandability comes with pretty steep tradeoffs.
Sure, if you want to make a comparison where the price tags aren't the same order of magnitude, then a recent server is obviously going to be powerful. But since the baseline of this comparison is a laptop and several Thunderbolt SSDs, the kind of servers or workstations with 1.5–2TB of RAM that you can reasonably compare against would have to be the really old ones.
And despite the theoretically high memory bandwidth of recent EPYC CPUs, approximately nobody who can afford one is doing LLM inference on them.
You would have the same issue with a GPU. I have 8 slots on my CPU but it cant handle 6TB of ram. I cant even buy 2TB for it because even though the mobo is rated for it, nobody ever made those chips (someone will sell me 8x256gb 8Rx4 for $15k but there’s no way the memory controller can handle that). 1TB is $10k and forced it to run slower.
There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate suggests, because a 512-token prompt already takes 6 minutes to load
Should LLMs be designed to be modular, so that instead of needing access to the whole model, for a given prompt, only a small subset of the model would be used? If knolwedge was sufficiently modularized, most of it could be ignored.
Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.
No, not really, current MoE limit the computation, not memory requirements. Router experts are not "sticky" enough to achieve what robrenaud describes - they'd have to be chosen per prompt, or at least per chunk, not per token.
Experts vary per token in MoE, there is maximum flexibility. Good for driving down loss, bad for locality/gpu memory/bandwidth.
If expert selection were more constrained, inference systems could take advantage of it. Keeping experts cached would mean not needing to load them from disk/ram every token.
132 comments
[ 3.3 ms ] story [ 9.7 ms ] threadso... enough for 1 question + 1 long answer?
Quick question - how much space is required on the SSD? e.g. on M5 Max 128GB 4TB SSD - would it fit on just internal SSD?
Such of a Claudism. Not criticizing, just noticing.
It is cool that they got it to work at all.
The entire history of innovation is filled with people doing something just to see they can get it to work, even if badly, and then people continue to iterate on that until it works better, then works well, and then is so obvious people would never even question it. But it all starts with someone doing it to scratch an itch.
Neural networks, the foundation of our current AI revolution, used to fit well into the "neat, but practically useless" category.
Sure there are countless "but why though?" experiments that don't pan out, but that's just the cost of exploration. There can be no step-function innovation in a world where people only do things that make immediate practical sense.
What are you going to do with a computer? I've always hated this attitude. We do these things because they are interesting to us, for the fun of exploration, because we enjoy learning, because we want to iterate and improve, to make the world better, or any plethora of reasons that involve intellectual curiosity of some sort.
> Just imagine someone asking this question today.
to be fair, considering how young folks live whole life in a smartphone nowadays...
"We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard".
Doing something like that at 1 token per second may be useful.
because prefill currently reads each layer's experts once per 64-row pass (~9 TB of reads for a 1.4 TB model).
Fix is scheduling and it's the next thing being built; once prefill reads each expert once per layer, the one-token-out classifier pattern becomes the sweet spot rather than the worst case.
Frontier models are amazing, but what will really be useful for us is having models and hardware so efficient that you can run useful LLMs locally. One of my favourite LLMs to this day is still my jail-broken gemma4 12b because it's small enough to run on my computer, but also 100% local and free as in liberty.
I get weird interruptions. I can only guess its related to some kind of timeouts in the harness or something.
Its not a problem of the model of course, but it seems impractical atm.
I wonder if anyone else had this kind of thing happening.
Which tasks? I think you're underestimating how token hungry current proposed workflows are.
GPT-6-astra runs at like ~40 tok/s, I have a hard time imagining what could be accomplished with that type of model at 10k+ tok/s when in the hands of the public. Will certainly make cybersecurity a challenge for older systems.
https://openai.com/index/cerebras-partnership/
I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to design an LLM architecture with an overt intention to make it better suited to GPUs and hard to target with ASICs.
And it seems more plausible to me that an ASIC architecture rather than GPUs would be able to best make use of something like wafer-bonded custom memory to approach the density of discrete DRAM while retaining the extremely high bandwidth that comes with arbitrarily wide interfaces and minimal PHYs.
So I'm extrapolating this same idea to LLM inference.
But it's only possible to make custom ASICs when you have a specific problem to solve. For newer crypto systems they can vary enough parameters that building a flexible enough ASIC to recoup the investment before the algorithm changes and makes your hardware useless.
For problems where the problem to solve remain in the problem space the ASIC can solve there is no point to use a thing else.
Maybe a dumb question.
https://github.com/argonautlabsai/deltafin/tree/main/k3-publ...
(0.535 vs 1.038 tok/s at 128 tokens), since one fast drive still has to serve all 16 reads per layer while four drives split the load
https://raw.githubusercontent.com/argonautlabsai/deltafin/ma...
Each enclosure tops out at about 7.1 GB/s on whole-file reads regardless of the drive inside (a 2 TB SN8100 measures the same as the 1 TB);
the drive behind the hub reads 5.7 GB/s and falls with queue depth.
Details in the README's hardware section
or 40 macbooks with each 4 ssd. to get 40 tps.
So your CPU, made by Apple, Intel, AMD etc., has a number of PCIe lanes and a number of USB/Thunderbolt ports for connecting peripherals.
Those have an aggregated throughput, which sets an upper limit for the amount of data that can be read per second from all the peripheral devices.
In a given computer, usually not all the lanes and ports of the CPU are actually connected, so the limit may be even lower.
In desktop PCs and mini-PCs, usually only 4 + 4 = 8 PCIe lanes are available for SSDs, and when there are more SSD sockets they share some of those lanes.
A much higher SSD throughput could be achieved in a desktop PC by using the GPU connector with an SSD adapter for M.2 SSDs, which has 16 PCIe 5.0 lanes, with a 64 GByte/s throughput.
Taking out the GPU might actually be OK for doing AI inference, because a beefy CPU like a Ryzen 9950X should be able to keep up with a reading throughput of 88 GB/s from 6 SSDs (2 on the motherboard and 4 on the add-on PCIe card), while computing inference in the INT8 or BF16 formats, so the absence of the GPU would not reduce the inference speed when it is limited by the speed of reading the weights.
In particular you can still get used Optane SSDs on eBay, although they’re fairly pricey. (not the bogus m.2 ones that are slower than a halfway decent consumer NVMe)
RAID0 works fine only when all the interfaces have the same speed.
If one SSD is twice faster than the other, in order to achieve maximal throughput, you must take care to place the data in such a way so that you will need to read twice more data from the twice faster SSD.
In general, you must distribute the data so that the amounts read from each SSD are proportional with the throughputs of the SSDs.
One could write a modified RAID0 device driver, which would use unequal stripes, with widths proportional with the SSD throughputs, but I am not aware of any such already existing RAID0 driver.
This would not work with HDDs, because the time to seek between 2 partitions of the same device would cause abysmal performance, but on SSDs this should work fine, even if some SSD controllers might still have a lower throughput when reading non-sequential pages.
They may have more M.2 sockets but usually except for the first all the others share the same PCIe 4.0 path.
Those 2 M.2 SSDs have a theoretical maximum aggregated throughput of 24 GByte/s, but what is achievable in practice is typically only slightly above 20 GB/s.
If you have 1 or 2 USB 4 / Thunderbolt 4 ports, the aggregated throughput can be increased with external SSDs, like in TFA.
With 2 internal SSDs and 2 fast external SSDs, a higher throughput than in TFA is possible.
So yes, Bill Gates denies that story. So it just depends on who you believe.
But to flat out say he never said it is too confident.
Designing for extreme expandability comes with pretty steep tradeoffs.
And despite the theoretically high memory bandwidth of recent EPYC CPUs, approximately nobody who can afford one is doing LLM inference on them.
Maybe a hyopthetical model of 5T of indexable weights could be used with only 50 GB of GPU ram, efficiently, because it stays resident in the GPU.
If expert selection were more constrained, inference systems could take advantage of it. Keeping experts cached would mean not needing to load them from disk/ram every token.