> Because of this I got a motherboard with slow GPU interconnect. It’s good for running many small experiments in parallel (which is my main use case) but horrible for any models split across gpus.
:( you paid a professional pc builder and you weren't told this?
It doesn't cover risk. If one or more gpus dies, who pays for it? If you rent, you are guaranteed to be insulated from this risk. But owning, you might not have the best return policy from the vendor. And if you are actually at fault for breaking it, they have every right to deny a return. Or if your apartment is burglarized or catches fire (possibly from overloading the circuit) you are out the entire investment.
In the article, he wrote "I tried to insure it under my renter’s insurance policy. They didn’t like that. I had to get business insurance to cover it.“， but he didn't say how much it cost, either.
I have four old 24gb Nvidia cards. They're not great but they're not useless either. The problem is that I haven't really figured out a good way to actually use them.
Genuine question; would anyone here recommend any specific motherboard to best utilize these cards?
Just curious OP (if you're the one posting) -- what do you mean by independent researcher? What are you researching and are you making $$ from it or are you living off previous built up savings? Seems like an interesting path. What research have you looked into so far?
In the last year, I have bought an M3 Ultra Mac Studio with 512 GB, a Macbook Pro M5 MAX with 128 GB and an RTX 6000 Pro. I have spent around $25k so far, not including electricity. I figured worst case scenario I can sell them in the next year and only take a haircut as opposed to losing my entire investment.
In comparison to just spending for tokens, the tokens would have been much cheaper and much much faster. I've been running against Gemma4:31b, Qwen3.5 and 3.6, and getting local LLMs to solve AMC 8/10 math questions and it's about 10-100x slower than just doing it online. When I tried it with ChatGPT late last year, it took about one night and $25 to solve about 1000 questions. Using my RTX 6000 and M3 Ultra and Gemma4:31b on both, it answered about 40 questions in 7 hours and I haven't checked how good the answer is yet. At 800 watts (600 for RTX and 200 for M3 Ultra) and running for 7 hours, it solved around 40 questions.
At the very least I'm going to try to sell my M3 Ultra if I can find a reliable place to sell it without getting ripped off by scammers.
This is, sadly, obvious and inevitable in retrospect.
The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, and it's much easier to move your AI workload than to move yourself.
This is a completely different dynamic than renting houses or apartments, as you can't really rent out the same house to different people at different times of day.
Only feee if you are using 100% efficient resistance heating. A heat pump can be up to 400% efficient. And natural gas is a lot cheaper per dollar. So for most people… it saves a tiny bit but not free.
Yep, the great theoretical promise of local models remains theoretical, no matter how much die hard-engineers want to push it...Who would have thought, right?
I don't think this changes the final conclusion - but have you considered calculating against depreciation -- i.e. figuring out how much your M3 ultra is worth today, and only charging yourself for the delta? In my mind you might even have made money on the hardware.
Same as it ever was with "cloud" eh? The advantages of small-scale on-prem are never cost or quality, they are strictly privacy and sovereignty. No one can rug pull you with a week of Claude regressions. No one has access to your sensitive data.
This is a difficult calculation to make because you wouldn't rent time on the exact same system in the cloud. Depending on what you're running, a bigger server with better inter-GPU interconnects in the cloud might complete the task so much faster that the additional per-hour expense is more than covered.
So some things have changed since this rig was first built (2024). The most relevant is that $6800 RTX 6000 Ada 48GB has arguably been supplanted by the $9500 RTX 6000 Pro 96GB.
The Ada has a memory bandwidth of 960GB/s. The Pro has 1.8TB/s and about 40-50% better performance so is at least equivalent in processing power, much better in memory bandwidth (important for inference) and can hold larger models on a single card.
I've considered buying a rig with 1-2 6000 Pros for similar reasons but I want to see what happens with this year's Mac Studios with a likely M5 Ultra. Macs have a shared memory architecture whereas NVidia segments the market based on max memory where the biggest consumer card (RTX 5090) has 32GB of VRAM but still excellent memory bandwidth (1.8TB/s). A RTX 5090 rig will still trounce a Mac Studio seems to be the conventional wisdom. Despite being able to hold larger models and being able to chain Mac Studios on TB5, their lower memory bandwidth (~900GB/s) and lower overall GFLOPS mean they still come out behind.
That being said, the current Mac Studios are relatively long in the tooth, being released in 2024.
I'm still not sure any of this is really wroth it because things are still changing so fast. I think there's a decent chance of a number of large AI companies going bust in the next 2-3 years such that you'll be able to buy enterprise AI hardware at cents on the dollar, a bit like how Google bought data centers in the post-dot-com crash.
But anyway, nowadays I'd be looking at the RTX 6000 Pro as the sweet spot, having anywhere from 1-4 in a single server.
The electricial issues the author mentions are interesting. I hadn't really thought about the max amperage on a residential circuit. In a DC, these would typically operate on three phase power and much higher overall amperage. I wonder if there's a device you can buy that can combine multiple residential circuits into a single power source for a server this power hungry?
The idea is similar to maintaining on-prem vs cloud
Cloud is optimized for development velocity but its nature of high margin business eventually makes on-prem more promising
It could be too late but it might be worth looking into tax saving if you have a business. Depreciation of asset is a loss and may deduct your income. (I'm NOT a tax expert)
out of curiosity, did you check how much would cost to rent a cage in a colocation space? Having to power your computer from two different outlets sounds wild..
I can't imagine spending $48K on a home GPU server, but I did just splurge and buy a PC with an RTX 5090, specifically to hold the largest models you can fit in 32GB. It's a top of the line PC with water cooled high end CPUs, 64GB RAM, RTX 5090 for $5K. To me the jury is still out whether this was a worthwhile investment, but I do expect to use this machine for a decade. I don't run it at 100% power (it's mostly idle, except for times when I'm training or doing batch inference). It has the nice property of being blackwell generation, similar to the machines we use at work.
It just scares me to own a box that is $48K in my house, especially if it breaks, or gets stolen.
I was looking at Ultras for sale, and had same worry, so didn't end up getting one. I have some peace of mind comfort about applecare and technical repair, but i couldn't find insurance that would cover theft (or rather, i did, but it was too expensive)
Last fall, seeing the writing on the wall, I pieced together an "AI" rig, 96GB ram, 2x RTX 3090, 9950X - not exactly top of the line, but it came in around CDN$3000 all in all, with most parts second hand. I don't think I could build that for CDN$10000 today.
I've been using it pretty steadily for a variety of personal projects, and the only improvement (aside from the obvious "more VRAM") I feel pressed to make is a portable AC unit / some kind of a focused cooling solution. The rig raises the ambient temperature in the office by 4C at least.
Now with the murmurs of even the large players reconsidering their AI spend, and usage-based pricing shifts, having a self-contained, owned, and independently administered compute resource is looking better and better.
The research that's presented in another article on the same site is way more interesting than the betteridges law article linked here. It'll be very useful in my own latest project if this research is incorporated into some model I can rent by the token!
(For reference I’m talking about the DFT post from the same blog.) I love that ML is still in the “gentleman researcher” stage where relatively small amounts of startup capital can buy a ticket into frontier research.
For a lot of research questions 6 GPUs is even overkill.
It’s one of the reasons I’m skeptical of the “trillion dollar supercluster” idea [0]. I think what we need is more reasonably smart people investigating medium-sized problems. A “GPU middle class” you might say.
FYI: If you're in a similar situation, think very carefully before you build your own. The $17000 might sound like a lot; but when you take into account your time and risk tolerance, renting might be a much better solution.
'If you google “plugging a PC into multiple outlets”, you get lots of warnings that if you even consider such a setup you will instantly burst into flames. So I hired a professional PC builder make sure it was safe.'
Agreed, if anything an electrician was needed, not a professional PC builder.
The picture shows two power supplies. Powering what is effectively one appliance from different circuits is a definite no-no, and I can't think of any circumstance where it wouldn't be in a home.
If his mains supply was sufficient to run the server and the house in the first place then the simplest solution would be to simply upgrade one of the MCBs/RCBOs on one of the circuits to the required capacity. I am not sure a landlord would even notice something like that, and if the house is wired correctly in the first place, it's unlikely to be dangerous. So going from say, 6A to 12A, on a 20A mains supply is generally fine if the gauge of wiring is correct.
> I thought that I could not get a standard datacenter server because my apartment wouldn’t let me upgrade the circuits, so I needed to have 2 power supplies plugged into different circuits.
Why didn't they just put a higher amp breaker in the box?
> Why didn't they just put a higher amp breaker in the box?
1) note the word "apartment" -- they rent, not own, and doing so not only would likely be illegal, but might also get them kicked out of the apartment.
2) Unless the wiring on the circuit drop, and all the end points are rated to handle the higher current, doing so would be an electrical code violation (and therefore trip into that "illegal" arena that might result in getting kicked out of the apartment).
Most residences are wired using the minimum size wire rated for the installed breaker (because doing so saves costs). So a 15a breaker in the box would mean 14gauge (the US NEC minimum size for 15a circuits) wiring in the walls and 15a rated outlets/switches. Installing a 20a breaker in the box would be a code violation, and in many jurisdictions also illegal.
And all the above is without considering that installing a 20a breaker on wires rated for 15a increases the fire risk tremendously if those wires are now asked to actually carry 20a for any length of time.
UPDATE: Launch was a success! 400K+ views, and multiple companies reached to use my IP. Read more here
It seems that he managed to get what he wanted from the hardware and I'm happy for them.
He said something interesting at the beginning of his post, he compared the cost of the hardware to the cost of his time based on his FAANG salary. Which is an interesting way to think of this, but the rest of the article didn't make me understand if at the end he did save money/time based compared to just rend on the cloud.
Also, outside of the power cost, hardware has other costs too, you need to operate it, maintain it, set it up, etc. all that require time. I mean, even the process of figuring out if it had a good enough ROI compared to cloud, takes from your time (collecting data, analyzing data, etc etc).
Doubt it, feels basically like just an ad to get attention "Oh look, that's where the magic happens" vs running their code on existing infrastructure and thus just showing the results, like everybody else. This "feels" more "tangible".
112 comments
[ 2.8 ms ] story [ 85.5 ms ] thread:( you paid a professional pc builder and you weren't told this?
Genuine question; would anyone here recommend any specific motherboard to best utilize these cards?
In comparison to just spending for tokens, the tokens would have been much cheaper and much much faster. I've been running against Gemma4:31b, Qwen3.5 and 3.6, and getting local LLMs to solve AMC 8/10 math questions and it's about 10-100x slower than just doing it online. When I tried it with ChatGPT late last year, it took about one night and $25 to solve about 1000 questions. Using my RTX 6000 and M3 Ultra and Gemma4:31b on both, it answered about 40 questions in 7 hours and I haven't checked how good the answer is yet. At 800 watts (600 for RTX and 200 for M3 Ultra) and running for 7 hours, it solved around 40 questions.
At the very least I'm going to try to sell my M3 Ultra if I can find a reliable place to sell it without getting ripped off by scammers.
The two major drivers of inference costs are GPUs and electricity. You can't get cheaper GPUs, but you can make existing GPUs not sit idle, and you do that by utilizing them 24/7, processing user B's request when user A is thinking, and handling many requests in parallel, neither of which you can do as an individual. You can get cheaper electricity... by moving, and it's much easier to move your AI workload than to move yourself.
This is a completely different dynamic than renting houses or apartments, as you can't really rent out the same house to different people at different times of day.
Slow? Yes, but ... private. Unconditionally.
And recently with https://github.com/antirez/ds4 one can use just one system to a very, very decent speed and ttft for chat inference. Again, private.
$48000 is equal to 12000 hours of renting an h100, which is about as long as you’d spend at your job for 6 years!
The Ada has a memory bandwidth of 960GB/s. The Pro has 1.8TB/s and about 40-50% better performance so is at least equivalent in processing power, much better in memory bandwidth (important for inference) and can hold larger models on a single card.
I've considered buying a rig with 1-2 6000 Pros for similar reasons but I want to see what happens with this year's Mac Studios with a likely M5 Ultra. Macs have a shared memory architecture whereas NVidia segments the market based on max memory where the biggest consumer card (RTX 5090) has 32GB of VRAM but still excellent memory bandwidth (1.8TB/s). A RTX 5090 rig will still trounce a Mac Studio seems to be the conventional wisdom. Despite being able to hold larger models and being able to chain Mac Studios on TB5, their lower memory bandwidth (~900GB/s) and lower overall GFLOPS mean they still come out behind.
That being said, the current Mac Studios are relatively long in the tooth, being released in 2024.
I'm still not sure any of this is really wroth it because things are still changing so fast. I think there's a decent chance of a number of large AI companies going bust in the next 2-3 years such that you'll be able to buy enterprise AI hardware at cents on the dollar, a bit like how Google bought data centers in the post-dot-com crash.
But anyway, nowadays I'd be looking at the RTX 6000 Pro as the sweet spot, having anywhere from 1-4 in a single server.
The electricial issues the author mentions are interesting. I hadn't really thought about the max amperage on a residential circuit. In a DC, these would typically operate on three phase power and much higher overall amperage. I wonder if there's a device you can buy that can combine multiple residential circuits into a single power source for a server this power hungry?
Cloud is optimized for development velocity but its nature of high margin business eventually makes on-prem more promising
It could be too late but it might be worth looking into tax saving if you have a business. Depreciation of asset is a loss and may deduct your income. (I'm NOT a tax expert)
- https://www.williamangel.net/blog/2026/05/17/offline-llm-ene... - Discussion: https://news.ycombinator.com/item?id=48168198
It just scares me to own a box that is $48K in my house, especially if it breaks, or gets stolen.
I'm not saying it's worth it just that it's not such a crazy amount in comparison.
I've been using it pretty steadily for a variety of personal projects, and the only improvement (aside from the obvious "more VRAM") I feel pressed to make is a portable AC unit / some kind of a focused cooling solution. The rig raises the ambient temperature in the office by 4C at least.
Now with the murmurs of even the large players reconsidering their AI spend, and usage-based pricing shifts, having a self-contained, owned, and independently administered compute resource is looking better and better.
For a lot of research questions 6 GPUs is even overkill.
It’s one of the reasons I’m skeptical of the “trillion dollar supercluster” idea [0]. I think what we need is more reasonably smart people investigating medium-sized problems. A “GPU middle class” you might say.
[0] https://situational-awareness.ai/racing-to-the-trillion-doll...
Not really sure how that makes it safe but OK!
The picture shows two power supplies. Powering what is effectively one appliance from different circuits is a definite no-no, and I can't think of any circumstance where it wouldn't be in a home.
If his mains supply was sufficient to run the server and the house in the first place then the simplest solution would be to simply upgrade one of the MCBs/RCBOs on one of the circuits to the required capacity. I am not sure a landlord would even notice something like that, and if the house is wired correctly in the first place, it's unlikely to be dangerous. So going from say, 6A to 12A, on a 20A mains supply is generally fine if the gauge of wiring is correct.
- muscle cars, with all the stuff, driven occasionally.
- boats, that don't get taken out much
- gamer x, where x=system or laptop or keyboard or mouse or desk or glasses or mousepad or speakers or ... usually with "> too much RGB"
- children
$48k for something constructive even if ai related? no problem, refreshing even.
Why didn't they just put a higher amp breaker in the box?
1) note the word "apartment" -- they rent, not own, and doing so not only would likely be illegal, but might also get them kicked out of the apartment.
2) Unless the wiring on the circuit drop, and all the end points are rated to handle the higher current, doing so would be an electrical code violation (and therefore trip into that "illegal" arena that might result in getting kicked out of the apartment).
Most residences are wired using the minimum size wire rated for the installed breaker (because doing so saves costs). So a 15a breaker in the box would mean 14gauge (the US NEC minimum size for 15a circuits) wiring in the walls and 15a rated outlets/switches. Installing a 20a breaker in the box would be a code violation, and in many jurisdictions also illegal.
And all the above is without considering that installing a 20a breaker on wires rated for 15a increases the fire risk tremendously if those wires are now asked to actually carry 20a for any length of time.
Someone needs to solve proper distribution of packaged GPUs with some Tesla-like wall connector for a consumer grade box that is plug and play.
Maybe John Ternus ends up doing that at Apple since they sit closer to this consumer profile.
It seems that he managed to get what he wanted from the hardware and I'm happy for them.
He said something interesting at the beginning of his post, he compared the cost of the hardware to the cost of his time based on his FAANG salary. Which is an interesting way to think of this, but the rest of the article didn't make me understand if at the end he did save money/time based compared to just rend on the cloud.
Also, outside of the power cost, hardware has other costs too, you need to operate it, maintain it, set it up, etc. all that require time. I mean, even the process of figuring out if it had a good enough ROI compared to cloud, takes from your time (collecting data, analyzing data, etc etc).