Thanks! Yeah this is a major consideration. I have looked at power consumption throughout runs in the past (https://esologic.com/gpu-server-benchmark/#gpu-box-benchmark) and found that for many of these enterprise class cards, they're happy to slam right into the max TDP. So, for doing actual work, you'll be living up closer to the rated TDP of the cards. Recording power consumption is easy on nvidia and I'll likely add this to future versions of the benchmarking tool.
Darn, I was hoping to see bc-250's (aka PS5 chips) in there. They've recently become popular for inference and they are only about $200 on ebay. They hold a special place in my heart because I deployed 20k of them and I'm glad to see they are finding a purpose now and not just e-waste.
"The results showed that these GPUs can still deliver significant compute power at a fraction of the cost of newer models, making them attractive for budget-conscious users." -Mistral AI
No mention of the venerable Tesla P4. 75W peak, 8GB VRAM, about $80 (£60).
I have 6x P4s, a Xeon E5 2696v3 (36 threads, 3.8ghz peak but all core turbo unlocked, so 6 cores at 3.8Ghz - about 8 cores at 3.5ghz, or all cores at 3.1ghz), 48GB DDR4, all fit into a micro atx case running on a 650W MSI psu.
This gives me a virtual 48GB GPU (llama.cpp ftw) to backup that 48GB of RAM.
I typically see scores of at least 7-12t/s on 20-30B Q4KM size dense models, on a 32K/48K/64K context, adequate for modern inference.
The pain point is the prompt loading, it is far far slower, minutes not seconds, than modern tensor core 8GB 5060s (my other machine's 2x GPUs) but is quite similar in regular inference speed once it has loaded.
For an easier change than HTML or SVG, try running them though pngcrush to make the graph images much smaller. Won't give you the lossless vector quality, but you should be able to keep these image files visually indistinguishable at much lower size.
When I wanted to tinker with self-hosted models, I bought a couple of Radeon Pro V620 GPUs, because they're 32GB, still supported by current ROCm releases, and a few years newer than the similar-priced 32GB Nvidia cards (which are all EOL). They're a little faster than the old Tesla stuff, as well. 64GB is enough to run Gemma 4 31b 4-bit QAT with pretty big context at a respectable interactive speed (30+ tokens per second sustained).
That said, even the old Radeon Pro stuff has gotten more expensive on eBay, so I'm not necessarily recommending cheap old server cards that need custom-printed fan shrouds to operate in a consumer PC. Probably better to buy the Radeon AI Pro R9700 for $1400, which will be faster, supported for many years, and has a fan already. Or, maybe even the Intel ARC B70 for $1000.
The B70 is woeful with respect to software performance today, unfortunately, and your stuck using Intels forks of things and it still doesn’t get the full expected throughput. Such a shame to be honest.
Yeah, I'd recommend spending a little more for the AMD. As I understand, it's 40%+ faster. And, while ROCm is less mature than CUDA, it is miles more mature than the Intel stack.
A few years ago we got rid of a bunch of K80s at work, they were not only obsolete but had gotten glitchy as hell. I suspect this is from the many heat/cool cycles they went through. When they were running flat out the exhaust air felt like a hair dryer.
Do you have any #s on how old they were at decomm time? There's some suspicion that part of the AI bubble is companies playing games with depreciation, eg assuming that H100/H200s will survive for 5 years.
Depends on the use case, as for hardware h265 codecs a rtx 5070 Ti works just as well as the rtx 6000 gpu. Legacy GPU don't support modern codecs, but modern Intel chips have h265 HDR hardware support. Lower <16GB VRAM GPU are not really useful for "AI" model labs, so are often far more economical for rendering media.
One metric that isn't considered is VRAM, as some rendering pipelines still rely on composited baked-scenes to reduce each areas memory requirements.
In general, the $/performance unit will depend on what you are doing, but there is 1 more thing to consider... Old GPU use mystery binary BLOB drivers no longer maintained on modern kernels. You might get the software to work with a legacy Windows GPU driver, but the key takeaway concept here is "might". =3
I’m obviously not the intended audience for this, and I understand this hardware is not useful for it, but I can’t help but feel an extra twinge of disappointment that there’s no mention of PC gaming anywhere in a post about GPUs in the comments here on HN. It says a lot.
37 comments
[ 2.6 ms ] story [ 26.4 ms ] thread> They hold a special place in my heart because I deployed 20k
Sounds like something I'd love to hear more about if you can share
I have 6x P4s, a Xeon E5 2696v3 (36 threads, 3.8ghz peak but all core turbo unlocked, so 6 cores at 3.8Ghz - about 8 cores at 3.5ghz, or all cores at 3.1ghz), 48GB DDR4, all fit into a micro atx case running on a 650W MSI psu. This gives me a virtual 48GB GPU (llama.cpp ftw) to backup that 48GB of RAM.
I typically see scores of at least 7-12t/s on 20-30B Q4KM size dense models, on a 32K/48K/64K context, adequate for modern inference.
The pain point is the prompt loading, it is far far slower, minutes not seconds, than modern tensor core 8GB 5060s (my other machine's 2x GPUs) but is quite similar in regular inference speed once it has loaded.
Man, I wish I lived where you guys lived.
That would cost me about $70/month ($0.15/kWh) or someone in California about $234/month ($0.50/kWh).
Do you pay $1/kWh and make ~$1500/month or something? I can’t make the math work for your case.
This is an interesting article though. Bookmarking since my dual e5-v4 system is unplugged until summer is over.
That said, even the old Radeon Pro stuff has gotten more expensive on eBay, so I'm not necessarily recommending cheap old server cards that need custom-printed fan shrouds to operate in a consumer PC. Probably better to buy the Radeon AI Pro R9700 for $1400, which will be faster, supported for many years, and has a fan already. Or, maybe even the Intel ARC B70 for $1000.
https://www.pugetsystems.com/pugetbench/creators/davinci-res...
https://www.pugetsystems.com/pugetbench/creators/premiere-pr...
In some cases is better in lower passmark scores:
https://www.videocardbenchmark.net/gpu.php?gpu=RTX+PRO+6000+...
Blender is heavily bottle-necked by ray-tracing and de-noising operations:
https://opendata.blender.org/benchmarks/query/?compute_type=...
One metric that isn't considered is VRAM, as some rendering pipelines still rely on composited baked-scenes to reduce each areas memory requirements.
In general, the $/performance unit will depend on what you are doing, but there is 1 more thing to consider... Old GPU use mystery binary BLOB drivers no longer maintained on modern kernels. You might get the software to work with a legacy Windows GPU driver, but the key takeaway concept here is "might". =3
I bough an Octominer for ~$150 which has power and PCIe slots and a basic Celeron and should let me expand to as many GPUs as I want.
I considered the P100s but I think the V100 16GBs are a better deal at $250. The 32GBs are way too much though.