An answer to: why is Nvidia making GPUs bigger when games don't need them?
Here is an example from a project I'm working on today. This is the console output of a model I'm training at the moment:
-----------------------------
I1015 20:04:51.426224 139830830814976 supervisor.py:1050] Recording summary at step 107041. INFO:tensorflow:global step 107050: loss = 1.0418 (0.453 sec/step)
I1015 20:04:55.421283 139841985250112 learning.py:506] global step 107050: loss = 1.0418 (0.453 sec/step) INFO:tensorflow:global step 107060: loss = 0.9265 (0.461 sec/step)
I1015 20:04:59.865883 139841985250112 learning.py:506] global step 107060: loss = 0.9265 (0.461 sec/step) INFO:tensorflow:global step 107070: loss = 0.7003 (0.446 sec/step)
I1015 20:05:04.328712 139841985250112 learning.py:506] global step 107070: loss = 0.7003 (0.446 sec/step) INFO:tensorflow:global step 107080: loss = 0.9612 (0.434 sec/step)
I1015 20:05:08.808678 139841985250112 learning.py:506] global step 107080: loss = 0.9612 (0.434 sec/step) INFO:tensorflow:global step 107090: loss = 1.7290 (0.444 sec/step)
I1015 20:05:13.288547 139841985250112 learning.py:506] global step 107090: loss = 1.7290 (0.444 sec/step)
-----------------------------
Check out that last line: with a batch size of 42 images (maximum I can fit on my GPU with 24Gb memory) I randomly get the occasional batch where total loss is more than double the moving average over the last 100 batches!
There's nothing fundamentally wrong with this, but it will throw a wrench in convergence of the model for a number of iterations, and it's probably not going to help in reaching the ideal ultimate state of the model within the number of iterations I have planned.
This is in part due to the fact that I have a fundamentally unbalanced dataset, and need to apply some pretty large label-wise weight rebalancing in the loss function to account for that... but this is the best representation of reality in the case I am working on!
--> The ideal solution RIGHT NOW would be to use a larger batch size, in order to minimise the possibility of getting these large outliers in the training set.
To get the best results in the short term I want to train this system (using this small backbone) with batches of 420 images instead of 42, which would require 240Gb of memory... so 3x Nvidia A100 GPUs for example!
--> Ultimately the next step is to make a version with a backbone that has 5x more parameters, and on the same dataset but scaled at 2x linear resolution... requiring probably around 500Gb of VRAM to have large enough batches to achieve good convergence on all classes! And bear in mind this is a relatively small model, which can be deployed on a stamp-size Intel Movidius TPU and run in real-time directly inside a tiny sensor. For non-realtime inference there are people out there working on models with 1000x more parameters!
So if anyone is wondering why NVIDIA keeps making more and more powerful GPUs, and wondering who could possibly need that much power... this is your answer: the design / development of these GPUs is now being pulled forward 90% by people who need these kinds of solutions for ML ops, and the gaming market is 10% of the real-world "need" for this type of power, where 10 years ago that ratio would have been reversed.
65 comments
[ 3.1 ms ] story [ 46.6 ms ] threadNow I'm just trying to run Stable Diffusion to generate things an it's 30 seconds per iteration at only 512x512 pixels. It looks like I'll have a GPU soon enough if I want to do any development with neural networks.
What an amazing ride... heck, I still remember thinking back in 1980 that the 10 Megabyte Corvus hard drive my friend was installing for a business would NEVER be filled... do you have any idea how much typing that is? ;-)
I’ve seen a single PDF which is the internal developer guide to a certain ARM based processor from years ago and that PDF was well over 1,000x in size to that hard drive. Yes, there were some pictures and charts, but most of that was just text, thousand upon thousand pages of detailed specs.
Whereas the huge machine learning (and also cryptomining) markets can't get enough hardware resources. And those companies are buying these GPUs by truckloads, every month.
Gamers are a very vocal but ultimately completely irrelevant market for these manufacturers. They account only for a tiny part of their income.
But, by definition, those cards aren't meant for average gamers, but those that do play at 4K@120Hz and at which they're hugely and massively useful.
I can't quite fathom the baseline misunderstanding of the market here - isn't it obvious that the most expensive piece of hardware isn't meant for average customer?
Games absolutely need more vector horsepower.
120hz feels so choppy when you're used to 240hz.
Don't get me wrong, there is nothing wrong with it - just you are simply not the market that is making them money anymore.
Every time I read comments like this I feel really good for being able to be perfectly happy with my crappy 24'1080p60 monitor. I swear, every time I see a 4k display or, god forbid, (120|240)Hz display in the wild I just close my eyes immediatly and dread that I could no longer be happy with my lesser experience.
This is one expensive treadmill to walk, moreso with gpu manufactors gutting consumers for the last few years.
[1] https://www.amazon.com/GIGABYTE-M27Q-Monitor-Display-Respons...
We actually have a long way to go before GPUs can really support fully immersive VR. The new 4090 is just barely enough to handle some racing sims like ACC in VR at high resolution and settings.
Even 3000 series cards couldn’t deliver playable frame rates at 4K with all of the graphical eye candy turned on (RayTracing, etc.) in games like Cyberpunk 2077. As with all games, you can simply play with lower settings and lower resolutions, but the full experience at 4K really does require something like a 4080 or 4090.
And of course, next generation games are being built to take full advantage of this new hardware. It doesn’t make any sense to suggest that GPU vendors should just stop making new progress and expect games to stay at current levels of advancement.
The difference between gaming at 4K and 1440p or even 1080p however is not that significant imo. Competitive (and many casual) gamers prefer a high frame rate to high resolution.
But as display technology improves people will just come to expect their games should run in native 4k resolution.
She could always tell the difference between 60 and 120/144, the way she saw the difference between 120 and 144 was comparing the pointer trail. She didn't notice that animations were smoother though, so that's definitely trained.
I hate myself so I run Linux on all my client machines, and if my display goes to 60 I notice by the first animation on my desktop (which is essentially only the start menu).
If you're curious it's easy to set different refresh rates on your monitors and try it out for yourself, the 60hz thing is bogus.
I also notice an awful lot of lag when movies have panoramic shots, they are less than 30hz. There are workarounds with computed frames in most modern tvs that help with this.
You are correct about the hardware requirements - but his point was that 90% of the applications of this high end hardware ends up being in machine learning (and or crypto mining) and not the tiny ultra-high-end gaming market that can afford to pay several thousands for such cards. Most gaming "rigs" are nowhere near those specs.
Nobody would manufacture such cards in quantity beyond a few "show pieces" only for the few very rich enthusiasts that can afford them if there wasn't this non-gaming market, it simply wouldn't be economical.
On top of all that, the halo product effect also plays a role, where it's easier to sell even the lower end models when you can claim to have the fastest model of the generation at the high end.
Furthermore, the current generation's high end tends to become the next generation's mid/low end, so it's really just buying into a product that could last a gamer one extra generation, which offsets the costs (ignoring that $1600 every ~2-3 years isn't really that crazy for someone with stable employment).
There’s only the same handful of graphics-intensive games that are being brought in every online argument and tech review outlet, like Digital Foundry. I can understand it when viewing games from a pure technological standpoint. But if viewed as actual games, you play through demanding single-player games like Metro, Cyberpunk or Control once, and then move on. The vast majority of produced titles play perfectly fine with barely any noticeable difference between the latest and greatest €2000 GPU, and something from a few years back like GTX1080, or a Playstation 5.
I don’t think there are enough demanding games come out to warrant the purchase, especially if you count only the ones that are actually fun to play as a game, and not just admire raytracing lighting or reflections.
If you build it, they will come.
I'm hoping the Steam Deck will save us from that, by presenting a fixed target that games have a good reason to optimize for.
Also game developers are making games on the basis of the average configuration, not the other way around.
You can downvote me for being lazy, yes. But my point is that there's a ton of BS marketing going on in the IT industry, and Nvidia is no exception. Look at the 3nm process, which isn't about gate distance anymore. Etc.
xxXx = Big X is "power". So 9 is highest, 8, 7, 6, 5 (3090/3080/3070) etc.
Bigger has more compute power, usually more RAM and more power requirements.
That's pretty much it. Figure out what you want to do, and then find a card that does that. Want ML? highest RAM you can get. Want high resolution gaming, high ram, high compute power.
The newer generations also have a spate of new gaming, video, and ML technologies that make some aspects of all those things faster. TDLR: More expensive = better.
All these questions need much more powerful GPUs than current top-of-the-line. And game makers know that better graphics often sells the game.
Better graphics sells games, no doubt. But there is also a tradeoff between investing huge amounts of resources into developing/optimizing for what is a tiny ultra-high-end niche that can afford the bleeding edge hardware.
The OP is not disputing this - all he is saying that this stuff is not what drives the GPU market since years ago anymore.
No, the same way he didn't try 200K / 1000fps. It's not really needed.
Even if some people are so extravagant as to got to having it, they're not enough to drive the sales of such cards at the moment, in fact few even have an 8K display or TV, so the combined expense for card and monitor makes it a non-starter anyway.
Which is why the vast majority of those cards goes to ML people.
However, it’s fun as hell
Or is this more like hi-fi zealotry?
Sorry but you are totally wrong. 4k +144hz absolutley needs this kinds of performance. Same as VR
Yes, but the average gamer doesn't do "4k + 144hz" and even if they wanted to, they couldn't afford to care for those Nvidia GPUs at these prices (often not also the required monitor). So this doesn't explain the niche those cards are addressed to (and sales demographics confirm this).
It's a general statement, not an absolute one about what any conceivable game and obsessive gamers would want/uses.
The average Mac user also isn't buying 8000$ Mac Studio with XDR display configuration. This isn't meant for "average users".
Probably I'm completely wrong, but surely scaling down those images, at least for the initial iterations, will reduce noise, and give you vastly better batches and faster iteration leading to better training -- then as it converges you would crank up the resolution
Note that scaling doesn't have to be cubic, there are algorithms that crop less 'meaningful' pixels; giving you smaller files with the same resolution
Also Nvidia would probably go more into custom accelerators (tensor cores) as whole industry is moving more towards bigger VRAM and more tf32/bf16 units for AI training over standard fp32 gaming workflow and that's isn't focus of RTX class gaming cards.
And 4090/3090/2080ti is old Titan class card for prosumers mostly focused on 3d rendering/video editing etc. and also gaming for AI pros they are milking sweet money out of quadro lineup.
If you are rly limited by VRAM you should just go fully TPU not wait for some magic consumer VRAM increase as it won't happen for a few years probably (most games in 4k can't utilise 16gb.)
Not at the rate ML does, and people still do fine playing even latest games with smaller GPUs. Which is why the vast percentage of the buyers for these beefed up cards are ML people, not gamers.
At 1080p, which is what I was using 20 years ago.
I'd rather have 640x420 with better gameplays than yet-another 4K shit
I haven’t seen a relatively modern 1080p title that didn’t look better at 4k.
You can accumulate the gradients of multiple batches before doing the weight update step. This allows you to run effectively much larger batch sizes than your GPU would allow without it.
I work with software for virtual studios. It's very similar to gaming in a lot of ways (even using Unreal Engine) and we really push GPUs to the limits. I'd still count this in the "for gaming" category as we;re still rendering 3D scene within a strict time limit.
What you are doing is the same type of work that I would put in the "pro user" bucket like ML. Actually I do the same thing with the same cards to render synthetic dataset for my ML pipeline, including using UE :)
My contention is not that it's impossible to use more power in games, or that developers are not working on games that will be able to fill all that compute power... but that higher quality graphics are no longer pulling the gaming industry forward like what was going on 20 years, and also that improvements in graphics are more and more about AI enhancements to image quality and less and less about having the power to push more vertices around in real time.
The market for video games doesn't really care about the top 0.5% of PC gamers that can afford to buy the latest GPU from Nvidia every 2 years. The PS4 is still outselling the PS5, if graphics power was the main criterion for the actual market of video gaming that could not happen...