Not supporting 8 bit quantizations is not just some minor optimization missing. An 8 bit quantization of a model is half the size of a 16 bit model. You can load a model with twice the number of parameters using 8 bit quantizations as compared to loading a 16 bit model. GPU ram is always in demand and since 8 bit quants give you <1% performance degredation re: perplexity there's no reason not to.
That's before getting to any potential 8 vs 16 bit processing speed differences the two companies are arguing over here.
The hardware supports 8 bit, but the chosen software (vLLM) does not. The reason AMD chose to do the comparison with vLLM is that it is by far the most used inference library. So for now at least the market disagrees with you on the relative merits of 8 vs 16 bit processing.
In any case, if the comparison was being done in 8 bit, the relative performance between the cards should be roughly similar as it is with 16 bit.
Its more nuanced than this. Sometimes pure 8 bit inference (without any quantization tricks) reduces llm performance enough to impact certain tasks like retrieval.
Its also important to distinguish 8 bit compute from the actual quantization scheme. vLLM in fact supports 4 bit AWQ, but the computation is not done in 4 bits on the GPU, and it wouldn't even be close to pure 4 bit inference even if the hardware and vLLM supported that.
Both GPUs support both formats, so comparing an 8 bit quantized model on one to a 16 bit model on another is nonsense. That'll massively affect the result.
It should also be noted that H100 has 80 GB VRAM and MI300X has 192 GB. So, it can happily load the largest 8 bit model H100 can in 16 bit. Also, like others noted, vLLM is behind on FP8 support, its not a hardware issue.
nvidia is winning with cuda, not hardware. as soon as AMD can crack the rocm/cross compile nut they will be on top. dominating server market already. and in pure raster perf, they’re dominating the graphics space as well.
PyTorch is using cuda on the backend. You might not see it, but that's what's going on. So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA". You can also say it's a lot of other things like ROCm. It's best to think of the python in PyTorch as more of an API.
It's not that simple. Pytorch has its own computing model, calling it just APIs is really downplaying the phenomenal works behind the framework. Pytorch put Tensorflow to shame even though the latter had huge first mover advantage.
For sure it's not that simple, but the person I was responding to was at the stage of understanding that they were thinking PyTorch was better than CUDA. I'm trying to communicate to them, not someone with a more detailed understanding (I mean one of the top comments here thinks ML is "just matrix multiplies"...)
Tensorflow had first mover advantage but they also had first mover disadvantage. TF isn't that easy to write nor to debug. PyTorch got to see everything that was wrong with TF and fix it. No doubt Meta has put in lots of amazing effort. It is an incredible piece of software.
I'll also add that PyTorch does a lot of great stuff that people don't use or recognize. There are a lot of good distributional works out there for prob and stats people that give you cuda acceleration. Really a lot of numpy can be replaced with pytorch and its great. But be warned of FMA and other optimization differences.
Their point seems sound. Restated: Using pytorch [is better than] using CUDA directly (i.e., ml devs interact w/ pytorch apis not CUDA apis). And if you're using a wrapper anyway, whether CUDA is used or some other backend is less crucial (than if directly writing to a GPGPU api) so long as the wrapper works as expected.
> So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA".
Not true. PyTorch has had an MPS backend[1] for Apple Silicon for quite some time now. And lately, they've also added a HIP backend[2] for AMD hardware. I've written CUDA kernels for PyTorch in the past, it'd have been a lot easier if PyTorch were a thin wrapper over CUDA, but it isn't.
Sure, I hope you can recognize that I was trying to convey that there were a lot of things. I mention ROCm because it is more than HIP and is what you'll see when you install pytorch. I hope you recognize that I was intending to convey that it was more than a wrapper. I'll admit that I could have phrased my intent much better, but I was trying to write to an audience of one with my best interpretation of their comment.
> So there's no "PyTorch vs CUDA" because "PyTorch __is__ CUDA".
==> You can also say it's a lot of other things like ROCm. <==
I mean my intent was just to communicate that PyTorch runs CUDA in the backend, which was what I meant by API, because it seemed the person I was responding to wasn't aware of this backend code you're interfacing with. I thought adding the same phrasing about it also "being" ROCm and using "such as" and "other things," that others would have allowed anyone with more knowledge to infer that I'm aware of the generalization but I guess I was deeply mistaken given the replies that are certain that I can't read the pytorch homepage. Though I'm not quite sure how you came to the conclusion that I wasn't aware of ROCm support since I did explicitly mention it.
Do you have feedback for how I can better communicate? I seem to be running into failures like this a lot lately where I feel like I can point to where I clearly stated something but that doesn't matter because communication is about getting the person on the other side to receive the message. I've been scratching my head about this and could definitely use the outside perspective.
"PyTorch uses the best backend it has available. If that's CUDA, it will use CUDA. It will also use ROCm on AMD GPUs and optimised BLAS libraries on CPUs. These backeds are selected at compile time though, so you need to install a version of PyTorch with optimised support for your hardware. "
I wrote that the usage of PyTorch is much higher than bare cuda, not making a claim that one is better than the other.
Obviously you can use cuda to write more than ML algos, but that’s its overwhelming use case.
Thus my point is if AMD ported PyTorch to their hardware (and PT is not cuda-only) it would create a more level playing field on which they could compete (in a cousin vs cousin battle) by eliminating the switching cost for the majority of developers.
This is the same reason cpu mfrs have compiler ports but generally don’t have to worry about instruction set level compatibility.
But you also suggested that cuda is less critical. But there's so much that is dependent on cuda, even if it is directly visible. You're right, that if AMD does fully integrate into Pytorch, something they've been working on btw, then yes this will be a big step up for AMD as ML is one of the biggest sectors. But it's important to realize that this is not as easy of a task as it sounds. They have been working on this for years and every step of the way Nvidia is still improving too.
> Thus my point is if AMD ported PyTorch to their hardware (and PT is not cuda-only) it would create a more level playing field on which they could compete
It is ported (I think by AMD). I use it daily. Biggest problem is that the people maintaining PyTorch aren't very interested as they don't use it themselves. Hence there are silly bugs in it. Users contribute fixes and they get closed down as "unsupported" or get plain misunderstood.
> ultimately dlss/frame gen is a hack that no one on a high end card would ever use.
The irony of those technologies is that they shine best on high end cards.
If you're on a low end card and say pushing the limit of a game and are upscaling 720p to 1080p, then dlss can't do miracles, there will be tons of defects due how little information about the scene the upscaler has. But if you're on a high end system trying to upscale 1440p to 4k then even not only is the upscaler less likely to make a mistake, it's also much less noticeable.
Same with framegen, on a low end system that's getting sub 60fps, the framegen not only adds noticeable input lag, but often makes noticeable defective frames in between, if your system was already pushing above 60 then not only is the input lag much more bearable, the possible defective frames stay on screen for so little that you'd have to be paying attention to it to notice them most of the time.
They're still useful for low end systems, they can prolong their lifespan quite a bit for current gen games, but as this tech becomes the norm, developers rely on them to hit performance targets and your card will be outdated just as fast as older cards used to.
As much as I wish this was true, a lot of new games require you choose between lower settings or DLSS if you want to maintain 1440p resolution and a decent FPS. This was true last generation with the 3090 (from personal experience) and it is true with this gen as well (based on second hand from my friends with current gen high end cards).
You may not be running DLSS with a super high scaling factor but I almost guarantee DLSS or DLAA is going to be on for most games nowadays if you want to keep framerates stable on ultra.
> but simply because they were first to market and have game studios using their stuff.
You say this as if momentum isn't one of the most powerful forces in the universe. Many a better technology has gone extinct because a competitor simply had more momentum. It's difficult to catch up and it's difficult to stop.
Nvidia also is aware of this and not making mistakes that many others do by sitting back and relaxing. They are doing everything they can to not let AMD even begin to compete, but they are also very aware that hardware is far from the full package. 75% max, but probably less.
Frame gen is a hack, but DLSS upscaling isn't a hack, it's excellent. Native raster looks pretty bad compared to it due to aliasing, and FXAA/TAA/MXAA don't work that well compared to DLSS at removing aliasing.
I don't disagree with frame gen, but I think DLSS, FSR (and XeSS) are going to replace traditional anti-aliasing. Especially with older techniques like MSAA not being compatible with newer deferred rendering methods. Even at native resolutions, you still want anti-aliasing. These are simply the current bests at that.
Alan Wake 2 will use FSR or DLSS for AA even at native resolutions, with no other AA option. I expect more games will do this in the future.
In my experience it’s virtually identical to the 4090 in gaming, particularly when you consider that a 4090 is 2x the cost. The 4090 will do better, but not substantially. And in lots of games optimized for consoles (PS5 and Xbox are AMD based) the 7900XTX is actually the better card. I play a lot of COD and the 7900XTX outperforms there on framerate.
It's even neck in neck in titles where Nvidia paid the developers to trash the console optimized path (which would have been ported as an identical DX12 path), and write an entirely new, much slower, DX11 path.
Why?
Because Nvidia's DX12/Vulkan-style path in the driver is absolute trash, while their DX11/OGL4-era path is very well tuned. Nvidia's "advantage" is entirely software, the emperor wears no clothes.
With games that Nvidia's money couldn't cripple games, the 7900XTX, the card that both costs less and uses less watts, beats the 4090 in 1080p, 1440p, and 4k targets.
It's the edge cases that are the problem. I regret getting a 7900xtx over a 4090 because RDNA3 is godawful at game console emulation. Even the 1070 I replaced runs rings around my 7900xtx when it comes to Switch emulation.
I have a hard time believing this. Switch is 720p low quality. Anything should be able to match that. Edge case indeed though - why not just play on an actual switch lol they are like $200
Not if you adjust for cost. 7900XTX is much better value raster in everything except perhaps raytracing (although even then there are some implementations where 7900XTX is better).
Gamers buy 4090 only because DLSS is significantly better than FSR.
The superior ML upscaling of DLSS is the why Nvidia dominates the gaming sector. AMD's FSR upscaler introduces noticeable atrefacts and aliasing especially during motion, the tech isn't close.
Better value I absolutely understand. But I thought they were claiming its out and out better/superior? As far as I was aware, the 4090 still edges out the 7900XTX in pure raster in most games, but again I'm not certain on that
I've seen multiple people mention training LLMs successfuly on AMD hardware, I think the software stack is close enough on these data center chips that you can productively use them and the bigger question, is if you can get your hands on them more easily than H100s.
I don't quite understand. Most of LLM inference is MatMul and Softmax. It's not rocket science to implement matrix multiplication in any GPU. We don't have very complex custom kernel for the LLM inference. If AMD can provide fast GPU with cheaper price, a lot of people don't mind to add support for them. GGML shows you can create a working LLM inference solution with one man's work.
Everyone has been writing cuda code. Hopefully we continue the shift towards using higher level frameworks/abstractions that can determine the optimum machine code for the underlying hardware. I don’t really get it either.
Nvidia is pouring $$$ into protecting their advantage as anemic as it might be. They give free compute and gpu resources to universities etc to cement the dependence on cuda. When you’re a beginner or someone who needs to just “get shit done” you’ll go nvidia at the expense of the greater good.
> They give free compute and gpu resources to universities etc to cement the dependence on cuda.
It's actually quite brilliant. It never occurred to me before that the resources we had access to at university weren't the most popular because that's what the library wanted. They were the most popular because the provider made them cheap/free to widen their moat over the competition.
It reminds me of Thomson Reuters launching Eikon, a superior alternative (imho) to Bloomberg terminals. Every business school has Bloomberg terminals. But few/none had Eikon, which is why nobody knows how to use one.
What the hell is the greater good here? I'm a researcher in a R1 university, I can quickly try stupid things on my personal 3080, then if it works I can virtually git clone the codes onto my lab's server and bam it chops through the problems. After a few weeks I'll get (hopefully) some publisable results that (hopefully) advance my field a little. Do you really expect most researchers to use half of their time debugging the unfinished ROCM stack that is just not there, sacrificing their output, and maybe even their tenure prospect? We love Nvidia solutions, at least in our field, not because it's a monopoly, but because it's actually the best solution right now.
I really like that you can prototype on your own consumer level card - the rocm stack has not matched that historically and has over promised and underdelivered for years now.
>Nvidia is pouring $$$ into protecting their advantage as anemic as it might be.
This description isn't fair. Nvidia was pouring money into CUDA for 16 years. AMD/Intel only woke up once ChatGPT launched and nvidia started printing money. It's not fair to call it "protecting their advantage" when nvidia invested into the ecosystem for years and AMD couldn't even bother to ship working examples with OpenCL. I'm not sure why choosing the company that decided to invest in the platform for years is at the expense of the "greater good." What "greater good" is there above actually working software?
Nvidia had boatloads of money to spend, and 6X the number of engineers. AMD was skating on the edge of bankruptcy. AMD bet the farm on Zen, and that bet paid off so now they have money to spend on other things.
If they would have split resources off of Zen to spend on CUDA they would have failed at both and AMD would be a bankrupt husk.
Inference-only hardware like this could be a temporary cost saving solution to scale up AI infrastructure, but I think a chip the size of this should be able to support training, otherwise it’s just a waste of money and energy. I predict inference will move to edge computing based on mobile chips like the ones from Qualcomm in the midterm.
This. When I was still doing infra for a large recommendation engine, the training was what got the big GPUs. Inference was on CPU or (later) ASICs (may have been FPGAs?).
> It's not rocket science to implement matrix multiplication in any GPU.
You're right, it's harder. Saying this as someone who's done more work on the former than the latter. (I have, with a team, built a rocket engine. And not your school or backyard project size, but nozzle bigger than your face kind. I've also written CUDA kernels and boy is there a big learning curve to the latter that you gotta fundamentally rethink how you view a problem. It's unquestionable why CUDA devs are paid so much. Really it's only questionable why they aren't paid more)
I know it is easy to think this problem is easy, it really looks that way. But there's an incredible amount of optimization that goes into all of this and that's what's really hard. You aren't going to get away with just N for loops for a tensor rank N. You got to chop the data up, be intelligent about it, manage memory, how you load memory, handle many data types, take into consideration different results for different FMA operations, and a whole lot more. There's a whole lot of non-obvious things that result in high optimization (maybe obvious __after__ the fact, but that's not truthfully "obvious"). The thing is, the space is so well researched and implemented that you can't get away with naive implementations, you have to be on the bleeding edge.
Then you have to do that and make it reasonably usable for the programmer too, abstracting away all of that. Cuda also has a huge head start and momentum is not a force to be reckoned with (pun intended).
Look at TensorRT[0]. The software isn't even complete and it still isn't going to cover all neural networks on all GPUs. I've had stuff work on a V100 and H100 but not an A100, then later get fixed. They even have the "Apple Advantage" in that they have control of the hardware. I'm not certain AMD will have the same advantage. We talk a lot about the difficulties of being first mover, but I think we can also recognize that momentum is an advantage of being first mover. And it isn't one to scoff at.
This comment is not going to age well. In the interest of curiosity, in this indictment of yours, is there any specific reason, about the AMD hardware itself, that leads you to believe that such implementations don't already exist, but are merely just not open sourced?
I have a specific reason to believe this is all already optimized: poor adoption of Infinity Fabric by unsophisticated people, whereas NVLink is on every 3090 and DGX/HGX box. But that doesn't mean DC users haven't optimized.
1) the user above me thinks ML is just matrix multiplies, which it is far from that.
2) It's more than hardware, and one team has (huge) momentum
3) You can have better tech and not win. It actually happens relatively frequently.
There may be nothing better for the ML space than AMD being a meaningful competitor. It will give all of us better tools. It'll make AMD better and it'll even make Nvidia better, because (healthy) competition is good for everyone. But let's not be naive in thinking it is just hardware and that one example is enough. It's a good sign, but a Tesla driving on the highway in nominal conditions isn't even close to fully self driving. It's still a year away as it's been for a decade.
So I do hope this ages poorly. But I'm not holding my breath because as AMD succeeding is one of the best things for ML, hype is the absolute worst and there's way too fucking much of it. And it it takes 5+ years to age, well I wouldn't say it aged poorly.
In an area so performance dominated people will spend the time porting to get the performance. I think this is especially true on the training side, because that's the area where companies have engineers waiting for hardware.
Unfortunately AMD only claim parity with Nvidia for training right now, and uplifts on inference. That's the wrong way round to get people to spend time porting as a priority.
I don't think that comment was an indictment of anything. As someone also in the space, it reads to me as a thoughtful explanation of why catching up to the CUDA ecosystem is so hard, even when the goal is "just" matrix multiplication (it's actually not, but for the sake of a cleaner argument that's ok). It's entirely possible AMD or Intel or someone else has some super secret CUDA killer hidden away somewhere, but nobody has seen any evidence of it. It's a bit of an extraordinary claim, and the community would love to see some extraordinary evidence.
None of your comment writes about why CUDA is so special compared to OpenCL or Metal.
Anybody who has worked with them knows that it's basically the same.
Optimizing kernels for all architectures and all models is a hard problem as there are lot of cases to handle, but getting good utilization for a particular model on a particular architecture (like MI300 / Mixtral) is not that hard.
You’re right. The hard part is to manage processing of your data as close to the metal as possible to optimize memory, data flow, and computation.
Coming from FPGA (both Verilog and VHDL), to me CUDA offers the best way to handle all three parts in comparison to let‘s say OpenCL, Vulkan, or Metal.
For game developers this might be different, as here you always use high-level language and packages (in most cases).
AMD need‘s to rethink there strategy regarding compute tool stack radically. If the HelloWorldMatrixMul is longer as ten lines and not compile on every new AMD powered notebook out of the box, they have no chance to beat NVIDIA. (I know, that the last point is not given for NVIDIA. But it would give AMD the doubleplus good, currently missing.)
> For game developers this might be different, as here you always use high-level language and packages (in most cases).
... I'm not sure what you mean that you always use high-level language and packages... But it doesn't sound right.
The reason game developers use DX/VK/GL as opposed to cuda is entirely:
A. You want access to non-nvidia customers
B. You need access to more aspects of the hardware then is exposed to Cuda. Cuda provides a nice interface to Nvidia's compute. Almost no access to the entire rest of the fixed function pipeline.
I was referencing DX/VK/GL and C++. What I can see on the lowest compute level, ML libs often are using pure CUDA in C or even PTX and are used especially if the code is generated by let's say via Python.
> I don't quite understand. Most of LLM inference is MatMul and Softmax.
Most operations are memory bound: slow because memory access takes longer than computation itself. The problem is we have to shove the whole model, billions of weights, in the SRAM once for every token generated. That creates slowness.
You are telling me what is supposedly a multi trillion dollar business being pursued by the brightest minds at Google and others would rather join the queue for a hardware drop from NVIDIA than enjoy a substantial competitive advantage being able to run on AMD hardware? For something that supposedly breaks down into such simple steps everyone is also building hardware to do it?
Something doesn’t quite add up for this dependence on CUDA. Sure, I’m installing some dependency hellscape Python research project from GitHub I reasonably expect idiosyncratic hardware dependencies, but surely the companies in this space are beyond that on the maturity timeline.
> Something doesn’t quite add up for this dependence on CUDA. Sure, I’m installing some dependency hellscape Python research project from GitHub I reasonably expect idiosyncratic hardware dependencies, but surely the companies in this space are beyond that on the maturity timeline.
I think part of the problem is something you're touching on here. The ML software space is horrifically put together. It's a bunch of research projects cobbled together. Hence the dependency hell, and very few people who actually know how to port it. Cuda siits at the bottom of the stack with its design assumptions baked into everything above it (see PyTorch for a good example of that). Nobody wrote a hardware agnostic layer over the top, and now the thought of pulling out a Jenga block at the bottom scares everyone.
Cuda is an intentionally leaky abstraction layer over the hardware, and its worked.
What about multi-gpu setups? Nvidia has nvlink, effectively combining any amount of GPUs into one speedy network, which can benefit both training and inference speed for bigger model sizes. AFAIK, AMD doesn't have anything like that, thus, limiting the usefullness of these chips.
Also, compute stack is too rough right now. Not sure ROCm will be comparable with CUDA in terms of usability in the next couple of years at least.
Could someone explain why ROCm is so behind Cuda and why wouldnt big tech and startups adopt MI300X instead of waiting in line begging Jensen for H100?
Nvidia has been working on CUDA for 16 years and for the past few years building ML use cases on CUDA. AMD is late in the game with ROCm. That said, I think the future is bright for AMD. The MI300X looks great and they're spending a lot on making ROCm better and fully PyTorch compatible, which will automatically make all of the PyTorch-based software AMD ready.
97 comments
[ 3.9 ms ] story [ 176 ms ] threadThat's before getting to any potential 8 vs 16 bit processing speed differences the two companies are arguing over here.
In any case, if the comparison was being done in 8 bit, the relative performance between the cards should be roughly similar as it is with 16 bit.
Its also important to distinguish 8 bit compute from the actual quantization scheme. vLLM in fact supports 4 bit AWQ, but the computation is not done in 4 bits on the GPU, and it wouldn't even be close to pure 4 bit inference even if the hardware and vLLM supported that.
Tensorflow had first mover advantage but they also had first mover disadvantage. TF isn't that easy to write nor to debug. PyTorch got to see everything that was wrong with TF and fix it. No doubt Meta has put in lots of amazing effort. It is an incredible piece of software.
I'll also add that PyTorch does a lot of great stuff that people don't use or recognize. There are a lot of good distributional works out there for prob and stats people that give you cuda acceleration. Really a lot of numpy can be replaced with pytorch and its great. But be warned of FMA and other optimization differences.
Not true. PyTorch has had an MPS backend[1] for Apple Silicon for quite some time now. And lately, they've also added a HIP backend[2] for AMD hardware. I've written CUDA kernels for PyTorch in the past, it'd have been a lot easier if PyTorch were a thin wrapper over CUDA, but it isn't.
[1]: https://pytorch.org/docs/stable/notes/mps.html
[2]: https://pytorch.org/docs/stable/notes/hip.html
==> You can also say it's a lot of other things like ROCm. <==
I mean my intent was just to communicate that PyTorch runs CUDA in the backend, which was what I meant by API, because it seemed the person I was responding to wasn't aware of this backend code you're interfacing with. I thought adding the same phrasing about it also "being" ROCm and using "such as" and "other things," that others would have allowed anyone with more knowledge to infer that I'm aware of the generalization but I guess I was deeply mistaken given the replies that are certain that I can't read the pytorch homepage. Though I'm not quite sure how you came to the conclusion that I wasn't aware of ROCm support since I did explicitly mention it.
Do you have feedback for how I can better communicate? I seem to be running into failures like this a lot lately where I feel like I can point to where I clearly stated something but that doesn't matter because communication is about getting the person on the other side to receive the message. I've been scratching my head about this and could definitely use the outside perspective.
Obviously you can use cuda to write more than ML algos, but that’s its overwhelming use case.
Thus my point is if AMD ported PyTorch to their hardware (and PT is not cuda-only) it would create a more level playing field on which they could compete (in a cousin vs cousin battle) by eliminating the switching cost for the majority of developers.
This is the same reason cpu mfrs have compiler ports but generally don’t have to worry about instruction set level compatibility.
It is ported (I think by AMD). I use it daily. Biggest problem is that the people maintaining PyTorch aren't very interested as they don't use it themselves. Hence there are silly bugs in it. Users contribute fixes and they get closed down as "unsupported" or get plain misunderstood.
Wait, is that actually true? The 4090 is superior in raster to the 7900XTX I thought? Happy to be corrected!
ultimately dlss/frame gen is a hack that no one on a high end card would ever use.
ray tracing nvidia is king but they got the jump. that will normalize over the next generation or two.
The irony of those technologies is that they shine best on high end cards.
If you're on a low end card and say pushing the limit of a game and are upscaling 720p to 1080p, then dlss can't do miracles, there will be tons of defects due how little information about the scene the upscaler has. But if you're on a high end system trying to upscale 1440p to 4k then even not only is the upscaler less likely to make a mistake, it's also much less noticeable.
Same with framegen, on a low end system that's getting sub 60fps, the framegen not only adds noticeable input lag, but often makes noticeable defective frames in between, if your system was already pushing above 60 then not only is the input lag much more bearable, the possible defective frames stay on screen for so little that you'd have to be paying attention to it to notice them most of the time.
They're still useful for low end systems, they can prolong their lifespan quite a bit for current gen games, but as this tech becomes the norm, developers rely on them to hit performance targets and your card will be outdated just as fast as older cards used to.
You may not be running DLSS with a super high scaling factor but I almost guarantee DLSS or DLAA is going to be on for most games nowadays if you want to keep framerates stable on ultra.
You say this as if momentum isn't one of the most powerful forces in the universe. Many a better technology has gone extinct because a competitor simply had more momentum. It's difficult to catch up and it's difficult to stop.
Nvidia also is aware of this and not making mistakes that many others do by sitting back and relaxing. They are doing everything they can to not let AMD even begin to compete, but they are also very aware that hardware is far from the full package. 75% max, but probably less.
For me DLSS looks worse than native. It looks all blurry. Even DLAA looks blurrier than native to me.
Alan Wake 2 will use FSR or DLSS for AA even at native resolutions, with no other AA option. I expect more games will do this in the future.
This is completely false.
Why?
Because Nvidia's DX12/Vulkan-style path in the driver is absolute trash, while their DX11/OGL4-era path is very well tuned. Nvidia's "advantage" is entirely software, the emperor wears no clothes.
With games that Nvidia's money couldn't cripple games, the 7900XTX, the card that both costs less and uses less watts, beats the 4090 in 1080p, 1440p, and 4k targets.
Have you seen fully realtime path-traced Cyberpunk?
You know very little about emulation.
Gamers buy 4090 only because DLSS is significantly better than FSR.
Nvidia is pouring $$$ into protecting their advantage as anemic as it might be. They give free compute and gpu resources to universities etc to cement the dependence on cuda. When you’re a beginner or someone who needs to just “get shit done” you’ll go nvidia at the expense of the greater good.
AMD's ML / ROCm support for their consumer cards is still awful.
I expect support will eventually be extended to a generation or two prior GPUs, but nothing beyond that.
https://rocm.docs.amd.com/projects/radeon/en/latest/index.ht...
It's actually quite brilliant. It never occurred to me before that the resources we had access to at university weren't the most popular because that's what the library wanted. They were the most popular because the provider made them cheap/free to widen their moat over the competition.
It reminds me of Thomson Reuters launching Eikon, a superior alternative (imho) to Bloomberg terminals. Every business school has Bloomberg terminals. But few/none had Eikon, which is why nobody knows how to use one.
It's a pretty solid strategy, but naturally favors the current incumbent who has the $$$ to throw around.
There's huge momentum on the "user" side of the equation that just getting cheap hardware and passable performance means Nvidia losing dominance.
This description isn't fair. Nvidia was pouring money into CUDA for 16 years. AMD/Intel only woke up once ChatGPT launched and nvidia started printing money. It's not fair to call it "protecting their advantage" when nvidia invested into the ecosystem for years and AMD couldn't even bother to ship working examples with OpenCL. I'm not sure why choosing the company that decided to invest in the platform for years is at the expense of the "greater good." What "greater good" is there above actually working software?
If they would have split resources off of Zen to spend on CUDA they would have failed at both and AMD would be a bankrupt husk.
The thing AMD doesn't have is the software stack.
You're right, it's harder. Saying this as someone who's done more work on the former than the latter. (I have, with a team, built a rocket engine. And not your school or backyard project size, but nozzle bigger than your face kind. I've also written CUDA kernels and boy is there a big learning curve to the latter that you gotta fundamentally rethink how you view a problem. It's unquestionable why CUDA devs are paid so much. Really it's only questionable why they aren't paid more)
I know it is easy to think this problem is easy, it really looks that way. But there's an incredible amount of optimization that goes into all of this and that's what's really hard. You aren't going to get away with just N for loops for a tensor rank N. You got to chop the data up, be intelligent about it, manage memory, how you load memory, handle many data types, take into consideration different results for different FMA operations, and a whole lot more. There's a whole lot of non-obvious things that result in high optimization (maybe obvious __after__ the fact, but that's not truthfully "obvious"). The thing is, the space is so well researched and implemented that you can't get away with naive implementations, you have to be on the bleeding edge.
Then you have to do that and make it reasonably usable for the programmer too, abstracting away all of that. Cuda also has a huge head start and momentum is not a force to be reckoned with (pun intended).
Look at TensorRT[0]. The software isn't even complete and it still isn't going to cover all neural networks on all GPUs. I've had stuff work on a V100 and H100 but not an A100, then later get fixed. They even have the "Apple Advantage" in that they have control of the hardware. I'm not certain AMD will have the same advantage. We talk a lot about the difficulties of being first mover, but I think we can also recognize that momentum is an advantage of being first mover. And it isn't one to scoff at.
[0] https://github.com/NVIDIA/TensorRT
I have a specific reason to believe this is all already optimized: poor adoption of Infinity Fabric by unsophisticated people, whereas NVLink is on every 3090 and DGX/HGX box. But that doesn't mean DC users haven't optimized.
1) the user above me thinks ML is just matrix multiplies, which it is far from that.
2) It's more than hardware, and one team has (huge) momentum
3) You can have better tech and not win. It actually happens relatively frequently.
There may be nothing better for the ML space than AMD being a meaningful competitor. It will give all of us better tools. It'll make AMD better and it'll even make Nvidia better, because (healthy) competition is good for everyone. But let's not be naive in thinking it is just hardware and that one example is enough. It's a good sign, but a Tesla driving on the highway in nominal conditions isn't even close to fully self driving. It's still a year away as it's been for a decade.
So I do hope this ages poorly. But I'm not holding my breath because as AMD succeeding is one of the best things for ML, hype is the absolute worst and there's way too fucking much of it. And it it takes 5+ years to age, well I wouldn't say it aged poorly.
Unfortunately AMD only claim parity with Nvidia for training right now, and uplifts on inference. That's the wrong way round to get people to spend time porting as a priority.
Anybody who has worked with them knows that it's basically the same.
Optimizing kernels for all architectures and all models is a hard problem as there are lot of cases to handle, but getting good utilization for a particular model on a particular architecture (like MI300 / Mixtral) is not that hard.
OP and OP’s OP never frame it about that. But you can still get the answer you are looking for from their answers.
Coming from FPGA (both Verilog and VHDL), to me CUDA offers the best way to handle all three parts in comparison to let‘s say OpenCL, Vulkan, or Metal.
For game developers this might be different, as here you always use high-level language and packages (in most cases).
AMD need‘s to rethink there strategy regarding compute tool stack radically. If the HelloWorldMatrixMul is longer as ten lines and not compile on every new AMD powered notebook out of the box, they have no chance to beat NVIDIA. (I know, that the last point is not given for NVIDIA. But it would give AMD the doubleplus good, currently missing.)
... I'm not sure what you mean that you always use high-level language and packages... But it doesn't sound right.
The reason game developers use DX/VK/GL as opposed to cuda is entirely: A. You want access to non-nvidia customers B. You need access to more aspects of the hardware then is exposed to Cuda. Cuda provides a nice interface to Nvidia's compute. Almost no access to the entire rest of the fixed function pipeline.
Most operations are memory bound: slow because memory access takes longer than computation itself. The problem is we have to shove the whole model, billions of weights, in the SRAM once for every token generated. That creates slowness.
Something doesn’t quite add up for this dependence on CUDA. Sure, I’m installing some dependency hellscape Python research project from GitHub I reasonably expect idiosyncratic hardware dependencies, but surely the companies in this space are beyond that on the maturity timeline.
I think part of the problem is something you're touching on here. The ML software space is horrifically put together. It's a bunch of research projects cobbled together. Hence the dependency hell, and very few people who actually know how to port it. Cuda siits at the bottom of the stack with its design assumptions baked into everything above it (see PyTorch for a good example of that). Nobody wrote a hardware agnostic layer over the top, and now the thought of pulling out a Jenga block at the bottom scares everyone.
Cuda is an intentionally leaky abstraction layer over the hardware, and its worked.
https://news.ycombinator.com/item?id=38669440
If you are a LLM startup blocked on compute, send me a dm
https://www.linkedin.com/in/gregory-diamos-1a8b9083
https://twitter.com/gregorydiamos
Also, compute stack is too rough right now. Not sure ROCm will be comparable with CUDA in terms of usability in the next couple of years at least.