I’ve been working on a toy bespoke computer, and eliminating accidental data dependencies like this has been joyful. Kind of embarassing that the V extension suffers this right now. Current deployment of older incompatible V specs is already a problem.
This is kind of interesting criticism, because as far as I can tell Casey doesn't actually have experience designing high-performance chips, he just appears to work on high-performance code. Now, I'm no expert in hardware design either (I guess we kind of work in the same area, actually…?), so I can't actually provide much feedback on the actual concern brought up here, but I think it's kind of new to get this specific concern from him and not an actual chip vendor. "I think this will make slow processors and that makes me sad" is a weird place to be, generally speaking. I've heard of a dialogue that goes "I want to do x, can you make this fast in hardware" or "we added these features in the new revision, can you use them for anything?" but this kind of concern is surprising. Maybe the concern is that RISC-V is making mistakes "so elementary" that it's leaking into the world of software engineering?
RISC-V's core philosophy, with V masks and elsewhere, is to make the smallest and simplest possible implementation as simple and small as possible (in this case, re-using the vector register file for masks reduces the minimum register file needed by 3%.) Even if it's at the expense of bigger cores needing to have additional logic to fix up the performance implications, and adding possible pitfalls like having to know what triggers a CPU switching its interpretation of v0 between "mask" and "vector".
This certainly has its place; there are way more tiny ARM cores running various bits of firmware in a typical computer than big cores running OS/application code. But way more programmers write code that runs on those big cores, and to be rude, a lot of the fixups RISC-V expects of big cores feel exactly like the workarounds Intel needed to implement as CPUs became faster and more complicated, at least to me. Except Intel has the excuse that x86 dates back to the 70's so unlike RISC-V it wasn't designed with that expectation.
Well, even ARM was designing new features with performance pitfalls in high-performance CPUs into the 2000's... (NEON 32/64-bit register views overlapping with the high bits of larger registers, and permute instructions with two destructive outputs, both of which were fixed in arm64)
While it is true that you can only use masks from v0, and this requires moving masks into v0 after calling a vector instruction, those moves don't actually copy data from one register to another. Instead, they just "rename" registers.
So...
...generate mask into v2... v2, .... <- put mask here
mov v0, v2 <- move mask into v0
vadd ... <- vector instruction, always use v0
doesn't really put some bits into v2, then copy them to v0, and then call the vector instruction.
Instead, the mov v0, v2 just disappears due to a register rename (e.g. v2 gets renamed as v0 for vadd), and vadd picks the mask directly from the register that was previously called v2 but is now called v0.
Any CPU would implement register renaming before actually even thinking of adding vector registers. So it is fair to assume that every CPU that implemenst the RISC-V V extension, supports it.
>
Although this may seem like a significant drawback, it's actually not that alarming, at least not to me. While it's true that you must insert various "move mask to v0" instructions that wouldn't otherwise be there, it's important to remember that these will not really be actual computation instructions. Moves from one vector register to another will always be simple register renames handled by the front end of any high-performance chip, and I would consider it highly unlikely that you would change masks so frequently as to overburden the front end.
If you only write to the lower 32-bit of the v0 register, which could be 1024 bit wide, that claims that the hardware somehow has to allocate a 1024-bit wide register to back those up, and then makes some "locality" arguments.
The hardware can back up the 1024-bit register with a pool of 32-bit registers, and if you only wrote to the first 32-bits, and all others are zero, it can use a single 32-bit register to back it up, making this "as good" as the single mask register solution, which the author thinks is good.
Determining that you only wrote to the bottom 32 bits of the register being copied from is hard for hardware to see; and if the compiler can see, it has no way to tell the hardware.
Personnally, I do expect a few ISA compatibity breakages for _really_ important fixes. It will probably happen at the start of RISC-V "never-ending" life.
I'm hoping for an eventual RISC-VI where all lessons learned in RISC-V are used and the instruction encoding is completely redone from scratch. I doubt it will happen, but it's what I hope for some day.
I call this Risc-6: The RISC-V "Good Parts" edition, leaving behind the academic complications and unfortunate premature pessimizations, crystallizing out a single compilation target that is simplest to implement for maximum performance for mainstream implementations.
The expensive formal verification work could be inherited by fixed mapping.
yep, I was more talking about instruction semantics, not really the encoding itself.
And even with fixes since software should come as assembly source files ultimately with risc-v, we should be sort of very fine, even if the encoding does change. We are not talking about absurdely and grotesquely complex c++ compiler backends!
I would be curious to know which RISC-V V implementation the author is talking about.
> If you imagine how a physical CPU or GPU has to be constructed in order to do large multi-input operations (...) You can imagine these inputs as being in "lanes" that are arranged across the chip such that the inputs to each lane are stored near the lane.
This is not how a GPU register bank works at all.
GPU register file are SRAM banks and operand collector are used to handle register-read latency. And there is a big cross-bar between the register banks and the operands collectors.
And for CPU Vector unit and SIMD unit, I only know two implementations (the CVA6 ARA vector unit and an industrial closed source one) but neither of them do registers storage within/near the lane.
Author's assumption on microarchitecture seems questionable to me.
PS: The ARA RISC-V V implementation used a mask unit to handle mask. Which makes the mentioned problem irrelevant
Wait, I'm pretty sure operand collection logic & banking is there to keep the number of ports on the SRAM low, so basically you're arbitrating and buffering requests coming from high register count instructions (say 3 input fma) and potentially multiple pipelined SMT threads (not thread in the nvidia sense, thread in the "a whole wavefront/warp" sense).
However to me it seems that's completely orthogonal to the vector lanes : I don't see why two parallel lanes in a single thread (eg a 64-element GCN wavefront) would need cross-connected logic at the register file, since almost all instructions _do not_ read/write data from another lane.
There are a few cross-lane shuffles / reduce instruction but it seems to me that those would be handled in a dedicated execution unit. (they are not really the fast-path/common case)
> There are a few cross-lane shuffles / reduce instruction but it seems to me that those would be handled in a dedicated execution unit. (they are not really the fast-path/common case)
Yes, you essentially need a (kind of) crossbar for shuffle and value broadcast.
But as far as I know there is no unit dedicated to this on Nvidia GPU.
However, depending on the GPU microarchitecture, shuffle and broadcast may be implemented differently (e.g. through the load/store units).
Note that I said "crossbar" for simplicity and because there is little information available, I doubt that all the paths really exist
I'm not qualified to comment whether this is a significant issue or not.
What puzzles me though is that as RISC-V is an open source ISA there should presumably be discussion of pros and cons and documentation of the decision to use this approach rather than that used in AVX and SVE. I would expect that this would help to settle the point.
And yet several hours after Casey has raised this issue I can see no references either here or on Github.
Edit: There is a thoughtful and extensive response now on GitHub (but without links to the original discussions) - which includes:
> Having separate mask registers as some, but by no means all, other ISAs do was explicitly considered by the working group, and rejected. Some members were for it, but not sufficient to do it. The implications of the decision were, I believe, well understood.
> There are many places in the design of RISC-V intended to make simple implementations simple, and high performance implementations possible, as possibly a little extra complexity in the already-complex implementations that can best afford it.
There likely has been a discussion, but maybe not documentation of the trade offs and logic behind the decisions, which are hard to capture and usually verbose and meandering.
The assumption here is that this fellow has discovered a flaw. Maybe that's the case or maybe it's not, but at this point my money is on the standards group and not one guy and his six day old blog.
In general — not necessarily this specific issue — I’d take the lone person pointing out a flaw. Things designed by committee are always compromises at best. Sometimes that’s good, and even necessary! But it doesn’t tend to lead to clean, elegant design.
This could be both the right thing for RISC-V and a wart on the design for high-performance implementations.
As brucehoult says on Github, that is the response:
"There are many places in the design of RISC-V intended to make simple implementations simple, and high performance implementations possible, at possibly a little extra complexity in the already-complex implementations that can best afford it."
Seems to me if we can make high speed x86 this one overloaded register in RISC-V is not gone make or break high speed implementation in terms of complexity.
Yes, the issue raised definitely not prevents a high-performance implementation.
But it's still interesting to ask ourselves if this is not an unnecessary cost?
The concern I have here is that to deal with this question you need very good microarchitecture knowledge about vector-unit, and the author doesn't seem to have them, but he reaches a confident conclusion.
How does one reach a conclusion with so much confidence on a technical subject that one does not know?
Are you saying that only people with very good micro architecture knowledge are allowed to ask questions about the RISC-V ISA?
Whilst the presentation of this as a ‘problem’ is debatable as may be some of the reasoning it doesn’t seem to me that it’s an unreasonable question to ask.
32 comments
[ 4.4 ms ] story [ 75.1 ms ] threadJust scroll a bit down to read it: https://github.com/riscv/riscv-v-spec/issues/811#issuecommen...
I don’t know much about this, I’m a programmer, but the explanation makes sense to me.
This certainly has its place; there are way more tiny ARM cores running various bits of firmware in a typical computer than big cores running OS/application code. But way more programmers write code that runs on those big cores, and to be rude, a lot of the fixups RISC-V expects of big cores feel exactly like the workarounds Intel needed to implement as CPUs became faster and more complicated, at least to me. Except Intel has the excuse that x86 dates back to the 70's so unlike RISC-V it wasn't designed with that expectation.
Well, even ARM was designing new features with performance pitfalls in high-performance CPUs into the 2000's... (NEON 32/64-bit register views overlapping with the high bits of larger registers, and permute instructions with two destructive outputs, both of which were fixed in arm64)
While it is true that you can only use masks from v0, and this requires moving masks into v0 after calling a vector instruction, those moves don't actually copy data from one register to another. Instead, they just "rename" registers.
So...
doesn't really put some bits into v2, then copy them to v0, and then call the vector instruction.Instead, the mov v0, v2 just disappears due to a register rename (e.g. v2 gets renamed as v0 for vadd), and vadd picks the mask directly from the register that was previously called v2 but is now called v0.
Any CPU would implement register renaming before actually even thinking of adding vector registers. So it is fair to assume that every CPU that implemenst the RISC-V V extension, supports it.
> Although this may seem like a significant drawback, it's actually not that alarming, at least not to me. While it's true that you must insert various "move mask to v0" instructions that wouldn't otherwise be there, it's important to remember that these will not really be actual computation instructions. Moves from one vector register to another will always be simple register renames handled by the front end of any high-performance chip, and I would consider it highly unlikely that you would change masks so frequently as to overburden the front end.
This is not the point the author was making.
The hardware can back up the 1024-bit register with a pool of 32-bit registers, and if you only wrote to the first 32-bits, and all others are zero, it can use a single 32-bit register to back it up, making this "as good" as the single mask register solution, which the author thinks is good.
The expensive formal verification work could be inherited by fixed mapping.
Why? That makes no sense. Its like, lets do 100 of man years of work so maybe we can do a few % better on encoding if you will do better at all.
And even with fixes since software should come as assembly source files ultimately with risc-v, we should be sort of very fine, even if the encoding does change. We are not talking about absurdely and grotesquely complex c++ compiler backends!
> If you imagine how a physical CPU or GPU has to be constructed in order to do large multi-input operations (...) You can imagine these inputs as being in "lanes" that are arranged across the chip such that the inputs to each lane are stored near the lane.
This is not how a GPU register bank works at all. GPU register file are SRAM banks and operand collector are used to handle register-read latency. And there is a big cross-bar between the register banks and the operands collectors.
And for CPU Vector unit and SIMD unit, I only know two implementations (the CVA6 ARA vector unit and an industrial closed source one) but neither of them do registers storage within/near the lane.
Author's assumption on microarchitecture seems questionable to me.
PS: The ARA RISC-V V implementation used a mask unit to handle mask. Which makes the mentioned problem irrelevant
However to me it seems that's completely orthogonal to the vector lanes : I don't see why two parallel lanes in a single thread (eg a 64-element GCN wavefront) would need cross-connected logic at the register file, since almost all instructions _do not_ read/write data from another lane.
There are a few cross-lane shuffles / reduce instruction but it seems to me that those would be handled in a dedicated execution unit. (they are not really the fast-path/common case)
Yes, you essentially need a (kind of) crossbar for shuffle and value broadcast. But as far as I know there is no unit dedicated to this on Nvidia GPU. However, depending on the GPU microarchitecture, shuffle and broadcast may be implemented differently (e.g. through the load/store units).
Note that I said "crossbar" for simplicity and because there is little information available, I doubt that all the paths really exist
What puzzles me though is that as RISC-V is an open source ISA there should presumably be discussion of pros and cons and documentation of the decision to use this approach rather than that used in AVX and SVE. I would expect that this would help to settle the point.
And yet several hours after Casey has raised this issue I can see no references either here or on Github.
Edit: There is a thoughtful and extensive response now on GitHub (but without links to the original discussions) - which includes:
> Having separate mask registers as some, but by no means all, other ISAs do was explicitly considered by the working group, and rejected. Some members were for it, but not sufficient to do it. The implications of the decision were, I believe, well understood.
> There are many places in the design of RISC-V intended to make simple implementations simple, and high performance implementations possible, as possibly a little extra complexity in the already-complex implementations that can best afford it.
which seems clear.
The assumption here is that this fellow has discovered a flaw. Maybe that's the case or maybe it's not, but at this point my money is on the standards group and not one guy and his six day old blog.
This could be both the right thing for RISC-V and a wart on the design for high-performance implementations.
"There are many places in the design of RISC-V intended to make simple implementations simple, and high performance implementations possible, at possibly a little extra complexity in the already-complex implementations that can best afford it."
Seems to me if we can make high speed x86 this one overloaded register in RISC-V is not gone make or break high speed implementation in terms of complexity.
But it's still interesting to ask ourselves if this is not an unnecessary cost?
The concern I have here is that to deal with this question you need very good microarchitecture knowledge about vector-unit, and the author doesn't seem to have them, but he reaches a confident conclusion.
How does one reach a conclusion with so much confidence on a technical subject that one does not know?
Whilst the presentation of this as a ‘problem’ is debatable as may be some of the reasoning it doesn’t seem to me that it’s an unreasonable question to ask.
Its a good blog-post and its a point worth talking about.