Yep. My wife just started as a professor (humanities) and she entered on H1B visa last week, as green card takes years to obtain. I have been offered a teaching job at the same institution as a partner hire and they…
Execution with masking is pretty much how broaching works on GPUs. What’s more relevant however is that conditional statements add overhead on terms of additional instructions and execution state management. Eliminating…
Quick note: I looked at the bindless proposal linked from the blog post and their description of Metal is quite outdated. MTLArgumentEncoder has been deprecated for a while now, the layout is a transparent C struct that…
No.
M3 GPU uses a new instruction encoding, among other things. Also, it has a new memory partitioning scheme (aka. Dynamic Caching), which probably requires a bunch of changes to both the driver interface and the shader…
Are you talking about Vulkan or about geometry shaders? The later is simple: because geometry shaders are a badly designed feature that sucks on modern GPUs. Apple has designed Metal to only support things that are…
Apple not supporting Vulkan is a business decision. They wanted a lean and easy to learn API that they can quickly iterate upon, and they want you to optimize for their hardware. Vulkan does not cater to either of these…
Have you looked at the Swift error model? I really like their design. They use a dedicated try statement to mark call sites that can fail — note that try is not the same as try...catch — Swift has an additional block…
Is this really a new approach? On a cursory look this seems like implicit error propagation with checked exceptions. I am Also curious about authors presentation of the topic. To me, an important feature of error…
While I understand the argument, it would also be good to see some empirical evidence. So far all x86 built need more power to reach the same performance level as ARM. Of course, Apple is still the outlier.
> Yes, that my understanding, and that's why I claim it's different from "classical" SIMD I understand, yes, it makes sense. Of course, other architectures can make other optimizations, like selecting warps that are…
Most NPUs are not directly end-user programmable. The vendor usually provides a custom SDK that allows you to run models created with popular frameworks on their NPUs. Apple is a good example since they have been doing…
> Not sure what you mean by lockstep here. When an operand-collector entry is ready it dispatch it to execute as soon as possible (write arbitration aside) even if other operand-collector entries from the same warp are…
Modern GPUs are exposing the SIMD behind the SIMT model and heavily investing into SIMD features such as shuffles, votes, and reduces. This leads to an interesting programming model. One interesting challenge is that…
In an operand-collector architecture the threads are still executed in lockstep. I don't think this makes the basic architecture less "SIMD-y". Operand collectors are a smart way to avoid multi-ported register files,…
How would you envision that working at the hardware level? GPUs are massively parallel devises, they need to keep the scheduler and ALU logic as simple and compact as possible. SIMD is a natural way to implement this.…
You need to consider this in the context of the relevant task. Nvidia GPUs have extremely high peak performance for GEMM, but when working with LLMs, bandwidth (and RAM capacity) becomes the limiting factor. There is a…
I remember last year (?) Quancomm proposing an ISA extension that brings ARM-like addressing modes and paired stores to RISC-V, and the community reaction being very negative. Happy to hear that there are now…
I fully support the idea of open instruction sets. I am not as much sold on the idea of cookie-cutter one-size-fits-all instruction sets. RISC-V is very nice for teaching CPU basics, and it is a great fit for tiny cores…
Bugs notwithstanding (which I agree are a significant concern for Metal), I'd frankly much prefer to work with a well-designed, streamlined API like Metal instead of a needlesly verbose and complex Vulkan.
> Note that compact/compress are not actually the key enablers: also with AVX-512 we use table lookups for u64 keys, because this allows us to actually partition a vector and write it both to the left and write sides,…
> Perhaps, though on VQSort it was more like 50% the performance. I looked at the paper and my interpretation is that the performance delta between M1 (Neon) and the Xeon (AVX2) can be fully explained by the difference…
> I agree with you we do not only want "very wide SIMD", and it seems to me that 2x512-bit (Intel) or 4x256 (AMD) are actually a good middle ground. I'd already classify this as "very wide". And the story is far from…
If you do streaming-type operations on long arrays, yes. If your data sizes are small, however, four smaller units might be more flexible. As a naive example, let's take the popular SIMD acceleration of hash tables.…
Yes, OoO is expensive — after all, that is the cost of performance. Very wide SIMD is great for energy efficiency if that is what your compute patterns require (there is a good reason why GPUs are in-order very wide SMT…
Yep. My wife just started as a professor (humanities) and she entered on H1B visa last week, as green card takes years to obtain. I have been offered a teaching job at the same institution as a partner hire and they…
Execution with masking is pretty much how broaching works on GPUs. What’s more relevant however is that conditional statements add overhead on terms of additional instructions and execution state management. Eliminating…
Quick note: I looked at the bindless proposal linked from the blog post and their description of Metal is quite outdated. MTLArgumentEncoder has been deprecated for a while now, the layout is a transparent C struct that…
No.
M3 GPU uses a new instruction encoding, among other things. Also, it has a new memory partitioning scheme (aka. Dynamic Caching), which probably requires a bunch of changes to both the driver interface and the shader…
Are you talking about Vulkan or about geometry shaders? The later is simple: because geometry shaders are a badly designed feature that sucks on modern GPUs. Apple has designed Metal to only support things that are…
Apple not supporting Vulkan is a business decision. They wanted a lean and easy to learn API that they can quickly iterate upon, and they want you to optimize for their hardware. Vulkan does not cater to either of these…
Have you looked at the Swift error model? I really like their design. They use a dedicated try statement to mark call sites that can fail — note that try is not the same as try...catch — Swift has an additional block…
Is this really a new approach? On a cursory look this seems like implicit error propagation with checked exceptions. I am Also curious about authors presentation of the topic. To me, an important feature of error…
While I understand the argument, it would also be good to see some empirical evidence. So far all x86 built need more power to reach the same performance level as ARM. Of course, Apple is still the outlier.
> Yes, that my understanding, and that's why I claim it's different from "classical" SIMD I understand, yes, it makes sense. Of course, other architectures can make other optimizations, like selecting warps that are…
Most NPUs are not directly end-user programmable. The vendor usually provides a custom SDK that allows you to run models created with popular frameworks on their NPUs. Apple is a good example since they have been doing…
> Not sure what you mean by lockstep here. When an operand-collector entry is ready it dispatch it to execute as soon as possible (write arbitration aside) even if other operand-collector entries from the same warp are…
Modern GPUs are exposing the SIMD behind the SIMT model and heavily investing into SIMD features such as shuffles, votes, and reduces. This leads to an interesting programming model. One interesting challenge is that…
In an operand-collector architecture the threads are still executed in lockstep. I don't think this makes the basic architecture less "SIMD-y". Operand collectors are a smart way to avoid multi-ported register files,…
How would you envision that working at the hardware level? GPUs are massively parallel devises, they need to keep the scheduler and ALU logic as simple and compact as possible. SIMD is a natural way to implement this.…
You need to consider this in the context of the relevant task. Nvidia GPUs have extremely high peak performance for GEMM, but when working with LLMs, bandwidth (and RAM capacity) becomes the limiting factor. There is a…
I remember last year (?) Quancomm proposing an ISA extension that brings ARM-like addressing modes and paired stores to RISC-V, and the community reaction being very negative. Happy to hear that there are now…
I fully support the idea of open instruction sets. I am not as much sold on the idea of cookie-cutter one-size-fits-all instruction sets. RISC-V is very nice for teaching CPU basics, and it is a great fit for tiny cores…
Bugs notwithstanding (which I agree are a significant concern for Metal), I'd frankly much prefer to work with a well-designed, streamlined API like Metal instead of a needlesly verbose and complex Vulkan.
> Note that compact/compress are not actually the key enablers: also with AVX-512 we use table lookups for u64 keys, because this allows us to actually partition a vector and write it both to the left and write sides,…
> Perhaps, though on VQSort it was more like 50% the performance. I looked at the paper and my interpretation is that the performance delta between M1 (Neon) and the Xeon (AVX2) can be fully explained by the difference…
> I agree with you we do not only want "very wide SIMD", and it seems to me that 2x512-bit (Intel) or 4x256 (AMD) are actually a good middle ground. I'd already classify this as "very wide". And the story is far from…
If you do streaming-type operations on long arrays, yes. If your data sizes are small, however, four smaller units might be more flexible. As a naive example, let's take the popular SIMD acceleration of hash tables.…
Yes, OoO is expensive — after all, that is the cost of performance. Very wide SIMD is great for energy efficiency if that is what your compute patterns require (there is a good reason why GPUs are in-order very wide SMT…