When I optimize stuff, I just think of the SIMD instructions as a long sandwich toaster. You can have a normal toaster that makes one sandwich, or you can have a 4x toaster that makes 4 sandwiches as once. If you have a bunch of sandwiches to make, obviously you want to align your work so that you can do 4 at a time.
If you want to make 4 at a time though, you have to keep the thing fed. You need your ingredients in the cache, or you are just going to waste time finding them.
I'm just happy that finally, with the popularity of zen4 and 5 chips, AVX512 is around ~20% of the running hardware in the steam hardware survey. It's going to be a long while before it gets to a majority - Intel still isn't shipping its own instruction set in consumer CPUs - but its going the right direction.
Compared to the weird, lumpy lego set of avx1/2, avx512 is quite enjoyable to write with, and still has some fun instructions that deliver more than just twice the width.
Personal example: The double width byte shuffles (_mm512_permutex2var_epi8) that takes 128 bytes as input in two registers. I had a critical inner loop that uses a 256 byte lookup table; running an upper/lower double-shuffle and blending them essentially pops out 64 answers a cycle from the lookup table on zen5 (which has two shuffle units), which is pretty incredible, and on its own produced a global 4x speedup for the kernel as a whole.
Interesting example! I've been learning AVX512 by using it to optimize Huffman coding. I found _mm512_permutexvar_epi8 and used it to do byte-indexed lookups, but _mm512_permutex2var_epi8 means I can get by with 2 shuffles and 1 comparison instead of 4 shuffles and 3 comparisons. In the end, on my CPU (AMD 9950x), changing to _mm512_permutex2var_epi8 only sped up compression by ~2%.
Compared to Huff0[1] (used by Zstd), my AVX512 code is currently ~40% faster at both compression and decompression. This requires using 32 datastreams instead of 4 used by Huff0.
Compared to GPU programming the gains from SIMD are limited but it's a small-multiple boost and available pretty much everywhere. C# makes it easy to use through Vector classes. WASM SIMD still has a way to go but even with the current 128-bit you can see dramatic improvements in some buffer-processing cases (I did a little comparison demo here showing a 20x improvement in bitwise complement of a large buffer: https://www.jasonthorsness.com/2)
The WASM folks should just include an arbitrary-length vector compute extension. We should also explore automatically compiling WASM to GPU compute as appropriate, the hardware independence makes it a rather natural fit for that.
Quick reminder that a 20x boost is better than going from O(n) to O(log n) for up to a million items. And, that log n algorithms often are simply not possible for many problems.
The high arithmetic bandwidth on GPUs is of course SIMD based as well. They just tend to have a ISPC style compilation model that doesn't expose the SIMD lanes in the source code. (Whereas SIMD even after decades is very lightly utilized by compilers on the CPU side).
You mean the gains from SIMD on the CPU, because all GPU programs use even wider SIMD than the CPUs, with width of 1024 bits or 2048 bits, for even greater die area and power consumption savings in comparison with non-SIMD.
The width of the SIMD instructions is not visible when programming with NVIDIA CUDA or with the similar compilers for Intel CPUs (ispc or oneAPI with SYCL targeting CPUs or OpenMP with appropriate pragmas), but only because the compiler takes care of that.
I remember having their manual around. I vaguely remember that the Athlon series used to be faster with floating points than the then new Pentium-4s. For that matter even the Pentium-3s had better floating point performance than Pentium-4s.
4 lanes of SIMD (like in say SSE) is not necessarily 4x faster because of the memory access, sometimes it's better than that (and often it's less).
PSHUFB wins in case of unpredictable access patterns. Though I don't remember how much it typically wins.
PMOVMSKB can replace several conditionals (up to 16 in SSE2 for byte operands) with only one, winning in terms of branch prediction.
PMADDWD is in SSE2, and does 8 byte multiplies not 4.
SSE4.1 FP rounding that doesn't require changing the rounding mode, etc.
The weird string functions in SSE4.2. Non-temporal moves and prefetching in some cases.
The cool thing with SIMD is that it's a lot less stress for the CPU access prediction and branch prediction, not only ALU. So when you optimize it will help unrelated parts of your code to go faster.
No mention of branches, which is a complementary concept. If you unwind your loop, you can get part of the way to SIMD performance by keeping the CPU pipeline filled.
Related: Go is looking to add SIMD intrinsics, which should provide a more elegant way to use SIMD instructions from Go code: https://go.dev/issue/73787
Why does such an abbreviation still exist in 2025?
They have been in the CPUs for so long that I expected them to be inseparable to the degree that people wouldn't even remember they were a separate thing in the past.
The main problem with SIMD instructions is that regular code doesn't use them. Almost always someone need to write SIMD code manually to achieve good performance, which is rarely done and if so, only in some tight loops and niche cases. Like cryptography-related code in a browser may be SIMD-based, but other code uses almost no SIMD.
Modern compilers are able sometimes to vectorize regular code, but this is done only occasionally, since compilers can't often prove that read/write operations will access valid memory regions. So one still needs to write his code in such a way that compiler can vectorize it, but such approach isn't reliable and it's better to use SIMD instruction directly to be sure.
While the blog post clearly highlights the core benefit of SIMD.
>Yet this new family of instructions was able to describe a great deal more work per instruction – the key benefit of SIMD.
It doesn't really explain why it is a benefit. From a ALU perspective, each SIMD lane could in theory execute a different operation. So why would simultaneous instructions as in SIMD ever be a thing? You could just keep cranking up the ILP (instruction level parallelism) by adding more ALUs, naturally extending to MIMD. Each SIMD lane would be equivalent to a full CPU core, so why is everyone stopping short of unlocking even more performance?
Because instruction memory (SRAM) is incredibly expensive.
By using SIMD you are reducing the number of instructions needed to describe repeating calculations. If you work with 32-bit floats and 512-bit instructions representing 16 lanes, you've made a 16-fold reduction to the required quantity of instruction memory and subsequent fetches from that memory to express the same parallel computation compared to using ILP.
22 comments
[ 5.7 ms ] story [ 54.1 ms ] threadIf you want to make 4 at a time though, you have to keep the thing fed. You need your ingredients in the cache, or you are just going to waste time finding them.
Of course memory bandwidth should increase proportionally otherwise the cores might have no data to process.
Why do we even need SIMD instructions? - https://news.ycombinator.com/item?id=44850991 - Aug 2025 (8 comments)
Compared to the weird, lumpy lego set of avx1/2, avx512 is quite enjoyable to write with, and still has some fun instructions that deliver more than just twice the width.
Personal example: The double width byte shuffles (_mm512_permutex2var_epi8) that takes 128 bytes as input in two registers. I had a critical inner loop that uses a 256 byte lookup table; running an upper/lower double-shuffle and blending them essentially pops out 64 answers a cycle from the lookup table on zen5 (which has two shuffle units), which is pretty incredible, and on its own produced a global 4x speedup for the kernel as a whole.
Compared to Huff0[1] (used by Zstd), my AVX512 code is currently ~40% faster at both compression and decompression. This requires using 32 datastreams instead of 4 used by Huff0.
[1] https://github.com/Cyan4973/FiniteStateEntropy
There wasn't much appetite for any of it on Emscripten.
https://github.com/WebAssembly/wasi-libc/pulls?q=is%3Apr+opt...
Quick reminder that a 20x boost is better than going from O(n) to O(log n) for up to a million items. And, that log n algorithms often are simply not possible for many problems.
The width of the SIMD instructions is not visible when programming with NVIDIA CUDA or with the similar compilers for Intel CPUs (ispc or oneAPI with SYCL targeting CPUs or OpenMP with appropriate pragmas), but only because the compiler takes care of that.
They were notable for several reasons, although they are no longer included in modern silicon.
https://en.wikipedia.org/wiki/3DNow!
PSHUFB wins in case of unpredictable access patterns. Though I don't remember how much it typically wins.
PMOVMSKB can replace several conditionals (up to 16 in SSE2 for byte operands) with only one, winning in terms of branch prediction.
PMADDWD is in SSE2, and does 8 byte multiplies not 4. SSE4.1 FP rounding that doesn't require changing the rounding mode, etc. The weird string functions in SSE4.2. Non-temporal moves and prefetching in some cases.
The cool thing with SIMD is that it's a lot less stress for the CPU access prediction and branch prediction, not only ALU. So when you optimize it will help unrelated parts of your code to go faster.
They have been in the CPUs for so long that I expected them to be inseparable to the degree that people wouldn't even remember they were a separate thing in the past.
Modern compilers are able sometimes to vectorize regular code, but this is done only occasionally, since compilers can't often prove that read/write operations will access valid memory regions. So one still needs to write his code in such a way that compiler can vectorize it, but such approach isn't reliable and it's better to use SIMD instruction directly to be sure.
>Yet this new family of instructions was able to describe a great deal more work per instruction – the key benefit of SIMD.
It doesn't really explain why it is a benefit. From a ALU perspective, each SIMD lane could in theory execute a different operation. So why would simultaneous instructions as in SIMD ever be a thing? You could just keep cranking up the ILP (instruction level parallelism) by adding more ALUs, naturally extending to MIMD. Each SIMD lane would be equivalent to a full CPU core, so why is everyone stopping short of unlocking even more performance?
Because instruction memory (SRAM) is incredibly expensive.
By using SIMD you are reducing the number of instructions needed to describe repeating calculations. If you work with 32-bit floats and 512-bit instructions representing 16 lanes, you've made a 16-fold reduction to the required quantity of instruction memory and subsequent fetches from that memory to express the same parallel computation compared to using ILP.