This is a good decision. SIMD.js never made sense to me. Writing vector code that outperforms scalar code requires not only fine-grained control over issues like alignment, but also requires targeting specific SIMD instruction sets. You must write your vector code with the emitted assembly in mind, because if you fall off the fast path and the compiler switches to scalar code or does a library call or something, you've lost any performance gain and then some.
A generic SIMD API on a high-level language like JavaScript is as bad as it gets. I looked at what it would take to port some of my vector code to SIMD.js and it wasn't remotely plausible.
Also saving 10% of the v8 code size is enormous.
The right path forwards is for JITs to emit SIMD code when possible, and for JS engines to provide OpenCL-like GPGPU-targeted APIs, following the trail that WebGL blazed with JS shaders.
This is a good decision. SIMD.js never made sense to me. Writing vector code that outperforms scalar code requires not only fine-grained control over issues like alignment, but also requires targeting specific SIMD instruction sets. You must write your vector code with the emitted assembly in mind, because if you fall off the fast path and the compiler switches to scalar code or does a library call or something, you've lost any performance gain and then some.
I think this highlights the absurdity of writing vector code in higher-level constructs rather than the direct ASM. Writing in ASM is not that hard, and the gains can be enormous. I've seen graphics operations sped up by 5x or more, just by writing hand-optimized vector ops in assembly.
This is one of Visual Studio's core strengths vs GCC. Writing an __asm { } block is much more pleasant than the syntax GCC forces on you. Unfortunately this inertia is one reason games tend to be windows-only, even in the post-Apple era.
In other words, phrases like "You must write your vector code with the emitted assembly in mind, because if you fall off the fast path and the compiler switches to scalar code" invokes a feeling of "What? Why does the compiler get any say in what the vector code looks like? Oh, right, everybody shies away from writing ASM by hand nowadays." Which is valid, but sometimes having ultra control is extremely nice. It just feels cool to get a 2x speedup by being persistent and clever. (Those speedups come with future technical debt, though, so there are good arguments against it.)
In my experience the compiler does a pretty poor job at this. If you really care about every cycle you want to hand code these... At least this was the case ~6 years ago, haven't looked at more recent compilers.
Yep, bingo. There's no substitute for understanding exactly what's being emitted in the innermost loops of intensive operations.
I wonder whether any of this matters much in GPU-land though. I haven't kept up with the game, so the young hotshots that programmed AlphaGo might be doing all of that without resorting to low-level-ASM-type sorcery. I wouldn't be surprised if massive parallelism makes these types of gains a rounding error.
"AlphaGo was powered by TPUs in the matches against Go world champion, Lee Sedol, enabling it to "think" much faster and look farther ahead between moves."
What are the common cases where you care about every cycle, though? In open source stuff, I've really only seen that in crypto code, where they measure using cycles per byte.
I've recently worked on optimizing a piece of integer-based AVX2 code which was running in a tight loop (dot product of 128-dimensional vectors with 8-bit components).
I inspected the assembly generated by MSVC17, and it did a really good job; no extra loads or stores or register spills. It was "as if" hand-written.
> In my experience the compiler does a pretty poor job at this.
A year ago I've worked with a guy optimizing ffmpeg's PRORES encoder; he also used AVX2 intrinsics. The result? 45% speed improvement (gcc) w/o any of the fuss of hand-coded assembly. [1] Sadly, the ffmpeg team suffers from the same prejudices as you, and won't accept intrinsics-based code, even though their "policy" is to accept all patches giving improvement of just a couple of percent points.
EDIT: Ironically, they'd most probably accept the output of gcc -S og forfeit future optimizations implemented in gcc and other compilers.
It is as if experienced programmers are forever stuck with their past experiences and won't ever look back.
Oh. He used 14 registers to manually unroll the inner loop. IME, on modern architectures, manual unrolling doesn't win you anything (benchmarked on, among other things, the above-mentioned dotproduct). I wouldn't be surprised if he got close to optimal code by just NOT manually unrolling. Fewer registers used, compiler less confused.
> It is as if experienced programmers are forever stuck with their past experiences and won't ever look back.
This is totally the case. IME, anything that makes them step back from how they would normally solve the problem and rethink other approaches is of benefit. It is good if you can frame that as an appeal to their ego, 'cos people get defensive quickly. Sometimes they just feel out of their depth with the new and instinctively push back.
Ha, now I'm closely working with a "junior" developer and it's awesome. I try to teach him stuff underway, but usually let him find solutions to problems on his own (within boundaries). Sometimes he digs out really nice stuff that I note down and eventually add to my toolbox. It's almost like having my own research team :)
Register allocation in general is np-complete[1][2]. Having polynomial-time coloring algorithm for SSA doesn't make the allocation optimal -- you still need to "lower" the SSA to proper machine code.
Their output is quite alright for straightforward algorithms (crypto, checksums, simpler data processing, nothing meaty like video codecs, don't know anything about that). Though it is of course a good idea to always check.
You are correct, but take a look at more recent compilers. They are much less embarrassing now --especially if you start using vectorcall. http://godbolt.org is really great for experimenting with SIMD with multiple compilers simultaneously.
Or even better, the clang/GCC vector extensions often let you write reasonably good fixed length SIMD code that’s largely portable between SSE, Neon, AltiVec, etc.
The compiler should vectorize functions automatically, without user intervention.
Writing assembly code to use SIMD instructions is a vote of no confidence on your compiler.
If your compiler of choice can't do automatic vectorization switch to one that can, or get people to implement that feature in your compiler of choice.
Well, the compilers don't do a good enough job, end of story (at least using C; not sure about OpenCL). To achieve top performance it is necessary to make sure everything in the computation is organized in the right way. And try out multiple strategies on the high level of the program, to see what achieves best performance at the low level. At that point, trying to obfuscate your SIMD logic in regular C so that the C compiler maybe figure out what you meant is just frustrating anyway.
What you are saying is basically "you don't need top performance, just deal with what the compiler gives you".
But sometimes you work on stuff where the whole point is to be best in class in performance (libsharp in my case).
If you think that what you are suggesting is possible, feel free to improve the compilers so that normal C code beats OpenBLAS. You will be famous. I will wait....
Writing inline assembly to get SIMD performance is likely to cost immense time of an architecture expert and doesn't scale.
So my point stands: If a compiler can't produce vectorised code, the compiler needs to be improved.
Spending time on improving the compiler is sustainably spent time. Spending time on programming SIMD in assembly by hand likely is not.
I believe you that the vectorization support at the moment is not good enough, and I completely understand that not everyone can spend time on improving the compiler.
However, it seems you know exactly what a compiler should be able to do and what feedback from the compiler, or annotations for the compiler, would be helpful for your use-cases where optimization somehow didn't figure out what to vectorize.
I was just pointing out that communicating with compiler people and getting them to improve the compiler is likely a better move than asking for better support for inline assembly.
The problem is not communication. To quote Paul Graham in a keynote from some years back: People have been waiting for "sufficiently smart compilers" to cut programmers out of the loop for 30 years. It is a neat idea in theory. In practice it is just very hard. You need to design the algorithm for the hardware, so it is sort of an AI problem. Programmers do fill a role usually in coming up with the best algorithms...
The point is simply that C doesn't have the concepts you want to program with. Vectorization only gets you so far, there are many other things you can do with AVX/SSE. It doesn't need to be ASM, it could be supersets of C instead (like CUDA).
Intrinsic functions for AVX is what I use and I don't see the problem with using those. It sort of is such a superset of C.
In practice a few numerical computations (BLAS, FFTs, etc) are reused a lot by many. It is worth it to write those few libraries in assembly (or at least intrinsics).
For the rest we just have to live with a small performance penalty.
I am just saying "if you want to go the extra mile to get high performance, you can beat compilers". In most scenarios of course it makes most sense to not bother.
However, I don't think it is useful to consider a compiler to be a replacement for programmers. Compilers are tools for programmers. The more time a compiler saves a programmer, the better it is.
I think there is a middle ground between writing inline assembly and fully automatic vectorization that would be less time intensive than manual vectorization and more predictable and available earlier than fully automatic vectorization. I wonder what would have to be done to find it and provide support for it in GCC/LLVM.
You also need to take into account that a huge chunk of the work in SIMD is the need to rearrange your data to be more amenable to the CPU. C++ compilers are very restricted in what they can do for you there.
>The problem is not communication. To quote Paul Graham in a keynote from some years back: People have been waiting for "sufficiently smart compilers" to cut programmers out of the loop for 30 years. It is a neat idea in theory. In practice it is just very hard.
I don't see how it's "very hard" in the general case. Compilers have been increasingly cutting programmers out of the loop for 30 years now.
It's the reason we program apps and even games in C++ and not in hand-rolled assembly as we did in the 80s, and why we can use crazily high level languages like JS and still get within an order of magnitude of C with a modern JIT.
__asm{} as described in the comment I am replying to is an MSVC extension which lets you do assembly sprinkled with a mix of C identifiers. GCC's equivalent feature (asm() or __asm___()) is quite different, with the list of inputs, outputs, clobbers etc.
> The right path forwards is for JITs to emit SIMD code when possible
"Just rely on autovectorization" is what people have wanted to do for over a decade. We have yet to see satisfactory results for C. Why would JS be any easier?
> and for JS engines to provide OpenCL-like GPGPU-targeted APIs, following the trail that WebGL blazed with JS shaders.
Please don't! Nobody wants more APIs that rely on passing a string for a driver blob to compile. Standardize an IR (Web Assembly makes the most sense here).
> We have yet to see satisfactory results for C...
That's not a fair assessment. Major compilers (LLVM, GCC, Intel) are able to vectorize real world code to a useful extent. No, they won't reorganize data to be more SIMD friendly but they can take the hard work out of writing ISA specific assemmbly.
I think the parent with "satisfactory" meant "possible to achieve the maximum performance the hardware will support" (after all why settle for less?) You are answering another question.
Also, the point where you have reorganized your whole computation to be AVX-friendly it is just extra obfuscation to try to wrtie regular C and then check that the compiler outputs what you meant. Just using SIMD instructions is WYSIWYG; writing pure C is like competing in an obfuscated C contest.
Keep in mind it is not only SIMD, one might use shuffles and permutations in registers etc to communicate between the SIMD "threads".
I think the SSE/AVX intrinsics are nice to work with; they do what they say, but GCC will also turn them into something else if not supported, and perhaps apply some extra optimizations.
Yeah I didn't mean that I would be over backwards to make it work, just that I happen to see great speed ups for my real world code with almost no effort.
For more complex stuff I'd be looking at ISPC or OpenCL before assembly since I can't afford to write ISA specific code.
Also, in my experience, maximum performance for most algorithms is not a clear cut line that I fall short of for lack of motivation. I see that maximum as a rather fuzzy unknown, approaching it is has diminishing returns on the time spent optimizing.
is contrived but the structure resembles code I maintain related to differential equation integration. Autovec provides a very nice speed up for very little effort.
> Major compilers (LLVM, GCC, Intel) are able to vectorize real world code to a useful extent.
Try compiling this with recent gcc:
uint8_t dumb_checksum(uint8_t *ar) {
uint8_t ans = 0;
for (int i = 0; i < 320; i++) ans += ar[i];
return ans;
}
You don't get satisfactory results. I, at least, don't have a clear mental model of when gcc will do something reasonable and when it will generate a large amount of bad code.
Someone notes that v8 is reduced by 500KB on Android.
Looks like it got reverted a few times for breaking Node.js tests, someone notes "can you wait a day until we sort out Node first?" then SIMD.js gets removed again the next day. So I assume they "sorted it out w/ Node" w/e that means?
Also, trying to learn how the macroassembler is laid out; seems pretty neat. Kinda baffled that s390 and ppc are supported. Was kinda for mips but I guess that's an Android supported platform...
A lot of the code that looks like it's being assembled by the macro assembler has double underscore followed by a space, followed by argument lists, which is curious looking at first glance.
This seemed shocking as I thought SIMD was still on track for inclusion in the next ECMAScript release as the proposal was on stage 3 the last time I checked (stage 4 means "it's in the spec", so stage 3 generally means "shipping ASAP"). I first thought Google had single-handedly decided to violate the spec.
So this isn't really "Google kills SIMD" but "Google kills implementation of dropped proposal".
It's still surprising to see proposals get canned this late into the process. The only other example of a late-stage retraction I can think of is Object.observe, which was originally expected to become part of the 2016 edition: https://esdiscuss.org/topic/an-update-on-object-observe
60 comments
[ 2.5 ms ] story [ 111 ms ] threadA generic SIMD API on a high-level language like JavaScript is as bad as it gets. I looked at what it would take to port some of my vector code to SIMD.js and it wasn't remotely plausible.
Also saving 10% of the v8 code size is enormous.
The right path forwards is for JITs to emit SIMD code when possible, and for JS engines to provide OpenCL-like GPGPU-targeted APIs, following the trail that WebGL blazed with JS shaders.
I think this highlights the absurdity of writing vector code in higher-level constructs rather than the direct ASM. Writing in ASM is not that hard, and the gains can be enormous. I've seen graphics operations sped up by 5x or more, just by writing hand-optimized vector ops in assembly.
This is one of Visual Studio's core strengths vs GCC. Writing an __asm { } block is much more pleasant than the syntax GCC forces on you. Unfortunately this inertia is one reason games tend to be windows-only, even in the post-Apple era.
In other words, phrases like "You must write your vector code with the emitted assembly in mind, because if you fall off the fast path and the compiler switches to scalar code" invokes a feeling of "What? Why does the compiler get any say in what the vector code looks like? Oh, right, everybody shies away from writing ASM by hand nowadays." Which is valid, but sometimes having ultra control is extremely nice. It just feels cool to get a 2x speedup by being persistent and clever. (Those speedups come with future technical debt, though, so there are good arguments against it.)
Or just use intrinsics and let the compiler arrange register allocation, scheduling, etc. for you.
I wonder whether any of this matters much in GPU-land though. I haven't kept up with the game, so the young hotshots that programmed AlphaGo might be doing all of that without resorting to low-level-ASM-type sorcery. I wouldn't be surprised if massive parallelism makes these types of gains a rounding error.
"AlphaGo was powered by TPUs in the matches against Go world champion, Lee Sedol, enabling it to "think" much faster and look farther ahead between moves."
I inspected the assembly generated by MSVC17, and it did a really good job; no extra loads or stores or register spills. It was "as if" hand-written.
> In my experience the compiler does a pretty poor job at this.
A year ago I've worked with a guy optimizing ffmpeg's PRORES encoder; he also used AVX2 intrinsics. The result? 45% speed improvement (gcc) w/o any of the fuss of hand-coded assembly. [1] Sadly, the ffmpeg team suffers from the same prejudices as you, and won't accept intrinsics-based code, even though their "policy" is to accept all patches giving improvement of just a couple of percent points.
EDIT: Ironically, they'd most probably accept the output of gcc -S og forfeit future optimizations implemented in gcc and other compilers.
It is as if experienced programmers are forever stuck with their past experiences and won't ever look back.
[1] https://ffmpeg.org/pipermail/ffmpeg-devel/2016-May/194575.ht...
See the old official pre-Matz gcc statements on the mess https://gcc.gnu.org/wiki/RegisterAllocation
This is totally the case. IME, anything that makes them step back from how they would normally solve the problem and rethink other approaches is of benefit. It is good if you can frame that as an appeal to their ego, 'cos people get defensive quickly. Sometimes they just feel out of their depth with the new and instinctively push back.
[1] http://www.cs.cmu.edu/afs/cs/academic/class/15745-s07/www/pa... [2] http://pharm.ece.wisc.edu/wddd/2006/papers/wddd_05.pdf
Writing assembly code to use SIMD instructions is a vote of no confidence on your compiler.
If your compiler of choice can't do automatic vectorization switch to one that can, or get people to implement that feature in your compiler of choice.
What you are saying is basically "you don't need top performance, just deal with what the compiler gives you".
But sometimes you work on stuff where the whole point is to be best in class in performance (libsharp in my case).
If you think that what you are suggesting is possible, feel free to improve the compilers so that normal C code beats OpenBLAS. You will be famous. I will wait....
So my point stands: If a compiler can't produce vectorised code, the compiler needs to be improved. Spending time on improving the compiler is sustainably spent time. Spending time on programming SIMD in assembly by hand likely is not.
I believe you that the vectorization support at the moment is not good enough, and I completely understand that not everyone can spend time on improving the compiler.
However, it seems you know exactly what a compiler should be able to do and what feedback from the compiler, or annotations for the compiler, would be helpful for your use-cases where optimization somehow didn't figure out what to vectorize.
I was just pointing out that communicating with compiler people and getting them to improve the compiler is likely a better move than asking for better support for inline assembly.
The point is simply that C doesn't have the concepts you want to program with. Vectorization only gets you so far, there are many other things you can do with AVX/SSE. It doesn't need to be ASM, it could be supersets of C instead (like CUDA).
Intrinsic functions for AVX is what I use and I don't see the problem with using those. It sort of is such a superset of C.
In practice a few numerical computations (BLAS, FFTs, etc) are reused a lot by many. It is worth it to write those few libraries in assembly (or at least intrinsics).
For the rest we just have to live with a small performance penalty.
I am just saying "if you want to go the extra mile to get high performance, you can beat compilers". In most scenarios of course it makes most sense to not bother.
However, I don't think it is useful to consider a compiler to be a replacement for programmers. Compilers are tools for programmers. The more time a compiler saves a programmer, the better it is.
I think there is a middle ground between writing inline assembly and fully automatic vectorization that would be less time intensive than manual vectorization and more predictable and available earlier than fully automatic vectorization. I wonder what would have to be done to find it and provide support for it in GCC/LLVM.
You also need to take into account that a huge chunk of the work in SIMD is the need to rearrange your data to be more amenable to the CPU. C++ compilers are very restricted in what they can do for you there.
I don't see how it's "very hard" in the general case. Compilers have been increasingly cutting programmers out of the loop for 30 years now.
It's the reason we program apps and even games in C++ and not in hand-rolled assembly as we did in the 80s, and why we can use crazily high level languages like JS and still get within an order of magnitude of C with a modern JIT.
Have you seen Halide? Might be a way to have your cake and eat it too.
http://halide-lang.org/
"Just rely on autovectorization" is what people have wanted to do for over a decade. We have yet to see satisfactory results for C. Why would JS be any easier?
> and for JS engines to provide OpenCL-like GPGPU-targeted APIs, following the trail that WebGL blazed with JS shaders.
Please don't! Nobody wants more APIs that rely on passing a string for a driver blob to compile. Standardize an IR (Web Assembly makes the most sense here).
That's not a fair assessment. Major compilers (LLVM, GCC, Intel) are able to vectorize real world code to a useful extent. No, they won't reorganize data to be more SIMD friendly but they can take the hard work out of writing ISA specific assemmbly.
Also, the point where you have reorganized your whole computation to be AVX-friendly it is just extra obfuscation to try to wrtie regular C and then check that the compiler outputs what you meant. Just using SIMD instructions is WYSIWYG; writing pure C is like competing in an obfuscated C contest.
Keep in mind it is not only SIMD, one might use shuffles and permutations in registers etc to communicate between the SIMD "threads".
I think the SSE/AVX intrinsics are nice to work with; they do what they say, but GCC will also turn them into something else if not supported, and perhaps apply some extra optimizations.
For more complex stuff I'd be looking at ISPC or OpenCL before assembly since I can't afford to write ISA specific code.
https://github.com/numba/llvmlite/issues/270
is contrived but the structure resembles code I maintain related to differential equation integration. Autovec provides a very nice speed up for very little effort.
Try compiling this with recent gcc:
You don't get satisfactory results. I, at least, don't have a clear mental model of when gcc will do something reasonable and when it will generate a large amount of bad code.That's because C is crap in that aspect and doesn't give enough guarantees and intents.
With a better language you could have great autovectorization in ever higher level code than C.
Supporting it in WebAssembly transparently is a much better way to go forward.
Dart VM SIMD API seems pretty nice to me: https://www.dartlang.org/articles/dart-vm/simd
You may not be able to squeeze out the same perf, but seems worth having to me.> JS engines to provide OpenCL-like GPGPU-targeted API
This is an interesting project, deep neural nets running on WebGPU: https://mil-tokyo.github.io/webdnn/
The bug links to the actual discussion ("The V8 binary has a considerable amount of code that could be trimmed"): https://bugs.chromium.org/p/v8/issues/detail?id=5948
There's a nice treemap of the size of pieces of v8 in the first comment on that bug.
Comment #3 on the bug notes that SIMD.js is large: https://bugs.chromium.org/p/v8/issues/detail?id=5948#c3
Finally, comment #6 "There's no reason to keep the simd.js stuff around. I'll take it out ASAP.": https://bugs.chromium.org/p/v8/issues/detail?id=5948#c6
Someone notes that v8 is reduced by 500KB on Android. Looks like it got reverted a few times for breaking Node.js tests, someone notes "can you wait a day until we sort out Node first?" then SIMD.js gets removed again the next day. So I assume they "sorted it out w/ Node" w/e that means?
Also, trying to learn how the macroassembler is laid out; seems pretty neat. Kinda baffled that s390 and ppc are supported. Was kinda for mips but I guess that's an Android supported platform...
A lot of the code that looks like it's being assembled by the macro assembler has double underscore followed by a space, followed by argument lists, which is curious looking at first glance.
Looks like the double underscores are defined as:
#define __ ACCESS_MASM(masm)
However it turns out the SIMD proposal has been dead since at least April: https://github.com/tc39/ecmascript_simd/commit/c6ca655dcbfc0...
So this isn't really "Google kills SIMD" but "Google kills implementation of dropped proposal".
It's still surprising to see proposals get canned this late into the process. The only other example of a late-stage retraction I can think of is Object.observe, which was originally expected to become part of the 2016 edition: https://esdiscuss.org/topic/an-update-on-object-observe