Man, when I see this stuff I sure hope there is maturation of auto-vectorization at the compiler level in clang etc.
Even more useful would be compiler-level feedback of how to stay within the constraints needed to auto-vectorize your C/C++ for loop. (I need to make this data access const etc)
As far as I know, the Intel compiler is ahead of MSVC/clang on this front without reverting to OpenMP or other annotations on your code.
The changes necessary to make efficient use of these instructions go well beyond just inner loops (e.g. struct of arrays vs. array of structs) that the compiler can't transparently perform for you.
> Even more useful would be compiler-level feedback of how to stay within the constraints needed to auto-vectorize your C/C++ for loop. (I need to make this data access const etc)
Specifically for clang you can use the new opt-viewer [1] to annotate the source to find hints on what could/couldn't be optimized/vectorized. Requires compiler flags only found in 4.x/trunk, unfortunately.
It was presented at the dev conf but I don't think the slides or video are up yet. :(
I've used it with CPython by adding `CFLAGS="-fsave-optimization-record -fdiagnostics-show-hotness"` to the env for `./configure`. Then run opt-viewer like so:
He's quite wrong.
In fact, he quotes me. I'm the one who said plenty of people have programs with flat profiles.
He simply asserts "this view is obsolete", with no data.
I pointed out to him that my data was based on the tens/hundreds of thousands of different programs running in google datacenters for many years.
The truth is, for larger customers who actually care about optimizing, profiles are pretty flat.
Because if they weren't, you'd hire people, switch languages, switch to gpus, make asics, whatever, until they were.
And if you didn't, you clearly don't care that much because it's not worth it, monetarily (IE not "important").
Which is fine, but it in fact, goes very against his point, because those people want optimizing compilers that do a great job without having to spend the money/time to mess with it by hand. He believes you hand these things to algorithm designers, but he misses that this just makes them flat again!
As far as i can tell, he's just handwaving.
I've talked with numerous colleagues elsewhere, and they see the same.
I have yet to hear of a lot of stories of incredibly important code that can't possibly be made faster, where people don't want compilers to do it.
Because it would just be dumb to have that code like that.
There may be code where, temporarily, it has to be like that while another solution becomes more cost effective, that happens for sure, but it's usually 2-3 year time periods.
Because otherwise, if my business could make more money/save time/etc by shoving that code on an FPGA, by convincing intel to add instructions, by funding compiler work, or whatever, not doing it is just dumb.
So they do it.
And then everything is flat again.
The only time people are doing those things by hand is for very short time periods. In the end, they want and need the optimizing compiler to catch up and be able to optimize it well for the architectures.
Even on GPU's, for example, people now depend heavily on optimizing compilers producing good code, even though, a few years ago, literally everything was done by hand.
Will there come a time where things can't be made faster, and we really do have hotspots again?
Maybe. We may change computing models so that doesn't happen :)
I'm not sure I completely agree and/or I misunderstand.
Compilers are still not used for many compute intensive applications that have been around for a long time. On both CPUs and GPUs the fastest gemm, convolution kernels, md force field calculations (and many more) are all hand written assembly and are at least twice as fast, sometimes much more, than compiled Fortran or C code.
"In the end, they want and need the optimizing compiler to catch up and be able to optimize it well for the architectures." This hasn't happened, as far as I can tell, for the compute intensive kernels I just mentioned. With every new architecture we write new assembly.
"Compilers are still not used for many compute intensive applications that have been around for a long time."
????
Citation needed :)
My experience is exactly the opposite.
"On both CPUs and GPUs the fastest gemm, convolution kernels, md force field calculations (and many more) are all hand written assembly and are at least twice as fast, sometimes much more, than compiled Fortran or C code."
Ditto. This is completely not correct, AFAIK :)
The fastest on GPU's is cublas, and even then, only in certain configs. gpucc (clang + cuda) and a normally compiled library we built for it already beats it for a lot of functions, and the only reason gpucc can't do better is because SASS is not open, so it generates PTX. If it was able to generate SASS, it'd win there too.
On CPU's, the pluto generated code also beats the hand generated code for these functions pretty handily, see the papers, or, heck, try it yourself.
This will happen for GPU's too sooner rather than later.
- fastest gpu conv libraries (cudnn/nervana neon - hand written assembly)
- Gromacs
gpucc is now basically in clang and I have not seen any gemm libraries written in CUDA C compiled by clang that are competitive. Or conv libraries.
I also looked at Pluto and again see no comparisons to the MKL suggesting it is competitive. It increases the speed of a triple nested for loop by a factor of 4, which is still nowhere close to the speed of the MKL gemm.
Intel is abandoning their compiler infrastructure and moving everything to Clang/LLVM. This does mean pushing their autovectorization work into LLVM, although judging from the quality of conversation in the vectorization BoF at the latest developer's meeting, it's not clear how much work they wish to put in to actually making acceptable upstreamable patches.
It sure would be nice if MKL could be integrated in clang. It's still the fastest LAPACK implementation. I use it on a relatively powerfull cluster (~11k cores) at university for doing quantum chemical calculations in Dalton, and having more research software as open source would benefit everyone in the end I believe.
I run into this often, it is amazing the speedup MKL provides for linear algebra heavy scientific computations and it is under this weirdly commercial, but permissively redistrubutable framework which means a lot of folks are using it unkowningly in a gray licensing area.
See the pre-build python Numpy/Scipy packages that use it and are often use by data science types:
Note that GCC at one point did pretty much everything icc did for vectorization (though this is circa 2008 and not sure if folks have kept to date on caring).
There isn't a lot of magic here, it's all just hard work and good cost models.
http://polaris.cs.uiuc.edu/~garzaran/doc/pact11.pdf suggests that gcc is only doing ~50% of what icc is doing for vectorization on TSVC, although the experimental speedups on PACT and MediaBench was fairly close (23% versus 28%).
MSVC provides that feedback. See https://msdn.microsoft.com/en-us/library/jj658585.aspx - the compiler option you want is /Qvec-report:2 which "Outputs an informational message for loops that are vectorized and for loops that are not vectorized, together with a reason code."
For effective use of such vector instructions, languages like C/C++ really fail at giving enough hints to their compiler. An advanced inspection tool in clang/gcc would certainly help humans to write compiler friendly code, but the real advance can only be taken with an improved programming language that designed specifically for such use. Perhaps the HN crowd is more knowledgable than me, but I fail to recognize any potentially useful language on the market to date. Perhaps Haskell or OCaml with compilers helped by advanced AI?
In the more mathematical sense the "vectors" are the matrices. Vector instructions are usually implemented as fixed-size SIMD operations that do some chunk-sized work on a problem. These new instructions would seemingly operate on an entire vector/matrix.
Historically vector processors had vector registers but scalar ALU execution units (although possibly more than one). Vector instructions were "just" a way to make sure that the ALU was fed a new operation every cycle without instruction fetch and loop overhead. It also made it easier to pipeline reading from main memory (main memory latency wasn't so high a that time, so the large vector operations made it possible to pipeline reads with processing without stalling the CPU). None of those issues has been a bottleneck for a while and the memory subsystem of a modern computer is significantly different, so classic vector processors have fallen out of favour.
In contrast more modern SIMD machines normally have the vector execution units as wide as the register themselves and the advantage, in addition to 1 N-vector ALU being more power and area efficient than N scalar ones, is that, in an OoO machine, less in-flight instructions need to be tracked. It is also easier to take advantage of wider memory/cache busses.
Because classic vectors machines processed elements one at a time, it was possible to have efficient accumulating operations, which is significantly harder on proper SIMD processors (so called horizontal operations).
Great. Will Intel still disable those features on lower-end chips, thus ensuring that the market share for such features to make sense for developers won't be reached anytime soon after release?
This is a KNL-only feature to workaround a microarchitectural limitation (max two instruction issued per clock) to speed up a few specific benchmarks^Wworkloads.
Funny: "It is not clear that any compiler will ever use this instruction — it looks like it is designed for Kazushige Goto‘s personal use." https://en.wikipedia.org/wiki/Kazushige_Goto
This sounds like exactly the instruction needed for the inner loop of xcorr_kernel(), the function at the heart of a bunch of algorithms used in the Opus codec. This falls under the "convolution kernel" use-case described in the article.
This instruction is surely targeted at deep learning applications. Convolutonal layers take up the majority of the compute time of deep networks.
People seem optimistic that compilers will auto generate such instructions, but even if a compiler could generate the instruction, you would need to carefully organize your data structures to take advantage of it.
Pipelining is as important as SIMD in achieving peak flops on current processors. You can do 16 flops in a single fmadd instruction, in five cycles. But ten consecutive independent fmadds also take 5 cycles, but perform 160 flops. Getting a pipeline going like that requires very careful design of data structures by the programmer.
Current publicly announced AVX512 does not support fp16. Skylake Server (SKX) and Knights Landing (KNL) are at a disadvantage here. They've not publicly said anything about extensions in Knights Hill (the long announced successor to KNL).
That said, Intel have announced the emergency "Knights Mill" processor jammed into the roundmap between KNL and Knights Hill. It's specifically targeted at deep learning workloads and one might expect FP16 support. They had a bullet point suggesting 'variable' precision too. I would guess that means Williamson style variable fixed point. (I also guess that the Nervena "flexpoint" is a trademarked variant of it).
I assume the FPGA inference card supports fp16. And Lake Crest (the first Nervena chip sampling next year) will support flex point of course. I would expect subsequent Xeon / Lake Crest successor integrations to do the same.
Fun times..
Aside on the compiler work -- I think it's not that hard to emit this instruction at least for GEMM style kernels where it's relatively obvious.
Yes a compiler can generate the instruction. But if it's alone in a for loop surrounded by random STL classes - which even if inlined - are bodging up the pipeline or (gasp) causing spurious random dram accesses, there's little performance gain. And that's what usually happens in c++ code that wasn't already designed for AVX ("it's using AVX, but it's not running any faster. i guess AVX doesn't make much difference").
Net-net, data and code need to be structured for AVX to achieve the potential performance gains, and that's 80% of the work.
Once you structure the data and code for AVX, yes you can use regular C statements, then experiment with optimization flags until the compiler generates the intended instructions (and hasn't introduced excessive register spills). But its hard to see how that's any easier than using the intrinsics.
The problem is less the spurious DRAM accesses etc, as awful as they would be. The compiler problem is really a mix of 1) understanding enough about fixed-bound unit-stride loops to nonoverlapping memory (or transforming access to such) and 2) data layouts that prevent that. E.g. while there are well understood data layouts at each point of the compilation pipeline, it's hard in general for compilers to profitably shift from array of structs to struct of array layouts.
You are correct that, generally speaking, most STL heavy code would be hard to vectorize and unlikely to gain much advantage. (Plus there are the valarray misadventures). You will sometimes see clang and gcc vectorize std::vector if the code is simple enough, and they can assume strict aliasing. Intel's compiler has historically been less aggressive about assuming strict aliasing.
Various proposals are working through the standard committee to add explicit support for SIMD programming. E.g. if something like http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n418... were to be standardized we could write matrix multiply explicitly as:
using SomeVec = Vector<T>
for (size_t i=0; i<n; ++i) {
for (size_t j=0; k<n; j+=SomeVec::size()) {
SomeVec c_ij = A[i][0] * SomeVec(&B[0],j, Aligned);
for (size_t k = 1; k < n; ++k) {
c_ij += A[i][k] * SomeVec(&N[k][j], Aligned);
}
c_ij.store(&C[i][j], Aligned);
}
}
For my own work on vector languages and compilers I've had an easier time of it since they have been designed to enable simpler SIMD code generation.
After 3 years of Xeon Phi I'm still waiting for OpenCL on Fortran with vector support so we can finally have a sane and performant programming model for these things. Instructions are neat, but the tooling support is just not there for widespread use IMO. If Intel had taken a more long term strategy oriented with OpenMP years ago, i.e. embracing accelerators of all kinds, instead of trying to hold on tight to it for market protection, I think they'd be in a better position now.
37 comments
[ 6.0 ms ] story [ 87.8 ms ] threadEven more useful would be compiler-level feedback of how to stay within the constraints needed to auto-vectorize your C/C++ for loop. (I need to make this data access const etc)
As far as I know, the Intel compiler is ahead of MSVC/clang on this front without reverting to OpenMP or other annotations on your code.
Specifically for clang you can use the new opt-viewer [1] to annotate the source to find hints on what could/couldn't be optimized/vectorized. Requires compiler flags only found in 4.x/trunk, unfortunately.
[1] https://github.com/llvm-mirror/llvm/tree/master/utils/opt-vi...
I've used it with CPython by adding `CFLAGS="-fsave-optimization-record -fdiagnostics-show-hotness"` to the env for `./configure`. Then run opt-viewer like so:
If that's not clear I can whip up a quick demonstration project.[1] https://cr.yp.to/talks/2015.04.16/slides-djb-20150416-a4.pdf
[2] https://news.ycombinator.com/item?id=9396950
[3] https://news.ycombinator.com/item?id=9202858
I pointed out to him that my data was based on the tens/hundreds of thousands of different programs running in google datacenters for many years.
The truth is, for larger customers who actually care about optimizing, profiles are pretty flat.
Because if they weren't, you'd hire people, switch languages, switch to gpus, make asics, whatever, until they were.
And if you didn't, you clearly don't care that much because it's not worth it, monetarily (IE not "important").
Which is fine, but it in fact, goes very against his point, because those people want optimizing compilers that do a great job without having to spend the money/time to mess with it by hand. He believes you hand these things to algorithm designers, but he misses that this just makes them flat again!
As far as i can tell, he's just handwaving. I've talked with numerous colleagues elsewhere, and they see the same.
I have yet to hear of a lot of stories of incredibly important code that can't possibly be made faster, where people don't want compilers to do it. Because it would just be dumb to have that code like that.
There may be code where, temporarily, it has to be like that while another solution becomes more cost effective, that happens for sure, but it's usually 2-3 year time periods. Because otherwise, if my business could make more money/save time/etc by shoving that code on an FPGA, by convincing intel to add instructions, by funding compiler work, or whatever, not doing it is just dumb.
So they do it. And then everything is flat again.
The only time people are doing those things by hand is for very short time periods. In the end, they want and need the optimizing compiler to catch up and be able to optimize it well for the architectures.
Even on GPU's, for example, people now depend heavily on optimizing compilers producing good code, even though, a few years ago, literally everything was done by hand.
Will there come a time where things can't be made faster, and we really do have hotspots again? Maybe. We may change computing models so that doesn't happen :)
Compilers are still not used for many compute intensive applications that have been around for a long time. On both CPUs and GPUs the fastest gemm, convolution kernels, md force field calculations (and many more) are all hand written assembly and are at least twice as fast, sometimes much more, than compiled Fortran or C code.
"In the end, they want and need the optimizing compiler to catch up and be able to optimize it well for the architectures." This hasn't happened, as far as I can tell, for the compute intensive kernels I just mentioned. With every new architecture we write new assembly.
???? Citation needed :) My experience is exactly the opposite.
"On both CPUs and GPUs the fastest gemm, convolution kernels, md force field calculations (and many more) are all hand written assembly and are at least twice as fast, sometimes much more, than compiled Fortran or C code."
Ditto. This is completely not correct, AFAIK :)
The fastest on GPU's is cublas, and even then, only in certain configs. gpucc (clang + cuda) and a normally compiled library we built for it already beats it for a lot of functions, and the only reason gpucc can't do better is because SASS is not open, so it generates PTX. If it was able to generate SASS, it'd win there too.
On CPU's, the pluto generated code also beats the hand generated code for these functions pretty handily, see the papers, or, heck, try it yourself.
This will happen for GPU's too sooner rather than later.
- MKL (hand written assembly for most routines)
- gotoBLAS (hand written assembly)
- fastest gpu gemm library https://github.com/openai/openai-gemm (hand written assembly)
- fastest gpu conv libraries (cudnn/nervana neon - hand written assembly)
- Gromacs
gpucc is now basically in clang and I have not seen any gemm libraries written in CUDA C compiled by clang that are competitive. Or conv libraries.
I also looked at Pluto and again see no comparisons to the MKL suggesting it is competitive. It increases the speed of a triple nested for loop by a factor of 4, which is still nowhere close to the speed of the MKL gemm.
I run into this often, it is amazing the speedup MKL provides for linear algebra heavy scientific computations and it is under this weirdly commercial, but permissively redistrubutable framework which means a lot of folks are using it unkowningly in a gray licensing area.
See the pre-build python Numpy/Scipy packages that use it and are often use by data science types:
http://www.lfd.uci.edu/~gohlke/pythonlibs/#numpy
There isn't a lot of magic here, it's all just hard work and good cost models.
Source?
I thought "vector" in the context of CPU instructions just meant more than one.
Is there a definition where vector implies consecutive?
In contrast more modern SIMD machines normally have the vector execution units as wide as the register themselves and the advantage, in addition to 1 N-vector ALU being more power and area efficient than N scalar ones, is that, in an OoO machine, less in-flight instructions need to be tracked. It is also easier to take advantage of wider memory/cache busses.
Because classic vectors machines processed elements one at a time, it was possible to have efficient accumulating operations, which is significantly harder on proper SIMD processors (so called horizontal operations).
2) I'm pretty sure this won't be in KNL. It's a proposed future extension, and I don't think there has been indication of where or when it might land.
From the article, it will supposedly end up on some furure knight variant (as mainstream Xeons have lesa of a need for the "hack").
Linear-time Matrix Transpose Algorithms Using Vector Register File With Diagonal Registers
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.13....
disclaimer: I'm the author.
People seem optimistic that compilers will auto generate such instructions, but even if a compiler could generate the instruction, you would need to carefully organize your data structures to take advantage of it.
Pipelining is as important as SIMD in achieving peak flops on current processors. You can do 16 flops in a single fmadd instruction, in five cycles. But ten consecutive independent fmadds also take 5 cycles, but perform 160 flops. Getting a pipeline going like that requires very careful design of data structures by the programmer.
Does anyone know if AVX512 will support fp16?
That said, Intel have announced the emergency "Knights Mill" processor jammed into the roundmap between KNL and Knights Hill. It's specifically targeted at deep learning workloads and one might expect FP16 support. They had a bullet point suggesting 'variable' precision too. I would guess that means Williamson style variable fixed point. (I also guess that the Nervena "flexpoint" is a trademarked variant of it).
I assume the FPGA inference card supports fp16. And Lake Crest (the first Nervena chip sampling next year) will support flex point of course. I would expect subsequent Xeon / Lake Crest successor integrations to do the same.
Fun times..
Aside on the compiler work -- I think it's not that hard to emit this instruction at least for GEMM style kernels where it's relatively obvious.
Net-net, data and code need to be structured for AVX to achieve the potential performance gains, and that's 80% of the work.
Once you structure the data and code for AVX, yes you can use regular C statements, then experiment with optimization flags until the compiler generates the intended instructions (and hasn't introduced excessive register spills). But its hard to see how that's any easier than using the intrinsics.
You are correct that, generally speaking, most STL heavy code would be hard to vectorize and unlikely to gain much advantage. (Plus there are the valarray misadventures). You will sometimes see clang and gcc vectorize std::vector if the code is simple enough, and they can assume strict aliasing. Intel's compiler has historically been less aggressive about assuming strict aliasing.
Various proposals are working through the standard committee to add explicit support for SIMD programming. E.g. if something like http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2014/n418... were to be standardized we could write matrix multiply explicitly as:
For my own work on vector languages and compilers I've had an easier time of it since they have been designed to enable simpler SIMD code generation.