For context, in other Python ML packages like scikit-learn running the Intel extension on a compatable CPU can result in a massive performance increase.
Although since TensorFlow models should be trained on a GPU unlike sklearn, that's less useful, and there are better tools for CPU inference. (e.g. SavedModels or ONNX)
You won't for some time as ARC is only supported on Linux kernel 6+. And honestly, their hardware is pretty bad for ML given you can pick up a used 3090 for $600. I just don't think there will be an ML push for ARC with current gen.
I think a recent Linux version requirement shouldn't hamper benchmarking if you're a somewhat technical user.
You can get the latest Linux pacakges for eg Ubuntu here: https://wiki.ubuntu.com/Kernel/MainlineBuilds - probably other distributions have similar easy ways since it's needed so often in bug reporting for users to tell if a hw support bug is in a distro specific kernel change or not.
I don’t know much about Intel’s toolchain: people say good stuff about TBB and things like that.
Historically doing inference on Intel gear was mostly about whether or not to target AVX2 or AVX512 when building Eigen or whatever. A few years ago the net win was AVX2 because the de-clock and re-clock just killed you.
What’s the game these days? Long term I doubt inference will be done on x86, but I think a lot of people still do it.
For the curious, Apple has an analogous Tensorflow Metal plugin to allow for Apple Silicon (and AMD GPU) acceleration using the same plugin architecture.
It is sorta the fault of tensorflow... Tensorflow should have had a test suite for plugins that verifies that all operations at least lead to correct results no matter which backend is in use.
Intel did a great thing for people interested in ML and numeric research by making their MKL library and compiler free and cross platform compatible. Even today on my AMD zen3 Ryzen machine intel's MKL linked numpy and pytorch are in some operations 10* (yes that is really ten times) faster in comparison with the next best alternative (openBlas etc). I was shocked to discover how much of a difference MKL makes for cpu workloads. This is mostly because it makes use of AVX2 cpu extensions which make certain matrix operations a lot faster.
Yes, this is true, but even back then (during the time they crippled it) it was possible to pretend to run on Intel hardware. I haven't done this, but I read it was possible.
That was my previous experience, but have you tried linking to AMD AOCL recently? I would not expect the performance gap between Intel MKL and AMD AOCL to still be as large as you describe on a Zen 3.
Since BLAS (and LAPACK) have an essentially well-defined ABI, anything dynamically ELF-linked doesn't need specific support. You can just LD_PRELOAD, say, OpenBLAS or blis+libflame (as in AOCL) to use them. You can also make a trivial libblas.so.3 shim to put on LD_LIBRARY_PATH. (Debian has a heavyweight approach to linear algebra alternatives, with separate builds of the whole library.)
See, e.g., https://fx.srht.site/posts/2019-10-20-library-aliases.html
Compiling eigen uses avx512 just fine. I don't think it is quite the discriminator it once was. However, I amp happy they made it open and it would be great of oneAPI became a real alternative to cuda for more tasks. I tried getting it to work with flux.jl a while back and ran into some difficulty.
That's simply implausible for any proper measurement of the large-dimension level-3 operations that normally matter (and probably others I haven't measured). OpenBLAS certainly uses AVX2 assembler on Zen if it's built and run correctly, but you can get >60% of MKL's AVX2 performance with a plain C micro-kernel. The only plausible reason for an order of magnitude difference on large GEMM would be multi-threaded versus serial.
I keep saying there's far too much mythology around MKL, disproved experimentally, and if you're using Zen I don't know why you wouldn't use AMD's version of BLIS. It doesn't even make sense to talk about MKL on Zen without the version, since the story keeps changing.
OpenBLAS and BLIS actually are free software and cross-platform. MKL is still proprietary, and certainly doesn't run on the POWER platform I support. Also note that it only got the small matrix performance relevant to tensorflow after libxsmm showed the way.
I love the hardware Nvidia makes, but for the good of our field and for progress, we need our ML stacks to work with all other vendors. CUDA is the wrong platform to build our future on.
34 comments
[ 2.1 ms ] story [ 16.0 ms ] threadAlthough since TensorFlow models should be trained on a GPU unlike sklearn, that's less useful, and there are better tools for CPU inference. (e.g. SavedModels or ONNX)
How do you know it's bad? Do you have benchmarks?
Just checked, used 3090s are going for $700-900 on eBay.
You can get the latest Linux pacakges for eg Ubuntu here: https://wiki.ubuntu.com/Kernel/MainlineBuilds - probably other distributions have similar easy ways since it's needed so often in bug reporting for users to tell if a hw support bug is in a distro specific kernel change or not.
Historically doing inference on Intel gear was mostly about whether or not to target AVX2 or AVX512 when building Eigen or whatever. A few years ago the net win was AVX2 because the de-clock and re-clock just killed you.
What’s the game these days? Long term I doubt inference will be done on x86, but I think a lot of people still do it.
https://developer.apple.com/metal/tensorflow-plugin/
For example, tf.sort only sorts up to 16 values and overwrites the rest with -0. Apparently not fixed for over one year: https://developer.apple.com/forums/thread/689299
Also, tf.random always returns the same random numbers: https://developer.apple.com/forums/thread/69705
Although I guess these bugs are not the fault of Tensorflow's plugin architecture but rather Apple's implementation.
reminds of this particular xkcd [1], How someone fucks up that bad is beyond me.
1. https://xkcd.com/221/
Intel's anti competitive behaviour follows them throughout their history.
I keep saying there's far too much mythology around MKL, disproved experimentally, and if you're using Zen I don't know why you wouldn't use AMD's version of BLIS. It doesn't even make sense to talk about MKL on Zen without the version, since the story keeps changing.
OpenBLAS and BLIS actually are free software and cross-platform. MKL is still proprietary, and certainly doesn't run on the POWER platform I support. Also note that it only got the small matrix performance relevant to tensorflow after libxsmm showed the way.