Torvalds clearly does not understand the brutality of business. Intel does not care at all about customers it have at the moment. Because they have them by the balls. After many years of struggle only Apple (with its unlimited resource) was able to switch to ARM. So they don’t care.
They only care about long term market, which is HPC and ML workloads. Because Nvidia is destroying anybody in that market. Look at their stock.
I’ve got a news for Torvalds, it going to get bad for Intel.
If there were ever a time where Intel had its customers by the balls the least in the past twenty years, it's now. AMD is extremely competitive in the server and desktop space, ARM/Apple is becoming competitive in the laptop space, Intel has no product in the phone/tablet space, and they are noncompetitive in the HPC/DL accelerator space.
If there's anything that has typified Intel in the past decade it's a generalized failure to execute. There's been very little innovation in the core product since Haswell in 2013.
Yeah for anyone who was a computer enthusiast in the 90s and 00s the rise of ARM has been an astonishing development comparable to the ubiquity of Linux and decline of Windows.
It could only be sarcastic if he's referring to "Linux on the desktop." I don't have any numbers, but I wouldn't be surprised if he was correct about servers.
Correct. I am referring to everything except desktop computing. I can see how this would be confusing as I was just referencing desktop prossesor market patterns of the 90s/00s.
But I do think Windows's position in the desktop space is somewhat unsustainable at this point. I personally prefer Windows but I just don't see how a heterogenous user interface experience (with users on iOS and Android for mobile and Windows for desktop) will abide for long.
I think it's only a matter of time before mobile OSes make some serious inroads to desktop. Though I suppose Google has been salivating at that thought for neaely a decade.
If you look at the overall "computing devices with a reasonably high performance CPU" market, Linux has won the server market, and through Android, won the phone/tablet market. If you squint a bit, you can lump MacOS and iOS in this in that they have a Unix core and use a lot of the same software stack as Linux.
50 years ago, IBM owned the computing market with mainframes. They still dominate the mainframe market, that's just become a smaller and smaller portion of the overall computing market as time has gone on.
Linux is ubiquitous. It runs on embedded devices, cell phones, and tablets which surround us. Linux runs most of the computers which underpin websites and services we interact with daily, and it is the only serious choice for HPC. A few people even run it on their laptops.
Windows has declined. It is still used extensively, but Microsoft no longer dictates the browser market. Microsoft Office is commonplace, but has genuine marketshare competition in alternatives like Google Docs. Developers increasingly build "desktop" applications as web-based or multiplatform-first instead of Windows-exclusive, with the notable exception of AAA games. The Linux subsystem for Windows is a direct admission by Microsoft that developers want to work with Linux-based software for many tasks.
Exactly. A single well-time and well-executed acquisition could change the whole game. Of course we all know that executing acquisitions well is easier said than done, and I don't have a specific target in mind when I say this. It's more of an "in principle" speculation than a concrete suggestion at the moment.
Very unlikely. Just because they are no longer "without alternatives" in many use-cases (for business customers) doesn't mean they go bankrupt. They will just lose money and maybe shrink a bit.
Also thy are in many other areas then just CPU's including FPGA and hard drives.
To be pedantic, Intel does have phone and tablet processors. This doesn't invalidate your point however. I can't easily find a breakdown of mobile cpu market share but I am guessing Intel isn't doing very well.
They canceled the atom for phones product. Right around the time that Microsoft announced the Continuum feature of Windows Phone 10 that gave a desktop(ish) experience when docked, and would have been perfect with an x86 phone. :(
The amoral mechanisms of business do not obviate criticisms that are independent of their bottom line, and this simple fact will not be unmade by any amount of the constant and fallacious insistence that people who make such criticisms "don't understand business".
Not really, like you mentioned in some ML/HPC contexts GPU computing with commonly Nvidia GPU is a better solution in the desktop marked AMD's recent CPU's are viable alternatives and that also applies for many usages in the server marked. And on mobile, well Intel CPUs never did really compete. Same for game consoles. Oh and for low power embedded systems it's often ARM and for thinks which also need GPU (self driving cars) it's ARM+Nvidea.
I mean sure parts of server the server marked are build on things like Intels (not so) secure enclave and then for some gaming workloads Intel CPU's still beat AMD and for not supper expensive software dev. workstations Intel also still is the best choice as the current CPU+embedded GPU chips from AMD for desktop still use their old CPU.
Through on thing which shouldn't be forgotten is that they have their hands in a lot of other things including:
- FPGA, if you use certain ML algorithm a lot putting them onto a FPGA can be a massive power and time saver. So the Intel CPU + FPGA are quite interesting for such cases.
- Graphics, Intel is about to bring out a dedicated graphic card. Probably not interesting for high end usages but it might be a use-full solution for budged setups or GPU servers (It's often monetary better to have multiple smaller cards then one big cards, it's not uncommon for GPU server to have Nvidea 2080ties instead of Quadros etc.).
- Hard drives, Optan "SSD" are not very well marked but while they are not necessary the fastest SSD's they have good latencies which can make them preferable for some use-cases.
- ...
Another thing which can be beneficial for them in the coming 10 years is that they do not depend on TSCM.
One funny thing I noticed is that Intel can build a good part of a computer today: CPU + GPU + Motherboard + Network Card + Drives + FPGA + OS (Clear Linux), so add RAM, PSU, Cooling, Cables & Case and you have a Intel only desktop ;-)
For >= 256 bit operations you need to decide to commit to them or not. If you go all-in on them and use them well, you will get the same work in less time and less joules than without them. If they won't help, don't use them. But, popping in and using them for tiny bits would be the slow and hot worst of both worlds.
There's not much use for AVX-512 in the kernel. But, from the people who are using it I've heard good things. It's an enormous collection of new instructions. But, a lot of them came from the Larrabee team --which was probably the greatest case of high-perf software engineers having direct influence over the direction of a CPU. Otherwise, the history SSE and AVX has largely been software engineers requesting features and hardware engineers replying "Do not understand. How does that feature improve our SPECfp score? Rejected."
On the one hand AVX-512 is incredible, the performance gains from AVX-2 to AVX-512 are often more than the 2x you would hope for. On the other hand, when you can use AVX-512 you can often use a GPU instead, to greater benefit. So linus may be right, overall.
Yes, these applications do exist. The speedups mainly come from the more flexible instruction set that AVX-512 provides (so in many cases you can replace several instructions with just one), plus the increased register set (32 architectural SIMD registers versus 16 on previous generations).
If your algorithm isn't sensitive to either one of these, then yes, the speedups are often smaller, particularly if the limiting factor in performance is memory bandwidth and/or latency (AVX-512 doesn't help at all there), or if the speedups aren't enough to overcome clock throttling (which IMO is a bit overblown; most people cite the Cloudflare blog article from a few years ago that complained about throttling on Xeon Silver processors, but throttling is much less of an issue on higher-TDP Gold/Platinum CPUs). However, I have seen specific applications that are >100% faster with AVX-512.
In my observations, GPUs are the way they are largely because Physics demands sacrifices if you want that kind of performance. Those same demands apply to CPUs, but Intel has been fighting against them because software engineers hate to change anything about how they code. Well... the fight was lost years ago and Intel is conceding to Physics by integrating architecture that was originally designed for a CPU-GPU hybrid.
With AVX-512 and CUDA, Intel and Nvidia are trying to find the best developer experience they can to deal with the inevitable future of computing that is somewhere between those two experiences currently.
> inevitable future of computing that is somewhere between those two experiences currently.
Not sure that ever happens. The two are very different in their performance characteristics. CPUs have to stay focused on single-threaded performance.
Single threaded performance = minimizing latency, that’s why in CPUs so many transistors and electricity is spent on cache hierarchy and coherency protocol, instructions reordering, ILP, microops fusion, prefetcher, branch prediction incl. indirect one, and so on.
GPUs don’t care about single-threaded performance, all they care is throughput on massively parallel tasks. They ignore memory latency, switching threads until data arrives, makes caches much simpler. They don’t need high clock frequencies (this alone improves their flops/watt by a large factor), don’t do branch prediction, memory model is less strict, and so on. That’s how they can spend that many transistors on FPUs and ALUs.
P.S. Intel tried many-slow-cores with Larrabee / knights landing / etc., all they got was a very expensive lesson, people didn’t want that stuff. I’ve heard a rumor initially Sony wanted to use another Cell CPU for the GPU in PS3, again, didn’t worked well enough and they negotiated with NVidia instead.
All true. Problem is that single-threaded latency minimization has been tapped out for a long time now. Physics says No. Memory is high-latency because of the speed of light. Larger caches have diminishing returns. Clocks can't go faster without melting the chips. There is no more prediction and pipelining to be dug out of a serial stream of instructions.
We can still add more transistors. But, there is nowhere for them to contribute but in going wider. High latency, high throughput. High concurrency and careful management of the memory hierarchy are the only ways to pull that off. When you accept that fate, you start out with the Cell processor and you pass through Nvidia's Ampere chip --which has somewhat more thread independence and better automatic caching than past GPUs, but also finally gained manual async DMAs to local SRAM like the Cell had. Where we'll end up, I don't know.
When I look at general-purpose single-threaded benchmarks, I see a good progress.
I’m pretty sure we’ll see very good progress on that, because Intel/AMD competition, also because inexpensive SSDs removed the storage bottleneck.
I agree on physical limitations, but the research still goes on, smart people still able to use these extra transistors in a way that’s useful for performance. Even for single-threaded one.
Another thing, software is progressively better at using many cores. It’s much easier to move large pieces of code to another CPU thread (like some browsers do with JS compiler) keeping it sequential code, than it is to re-write code as a large set of small tasks / threads. Many useful algorithms can’t use fine-grained parallelism much, or at all, e.g. streaming stuff like gzip or parsers.
I really miss the 286->Pentium days when you had to buy a new machine every year because it would be literally 50-100% faster at everything. Every year! These days people call a 9% single-thread improvement a good year. https://www.cpubenchmark.net/year-on-year.html
> All true. Problem is that single-threaded latency minimization has been tapped out for a long time now. Physics says No. Memory is high-latency because of the speed of light. Larger caches have diminishing returns. Clocks can't go faster without melting the chips. There is no more prediction and pipelining to be dug out of a serial stream of instructions.
Someone should probably tell AMD and Intel they need to stop, then, because Ice Lake was +18% IPC, Tiger Lake will likely be another 5-10%, and Zen3 will be around 15% higher IPC.
What has truly stalled IPC is 10nm. Intel has had higher-IPC architectures for a long time, they simply couldn't manufacture them. And AMD was coming so far from behind that it took them 3 generations of Zen to even catch up to Intel.
The last few years have been a rather unfortunate "CPU dark age" but things are finally breaking loose once again. Ice Lake launched 9 months ago, Intel backported Tiger Lake to 14nm and that will launch probably around Christmas or CES, Milan is launching probably within the next 6-8 weeks, and Tiger Lake is launching this month.
CUDA is night-and-day different from Intel's nightmare.
CUDA in particular hides the exact width of the processing elements from the programmer, so you don't have to rewrite your CUDA applications to take advantage of the next generation of GPU the way you have to do with AVX1,2,3,...
Mind you, NVIDIA's drivers compile your shaders to their proprietary instruction set, which is a path not taken in conventional CPU architecture even though it keeps coming up. (e.g. the belt)
FWIW, Intel supports OpenCL for their CPUs, so you can use similar general parallel programming code which is compiled down to AVX-512 (fairly) transparently.
> On the one hand AVX-512 is incredible, the performance gains from AVX-2 to AVX-512 are often more than the 2x you would hope for.
Your assertion sounds like a gross exaggeration. Some benchmarks show only a few percent improvements (i.e.14% improvements on synthetic benchmarks), and results range between no improvements and almost 2x in some synthetic benchmarks.
Those look like benchmarks comparing just autovectorized compilation of some tools and measuring overall runtime. I am referring to cases of hand tuned use SIMD. The bSIMD project used to have a suite of benchmarks up and some where very very impressive for AVX-512. The page appears to be gone or moved now though.
To be clear I am not suggest "avx-512 will cause programs to run twice as fast or more", just that specific functions that used to use AVX-2, may be more than twice as fast if you move them to AVX-512.
Some SIMD optimization cases can occasionally exceed scalar versions by more than lane width multiply. Often because of branch elision, but also less dependency chains stalls due to the larger register file. There are also some special instructions that can be very useful in SIMD case (packing, unpacking, permutation, etc.).
> Those look like benchmarks comparing just autovectorized compilation of some tools and measuring overall runtime. I am referring to cases of hand tuned use SIMD.
I presented data that showed that AVX-512 in reality only brings margins performance improvements.
If you have any data that shows otherwise then feel free to present it.
Still, hand-tuned assembly is hardly a relevant example, let alone selling point, as it creates hypothetical expectations that are never ever matched in reality.
"Still, hand-tuned assembly is hardly a relevant example, let alone selling point, as it creates hypothetical expectations that are never ever matched in reality."
There's a ton of hand tuned assembly running on your system in numerous libraries. Media and data compression? Hand tuned assembly. Matrix math? You guessed it. C-standard library functions? Yes, really.
Maybe for media, crypto, things like that, but we all have the source for the C-standard libaries, and there is not all that much assembly in there given the size of the code base.
Certainly you have required architecture instructions for things like instruction scheduling (isync/dsync), stack manipulation (setjump/longjump and others) or floating point, but though it looks like a lot of assembly source, it is really just a little bit for each of the many supported architectures in the tree. And most of it isn't performance related, it is just plain required because architectures vary on things like instruction scheduling and stack/cache manipulation.
In many cases within regulated environments getting access to GPU in e.g. a customer's on-prem solution is prohibitive.
In enterprise ML contexts on tabular data problems we found there are a lot of cases where even training can be greatly sped up by leveraging AVX instruction support in e.g. tensorflow builds. The gains from AVX instructions could boost training time by ~20% on the GAN use cases I profiled.
So you're saying that we need to solve organizational stupidity by complicating all chips for everyone so that bad organizations can get a performance boost on specialized tasks?
What percentage of consumers with these chips installed do you think are getting a performance win? Do you think it might be as high as 1%?
Do you think that the same resources devoted elsewhere might be worth more than 1% to that 99%? Whether that is in reduced cost, reduced bugs, or a boost for more widely used operations.
Yes, if you target a specific use case for a specific set of people, you can give them a nice win. But you shouldn't lose sight of the fact that CPUs cover a lot of use cases for a lot of people. And simplifying then focusing on the core mission is better for everyone in the end.
Linus somewhat waved away the idea of tradeoffs, which is fine, he was speaking in generalities.
Turning factual reports of where the tradeoff was helpful into a strawman insulting the reporter, and the users who benefit, is neither charitable nor illuminating.
I agree about GPUs. Intel used to have a low-end CPU with a 128MB L4 cache that was shared with the iGPU. It turns out having that large cache improved a lot of regular software too. I'd rather see die space used for that than AVX512.
The only AVX512 instructions that actually seem essential imo are the VNNI extensions.
There are a lot of use cases for a neural network that is trained on the GPU but where inference is done on a bunch of CPUs because it's cheaper than buying/renting inference hardware.
I think you are thinking AVX takes up more diespace than it does. Only a tiny fraction of all CPU diespace is dedicated to the actual CPU. The rest goes primarily to cache and partially to the various controllers on the CPU (memory controller, PCI-Express controller, etc).
There's a reason AMD is and Intel are able to cram full GPUs onto the same die as the CPU.
Yeah, here's a good image of how much space it costs to add a 'second' AVX-512 unit to certain SKUs. Cutting AVX-512 support entirely would save a bit more space, and the total would be enough to make a difference, but it's not an extreme amount.
Yes, in my opinion, the CPU model is dying as we fail to miniaturize we will be forced towards data parallel algorithms which benefit from speedups through SIMD.
Ultimately AVX will be the future of Intel if they want to compute with GPUs and if everyone stays silicon.
My view is that data parallel is here to stay and what's likely to happen is we'll see more power pumped into the instructions. Things like "If lane A > lane B add C to lane D"
Eventually, I see us going basically full EPIC. That is "here's a batch of instructions that can happen in parallel, run it".
I might suggest thinking along the lines that hardware engineers reject many features because of other constraints that impact the processor.
The failure of the larrabee team to bring something competitive to the market a decade ago doesn't necessarily imply it will become competitive just because it was brought "on-chip" instead of as an additional card or a co-processor.
If it worked all that well, it would be a product of its own, able to compete against Nvidia in GPU and GPGPU tasks.
An example of what we are talking about is _mm_dp_ps (4-component dot product). It was heavily requested by game devs for SSE2 but didn't show up until 4.1. And, even then it wasn't much better than hand-rolling it with old instructions.
The hardware engineers were "correct". It's a difficult instruction to make fast compared to completely independent mul-adds. Game devs "should just" re-architect their code to not deal with small situations individually and instead be more bulk-data-processing like SPECfp because that's what's friendly to the hardware. But, that's dismissing how the software actually worked --especially back then.
16 years later, many, but far from all of AAA game engines more closely resemble SPECfp. But, all along the way, and in tons of 3D graphics and gameplay code still today, it would have been nice to reliably have one of the most common and fundamental operations in games well-baked into the chips.
Any performance gain from fewer cycle cost may be lost from the CPU being slower. Maybe there are situations where AVX-512 is faster but how is one to reason about performance when some instructions can slow the CPU.
> but how is one to reason about performance when some instructions can slow the CPU.
The same way one should always do so on an advanced out-of-order CPU largely being programmed in modern high level languages: through thorough profiling and performance testing.
I appreciate the humor but that's more time consuming than comparing instruction cycle counts and data dependencies. It's also workload dependent when it wouldn't be if they didn't come up with this throttling thing even if out of order execution makes it complicated.
Most programmers are not regularly writing primarily assembly or reviewing the assembly generated by their compiler. These are the scenarios where instruction cycle counts become relevant - everything else is abstracted too far to rely on anything other than profiling and testing. If that's you, then I'm sorry for my response which didn't really take you into account.
When we got our processors last year for our new cluster (I work in an HPC center), Intel explicitly said that it's confidential information and cannot be shared.
I remember reading the datasheets myself and not being able to find it, together with my teammates.
So they changed their mind two months ago, and shared this info for their processors.
Guess that the information lost its significance.
Either way, a processor significantly throttles under AVX-512 load. Nice.
Edit: I re-read the document and found our SKUs under the "newly added" table and, clarified the comment.
It has been publicly available since years ago(I remember reading the specification explaining how it worked), it was never a surprise like Cloudflare made it out to be.
The base frequencies for first generation scalable was available however, not very publicly. You either needed to dig very deep in Intel's site or, find the blog post of some guy who nicely tabulated the data.
Our mostly used SKUs are released in 2019 so, its data was not available since years ago. Intel was very open about how their tech worked when it was new, needed to show their tech against the competition and not limited by thermal constraints.
When they became the dominant player, they didn't feel bad about hiding small but important details from their consumers.
Until EPYC came and pushed & prodded them around, they didn't bother to release both "R" processors and the related frequency scaling data for AVX-512 loads.
For the throttling operations you need to decide to commit to using them heavily or not. If you go all-in on them and use them well, you will get the same work done in less time and less joules than without them. If they won't help, don't use them. But, popping in and using them for tiny bits would be the slow and hot worst of both worlds.
Here's how I think of Intel chip features now after years of getting burned :
1)Has the feature been in the chip for one generation?
True -> Skip this chip, (if I need to feature, that is) it'll have the feature disabled by microcode after someone realizes it may be a security or inaccuracy issue
False -> Go ahead and buy it, probably at least a 50/50 coin flip that the feature will stay stable
The throttling will surely be reduced when Intel gets its process issues resolved(7nm), which is probably around when they finally ship AVX512 on desktop.
No point. He just made a commend about something he things is annoying in a mailing list and some media makes a big deal out of it because it's Linus Torvalds. That's all there is to it.
Before I get downvoted to oblivion - a lot of good people worked on that feature because it was their job, presumably some women, and I'm sure they brought passion and creativity to the endeavor, even if it was a dumb management decision. I guess HN approves of telling those people that their work should die a painful death.
I didn't say keep it. I suggested using words less harsh than I hope it dies a painful death. If someone said that about something I worked hard on it would cause me pain
His idea that AVX-512 is something that's exclusively for floating-point is completely off-base, AVX has included integer operations since AVX2. Widely used in JIT, database, etc.
Furthermore, AVX-512 is about much more than doubling the vector width, it is a significant overhaul to the instruction set and adds many new operations and "fills in gaps" that were missing from previous instruction sets. It in fact would be perfectly valid and good to implement AVX-512 with a 256-bit unit that takes twice as long to run 512-bit width instructions. This completely negates all his points about die space utilization right from the start - AVX-512 support does not imply a significantly larger use of space than previous AVX instructions. This would also fix some of the power-related problems on Skylake-SP - after all if you go from 2 512-bit wide units to 2x256 gangable units or 1x256 running at half-rate, that reduces power correspondingly and you no longer need to drop clocks so strongly to offset this, but you keep the functionality added in AVX-512.
Furthermore, it's not like there are massive gains in general IPC that haven't been tapped. AVX-512 has taken 25% of the die area in some instances, if you dropped that to AVX2 (assume 12.5% of die area) then it's not like the processor would be 12.5% faster in general, that would translate to maybe 2-3% faster in general and a 30%+ loss in specialty applications. Once you've mostly explored general-purpose gains and are into diminishing returns territory (which modern processors certainly are), it makes sense to start looking at "specialty units", like AVX, or on GPUs you've got tensor cores and BVH traversal units, and so on. These can provide big speedups in key tasks at the cost of very little "general" performance (since that's already in diminishing returns territory).
The biggest thing slowing down AVX-512 adoption has been, yet again, 10nm. Right now it is only available on Skylake-X and Skylake-SP products, and more recently Ice Lake (which came out September of last year, in only the ultrabook segment, supplemented by 14nm in the mobile workstation segment as well as the ultrabook segment). So right now it is available in less than 1% of the desktop "fleet" and probably less than 1/8th of the laptop "fleet". There is very little reason to implement code paths for an instruction set that nobody can execute. Over time, as Ice Lake and Tiger Lake build share of the laptop "fleet", Rocket Lake implements it on desktop, and AMD implements it whenever, it will see more usage, just like prior AVX sets.
It really is wider-market than people realize. I have seem many people scoff and say "well you'll never see it used in games or whatever", but for many years now there have been games that simply will not run if you don't have AVX (notably many Ubisoft titles), there is no fallback SSE/scalar codepath. In another 10 years you will probably have AVX-512 mandatory games as well.
With all due respect to his long career in software engineering, that doesn't necessarily translate to processor design. This is just one person's opinion and you are under no obligation to accept it as gospel just because it's Torvalds'. See also: his weird ZFS rant.
(This seems to be a common thing with software engineers in particular, including many on this site - can't count how many "one weird fix from a software engineer to fix [complex domain problem] in [chemical/materials/aerospace engineering]" I've seen. I of course have no particular expertise in processor design either, but the engineers at Intel presumably do, and they thought it was a good idea.
His experience does translate pretty well to typical computer workload analysis. Realize he has to digest, at least partially, most of the kernel commits.
Someone made a comment about special purpose instructions being the future. I think that is true, in part, but those instructions might be better off on an accelerator of some kind, or at least interfaced to the chip in a way that is better than how AVX512 must be used.
> Someone made a comment about special purpose instructions being the future. I think that is true, in part, but those instructions might be better off on an accelerator of some kind, or at least interfaced to the chip in a way that is better than how AVX512 must be used.
The problem with the coprocessor model is that it adds an enormous amount of latency. A lot of things aren't worth pushing off to a GPU because it adds so much time, and because there is a limited amount of bandwidth to push it all there and back.
Furthermore, the farther you are sending something, the more power you need to do it. Sending it off-chip is significantly less efficient than doing it on-chip, all things equal.
RTX is sort of a microcosm of these problems - people really wanted NVIDIA to implement RTX as an off-die "accelerator chip" (and some dumb rumors are even bringing this topic back for the 3000 series despite all the reasons it doesn't work). But it actually makes no sense to separate this - you can't just push it off to the RT core and be done with it, after RT finds collisions you need to go back to shaders and do stuff with it, so it needs to be feasible to flip back and forth between shading, RT, and back to shading. Moving it off die would mean that you now have some kind of async thing or buffer going on and that would kill latency and make it infeasible for real-time shading at least. Furthermore you have cache benefits if everything stays paged in while you are finishing up one tile.
Or another example would be Fusion - sure the idea of having coherent memory and triggering the GPU "like an AVX unit" sounds great but it's only unidirectional cache coherency - the GPU reads an up-to-date picture of the CPU but the CPU is not optimized to read out of the GPU, so while it can work in this mode the performance is "singularly poor". This means that "flipping back and forth" does not really work here either, it is structured as a CPU-to-GPU pipeline, not a general purpose coprocessor. With as much as they were pushing Heterogeneous Systems Architecture, presumably they couldn't reasonably implement that without trashing GPU performance, cause if they could I bet they would.
Integrating specialty units into the core is the tradeoff between latency and performance - it's nowhere near as much performance as a dedicated accelerator, but it's nowhere near as much latency either. Depending on how much work each item needs done, it may be better to do it on-core and keep trucking. And often this is not particularly expensive - things like crypto acceleration instructions are basically uncontroversial.
And that is analogous to something like BVH traversal units on NVIDIA, where a ~5% die area expenditure speeds up that segment of the program by 8x or so and results in 2x performance improvements program-wide in some cases. The NVIDIA equivalent to AVX-512 would be tensor cores I suppose - not all that useful to general workloads (apart from DLSS), consuming a fairly significant amount of die area (around 10-15% iirc) but greatly desired by some subset of HPC customers to whom NVIDIA really wants to cater.
>Or another example would be Fusion - sure the idea of having coherent memory and triggering the GPU "like an AVX unit" sounds great
As we see though, if you need to "trigger" the unit you may have problems. In this case AVX triggers a cooldown. Running the AVX instructions right now is just as if you triggered an alternate mode.
The examples you give citing NVIDIA seem to be more inline with what I'm suggesting. The shared memory fabric allows fast offload the the extremely specialized processing units.
The issue is these terms are fluid. It used to be that PCIe was the fabric, but now that's too slow, so what is on the die is the fabric, etc.
> As we see though, if you need to "trigger" the unit you may have problems. In this case AVX triggers a cooldown. Running the AVX instructions right now is just as if you triggered an alternate mode.
Not all AVX instructions trigger a cooldown to build voltage, only 512-bit ones or 256-bit multiplies. So the penalty in these cases is in fact zero unless you are doing a lot of them and triggering cumulative power cooldowns (in which case you are seeing a large benefit in throughput, although potentially you should consider offloading them at that point, yes).
Also, AMD doesn't have cumulative power cooldowns or clockdowns at all, that is an Intel implementation detail, not a fault of the instruction set.
> The examples you give citing NVIDIA seem to be more inline with what I'm suggesting. The shared memory fabric allows fast offload the the extremely specialized processing units.
No, actually, despite some confusing block diagrams, RTX and tensor cores are actually part of the SM engine, not "standalone units" in a different part of the chip. They live right next to the shaders.
There is no "fabric" there, it is execution units working on registers just like any other instruction.
> Not all AVX instructions trigger a cooldown to build voltage
Ok, but the AVX instructions that do trigger the cooldown may be the ones you wish to use, and the statement is still valid. I never mentioned AMD.
> not "standalone units"
I didn't say they were standalone, in fact, the opposite. It just depends on what you want to call a boundary. The tighter integration is what is driving these special purpose units but up until now they've not affected package TDP so heavily.
> Ok, but the AVX instructions that do trigger the cooldown may be the ones you wish to use, and the statement is still valid. I never mentioned AMD.
Again, it's not that AVX instructions inherently must be clocked down for. It is an implementation detail of Intel's processor. AMD already has different boost behavior for AVX2 and will probably have different boost behavior for AVX-512 when they implement that.
> It just depends on what you want to call a boundary
What you were saying is that they are units attached over some "fabric", which isn't the case.
> The shared memory fabric allows fast offload the the extremely specialized processing units.
They aren't "multiple processors that share memory", it is an execution unit within the Streaming Multiprocessor. It is no different from an AVX instruction, they are driven by registers, not by a fabric and not by memory.
At risk of reigniting the old Bulldozer "what is a core really anyway", AVX and tensor cores don't really represent an independent core, they are an execution unit within a larger core. They can't operate independently without the other execution resources provided by the core.
Doesn't avx-512 crush the clock. Then they still spec using the non avx-512 clock speeds for floating point somehow. I always felt there was game playing here.
That's something I addressed, perhaps not clearly enough.
Having 2x 512b units crushes the clocks. That is a factor of vector width (and having multiple units), not the instruction set itself. Half-rate AVX-512 would in all probability run at the same speeds as AVX2, and allows significantly more flexibility than "mere" 256-bit AVX2.
That's perhaps something that is not clear from Linus - whether he is railing against the instruction set, or the implementation in Skylake-SP, or both (I think it's both). I would probably agree that Skylake-SP takes it too far with dual AVX-512 units, but the instruction set itself ("stop inventing magic instructions"), yeah he's way off base.
I am really hoping that one of these Zen generations, AMD includes half-rate AVX-512 like they had half-rate AVX2 in Zen1/Zen+. That really seems like the best of both worlds, you get the flexibility of AVX-512 but without the die area or the power issues. If not, well, Rocket Lake is coming end of this year/early next year and that should bring AVX-512 to consumer platform desktop (as it's in Tiger Lake and Rocket Lake will be a Tiger Lake backport) and will probably have only a single AVX-512 unit since it's mobile-derived.
Do you / Intel GUARANTEE no frequency scaling to handle 512?
Intel quotes these specs on throughput and seem to do it using full frequency for FP. But if you have 512 in your mix (ie, 1-2% 512) do you really get that speed. Cloudflare talks about frequency dropping to 1.4Ghz
Lack of adoption of AVX-512 would be Intel's own issue. If they wanted it to succeed, they would try to put it on more chips than just the few lines you mentioned. As it currently stands, Intel is setting up AVX-512 to fail by only putting it in so few products.
I doubt you will find much AVX512 code with Daniel. The costs are just too high to justify it.
Downclocking because of overheating and insane waitcycles to get out of this mess which is AVX512, plus poor HW support (only on insecure Intel CPU's which need to be probed at runtime) makes it an esoteric extension. AVX256 is good enough for most purposes and is not that insane.
Search "daniel lemire" "AVX-512". Vectorizing random number generators for greater speed: PCG and xorshift128+ (AVX-512 edition) [1]. AVX-512: when and how to use these new instructions [2]. The dangers of AVX-512 throttling: a 3% impact [3].
> Downclocking because of overheating and insane waitcycles to get out of this mess which is AVX512, plus poor HW support (only on insecure Intel CPU's which need to be probed at runtime) makes it an esoteric extension. AVX256 is good enough for most purposes and is not that insane.
> Lack of adoption of AVX-512 would be Intel's own issue. If they wanted it to succeed, they would try to put it on more chips than just the few lines you mentioned. As it currently stands, Intel is setting up AVX-512 to fail by only putting it in so few products.
It will be fun to revisit these sorts of comments next year when AMD releases Zen4 based processors that support it. Like with raytracing, I suspect people's tune will change when the red man does it instead of the blue man or green man ;)
(it may even be in Zen3, that "50% improved FP performance" sure is interesting if you figure that's about what you would get from ideal-case speedups of AVX-512... I suspect they are moving to dual 256b units and at that point you might as well make them gangable into a 512b unit. If they don't go there with Zen3 it is an obvious low-hanging fruit for Zen4.)
> With all due respect to his long career in software engineering, that doesn't necessarily translate to processor design.
However, his particular career includes a stint at Transmeta, so he has probably thought about processor design a bit more than the average tenured software engineer. And it might also account for some of his bitterness toward Intel.
>> It in fact would be perfectly valid and good to implement AVX-512 with a 256-bit unit that takes twice as long to run 512-bit width instructions. This completely negates all his points about die space utilization right from the start - AVX-512 support does not imply a significantly larger use of space than previous AVX instructions.
The extra registers have to be there even if the execution unit is half size. The context switching overhead grows, which is something kernel developers care about. And then with all these variants there's a push to have a single binary support them all which complicates the code too. Meanwhile the world is moving toward parallelism which is going to mean more thread creation and context switching.
Are those AVX-512 registers saved on context switches? At least on syscalls, the kernel could just have a policy of not using those registers? I'm not exactly sure how that works, this is part of the ABI right?
Intel? Dude, have you seen Arm's JavaScript instructions? Or RISC-V's user-defined instructions? (and Arm announced the same thing at Arm Tech Con last year)
Someone is missing the boat: custom ISA is the future.
> Someone is missing the boat: custom ISA is the future.
Custom ISA are 99% of the case useless marketing porn for hardware provider.
The reality is that most Linux distributions in 2020 are still binary compiled with SSE41 instructions (maybe AVX at best). Instructions that shipped decades ago.
AVX2 and AVX512 being barely supported in some HPC parts and the libc itself.
SIMD is good, but fragmentation of instruction set is much more of a problem in most case.
Nobody will dare to support your fancy new instruction if it covers 2% of the CPU used world wide and make their software a pain in the butt to distribute.
A lot of software these days is run by JIT compiling VMs, anything Java, anything JavaScript, and that trend is likely to accelerate in future with new tech like Graal that can JITC even stuff like LLVM bitcode or Python.
The nice thing about JITCs is:
1. They run parallel to the application so can use those spare cores your program isn't using.
2. Especially true as they run a lot during startup when your app is probably single threaded anyway, even for something like a web server where it's going to be multi-threaded soon.
3. Upgrading the VM upgrades the compiler, so new instructions can be used almost immediately, as long as your software is tested/runs on the newest VMs.
JVM has historically tried to auto-vectorise everything and not been all that good at it, but now it's getting vector extensions that generalise to newer instruction sets fairly well, so we might start to see more HPC done in higher level languages.
>A lot of software these days is run by JIT compiling VMs, anything Java, anything JavaScript, and that trend is likely to accelerate in future with new tech like Graal that can JITC even stuff like LLVM bitcode or Python.
That's theory. Practice is that JIT-compiled language generally performs even worst than compiled language with an outdated instructions set.
And there is reasons to that, compiler passes are expensive, including vectorization, and you specially do not want anything expensive running in your critical JIT passes.
Some language like Julia still do it but to the price of a very long compiling and starting time.
Something that has never been acceptable over JVM or in Nodejs.
Comparing language performance isn't the same thing as comparing JIT vs AOT performance.
Generally JIT outperforms AOT for a language like Java by about 20%. For Scala, it's even more. For a more dynamic language like JavaScript or Ruby you don't even try to AOT it at all the difference is so huge.
Even in languages that are normally AOT compiled, profile guided optimisations make a big difference. For C++ I've seen figures in the 15%+ range. That's a lot! Most projects don't use PGO though because it's a pain to deploy.
C2 does auto-vectorisation and it's not a slow compiler. It runs in parallel with the app, it's not a problem. The difficulty with auto-vectorisation isn't how much time you have to compile it, it's more that matching all the different loop constructs to the instructions is a very complicated problem that ends up needing tons of special cases in the compiler. It's a problem of code complexity rather than runtime performance. And it's opaque to the user: if they change their code structure a bit and it's no longer recognised as a vectorizable template, it'll stop being vectorised and performance drops off a cliff. You can't really see that though as a developer ahead of time because it's all just "best effort" optimisation. Game devs in particular hate that and would rather have a compiler error than silently bailed out optimisation passes.
That's why Java is switching to explicit vectorisation. It's not about JITs not having enough CPU time, it's about giving the developer reliable and predictable performance even in the face of arbitrary refactorings.
I don't think many people have a problem with special case, highly-complex instructions when they make a lot of sense. Reading into FJCVTZS leads me to believe that this is a fairly common use case for a type of code execution that could hypothetically occur with any arbitrary website on any ARM device (aka smartphones). I also assume this instruction does not restrict the clock speed of the CPU based on its behavior or otherwise impact the general case adversely. The availability of this instruction likely only improves the experience of the user when utilized.
I personally have a problem when compromises are made for the general case to satisfy a special case, especially when the special case motivations appear to be bound mostly in marketing reasons. Even more so if the special case is well-known to adversely impact performance in the general case.
Before I saw this post, I didn't even know ARM had some complex javascript instruction. Contrast that with - You can't watch or read a review for any current gen Intel server product without being presented with at least one AVX benchmark. Is this marketing or is it practical engineering? For how many applications is AVX availability actually a hard constraint?
I find it amusing, er, ironic, er, amusing -- that after this article, on the same web page, there's a link/blurb which reads:
"Nvidia is worth more than Intel for the first time in history. Nvidia is now worth more than Intel, according to the NASDAQ. The GPU company has finally topped the CPU company's market cap (the total value of its outstanding shares) by $251bn..."
Well no surprise there.
On the one hand, Intel (especially early Intel employees) should be thanked, profusely, for giving us PC history as we know it today.
On the other hand, we should seek to honestly recognize what Intel has become today.
A mega-corporation, driven by corporate mentality, which always seeks to maximize profits for their shareholders at the expense of all other virtues.
Keep in mind I am not criticizing Intel's employees -- only the marching orders that come from the top down.
But, as far as I can tell, Intel, as we know it, will not be around in another 30 years.
The future of semiconductors is in the following areas:
1) Simple, non-proprietary instruction sets (RISC-V and others)
2) Transparent (or as transparent as possible) and publicly auditable engineering and manufacturing processes
3) Conscientous companies that place virtue first, and are not driven by maximimizing profits for shareholders
So on the one hand, Intel is to be thanked, lauded, praised for its role, especially its early role in the beginning of the PC revolution, but on the other hand, I don't see Intel existing as company more than 30 years from now...
Although, at that point in time, Intel's past role will always remain important and relevant -- to future students of early computer history...
Your "future of semiconductors" could have been written 30 years ago, and seems no more likely to be true today than it was then. Do you think Nvidia is less proprietary, more transparent, or more virtuous than Intel?
Well, combination of the lead in x86 architecture (a lot of times close to 100% of market supply) combined together with lead in manufacturing (first in process shrink down to 14nm) was unique and highly unlikely to be repeated.
Commoditization of architecture (ARM seems to be in the lead by RISC-V might go up) plus separation between chip design houses and manufacture fabs seems to be more efficient market solution. Where it leaves Intel?
> This is not the first time Torvalds has directed his ire at Intel. In 2018, Torvalds referred to Intel's Meltdown and Spectre patches as "COMPLETE AND UTTER GARBAGE," in all caps to emphasize his level of anger.
I'm 1000% behind him on this point. Fix that first and fix it well. Then get onto fancier stuff that i'll never ever code against.
The thing is, "that I'll never ever code against" is context specific. The set of applications that want to be embarrassingly-serial and not utilize any massively-parallel features is shrinking (even problems traditionally solved via serial code can benefit from a parallel re-architecting, assuming the path from language to CPU architecture supports true parallelism).
I think they mean that SIMD only provides a very particular, limited kind of parallelism. It's only useful for some kinds of parallel problems.
Anything where the execution traces diverge significantly requires more execution cores, to sequence them separately. For example serving multiple web pages in parallel.
Anything where the executation traces are identical or near-identical across many repetitions in lockstep, is fine with one execution core and SIMD ALUs. For example dense or block matrix arithmetic.
Modern GPUs are a hybrid of both ideas. Shared execution cores and SIMD ALUs because of high repetition of identical logic, but still many execution cores (but fewer than ALUs) so there is some amount of decoupled parallelism as well, running different tasks at the same time.
GPUs are somewhat more like "Super powerful SIMD". That is, you tell a GPU "I want n cores to run this block of code" and that's just what it will do.
SIMD is much more primitive. It is more along the scale of "Multiple these 4 numbers by 5" and that's it. A GPU kernel can have branching, conditionals, etc. SIMD is limited to one operation on a small set of data (512 bytes for AVX-512).
CPU SIMD can do conditionals and branches in the same way as GPUs, using per-lane conditionals.
A modern GPU isn't limited to running one block of code at a time (on n cores). It will schedule a few different blocks of code independently - up to the number of execution units on the GPU, multiplied by the number of latency-hiding time slots per execution unit.
I'm still waiting for languages to support 2,3, and 4D vectors as first class data types. These are so common it's silly to have people define them. We end up with different implementations sometimes too.
Well... MATLAB (matrix laboratory), and its GNU counterpart Octave kind of do this. The primary data type is an n-dimensional matrix. Of course, there's a lot to say about the rest of the language (and it's untyped/interpreted, though BLAS handles matrix operations so it's extremely fast for vectored code), but it is very nice to work with vast amounts of data in it, and numpy tries to mimic parts of its syntax. That's something I would really like to see in other languages, but depending on the use-case, you can also mix languages...
u=rand(5) % 5-by-5 matrix
v=rand(5,5,5,5,5,5) % 6-dimensional matrix, each dimension 5 long
v2=v(1,2,3,:,:) % 1-by-1-by-1-by-5-by-5 matrix
w=u(:,3) % take every line in the third column, so that's a vertical 5-by-1 vector
x=5*w*w' % matrix multiplication of w and its transpose, times a scalar
y=rand(5).*x % element-wise multiplication
While I love Linus and I think he's excellent at what he does, he's utterly missing the fact that things like AVX-512 sell chips, and that's what Intel's in the business of doing. And best practice in how one writes software will increasingly look like "write it so it executes well in a massively-parallel context" if the driver for chip sales is massively-parallel problems.
We've smacked pretty hard into the wall of how fast we can make CPUs by miniaturizing components, and the low-hanging fruit now is parallelization, predictive execution, and the whole host of ways to do more per clock cycle, not speed up clock cycles. But that flies in the face of the traditional embarassingly-serial pattern of the x86 instruction set and computing environment. Much of what Intel's doing these days is trying to open up opportunities for people to change tools to speed up code without having to boil the ocean by throwing the whole serial-instruction model completely out the window (even though, increasingly, code written to that instruction set is really operating like a language that is emulated by the underlying parallel-and-predictive CPU hardware).
I think what Linus calls "regular code" isn't going to move chips and we've wrung all the cheap optimizations out of that critical path (and, depending on the details of what is meant by "regular," such code is increasingly going to ram up against an efficiency ceiling unless it can be moved to a model of "Prepare work to do in parallel, execute in parallel, merge results and present to UI").
Pretty sure this is exactly what Linus is complaining about. It's not that he doesn't get it, he's upset that intel is spending resources on what he sees as advertising instead of making the product better.
But more direct access to parallel execution hardware is making the product better. It's not making it better for his use case, which is not surprising because Intel is in the business of selling chips, not optimizing for the Linux kernel use case.
Marketing isn't the only thing that sells chips. OP is talking about actual usefulness. It's not that useful to Linus and what he needs. OP is arguing that it is useful to others, doing different things, and that those use cases sell chips.
> he's utterly missing the fact that things like AVX-512 sell chips, and that's what Intel's in the business of doing.
But wouldn't something more useful than AVX-512 sell more chips?
Massively parallel floating point workloads on the CPU are a modest niche compared to other ways Intel could have spent engineering effort and chip area, like not ruining the performance of just about everything else with security vulnerabilities.
Nobody saw those security vulnerabilities coming. And ironically, they were introduced due to changes that are intended to make the chips faster without explicit parallelization. This goes back to the problem that a modern architecture for a CPU is a massively parallel machine emulating an instruction set design for an embarrassingly serial machine based on best practices from the 1970s.
Two years ago, I wrote an LLVM compiler pass that automatically upgrades SSE or AVX-2 code to AVX-512 when those instructions are available. It's helpful for getting performance gains for hand engineered code that you don't want to touch. We saw some good gains on integer workloads (unpacking, useful for databases - something I'd call "regular code"): 1.16x speedup going from SSE to AVX-2, and 1.43x speedup going from SSE to AVX-512. Even better speedups are available for FP workloads, though hand-writing the AVX-512 version can work a lot better since programmers can exploit the full range of new instructions available in AVX.
The catch is "hand engineered code that you don't want to touch."
I helped put an early deep network application into production that used one particular generation of vector instructions and it was clear everyone was traumatized at the process of implemented it to use those vector instructions and would never do it again.
We later bought servers that had instructions that we weren't using, but we were so focused on getting the product out the door (which we did) that we left those performance gains on the table.
The consequence of the "new instructions every two years" route is that end users don't really experience the performance benefits engineered into the chips. The vast majority of software development organizations are more concerned about support costs than they are about getting the last bit of performance out. (Also, end users are already tired of 200-megabyte binaries, long start times, and other performance decrements that come from high-complexity libraries that contain a huge number of code paths for different CPUs)
Many working assembly programmers who are good at SIMD love the Intel approach, maybe because it keeps them in business. People are doing very cool things in the parsing space (e.g. it is shocking how slow it is to parse ascii-formatted numbers, JSON, XML, RDF, ...) with SIMD instructions.
However, the old tradition in vector computing is the variable-sized vector unit like what the ILLIAC had or the old Crays, the vector accelerator for the IBM 3090, etc. ARM has the scalable vector instructions which are organized that way.
ARM's Helium had a nice middle ground of having "small, medium and large" implementations available which all run the same code. You might get higher peak peformance on Intel's road, but higher realized performance with ARM.
Yep! You got the motivation exactly right. The goal is automatic program "rejuvenation" w/o all the engineering and maintenance costs. Revectorization in the compiler allows programmers to capture much of the performance gain possible with wider instruction sets without having to write and maintain multiple code paths. It can be a big pain to understand and rewrite this vector code -- especially if it's in some external library or another team's code. But agreed that some really nice results are possible with the right engineering such as in parsing.
Do the binaries really get into the 200 MB range from different code paths, or is that due to data?
Cool technique. I read the evaluation and there is afaik no mention if the tests were run multi-threaded or single threaded.
Considering the rather complex dynamic of Intel's downclocking with AVX512 workloads, that would be pretty interesting.
Similarly the AWS VM CPUs were clocked at 2.0Ghz, which is pretty low and possibly does not even downlock in that configuration, were tests run on desktop CPUs that go to 5Ghz regularly on non-AVX512 workloads?
How is the advantage there?
Thanks! The tests were run on a single thread since the source kernels & test programs were single threaded. It was important for us to find the largest possible instruction graphs to revectorize at a time so that the whole kernel or most of the kernel runs with the target instruction set. If our search terminated early due to e.g. a pair of low-width instructions that the pass didn't know how to merge, there's a performance loss, so we spent quite a bit of time to synthesize comprehensive lookup tables and implement shuffle merging and reductions. My impression is that kernels running exclusively AVX512 rather than interleaved scalar/low width code generally perform better than the low width code even with downclocking, but please let me know if that's not the case.
We originally ran experiments on a multisocket lab server which should have a faster clock, but I don't have a record of the specs. That server did fine for SSE to AVX2 conversion, but didn't support AVX512 so we ultimately used Google Cloud VMs.
x86 instructions I'd like to see are atomic instructions with lower global memory ordering guarantees. TSO is nice, but there's enough performance sensitive concurrent code where opting out would be useful.
I see a huge bump in numpy performance with AVX-512, to the point where I wouldn’t buy a cpu without it. I don’t know that I understand the critique—-is it because these instructions are Intel specific and not on AMD? Seems obviously useful for scientific computing.
The word "founder" just means "person who started a thing." The use as a synonym for entrepreneur is something I basically haven't seen anywhere outside of very business-focused contexts.
Seems like many people here haven't actually read Linus's specific thoughts on AVX-512, and are accusing him of rejecting SIMD entirely or the usefulness of FP performance entirely. This is not an accurate representation of his position and he singles out AVX-512 differently from SSE/AVX/AVX2.
Here is a link to the thread directly from Linus explaining his position on MMX/SSE, AVX/AVX2, AVX512. His complaints on AVX512 are both about fragmentation and regressing from the learned lessons from their prior generations. And he also looks to NEON and SVE2 and suggesting ARM looks seems much saner to him:
"So just as a bystander, I'm looking at AVX512, and I'm looking at SVE2, and I'm going "AVX512 really is nasty, isn't it"?""
163 comments
[ 3.2 ms ] story [ 206 ms ] threadThey only care about long term market, which is HPC and ML workloads. Because Nvidia is destroying anybody in that market. Look at their stock.
I’ve got a news for Torvalds, it going to get bad for Intel.
If there's anything that has typified Intel in the past decade it's a generalized failure to execute. There's been very little innovation in the core product since Haswell in 2013.
Should I read this as sarcasm?
> Should I read this as sarcasm?
It could only be sarcastic if he's referring to "Linux on the desktop." I don't have any numbers, but I wouldn't be surprised if he was correct about servers.
But I do think Windows's position in the desktop space is somewhat unsustainable at this point. I personally prefer Windows but I just don't see how a heterogenous user interface experience (with users on iOS and Android for mobile and Windows for desktop) will abide for long.
I think it's only a matter of time before mobile OSes make some serious inroads to desktop. Though I suppose Google has been salivating at that thought for neaely a decade.
50 years ago, IBM owned the computing market with mainframes. They still dominate the mainframe market, that's just become a smaller and smaller portion of the overall computing market as time has gone on.
Windows has declined. It is still used extensively, but Microsoft no longer dictates the browser market. Microsoft Office is commonplace, but has genuine marketshare competition in alternatives like Google Docs. Developers increasingly build "desktop" applications as web-based or multiplatform-first instead of Windows-exclusive, with the notable exception of AAA games. The Linux subsystem for Windows is a direct admission by Microsoft that developers want to work with Linux-based software for many tasks.
Also thy are in many other areas then just CPU's including FPGA and hard drives.
To be pedantic, Intel does have phone and tablet processors. This doesn't invalidate your point however. I can't easily find a breakdown of mobile cpu market share but I am guessing Intel isn't doing very well.
https://www.intel.in/content/www/in/en/products/devices-syst...
https://www.intel.com/content/www/us/en/products/devices-sys...
Not really, like you mentioned in some ML/HPC contexts GPU computing with commonly Nvidia GPU is a better solution in the desktop marked AMD's recent CPU's are viable alternatives and that also applies for many usages in the server marked. And on mobile, well Intel CPUs never did really compete. Same for game consoles. Oh and for low power embedded systems it's often ARM and for thinks which also need GPU (self driving cars) it's ARM+Nvidea.
I mean sure parts of server the server marked are build on things like Intels (not so) secure enclave and then for some gaming workloads Intel CPU's still beat AMD and for not supper expensive software dev. workstations Intel also still is the best choice as the current CPU+embedded GPU chips from AMD for desktop still use their old CPU.
Through on thing which shouldn't be forgotten is that they have their hands in a lot of other things including:
- FPGA, if you use certain ML algorithm a lot putting them onto a FPGA can be a massive power and time saver. So the Intel CPU + FPGA are quite interesting for such cases.
- Graphics, Intel is about to bring out a dedicated graphic card. Probably not interesting for high end usages but it might be a use-full solution for budged setups or GPU servers (It's often monetary better to have multiple smaller cards then one big cards, it's not uncommon for GPU server to have Nvidea 2080ties instead of Quadros etc.).
- Hard drives, Optan "SSD" are not very well marked but while they are not necessary the fastest SSD's they have good latencies which can make them preferable for some use-cases.
- ...
Another thing which can be beneficial for them in the coming 10 years is that they do not depend on TSCM.
One funny thing I noticed is that Intel can build a good part of a computer today: CPU + GPU + Motherboard + Network Card + Drives + FPGA + OS (Clear Linux), so add RAM, PSU, Cooling, Cables & Case and you have a Intel only desktop ;-)
[0] https://news.ycombinator.com/item?id=23810654
If your algorithm isn't sensitive to either one of these, then yes, the speedups are often smaller, particularly if the limiting factor in performance is memory bandwidth and/or latency (AVX-512 doesn't help at all there), or if the speedups aren't enough to overcome clock throttling (which IMO is a bit overblown; most people cite the Cloudflare blog article from a few years ago that complained about throttling on Xeon Silver processors, but throttling is much less of an issue on higher-TDP Gold/Platinum CPUs). However, I have seen specific applications that are >100% faster with AVX-512.
Current AVX512 implementations might not always get there, as I believe some instructions are dual issuing at 256bit under the hood.
With AVX-512 and CUDA, Intel and Nvidia are trying to find the best developer experience they can to deal with the inevitable future of computing that is somewhere between those two experiences currently.
Not sure that ever happens. The two are very different in their performance characteristics. CPUs have to stay focused on single-threaded performance.
Single threaded performance = minimizing latency, that’s why in CPUs so many transistors and electricity is spent on cache hierarchy and coherency protocol, instructions reordering, ILP, microops fusion, prefetcher, branch prediction incl. indirect one, and so on.
GPUs don’t care about single-threaded performance, all they care is throughput on massively parallel tasks. They ignore memory latency, switching threads until data arrives, makes caches much simpler. They don’t need high clock frequencies (this alone improves their flops/watt by a large factor), don’t do branch prediction, memory model is less strict, and so on. That’s how they can spend that many transistors on FPUs and ALUs.
P.S. Intel tried many-slow-cores with Larrabee / knights landing / etc., all they got was a very expensive lesson, people didn’t want that stuff. I’ve heard a rumor initially Sony wanted to use another Cell CPU for the GPU in PS3, again, didn’t worked well enough and they negotiated with NVidia instead.
We can still add more transistors. But, there is nowhere for them to contribute but in going wider. High latency, high throughput. High concurrency and careful management of the memory hierarchy are the only ways to pull that off. When you accept that fate, you start out with the Cell processor and you pass through Nvidia's Ampere chip --which has somewhat more thread independence and better automatic caching than past GPUs, but also finally gained manual async DMAs to local SRAM like the Cell had. Where we'll end up, I don't know.
I agree on physical limitations, but the research still goes on, smart people still able to use these extra transistors in a way that’s useful for performance. Even for single-threaded one.
Another thing, software is progressively better at using many cores. It’s much easier to move large pieces of code to another CPU thread (like some browsers do with JS compiler) keeping it sequential code, than it is to re-write code as a large set of small tasks / threads. Many useful algorithms can’t use fine-grained parallelism much, or at all, e.g. streaming stuff like gzip or parsers.
https://www.karlrupp.net/wp-content/uploads/2018/02/42-years...
BTW: Thanks for http://const.me/articles/simd/simd.pdf I came across it recently and posted it to https://www.reddit.com/r/simd/
Someone should probably tell AMD and Intel they need to stop, then, because Ice Lake was +18% IPC, Tiger Lake will likely be another 5-10%, and Zen3 will be around 15% higher IPC.
What has truly stalled IPC is 10nm. Intel has had higher-IPC architectures for a long time, they simply couldn't manufacture them. And AMD was coming so far from behind that it took them 3 generations of Zen to even catch up to Intel.
The last few years have been a rather unfortunate "CPU dark age" but things are finally breaking loose once again. Ice Lake launched 9 months ago, Intel backported Tiger Lake to 14nm and that will launch probably around Christmas or CES, Milan is launching probably within the next 6-8 weeks, and Tiger Lake is launching this month.
CUDA in particular hides the exact width of the processing elements from the programmer, so you don't have to rewrite your CUDA applications to take advantage of the next generation of GPU the way you have to do with AVX1,2,3,...
Mind you, NVIDIA's drivers compile your shaders to their proprietary instruction set, which is a path not taken in conventional CPU architecture even though it keeps coming up. (e.g. the belt)
Your assertion sounds like a gross exaggeration. Some benchmarks show only a few percent improvements (i.e.14% improvements on synthetic benchmarks), and results range between no improvements and almost 2x in some synthetic benchmarks.
https://www.phoronix.com/scan.php?page=article&item=gcc9-sky...
To be clear I am not suggest "avx-512 will cause programs to run twice as fast or more", just that specific functions that used to use AVX-2, may be more than twice as fast if you move them to AVX-512.
I presented data that showed that AVX-512 in reality only brings margins performance improvements.
If you have any data that shows otherwise then feel free to present it.
Still, hand-tuned assembly is hardly a relevant example, let alone selling point, as it creates hypothetical expectations that are never ever matched in reality.
There's a ton of hand tuned assembly running on your system in numerous libraries. Media and data compression? Hand tuned assembly. Matrix math? You guessed it. C-standard library functions? Yes, really.
Certainly you have required architecture instructions for things like instruction scheduling (isync/dsync), stack manipulation (setjump/longjump and others) or floating point, but though it looks like a lot of assembly source, it is really just a little bit for each of the many supported architectures in the tree. And most of it isn't performance related, it is just plain required because architectures vary on things like instruction scheduling and stack/cache manipulation.
In enterprise ML contexts on tabular data problems we found there are a lot of cases where even training can be greatly sped up by leveraging AVX instruction support in e.g. tensorflow builds. The gains from AVX instructions could boost training time by ~20% on the GAN use cases I profiled.
What percentage of consumers with these chips installed do you think are getting a performance win? Do you think it might be as high as 1%?
Do you think that the same resources devoted elsewhere might be worth more than 1% to that 99%? Whether that is in reduced cost, reduced bugs, or a boost for more widely used operations.
Yes, if you target a specific use case for a specific set of people, you can give them a nice win. But you shouldn't lose sight of the fact that CPUs cover a lot of use cases for a lot of people. And simplifying then focusing on the core mission is better for everyone in the end.
Turning factual reports of where the tradeoff was helpful into a strawman insulting the reporter, and the users who benefit, is neither charitable nor illuminating.
The only AVX512 instructions that actually seem essential imo are the VNNI extensions.
There are a lot of use cases for a neural network that is trained on the GPU but where inference is done on a bunch of CPUs because it's cheaper than buying/renting inference hardware.
There's a reason AMD is and Intel are able to cram full GPUs onto the same die as the CPU.
Yeah, here's a good image of how much space it costs to add a 'second' AVX-512 unit to certain SKUs. Cutting AVX-512 support entirely would save a bit more space, and the total would be enough to make a difference, but it's not an extreme amount.
https://software.intel.com/sites/default/files/managed/5f/99... https://twitter.com/GPUsAreMagic/status/1256866465577394181 these too (speculative)
Ultimately AVX will be the future of Intel if they want to compute with GPUs and if everyone stays silicon.
Eventually, I see us going basically full EPIC. That is "here's a batch of instructions that can happen in parallel, run it".
The failure of the larrabee team to bring something competitive to the market a decade ago doesn't necessarily imply it will become competitive just because it was brought "on-chip" instead of as an additional card or a co-processor.
If it worked all that well, it would be a product of its own, able to compete against Nvidia in GPU and GPGPU tasks.
The hardware engineers were "correct". It's a difficult instruction to make fast compared to completely independent mul-adds. Game devs "should just" re-architect their code to not deal with small situations individually and instead be more bulk-data-processing like SPECfp because that's what's friendly to the hardware. But, that's dismissing how the software actually worked --especially back then.
16 years later, many, but far from all of AAA game engines more closely resemble SPECfp. But, all along the way, and in tons of 3D graphics and gameplay code still today, it would have been nice to reliably have one of the most common and fundamental operations in games well-baked into the chips.
Your praise here is for the former, and Linus' complaints seem to generally be targeted at the latter.
https://lemire.me/blog/2018/08/25/avx-512-throttling-heavy-i...
https://lemire.me/blog/2018/08/24/trying-harder-to-make-avx-...
https://lemire.me/blog/2018/08/15/the-dangers-of-avx-512-thr...
Any performance gain from fewer cycle cost may be lost from the CPU being slower. Maybe there are situations where AVX-512 is faster but how is one to reason about performance when some instructions can slow the CPU.
The same way one should always do so on an advanced out-of-order CPU largely being programmed in modern high level languages: through thorough profiling and performance testing.
You can only see the performance values for yourself, or believe the SPEC benchmarks.
If you're working in an HPC center, you may ask your fellow centers and get off-the-record answers, sometimes.
https://www.intel.com/content/www/us/en/products/docs/proces...
I remember reading the datasheets myself and not being able to find it, together with my teammates.
So they changed their mind two months ago, and shared this info for their processors.
Guess that the information lost its significance.
Either way, a processor significantly throttles under AVX-512 load. Nice.
Edit: I re-read the document and found our SKUs under the "newly added" table and, clarified the comment.
Original Datasheets:
- https://www.intel.com/content/www/tr/tr/products/docs/proces...
- https://www.intel.com/content/www/tr/tr/products/docs/proces...
Spec Update - For re-referencing:
- https://www.intel.com/content/dam/www/public/us/en/documents...
Our mostly used SKUs are released in 2019 so, its data was not available since years ago. Intel was very open about how their tech worked when it was new, needed to show their tech against the competition and not limited by thermal constraints.
When they became the dominant player, they didn't feel bad about hiding small but important details from their consumers.
Until EPYC came and pushed & prodded them around, they didn't bother to release both "R" processors and the related frequency scaling data for AVX-512 loads.
1)Has the feature been in the chip for one generation?
True -> Skip this chip, (if I need to feature, that is) it'll have the feature disabled by microcode after someone realizes it may be a security or inaccuracy issue
False -> Go ahead and buy it, probably at least a 50/50 coin flip that the feature will stay stable
It would at a first glance appear the throttling is not caused by temperature spikes but rather just executing the instructions at all.
I wonder how much more FP performance one could extract out of an AVX512 CPU with extreme cooling.
Indeed the boost clocks are on a hard timer in general and cannot run continuously.
I see his month break to work on "unprofessional" behavior didn't include a course on non-violent communication (NVC - https://www.cnvc.org/)
Furthermore, AVX-512 is about much more than doubling the vector width, it is a significant overhaul to the instruction set and adds many new operations and "fills in gaps" that were missing from previous instruction sets. It in fact would be perfectly valid and good to implement AVX-512 with a 256-bit unit that takes twice as long to run 512-bit width instructions. This completely negates all his points about die space utilization right from the start - AVX-512 support does not imply a significantly larger use of space than previous AVX instructions. This would also fix some of the power-related problems on Skylake-SP - after all if you go from 2 512-bit wide units to 2x256 gangable units or 1x256 running at half-rate, that reduces power correspondingly and you no longer need to drop clocks so strongly to offset this, but you keep the functionality added in AVX-512.
Furthermore, it's not like there are massive gains in general IPC that haven't been tapped. AVX-512 has taken 25% of the die area in some instances, if you dropped that to AVX2 (assume 12.5% of die area) then it's not like the processor would be 12.5% faster in general, that would translate to maybe 2-3% faster in general and a 30%+ loss in specialty applications. Once you've mostly explored general-purpose gains and are into diminishing returns territory (which modern processors certainly are), it makes sense to start looking at "specialty units", like AVX, or on GPUs you've got tensor cores and BVH traversal units, and so on. These can provide big speedups in key tasks at the cost of very little "general" performance (since that's already in diminishing returns territory).
The biggest thing slowing down AVX-512 adoption has been, yet again, 10nm. Right now it is only available on Skylake-X and Skylake-SP products, and more recently Ice Lake (which came out September of last year, in only the ultrabook segment, supplemented by 14nm in the mobile workstation segment as well as the ultrabook segment). So right now it is available in less than 1% of the desktop "fleet" and probably less than 1/8th of the laptop "fleet". There is very little reason to implement code paths for an instruction set that nobody can execute. Over time, as Ice Lake and Tiger Lake build share of the laptop "fleet", Rocket Lake implements it on desktop, and AMD implements it whenever, it will see more usage, just like prior AVX sets.
It really is wider-market than people realize. I have seem many people scoff and say "well you'll never see it used in games or whatever", but for many years now there have been games that simply will not run if you don't have AVX (notably many Ubisoft titles), there is no fallback SSE/scalar codepath. In another 10 years you will probably have AVX-512 mandatory games as well.
With all due respect to his long career in software engineering, that doesn't necessarily translate to processor design. This is just one person's opinion and you are under no obligation to accept it as gospel just because it's Torvalds'. See also: his weird ZFS rant.
(This seems to be a common thing with software engineers in particular, including many on this site - can't count how many "one weird fix from a software engineer to fix [complex domain problem] in [chemical/materials/aerospace engineering]" I've seen. I of course have no particular expertise in processor design either, but the engineers at Intel presumably do, and they thought it was a good idea.
Someone made a comment about special purpose instructions being the future. I think that is true, in part, but those instructions might be better off on an accelerator of some kind, or at least interfaced to the chip in a way that is better than how AVX512 must be used.
The problem with the coprocessor model is that it adds an enormous amount of latency. A lot of things aren't worth pushing off to a GPU because it adds so much time, and because there is a limited amount of bandwidth to push it all there and back.
Furthermore, the farther you are sending something, the more power you need to do it. Sending it off-chip is significantly less efficient than doing it on-chip, all things equal.
RTX is sort of a microcosm of these problems - people really wanted NVIDIA to implement RTX as an off-die "accelerator chip" (and some dumb rumors are even bringing this topic back for the 3000 series despite all the reasons it doesn't work). But it actually makes no sense to separate this - you can't just push it off to the RT core and be done with it, after RT finds collisions you need to go back to shaders and do stuff with it, so it needs to be feasible to flip back and forth between shading, RT, and back to shading. Moving it off die would mean that you now have some kind of async thing or buffer going on and that would kill latency and make it infeasible for real-time shading at least. Furthermore you have cache benefits if everything stays paged in while you are finishing up one tile.
Or another example would be Fusion - sure the idea of having coherent memory and triggering the GPU "like an AVX unit" sounds great but it's only unidirectional cache coherency - the GPU reads an up-to-date picture of the CPU but the CPU is not optimized to read out of the GPU, so while it can work in this mode the performance is "singularly poor". This means that "flipping back and forth" does not really work here either, it is structured as a CPU-to-GPU pipeline, not a general purpose coprocessor. With as much as they were pushing Heterogeneous Systems Architecture, presumably they couldn't reasonably implement that without trashing GPU performance, cause if they could I bet they would.
https://www.realworldtech.com/fusion-llano/3/
Integrating specialty units into the core is the tradeoff between latency and performance - it's nowhere near as much performance as a dedicated accelerator, but it's nowhere near as much latency either. Depending on how much work each item needs done, it may be better to do it on-core and keep trucking. And often this is not particularly expensive - things like crypto acceleration instructions are basically uncontroversial.
And that is analogous to something like BVH traversal units on NVIDIA, where a ~5% die area expenditure speeds up that segment of the program by 8x or so and results in 2x performance improvements program-wide in some cases. The NVIDIA equivalent to AVX-512 would be tensor cores I suppose - not all that useful to general workloads (apart from DLSS), consuming a fairly significant amount of die area (around 10-15% iirc) but greatly desired by some subset of HPC customers to whom NVIDIA really wants to cater.
As we see though, if you need to "trigger" the unit you may have problems. In this case AVX triggers a cooldown. Running the AVX instructions right now is just as if you triggered an alternate mode.
The examples you give citing NVIDIA seem to be more inline with what I'm suggesting. The shared memory fabric allows fast offload the the extremely specialized processing units.
The issue is these terms are fluid. It used to be that PCIe was the fabric, but now that's too slow, so what is on the die is the fabric, etc.
Not all AVX instructions trigger a cooldown to build voltage, only 512-bit ones or 256-bit multiplies. So the penalty in these cases is in fact zero unless you are doing a lot of them and triggering cumulative power cooldowns (in which case you are seeing a large benefit in throughput, although potentially you should consider offloading them at that point, yes).
Also, AMD doesn't have cumulative power cooldowns or clockdowns at all, that is an Intel implementation detail, not a fault of the instruction set.
> The examples you give citing NVIDIA seem to be more inline with what I'm suggesting. The shared memory fabric allows fast offload the the extremely specialized processing units.
No, actually, despite some confusing block diagrams, RTX and tensor cores are actually part of the SM engine, not "standalone units" in a different part of the chip. They live right next to the shaders.
There is no "fabric" there, it is execution units working on registers just like any other instruction.
Ok, but the AVX instructions that do trigger the cooldown may be the ones you wish to use, and the statement is still valid. I never mentioned AMD.
> not "standalone units"
I didn't say they were standalone, in fact, the opposite. It just depends on what you want to call a boundary. The tighter integration is what is driving these special purpose units but up until now they've not affected package TDP so heavily.
Again, it's not that AVX instructions inherently must be clocked down for. It is an implementation detail of Intel's processor. AMD already has different boost behavior for AVX2 and will probably have different boost behavior for AVX-512 when they implement that.
> It just depends on what you want to call a boundary
What you were saying is that they are units attached over some "fabric", which isn't the case.
> The shared memory fabric allows fast offload the the extremely specialized processing units.
They aren't "multiple processors that share memory", it is an execution unit within the Streaming Multiprocessor. It is no different from an AVX instruction, they are driven by registers, not by a fabric and not by memory.
At risk of reigniting the old Bulldozer "what is a core really anyway", AVX and tensor cores don't really represent an independent core, they are an execution unit within a larger core. They can't operate independently without the other execution resources provided by the core.
Having 2x 512b units crushes the clocks. That is a factor of vector width (and having multiple units), not the instruction set itself. Half-rate AVX-512 would in all probability run at the same speeds as AVX2, and allows significantly more flexibility than "mere" 256-bit AVX2.
That's perhaps something that is not clear from Linus - whether he is railing against the instruction set, or the implementation in Skylake-SP, or both (I think it's both). I would probably agree that Skylake-SP takes it too far with dual AVX-512 units, but the instruction set itself ("stop inventing magic instructions"), yeah he's way off base.
I am really hoping that one of these Zen generations, AMD includes half-rate AVX-512 like they had half-rate AVX2 in Zen1/Zen+. That really seems like the best of both worlds, you get the flexibility of AVX-512 but without the die area or the power issues. If not, well, Rocket Lake is coming end of this year/early next year and that should bring AVX-512 to consumer platform desktop (as it's in Tiger Lake and Rocket Lake will be a Tiger Lake backport) and will probably have only a single AVX-512 unit since it's mobile-derived.
Cloudflare had complaints as well. Even if you just have a fraction of your code running 512, you take a big hit.
Cloudflare talks about a 0.3% 512 workload (ie, a cryto library for SSL) blowing up their throughput.
"It is equivalent to giving up on two cores, for nothing."
https://blog.cloudflare.com/on-the-dangers-of-intels-frequen...
Do you / Intel GUARANTEE no frequency scaling to handle 512?
Intel quotes these specs on throughput and seem to do it using full frequency for FP. But if you have 512 in your mix (ie, 1-2% 512) do you really get that speed. Cloudflare talks about frequency dropping to 1.4Ghz
If in doubt, search on "daniel lemire" "simd".
Downclocking because of overheating and insane waitcycles to get out of this mess which is AVX512, plus poor HW support (only on insecure Intel CPU's which need to be probed at runtime) makes it an esoteric extension. AVX256 is good enough for most purposes and is not that insane.
[1] https://lemire.me/blog/2018/06/07/vectorizing-random-number-...
[2] https://lemire.me/blog/2018/09/07/avx-512-when-and-how-to-us...
[3] https://lemire.me/blog/2018/08/15/the-dangers-of-avx-512-thr...
> Lack of adoption of AVX-512 would be Intel's own issue. If they wanted it to succeed, they would try to put it on more chips than just the few lines you mentioned. As it currently stands, Intel is setting up AVX-512 to fail by only putting it in so few products.
It will be fun to revisit these sorts of comments next year when AMD releases Zen4 based processors that support it. Like with raytracing, I suspect people's tune will change when the red man does it instead of the blue man or green man ;)
(it may even be in Zen3, that "50% improved FP performance" sure is interesting if you figure that's about what you would get from ideal-case speedups of AVX-512... I suspect they are moving to dual 256b units and at that point you might as well make them gangable into a 512b unit. If they don't go there with Zen3 it is an obvious low-hanging fruit for Zen4.)
However, his particular career includes a stint at Transmeta, so he has probably thought about processor design a bit more than the average tenured software engineer. And it might also account for some of his bitterness toward Intel.
The extra registers have to be there even if the execution unit is half size. The context switching overhead grows, which is something kernel developers care about. And then with all these variants there's a push to have a single binary support them all which complicates the code too. Meanwhile the world is moving toward parallelism which is going to mean more thread creation and context switching.
Linus does have valid concerns here.
Someone is missing the boat: custom ISA is the future.
Name two
Did you miss the part about custom instructions?
Custom extensions aren't being made to beat benchmarks.
Custom ISA are 99% of the case useless marketing porn for hardware provider.
The reality is that most Linux distributions in 2020 are still binary compiled with SSE41 instructions (maybe AVX at best). Instructions that shipped decades ago.
AVX2 and AVX512 being barely supported in some HPC parts and the libc itself.
SIMD is good, but fragmentation of instruction set is much more of a problem in most case.
Nobody will dare to support your fancy new instruction if it covers 2% of the CPU used world wide and make their software a pain in the butt to distribute.
On that Linus is perfectly right.
The nice thing about JITCs is:
1. They run parallel to the application so can use those spare cores your program isn't using.
2. Especially true as they run a lot during startup when your app is probably single threaded anyway, even for something like a web server where it's going to be multi-threaded soon.
3. Upgrading the VM upgrades the compiler, so new instructions can be used almost immediately, as long as your software is tested/runs on the newest VMs.
JVM has historically tried to auto-vectorise everything and not been all that good at it, but now it's getting vector extensions that generalise to newer instruction sets fairly well, so we might start to see more HPC done in higher level languages.
That's theory. Practice is that JIT-compiled language generally performs even worst than compiled language with an outdated instructions set.
And there is reasons to that, compiler passes are expensive, including vectorization, and you specially do not want anything expensive running in your critical JIT passes.
Some language like Julia still do it but to the price of a very long compiling and starting time.
Something that has never been acceptable over JVM or in Nodejs.
Generally JIT outperforms AOT for a language like Java by about 20%. For Scala, it's even more. For a more dynamic language like JavaScript or Ruby you don't even try to AOT it at all the difference is so huge.
Even in languages that are normally AOT compiled, profile guided optimisations make a big difference. For C++ I've seen figures in the 15%+ range. That's a lot! Most projects don't use PGO though because it's a pain to deploy.
C2 does auto-vectorisation and it's not a slow compiler. It runs in parallel with the app, it's not a problem. The difficulty with auto-vectorisation isn't how much time you have to compile it, it's more that matching all the different loop constructs to the instructions is a very complicated problem that ends up needing tons of special cases in the compiler. It's a problem of code complexity rather than runtime performance. And it's opaque to the user: if they change their code structure a bit and it's no longer recognised as a vectorizable template, it'll stop being vectorised and performance drops off a cliff. You can't really see that though as a developer ahead of time because it's all just "best effort" optimisation. Game devs in particular hate that and would rather have a compiler error than silently bailed out optimisation passes.
That's why Java is switching to explicit vectorisation. It's not about JITs not having enough CPU time, it's about giving the developer reliable and predictable performance even in the face of arbitrary refactorings.
I personally have a problem when compromises are made for the general case to satisfy a special case, especially when the special case motivations appear to be bound mostly in marketing reasons. Even more so if the special case is well-known to adversely impact performance in the general case.
Before I saw this post, I didn't even know ARM had some complex javascript instruction. Contrast that with - You can't watch or read a review for any current gen Intel server product without being presented with at least one AVX benchmark. Is this marketing or is it practical engineering? For how many applications is AVX availability actually a hard constraint?
"Nvidia is worth more than Intel for the first time in history. Nvidia is now worth more than Intel, according to the NASDAQ. The GPU company has finally topped the CPU company's market cap (the total value of its outstanding shares) by $251bn..."
Well no surprise there.
On the one hand, Intel (especially early Intel employees) should be thanked, profusely, for giving us PC history as we know it today.
On the other hand, we should seek to honestly recognize what Intel has become today.
A mega-corporation, driven by corporate mentality, which always seeks to maximize profits for their shareholders at the expense of all other virtues.
Keep in mind I am not criticizing Intel's employees -- only the marching orders that come from the top down.
But, as far as I can tell, Intel, as we know it, will not be around in another 30 years.
The future of semiconductors is in the following areas:
1) Simple, non-proprietary instruction sets (RISC-V and others)
2) Transparent (or as transparent as possible) and publicly auditable engineering and manufacturing processes
3) Conscientous companies that place virtue first, and are not driven by maximimizing profits for shareholders
So on the one hand, Intel is to be thanked, lauded, praised for its role, especially its early role in the beginning of the PC revolution, but on the other hand, I don't see Intel existing as company more than 30 years from now...
Although, at that point in time, Intel's past role will always remain important and relevant -- to future students of early computer history...
Commoditization of architecture (ARM seems to be in the lead by RISC-V might go up) plus separation between chip design houses and manufacture fabs seems to be more efficient market solution. Where it leaves Intel?
Arguably, what I've written applies equally to most of the large companies at this point in time in the semiconductor/CPU space...
I'm 1000% behind him on this point. Fix that first and fix it well. Then get onto fancier stuff that i'll never ever code against.
Anything where the execution traces diverge significantly requires more execution cores, to sequence them separately. For example serving multiple web pages in parallel.
Anything where the executation traces are identical or near-identical across many repetitions in lockstep, is fine with one execution core and SIMD ALUs. For example dense or block matrix arithmetic.
Modern GPUs are a hybrid of both ideas. Shared execution cores and SIMD ALUs because of high repetition of identical logic, but still many execution cores (but fewer than ALUs) so there is some amount of decoupled parallelism as well, running different tasks at the same time.
SIMD is much more primitive. It is more along the scale of "Multiple these 4 numbers by 5" and that's it. A GPU kernel can have branching, conditionals, etc. SIMD is limited to one operation on a small set of data (512 bytes for AVX-512).
A modern GPU isn't limited to running one block of code at a time (on n cores). It will schedule a few different blocks of code independently - up to the number of execution units on the GPU, multiplied by the number of latency-hiding time slots per execution unit.
SIMD is still very useful elsewhere of course, but there is a lot of software for which it's not very relevant.
works great if you have, literally, one instruction you want to apply to multiple data.
not relevant if "parallelism" means "thread 1 is doing foo, thread 2 is doing bar, and thread 3 is doing baz".
I assume the Fab/process issues, Spectre issues, and AVX-512 are each handled by different teams.
Please Rust, please!
https://docs.microsoft.com/en-us/dotnet/api/system.numerics....
https://www.khronos.org/sycl/
We've smacked pretty hard into the wall of how fast we can make CPUs by miniaturizing components, and the low-hanging fruit now is parallelization, predictive execution, and the whole host of ways to do more per clock cycle, not speed up clock cycles. But that flies in the face of the traditional embarassingly-serial pattern of the x86 instruction set and computing environment. Much of what Intel's doing these days is trying to open up opportunities for people to change tools to speed up code without having to boil the ocean by throwing the whole serial-instruction model completely out the window (even though, increasingly, code written to that instruction set is really operating like a language that is emulated by the underlying parallel-and-predictive CPU hardware).
I think what Linus calls "regular code" isn't going to move chips and we've wrung all the cheap optimizations out of that critical path (and, depending on the details of what is meant by "regular," such code is increasingly going to ram up against an efficiency ceiling unless it can be moved to a model of "Prepare work to do in parallel, execute in parallel, merge results and present to UI").
But wouldn't something more useful than AVX-512 sell more chips?
Massively parallel floating point workloads on the CPU are a modest niche compared to other ways Intel could have spent engineering effort and chip area, like not ruining the performance of just about everything else with security vulnerabilities.
https://www.nextgenvec.org/ https://arxiv.org/abs/1902.02816
I helped put an early deep network application into production that used one particular generation of vector instructions and it was clear everyone was traumatized at the process of implemented it to use those vector instructions and would never do it again.
We later bought servers that had instructions that we weren't using, but we were so focused on getting the product out the door (which we did) that we left those performance gains on the table.
The consequence of the "new instructions every two years" route is that end users don't really experience the performance benefits engineered into the chips. The vast majority of software development organizations are more concerned about support costs than they are about getting the last bit of performance out. (Also, end users are already tired of 200-megabyte binaries, long start times, and other performance decrements that come from high-complexity libraries that contain a huge number of code paths for different CPUs)
Many working assembly programmers who are good at SIMD love the Intel approach, maybe because it keeps them in business. People are doing very cool things in the parsing space (e.g. it is shocking how slow it is to parse ascii-formatted numbers, JSON, XML, RDF, ...) with SIMD instructions.
However, the old tradition in vector computing is the variable-sized vector unit like what the ILLIAC had or the old Crays, the vector accelerator for the IBM 3090, etc. ARM has the scalable vector instructions which are organized that way.
ARM's Helium had a nice middle ground of having "small, medium and large" implementations available which all run the same code. You might get higher peak peformance on Intel's road, but higher realized performance with ARM.
Do the binaries really get into the 200 MB range from different code paths, or is that due to data?
Considering the rather complex dynamic of Intel's downclocking with AVX512 workloads, that would be pretty interesting.
Similarly the AWS VM CPUs were clocked at 2.0Ghz, which is pretty low and possibly does not even downlock in that configuration, were tests run on desktop CPUs that go to 5Ghz regularly on non-AVX512 workloads? How is the advantage there?
We originally ran experiments on a multisocket lab server which should have a faster clock, but I don't have a record of the specs. That server did fine for SSE to AVX2 conversion, but didn't support AVX512 so we ultimately used Google Cloud VMs.
Folks, anyone who knows what Linux is also knows who Linus Torvalds is.
Here is a link to the thread directly from Linus explaining his position on MMX/SSE, AVX/AVX2, AVX512. His complaints on AVX512 are both about fragmentation and regressing from the learned lessons from their prior generations. And he also looks to NEON and SVE2 and suggesting ARM looks seems much saner to him:
"So just as a bystander, I'm looking at AVX512, and I'm looking at SVE2, and I'm going "AVX512 really is nasty, isn't it"?""
https://www.realworldtech.com/forum/?threadid=193189&curpost...