The article talks about ai development workloads and how there's little need for a low power low performance accelerator compared to a high end GPU but I think the point is end user workloads.
Software is coming with AI integrations. IDEs, spreadsheets, word processors, graphics programs. These absolutely will benefit from low power ai acceleration. Sure if i want best in class acceleration I'll buy the damn 300w GPU. But really I just want some autocompletion without burning my lap or requiring cloud connectivity.
I always feel that accelerators are in an awkward spot, because at some point with a sufficiently powerful CPU most workloads finish quickly enough.
And let's be real, even lowly Core i3s these days are overkill for most people; let alone mid tier offerings like Core i5s which are the most popular for commercial users.
Isn’t a big advantage of an accelerator is that its more optimized to do this sort of work than a cpu? This npu is a tiny dot on the Meteor Lake die while the cpus take up a whole tile and accounts for much of the power usage of the soc at any given time.
That’s the idea but it looks like this might not pull it off unless Intel aggressively gives away optimized libraries. If it’s too hard to get to the point where it’s actually competitive, most developers are going to use the faster iGPU.
You hit the nail on the head. Software, libraries, dev tools, profiling resources, mindshare... And for Intel, a good deal of maintained interest. For those of us who used and programmed SHAVE cores, to see everything slowly close down behind openvino, no apparent interest from Intel in years, the writing was on the wall. Repeatedly, they taught us: use nothing from Intel except main-mainline x86_64.
What’s sad is that I remember the same lesson from Itanium: to preserve a small revenue stream from their compiler and optimized library business, they made almost every benchmark even worse at a time when they most needed businesses to be comfortable switching.
It makes systems "feel" way more responsive. It's a huge quality of life increase. Apple Silicon's user/product story has basically changed how people view this stuff forever.
4 TFLOPs is significantly faster than your CPU. Even 1.35 TFLOPS represents a reasonable edge. The question should be why Intel added another processor instead of extending AVX.
It's better than the TOPS nonsense where accuracy is completely thrown out the window. The quantization schemes used by llama.cpp do not operate directly on integers. They dequantize back to floats at runtime.
You have to forgive me, but these NPUs were specifically designed for inference. You can indeed use them as general purpose accelerators, but I think you will struggle to find an application for them.
That is the big "if", to be sure. I spent three years working for Intel after they acquired Vertex.AI, trying to help build that software story, and it was rough going; we didn't accomplish much, and it doesn't sound like they're using any of it.
I was impressed by their _CPU_ kernels in OpenVINO which I studied very closely after discovering I cannot beat their performance even on kernels specialized for my specific tensor shapes. Turns out the reason I can't beat them is because they jitted their kernels with xbyak, and specialized them for those shapes, too. They also obviously knew exactly how to maximize the ALU utilization, and counted every cycle, something that's very hard to do unless you specialize _just_ on CPU performance and have access to the internal documentation. Top notch work. But that team used to be in Russia back then, in Nizhny Novgorod, IDK if it's still a part of Intel.
Yes the _CPU_ perf of OpenVINO is a thing of beauty. Debuggability is a nightmare story but if it works you're GOLD. The GPU/Myriad story was very clunky last times I tried.
I hope some Intel PHB reads this and purely stochastically makes the right decision for the first time in his/her life. It seems pretty existential to be competitive at selling shovels during a gold rush.
There's a Matt Pharr-level story to tell here if you feel like it one day. I'm also looking at what the very cool maxas authors became. Intel seems like a dark hole of blackness and holeness for acquired tech these days.
1024 fp16 macs is pretty good but the 128b vector datapath is weak sauce. on the other hand 2MB SRAM is legit. i wonder how many tiles (i don't think it's in the post).
I think the main advantage of Movidius was its memory architecture. They had to disclose the HW arch for some sort of a tender in the EU which is how I stumbled upon their HW documentation years ago. Basically IIRC even though you’re right and 128 bit is weak sauce, the strength of that arch was that the memory was much “closer” to the cores and it was partitioned and accessed by multiple cores at the same time, boosting overall available bandwidth. The weakness of it was that it was (IMO) overly complicated for no good reason, and imposed a rather inflexible programmability model which if their software layer didn’t do certain things, you couldn’t do them at all. Which was a problem before transformers, because models tended to use a greater variety of ops
"Instead of building a custom command processor, Intel uses Movidius’s 32-bit LEON microcontrollers. LEON use the SPARC instruction set and run a real time operating system."
Your Intel core now runs two ISAs and three OSs. Managment engine runs Minix (and used to use the ARC isa).
this is very common. lots of companies around (e.g. cadence) that sell SIPs as synthesizable RTL for easy integration into chip designs (and lots of hardware companies buy these SIPs).
It's always weird to see Intel using "foreign" ISAs, but not too surprising for something they acquired. The ME's use of SPARC and ARC before that is more unusual; they could've at least used a 386/486-class core, or even the i860/960 if they wanted something RISC.
A 32-bit Linux-capable microcontroller at that. I still keep my Galileo board. Will probably assemble it into a picture frame and run a heterogeneous cluster with it and some of the other boards I accumulated over the years.
It must be a bit depressing for Intel engineers that they are forced to keep themselves tied to the ever extending x86 ISA and can't explore all the other ISAs they use (or invented). The 860/960 generated a lot of buzz back then and there was some pretty capable hardware using it.
Fortunately, with open-source becoming dominant in the data center and the relative ease of bootstrapping a new ISA on top of it, we will see some extra diversity in this space, but I don't think Intel will abandon x86 anytime soon (and "soon" is in almost geological scales here).
Intel could just as well become a pure fab and give up on designing processors. Switching to another ISA might tie them up for years before they can deliver state-of-the-art processors again. Intel can't afford that after the recent year's delays with switching to new nodes.
Intel has been thinking in detail about a streamlined x86 architecture that would ditch all of the cruft: 64-bit only, no memory segmentation, no real mode.
More than two. It has x86_64 on the main CPU cores, x86 on the ME, SPARC on this NPU and whatever the iGPU ISA is called. And I wouldn't be surprised if there's an ARM microcontroller or three in there as well.
So how do you program these things? I tried looking into the AMD equivalent and the development story was very primitive. Is the future going to be programming against a different opaque binary blob for each NPU one wants to support?
I am sure Intel will improve the shortcomings in the next generation, so it is a good start. But I wonder how this will compare with Qualcomm's new PC CPU (Snapdragon X Elite) which also has its own NPU.
> Even economy class plane seats have power outlets these days. Hopefully Intel will iterate on both hardware and software to expand NPU use cases going forward.
I can totally see myself doing inference on my phone, but arguably - this is not intel's territory.
Not the best time for testing OpenVINO at least, they plan to release NPU support in 2024.1 as I understand, current version 2024.0 has very limited NPU support if any.
55 comments
[ 5.3 ms ] story [ 116 ms ] threadSoftware is coming with AI integrations. IDEs, spreadsheets, word processors, graphics programs. These absolutely will benefit from low power ai acceleration. Sure if i want best in class acceleration I'll buy the damn 300w GPU. But really I just want some autocompletion without burning my lap or requiring cloud connectivity.
And let's be real, even lowly Core i3s these days are overkill for most people; let alone mid tier offerings like Core i5s which are the most popular for commercial users.
- Computational photography features like Night Mode and Smart HDR
- Real-time video processing for effects and stabilization
- Object and action recognition in videos
- Faster on-device speech recognition and natural language understanding for Siri
- Real-time translation, object detection, and scene understanding in AR
They would be, if it weren't for the egregious insanity of software bloat, particularly the web-based stuff.
The registers in this CPU are all quantumly entwined, with Intel having updates signed.
Thus, Intel can one-way update one or all CPUs remotely. No Faraday cage will protect you.
Further, this CPU has access to caches on all other cores, so Intel can dynamically inhibit, or modify code as they please.
We'll find this out in 2027, I dare not say more, they'll track me through time and afg#.<NO CARRIER>
It seems that efficient cores don't support much of AVX extensions. You have to turn those cores off if you want to use AVX512, for example.
You have to forgive me, but these NPUs were specifically designed for inference. You can indeed use them as general purpose accelerators, but I think you will struggle to find an application for them.
Your Intel core now runs two ISAs and three OSs. Managment engine runs Minix (and used to use the ARC isa).
This is Gibson, the 4 RPi Zero W's running Docker Swarm: https://twitter.com/0xDEADBEEFCAFE/status/172986300775544462...
You did it, you hacked the Gibson!
Please tell me it has a slick 90s neon low poly 3D interface.
Fortunately, with open-source becoming dominant in the data center and the relative ease of bootstrapping a new ISA on top of it, we will see some extra diversity in this space, but I don't think Intel will abandon x86 anytime soon (and "soon" is in almost geological scales here).
https://www.intel.com/content/www/us/en/developer/articles/t...
I can totally see myself doing inference on my phone, but arguably - this is not intel's territory.
“Do you guys not have power outlets? Yeah, everybody’s got a power outlet.”
What a time to be alive.