Can this really be everything publically known about the ANE? Sounds hard to believe, I would have thought someone would have reverse engineered something about it by now.
Unfortunately not. You can only use the Neural Engine through Core ML at the moment.
There currently is no public framework for programming the ANE. There are several private, undocumented frameworks but obviously we cannot use them as Apple rejects apps that use private frameworks.
(Perhaps in the future Apple will provide a public version of AppleNeuralEngine.framework.)"
No those initials were already taken by Atlassian Software Systems. They seem to have lodged the paperwork with that name in 2002, and to have dropped it later on (they rather went with TEAM when going IPO in 2015), but back in 2010 when I applied, there was a book (collection of news articles) in the waiting room for candidates titled “Atlassian Software Systems”.
It's really terrible that Apple markets this as the next big thing but forgets to include detailed documentation so people have to experiment and figure out what works...
Part Apple's docs haven't been great for a while, part that's just how they roll, and part trying (like most everyone) to figure out what their strategy is going to be in a post GPT4 world [0].
[0] Persist with their own models running locally, how much to integrate with rest of the OS and maintain privacy moral ground, that sort of thing.
It's not that simple. If you have a model that actually does something useful (e.g. not just doing matmul & conv2d) your model will fail to run on ANE and, instead, the device will move it over to CPU/GPU and turn your iPhone into a heater. I literally had to continually wipe down my iPhone with a wet towel to keep it from overheating so I could build, ct.convert, run, and debug a model I was working on. Apple doesn't document how to keep operations on ANE. A model created by coremltools may run on either CPU, GPU, ANE, but you don't get to choose. And, if you don't know what you're doing and naively build a model, you will likely run on CPU/GPU only. If your batch is too large, too small, if you need to transpose tensors, if you need to expand mismatched tensors to matmul them together, if your model has an IF branch, or a loop, if you breathe the wrong way, your model silently falls off ANE. But you don't know what caused it. You have to open Netron and guess. Also, it may run on ANE on one device, but not another. There's no documentation from Apple about any of this. So, no, you do not simply "use CoreML to program it".
It's probably more nuanced than what everyone else is saying. The Neural Engine compute block can change pretty significantly from one chip generation to the next, and instead of exposing the unstable raw capabilities, they use CoreML as an abstraction to keep the changes out of sight. Is it rather annoying? Yes, but it's a more stable method to keep everyone using the Neural Engine from having their software break regularly.
I don't know for sure, but things like text recognition (Live Text) or object recognition in Photos (Visual Look Up) are obvious candidates.
I think neural engine is absolutely key to Apple's strategy. They want people to buy expensive devices and they don't want to process user data on their servers.
Users get privacy. Apple gets money. It's a pretty coherent business model.
It’s not a subsidy. It’s a pricing structure for a commercial transaction. Fundamentally a business can not just give out free compute. In the long run the user of computation needs to pay for it. It’s a question of whether people feel more satisfied paying for it in a lump sum bundled with a device or through a subscription plan on the cloud.
For frequently, on demand, low latency applications I would suspect that people will always be happier running the computations locally.
> Apple also gets users to subsidize the cost of compute indefinitely (by buying the expensive phone), rather than using their servers.
Subsidizing the smart speaker hardware and running everything on their servers hasn't worked out well for Alexa and Google Assistant.
> The Alexa division is part of the "Worldwide Digital" group along with Amazon Prime video, and Business Insider says that division lost $3 billion in just the first quarter of 2022, with "the vast majority" of the losses blamed on Alexa. That is apparently double the losses of any other division, and the report says the hardware team is on pace to lose $10 billion this year. It sounds like Amazon is tired of burning through all that cash.
Google expressed basically identical problems with the Google Assistant business model last month. There's an inability to monetize the simple voice commands most consumers actually want to make, and all of Google's attempts to monetize assistants with display ads and company partnerships haven't worked. With the product sucking up server time and being a big money loser, Google responded just like Amazon by cutting resources to the division.
On the downside, we have to acknowledge that it is hugely inefficient for everyone to own expensive hardware that has to sit idle most of the time because it would otherwise drain the battery.
Where low latency is not an absolute necessity, the economic pull of the cloud will be tremendous, especially if mobile networks become ubiquitous and fast.
That's a weak argument. Lots of hardware sits idle in the cloud as well. And on your phone its not expensive. In fact, the $/tflop is cheaper on phone than in the cloud – cloud has to deal with all kinds of complexity that you assume away in your local single-tenant phone context.
I wouldn't be so sure. A quick web search brings up average server utilisation numbers for large-scale cloud providers between 45% and 65%. That's probably an order of magnitude or two higher than what you could do on a mobile device without absolutely annihilating the battery.
> On the downside, we have to acknowledge that it is hugely inefficient for everyone to own expensive hardware that has to sit idle most of the time
Expensive hardware like millions of 5G/6G modems, stations and fiber optic cables nessesary to send vast quantity of photos and videos to the cloud for analysis?
ait is more expensive to build out bandwidth than to give everyone compute
Also average car is unused 95/% of the time, so by this principle everyone should take a bus
>ait is more expensive to build out bandwidth than to give everyone compute
I'm not sure about that. Storing all data on all devices (not just user data but also the pretrained models) means higher data transfer volumes.
On the other hand, syncing data doesn't require low latency and a lot of the data can be transferred over cheaper landline connections rather than mobile towers.
Right now though the default appears to be to upload everything to the cloud right away, regardless of where the data is processed.
And there's also communication and media streaming, which consumes the vast majority of network resources.
Net net I'm not sure whether on-device vs cloud processing will make a big difference for networking costs one way or another.
>Also average car is unused 95/% of the time, so by this principle everyone should take a bus
Yes, and it is hugely inefficient. People who own a car (I don't) are doing it for the benefits it has, not because it's efficient.
I hope it's not disrespectful to point this out, less than 24 hours after his passing, but I don't think Gordon would object to my pointing out that Moore's Law has a finite length. Some have argued it expired up to 13 years ago; Moore himself predicted another 2 years or so.
I built an always on local OCR system that used ML on CPU/GPU a few years ago and I can say with confidence it doesn’t use much. We literally scanned your entire screen every two seconds and it used less than 1% in total, and this was before CoreML which is far more efficient. I think it’s FUD that it is that significant.
RF transmission and reception have been bumping smack up against physics limits for a long time. Regardless of whether or not Moore's Law is Dead, it is an extremely good bet that there is more juice left in compute than in RF. For instance, even the "specialized" ML hardware of today has a comfortable factor of 10 in further possible optimization and probably considerably more. Also, due to the fact that these innovations tend to compound, there isn't just more juice, there is e to the power of more juice left.
"Moore's Law is Dead" is a bit of a joke in the hardware space because a staggering number of smart people have made staggeringly wrong predictions of this nature (well, typically correct in a narrow sense and wrong in a broad sense). Jim Keller frequently talks about this and has a convincing theory as to why it happens: the industry is full of specialists who are all chasing one particular S-curve and fully understand to the point of conservatively believing in another S-curve or two. Inevitably, this gives them the impression that Moore's Law has just a few years of gas left -- however, it's actually a consequence of limits on human communication, curiosity, and cognition that determines the number of promising S-curves a typical engineer is thinking about. It's not a true evaluation of the supply of additional S-curves waiting in the wings. There's a limit somewhere out there but it's not really "in sight."
> typically correct in a narrow sense and wrong in a broad sense
I'm not quite sure what people think as noone seems to be stating it explicitly, but getting the impression some commenters here have their own personal definition of Moore's Law. The actual law is narrowly defined: if you're looking to discuss things related to it "in a broad sense" cool, but that isn't what I was referring to above. I was referring to Moore's Law.
To be clear, Moore's Law is about processor manufacturing tolerances & states some pretty concrete predictions for rates of progress in the manufacturing process. It doesn't state anything related to compute (i.e what uses those processors can be put to), it's purely about the physical properties thereof.
From my understanding (which seems to differ from OP), the benefit of running models on ANE is for much lower power usage, not necessarily speed.
Apple’s always-on Hey Siri wake word detection uses ANE for minimal battery life.
I’m not sure but I think android’s “Now Playing” feature (the always-on Shazam thing that shows the names of songs playing around you on your lock screen ) also uses Edge TPU for similar reasons.
To my knowledge this is mostly used by internal tools, though a number of common 3rd party apis (qr code scan) use hw acceleration under the hood. Internally there is a ton of ML running on the device. The most obvious is touch screen and inputs and the camera.
3rd party developers have acres to this via CoreML, but unless latency is critical it is usually easier to develop and run ml on the cloud. For camera apps using ml, this chip is going to be used either explicitly or implicitly.
If you think about it a capacitive touch sensor provides a noisy Grayscale image and the goal is to detect and classify blobs as touch gestures as quickly and accurately as possible. Since it is running at all times and latency really burns the UX. Consequences this has always been done on a HW accelerator.
>Machine learning is used to help the iPad’s software distinguish between a user accidentally pressing their palm against the screen while drawing with the Apple Pencil, and an intentional press meant to provide an input.
I know you probably didn't mean this, but in case anyone is confused ANE is not doing 15 trillion matrix multiplications per second. It is doing 15 trillion scalar operations in order to multiply a much smaller number of matrices.
I can imagine some thing like Siri running on device much more effectively against local content. The cynic in me doesn’t want to hope too much for cloudless services like this, but one can hope.
No but the first party users should not consume all the compute on the chip. The bigger the margin the better for the device. The other aspect of this is speed and power consumption (battery life is a top 3 phone feature across pretty much all consumers).
I got one recently and generally think the phone is garbage, but the OCR built into pictures is really something else. I took a photo of a label for a barcode when I couldnt see it myself but could get my hand nearby, it was at an odd angle, but when I pressed my finger to the text I was interested in the phone captured it immediately, highlighted it, and I copied it nice as you please.
You are looking at the wrong top-line number as the “constraint”. My understanding is that the power (and die area) involved in moving the data around is a far greater bottleneck than the actual matmuls themselves.
Your iPhone won’t analyze your photos unless it is plugged in. It’s power intensive primarily due to the amount of data that needs to move to classify faces in a photo. Image files are quite large these days, and even if you downscale them you still need to load them from disk and decompress them in the first place.
The scarce resource on the iPhone is RAM, not compute. The Camera’s image pipeline uses so much memory that the phone is effectively unable to multi-task when the camera is open.
Newer phones are running on faster process nodes, so that is always a boost, but the phones also have more RAM for bigger cameras.
Apple’s unified memory architecture is a key advantage, because they can ship less RAM if it can be shared across the CPU/GPU, but all mobile SoCs have unified memory for this reason, so it’s not Apple’s secret sauce, just part of it.
I think the primary constraint will be how many models can fit in memory at once, not how many can run at once.
They also don’t often all run at once: you only use the TTS model when Siri speaks, so Siri only loads it when it’s needed. But, in that case, then the latency of loading the model from disk is a concern…
In short, performance for ML is really a small subset of performance more generally, and Apple takes performance seriously because efficiency is their game. That is how good battery life pops out on the other side.
Arkit makes use of it on the phone, there’s plane detection and classification, image and object detection, segmentation for people occlusion, probably more behind the scenes.
I find it a little frustrating we aren’t using the built in capabilities of iphones more in our company, i still kinda think apple tech is kind of a pariah in some circles, so we have to run with stuff that runs on cloud that costs us money over, heaven forbid something you could run on an iphone
While it’s true you can run some parts of a machine learning stack, Apple still lags way behind nvidia due to not having CUDA support. It makes a huge difference in both speed and quality of output.
https://www.rewind.ai/ is one that I know of. it only works on apple Silicon but their software records the shit out of everything you do on your computer, so you never have that moment when you half remember something you did the past. just pull up rewind and find it!
This is the answer. CoreML gives you an abstraction over different generations and sizes of underlying NPU.
You might not want the abstraction, but love it or hate it, that’s kind of the Apple way.
It will be very interesting to see what their next chips look like since we’re getting to the point where HW designs will reflect the rise of the, uh, transformers.
Anyone done any work on using a model for transcription on the local device using the ANE? I've heard it kills the battery. Having to transcribe voice in the cloud is a serious impediment to end to end encryption for certain applications.
My understanding is that the CPU and GPU both have DMA to the memory at some incredible speed since it’s all on the same chip. Does the ANE have that same DMA speed and latency?
I believe so as it’s used by adobe amongst others, this was from a convo with an adobe engineer gushing about the UMA/DMA and what an improvement it was from the fans whirring jet engine end of Intel era.
I can’t find any documentation about it though just everyone working under that assumption.
100 comments
[ 3.3 ms ] story [ 211 ms ] thread"> Can I program the ANE directly?
Unfortunately not. You can only use the Neural Engine through Core ML at the moment.
There currently is no public framework for programming the ANE. There are several private, undocumented frameworks but obviously we cannot use them as Apple rejects apps that use private frameworks.
(Perhaps in the future Apple will provide a public version of AppleNeuralEngine.framework.)"
The last part links to this bunch of headers:
https://github.com/nst/iOS-Runtime-Headers/tree/master/Priva...
So might it be more accurate to say you can program it directly, but won't end up with something that can be distributed on the app store?
https://github.com/geohot/tinygrad/tree/master/accel/ane
Anyone disappointed, here be full details on everything.
https://github.com/eiln/ane
https://github.com/eiln/anecc
https://github.com/AsahiLinux/m1n1/pull/296/files
Just sayin'.
Great guys.
https://youtu.be/VfyUbuFoiBU
[0] Persist with their own models running locally, how much to integrate with rest of the OS and maintain privacy moral ground, that sort of thing.
What kind of things are taking advantage of this right now? It's gotta be more than just Face ID right?
What's my laptop likely to be doing with that?
I think neural engine is absolutely key to Apple's strategy. They want people to buy expensive devices and they don't want to process user data on their servers.
Users get privacy. Apple gets money. It's a pretty coherent business model.
> Users get privacy. Apple gets money.
Apple also gets users to subsidize the cost of compute indefinitely (by buying the expensive phone), rather than using their servers.
Subsidizing the smart speaker hardware and running everything on their servers hasn't worked out well for Alexa and Google Assistant.
> The Alexa division is part of the "Worldwide Digital" group along with Amazon Prime video, and Business Insider says that division lost $3 billion in just the first quarter of 2022, with "the vast majority" of the losses blamed on Alexa. That is apparently double the losses of any other division, and the report says the hardware team is on pace to lose $10 billion this year. It sounds like Amazon is tired of burning through all that cash.
Google expressed basically identical problems with the Google Assistant business model last month. There's an inability to monetize the simple voice commands most consumers actually want to make, and all of Google's attempts to monetize assistants with display ads and company partnerships haven't worked. With the product sucking up server time and being a big money loser, Google responded just like Amazon by cutting resources to the division.
https://arstechnica.com/gadgets/2022/11/amazon-alexa-is-a-co...
On the downside, we have to acknowledge that it is hugely inefficient for everyone to own expensive hardware that has to sit idle most of the time because it would otherwise drain the battery.
Where low latency is not an absolute necessity, the economic pull of the cloud will be tremendous, especially if mobile networks become ubiquitous and fast.
And it's not even particularly wasteful to produce. While there are lots of devices the amount of materials in each chip is incredibly small.
In terms of financial cost the material cost is so low that most chip vendors include features in low end chips that are disabled but shipped anyway.
The battery is a key part as well. Doing a lot of on-device processing requires a bigger battery and/or more frequent battery replacements.
Expensive hardware like millions of 5G/6G modems, stations and fiber optic cables nessesary to send vast quantity of photos and videos to the cloud for analysis?
ait is more expensive to build out bandwidth than to give everyone compute
Also average car is unused 95/% of the time, so by this principle everyone should take a bus
I'm not sure about that. Storing all data on all devices (not just user data but also the pretrained models) means higher data transfer volumes.
On the other hand, syncing data doesn't require low latency and a lot of the data can be transferred over cheaper landline connections rather than mobile towers.
Right now though the default appears to be to upload everything to the cloud right away, regardless of where the data is processed.
And there's also communication and media streaming, which consumes the vast majority of network resources.
Net net I'm not sure whether on-device vs cloud processing will make a big difference for networking costs one way or another.
>Also average car is unused 95/% of the time, so by this principle everyone should take a bus
Yes, and it is hugely inefficient. People who own a car (I don't) are doing it for the benefits it has, not because it's efficient.
That’s exactly right.
Back when dictation was done in the cloud, I could dictate all day on my iPhone no problem.
Now that it's on-device it kills my battery in a couple of hours.
The latency is absolutely improved, and continuous dictation (not stopping every 30s) is a godsend.
But it does absolutely destroy your battery life.
"Moore's Law is Dead" is a bit of a joke in the hardware space because a staggering number of smart people have made staggeringly wrong predictions of this nature (well, typically correct in a narrow sense and wrong in a broad sense). Jim Keller frequently talks about this and has a convincing theory as to why it happens: the industry is full of specialists who are all chasing one particular S-curve and fully understand to the point of conservatively believing in another S-curve or two. Inevitably, this gives them the impression that Moore's Law has just a few years of gas left -- however, it's actually a consequence of limits on human communication, curiosity, and cognition that determines the number of promising S-curves a typical engineer is thinking about. It's not a true evaluation of the supply of additional S-curves waiting in the wings. There's a limit somewhere out there but it's not really "in sight."
I'm not quite sure what people think as noone seems to be stating it explicitly, but getting the impression some commenters here have their own personal definition of Moore's Law. The actual law is narrowly defined: if you're looking to discuss things related to it "in a broad sense" cool, but that isn't what I was referring to above. I was referring to Moore's Law.
To be clear, Moore's Law is about processor manufacturing tolerances & states some pretty concrete predictions for rates of progress in the manufacturing process. It doesn't state anything related to compute (i.e what uses those processors can be put to), it's purely about the physical properties thereof.
Apple’s always-on Hey Siri wake word detection uses ANE for minimal battery life.
I’m not sure but I think android’s “Now Playing” feature (the always-on Shazam thing that shows the names of songs playing around you on your lock screen ) also uses Edge TPU for similar reasons.
Gesture recognition has been around since before the neural engine shipped and doesn't appear to be different on devices with or without it.
>Machine learning is used to help the iPad’s software distinguish between a user accidentally pressing their palm against the screen while drawing with the Apple Pencil, and an intentional press meant to provide an input.
https://yugalchoubisa.medium.com/how-machine-learning-and-ar...
- Biometrics (Face ID and Touch ID)
- Image analysis (face matching, aesthetic evaluations, etc)
- Text to speech and speech to text (smaller models on device, used for privacy/latency/reliability)
- Small ad-hoc models like Raise to Speak on Apple Watch, the Hey Siri detector (https://machinelearning.apple.com/research/hey-siri)
These things have been in phones for 5 years now and have been used from day one
I could only find a blurry YouTube video of the instruction manual for an old old heater in my house.
I paused the video on the bit I needed the guy had zoomed into and was able to copy and paste the text that I could barely read into a notes doc.
There’s no one splashy thing just lots of little quality of life improvements.
Your iPhone won’t analyze your photos unless it is plugged in. It’s power intensive primarily due to the amount of data that needs to move to classify faces in a photo. Image files are quite large these days, and even if you downscale them you still need to load them from disk and decompress them in the first place.
The scarce resource on the iPhone is RAM, not compute. The Camera’s image pipeline uses so much memory that the phone is effectively unable to multi-task when the camera is open.
Newer phones are running on faster process nodes, so that is always a boost, but the phones also have more RAM for bigger cameras.
Apple’s unified memory architecture is a key advantage, because they can ship less RAM if it can be shared across the CPU/GPU, but all mobile SoCs have unified memory for this reason, so it’s not Apple’s secret sauce, just part of it.
I think the primary constraint will be how many models can fit in memory at once, not how many can run at once.
They also don’t often all run at once: you only use the TTS model when Siri speaks, so Siri only loads it when it’s needed. But, in that case, then the latency of loading the model from disk is a concern…
In short, performance for ML is really a small subset of performance more generally, and Apple takes performance seriously because efficiency is their game. That is how good battery life pops out on the other side.
I find it a little frustrating we aren’t using the built in capabilities of iphones more in our company, i still kinda think apple tech is kind of a pariah in some circles, so we have to run with stuff that runs on cloud that costs us money over, heaven forbid something you could run on an iphone
I know the Swift version of Stable Diffusion from Hugging Face uses this too.
Or they are keeping it obscure for commercial reasons?
Or just not very competent/don't care?
Seems weird having these amazing chips and only blunt tools
Directly exposing the ANE wouldn't make much sense, as it's an IP block that changes between generations in incompatible ways.
You might not want the abstraction, but love it or hate it, that’s kind of the Apple way.
It will be very interesting to see what their next chips look like since we’re getting to the point where HW designs will reflect the rise of the, uh, transformers.
I can’t find any documentation about it though just everyone working under that assumption.