For single batch inference of anything remotely LLM you'll hit the memory bound way before FLOPs, so I haven't actually looked at FLOPs much. For raw performance GPU is certainly better. ANE is more energy efficient,…
coremltools is the only way to run on ANE, so less of a trick and more of a requirement. The tricks are more around optimizing for the hardware capabilities/constraints. For instance: - conv2d is faster than linear (see…
What do you mean by less wide? The main bottleneck for transformers is memory bandwidth. ANE has a much lower ceiling than CPU/GPU (yes, despite unified memory). Chunking is actually beneficial as long as all the chunks…
Not a public follow-up but the iOS 17 speech-to-text model has a clever approach to KV caching that works within the ANE’s constraints (fixed size inputs). I wrote about it here[0] but the gist is you can have a fixed…
I bet these can all run on ANE. I’ve run gpt2-xl 1.5B on ANE [1] and WhisperKit [2] also runs larger models on it. The smaller ones (1.1B and below) will be usably fast and with quantization I suspect the 3B one will be…
Has perplexity fallen out of favor? I didn't see it mentioned anywhere. I tried using lm-eval for the 2B model but the results seem wrong (46.1288).
The jump was due to them fixing a bug. There’s a footnote about it on the bottom of page 5. In the Discord, they mentioned a TinyLLaMa v2, presumably that would have this bug (and another bug, footnote page 4) fixed.
MobileVLM [1] is another recent small multimodal model. They trained their own 1.4B/2.7B LLaMa from scratch using RedPajama and Vicuna instead of leveraging Phi-2. The papers only have one common benchmark (GQA,…
> Multi-device: Operations can run on any of the supported devices (currently, the CPU and GPU). Probably reading into this too much, but is this hinting at future Neural Engine support? It’d be nice to access that…
Yeah, I believe it is. You trade off speed for lower power usage and CPU. 8 tokens/sec is usable though.
You can do autoregressive decoding with KV caching on the Neural Engine. You have to make a bit of a trade off and use fixed size inputs [1] but the speed up over no caching is meaningful. There's a Whisper…
I've been looking at these files too and have another data point for unilm.bundle being the new text prediction. If you take an iOS simulator, turn off "Settings > General > Keyboard > Predictive", reboot it and then…
Sorry for the confusing phrasing about STT vs TTS. I'm not familiar with cases where you would use something like this 'at the edge' instead of say a laptop. I was thinking maybe some sort of offline setup with a…
What's an example use case for something like this? "At the edge" makes me think offline but are you generating audio at anything faster than real time in that case? Would be curious to see an even lower cost/lower…
Also not an expert, but I believe it is a little bit of both for inference. If you are generating token-by-token naively, you do need to pay the n^2 cost since every token must attend to all other tokens. Generating a…
For single batch inference of anything remotely LLM you'll hit the memory bound way before FLOPs, so I haven't actually looked at FLOPs much. For raw performance GPU is certainly better. ANE is more energy efficient,…
coremltools is the only way to run on ANE, so less of a trick and more of a requirement. The tricks are more around optimizing for the hardware capabilities/constraints. For instance: - conv2d is faster than linear (see…
What do you mean by less wide? The main bottleneck for transformers is memory bandwidth. ANE has a much lower ceiling than CPU/GPU (yes, despite unified memory). Chunking is actually beneficial as long as all the chunks…
Not a public follow-up but the iOS 17 speech-to-text model has a clever approach to KV caching that works within the ANE’s constraints (fixed size inputs). I wrote about it here[0] but the gist is you can have a fixed…
I bet these can all run on ANE. I’ve run gpt2-xl 1.5B on ANE [1] and WhisperKit [2] also runs larger models on it. The smaller ones (1.1B and below) will be usably fast and with quantization I suspect the 3B one will be…
Has perplexity fallen out of favor? I didn't see it mentioned anywhere. I tried using lm-eval for the 2B model but the results seem wrong (46.1288).
The jump was due to them fixing a bug. There’s a footnote about it on the bottom of page 5. In the Discord, they mentioned a TinyLLaMa v2, presumably that would have this bug (and another bug, footnote page 4) fixed.
MobileVLM [1] is another recent small multimodal model. They trained their own 1.4B/2.7B LLaMa from scratch using RedPajama and Vicuna instead of leveraging Phi-2. The papers only have one common benchmark (GQA,…
> Multi-device: Operations can run on any of the supported devices (currently, the CPU and GPU). Probably reading into this too much, but is this hinting at future Neural Engine support? It’d be nice to access that…
Yeah, I believe it is. You trade off speed for lower power usage and CPU. 8 tokens/sec is usable though.
You can do autoregressive decoding with KV caching on the Neural Engine. You have to make a bit of a trade off and use fixed size inputs [1] but the speed up over no caching is meaningful. There's a Whisper…
I've been looking at these files too and have another data point for unilm.bundle being the new text prediction. If you take an iOS simulator, turn off "Settings > General > Keyboard > Predictive", reboot it and then…
Sorry for the confusing phrasing about STT vs TTS. I'm not familiar with cases where you would use something like this 'at the edge' instead of say a laptop. I was thinking maybe some sort of offline setup with a…
What's an example use case for something like this? "At the edge" makes me think offline but are you generating audio at anything faster than real time in that case? Would be curious to see an even lower cost/lower…
Also not an expert, but I believe it is a little bit of both for inference. If you are generating token-by-token naively, you do need to pay the n^2 cost since every token must attend to all other tokens. Generating a…