HN Explorer

Ask HN: Embedded NN Libraries

1 points by earthnail ↗ HN
Dear HN,

Do you have recommendations for libraries for running NNs on embedded devices?

- must compile using a C or C++ compiler

- must be efficient (runtime speed on embedded obv matters)

- must support conv, lstm

- easy export from PyTorch

- bonus points for rfft / torch.stft support

Any advice on how to choose/evaluate libraries would also be welcome. So far I've had a look at tinygrad, ONNX, and ExecuTorch. It seems ONNX is the most supported, stable option, but I'd very much welcome advice from people who have actual experience running NNs on embedded devices.

2 comments

[ 2.5 ms ] story [ 10.7 ms ] thread
fwsgonzo
I've embedded llama.cpp for a while now. There may be a point where it's not embeddable anymore, but for now it works great. My experience with the others is don't bother. The build systems and dependency management is truly awful.

Also, I'm obviously talking about porting here. You will need to tailor the code to your needs, thinking of it as a library. You cannot run an unmodified program.

earthnail
Can you elaborate a bit more why you chose llama.cpp? From their docs:

> "The main goal of llama.cpp is to run the LLaMA model using 4-bit integer quantization on a MacBook"

I'm not interested in running an LLM on embedded devices; the NN I want to deploy is really tiny and used for audio signal processing. It extracts information out of a small chunk of audio. The code needs to run on various embedded ARM chips.

What would be the advantage of using llama.cpp for this?