I think Raspberry lost the magic of the older Pis, they lost that sense of purpose. They basically created a niche with the first Pis, now they're just jumping into segments that others created and are already filled to the brim with perhaps even more qualified competition.
Are they seeing a worthwhile niche for the tinkerers (or businesses?) who want to run local LLMs with middling performance but still need full set of GPIOs in a small package? Maybe. But maybe this is just Raspberry jumping on the bandwagon.
I don't blame them for looking to expand into new segments, the business needs to survive. But these efforts just look a bit aimless to me. I "blame" them for not having another "Raspberry Pi moment".
P.S. I can maybe see Frigate and similar solutions driving the adoption for these, like they boosted Coral TPU sales. Not sure if that's enough of a push to make it successful. The hat just doesn't have any of the unique value proposition that kickstarted the Raspberry wave.
Nah, they released products better suited to what people were already using Pis for.
The Picos are great for the smaller stuff, new Pis are great for bigger stuff, and old Pis and Zeros are still available. They've innovated around their segment.
The AI stuff is just an expression of that. People are doing AI on Pi5s and this is just a way to make that better.
People have for quite some time been using Googles Tensor chip to accelerate AI workloads on the Pi. I doubt that anyone runs Llms on Pis but stuff like security cameras with object detection...
> I think Raspberry lost the magic of the older Pis, they lost that sense of purpose. They basically created a niche with the first Pis, now they're just jumping into segments that others created and are already filled to the brim with perhaps even more qualified competition.
I don't think you will find anything on the market enabling you to create your own audiophile quality AMP, DAC, or AMP+DAC for a pretty attractive price except a Pi 3/4/5 with a HifiBerry (https://www.hifiberry.com/) HAT.
IMO this is a consequence of Raspberry Pi going for-profit and IPOing. Now they are incentivised to chase the same hype trains as every other public tech company. I can't see them having another "Raspberry Pi moment", those are too risky now.
That said, more options at the (relatively speaking) low end of the AI hardware market probably isn't a bad thing. I'm not particularly an AI enthusiast generally, but if it is going to infest everything anyway, then at least I would like a decent ecosystem for running local models.
I think this is a miss on what the Pi is: an experiment. Sure, it stood on the shoulders of other SFF boards that came before it - but it broke into the general computing landscape targeting makers and builders. If the AI hat doesn't work out, so be it. The use cases for this type of hat may yet to be seen. On one hand it may feel shortsighted to bringing hardware to market with no explicit use case, but that's part of the Pi brand.
As someone else mentioned: if the hat could efficiently be leveraged with the YOLO models on Frigate for a low volume camera setup that could be a nice niche use case for it.
Either way I hope the RPi org keeps dropping things like this and letting the users sort out the use cases with their dollars.
In regard to their niche, their niche is a ridiculously well-documented ecosystem for SBCs. Want to do something with your RPi? You can find it on Google, and the LLM of your choice is probably trained to give you the answer on how to do it. If you're just tinkering or getting a POC ready, that's a big help.
Of course, if you're in the business of hardware prototyping, and have a set of libraries and programs you know you're going to work with, you don't need to care as much.
My biggest issue is the lack of really good cases. There are all those fancy peripherals you can buy, but it's really hard to find simple case that works without overheating and no cables sticking out on all four sides.
This is why I keep buying 3B+ and Zero 2 W and not any of the newer versions...it's much more in keeping with the relatively low cost board with GPIO and reasonable compute. It's kind of the last one they made that does what I kind of expected out of a Raspberry Pi at a reasonable price point. If I needed more compute I would have skipped the travesty that is ARM and just bought an x86 system.
I feel like if RPI doubled down heavily into education, they would be in a much better spot. They really could never win on price in the long run. But having a bit of K-12 and university budgets going to RPIs every year, especially during the "teach the kids programming" era, would I think make them a much healthier business.
I still remember the email I got telling me that they were going to upgrade the RAM of the 256Mb Model B I ordered and that I would receive a brand new model with 512Mb for no extra cost. Hard to believe that was nearly 14 years ago.
Was very helpful in me learning Linux. The only alternative I had at the time was a few old Pentium 4 machines, which were very noisy and my parents didn't like me leaving turned on for a long time.
The original Pis are still for sale, are cheap, and still do everything you need. This doesn't conflict with an expanded product line. The whole reason for Pi is still GPIO plus general purpose computing. AI is now a part of general purpose computing, so it only makes sense to adopt it too.
The things you can do locally with AI now are amazing. For several years there's been multiple open source products that can do both audio and visual processing locally using AI models. Local-only Home Assistant is almost equivalent to Siri. The more things you throw at it, the more computing power it needs (especially for low latency), and that's where the dedicated GPUs/NPUs (previously ASICs) are needed. And consider the expanded use cases; drones and robots can now navigate the world autonomously using a $150 SoC and some software.
Case closed. And that's extremely slow to begin with, the Pi 5 only gets what, a 32 bit bus? Laughable performance for a purpose built ASIC that costs more than the Pi itself.
> In my testing, Hailo's hailo-rpi5-examples were not yet updated for this new HAT, and even if I specified the Hailo 10H manually, model files would not load
Laughable levels of support too.
As another datapoint, I've recently managed to get the 8L working natively on Ubuntu 24 with ROS, but only after significant shenanigans involving recompiling the kernel module and building their library for python 3.12 that Hailo for some reason does not provide outside 3.11. They only support the Pi OS (like anyone would use that in prod) and even that is very spotty. Like, why would you not target the most popular robotics distro for an AI accelerator? Who else is gonna buy these things exactly?
What’s the current state of the art in low power wake word and speech to text? Has anyone written a blog post on this?
I was able to run a speech to text on my old Pixel 4 but it’s a bit flaky (the background process loses the audio device occasionally). I just want to take some wake word and then send everything to remote LLM and then get back text that I do TTS on.
Is there any usefulness with the small large language models, outside perhaps embeddings and learning?
I fail to see the use-case on a Pi. For learning you can have access to much better hardware for cheaper. Perhaps you can use it as a slow and expensive embedding machine, but why?
A natural language based smart home interface, perhaps?
Tiny LLMs are pretty much useless as general purpose workhorses, but where they shine is when you finetune them for a very specific application.
(In general this is applicable across the board, where if you have a single, specific usecase and can prepare appropriate training data, then you can often fine-tune a smaller model to match the performance of a general purpose model that is 10x its size.)
It's more about demonstrating what's possible on a Pi than expecting GPT-4 level performance. It's designed for LLMs that specialize in tiny, incredibly specific tasks. Like, "What's the weather in my ant farm?" ;)
The vision processing boost is notable, but not enough to justify the price over existing HATs. The lack of reliable mixed-mode functionality and sparse software support are significant red flags.
(This reply generated by an LLM smaller than 8GB, for ants, using the article and comment as context).
You can run some models pretty decently using CPU inference only, things like Gemma 3 that are built for exactly that use case or some tiny speech to text models via llama.cpp that I have tested out (not so good). Although not the best for "heavy" tasks, if you just need a decent text generator that can produce more or less sensible, generic output you are good to go.
"For example, the Hailo 10H is advertised as being used for a Fujitsu demo of automatic shrink detection for a self-checkout."
... why though? CV in software is good enough for this application and we've already been doing it forever (see also: Everseen). Now we're just wasting silicon.
As an edge computing enthusiast, this feels like a meaningful leap for the Raspberry Pi ecosystem. Having a low-power inference accelerator baked into the platform opens up a lot of practical local AI use cases without dragging in the cloud. It’s still early, but this is the right direction for real edge workloads.
At this moment my two questions for these things are:
1. Can I run a local LLM that allows me to control Home Assistant with natural language? Some basic stuff like timers, to do/shopping lists etc would be nice etc.
2. Can I run object/person detection on local video streams?
I want some AI stuff, but I want it local.
Looks like the answer for this one is: Meh. It can do point 2, but it's not the best option.
> Can I run a local LLM that allows me to control Home Assistant with natural language? Some basic stuff like timers, to do/shopping lists etc would be nice etc.
No. Get the larger PI recommended by the article.
Quote from the article:
> So power holds it back, but the 8 gigs of RAM holds back the LLM use case (vs just running on the Pi's CPU) the most. The Pi 5 can be bought in up to a 16 GB configuration. That's as much as you get in decent consumer graphics cards1.
> Because of that, many quantized medium-size models target 10-12 GB of RAM usage (leaving space for context, which eats up another 2+ GB of RAM).
…
> 8 GB of RAM is useful, but it's not quite enough to give this HAT an advantage over just paying for the bigger 16GB Pi with more RAM, which will be more flexible and run models faster.
The model specs shown for this device in the article are small, and not fit for purpose even for the relatively trivial use case you mentioned.
I mean, look, lots of people have lots of opinions about this (many of them wrong); it’s cheap, you can buy one and try… but, look. The OP really gave it a shot, and results were kind of shit. The article is pretty clear.
Don’t bother.
You want a device with more memory to mess around with for what you want to do.
can't wait to not be able to buy it, and also for it to be more expensive than a mini-computer
I buy a raspberry pi because I need a small workhorse - I understand adding RAM for local LLMs, but it would be like a raspberry pi with a GPU, why do i need it when a normal mini machine will have more ram, more compute capacity and better specs for cheaper?
a lot of people buy rpis because they are the only reasonable option for connectivity with power. I'm not sure what other devices you can get that have gpio and mipi connectivity with the ability to (potentially) run vlms and llms on them.
I daresay they could charge more than a comparably specced computer (if they don't already) and they would still be a viable purchase.
In the UK I've never seen the hailo hats (which are quite old BTW) advertised for LLMs. The presented usecase has been object detection from lots of video cameras in realtime.
They seem very fast and I certainly want to use that kind of thing in my house and garden - spotting when foxes and cats arrive and dig up my compost pit, or if people come over when I'm away to water the plants etc.
[edit: I've just seen the updated version in Pimonori and it does claim usefulness for LLMs but also for VLMs and I suspect this is the best way to use it].
That said, perhaps there is a niche for slow LLM inference for non-interactive use.
For example, if you use LLMs to triage your emails in the background, you don't care about latency. You just need the throughput to be high enough to handle the load.
This looks pretty nice for what it is. However, the RAM is a bit oversized for the vast majority of applications that will run on this, which is giving a misleading impression of what it is useful for.
I once tried to run a segmentation model based on a vision transformer on a PC and that model used somewhere around 1 GB for the parameters and several gigabytes for the KV cache and it was almost entirely compute bound. You couldn't run that type of model on previous AI accelerators because they only supported model sizes in the megabytes range.
68 comments
[ 2.8 ms ] story [ 78.3 ms ] thread8GB RAM for AI on a Pi sounds underwhelming even from the headline
Are they seeing a worthwhile niche for the tinkerers (or businesses?) who want to run local LLMs with middling performance but still need full set of GPIOs in a small package? Maybe. But maybe this is just Raspberry jumping on the bandwagon.
I don't blame them for looking to expand into new segments, the business needs to survive. But these efforts just look a bit aimless to me. I "blame" them for not having another "Raspberry Pi moment".
P.S. I can maybe see Frigate and similar solutions driving the adoption for these, like they boosted Coral TPU sales. Not sure if that's enough of a push to make it successful. The hat just doesn't have any of the unique value proposition that kickstarted the Raspberry wave.
The Picos are great for the smaller stuff, new Pis are great for bigger stuff, and old Pis and Zeros are still available. They've innovated around their segment.
The AI stuff is just an expression of that. People are doing AI on Pi5s and this is just a way to make that better.
- I can boot it w/o having to learn about custom U-Boot implementations
- I, as a consumer or small business, can buy
- Can not only buy today but also still buy in 2 years
- Doesn't cost a small fortune
- Can be tugged away behind TVs and other small niches
I don't think you will find anything on the market enabling you to create your own audiophile quality AMP, DAC, or AMP+DAC for a pretty attractive price except a Pi 3/4/5 with a HifiBerry (https://www.hifiberry.com/) HAT.
That said, more options at the (relatively speaking) low end of the AI hardware market probably isn't a bad thing. I'm not particularly an AI enthusiast generally, but if it is going to infest everything anyway, then at least I would like a decent ecosystem for running local models.
As someone else mentioned: if the hat could efficiently be leveraged with the YOLO models on Frigate for a low volume camera setup that could be a nice niche use case for it.
Either way I hope the RPi org keeps dropping things like this and letting the users sort out the use cases with their dollars.
In regard to their niche, their niche is a ridiculously well-documented ecosystem for SBCs. Want to do something with your RPi? You can find it on Google, and the LLM of your choice is probably trained to give you the answer on how to do it. If you're just tinkering or getting a POC ready, that's a big help.
Of course, if you're in the business of hardware prototyping, and have a set of libraries and programs you know you're going to work with, you don't need to care as much.
Chromebooks did what RPI should have done.
Was very helpful in me learning Linux. The only alternative I had at the time was a few old Pentium 4 machines, which were very noisy and my parents didn't like me leaving turned on for a long time.
The things you can do locally with AI now are amazing. For several years there's been multiple open source products that can do both audio and visual processing locally using AI models. Local-only Home Assistant is almost equivalent to Siri. The more things you throw at it, the more computing power it needs (especially for low latency), and that's where the dedicated GPUs/NPUs (previously ASICs) are needed. And consider the expanded use cases; drones and robots can now navigate the world autonomously using a $150 SoC and some software.
Although the op is not wrong, maybe their decisions are data driven and pay off?
My rpi3 (that's been running since 2019) died last year and I was able to buy another and just plug in the SD card.
Case closed. And that's extremely slow to begin with, the Pi 5 only gets what, a 32 bit bus? Laughable performance for a purpose built ASIC that costs more than the Pi itself.
> In my testing, Hailo's hailo-rpi5-examples were not yet updated for this new HAT, and even if I specified the Hailo 10H manually, model files would not load
Laughable levels of support too.
As another datapoint, I've recently managed to get the 8L working natively on Ubuntu 24 with ROS, but only after significant shenanigans involving recompiling the kernel module and building their library for python 3.12 that Hailo for some reason does not provide outside 3.11. They only support the Pi OS (like anyone would use that in prod) and even that is very spotty. Like, why would you not target the most popular robotics distro for an AI accelerator? Who else is gonna buy these things exactly?
I was able to run a speech to text on my old Pixel 4 but it’s a bit flaky (the background process loses the audio device occasionally). I just want to take some wake word and then send everything to remote LLM and then get back text that I do TTS on.
I fail to see the use-case on a Pi. For learning you can have access to much better hardware for cheaper. Perhaps you can use it as a slow and expensive embedding machine, but why?
Tiny LLMs are pretty much useless as general purpose workhorses, but where they shine is when you finetune them for a very specific application.
(In general this is applicable across the board, where if you have a single, specific usecase and can prepare appropriate training data, then you can often fine-tune a smaller model to match the performance of a general purpose model that is 10x its size.)
The vision processing boost is notable, but not enough to justify the price over existing HATs. The lack of reliable mixed-mode functionality and sparse software support are significant red flags.
(This reply generated by an LLM smaller than 8GB, for ants, using the article and comment as context).
... why though? CV in software is good enough for this application and we've already been doing it forever (see also: Everseen). Now we're just wasting silicon.
1. Can I run a local LLM that allows me to control Home Assistant with natural language? Some basic stuff like timers, to do/shopping lists etc would be nice etc.
2. Can I run object/person detection on local video streams?
I want some AI stuff, but I want it local.
Looks like the answer for this one is: Meh. It can do point 2, but it's not the best option.
No. Get the larger PI recommended by the article.
Quote from the article:
> So power holds it back, but the 8 gigs of RAM holds back the LLM use case (vs just running on the Pi's CPU) the most. The Pi 5 can be bought in up to a 16 GB configuration. That's as much as you get in decent consumer graphics cards1.
> Because of that, many quantized medium-size models target 10-12 GB of RAM usage (leaving space for context, which eats up another 2+ GB of RAM).
…
> 8 GB of RAM is useful, but it's not quite enough to give this HAT an advantage over just paying for the bigger 16GB Pi with more RAM, which will be more flexible and run models faster.
The model specs shown for this device in the article are small, and not fit for purpose even for the relatively trivial use case you mentioned.
I mean, look, lots of people have lots of opinions about this (many of them wrong); it’s cheap, you can buy one and try… but, look. The OP really gave it a shot, and results were kind of shit. The article is pretty clear.
Don’t bother.
You want a device with more memory to mess around with for what you want to do.
I buy a raspberry pi because I need a small workhorse - I understand adding RAM for local LLMs, but it would be like a raspberry pi with a GPU, why do i need it when a normal mini machine will have more ram, more compute capacity and better specs for cheaper?
I daresay they could charge more than a comparably specced computer (if they don't already) and they would still be a viable purchase.
They seem very fast and I certainly want to use that kind of thing in my house and garden - spotting when foxes and cats arrive and dig up my compost pit, or if people come over when I'm away to water the plants etc.
[edit: I've just seen the updated version in Pimonori and it does claim usefulness for LLMs but also for VLMs and I suspect this is the best way to use it].
That said, perhaps there is a niche for slow LLM inference for non-interactive use.
For example, if you use LLMs to triage your emails in the background, you don't care about latency. You just need the throughput to be high enough to handle the load.
[1] https://rubikpi.ai/
I once tried to run a segmentation model based on a vision transformer on a PC and that model used somewhere around 1 GB for the parameters and several gigabytes for the KV cache and it was almost entirely compute bound. You couldn't run that type of model on previous AI accelerators because they only supported model sizes in the megabytes range.
That's also limited to 8Gb RAM so again you might be better off with a larger 16Gb Pi and using the CPU but at least the space is heating up.
With a lot of this stuff it seems to come down to how good the software support is. Raspberry Pis generally beat everything else for that.
A NPU that adds to price but underperforms a rasp cpu?
You get SBC with 32gb ram…
Nevermind the whole minipc ecosystem which will crush this