Disappointing on the NPU. I have found it's a point where industry wide improvement is necessary. People talk tokens/sec, model sizes, what formats are supported... But I rarely see an objective accuracy comparison. I repeatedly see that AI models are resilient to errors and reduced precision which is what allows the 1 bit quantization and whatnot.
But at a certain point I guess it just breaks? And they need an objective "I gave these tokens, I got out those tokens". But I guess that would need an objective gold standard ground truth that's maybe hard to come by.
The even more confounding factor is there are specific builds provided by every vendor of these Cix P1 systems: Radxa, Orange Pi, Minisforum, now MetaComputing... it is painful to try to sort it out, as someone who knows where to look.
I couldn't imagine recommending any of these boards to people who aren't already SBC tinkerers.
I was also onboard until he got to the NPU downsides. I don't care about use for an LLM, but I would like to see the ability to run smallish ONNX models generated from a classical ML workflow. Not only is a GPU overkill for the tasks I'm considering, but I'm also concerned that unattended GPUs out on the edge will be repurposed for something else (video games, crypto mining, or just straight up ganked)
There are a couple outfits making M.2 AI accelerators. Recently I noticed this one: DeepX DX-M1M 25 TOPS (INT8) M.2 module from Radxa[1]: https://radxa.com/products/aicore/dx-m1m
If you're in the business of selling unbundled edge accelerators, you're strongly incentivized to modularize your NPU software stack for arbitrary hosts, which increases the likelihood that it actually works, and for more than one particular kernel.
If I had an embedded AI use case, this is something I'd look at hard.
So, this is slightly off topic, but out of curiousity, what are NPUs good for right this very second? What software uses them? What would this NPU be able to run if it was in fact accessible?
This is an honest, neutral question, and it's specifically about what can concretely be done with them right now. Their theoretical use is clear to me. I'm explicitly asking only about their practical use, in the present time.
(One of the reasons I am asking is I am wondering if this is a classic case of the hardware running too far ahead of the actual needs and the result is hardware that badly mismatches the actual needs, e.g., an "NPU" that blazingly accelerates a 100 million parameter model because that was "large" when someone wrote the specs down, but is uselessly small in practice. Sometimes this sort of thing happens. However I'm still honestly interested just in what can be done with them right now.)
I love that OrangePi is making good hardware, but after my experience with the OrangePi 5 Max, I won’t be buying more hardware from them again. The device is largely useless due to a lack of software support. This also happened with the MangoPi MQ-Pro. I’ll just stick with RPi. I may not get as much hardware for the money, but the software support is fantastic.
I was planning to build a NAS from OPi 5 to minimise power consumption, but ended up going for a Zen 3 Ryzen CPU and having zero regrets. The savings are miniscule and would not justify the costs.
> The device is largely useless due to a lack of software support.
I think everyone considering an SBC should be warned that none of these are going to be supported by upstream in the way a cheap Intel or AMD desktop will be.
Even the Raspberry Pi 5, one of the most well supported of the SBCs, is still getting trickles of mainline support.
The trend of buying SBCs for general purpose compute is declining, thankfully, as more people come to realize that these are not the best options for general purpose computing.
On a related note: I pulled my pinebook pro out of a drawer this week, and spent an hour or so trying to figure out why the factory os couldn’t pull updates.
I guess manjaro just abandoned arm entirely. The options are armbian (probably the pragmatic choice, but fsck systemd), or openbsd (no video acceleration because the drivers are gpl for some dumb reason).
This sort of thing is less likely to happen to rpi, but it’s also getting pretty frustrating at this point.
> device is largely useless due to a lack of software support.
Came looking for this. It's the pitfall of 99% of hardware projects. They get a great team of hardware engineer, they go through the maddening of actually producing a thing (which is crazy complex) at scale, economically viable (hopefully), logistic hurdles including tax worldwide, tariffs, etc... only to have only people on their team be able to build and run a Hello World example.
To be fair even big player, e.g. NVIDIA, sucks at that too. Sure they have their GPU and CUDA but if you look at the "small" things like Jetson everybody I met told me the same thing, great hardware, unusable because the stack worked once when shipped then wasn't maintained.
Every time there's a new discussion of some arm board, I compare the price / features / power use with the geekom n100 SBC I picked up awhile back.
As far as I can tell, the OrangePi 6 remains distinctly uncompetitive with SBCs based on low-end intel chips.
- Orange pi consumes much more power (despite being an arm CPU)
- A bit faster on some benchmarks, a bit slower on others
- Intel SBC is about 60% the price, and comes with case + storage
- Intel SBC runs mainline linux and everything has working drivers
I don't understand why many say that RPi software/firmware support is 'fantastic'. Maybe it used to be in the beginning compared to other chips and boards, but right now it's a bit above average: they ignore many things which is out of their control/can't debug and fix (as in Wi-Fi chip firmware).
The main issue was that they forked UBoot and did not release their modifications, making it hard to run anything other than their Armbian fork. They forked Armbian a long time ago and kind of hacked things together rather than adding support for their HW to Armbian. After a while I gave up running anothing other than their releases, I had good experiences with the Orange Pi 3 and 5.
But it's really uncool that they don't release their UBoot build! Lame!
Their WiFi chip (Dragon something brand), had a bug where the WiFI beacon frames had incorrect element orderings, causing inconsistent results with some clients. Overall their Wifi was pure garbage. But apart from the Wifi it's robust stuff.
Looks like the SoC (CIX P1) has Cortex-A720/A520 cores which are Armv9.2, nice.
I've still been on the hunt for a cheap Arm board with a Armv8.3+ or Arvm9.0+ SoC for OSDev stuff, but it's hard to find them in hobbyist price range (this board included, $700-900 USD from what I see).
The NVIDIA Jetson Orin Nanos looked good but unfortunately SWD/JTAG is disabled unless you pay for the $2k model...
Unfortunately only available atm for extremely high prices. I'd like to pick some up to create a ceph cluster (with 1x 18tb hdd osd per node in an 8 node cluster with 4+2 erasure coding)
This seems to be an overkill for most of my workloads that require an SBC.
I would choose Jetson for anything computationally intensive, as Orange Pi 6 Plus's NPU is not even utilized due to lack of software support.
For other workloads, this one seems a bit too large in terms of formfactor and power consumption, and older RK3588 should still be sufficient
But I just don't get... everything, I don't get the org, I don't get the users on hn, I'm like skinner in the 'no the kids are wrong' meme.
It's a lambda. It's a cheap, plug in, ssh, forget. And it's bloody wonderful.
If you buy a 1 or 2 off ebay, ok maybe a 3.
After that? Get a damn computer.
Want more bandwidth on the rj45? Get a computer.
Want faster usb? Get a computer.
Want ssd? Get a computer
Want a retro computing device? Get a computer.
Want a computer experience?
Etc etc etc, i don't need to labour this.
Want something that will sit there, have ssh and run python scripts for years without a reboot? Spend 20 quid on ebay.
People demanded faster horses. And the raspi org, for some, damn fool, reason, tried to give them.
There are people bemoaning the fact that raspberry pi's aren't able to run LLM's. And will then, without irony, complain that the prices are too high. For the love of God, raspi org, stop listening to dickheads on the Internet. Stop paying youtubers to shill. Stop and focus.
You have to go in with your eyes open wth SBCs. If you have a specific task for it and you can see that it either already supports it or all the required software is there and it just needs to be gathered, then they can be great gadgets.
Often they can go their entire lifespan without some hardware feature being usable because of lack of software.
The blunt truth is that someone has to make that software, and you can't expect someone to make it for you. They may make it for you, and that's great, but really if you want a feature supported, it either has to already be supported, or you have to make the support.
It will be interesting to see if AI gets to the point that more people are capable of developing their own resources. It's a hard task and a lot of devices means the hackers are spread thin. It would be nice to see more people able to meaningfully contribute.
Excellent! There is an OrangePi Zero 3W. That means my radxa zero tv-computer now has some competition. It is sad that rasspberry pi abandoned the small, zero computer. Will keep the OrangePi Zero 3W in mind next time I need to cobble together a new tv computer.
I bought a NanoPi R6C in the past in the hope that it's going to be a nice mini pc to run all my containers with super low power usage or router. But the software was bad, really bad. I found https://github.com/Joshua-Riek/ubuntu-rockchip/ , it was godsend but still had some shortcomings. after 2 years, it's bit stable but I just keep it around as a backup route to access my homelab incase the main machines go down.
It should be noted that the CIX P1(this board's SoC) has ongoing efforts to be upstreamed. Last I checked, the GPU drivers were still not available(due to them not supporting ACPI? I may be wrong on this) and power draw being weird and stuck at 10-15ish watts. It seems like this blog confirms nothing has changed on those 2 points.
With that being said, CIX and their main board partner, Radxa, have been open with the UEFI.
I am not an expert in low-level environments such as the kernel or the UEFI, but if these tidbits sound interesting I would encourage anyone who is to look further into the CIX P1. To my untrained eyes, CIX looks like a company that is working towards a desktop/laptop chip with real UEFI/ACPI support. I look forward to the day it is polished up a bit.
> and you end up diving far more into boot chains, vendor GPU blobs and inference runtimes than you ever intended.
Yep, I'll pass. I'm done dealing with that kind of crap that is spread like a nasty STD through the ARM world. I'm sticking with x64 unless/until ARM gets this crap together.
> lspci is a bit more revealing, especially because you get to see where the dual 5GbE setup and Wi-Fi controller are placed–each seems to get its own PCI bridge:
That's how PCIe works. A PCIe port - both upstream and downstream - is a "PCI bridge". The link is one bus. A switch chip's "interior" is another bus. The next links are each their own bus again. One per port. There's no switch here, bus 0 ( / 30 / 60)is "in" the CPU, each port is it's own bus.
The more interesting thing is the PCI domain, the first 4 digits:
This generally (caveat emptor) means the ports aren't handled in some common PCIe subsystem, rather each port is independently connected to the CPU crossbar. The ports may also not be able to access each other, or non-transparent mapping rules apply.
Doesn't have to, though; it might be due to some technicality, driver bug, misunderstanding, whatever else.
38 comments
[ 3.1 ms ] story [ 59.1 ms ] threadBut at a certain point I guess it just breaks? And they need an objective "I gave these tokens, I got out those tokens". But I guess that would need an objective gold standard ground truth that's maybe hard to come by.
I couldn't imagine recommending any of these boards to people who aren't already SBC tinkerers.
There are some perplexity comparison numbers for the previous gen - Orange pi 5 in link below.
Bit of a mixed bag, but doesn't seem catastrophic across the board. Some models are showing minimal perplexity loss at Q8...
https://github.com/invisiofficial/rk-llama.cpp/blob/rknpu2/g...
If you're in the business of selling unbundled edge accelerators, you're strongly incentivized to modularize your NPU software stack for arbitrary hosts, which increases the likelihood that it actually works, and for more than one particular kernel.
If I had an embedded AI use case, this is something I'd look at hard.
This is an honest, neutral question, and it's specifically about what can concretely be done with them right now. Their theoretical use is clear to me. I'm explicitly asking only about their practical use, in the present time.
(One of the reasons I am asking is I am wondering if this is a classic case of the hardware running too far ahead of the actual needs and the result is hardware that badly mismatches the actual needs, e.g., an "NPU" that blazingly accelerates a 100 million parameter model because that was "large" when someone wrote the specs down, but is uselessly small in practice. Sometimes this sort of thing happens. However I'm still honestly interested just in what can be done with them right now.)
I think everyone considering an SBC should be warned that none of these are going to be supported by upstream in the way a cheap Intel or AMD desktop will be.
Even the Raspberry Pi 5, one of the most well supported of the SBCs, is still getting trickles of mainline support.
The trend of buying SBCs for general purpose compute is declining, thankfully, as more people come to realize that these are not the best options for general purpose computing.
It's now way easier to write drivers/libraries etc whereas before, smaller hardware wasn't worth dedicating developer cycles.
https://www.armbian.com/boards?vendor=xunlong
It's pretty hacky for sure but wouldn't classify it as useless. e.g. I managed to get some LLMs to run on the NPU of an Orange pi 5 a while back
I see there is now even a NPU compatible llama.cpp fork though haven't tried it
I guess manjaro just abandoned arm entirely. The options are armbian (probably the pragmatic choice, but fsck systemd), or openbsd (no video acceleration because the drivers are gpl for some dumb reason).
This sort of thing is less likely to happen to rpi, but it’s also getting pretty frustrating at this point.
Came looking for this. It's the pitfall of 99% of hardware projects. They get a great team of hardware engineer, they go through the maddening of actually producing a thing (which is crazy complex) at scale, economically viable (hopefully), logistic hurdles including tax worldwide, tariffs, etc... only to have only people on their team be able to build and run a Hello World example.
To be fair even big player, e.g. NVIDIA, sucks at that too. Sure they have their GPU and CUDA but if you look at the "small" things like Jetson everybody I met told me the same thing, great hardware, unusable because the stack worked once when shipped then wasn't maintained.
As far as I can tell, the OrangePi 6 remains distinctly uncompetitive with SBCs based on low-end intel chips.
- Orange pi consumes much more power (despite being an arm CPU) - A bit faster on some benchmarks, a bit slower on others - Intel SBC is about 60% the price, and comes with case + storage - Intel SBC runs mainline linux and everything has working drivers
Unreliable USB: https://github.com/raspberrypi/linux/issues/3259
Unreliable Wi-Fi:
* https://github.com/raspberrypi/linux/issues/7092
* https://github.com/raspberrypi/linux/issues/7111
* https://github.com/raspberrypi/linux/issues/7272
I don't understand why many say that RPi software/firmware support is 'fantastic'. Maybe it used to be in the beginning compared to other chips and boards, but right now it's a bit above average: they ignore many things which is out of their control/can't debug and fix (as in Wi-Fi chip firmware).
There's a reason people just default to RaspberryPi even though better _hardware_ exists. RPi at least gets drivers and software support consistently.
This is the problem with every SBC that is not a Pi. I don't understand how they just ignore the software problem. I guess people keep buying them?
Can also plug in a power bank. https://us.ugreen.com/collections/power-bank?sort_by=price-d...
The advantage is that if the machine breaks or is upgraded, the dock and pb can be retained. Would also distribute the price.
The dock and pb can also be kept away to lower heat to avoid a fan in the housing, ideally.
Better hardware should end up leading to better software - its main problem right now.
This 10-in-1 dock even has an SSD enclosure for $80 https://us.ugreen.com/products/ugreen-10-in-1-usb-c-hub-ssd (no affiliation) (no drivers required)
I'd have another dock/power/screen combo for traveling and portable use.
Right?
I've still been on the hunt for a cheap Arm board with a Armv8.3+ or Arvm9.0+ SoC for OSDev stuff, but it's hard to find them in hobbyist price range (this board included, $700-900 USD from what I see).
The NVIDIA Jetson Orin Nanos looked good but unfortunately SWD/JTAG is disabled unless you pay for the $2k model...
``` alias findpi='sudo nmap -sP 192.168.1.0/24 | awk '\''/^Nmap/{ip=$NF}/B8:27:EB|DC:A6:32|E4:5F:01|28:CD:C1/{print ip}'\''' ```
On every `.bashrc` i have.
But I just don't get... everything, I don't get the org, I don't get the users on hn, I'm like skinner in the 'no the kids are wrong' meme.
It's a lambda. It's a cheap, plug in, ssh, forget. And it's bloody wonderful.
If you buy a 1 or 2 off ebay, ok maybe a 3.
After that? Get a damn computer.
Want more bandwidth on the rj45? Get a computer.
Want faster usb? Get a computer.
Want ssd? Get a computer
Want a retro computing device? Get a computer.
Want a computer experience? Etc etc etc, i don't need to labour this.
Want something that will sit there, have ssh and run python scripts for years without a reboot? Spend 20 quid on ebay.
People demanded faster horses. And the raspi org, for some, damn fool, reason, tried to give them.
There are people bemoaning the fact that raspberry pi's aren't able to run LLM's. And will then, without irony, complain that the prices are too high. For the love of God, raspi org, stop listening to dickheads on the Internet. Stop paying youtubers to shill. Stop and focus.
You won't win this game
Often they can go their entire lifespan without some hardware feature being usable because of lack of software.
The blunt truth is that someone has to make that software, and you can't expect someone to make it for you. They may make it for you, and that's great, but really if you want a feature supported, it either has to already be supported, or you have to make the support.
It will be interesting to see if AI gets to the point that more people are capable of developing their own resources. It's a hard task and a lot of devices means the hackers are spread thin. It would be nice to see more people able to meaningfully contribute.
With that being said, CIX and their main board partner, Radxa, have been open with the UEFI.
I am not an expert in low-level environments such as the kernel or the UEFI, but if these tidbits sound interesting I would encourage anyone who is to look further into the CIX P1. To my untrained eyes, CIX looks like a company that is working towards a desktop/laptop chip with real UEFI/ACPI support. I look forward to the day it is polished up a bit.
Yep, I'll pass. I'm done dealing with that kind of crap that is spread like a nasty STD through the ARM world. I'm sticking with x64 unless/until ARM gets this crap together.
Massively simplified, 2.5G is 1G sped up while 5G is 10G slowed down. It makes no sense and the market agrees. The ladder of popularity goes:
1000base-T, <long break>, 10Gbase-T, 2.5Gbase-T, <long break>, 5Gbase-T. (Depends on context ofc, 2.5G is quite popular on APs for example.)
And note a lot of 10Gbase-T hardware is not Nbase-T compatible, and there are chips that do only 1G, 2.5G and 10G - no 5G.
I guess if your design doesn't work at 10GbT you try with 5? Ugh.
That's how PCIe works. A PCIe port - both upstream and downstream - is a "PCI bridge". The link is one bus. A switch chip's "interior" is another bus. The next links are each their own bus again. One per port. There's no switch here, bus 0 ( / 30 / 60)is "in" the CPU, each port is it's own bus.
The more interesting thing is the PCI domain, the first 4 digits:
This generally (caveat emptor) means the ports aren't handled in some common PCIe subsystem, rather each port is independently connected to the CPU crossbar. The ports may also not be able to access each other, or non-transparent mapping rules apply.Doesn't have to, though; it might be due to some technicality, driver bug, misunderstanding, whatever else.