Nucs also tend to win on performance, software compatibility/upkeep (yay commodity x86), price per performance, and often price outright (pis aren't really $35 if you get a newer/better model and once you buy storage/power/case). AFAICT pis only consistently win on power consumption, and that only just (x86 can get well under 10W, you just have to get the right model)
Maybe professional bias but I prefer to follow a cattle vs pets approach here and treat them as disposable, meaning: let the sd card break and optimize for quick replacement.
Current Pi 4s & 5s support booting from USB out of the box with no configuration (took them long enough), so I don't think it's worth the downtime and wasted SD cards.
NVMe SSDs cost the same as SD cards nowadays. You can treat the SSDs as disposable if you want, but you are going to quickly realize that this mindset isn't exactly actionable, because there is nothing that needs disposing of.
Before I opened the article I was thinking "Don't use an SD card. Don't use an SD card. Don't use an SD card."
People don't understand that 90% of the SD card problems are power related (only relevant if you don't use the official power supply) and 10% are simply because of poor SD card quality.
People haven't gotten the message that a charger has lower QC standards since an interruption of the charging process does not shut the device down. A bug that leads to a few milliseconds of power loss will pass QC, but also corrupt your SD card.
Penny drops. Yes this jives much better with the observed data -- that Rpi break SD cards orders of magnitude more than anything else that uses SD cards no matter how crappy said cards are.
Problem could have been solved by adding a decent sized capacitor on the 5v supply rail.
You can also use a Raspberry Pi compute module in an appropriate carrier. I use the CM4 for work and consequently it’s all I use at home too. Lovely versatile device that fits in whatever carrier you want (including one that mocks a regular pi) and it has industrial grade flash.
An IO board that adds the IO ports that the e.g. CM4 lacks. There's a decent range of them from different suppliers, but Raspberry Pi have one themselves called "Raspberry Pi Compute Module 4 IO Board".
I'm shocked the SD card bit isn't first, and more surprised that the post doesn't suggest USB boot (I have one pi that's been on ~24/7 for years now, and I attribute its lack of problems to 1. using Alpine configured to barely touch disk, and 2. not having an SD card to corrupt - I don't know why USB would be more reliable, but anecdotally it is)
I've had a bunch of Pi cards, running on SD without problems. But a single one suddenly developed a super-hot SD card, this was a brand new Pi which I was just setting up. Got the card out, and that one and the next Pi got a USB SSD and are now using those. That was a bit scary. But as mentioned I've also been running Pi with micro SD as Cups servers for years, with no problems at all.
I don't really know. I assume they would be slower than SSDs though, but I have never measured USB flash. The SSBs get some 300MB/sec on my Pi 4 boards.
Sample size of 1, but I ran an RPi4 just fine on a USB thumb drive for about a year.
Still upgraded to an SSD later because I wanted even more storage space (and SSD seemed to have better random IOPS than a thumb-drive), but I'd say go for it.
Yeah, exactly - as best as I can remember, and https://wiki.alpinelinux.org/wiki/Raspberry_Pi seems to agree, Alpine on the pi defaults to a mode where it mosly loads itself into a ram disk on boot and just stays there. Which means that you're constrained on "disk" space (your root filesystem is no larger than ram), but if your workload fits in that then you have the advantage that after boot the system pretty much doesn't touch disk.
I'm still using a Sandisk 8GB microSD from 2008 running 24/7: smartphone expansion 6 years -> Orange Pi as a router for 3 years, Pi 4 running Frigate for 3 years.
I'm guessing it's SLC/MLC.
Had a Transcend 32GB in 2016 die after a year.
The biggest issues with set-and-forget setups is software upgrade for security or other reasons, jumping major versions ends up breaking things. Compared to cloud which is (usually) regularly updated.
Admittedly, I only have one long-running Raspberry Pi, but it's currently sitting at a few months uptime. And that was an intentional reboot. I've never had to take any measures like these in the four or so years I've had it up.
a) Cable ethernet
b) SSD (via USB3.0 adapter on my RPI4)
c) Ubuntu Server LTS 22.04.
d) cheap UPS.
Mine runs Yggdrasil network, HAproxy, Caddy server, a couple of webservers in containers, and a TMUX instance that I log into almost daily to write code (slow computer reveals bad code much better). Since I put it (and my router) on the UPS, in the last 2 years it has literally never gone offline other than a couple of times I rebooted it for firmware upgrades.
This is mostly what I do for “critical” Pi things. The cable Ethernet and the SSD are major. Do you have any recommendations for a cheap UPS? What are your considerations here?
I went with a CyberPower BR1200 which was about 150 pounds (circa 200 USD) but that's overkill because it will keep my pi+router going for several hours, whereas power-downs typically last less than 15 minutes max where I live and I haven't seen > 1 hour in 10 years. It's usually "human error" related someone plugs the vacuum in or some appliance trips a circuit. Keeping the router on the UPS (and fiber box in my case) is important though because then everything is completely uninterrupted from a serving perspective. I also give the standard RPI power adapter enough headroom by not overclocking the pi, because in the past this has caused problems when SSD attached with lots of writing, and "big" (for a pi) compute load at the same time. Since turning off the overclocking, zero problems. Probably an RPI5-class power supply would be even better. No clue if this matters but on a jetson nano (which was notoriously power-spikey), crucial (micron) or samsung SSDs tended to be better than budget alternatives.
I got one of those super low profile USB thumb drives and then set the Pi to boot from that. It ran automation here in my house for 2.5 years without a blip.
Also no mentioning of the power connector? I have too little experience with USB-C, but the micro USB connector used on early Raspberry Pi's is just asking for trouble. That might be (barely) good enough for charging, but a computing device w/o battery-backup won't take lightly the power interruptions when jiggling the cable a little. I finally got around to replace it with an old-fashioned (time tested!) barrel connector. Easy way to improve the robustness significantly.
As with my DSLRs/mirrorless cameras (particularly the ones that use the wretched USB mini) when tethering I have ended up with a right-angle adapter cable secured with a cable tie (or the super-strong Tethertools Jerkstopper), and a USB micro cord into that.
USB micro is designed to snap at the connector, not the board. And the connectors will indeed snap.
I've been running a bunch of Pi's for years now, and the biggest problem I've had is the Pi itself dying: 24/7 usage is hard on a small device. I've also found that stable power is essential, and to that end I've always used 5v 3a branded power cubes, plugged into a pure sine wave UPS. Choice of micro-SDHC cards is important and I ended up getting ATP industrial cards (https://www.atpinc.com/products/industrial-sd-cards) - expensive but really long-lived. Finally, using RPi-clone (https://github.com/billw2/rpi-clone) on a regular basis has been a life-saver. I clone to Sandisk Extreme micro-SDHCs and can recover from an outage in minutes.
I did exactly zero of any of those things and have had some Pis run for multiple years without any issues until being replaces by a newer model (my HomeKit/Zigbee gateway and data logger is now a Pi 4). I guess it all boils down to good SD cards and stable power supplies.
Same here - I've been running a couple of Pi3 as Cups servers for years of uptime (the only time that uptime gets reset is when there's a power outage - and that's very rare indeed). Did nothing more than install Raspbian on a micro SD card, set up Cups, connect USB to printer (for one of them - the other manages a networked printer). And left them alone after that.
My Raspberry Pi 2 ran for a while, used as a pihole, a VLC-hacked media center, and a weather stations for simple sensors, until it didn’t :(
In ~2022 I started to get random errors, then found out the SD card was failing. I never took the time to fix it, it had lots of little things locally I don’t feel like doing again.
It may seem like overkill, but I learnt how to use ansible to manage the software on my pi. I've actually just upgraded my rpi4 to use an ssd so am choosing to start from scratch. I'm hoping my ansible playbooks are written well.
Haha, yeah, that’s a good point. I may still have some ansible playbook from a while ago but I’m pretty sure it wouldn’t be up to date. I guess it would be nice to have a system like CoreOS was, where you just provide a setup script, systemd unit files, and at runtime most of the file system is expected to be read-only. That way you’re confident you keep all your setup in a git repo and on reboot the whole thing is reset (outside of data stores).
It’s just so tempting to quickly ssh into the machine to hack something around, then you forget about it b cause it ~works.
But a rpi4 can run containers, so that may also be an alternative.
I've been trying (and failing) to get my ansible playbooks to work for about 2 hours - so definitely not a perfect solution by any stretch. This reminds me why I don't daily drive a linux machine! I don't know why I'm encountering so many errors around installing docker-compose, pip, apt packages etc.
I suspect ansible may be a better solution for when you're using it to build machines from scratch very frequently, and you'd have more of a chance to maintain and ensure they're working over time vs once every couple of years.
The other annoying thing with ansible is that while it may install the software, it can't log me in. So still having to manually sign in to syncthing/tailscale etc even after the software is installed.
> I guess it all boils down to good SD cards and stable power supplies.
I would say your sample size has a stronger effect on your experiences. With a large enough pool of devices everything will go wrong, what can go wrong. And there will be also new failure modes that you have never even dreamed of.
6 years of the same Orange Pi PCs/PC2s often on the same cheap Sandisk SD cards, some with postgresql databases on them logging environment data every 5 seconds (though batched to 15 minutes), frequent large system updates (Arch Linux ARM is a bit more churny). And not much issues, to be concerned about doing something annoying like other people suggest here with A/B updates and readonly stuff, that prevents comfy normal use of a general purpose Linux OS.
I don't run a completely storage oblivious setup, I use f2fs, which has much better write patterns than ext4, and I disable stats collection in postgresql, which causes constant churn, but logging, and other stuff has negligible effects and I don't care doing anything special about it, and I leave it default OS configuration.
I have probably 4-5 uSD card failures over 60 year total runtime of my 15 SBCs that I run 24/7. So that's 1 failure per 12 years. Nothing that can't be dealt with using backups. (I didn't even need them that much, yet, because all uSD cards so far failed by turning read only, which for f2fs means you'll get the last consistent snapshot on the card, that you can just copy to a new card and continue. 10-15 minutes of recovery time.
All that complexity and limitations that people talk about seem way overkill for a home setup.
And I think other reason why I don't get many uSD card failures is because I run most cards in HS25 mode. Not in SDR104 or whatnot. 3-4x the frequency really causes the cards to heat up a LOT during activity. Can't be great for the flash chips. 2 of those failures were in SDR104 enabled hosts. Copying data to uSD card using SDR104 capable USB3.0 adapter really makes the card scarry hot in just a few seconds.
In the grand scheme of things a sample size of 15 SBC is the same as OP's: 1.
While I definitely agree that for a home setup it is overkill, in my dayjob I work on industrial embedded gadgets, sold 10-100k pcs/design, expected to run without reboot for years, sometimes for decades. And most of the weird issues end up usually on my desk for investigation. While I admit that this might not have been completely obvious, but when I referred to the sample size, I was referring to such numbers, taken from this experience.
Sorry, but when discussing the article, I assume the context of the article.
Context of the article is "I use Raspberry Pis around my home as everything". I don't really care about industrial anything in that context, nor 10k+ unit scale. And nobody sane does/should. At home/hobby I'm optimizing for very different things.
It's interesting if someone adds to the discussion that things predictably fail at 10k+ scale, but completely irrelevant. It's like people who read backblaze stats to go buy 2 disks from the same batch to put in raid1. That those disk mdoels fail at 2x lower rate at scale than other models is completely irrelevant to their data safety.
Older Raspbian OS around year 2016 and below had access time enabled, with any file read there was a write, which is probably why there were many reports of corrupted cards ever since.
With one command it for all Pi’s for both Raspbian and bookworm it:
* Shrinks the file system (Gee, how does it do that with just one disk ? ;-) )
* Creates new partitions
* Installs a memory overlayFS
* Installs and configures the system as an audio recorder with micro second time accuracy
* updates /etc/rc to do a forced repair of the data and config portions, in case they were damaged. This avoids system hangs waiting for human interaction with fsck
For the partitioning scheme it creates a swap partition, not as a wow but as an enabler if you really need it to install some large software.
It creates a small config partition. The idea here is that you keep it read only and remount it read write if you need to change config then remount it read only again.
And finally a data partition, which in this projects case is where the audio files are written.
I maintain a version of an overlayFS boot for the Pi but it needs revisiting for bookworm. The easiest way to use do this is to install the sbts-aru and then just don’t use it. Then everything is done for you in one command. And that version works for all Pi’s.
I also do this for the Jetson SBCs. But I need to revisit this for the Orin series. I have it working here for myself and friends but need to update the installer. Note, due to kernel behavior changes with Orin the older Pi like overlayFS code will not work. But I solved this and will release it when I release the Orin release of sbts-install soon.
I’ve been using memory overlayFS like installs for years for long running Pi systems.
For the life of me I don't really get it why Raspberry Pi Foundation does not include onboard eMMC or SSD storage in their non-compute modules products.
Yes with the new PCIe expansion in the latest RPi 5 you can have external SSD for example, but if you decided to use it for other purposes as well like extra Ethernet port expansion then you cannot use it for booting anymore.
> For the life of me I don't really get it why Raspberry Pi Foundation does not include onboard eMMC or SSD storage in their non-compute modules products.
Cost. It's always cost.
If the eMMC or SSD storage is not enough to hold a general purpose OS for all users, then only some users get the value. And if it is big enough, it's put the cost of the machine up above the point at which they feel happiest, when an SD card is perfectly fine for the majority of their target users.
Eben Upton is regularly on record talking about how the cost-per-component/users-per-component tradeoff will lead them to avoid adding a component, and has motivated removing some (composite video for example)
It's curious that there are people who need this, are aware of the compute module, and still complain about it. Is the the availability of the compute module that's a problem?
First advice must be to mount FS in read only mode, mount /var in memory and forward al logs to one nide, which may be not RPi but something with proper UPS and nut running. Power loss becomes absolutely bening if your FSes RO or temporary.
It is overkill if you have one RPi but author claims that he uses multiple RPis all around a house.
Also, good idea to have A/B system partitions and upgrade system with full partition rewrite and changing active one. Thus way your system will have one good system partition in any case, even if new version has fatal bugs, and recovery become trivial.
I'm using several small/single board PCs im different roles in such way for 20+ years with great success.
During installation the guide tells you to run a program, which generates an fstab based on current mounts. So by default it will configure /tmp the same way it’s configured in the live cd.
Actually the installation guide for arch asks you to use `genfstab -U /mnt >> /mnt/etc/fstab` which basically copies over whatever was mounted at the live environment (taking off the /mnt), I'm pretty sure tmpfs for /tmp was there by default last time I installed.
A long time ago, it used to be. It doesn't seem to be now, and I don't have any idea why the distros changed. (Maybe it's due to the mv semantics? But I though people considered the idea of creating a file in /tmp and then renaming it to be bad instead.)
Copying or moving through tmpfs loses silently some file metadata, e.g. extended attributes or high-resolution filestamps.
Copying through /tmp is a frequently used method for transferring files between users on a multi-user computer.
It is said that the Linux kernel will include soon an improved version of tmpfs, which will allow user extended attributes, within certain limitations.
When this will happen, one of the most annoying misfeatures of tmpfs that has persisted for much too many years will finally be gone away.
If I had to guess, I would assume they chose not to use tmpfs because the earlier Pis had very limited RAM. With a 4GB or 8GB pi 4 or 5 this should not be a problem.
Anything transient writable must be in memfs/tmpfs/how it is mamre im your OS, of course.
Logs to log server, if system needs non-volatile writable storage — NFS or second storage device, depending on requirements.
Of course, it is too much hassle for single such system (but I started to use it from very beginning out of curiosity) but uf you have many single-task small devices it is very convenient.
I have one «beefy» server/NAS (now it is EPYC2 on SuperMicro platform becuse they are cheap used on AliExpress, before it was Intel E3 12xx-badsed systems, two ir three generation old, always bought used). It is not my router, but NAS for all my data, NFS and build server.
I'm using FreeBSD, and here is script for preparing such installations named NanoBSD. To be honest, it is nothing special — build system with provided config (to strip it down, for example full FreeBSD installation includes toolchain and it is waste of space on «embedded» systems), mount file as loop device, create FS, install system to this FS by standard system means, add needed packages.
I'm building system once, make several images (as each device needs its own set of packages, of course), login to each device via ssh and run simple sh script which detects current active partition (by simply looking at output of mount command — from whic root is mounted) and then «dd if=/net/images/$hostname.img of=/dev/da0p$otherpart bs=128k», set this updated partition as «bootonce» in bootmanager and reboots. Last startup script in boot sequence check availability of network and liveleness of sshd, and if these simple checks are Ok, set this partition as alwaysboot (it is all UEFI features, mostly). If something goes wrong — one power cycle and device will boot from previous partition.
I don't have enough devices to automate «login and call script» part :-)
Note that mounting /var in memory might exceed your device's memory if you're using something like Docker. You might have to move /var/lib/docker to secondary storage.
I have a few long time running Pis - and have been keeping them up for a decade now. No SD Card corruption ever and i got close to 1000 days uptime on one.
The biggest problem is loss of wifi, after a few months one will lose wifi, but keep working - it’s constantly recording data so a reboot is not a good idea. I’d prefer a solution where I could just reset the wifi, but all attempts to script that reliably so far failed.
I've got a few in factories (doing non critical stuff) at work. They auto reboot regularly because otherwise they'll lose whatever's on usb (reading those usb metering devices being the whole point of the PIs).
I've been plagued with the wifi problem since changing routers. Devices on the local network will randomly lose the ability to connect to it, but everything else is fine.
I added a cron job to one Pi that checks if wifi is up and tries to restart it if not.
I also had an LG monitor that had so much feedback it would disable the WiFi interface completely. So I would check your monitor, if the pi is connected to one.
Yes. And I monitor my network link and automatically power-cycle the modem, if the worst happens. (Which is rare, but the network link has been the source of most of the few problems I've had.)
And use a wired network connection for anything important!
I’ve got 2 rPI’s thats been running for 4+ years each. Only thing I do is update them once in a while. Both have workloads running 24/7. no issues. I have however experienced issues in the past but those were due to a faulty power adapter not keeping voltage within spec.
It's worth noting that SD card firmware is usually optimised for the FAT32 filesystem, which has a very predictable access (specifically write) pattern, and using filesystems that have a more "free-form" layout can lead to lower performance and higher write amplification:
Good note, I wonder if wear leveling depends on that since it's local wear leveling compared to SSD global.
SDXC probably requires exFAT and free space bitmap is considered.
This covers the readonly filesystem, but doesn't cover the write protect flag that you can set on the microSD card itself[1]. The flag will configure the card's controller to drop any writes, and is thought to resist the corruption issues that can still occur even when the filesystem is readonly.
Also, creating a readonly root out of an existing disto is a bit of a pain, my preference is to use a distro (like TinyCore) that's already a readonly root.
All my home Pis network boot, so there is no card to fail. You can also change what OS they boot into by just renaming a symlink on the server and rebooting them. Very convenient.
I'm curious what you're using all these Pis for when you've got a server on the network to provide network boot. Can you not just use the device that's providing network boot to provide whatever you're running on the Pis?
No, because the server is in the wrong place. Some Pis are connected to televisions and play videos, or emulated games. One is connected to my hi-fi. One has a GPS receiver on it for NTP. One is a photo frame. Basically, the Pis exist to interface with the real world.
As someone using many, many Pi's at home and many times that at work, the preferred approach is to boot them all diskless (and for those that actually need an SD card, boot them off a read-only SD card and get everything else off the home server). This is so much easier than having lots of different SD cards/Pi versions etc, and makes them trivial to replace in the event of failure.
Look up diskless booting, it's a very general thing on linux (i.e. not Pi-specific, although there are plenty of Pi-specific tutorials on the 'net.
As a minimal first step, install an NFS server (which can be X86, Pi, other) on your LAN and make sure you can mount it from the pi ("mount $SERVER:/some/dir /mnt/tmp"). Then copy the contents of a Pi SD card to the server, make it exportable (see '/etc/exports') then edit '/etc/fstab' on the Pi to mount the (now remote) copy of the SD card instead of the usual root. That should get you started - beyond that, with some of the Pi's you don't even need to have an SD card installed (however you'll then need to set up things like DHCP and TFTP on your server).
i very much so enjoyed these articles even if i don’t use raspberry pi anymore. it has a very much so research notes cleaned up for sharing feel and reminds me
of some of my teams internal articles, particularly on research the author wasn’t sure on but wanted to put out there regardless. it was very easy to get their decisions and the limit of their research and knowledge so i had pretty clear idea what i need to still check on my own and what i might adopt if i were to use rpi devices.
i don’t really get what to use a rpi for but i guess not important, it was just a nice series of articles
349 comments
[ 3.8 ms ] story [ 396 ms ] threadhttps://www.notebookcheck.net/Ryzen-5-5560U-performance-debu...
Similar power consumption but power and case/cooling/storage is included.
Depending on what you choose you get eMMC or SATA or M.2 SATA or M.2 NVMe for storage.
For anyone that is interested: https://www.parkytowers.me.uk/thin/
People don't understand that 90% of the SD card problems are power related (only relevant if you don't use the official power supply) and 10% are simply because of poor SD card quality.
People haven't gotten the message that a charger has lower QC standards since an interruption of the charging process does not shut the device down. A bug that leads to a few milliseconds of power loss will pass QC, but also corrupt your SD card.
Problem could have been solved by adding a decent sized capacitor on the 5v supply rail.
I think if you take into account how often they're powered on you'd find cameras destroy flash memory cards comparably to Raspberry Pis.
> Problem could have been solved by adding a decent sized capacitor on the 5v supply rail.
Or you could just buy the Raspberry Pi branded power supply.
https://sourcekit.cc/#/
Think I really want something like the Uptime Labs Blade https://pipci.jeffgeerling.com/boards_cm
https://raspberrypi.stackexchange.com/questions/124628/raspb...
Still upgraded to an SSD later because I wanted even more storage space (and SSD seemed to have better random IOPS than a thumb-drive), but I'd say go for it.
Only reason why it fails is when the electricity goes out. A battery that could keep it running for 10 minutes would completely be enough.
I'd like to learn some tips on how to do that... do you mean something like https://wiki.alpinelinux.org/wiki/Installation#Diskless_Mode ?
I'm guessing it's SLC/MLC.
Had a Transcend 32GB in 2016 die after a year.
The biggest issues with set-and-forget setups is software upgrade for security or other reasons, jumping major versions ends up breaking things. Compared to cloud which is (usually) regularly updated.
Only buy Sandisk I guess!
USB micro is designed to snap at the connector, not the board. And the connectors will indeed snap.
In ~2022 I started to get random errors, then found out the SD card was failing. I never took the time to fix it, it had lots of little things locally I don’t feel like doing again.
It’s just so tempting to quickly ssh into the machine to hack something around, then you forget about it b cause it ~works.
But a rpi4 can run containers, so that may also be an alternative.
I suspect ansible may be a better solution for when you're using it to build machines from scratch very frequently, and you'd have more of a chance to maintain and ensure they're working over time vs once every couple of years.
The other annoying thing with ansible is that while it may install the software, it can't log me in. So still having to manually sign in to syncthing/tailscale etc even after the software is installed.
Agree, I've also been running a number of PIs and when they broke, it was because of failing SD cards.
I found pibenchmark to be a good source of info --> https://pibenchmarks.com/
Definitely compare SD cards before buying.
I don't run a completely storage oblivious setup, I use f2fs, which has much better write patterns than ext4, and I disable stats collection in postgresql, which causes constant churn, but logging, and other stuff has negligible effects and I don't care doing anything special about it, and I leave it default OS configuration.
I have probably 4-5 uSD card failures over 60 year total runtime of my 15 SBCs that I run 24/7. So that's 1 failure per 12 years. Nothing that can't be dealt with using backups. (I didn't even need them that much, yet, because all uSD cards so far failed by turning read only, which for f2fs means you'll get the last consistent snapshot on the card, that you can just copy to a new card and continue. 10-15 minutes of recovery time.
All that complexity and limitations that people talk about seem way overkill for a home setup.
And I think other reason why I don't get many uSD card failures is because I run most cards in HS25 mode. Not in SDR104 or whatnot. 3-4x the frequency really causes the cards to heat up a LOT during activity. Can't be great for the flash chips. 2 of those failures were in SDR104 enabled hosts. Copying data to uSD card using SDR104 capable USB3.0 adapter really makes the card scarry hot in just a few seconds.
While I definitely agree that for a home setup it is overkill, in my dayjob I work on industrial embedded gadgets, sold 10-100k pcs/design, expected to run without reboot for years, sometimes for decades. And most of the weird issues end up usually on my desk for investigation. While I admit that this might not have been completely obvious, but when I referred to the sample size, I was referring to such numbers, taken from this experience.
Context of the article is "I use Raspberry Pis around my home as everything". I don't really care about industrial anything in that context, nor 10k+ unit scale. And nobody sane does/should. At home/hobby I'm optimizing for very different things.
It's interesting if someone adds to the discussion that things predictably fail at 10k+ scale, but completely irrelevant. It's like people who read backblaze stats to go buy 2 disks from the same batch to put in raid1. That those disk mdoels fail at 2x lower rate at scale than other models is completely irrelevant to their data safety.
I’ve switched to SSDs for most of them now. It’s too much of a dice roll otherwise
https://github.com/hcfman/sbts-aru
https://hackaday.com/2023/12/30/localizing-fireworks-launche...
With one command it for all Pi’s for both Raspbian and bookworm it:
* Shrinks the file system (Gee, how does it do that with just one disk ? ;-) )
* Creates new partitions
* Installs a memory overlayFS
* Installs and configures the system as an audio recorder with micro second time accuracy
* updates /etc/rc to do a forced repair of the data and config portions, in case they were damaged. This avoids system hangs waiting for human interaction with fsck
For the partitioning scheme it creates a swap partition, not as a wow but as an enabler if you really need it to install some large software.
It creates a small config partition. The idea here is that you keep it read only and remount it read write if you need to change config then remount it read only again.
And finally a data partition, which in this projects case is where the audio files are written.
I maintain a version of an overlayFS boot for the Pi but it needs revisiting for bookworm. The easiest way to use do this is to install the sbts-aru and then just don’t use it. Then everything is done for you in one command. And that version works for all Pi’s.
I also do this for the Jetson SBCs. But I need to revisit this for the Orin series. I have it working here for myself and friends but need to update the installer. Note, due to kernel behavior changes with Orin the older Pi like overlayFS code will not work. But I solved this and will release it when I release the Orin release of sbts-install soon.
I’ve been using memory overlayFS like installs for years for long running Pi systems.
Yes with the new PCIe expansion in the latest RPi 5 you can have external SSD for example, but if you decided to use it for other purposes as well like extra Ethernet port expansion then you cannot use it for booting anymore.
Cost. It's always cost.
If the eMMC or SSD storage is not enough to hold a general purpose OS for all users, then only some users get the value. And if it is big enough, it's put the cost of the machine up above the point at which they feel happiest, when an SD card is perfectly fine for the majority of their target users.
Eben Upton is regularly on record talking about how the cost-per-component/users-per-component tradeoff will lead them to avoid adding a component, and has motivated removing some (composite video for example)
It's curious that there are people who need this, are aware of the compute module, and still complain about it. Is the the availability of the compute module that's a problem?
First advice must be to mount FS in read only mode, mount /var in memory and forward al logs to one nide, which may be not RPi but something with proper UPS and nut running. Power loss becomes absolutely bening if your FSes RO or temporary.
It is overkill if you have one RPi but author claims that he uses multiple RPis all around a house.
Also, good idea to have A/B system partitions and upgrade system with full partition rewrite and changing active one. Thus way your system will have one good system partition in any case, even if new version has fatal bugs, and recovery become trivial.
I'm using several small/single board PCs im different roles in such way for 20+ years with great success.
In fact, I didn't notice the change until now.
Copying through /tmp is a frequently used method for transferring files between users on a multi-user computer.
It is said that the Linux kernel will include soon an improved version of tmpfs, which will allow user extended attributes, within certain limitations.
When this will happen, one of the most annoying misfeatures of tmpfs that has persisted for much too many years will finally be gone away.
Be careful to check inside your docker containers also. It can end up different.
Here is a good article btw, with someone advocating for moving to tmpfs https://ubuntu.com/blog/data-driven-analysis-tmp-on-tmpfs
Logs to log server, if system needs non-volatile writable storage — NFS or second storage device, depending on requirements.
Of course, it is too much hassle for single such system (but I started to use it from very beginning out of curiosity) but uf you have many single-task small devices it is very convenient.
How then do you update the system or install new software?
Of course, devices don't build their own systems.
https://www.dzombak.com/blog/2021/11/Reducing-SD-Card-Wear-o...
What’s your upgrade process like? How do you make the new disk image? Do you log into each device to upgrade it, or do you have automation?
I'm using FreeBSD, and here is script for preparing such installations named NanoBSD. To be honest, it is nothing special — build system with provided config (to strip it down, for example full FreeBSD installation includes toolchain and it is waste of space on «embedded» systems), mount file as loop device, create FS, install system to this FS by standard system means, add needed packages.
I'm building system once, make several images (as each device needs its own set of packages, of course), login to each device via ssh and run simple sh script which detects current active partition (by simply looking at output of mount command — from whic root is mounted) and then «dd if=/net/images/$hostname.img of=/dev/da0p$otherpart bs=128k», set this updated partition as «bootonce» in bootmanager and reboots. Last startup script in boot sequence check availability of network and liveleness of sshd, and if these simple checks are Ok, set this partition as alwaysboot (it is all UEFI features, mostly). If something goes wrong — one power cycle and device will boot from previous partition.
I don't have enough devices to automate «login and call script» part :-)
Are there any solutions available for this?
The biggest problem is loss of wifi, after a few months one will lose wifi, but keep working - it’s constantly recording data so a reboot is not a good idea. I’d prefer a solution where I could just reset the wifi, but all attempts to script that reliably so far failed.
I also had an LG monitor that had so much feedback it would disable the WiFi interface completely. So I would check your monitor, if the pi is connected to one.
While one could argue that you should figure out the source of your device freezing in the first place;
Nothing is better than having to ask someone to power cycle your Raspberry Pi while you're away.
And use a wired network connection for anything important!
They state the endurance on thousands of hours of FHD video, but what assumptions do they make in bitrate etc?
Can‘t they state total TB written or drive writes per day or something sensible?
Specifying endurance in "thousands of hours of FHD video" implies large sequential writes, or in other words the best-case for write amplification.
I used one of these Micron cards in my RPi4, it has high write endurance and also an A2 performance rating so it can support the IOPS needs of a boot drive. https://www.mouser.com/new/micron-technology/micron-i400-mic...
https://lwn.net/Articles/428584/
Also, creating a readonly root out of an existing disto is a bit of a pain, my preference is to use a distro (like TinyCore) that's already a readonly root.
https://github.com/BertoldVdb/sdtool
As a minimal first step, install an NFS server (which can be X86, Pi, other) on your LAN and make sure you can mount it from the pi ("mount $SERVER:/some/dir /mnt/tmp"). Then copy the contents of a Pi SD card to the server, make it exportable (see '/etc/exports') then edit '/etc/fstab' on the Pi to mount the (now remote) copy of the SD card instead of the usual root. That should get you started - beyond that, with some of the Pi's you don't even need to have an SD card installed (however you'll then need to set up things like DHCP and TFTP on your server).
There's also a tutorial at https://www.raspberrypi.com/documentation/computers/remote-a...
i don’t really get what to use a rpi for but i guess not important, it was just a nice series of articles