Yep - for home automation style stuff (which the author seems to have built it for - see the showcase section), having a full Linux distro is actually a real pain. I've taken to building my own stuff with the ESP series of microcontrollers, home made PCBs, etc - once it's built - it just keeps chugging along, unlike my original stuff I built atop RaspberryPis with Linux.
I've done a lot of this sort of this as well. The ESP8266 is so great for simple on/off switches and sensors.
But-- there are so many applications for having a full IoT Linux distribution that I can't believe this isn't already a solved problem. I'm imagining an ultra-stable distro that is easy to deploy to a headless machine, handles security updates, handles flaky wi-fi issues, tests the SD card (I've had more than a few burn out on me), and automatically logs to the cloud for easy debugging.
In other words: Why am I still maintaining my RasPis as if they are apache servers from the 1990s?
People have been building stuff like this. Often for firewall appliances (eg SCC Sidewinder, Immunix Linux, genua w/ OpenBSD). They usually do one of these: (a) get big under new management that doesnt care about either security or the little people (enterprise focus); (b) sell out to company like that; (c) go bankrupt cuz market values assurance of security the least.
Sad but true. There's still a few companies selling microkernel-based solutions in defense and mobile with Linux user-lands. They'd OEM license their stacks for a new set of products aimed to do this. They cost a fortune. Whoever pays it wont get it back, though, cuz folks dont usually pay for security in that volume. So, nobody is doing it for small players for now.
You have to keep it up to date because of all the security issues in other apps that are installed.
If the system only contains the Linux kernel and your software you only have to worry about your security issues (unavoidable) and Linux security issues (also difficult to avoid - too much work to write a new kernel too, plus remote Linux vulnerabilities are quite rare).
So why not just create a stripped down Linux distribution instead of reinventing the wheel and creating a new userland from scratch (with probably a few brand new and undiscovered security issues already)?
Replacing the implementation language eliminates whole classes of attacks, while stripping down an existing Linux distribution can only go so far. Plus, it’s not clear to me how one would maintain such a stripped-down distribution in a way that isn’t a lot of effort.
I still find this effort odd. You already have a well tested and widely used userland, even if it is written in a non memory safe language.
The easiest solution to me seems to be to grab the tools you need from the existing userland, instead of reimplementing them in Go. Currently you have reimplemented some of the userland, and packaged it up into a distro. So you have to maintain a stripped down distribution and the tools that you reimplemented. If you just took existing tools you would only have to do the latter. Am I missing something here?
The userland in question is an NTP, a DHCP client and the init system / web interface. It really isn’t much code. Implementing these in Go reduces the number of moving parts, both in terms of libraries involved in the project and more importantly in terms of programming languages involved.
Have a look at the git history to see how little maintenance is required in practice. I have barely had to touch the code over the last year of the project. Just incorporating new upstream versions of a typical Linux userland would have been more work.
Also note that “distribution” is a rather big word for the few files we’re talking about:
If my reasoning doesn’t convince you, so be it. Not every project has to make sense to everybody :). I wanted to do a pure-Go userland, and I’m happy with the amount of work this results in and the properties that come out of this decision (easily automated updates, massively reduced attack surface).
It's everything the kernel doesn't do. In general, that means the surface with which you interact as a user. So if you're used to using a terminal, your SSH connection, your Bash session, are all userland programs (that use kernel features to do things). You can think of it as kernel is a framework like Rails and userland stuff is developers using and expending the interfaces... sort of.
When your computer runs headless, the init/process tree system is userland. Background (cron) jobs are userland, daemons are userland. Etc.
It’s not a full-blown distribution; it doesn’t distribute third-party software. Instead, it allows application authors to build their own distribution (at which point “appliance” is a more fitting word than “distribution”).
Updates are easy to fully automate with gokrazy (and I’ve been using that mechanism for a year at this point) because assembling a new image and installing it over the network is just one simple command. New kernel and firmware versions are provided automatically once they are verified to boot on real hardware.
There is no maintenance aspect in the actual installations (i.e. you don’t need to SSH into them, ran commands to update, etc.) — the root file system is read-only, and an update overwrites it completely. Unless the applications you want to use with gokrazy introduce state, gokrazy defaults to being stateless.
Hope that clarifies things, let me know if you have further questions.
Forgive me if this sounds dumb, but if the image just contains the firmware + kernel + a light layer for managing processes, do this mean it doesn't include stuff like SSHD?
That’s correct: there is no SSH daemon by default. You can optionally include https://github.com/gokrazy/breakglass, though, which is an SSH daemon that can be started on demand to facilitate interactive debugging.
This is great! I think it could have uses far beyond the Raspberry Pi. As someone who's worked in the consumer electronics industry for years, I think we desperately need to rethink our approach to Linux-based electronics with respect to both the userland and the primary application. A legacy-free framework written in a safe language could go a long way to improving our currently lousy reputation with respect to security and reliability. (Yes, fresh code will have fresh bugs even in a safe language -- but it's a start!) I'm also keeping my eyes open for similar ideas using Rust.
Doesn't look like it. U-boot is a boot loader for PowerPC/ARM/MIPS/<other CPUs> boards. A typical Linux-based embedded system would consist of U-Boot, the Linux Kernel and a filesystem. u-root would be one of the many ways to construct this filesystem.
A large portion of the recent widely exploited NAT box / IP camera vulnerabilities and malware are exploiting logic bugs, backdoors and default passwods - instead of memory safety bugs. I agree memory safe systems are needed, but nontechnical changes are also needed to fix the widespread IoT vendor negligence.
Exactly, the security in legacy systems is non-existent. I've been pretty pleased moving systems to the nerves-project [1] (which like GoKrazy has an A/B root file-system) to replace deprecated and ancient firmware systems. One of those setups required a Windows XP machine to flash the particular MCU that's obsolete now, a huge pain. It was almost impossible to update / patch.
Nerves is based on a Buildroot core but most of the system is built with Elixir and BEAM/OTP. It's fantastic having a high level language built for robustness and reasonable security. Still, I have dreams of porting BEAM to Redox OS one day [2] and replacing the C userland-bits with Rust-bits. Especially if you could represent the micro-kernel services as actors. That would be amazing!
> As someone who's worked in the consumer electronics industry for years, I think we desperately need to rethink our approach to Linux-based electronics with respect to both the userland and the primary application.
Google has created a stack called NERF (Non-Extensible Reduced Firmware) for this:
E.g. Fedora has for years been using for RPi the same close-to-mainline kernel source that is used for other ARM boards and x86-64 and any other supported hw platform.
Thanks! That’s good to know. Perhaps getting gokrazy going on the Pi Zeroes will not be a huge task. 32-bit userland probably already works on the 64-bit Pi 3 kernel.
I've only dipped into Go recently (and not for any production code) but your article very nicely summarises my impressions with it so far. I've found a kind of fluid pleasure in writing Go that wouldn't be apparent from a bullet list of its (lack of) features. It really allows my attention to fall mainly on the problem domain in a way few other languages afford.
I can't resist while here thanking you for i3. I've been a mac user recently, but will likely be back to linux in the next year or so. Using i3 again will be probably be the greatest pleasure involved in that return ;)
I've added gokrazy to my list of things to play with over the weekend (when I theoretically have time, hah). One question I'd ask in context of the other poster, can I easily run a Lua VM and tie it into the gocrazy env so that a Lua app would appear like a go app?
It might be possible to do this, but certainly not easily.
gokrazy’s unit of application is a Go package, so you’d need to write a Go program which then executes the Lua VM. I don’t know what dependencies Lua has — if it cannot be compiled statically, I’d recommend against using Lua in this context.
Are there not many statically typed, compiled languages that ship a runtime with a garbage collector?
Not really a huge number, no. Java is probably the most obvious, if you count that as compiled. Maybe Haskell or D? I suppose even Swift of Objective C if you include RC in the definition of GC.
I would say the focus shouldn't be entirely on GC, however. Safety as a whole is the most important factor, and it's possible to be say without GC, as you point out. I reckon Go is actually quite an attractive language for this sort of application regardless – quite safe, quite lightweight, good libraries.
There are a number of languages that meet the memory safety criteria, and they each have tradeoffs that may make them a good choice for a certain range of use cases. This project happens to be based on Go, but similar projects in other languages would be interesting as well.
Go is not memory safe (though is certainly safer than C++). Various Go constructs like interfaces and slices are implemented via non-atomic multiword structs, and data races may result in invalid values leading to memory corruption, etc. See https://blog.stalkr.net/2015/04/golang-data-races-to-break-m...
Many. I like bashing C++ as much as the next dev, but modern C++ has nice type checks, interesting types/wrappers which help with common patterns (shared, unique ptrs, etc), and the compilers learned some great tricks which allow you to run memory safety checks yourself. (Asan and others)
Yours is a stricter and not-commonly-used definition of "memory safe", which typically means "in the C sense" i.e. pointer arithmetic and bounds checking. Go is certainly memory safe by the typical definition.
What's the typical definition? With data races, you can access array values out-of-bounds, which seems like the most fundamental thing you'd want protection from in a memory safe language.
You can't access data out of bounds. Go will panic with a runtime exceptions if you do. Memory safety doesn't mean "memory IO errors never happen" it means "if a memory IO error happens, you'll know"
If you have an out of bound in C, your program keeps running in 9/10 cases. In Go you'll cause a panic in 10/10 cases which you can either handle and propagate as error or let the program crash entirely, the memory will be in a defined state if you choose to recover (your array might be garbage but you won't have written data into random memory belonging to other threads)
But with Go you CAN access data out of bounds. See the "Exploiting the slice type" section where writing to a slice modifies a function pointer which happens to be stored after the slice. It's an OoB write which does not panic.
an OoB write under a data race in a program that won't occur like this in the wild. The conclusion section of the post you mean contains that statement too.
The %p formatting used to gain access to the pointer of a variable uses the unsafe package, the formatting itself is incredibly rarely use to begin with.
I don't see anybody doing any security-related exploits doing that unless you explicitly write for it.
Go won't do out of bounds, edges cases do happen. You can find the same stuff on Java and Rust (yes, I had OoB in Rust, though on a baremetal environment without OS, even Rust ain't safe.)
I don't see why it's a different definition, though. What's he's saying is that it's very easy to break Go's bounds checking with race conditions, and produce memory corruption. This isn't true of languages traditionally considered as memory safe - e.g. Java or Python.
This is only true if you can prove that data races will never cause any bounds checks to be wrong. The blog post linked upthread suggests otherwise, and even appears to provide a counter-example to your claim.
> You'll notice my address() trick to get the address of a function or variable that I used in my previous exploit. We're relying on package fmt and its %p format specifier which is implemented by using the unsafe package, so we don't need to use it ourselves.
unsafe code is not required or used. The exploit constructs a confused interface value, whose vtable refers to a different type than is what is stored. The result is that Go treats an integer as a function and jumps to it.
The unsafe address() is used to get the address of a function to store in the int, to demonstrate the potential for shellcode injection. But you could just as easily assign pp := 12345 and produce a SIGSEGV or SIGILL, which would not be possible if Go were memory safe.
On the contrary, if Go was unsafe that would just corrupt memory and the program would just carry on without anyone noticing the memory was corrupted until it was too late, in another completely different stack.
The exploit was only possible thanks to using the unsafe package, even if indirectly.
It is exactly because Go is memory safe that a SIGSEGV or SIGILL is produced the exact moment the memory gets corrupted and not 2 hours later in a complete different stack trace.
So? The code they wrote doesn't use `unsafe`. So presumably, you need to modify your claim to this then I guess?
> A crash on out of bounds will always happen, unless unsafe package is being used to subvert memory accesses, or if you've used something that exposes details that only the unsafe package exposes, such as fmt's %p specifier.
I mean, you are like, the ultimate pedant, so I figured you'd uniquely appreciate this.
You can't just say, "use unsafe or anything that uses unsafe," because it's turtles all the way down. You've got to draw that line somewhere. If the %p specifier really is required for an exploit and there's nothing else that exposes that (sans unsafe), then that's great news! But is that the whole story?
> The motivation is that @stapelberg spends way more time on C software and their various issues than he would like. Hence, he is going Go-only where feasible.
This is so cool! I’m very happy to learn an effort like this exists, and looking forward to perusing the code for it... in Go (a language I really enjoy reading).
Super nice. Glad to see A/B partiton updating scheme, that is the way to go on appliance / embedded device.
Want something like this for one of the languages that I know... Node.js, Python or Rust.
I'd include C++11, though for this kind of deployment method I think one needs packages easily available in a way separate from Linux distro (since there is none).
The A/B updating scheme is a massive waste of space in most implementations... In many hardware devices, flash storage space for the OS is actually one of the most expensive components, and one of the main reasons that many devices run little hard-to-program embedded OS's rather than linux or something else high level.
Instead, a filesystem like btrfs should be used which can give the appearance of multiple versions, yet only use actual storage space where versions differ.
I'd prefer the root filesystem to be something using a FUSE-like layer which can store format specific binary diff/patches between versions, and then transparently apply (and cache) them on access.
Taking it a step further, I wish desktop linux distros had a package-manager-fs, which is a readonly virtual filesystem which displayed all the files installed by all packages, but the underlying real filesystem only contained all the original compressed packages and a cache of recently used files.
Sure it is quite inefficient. Both in storage and in transmission of updates.
But it is very simple and robust. What one tests pre rollout is identical to what will go on production devices, down to the byte. Each device with same version is identical. It can be cryptographically verified easily. Automatic rollback can be performed on update fail, including self-checks.
This approach of shipping a custom userland without any of the generic Linux userland should drastically bring down the size.
Transmission of updates can be quite small: For the info-beamer hosted (https://info-beamer.com/hosted) OS, we also do A/B booting. Updates are implemented using zsync.
The initial installation of the OS is simply done by unzipping a carefully crafted [1] install.zip file to the Raspberry Pi SD card. By exploiting zsync features, it's quite efficient to reconstruct the original install.zip from the unpacked files. Once that's done, all future updates also use zsync and only fetch changes to the locally stored 'install.zip' file. After updating this file we then unpacked everything again to the next A/B partition, so the exact state of the system is always known. Updates are quite small that way. It also helps that the complete OS is only ~37MB compressed at the moment.
[1] The root file system is squashfs and is stored uncompressed in the zip file to avoid double compression and allows zsync to use the squashfs blocksize to find a minimal set of changes. Similarly the initramfs.gz is compressed --rsyncable and also also stored uncompressed in the zip file.
I’d love to see this, but note that you’ll need to use a pure-Go graphical toolkit (or implement all graphics yourself), and likely interface the touchscreen yourself, too. If you’re not comfortable with this, it might be better to use something else.
Does anyone know of a good tool for building custom PI distributions based on the Raspbian / Raspbian Lite or other distributions that does not involve forking the official image builder and hacking it up? My ideal tool would allow me to do this with a Dockerfile and just export the image after the configuration is complete.
89 comments
[ 3.6 ms ] story [ 158 ms ] thread"For a long time, we were unhappy with having to care about security issues and Linux distribution maintenance on our various Raspberry Pis."
But-- there are so many applications for having a full IoT Linux distribution that I can't believe this isn't already a solved problem. I'm imagining an ultra-stable distro that is easy to deploy to a headless machine, handles security updates, handles flaky wi-fi issues, tests the SD card (I've had more than a few burn out on me), and automatically logs to the cloud for easy debugging.
In other words: Why am I still maintaining my RasPis as if they are apache servers from the 1990s?
It's container-based. Works super-well on a Pi3.
Sad but true. There's still a few companies selling microkernel-based solutions in defense and mobile with Linux user-lands. They'd OEM license their stacks for a new set of products aimed to do this. They cost a fortune. Whoever pays it wont get it back, though, cuz folks dont usually pay for security in that volume. So, nobody is doing it for small players for now.
If the system only contains the Linux kernel and your software you only have to worry about your security issues (unavoidable) and Linux security issues (also difficult to avoid - too much work to write a new kernel too, plus remote Linux vulnerabilities are quite rare).
A big part of the motivation for gokrazy is to use a safe, high-level programming language.
This allows us to re-implement the userland with small amounts of code, see e.g. the NTP client at https://github.com/gokrazy/gokrazy/blob/master/cmd/ntp/ntp.g...
Replacing the implementation language eliminates whole classes of attacks, while stripping down an existing Linux distribution can only go so far. Plus, it’s not clear to me how one would maintain such a stripped-down distribution in a way that isn’t a lot of effort.
The easiest solution to me seems to be to grab the tools you need from the existing userland, instead of reimplementing them in Go. Currently you have reimplemented some of the userland, and packaged it up into a distro. So you have to maintain a stripped down distribution and the tools that you reimplemented. If you just took existing tools you would only have to do the latter. Am I missing something here?
Have a look at the git history to see how little maintenance is required in practice. I have barely had to touch the code over the last year of the project. Just incorporating new upstream versions of a typical Linux userland would have been more work.
Also note that “distribution” is a rather big word for the few files we’re talking about:
/mnt/gokrazy-root % find -type f ./cacerts ./gokr-pw.txt ./gokrazy/dhcp ./gokrazy/ntp ./gokrazy/init ./user/hello ./localtim ./hostname
If my reasoning doesn’t convince you, so be it. Not every project has to make sense to everybody :). I wanted to do a pure-Go userland, and I’m happy with the amount of work this results in and the properties that come out of this decision (easily automated updates, massively reduced attack surface).
https://en.wikipedia.org/wiki/User_space
When your computer runs headless, the init/process tree system is userland. Background (cron) jobs are userland, daemons are userland. Etc.
... that will have security issues, require updates, distribution maintenance, and all that stuff they claim is unacceptable ?
Updates are easy to fully automate with gokrazy (and I’ve been using that mechanism for a year at this point) because assembling a new image and installing it over the network is just one simple command. New kernel and firmware versions are provided automatically once they are verified to boot on real hardware.
There is no maintenance aspect in the actual installations (i.e. you don’t need to SSH into them, ran commands to update, etc.) — the root file system is read-only, and an update overwrites it completely. Unless the applications you want to use with gokrazy introduce state, gokrazy defaults to being stateless.
Hope that clarifies things, let me know if you have further questions.
https://news.ycombinator.com/item?id=13792287
(The gokrazy and u-root people all work at Google, I just contacted them to see if we can help each other. Not a competing project.)
There are a few Basic, Pascal, Oberon and Java (not Android related) options available, but only a minority cares for them.
Still, big enough to keep those vendors doing business.
Having a minimal bare metal runtime for Go would also be nice, but it is a gigantic effort to make it portable across all types of hardware.
gokrazy generates strong passwords by default and listens only on private IP addresses by default to reduce the attack surface.
Nerves is based on a Buildroot core but most of the system is built with Elixir and BEAM/OTP. It's fantastic having a high level language built for robustness and reasonable security. Still, I have dreams of porting BEAM to Redox OS one day [2] and replacing the C userland-bits with Rust-bits. Especially if you could represent the micro-kernel services as actors. That would be amazing!
1: https://nerves-project.org/ 2: https://github.com/redox-os/redox
[1]: https://www.tockos.org/
Google has created a stack called NERF (Non-Extensible Reduced Firmware) for this:
https://firmwaresecurity.com/2017/07/23/google-nerf-non-exte...
There's a presentation about it on youtube:
https://www.youtube.com/watch?v=iffTJ1vPCSo
https://github.com/gokrazy/kernel/blob/3ecdc901da51c2c5b6bf7...
Why is this choice better than a language that provides memory safety using linear types so you don't even need a runtime or garbage collector?
Of course I used my favorite programming language for building gokrazy :).
I love your stuff! I enjoyed your episode on Go Time as well!
Cheers!
I can't resist while here thanking you for i3. I've been a mac user recently, but will likely be back to linux in the next year or so. Using i3 again will be probably be the greatest pleasure involved in that return ;)
gokrazy’s unit of application is a Go package, so you’d need to write a Go program which then executes the Lua VM. I don’t know what dependencies Lua has — if it cannot be compiled statically, I’d recommend against using Lua in this context.
Not really a huge number, no. Java is probably the most obvious, if you count that as compiled. Maybe Haskell or D? I suppose even Swift of Objective C if you include RC in the definition of GC.
I would say the focus shouldn't be entirely on GC, however. Safety as a whole is the most important factor, and it's possible to be say without GC, as you point out. I reckon Go is actually quite an attractive language for this sort of application regardless – quite safe, quite lightweight, good libraries.
Also multithreading gets some improvements which will take more of the manual sync details away from the business logic. (http://www.modernescpp.com/index.php/multithreading-in-c-17-...)
It's still delivered attached to a foot-gun, but you could do worse.
On safe languages an out-of-bounds will cause a crash on the access point, while on unsafe languages it will silently corrupt memory.
If you have an out of bound in C, your program keeps running in 9/10 cases. In Go you'll cause a panic in 10/10 cases which you can either handle and propagate as error or let the program crash entirely, the memory will be in a defined state if you choose to recover (your array might be garbage but you won't have written data into random memory belonging to other threads)
The %p formatting used to gain access to the pointer of a variable uses the unsafe package, the formatting itself is incredibly rarely use to begin with.
I don't see anybody doing any security-related exploits doing that unless you explicitly write for it.
Go won't do out of bounds, edges cases do happen. You can find the same stuff on Java and Rust (yes, I had OoB in Rust, though on a baremetal environment without OS, even Rust ain't safe.)
> You'll notice my address() trick to get the address of a function or variable that I used in my previous exploit. We're relying on package fmt and its %p format specifier which is implemented by using the unsafe package, so we don't need to use it ourselves.
The unsafe address() is used to get the address of a function to store in the int, to demonstrate the potential for shellcode injection. But you could just as easily assign pp := 12345 and produce a SIGSEGV or SIGILL, which would not be possible if Go were memory safe.
The exploit was only possible thanks to using the unsafe package, even if indirectly.
It is exactly because Go is memory safe that a SIGSEGV or SIGILL is produced the exact moment the memory gets corrupted and not 2 hours later in a complete different stack trace.
> A crash on out of bounds will always happen, unless unsafe package is being used to subvert memory accesses, or if you've used something that exposes details that only the unsafe package exposes, such as fmt's %p specifier.
I mean, you are like, the ultimate pedant, so I figured you'd uniquely appreciate this.
You can't just say, "use unsafe or anything that uses unsafe," because it's turtles all the way down. You've got to draw that line somewhere. If the %p specifier really is required for an exploit and there's nothing else that exposes that (sans unsafe), then that's great news! But is that the whole story?
This is so cool! I’m very happy to learn an effort like this exists, and looking forward to perusing the code for it... in Go (a language I really enjoy reading).
Want something like this for one of the languages that I know... Node.js, Python or Rust. I'd include C++11, though for this kind of deployment method I think one needs packages easily available in a way separate from Linux distro (since there is none).
Instead, a filesystem like btrfs should be used which can give the appearance of multiple versions, yet only use actual storage space where versions differ.
I'd prefer the root filesystem to be something using a FUSE-like layer which can store format specific binary diff/patches between versions, and then transparently apply (and cache) them on access.
Taking it a step further, I wish desktop linux distros had a package-manager-fs, which is a readonly virtual filesystem which displayed all the files installed by all packages, but the underlying real filesystem only contained all the original compressed packages and a cache of recently used files.
But it is very simple and robust. What one tests pre rollout is identical to what will go on production devices, down to the byte. Each device with same version is identical. It can be cryptographically verified easily. Automatic rollback can be performed on update fail, including self-checks.
This approach of shipping a custom userland without any of the generic Linux userland should drastically bring down the size.
A minimal gokrazy root file system is 16MB in size. On a 8GB SD card, I certainly am willing to sacrifice another 16MB to get A/B updates :).
The initial installation of the OS is simply done by unzipping a carefully crafted [1] install.zip file to the Raspberry Pi SD card. By exploiting zsync features, it's quite efficient to reconstruct the original install.zip from the unpacked files. Once that's done, all future updates also use zsync and only fetch changes to the locally stored 'install.zip' file. After updating this file we then unpacked everything again to the next A/B partition, so the exact state of the system is always known. Updates are quite small that way. It also helps that the complete OS is only ~37MB compressed at the moment.
[1] The root file system is squashfs and is stored uncompressed in the zip file to avoid double compression and allows zsync to use the squashfs blocksize to find a minimal set of changes. Similarly the initramfs.gz is compressed --rsyncable and also also stored uncompressed in the zip file.
I really like Go, and suspect a Go app would beat the pants off anything powered by Electron (performance wise).
In case any of periph.io’s features require changes to the kernel configuration, I’m happy to change it accordingly.
Apologies if this is not helpful.
[0] https://nerves-project.org