I have to wonder just how relevant all of this effort is anymore. They're talking about faster reads and saving space, but the savings is seven megabytes. And even then it comes at the cost of leaving dead space underneath your kernel space in main memory where the old compressed kernel used to be.
They're doing all of this on systems frequently running off of 16 or 32GB flash devices. The difference in cost between manufacturing a 16MB flash device and a 512MB flash device these days must be minuscule.
It's cents... but consider this. Linux is run on devices embedded in just about everything. Places where even doubling from 128 to 256MB of flash ROM could seriously hurt margins. So as the old saying goes.. a penny here and penny there sooner or later you're talking real money. And it is real money sometimes millions of dollars in margin.
When I made consumer electronics our production team would swap parts based on cost. 1c on a million units is $10,000. They only had to save a few cents a year each to easily pay for themselves.
The rest was margin for the business.
Saving cents on a $80 BOM (bill of materials) may not sound like much but it really does add up when you're doing volume manufacturing.
Exactly this, you can even look at the old consoles and see the same thing. They were built to a VERY specific margin and tuned for the same. There is as best I can tell no fat in any of the systems shipped except newer consoles and the original xbox. The only reason I'd say there is 'fat' in newer consoles is because they have much wider requirements than old ones do.
And if I learned anything from the recent fiasco over BIOS support for AMD's Zen 3 chips, it's that 128 megabit flash ROMs have massive availability and widespread use. That specific size is very cheap, and going above it is disproportionately expensive.
And at that size, compression more than doubles what you can pack alongside the kernel.
The bootloader's driver for your storage device is often very simplistic. Eg: it might only read one sector at a time, might not have the FAT cached in RAM, etc. It is not designed for speed - only for simplicity and size. So, reading a few MB fewer from storage is often actual time savings.
Plus there are things like [e]MMC's "boot" area, where a special storage area is simpler to read using a simpler special protocol. That area has a limited size.
What kind of time savings are we talking about for 7 megabytes here? A second? Three seconds? Or is it more like 20ms? Is it still common to source components from 1995 in these devices?
20ms is a long time in low-level systems programming, especially where concurrency might be low or limited. You only need to find five 20ms savings opportunities to shave a tenth of a second off of your boot time.
Edited to add: A lot of performance loss for large, complex systems is death-by-a-thousand-cuts -- 20ms here and there can make a real, user-perceivable difference.
In case of the PinePhone (which is arm64) the actual loading of the kernel image from eMMC can take a second because the MMC controller isn't initialized for max speed by the bootloader, so having a smaller kernel saves some time between pressing the power button and having the splash screen show up (since the bootloader can't control the display)
I don't have exact numbers on this any more. It's been couple of years since I last worked in embedded storage and boot optimization.
When optimizing boot time of an embedded system something like 100 ms is a significant saving once you've picked all the low hanging fruit in the user space init side. In the end, every millisecond counts as it all adds up when there are hundred or so places where they may end up being spent needlesly. An embedded product, especially automotive, might have rather strict boot time requirements.
The time it takes for a bootloader to load a kernel image can be a decent part of the boot time. The bootloader most likely won't be running the SDHCI controller in any of the faster modes that require implementing bus training, calibration, and tuning algorithms of varying complexity.
I mean just today I had to contemplate whether I could afford including a printf implementation in my code. Granted, this thing does not run Linux, but there's plenty of resource-constrained systems out there that do.
It might (or might not) be less relevant, but it's also to a great extent already-expended effort. Arm kernels have had a built-in decompressor for over a decade, so the bulk of the effort went in many years back when the gains were clear and worthwhile, and now while there's some maintenance cost to keeping it rather than just ripping it out, it's not a big cost. And dropping the decompression has the catastrophically huge cost of breaking an enormous number of already-deployed systems when they update to a decompressor-less kernel.
Incidentally, aarch64 kernels do not include a decompressor -- if you want to compress the kernel image then it is the job of the bootloader to uncompress it. Different primary market, different time, different constraints, opportunity to choose differently, different outcome...
For NAND flash, maybe the difference is miniscule. For NOR flash, which is what many small embedded devices run from, the difference is huge:
128Mbit (16MiB): 1.01$
2Gbit (256MiB): 16.38$
2Gbit is the highest commercially available package (with very few suppliers, so it carries extra risk).
Pretty much every security camera, most of the networking equipment, action cameras and set-top boxes run Linux from NOR flash (cost and simplicity reasons- no FTL, IIRC better retention). Altogether it's probably billions of devices.
There are also quite a few lines of MCUs without built-in flash memory and those typically boot from external NOR flash.
NAND is the new industry standard, but most embedded systems, several billion units, produced in the early 2010s were powered by NOR flash, and many use Linux. One example was the vast majority of home routers. I once replaced the 4 MiB NOR flash in my home router to 16 MiB for OpenWrt (yes, you can run Linux with 4 MiB of NOR flash, this configuration was officially supported by OpenWrt until it's retired in 2019).
See also: The most expensive chip in the ARM world by Marcin Juszkiewicz [0]. It's written by a core AArch64 developer on the lack of a full-function boot ROM on most ARM single-board computers, forcing developers and users to fight with vendor-specific bootloaders.
> So SBC… Which means playing with U-Boot. In the past it meant random fork of it. [...] you have bootloader(s) built. The next step is taking microsd card, finding out which offsets to use. Then put bootloader(s) at proper places so CPU can read them and boot. Usually it also means that you have to use MBR style formatting [...]
The reason? A EEPROM chip is too expensive, an astronomical cost of $3! The author was half-joking, but only serious - $3 is really expensive for most single-board computers.
> The solution is SPI flash chip. But it is very expensive - 3 (three) EUR for 512Mb (64MB) chip (if bought 1000 of them). Far too much for 99.99% of SBC probably. Probably 64/128Mb will be enough even. And they are cheaper (128Mb is 1.7 EUR). Still too expensive…
> I have to wonder just how relevant all of this effort is anymore
Note that this is ARM32, which is more or less a legacy platform, raising the concern of whether compression is relevant anymore is irrelevant since it's legacy already. As the article says,
> ARM __traditionally__ uses compressed kernels. This is intended as a comprehensive rundown of how the Linux kernel self-decompresses on ARM 32-bit __legacy systems__.
Legacy, but how many devices do you think are still functioning with them now? How many are sitting on store shelves?
I wonder if the Google Home next to me is aarch64. The "new" router that is sending my packets to this site is MIPS on an even more ancient software stack.
This doesn't change my argument. Legacy can be interpreted as something that will be phased out in the future, but if you want, you can also think of it as the established status quo, which is hard to change. Whether the compressing scheme should be removed is of no interests - if it's not broken, don't fix it, just keep it as is. If a new ARM32 system is to be designed, it's likely that the earlier limitations such as Flash space is still a factor, so the compression is here to stay.
It is so sad to see gzip still used these days. If boot time matters, how much faster would it be with lz4 or even zstd? Or are those actually used now, on ARM bootloaders?
33 comments
[ 3.2 ms ] story [ 60.5 ms ] threadThey're doing all of this on systems frequently running off of 16 or 32GB flash devices. The difference in cost between manufacturing a 16MB flash device and a 512MB flash device these days must be minuscule.
When I made consumer electronics our production team would swap parts based on cost. 1c on a million units is $10,000. They only had to save a few cents a year each to easily pay for themselves.
The rest was margin for the business.
Saving cents on a $80 BOM (bill of materials) may not sound like much but it really does add up when you're doing volume manufacturing.
Once, the production team swapped a component as part of a running change and it took a programmer nearly a month to write the new driver.
I think we still came out ahead in the end but there was some worry along the way as it wasn't clear how long it was going to take.
And at that size, compression more than doubles what you can pack alongside the kernel.
Plus there are things like [e]MMC's "boot" area, where a special storage area is simpler to read using a simpler special protocol. That area has a limited size.
Edited to add: A lot of performance loss for large, complex systems is death-by-a-thousand-cuts -- 20ms here and there can make a real, user-perceivable difference.
When optimizing boot time of an embedded system something like 100 ms is a significant saving once you've picked all the low hanging fruit in the user space init side. In the end, every millisecond counts as it all adds up when there are hundred or so places where they may end up being spent needlesly. An embedded product, especially automotive, might have rather strict boot time requirements.
The time it takes for a bootloader to load a kernel image can be a decent part of the boot time. The bootloader most likely won't be running the SDHCI controller in any of the faster modes that require implementing bus training, calibration, and tuning algorithms of varying complexity.
Incidentally, aarch64 kernels do not include a decompressor -- if you want to compress the kernel image then it is the job of the bootloader to uncompress it. Different primary market, different time, different constraints, opportunity to choose differently, different outcome...
2Gbit is the highest commercially available package (with very few suppliers, so it carries extra risk).
There are also quite a few lines of MCUs without built-in flash memory and those typically boot from external NOR flash.
> So SBC… Which means playing with U-Boot. In the past it meant random fork of it. [...] you have bootloader(s) built. The next step is taking microsd card, finding out which offsets to use. Then put bootloader(s) at proper places so CPU can read them and boot. Usually it also means that you have to use MBR style formatting [...]
The reason? A EEPROM chip is too expensive, an astronomical cost of $3! The author was half-joking, but only serious - $3 is really expensive for most single-board computers.
> The solution is SPI flash chip. But it is very expensive - 3 (three) EUR for 512Mb (64MB) chip (if bought 1000 of them). Far too much for 99.99% of SBC probably. Probably 64/128Mb will be enough even. And they are cheaper (128Mb is 1.7 EUR). Still too expensive…
[0] https://marcin.juszkiewicz.com.pl/2020/01/29/the-most-expens...
Note that this is ARM32, which is more or less a legacy platform, raising the concern of whether compression is relevant anymore is irrelevant since it's legacy already. As the article says,
> ARM __traditionally__ uses compressed kernels. This is intended as a comprehensive rundown of how the Linux kernel self-decompresses on ARM 32-bit __legacy systems__.
I wonder if the Google Home next to me is aarch64. The "new" router that is sending my packets to this site is MIPS on an even more ancient software stack.