How to handle Linux system integrity and secrets on physical product?

There's multiple steps involved in this, but some of the important ones are:

- encrypt all the contents of your flash (e.g. with LUKS)

- store the key in a crypto hardware module

- Typically you'd then have a System on Chip which supports a secure boot with a fused certificate key as hardware root of trust. Each subsequent phase of the boot process needs to be attested by the previous phase to maintain this chain of trust. Have the boot process pull the key for your Flash partition and unlock the flash

- Ensure your board is fully locked down, so no debug access, no boot overrides, no sideloading, ....

- make sure the key can't be accessed with the same process privileges as the server running on the unit, for when someone inevitably pops it

- select a SoC with protections against side channel attacks and fault injection (glitching), not some 0.3 USD microcontroller

This is a very hard problem to solve. It isn’t possible to prevent a well-resourced determined adversary from getting into such a device (iPhones are some of the most well locked-down devices in the world, and there are still jailbreaks). For ordinary hardware you’re probably looking at using something like a TPM, which can be used to decrypt drives based on the state of the bios and hardware. Your best bet is to have trustworthy customers and strong legal protections. If that’s not good enough, a TPM will be the next best thing. Have a look at the red hat documentation on how to set up a TPM for LUKS. Also be sure to set up secure boot correctly with a custom key and lock the BIOS.

https://access.redhat.com/documentation/en-us/red_hat_enterp...

Edit: I’m reading through your question again. In terms of secrets used to communicate with a back end, it’s very very very important that the secrets used for each device/customer are different, and can be revoked. If you’ve got something like AWS access keys in there, don’t. Use per-user secrets that aren’t the keys to the kingdom.

While there are some good recommendations here, I would defintely hire an expert (either in-house or some external contractor) to design and test the final solution.