I don't understand all the worry about being able to modify the firmware of a computing device if one has physical access to it. If you have physical access then you can completely replace the computing device with one under the attacker's control.
Raspberry pi's might not look professional but they are easy to procure and develop with whilst only adding a small amount to a BOM. Even when using a different SoC very few will be setup to use a fully secure boot process.
The original vulnerabilities found were mostly in the web apis of the servers that these chargers connect to which makes vulnerabilities in the hardware irrelevant.
I agree that the 'o noez, raspberry pi!' is high here, but one of the things you can do on other devices is something like secure boot, where the OS image on disk is signed and the device will boot to a 'rescue' firmware if it is compromised. Aside from physical access deterioration it also has the effect that an attack over the Internet generally cannot get permanence, that is, you can get rid of it by rebooting the device.
In theory, sure. But most "real world IoT" devices are appliance-like and seldom designed for the real world nor for non-technical owners. I'd estimate the number of IoT devices with "good and wise security in the design" at being far smaller than 0.1%.
Most people designing IoT are NOT CompSci people who remember Therac-200 in class. Most are EEs who often have never had more than one or two programming classes and often often have had no formal training at all. Never mind "security theory" courses. Never mind case studies around that.
It's like expecting the average man on the street to be able to perform brain surgery.
Worse: no employer is willing to pay for the talent to do the job right. They want it to "work well enough" and "on the cheap" at the same time. Once it's sold, any security issue is someone else's problem (even if it's someone in the same company).
And I'm saying this as a degreed EE (I just happened to get into software long before I went to engineering school so I know what I don't know).
> Selling EV charging products based on Raspberry Pi, which is a prototyping board, is unwise.
What's the difference between a prototyping board and a product built using the lessons learned while integrating the prototyping board? The compute module is intended for use as an embedded board that simplifies development by mass producing the more difficult, expensive parts of embedded computers and letting the end user install it in a product. It's intended for exactly this sort of use case.
> When both device manufacturers were contacted by the white hat penetration testers who conducted the penetration testing, the response was typically reactive seeing the device makers put in place a new server, app updates and firmware updates to the chargers. This is symptomatic of the general lack of strategic security thinking that goes into the manufacture of many embedded devices today. For example, one of the chargers had a Raspberry Compute Module 3 (CM 3) which lacked the necessary security features for such a use case.
They updated with patches for the vulnerabilities. What more do you want?
It's not like there are two kinds of computers, on one side insecure prototyping computers like Raspberry Pis that can be made to do unintended things, and on the other side secure IoT EV charger computers that only do what they are intended to do. There are only EV chargers that are controlled by general-purpose computers.
If you want a material example, the raspberry pi's SOC is designed to be mounted in read only mode with industrial on-board flash. It is meant to be used with the broadcom toolkit that includes multiple boot partitions for safe and validated updates.
In addition, (afaik) that SDK includes a hardened and stripped down linux kernel with watchdog, chain of trust and binary signature verification - a far cry from the raspberry pi's kinda "open like whatever" standard desktop and server grade distro.
A material consequence of this is that raspberry pis (pre 4, at least) that lose power during a SD write have had a common tendency to leave the flash in a really bad state, rendering the device unusable until a reflash happens.
The compute model suffers from all of this. I use the compute modules for lighting and other fun such activities, at art and music festivals. I have gotten used to carrying spare SD cards (for non compute) and a laptop with flash snapshots (for CMs). More than once, a breaker has blown and the compute module wouldn't come up after due to a trashed filesystem.
I will admit, the CMs are wayyy more robust than the normal ones, due to whatever emmc they're using. I could probably mount the flash as RO also but that tends to cause "weird things" to happen. If I have to flash a CM once in a weekend, that's a small price to play
> What's the difference between a prototyping board and a product built using the lessons learned while integrating the prototyping board?
The difference is that the lessons learned likely won't cover the large amount of pitfalls you'd encounter in the field, but those pitfalls still exist.
TLDR: Raspberry pi doesn't include a hardened sdk and distro intended for robust, embedded applications.
The issue is likely the general purpose potentially network attached computer should not be trusted for the safety critical portion of the system such as power control and monitoring. That should be a fully isolated computer with a light weight communications protocol with the host over SPI/I2C etc. The isolated processor can run an RTOS with some deadline guarantees as well then. If the host is compromised, the physical hardware cannot be controlled past trivial (audited) requests issued to the isolated processor.
Also on top of that, not joking but the Pi hardware, even the compute model, is not very good quality compared to typical industrial computers assigned to or designed for these tasks.
The only reason people do this with the pi is to trade safety for cost savings which is a stupid idea when you’re playing with hundreds of amps.
There seems to be this mythos that embedded systems developers are the hallowed priests of high-quality software development and embedded systems as these robust combinations of specialty electronics that can withstand everything up to a nuke dropped nearby.
In truth, embedded systems engineers and the hardware they use/design have the same span of quality you'll find in most other types of software developments outside Safety Critical products.
I've seen some really dumb things done by people who should know better and on the flip side, some systems that I'd happily trust my life to. There's no magic.
There is a risk of a tragedy of the commons situation with EV charging and other wider use of electricity for heat and as a primary source of energy -- there will, in the short and medium term, need to either be massive investment in significantly increasing grid capacity, or greater control of loads being connected to the grid.
Putting more copper in the ground can help remove local constraints on how much energy can flow into a given area at a time, and ensure everyone in the street can put their car onto charge at 6:30pm. But the rest of the grid needs to be adjusted in response - generation will need to be greater for these swings in demand, and this might not align with when renewable energy is abundant.
For large loads like EV chargers (or really anything else beyond basic domestic usage), there's likely to be significant correlation in usage (charging cars arriving home from work, or upon arrival at the office, turning on heating during a cold snap an hour before arriving home, etc.) over time, that the energy networks can't deliver.
Large loads will therefore likely need to be remotely switchable and controlled by a grid controller, taking into consideration local load (to avoid melting the cable in your street and taking everyone offline) and wider regional and national load (if solar production in your region has dropped due to a large storm front passing over, there's less energy to go around). Switchable loads need to be secure, otherwise an attacker who can remotely switch them can do harm to the wider energy network, especially if the vulnerable device is widely deployed and can be trivially controlled. For example, even simply turning off a large number of active loads simultaneously could destabilise the grid and lead to an outage. Done at scale due to vulnerable devices, this could cause widespread problems.
There's a bigger discussion to be had around people's expectations to charge their car instantly at full rate - perhaps differentiated pricing incentives will incentivise users to plug their vehicle in and let it charge at the best price it can find overnight, rather than immediately charging, to spread loads. But measures like this require a secure set of basic infrastructure to control large load devices, otherwise everyone will insist on bringing theirs online immediately, resulting in an outage and everybody being disconnected due to a breaker tripping or a wire melting.
If the price difference is large enough people will all by themselves wait until a certain time to charge their vehicles or do their laundry or start the dishwasher etc.
No fancy centralized control infrastructure is required, The user or the user’s device can decide based on time or price easily downloaded by https when to start the load.
Currently Everybody can use as much power as they want whenever they want, up to the size of their service, and everywhere I’ve lived has not had issues with the street being under provisioned. I doubt everyone charging their car overnight will be too much load.
The bulk electrical system in western North American can already handle instantaneous changes in load on the order of thousands of megawatts. As new generation comes online it will be important to retain that characteristic by pairing solar/wind with storage to firm it up.
11 comments
[ 2.4 ms ] story [ 44.8 ms ] threadRaspberry pi's might not look professional but they are easy to procure and develop with whilst only adding a small amount to a BOM. Even when using a different SoC very few will be setup to use a fully secure boot process.
The original vulnerabilities found were mostly in the web apis of the servers that these chargers connect to which makes vulnerabilities in the hardware irrelevant.
We now have things like platform attestation that - unless your adversary is a state actor - stops some physical attacks.
(Not available rpi right now, but maybe in 4-6 years?)
Most people designing IoT are NOT CompSci people who remember Therac-200 in class. Most are EEs who often have never had more than one or two programming classes and often often have had no formal training at all. Never mind "security theory" courses. Never mind case studies around that.
It's like expecting the average man on the street to be able to perform brain surgery.
Worse: no employer is willing to pay for the talent to do the job right. They want it to "work well enough" and "on the cheap" at the same time. Once it's sold, any security issue is someone else's problem (even if it's someone in the same company).
And I'm saying this as a degreed EE (I just happened to get into software long before I went to engineering school so I know what I don't know).
What's the difference between a prototyping board and a product built using the lessons learned while integrating the prototyping board? The compute module is intended for use as an embedded board that simplifies development by mass producing the more difficult, expensive parts of embedded computers and letting the end user install it in a product. It's intended for exactly this sort of use case.
> When both device manufacturers were contacted by the white hat penetration testers who conducted the penetration testing, the response was typically reactive seeing the device makers put in place a new server, app updates and firmware updates to the chargers. This is symptomatic of the general lack of strategic security thinking that goes into the manufacture of many embedded devices today. For example, one of the chargers had a Raspberry Compute Module 3 (CM 3) which lacked the necessary security features for such a use case.
They updated with patches for the vulnerabilities. What more do you want?
It's not like there are two kinds of computers, on one side insecure prototyping computers like Raspberry Pis that can be made to do unintended things, and on the other side secure IoT EV charger computers that only do what they are intended to do. There are only EV chargers that are controlled by general-purpose computers.
In addition, (afaik) that SDK includes a hardened and stripped down linux kernel with watchdog, chain of trust and binary signature verification - a far cry from the raspberry pi's kinda "open like whatever" standard desktop and server grade distro.
A material consequence of this is that raspberry pis (pre 4, at least) that lose power during a SD write have had a common tendency to leave the flash in a really bad state, rendering the device unusable until a reflash happens.
The compute model suffers from all of this. I use the compute modules for lighting and other fun such activities, at art and music festivals. I have gotten used to carrying spare SD cards (for non compute) and a laptop with flash snapshots (for CMs). More than once, a breaker has blown and the compute module wouldn't come up after due to a trashed filesystem.
I will admit, the CMs are wayyy more robust than the normal ones, due to whatever emmc they're using. I could probably mount the flash as RO also but that tends to cause "weird things" to happen. If I have to flash a CM once in a weekend, that's a small price to play
> What's the difference between a prototyping board and a product built using the lessons learned while integrating the prototyping board?
The difference is that the lessons learned likely won't cover the large amount of pitfalls you'd encounter in the field, but those pitfalls still exist.
TLDR: Raspberry pi doesn't include a hardened sdk and distro intended for robust, embedded applications.
Also on top of that, not joking but the Pi hardware, even the compute model, is not very good quality compared to typical industrial computers assigned to or designed for these tasks.
The only reason people do this with the pi is to trade safety for cost savings which is a stupid idea when you’re playing with hundreds of amps.
There seems to be this mythos that embedded systems developers are the hallowed priests of high-quality software development and embedded systems as these robust combinations of specialty electronics that can withstand everything up to a nuke dropped nearby.
In truth, embedded systems engineers and the hardware they use/design have the same span of quality you'll find in most other types of software developments outside Safety Critical products.
I've seen some really dumb things done by people who should know better and on the flip side, some systems that I'd happily trust my life to. There's no magic.
Putting more copper in the ground can help remove local constraints on how much energy can flow into a given area at a time, and ensure everyone in the street can put their car onto charge at 6:30pm. But the rest of the grid needs to be adjusted in response - generation will need to be greater for these swings in demand, and this might not align with when renewable energy is abundant.
For large loads like EV chargers (or really anything else beyond basic domestic usage), there's likely to be significant correlation in usage (charging cars arriving home from work, or upon arrival at the office, turning on heating during a cold snap an hour before arriving home, etc.) over time, that the energy networks can't deliver.
Large loads will therefore likely need to be remotely switchable and controlled by a grid controller, taking into consideration local load (to avoid melting the cable in your street and taking everyone offline) and wider regional and national load (if solar production in your region has dropped due to a large storm front passing over, there's less energy to go around). Switchable loads need to be secure, otherwise an attacker who can remotely switch them can do harm to the wider energy network, especially if the vulnerable device is widely deployed and can be trivially controlled. For example, even simply turning off a large number of active loads simultaneously could destabilise the grid and lead to an outage. Done at scale due to vulnerable devices, this could cause widespread problems.
There's a bigger discussion to be had around people's expectations to charge their car instantly at full rate - perhaps differentiated pricing incentives will incentivise users to plug their vehicle in and let it charge at the best price it can find overnight, rather than immediately charging, to spread loads. But measures like this require a secure set of basic infrastructure to control large load devices, otherwise everyone will insist on bringing theirs online immediately, resulting in an outage and everybody being disconnected due to a breaker tripping or a wire melting.
No fancy centralized control infrastructure is required, The user or the user’s device can decide based on time or price easily downloaded by https when to start the load.
Currently Everybody can use as much power as they want whenever they want, up to the size of their service, and everywhere I’ve lived has not had issues with the street being under provisioned. I doubt everyone charging their car overnight will be too much load.
The bulk electrical system in western North American can already handle instantaneous changes in load on the order of thousands of megawatts. As new generation comes online it will be important to retain that characteristic by pairing solar/wind with storage to firm it up.
That's my primary problem with ALL "IoT" businesses and products - security is very much and absolutely the last thing considered.