61 comments

[ 0.19 ms ] story [ 137 ms ] thread
The "S" in IoT stands for security. :-/
Reminder for everyone here who ever feels "Imposter Syndrome":

If you built an IoT Device, would you ever:

- Not consider changing the root username and password?

- Not consider shutting off the SSH server?

- Not consider writing the most delicate parts in a memory-safe language?

- Not consider enabling Secure Boot or tamper detection?

- Not consider building a system for easy and automatic security updates, perhaps with an opt-out for those interested?

Almost all of us here would consider the above. Yet most IoT manufacturers never thought of any of them. You're competent.

"Yet most IoT manufacturers never thought of any of them."

It's possible they did think of these and just didn't do them for various reasons. Like it wasn't worth the cost (not necessarily due to changing the setting, but more from a support side), or they might have an incentive to leave it unsecured (Chinese tech), or something else.

The sad part is, the people making the decisions don't have imposter syndrome because they're making money with a success product and any shortcomings must be from the people under them.

Hey boss lets secure this system. We need to do these 10 things.

Sure thing... How long will it take and will it get in the way of these 4 features we promised to the customer that they wanted last week?

Probably like about a month of work.

Well add it to the backlog and we will get to it eventually.

Firmware engineer here with an opinion from down in the trenches:

The reality is that it takes a ton of work to get a microcontroller to blink an LED. Various clocks need to be configured, various peripheral devices (I2C controllers, GPIO controllers etc) need to be configured, interrupt vector tables need to be set up and so on.

Who is expected to do this work?

Today, it is done by the semiconductor vendor. When you buy a microcontroller, you’ll use an SDK and a toolchain that the vendor offers.

The vendor is incentivized to keep their SDK and tool chain as general as possible to sell into as many different designs as possible.

The OEM wants to ship a product as quickly as possible to make their cashflow model work.

There’s a huge gap between these two ends where a ton of important stuff goes unaddressed. At no point are the OEM and semiconductor vendor incentivized to sit together and addresses all the issues that fall through the cracks.

The current school of thought dominating MBA courses would see addressing these issues as a cost with no tangible benefits to the business. I can see their rationale: if consumers don’t signal that they care about security, then it is actually hard to argue that addressing security is a benefit to the business.

Most consumers today don’t know how bad the situation is to care about this stuff. It sucks because we firmware engineers want to do the right thing but we can’t justify it.

> The reality is that it takes a ton of work to get a microcontroller to blink an LED. Various clocks need to be configured, various peripheral devices (I2C controllers, GPIO controllers etc) need to be configured, interrupt vector tables need to be set up and so on.

No, that's the easy part (even if done manually). The hard part starts with networking

> There’s a huge gap between these two ends where a ton of important stuff goes unaddressed. At no point are the OEM and semiconductor vendor incentivized to sit together and addresses all the issues that fall through the cracks.

Most SDKs I've seen end up somewhere at level of TCP stack, sometimes at level of simple web server. Because everything above will be very application specific.

It's near-entirely on the app vendor to fuck stuff above that. Don't try to push blame on manufacturer of the chip here, they didn't force your password to be 12345678

But SSH etc are totally inappropriate for embedded devices (microcontroller level) anyway. Suddenly you need to deal with either a clock battery or secure NTP on boot . . . This is opening the pandora’s box of replays etc.

Frankly this whole idea the same protocols that are used on the web should go anywhere near microcontrollers is braindead.

Edit: apologies if this was too strongly/inappropriately worded, but I stand by the underlying point.

Not so much microcontrollers, but IoT devices. That's the reality we live in - as devices grow more complex and hardware becomes more mass produced and less expensive, commercial interests will find their way to just including a linux kernel with everything. And once that happens, ssh will be bundled with it.

sshd has no place on most of these devices. But it comes packaged by default in many distributions. Real products are shipping with it right out of the box, and it's a problem.

This right here is why I am a rabid fan of Apple's vertically integrated way of doing things.
I needed a computer to sit in a corner and run some application, performance irrelevant. I had a Mac Mini that Apple calls "obsolete" and doesn't support, but it's really a PC so I put Linux on it. An old iPhone would have used less power, but "vertical integration" means when the vendor stops supporting it neither can anyone else. Vertical integration is not the solution, it's the problem.

The hardware vendors are providing the firmware instead of a spec that allows you to write your own. Then you're stuck using their software to use their hardware, even if it's awful or they stop updating it.

It's perfectly fine to provide a reference implementation, but that's not a substitute for hardware documentation sufficient to let you make your own if you don't want the vertical integration.

Because the party with the economic incentive to make it better is the end user. Not all of them have the capacity to do that, but only one of them needs to and you have better code available to the whole world.

> The current school of thought dominating MBA courses would see addressing these issues as a cost with no tangible benefits to the business. I can see their rationale: if consumers don’t signal that they care about security, then it is actually hard to argue that addressing security is a benefit to the business.

At least in the US, customers are very price conscious and will sacrifice a lot to save a dollar. To frame this from their standpoint, "Why do I have to pay more for ¨features¨ that I don't want?"

I think folks _should_ demand more secure devices, but they also must be willing to pay for it.

(Many folks will immediately respond, "We'll use the government to force companies to do this!" but that can cost even more, as the company provides the extra features and also pays for the bureaucracy.)

I think I'd start by removing, at the hardware level, the ability of the IoT devices to do general computing and networking. They accept authenticated external control instructions, they execute the instructions, they physically cannot do more. Most likely there would be no updates at all. There would also be a physical off switch.

My idea is that, if you limit the IoT device - say, a thermostat or a smart bulb - to only have the exact capabilities it is supposed to have, then the damage from any external hack is limited to messing with the user's house. And, if the user has the ability to turn it off, the damage of the hack is limited to "the device doesn't do anything anymore".

I figure this is much better than the possibility that breaking the security enables the IoT device to spy on the user's network traffic, or their house, or be part of a botnet, or a bounce point for hacking attempts.

EDIT: I recognize there are good real-world reasons why IoT companies don't do it this way. If we actually valued security, then there would be an incentive for someone to build up the systems to facilitate this kind of limited-attack-value design.

Given how utterly trivial it is to accidentally achieve turing completeness, I suspect any reasonable subset you could come up with would also be capable of general computation. Moreover, such a design would be inherently expensive and inefficient, because it can't achieve the economies of scale modern uCs do, nor can it do anything without an external (presumably networked) command for every single operation.
The challenge is that even a minimal implementation requires quite a lot of functionality.

Take a Wifi-based smart outlet, for example. The functionality is literally a single output that drives a relay. But in order for that to work, it needs to join a Wifi network - that means that you need some way to get an SSID and password onto the device and it needs a DHCP client to get an IP address. There also needs to be a way for the controller to discover the device's IP address, which means there's now likely an mDNS implementation on the device. Then there needs to be a way for the device to receive commands, so now we've got an HTTP server.

If you want out-of-home access, which I suspect most people do, there's even more work required. Usually this means a central server that the device connects to since most regular people don't have a VPN connection back home. Another option is HomeKit, which is explicitly about local control with all of the remote access magic handled by Apple. But HomeKit is yet more stuff to run on the device, and only works if you've bought into the Apple ecosystem.

This is a ton of scope creep for a pretty minimal device.

A Zigbee or Z-Wave based outlet would be better, but that really just moves all of that scope creep from the individual device to the hub.

> A Zigbee or Z-Wave based outlet would be better, but that really just moves all of that scope creep from the individual device to the hub.

This is actually fine as long as the hub is open source software running on a general purpose computer, because then you're not reliant on the hardware vendor to support it.

The fundamental problem is that the hardware vendors (1) don't provide what's necessary for anyone but themselves to support the hardware, and then (2) don't adequately support it themselves.

Far better to address (1) than (2).

The configuration is a problem here.

You can do exactly what you propose, right now, with stuff like ESPhome. You pre-bake what device does and which WiFi network it uses into image, burn it in, and you're done.

But that's for "us"

Normal user needs a way to get the box to configure to his needs, and do it easily. So suddenly you can "log" onto device and edit its config, and you need to find a way to make sure only legit user does it etc.

And now it is a big blob of code and you DONT, UNDER ANY CIRCUMSTANCE want to have big blob of code that's hard to update in the wild.

- Not consider changing the root username and password?

I would bake it in the bios or similar, and print it on the bottom of the device.

- Not consider shutting off the SSH server?

I would keep sshd running, precisely for the fact that the person who bought this could modify it I'm a reasonable and sane manner, like any Linux peripheral. It also allows for more extensibility past what I thought of.

- Not consider writing the most delicate parts in a memory-safe language?

I wouldn't reinvent the wheel. Use Linux, QNX, or Tron. Probably not Tron, but it IS open source.

- Not consider enabling Secure Boot or tamper detection?

Are we assuming the person who buys this as the enemy, or are we trying to evade Mission Impossible level national spies from looking how some IoT shit works?

- Not consider building a system for easy and automatic security updates, perhaps with an opt-out for those interested?

Most "security" updates are just malware in disguise. I've seen enough de-featurization and others in the name of security.

The issue that needs to be addressed is economic. Devices are paid for once, but security is ongoing work. The Android world has shown that no-one has figured out how to deal with the mismatch.

I'd expect that devices on subscription will do better on security because they have an economic incentive to do so. Even then vendors will want to end of life hardware that is no longer economic to support.

We are too spoiled by cheap garbage and the true cost of these things is hidden.
Yup, all we have to do is make expensive garbage instead, and then the problem would be solved.
"If you want to stop supporting it before 10 years, open source everything including getting the new firmware onto device (so all the keys etc)"

So keep supporting it or give away how your stuff works underneath and allow someone else to. Bricking device into a landfill should not be decision corporation is allowed to take on most (all?) customer devices.

It shouldn't be time-based it should be support-based. As in, the moment you stop supporting your product is the moment you're legally required to publish the source code and any signing keys, tools, etc necessary to use it.
The problem is "oh, we're still supporting it, we just released an update yesterday that fixed an issue with kerning when selecting certain fonts on the built-in LCD. What do you mean 12 critical CVEs unpatched for years?" So now you have to define "supported," which will be lobbied to death by every vendor in the world.
Well, if the company is willing to live with the risks/liability of a still-supported product containing unfixed vulnerabilities then they're clearly not long for this world. Eventually someone will sue--no matter what the support agreement/TOS states--and they will win. Even if they don't it'll still be a pointless large expense for the business.
Then by the time you can see what they actually provided, the company is already bankrupt and if what they provided is non-compliant or otherwise useless, it's too late.

There is no excuse for not providing it from day one. And more than one reason to do it -- not only do you verify that you have what you need before they're gone, it allows people to find vulnerabilities sooner so they get patched before more people have the vulnerable device, and allows them to improve the software in general even when the hardware vendor is in a commodity market with margins to slim too do it themselves.

So any company that makes a device knows that all their proprietary work will become public if they end support -- that seems like a great reason for companies to not build devices.
The FCC actually could have a decent mechanism to force this. The documentation for Part 15 compliance often has a confidentiality request for certain device internals. They could have some uncloaking mechanism where that information is released if the company is no longer supporting the device.
That's a pretty slick idea actually. I've been battling on the upstream reference design angle as I think that's where a lot of security issues truly come from. Release of device internals probably wouldn't help fix those upstream references unless it also covered support the drivers of individual electronic chips from their producers as well, but it definitely would help opensource devs to potentially fix things for devices they love.
This is not viable because what of the vendor used a proprietary build system? They can't open source it.

Setting that aside, I have yet to see a complex system that you can rebuild without a lot of undocumented know-how just by having the source code. Companies don't typically keep meticulous documentation on this, they keep people around to do it. When the product goes out of support, the people get transferred or fired.

It just doesn't work in reality. I work somewhere that makes IoT devices, our microphone noise-cancellation code is written by a third party with a license not to redistribute. They sell their code to multiple companies. If we stopped supporting some device for X years, does that mean we're legally allowed to open source their noise cancelation code?
If you look deeper into the issue, there is a lot of people willing to do volunteer security work to keep using devices they love. I don't think we should have to rely on this and without it I 100% agree with what you're saying.

The other side of the coin, the other option we shouldn't really rely on but we can, is entirely blocked by the way these vendors operate. Android has had its fragmentation issues because the vendors aren't using the stock Android, and aren't upstreaming the drivers and tweaks needed to actually run Android on their hardware. There is usually an economic incentive to not do so (if a custom ROM can be made for their phone, they won't get the ad and deal revenue from the garbage they've preloaded into the phone).

Some do this better than others and there are custom ROMs out there that allow you to update, but custom ROMs are a horrible solution for most customers. It's really only accessible to technical people that have the time and energy to put into it and maintain it directly.

The same thing is true for most IoT devices, the drivers aren't upstreamed, the source is closed, third parties can't fix the software or use it outside the designed ecosystem and the manufacturer doesn't have an economic incentive to give you any time after you've purchased their product.

A lot of manufacturers of devices are mostly white labeling reference designs with a small modification for an extra sensor or other relatively small benign change to the electrical schematic and they'll correspondingly use the reference software for that design which hasn't been updated in 20 years since the reference design was created. Those designs usually have the same problem, the manufacturer of the chip had some devs hand edit a "stable" version of the Linux kernel outside of any source control or public repository and shipped a binary with custom drivers (probably not up to the normal quality expected of code merged into the kernel either).

I think that last one is really the sticking point. The fractured Android ecosystem isn't great but its fracturing issues have been much better in the past 6-8 years than it was up till about Android 8 (based on my recollections) so is kind of an outdated example at this point.

If manufacturers of custom ICs kept reference designs up to date, or upstream their drivers into mainline Linux that becomes the economically cheapest, lowest effort, and widest impact change that could make meaningful security improvements to all devices produced. I have zero idea how to turn that into something that can be legislated though.

> The other side of the coin, the other option we shouldn't really rely on but we can, is entirely blocked by the way these vendors operate. Android has had its fragmentation issues because the vendors aren't using the stock Android, and aren't upstreaming the drivers and tweaks needed to actually run Android on their hardware. There is usually an economic incentive to not do so (if a custom ROM can be made for their phone, they won't get the ad and deal revenue from the garbage they've preloaded into the phone).

That's a really long-winded way to say "Violate GPL of Linux Kernel".

Compiled in closed source modules is explicitly against terms of GPL2, and yet billions of violating portable computers.

And people wonder why I pirate? Our rights are constantly shat on by the same interests that would imprison us for copying software or movies.

Compiled or not matters nothing to GPL2. What matters is whether the code is derivative. And a lot of the problematic blobs are not derivative of kernel code, hell they often don't run in kernel.
The Android shitshow has a major problem: Qualcomm
While I'm sure the FCC has an end game of actually mandating these labels, the vast majority of IoT devices are not exposed to the internet and just aren't a major attack vector in most environments. How much money and time needs to be spent to secure an RGB lightbulb or a wireless speaker?

There is approximately one class of consumer devices that I suppose fall under the IoT umbrella and that are commonly attacked: modems and wifi routers. But these generally get security support. And if you had product labels, would it change shopping behaviors in any way? "This NetGear router will get security updates for 8 years" sounds great. But then, in 10 years, you might have the same router in your closet. Will you even remember the label by then?

If the device isn't internet-connected, it's not an IoT device. That's what the I stands for.

If what you're getting at is that most networked devices sit behind a consumer firewall, and that's probably good enough -- well, I mostly agree.

Disagreeing on being good enough. The problem is that many of these devices regularly poll their mothership for commands and updates. We are one (feasibly, already done but unknown) server compromise away from millions of light bulbs or outlets or whatever turning into a botnet.
Most consumer routers these days are automatically assigning global IPv6 addresses to every device on their network. The only security feature protecting them is the difficulty of (random) discoverability (no firewall rules by default). As in, you can't just scan the entire IPv6 Internet looking for insecure devices as it would take too long (e.g. thousands of years) but if you can figure out their address they're right there, ready for hacking, from anywhere in the world.

The truth is that there's always other ways to find the IPv6 address of various devices inside a home. Many of them will happily tell you if you just send out the right broadcast (e.g. zeroconf) or they connect to services on the Internet that can be spoofed or just have generally terrible security (e.g. the addresses of all devices are publicly discoverable).

Another fun way to find these devices is buying up dead domain names (e.g. because the company no longer exists) and setting up services that auto-hack the insecure devices once they can finally "phone home" again due to the malicious domain suddenly coming back online. This kind of hack works regardless of firewall rules (assuming the device is allowed to "phone home" at all).

Can you give an example of a consumer router that does not provide a default deny inbound (tonight in noun, according to voice transcribe) for IPv6 traffic? I'm not arguing with you, I'm curious. As a network and security guy, it seems like step zero in IPv6 security to have a default deny inbound firewall rule to make up for the lack of NAT.
I've never seen one that did but I've only looked at IPv6 on Netgear and TP-LINK routers. Let's try the other route: Find a consumer router that both hands out IPv6 addresses and blocks inbound IPv6 traffic by default.
There was a CVE for my router which permitted some sort of traffic over IPv6 that should've been blocked. IIRC, it was some sort of malicious firmware update vector, actually. Good times.

I found out retroactively after my router had been pwned and was acting as some sort of shady DNS server. I'll never actually know the method by which it was compromised, but I made a few educated guesses.

>If the device isn't internet-connected, it's not an IoT device. That's what the I stands for

in practice, the I in IoT means that the device connects to your wi-fi. whether that extends to the open web or not, it's still an IoT device, even if it doesn't conform to the word "internet" in the strictest sense

I disagree. Regardless of the use of the word Internet, I argue IoT is a broad term describing devices not traditionally connected to networks which now are.

Just because I build a private network of cameras, power monitors and weather sensors at my house doesn't mean those don't qualify as an IoT device.

In the same way a butterfly is made out of butter.

Many devices work just fine with local only connection. If an IOT devices does not work without internet that is a reason to not buy it.

More like buttercream frosting is made out of butter. Sometimes people make something similar from margarine, but it isn't quite the same, and isn't as common.

But even that analogy isn't perfect, because while buttercream frosting made from margerine is usually cheaper and lower quality, from what I've seen, "IoT" devices that don't depend on remote servers, or send back telemetry tend to be higher end and more expensive.

The FCC commissioner that was here the other day explained that if companies put these labels on their products then they will be liable for cyber security issues which would be enforceable under contract law. The label being a contractual agreement.
> the vast majority of IoT devices are not exposed to the internet

IoT stands for "internet of things". I am by no means an expert in the area, but my understanding was that am IoT device is by definition connected to the Internet.

I started automating part of my house some time ago and used an ezlo zwave hub, which is a small Linux box where you can ssh in. If you change the default 8char password that's printed on the box, it resets to default upon reboot.

That's their answer per why:

> Unfortunately, the current configuration of the controller for networking connections will be taken back for default, including new credential set for SSH connections after a simple reboot because of security reasons

Why wouldn't you go with Home Assistant? When I was researching, there basically seemed to be no other competent solution that worked across devices. The only other reliable solutions seemed to force you to lock yourself into Apple or Google's ecosystem
I wish there was better designed competitor. I don't like the all-in-one blob architecture as any upgrade is pretty much "your automation will not work during it", and there is no good way to run it in HA.

I'd like to see some that have separate "rule running engine hub" (that can be run in some kind of active/passive HA) + "UI/editor" (that if required can be run on separate machine so the "hub" could run on even weaker hardware if needed)

Mostly cause I wanted something boxed where I don't have to invest too much time.

But yeah I regret that decision now.

Because of security reasons they're reverting to a default password? That has to be the most insane thing I've read in a while...
Yeah, that was my answer

> “for security reasons” is a pretty bad reason to reset the password to a default 8 characters printed under the box!

Glob Love Karl Bode for getting to the meat of the issue,

for not gushing with tech-journo love for the subject of the article,

and for not gifting endless benefit of the doubt for Gov/Corp.

The main issue caused by vulnerable devices is that they can be used in DDoS attacks. If there were no attacks, there would be almost no merit to hack those devices. So why don't we take measures against DDoS attacks instead?

What I suggest is to make all Tier-1 ISPs adapt a protocol that would allow any host to block incoming traffic from specific IP/subnet at all upstream providers. So during attack the malicious traffic would be blocked at source network level or at Tier-1 level and never reach the target.

Wouldn't networks be more secure in this case? Critical infrastructure like banks or government services would be better protected. Also, middlemen like Cloudflare would become unnecessary.

What surprises me is why we use technologies that allow to perform DDoS attacks with mimimun costs. ISPs turn a blind eye to this and route malicious traffic to the victim without trying to stop it. Security updates for IoT devices won't fix this. ISPs are the ones who need to be fixed.

Also, if IoT devices were restricted to local network, then outdated firmware wouldn't be a problem too.

> What I suggest is to make all Tier-1 ISPs adapt a protocol that would allow any host to block incoming traffic from specific IP/subnet at all upstream providers.

Cool story bro, and this is authenticated and secured, how? What happens when malware starts black-holing Google and Microsoft for all my devices? What happens when a botnet starts black-holding Cloudflare for all their neighbors? You want to prevent a floody-floody DDOS by creating opportunities for another variety of DDOS?

> What happens when malware starts black-holing Google and Microsoft for all my devices?

You will send an unblock packet. Well, maybe this protocol should only work for commercial IP addresses, not for residential.

> this is authenticated and secured, how?

By your source IP address. You can only block traffic to your own IP address.

> By your source IP address.

Hey, 1985 called, and they want their authentication methods back.

I would like involuntary labeling for:

1. When the device manufacturer will discontinue support.

2. Up front disclosure of subscription fees if required to make the device work.

3. Software or hardware locks that prevent the user from modifying the device's firmware.