Not bad. Could go further and call them "dependent" to make it unambiguous that they are not standalone devices but out of your control and potentially worthless if the connected services close or fail.
I've worked with embedded device makers that had better ideas about building redundancy and failsafe procedures into their products
While software mistakes do happen, it seems any moron these days thinks they can put out an IoT product because they know how to flash a led on Arduino and they make no effort whatsoever in thinking how to make their devices have minimum reliability
Have a user accessible USB port (from the inside) from which a signed payload can be loaded for cases like these. Would have saved several days for their customers and several shipping fees.
Well, good points I suppose. On the other hand if you have a USB port on the thing, what's to keep someone from flashing an update that backdoors the lock? Wait a few weeks, rob the place, rinse, repeat.
Now that I think about it, I hope these devices are validating the certificate of publisher and the updates are signed.
> I hope these devices are validating the certificate of publisher and the updates are signed.
You're joking right? Really, the vast majority of this iot stuff is ridiculously bad in this respect. Plain text data transfer, unsigned updates from any source based on a text message and so on. It's a disaster waiting to happen.
No, I'm thinking along the lines of large vehicle manufacturer doing an over the air update of the firmware of their vehicles bricking all of them, including the ones on the road. That kind of disaster.
USB would add to the BoM. Much easier (methinks) to have the lock revert to the last known good configuration if you operate an I/O on the inside in an unusual way - say, turning the lock thingamajig towards the 'open' direction for >20s or so.
Also, no update should be carried out unless the user initiated it - for instance, a flashing led or an app notification tells the user an update is available - and it needs to be initiated by using the same controls as would be used for a roll-back, ensuring you do not end up locked out after updating.
That being said, the idea of a lock being online scares the bejeezus out of me. Call me a luddite, but as a minimum I want my burglars or vandals to show up, rather than nessing with me from wherever.
You could also do A/B partition booting. Whatever partition is currently running, the other one gets updated. When the update is complete, the system reboots to the updated partition. If it doesn't survive the reboot (like if the system never reaches the app and the hardware watchdog timer stops being reset), then it reboots again, back to the first partition.
Have two bootloaders, one is trivially provable and can always be relied on in case the entire world falls apart, and can never be updated. The second one is updatable if needs be.
Of course, the good thing about the bootloader is it is less likely to need updates in the first place, and it's also likely easier to verify an update with a high level of confidence, as its functionality is so tightly bounded.
Of course. The hardware watchdog is just one of many things that should be taken into account. I picked it as an example, specifically because a lot of people don't know such a thing exists on many embedded systems.
In hindsight I'm sure they will do something different. For folks who are maintaining systems with OTA updates that can't survive an OTA failure, hopefully they now realize they are in the middle of a train wreck and should get to work.
An 8 MB (64 megabits) SPI flash chip[1] is about $2. Two jumpers to power the SPI flash and set the boot/load option is another 3 - 4 cents. These take up less than 30 sq mm of board space. And they give you a tremendous back up in case of failure.
If you're using external flash rather than on-die flash, doesn't that mean you'll have to copy it into RAM before running it instead of running it directly, and therefore more RAM will also be part of the extra cost?
All of the ARM chips I'm playing with these days have a serial boot loader in ROM that you can get to by setting a pin on the CPU and powering it up that way. In such a scenario it would take no extra parts.
I believe they emulate or copy the system used by FPGAs to read a configuration from a dedicated SPI chip. Its well understood and a lot of tools can program the SPI chip to have the right sequence of bits in it to do the right thing.
If I were designing a system with OTA updates, then I would design something like this with it for this very reason.
From looking at the product website, it looks like these locks have traditional keys as well, so I would hope that nobody is locked out of their own house while waiting for their lock to be fixed. The article didn't provide much detail, so I was picturing people getting locked out (or worse, locked in) to their own apartments.
What will it take to get IoT device manufacturers to adopt development practices that prevent this kind of thing happening? It's bad enough that most software is bug-ridden and poorly tested, but it's inexcusable for IoT software if they ever expect IoT devices to replace traditional alternatives.
Agreed. I don't see any major changes happening (WRT IoT insecurity) until there's regulation that requires it. The industry certainly isn't trying to solve this on their own (if they are, they're hiding it very well), they're only worried about getting to market as quickly and cheaply as possible.
I expect this to continue getting worse. At some point, hopefully, things will change and begin to improve. I think we'll see many more -- and worse -- incidents like this before that happens, though.
Affected product owner here, I can clarify. First they do not lock from the inside, so they never prevent people from leaving, there's no issue there. Also they fail closed so at least my venues weren't open for thieves to just walk in (SOMA in SF is not exactly the best place to leave your doors unlocked).
They do accept traditional keys but the whole point of the lock is to delegate restricted access to people at certain times and with logs/notifications. So a physical key is like an admin override, not to be used in normal circumstances. As such, only me and my partner have them, and we generally never use them. In a wonderful example of Murphy's Law we were both out of town this weekend so all the people that were supposed to be coming to the venue to set up for a show were locked out with no recourse, it was a huge mess. Long story short, the use case of these locks renders the traditional key fallback less than ideal.
Frankly, this is why OTA updates for critical infrastructure (locks, heating, phones, vehicles) always scares me. It's too frequent an issue that someone's device gets bricked as a result, leaving folks with no practical remedy in many cases.
I've pushed dozens of OTA updates to WiFi devices (telepresence robots), and it's always terrifying. We'd test thoroughly, and we'd always start with devices in our office, and wait to see them reboot before pushing to customers. In addition, we had a scheme where we'd push an update, it'd reboot, and if it didn't connect to our server within 20 minutes it'd revert to the previous version and reboot again. WCGW? We never bricked one in the field, but I can think of a million ways it could have happened.
The scariest times were pushing updates that affected WiFi connection for customers with unusual network configurations. In particular, we had one with a multi-campus Cisco WiFi system with WPA-Enterprise, which requires logging into a RADIUS server to get WiFi. We were running on FreeBSD which had a half-assed port of the Linux WPA authenticator. We never managed to replicate their network perfectly for in-house testing, so pushing updates to them was always nerve-wracking.
we had a scheme where we'd push an update, it'd reboot, and if it didn't connect to our server within 20 minutes it'd revert to the previous version and reboot again
Was this handled in userland? Or with something lower level (e.g., a hardware watchdog)?
I'm asking because the question has come up a few times recently about making FreeBSD recover (unattended) in the event that an unbootable kernel is installed.
Userland. Basically during upgrades we installed a shell script in /etc/init.d:
sleep 1200
if ping -c 5 central-services.anybots.com; then
rm <this file>
else
cd /home/dist && git checkout STABLE && make install
reboot
fi
(We used git to distribute binaries, so we could checkout and install a previous version easily).
This was a last ditch recovery. I could imagine lots of things that break, but which still allow ping. And installing a previous version over a newer version is flaky, because a newly created file won't be deleted.
An unbootable kernel is a whole different kettle of fish. I believe GRUB can be told to fallback to a safe mode.
I believe GRUB can be told to fallback to a safe mode.
I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.
This will be the time where setup HW watch dog should help.
After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader.
If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader.
As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.
It's fun to poke around in these embeded firmwares. Most of them are strung together with shell scripts for everything but the actual application, this kind of stuff is generally just another script.
Gradual rollout also helps (whenever possible), especially with stuff that has to work on many hardware platforms etc, so you can roll out only a few to a new hardware platform, see that it works, then roll out the rest.
I've been involved with building a similar system. We have resolved to having a backup un-touchable recovery version that is capable of finding it's way to our servers and updating. As well as an onion roll-out process, with exponentially sized groups prioritized by distance from our office, and risk of customer loss due to downtime.
I think this all is a fascinating problem. I'm running a digital signage service for the Raspberry Pi (see profile) and also offer OTA updates. I've initially built my OTA process around the same idea: Keep a bare minimum bootable kernel and minimal userland that is always able to connect to the update servers. In case of a broken update I would simply rely on this part to fetch a fixed version. Unfortunately it turned out that most updates I did also updated the kernel (for new features/bug fixes) and it felt like I superfluously split my system into two parts that both need maintenance and testing. I've since changed to an A/B booting system that automatically falls back to the previous version if it isn't able to reach the servers within a few minutes. It feels way more unified and all I need for it to fall back to the previous version is a working kernel (which, on the Pi, is easy to test), the busybox shell and the first couple of lines of my init script. Everything else might break and the system will revert. It wasn't easy to reach this point but thinking about OTA updates doesn't feel so scary now.
our app updates via the app itself. This gets regularly bricked but we have a Cronjob where we can put bash commands in a .txt file and upload that to our server, and devices will execute it regularly. This has always saved the day but apparently the past developers before me used to drive around with a USB stick on a regular basis, to re-image bricked devices.
I can also confirm the wifi problem. Not knowing if a device is bricked or just on bad wifi is a huge problem you have to somehow account for. It was terrifying before I built tools to keep track of which devices those are
Your end devices execute arbitrary code put in a specific url on your website? I assume this is a purely in-house device and not one in the hands of consumers?
Just in case the domain name lapses. (Or for any of a hundred other reasons why having a system that reads plain text commands from the network is a bad idea)
If someone is not competent enough to keep their domain renewed, what's to say they're competent enough to keep private keys secured, and not lose or leak them. Then again I saw something on here recently about someone hijacking the whole .io TLD, so what you're saying is a real issue.
Slightly, but not exactly. Any remote code execution could lead to "example.com/run_this.txt" being filled with "evil_hacker.sh". However if "run_this.txt" were somehow signed and verified prior to downloading (with signing keys kept off the distribution box) then it's much safer.
Practically, there are tons of instances where executing code downloaded from URL's is how software is updated, but again most of those instances are human-driven, not automated.
Our main issue is if someone tampers with content stored on the server, which gets copied to devices. For example someone replaces a JPG with the wrong JPG that is something offensive.
If someone gains access to our server, its already game over. Being able to execute code on the devices is then moot because we were already hacked & potentially lost all credibility in the industry.
To everyone saying not to execute code at URL, this is how oh my zsh, and many other open source projects are installed. And SSL should prevent any kind of MITM attacks. The point about keeping keys off the server is a good idea, but is not always possible when the server needs to dynamically send content down to the device. The server would need to access the private key in order to sign this content dynamically. For statically built executables this is a good idea though.
All hardware devices that allow firmware updates should have two things:
1. A backup recovery read-only image which can flash everything back to square zero via some sort of hardware over-ride (like a pin-hole button that has to be held down for 10 seconds)
2. A temporary place in memory to buffer and validate the checksum of any new firmware that is uploaded before flashing it.
Real time flashing over spotty WiFi (without any sort of post-transfer checksum validation) has got to be the dumbest, laziest, cheapest, riskiest thing a hardware manufacturer can put out, and any manufacturer that does that should be ashamed of themselves.
All hardware devices that allow firmware updates should have two things:
1. A backup recovery read-only image which can flash everything back to square zero via some sort of hardware over-ride (like a pin-hole button that has to be held down for 10 seconds)
2. A temporary place in memory to buffer and validate the checksum of any new firmware that is uploaded before flashing it.
Real time flashing over spotty WiFi (without any sort of pre-flashing validation) has got to be the dumbest, laziest, cheapest, riskiest thing a hardware manufacturer can put out, and any manufacturer that does that should be ashamed of themselves.
I can't understand it. You can mail them the guts of your lock, have them fix it, then mail it back to you, all in a week or less. Or they can just send you a new one, and it takes at least two weeks? Do new ones come by courier snail from China, or is this just a scheme to discourage people from opting for a replacement, or what?
Ouch. Clearly someone missed a possible outcome on their OTA update flow. Worse, they didn't devise a way that you could locally get the device sane enough to do another OTA. Counts as a double fault :-(
I've pushed OTA updates to smartphones before: there is a rigorous certification process that the software has to go through before reaching the customers. We've had times where we had to push emergency OTAs soon after, but never a time where we had to do any sort of recall due to an OTA.
I'm certain that these vendors don't have nearly the same amount of checks compared to smartphones due to much less overhead (like we had from carriers, Google, SoCs), but it's still astonishing how this could happen, but it's a good lesson for them, mistakes happen and I hope they implement a system where this is not possible to happen again.
This is why I always look extremely carefully at companies that operate machinery (cars, machine tools and other connected devices) and do OTA updates. This sometimes leads to interesting discoveries and blind spots. Who ever thought that updating software connected to the cars' buses without first checking that the car is stationary was a good idea?...
I had a flight get delayed due to needing an OS reinstall on a 737-800, the OTA update failed and they had to send someone out to connect to the USB port to recover the OS. The plane was bricked essentially and couldn't take off. Fun.
That's better than my flight 18 years ago where the onboard computers all died mid-takeoff, we had to abort takeoff and sat on the tarmac for about 3 hours while they sourced alternate computers and did the replacement.
Boy that suggests an interesting attack vector. You didn't happen to see where the USB port to update the OS on the plane is, did you? Hopefully the in-seat USB ports aren't wired to this somehow...
Could have been the Electronic Flight Bag, those can run commodity OSs. I can see an OTA failing there and it needing an OS reinstall. It'd be odd, but not impossible.
Seems like it would save a lot of time, energy, and money if they did (assuming they had the procedure worked out). Push out an update while it's sitting at the gate, powered down, instead of taking it out of service and having to bring it into a hangar (where space is already limited).
Nothing gets changed unless it's for maintenance or airworthiness reasons (or fuel economy)
Updates can be done during scheduled maintenance if there's a good reason for it (see above) or if it's really something urgent it is done overnight if possible or the plane gets scheduled for this specific maintenance
Stuff like this is why I don't own any "smart" devices.
I never think about my deadbolt unless I am opening my door. I never think about my smoke alarm unless starts to chirp because of a low battery. I never think about my light bulbs unless they burn out.
Any "smart" features, like notifications or a phone app, are going to make me spend more time thinking about these products than I care to, even if they are working perfectly. Failed OTA updates make things even worse. That door lock is going to need to introduce an extremely compelling new feature to convince me to spend time thinking about such a basic component of my home's infrastructure.
I think the feature is fairly compelling in this case. Smart locks allow airbnb hosts to grant limited entry to guests without sharing their house keys. Any frequent airbnb renter runs the risk of their keys being duped and used for illicit access later on. While the implementation still needs work (obviously), the core idea is sound.
My neo-luddite instincts told me to agree with parent, but you make a good point. Copying keys is trivial (I used an automated booth in a pharmacy that was android based the other week, cost like 10 bucks for 3 copies) when having to deal with potentially malicious guests adding complexity to the simple act of locking serves a real purpose.
My initial reaction on reading this was "Bahahahah stupid smart locks, internet of shit blah blah blah," but I am compelled to agree with your statement. Any lock is merely a deterrent, and adding complexity to a deterrent inherently increases it's security, especially when the threat profile of a door lock is generally "idiot burglar."
For the most part, adding complexity to anything increases the chance of failure. For average people (who don't generally give out their keys) it's not a great idea, but for airbnb hosts, making a key that is hard to copy is a good idea.
However, I've seen some airbnbs that have a pin door code or a key fob, which are arguably more secure than a regular key lock but have less chance of failure in my experience.
There are lock brands that have service keys that will allow you to pull and change a cylinder, from the outside, without having to take the door/lock apart. A property management company and a set of cylinders to rotate through solves this problem easily.
I have a mechanical deadbolt with a keypad that can be opened by anyone with the right combination. It's handy when I'm not at home but need to let someone in.
For Airbnb, etc., you'd have to change the combination regularly (which is easily done in about 30 seconds) but at least I don't have to worry about getting locked out if I forget my phone, it's battery is drained, cell service is out, etc.
Although not as convenient as a smart lock, I can re-key my kwikset deadbolts to any key that fits the lock very easily using a provided tool: https://www.kwikset.com/smart-security.aspx .
But I've been quite happy with my smart thermostat and sprinkler controller.
Maybe the difference is in these cases, a controller has been replaced with a smarter controller.
It's a natural line of progress, and there have been obvious deficiencies with the logic of the "dumb" controllers since their inception (heating the home when you're away, watering while it rains)
I have to admit, I'm coming around to your way of thinking. I have a bunch of Nest smoke alarms and I think I've reached the point where I just want to rip them out. Too many false alarms, too many emails, too much noise, too many "Hi from Nest!" alerts. It's gotten to the point where my dog goes crazy, my daughter seems to have developed anxiety and my neighbours must all hate me ... all for the sake of a smoke alarm.
Happens to Google Chromecasts kinda often, by a quick Google - and you can't reflash them via USB. Same for phones, especially cheap ones tend to brick themselves on updates, and it's a hassle to get them reflashed.
The solution is simple, though: Use U-boot, two partitions for firmware and a tiny script using a boot counter/flag. Only after successful boot set the flag "do not boot back to old firmware". But doing that well requires recent versions of U-boot, and people are STILL shipping devices with 2012 or earlier versions...
Why are they even pushing out OTA updates for a deadbolt? Who designs a deadbolt that locks/unlocks via expiring codes and thinks "we're going to need to release so many updates for this that we should use OTA updates instead of the customer manually applying an occasional patch."
I own 2 of these smartlocks (for my theater and my speakeasy, it's nice to be able to let performers/crew in on set schedules with SMS confirmation). They have both been down since last week. It's been hugely annoying, especially considering we paid over $600 per lock. They overnighted us new circuitboards that we have to manually replace, but the timing was horrible as both me and my partner were out of town all weekend and couldn't do the repair, so it impacted load-in for a show on Sunday.
These locks have some nice advantages but sometimes I feel like they are way too internet-reliant and a bit outside my control. Furthermore the company isn't exactly the picture of professionalism (their IOS app has actually gotten worse during the year+ I have owned the locks) and I wonder if there will be any functionality left in my $600 locks if/when they go out of business someday.
> Maybe there just isn't enough space in a door lock to do it properly.
Doubtful, dead simple OTA with recovery requires a 2x allocation of space. $2 or so of parts covers it for 99% of embedded scenarios, and if the product doesn't have enough storage to do a recovery OTA then the product wasn't spec'd properly. More clever but just as robust systems can get away with less (however much a recovery partition takes up, if doing WiFi that can be a lot).
> If a system can go back the last working version if the latest fails to work, even a bad version shouldn't brick the device.
Knowing when the current version has failed can be hard. If the system comes up 99% of the way but one driver fails, that can be hard to determine programmatically. Obviously the manufacturer here didn't have that robust of tests running at startup to determine if there was a need to rollback.
To be fair, most software updates don't have that level of rigor. Pre-release testing catches the majority of issues, as it should. Now days, phased rollouts to opt-in beta testers is standard to try and catch any remaining issues.
I hate just posting because I have an axe to grind here but wifi locks for airbnb are an incredibly stupid thing to have for all sorts of reasons.
Just get normal locks and hire a damn property manager. When I used to work for an ISP in a resort town, our support desk would get tons of calls when power went out (a frequent issue there) or our service went out.
It was always property owners who lived multiple states away who couldn't get their renters access to their property. Always furious. Always without someone local who could let someone on to the property.
Lockitron locked me out of my apartment. Danalock jammed in the middle of unlocking and made it very difficult to get inside. You know what works 24x7? My car's door. Why is this so hard?
They can fail though (and I've seen it happen, overload, excessive moisture, etc.,). How do you know this before you touch the doorknob and get the (possibly final) zap of your life?
If the failure mode is potentially "you die", then we're probably better off low tech.
My car door is currently broken, the lock somehow got pushed into the door and the keyless entry broken years ago. Luckily, its only the passenger door so I just unlock it from inside but if it was the driver door I'd have to take the door apart to fix it.
I guess part of it is that these are all after-the-fact upgrades with limited integration into the lock/door. Electronic locks in commercial buildings etc. seem to work just fine, but they are not an easy "order of Amazon and install in <1h" setup.
Sorry, that sucks. There were painful lessons from crowdfunded Lockitron which is why we stopped shipping it at the end of 2014.
We had a similar debacle in early 2014 with automated updates. The crowdfunded Lockitron used a gear train that required it to back turn in order to unload the mechanism such that a user could turn the lock manually.
A hastily deployed update with insufficient QA would overturn the gear train past the limit of the underlying lock mechanism. At this point the stress would be released in one of three ways; (1) the main worm gear would pop off from the motor shaft, (2) a secondary gear would lose some teeth or, in a few rare cases, (3) the underlying lock would be damaged. We luckily had enough units to quickly ship replacements and rolled up an update that reversed the changes.
As a result, we made an explicit decision not to implement remote OTA firmware update capability for Lockitron Bolt. Instead we require users to be within Bluetooth range of the door so they can immediately verify the device is working as expected after an update. Given we don't want to nag users on a regular basis to update their lock, we opt for very conservative release cycles.
Additionally Lockitron Bolt uses a vastly simplified clutch mechanism that doesn't require any back turn behavior. This means you can always turn the lock manually (by hand or key), even if the device is dead.
Does anyone here have experience with Mender or similar? I'm looking to do some IoT work and over-the-air updates are still my biggest unsolved problem. I'd love any guidance on any good tools to research & investigate.
This type of headline is why large enterprises are often careful about working with unproven 3rd parties. And consequently why the enterprise sales cycle feels so slow and bureaucratic.
LockState makes the mistake, and AirBnB takes collateral damage.
106 comments
[ 2.4 ms ] story [ 89.5 ms ] threadSmart implies some sort of intuitiveness or responsiveness, which is often not the case.
While software mistakes do happen, it seems any moron these days thinks they can put out an IoT product because they know how to flash a led on Arduino and they make no effort whatsoever in thinking how to make their devices have minimum reliability
Have a user accessible USB port (from the inside) from which a signed payload can be loaded for cases like these. Would have saved several days for their customers and several shipping fees.
Now that I think about it, I hope these devices are validating the certificate of publisher and the updates are signed.
You're joking right? Really, the vast majority of this iot stuff is ridiculously bad in this respect. Plain text data transfer, unsigned updates from any source based on a text message and so on. It's a disaster waiting to happen.
> from which a signed payload can be loaded
All updates should be digitally signed by the lock manufacturer
But then we complain because we don't "own" the hardware and can't hack around on it.
Also, no update should be carried out unless the user initiated it - for instance, a flashing led or an app notification tells the user an update is available - and it needs to be initiated by using the same controls as would be used for a roll-back, ensuring you do not end up locked out after updating.
That being said, the idea of a lock being online scares the bejeezus out of me. Call me a luddite, but as a minimum I want my burglars or vandals to show up, rather than nessing with me from wherever.
Well they can't be burglars if they never show up :p
In this scenario, the biggest problem is updating the bootloader, which cannot do this type of fallback.
Generally people avoid updating the bootloader (for good reason) but it is necessary sometimes...
Worst case scenario is if the system does come up enough for the hardware watchdog to get kicked in a timely manner, but nothing else works.
The watchdog needs to be the last initialized bit of code on the system.
Even so, determining when a partition is "bad" is hard.
An 8 MB (64 megabits) SPI flash chip[1] is about $2. Two jumpers to power the SPI flash and set the boot/load option is another 3 - 4 cents. These take up less than 30 sq mm of board space. And they give you a tremendous back up in case of failure.
[1] http://www.mouser.com/Semiconductors/Memory/Flash-Memory/_/N...
All of the ARM chips I'm playing with these days have a serial boot loader in ROM that you can get to by setting a pin on the CPU and powering it up that way. In such a scenario it would take no extra parts.
I believe they emulate or copy the system used by FPGAs to read a configuration from a dedicated SPI chip. Its well understood and a lot of tools can program the SPI chip to have the right sequence of bits in it to do the right thing.
If I were designing a system with OTA updates, then I would design something like this with it for this very reason.
What will it take to get IoT device manufacturers to adopt development practices that prevent this kind of thing happening? It's bad enough that most software is bug-ridden and poorly tested, but it's inexcusable for IoT software if they ever expect IoT devices to replace traditional alternatives.
I wonder if a certification body (like UL) could work though. Probably is it is hard to boil "works good" down to a set of clear requirements.
I expect this to continue getting worse. At some point, hopefully, things will change and begin to improve. I think we'll see many more -- and worse -- incidents like this before that happens, though.
They do accept traditional keys but the whole point of the lock is to delegate restricted access to people at certain times and with logs/notifications. So a physical key is like an admin override, not to be used in normal circumstances. As such, only me and my partner have them, and we generally never use them. In a wonderful example of Murphy's Law we were both out of town this weekend so all the people that were supposed to be coming to the venue to set up for a show were locked out with no recourse, it was a huge mess. Long story short, the use case of these locks renders the traditional key fallback less than ideal.
Having to pay to send a locksmith to every lock they bricked.
The scariest times were pushing updates that affected WiFi connection for customers with unusual network configurations. In particular, we had one with a multi-campus Cisco WiFi system with WPA-Enterprise, which requires logging into a RADIUS server to get WiFi. We were running on FreeBSD which had a half-assed port of the Linux WPA authenticator. We never managed to replicate their network perfectly for in-house testing, so pushing updates to them was always nerve-wracking.
Was this handled in userland? Or with something lower level (e.g., a hardware watchdog)?
I'm asking because the question has come up a few times recently about making FreeBSD recover (unattended) in the event that an unbootable kernel is installed.
This was a last ditch recovery. I could imagine lots of things that break, but which still allow ping. And installing a previous version over a newer version is flaky, because a newly created file won't be deleted.
An unbootable kernel is a whole different kettle of fish. I believe GRUB can be told to fallback to a safe mode.
I'd assume so; FreeBSD's loader can also do that. The problem was that if a kernel hanged during boot, we wouldn't get back to the loader until someone power cycled the system, so the "try the other kernel" logic wouldn't run.
After the upgrade, before the reboot, setup the HW watch dog for 2-5 minutes. Setup the boot-old-kernel flags as default in the uboot/EFI/grub/or your system boot loader.
If the upgrade the successful (connect to back to the server) and function correctly, disable the "boot-old-kernel" flags in your bootloader.
As always, do a lot of testing and automate those tests. You will be surprise how often a simple boot/reset/power cycle can fail if you repeat it over and over again in a weekend.
If you set up your watchdog correctly and the system loops too many times, your u-boot launch script can swap booting to safe/backup partition.
I can also confirm the wifi problem. Not knowing if a device is bricked or just on bad wifi is a huge problem you have to somehow account for. It was terrifying before I built tools to keep track of which devices those are
Just in case the domain name lapses. (Or for any of a hundred other reasons why having a system that reads plain text commands from the network is a bad idea)
Practically, there are tons of instances where executing code downloaded from URL's is how software is updated, but again most of those instances are human-driven, not automated.
If someone gains access to our server, its already game over. Being able to execute code on the devices is then moot because we were already hacked & potentially lost all credibility in the industry.
To everyone saying not to execute code at URL, this is how oh my zsh, and many other open source projects are installed. And SSL should prevent any kind of MITM attacks. The point about keeping keys off the server is a good idea, but is not always possible when the server needs to dynamically send content down to the device. The server would need to access the private key in order to sign this content dynamically. For statically built executables this is a good idea though.
1. A backup recovery read-only image which can flash everything back to square zero via some sort of hardware over-ride (like a pin-hole button that has to be held down for 10 seconds)
2. A temporary place in memory to buffer and validate the checksum of any new firmware that is uploaded before flashing it.
Real time flashing over spotty WiFi (without any sort of post-transfer checksum validation) has got to be the dumbest, laziest, cheapest, riskiest thing a hardware manufacturer can put out, and any manufacturer that does that should be ashamed of themselves.
1. A backup recovery read-only image which can flash everything back to square zero via some sort of hardware over-ride (like a pin-hole button that has to be held down for 10 seconds)
2. A temporary place in memory to buffer and validate the checksum of any new firmware that is uploaded before flashing it.
Real time flashing over spotty WiFi (without any sort of pre-flashing validation) has got to be the dumbest, laziest, cheapest, riskiest thing a hardware manufacturer can put out, and any manufacturer that does that should be ashamed of themselves.
I'm certain that these vendors don't have nearly the same amount of checks compared to smartphones due to much less overhead (like we had from carriers, Google, SoCs), but it's still astonishing how this could happen, but it's a good lesson for them, mistakes happen and I hope they implement a system where this is not possible to happen again.
Could have been the Electronic Flight Bag, those can run commodity OSs. I can see an OTA failing there and it needing an OS reinstall. It'd be odd, but not impossible.
Nothing gets changed unless it's for maintenance or airworthiness reasons (or fuel economy)
Updates can be done during scheduled maintenance if there's a good reason for it (see above) or if it's really something urgent it is done overnight if possible or the plane gets scheduled for this specific maintenance
I never think about my deadbolt unless I am opening my door. I never think about my smoke alarm unless starts to chirp because of a low battery. I never think about my light bulbs unless they burn out.
Any "smart" features, like notifications or a phone app, are going to make me spend more time thinking about these products than I care to, even if they are working perfectly. Failed OTA updates make things even worse. That door lock is going to need to introduce an extremely compelling new feature to convince me to spend time thinking about such a basic component of my home's infrastructure.
However, I've seen some airbnbs that have a pin door code or a key fob, which are arguably more secure than a regular key lock but have less chance of failure in my experience.
Many Airbnb owners do not.
https://www.techdirt.com/articles/20120823/10320820137/hotel...
For Airbnb, etc., you'd have to change the combination regularly (which is easily done in about 30 seconds) but at least I don't have to worry about getting locked out if I forget my phone, it's battery is drained, cell service is out, etc.
But I've been quite happy with my smart thermostat and sprinkler controller.
Maybe the difference is in these cases, a controller has been replaced with a smarter controller.
It's a natural line of progress, and there have been obvious deficiencies with the logic of the "dumb" controllers since their inception (heating the home when you're away, watering while it rains)
The solution is simple, though: Use U-boot, two partitions for firmware and a tiny script using a boot counter/flag. Only after successful boot set the flag "do not boot back to old firmware". But doing that well requires recent versions of U-boot, and people are STILL shipping devices with 2012 or earlier versions...
These locks have some nice advantages but sometimes I feel like they are way too internet-reliant and a bit outside my control. Furthermore the company isn't exactly the picture of professionalism (their IOS app has actually gotten worse during the year+ I have owned the locks) and I wonder if there will be any functionality left in my $600 locks if/when they go out of business someday.
If a system can go back the last working version if the latest fails to work, even a bad version shouldn't brick the device.
Doubtful, dead simple OTA with recovery requires a 2x allocation of space. $2 or so of parts covers it for 99% of embedded scenarios, and if the product doesn't have enough storage to do a recovery OTA then the product wasn't spec'd properly. More clever but just as robust systems can get away with less (however much a recovery partition takes up, if doing WiFi that can be a lot).
> If a system can go back the last working version if the latest fails to work, even a bad version shouldn't brick the device.
Knowing when the current version has failed can be hard. If the system comes up 99% of the way but one driver fails, that can be hard to determine programmatically. Obviously the manufacturer here didn't have that robust of tests running at startup to determine if there was a need to rollback.
To be fair, most software updates don't have that level of rigor. Pre-release testing catches the majority of issues, as it should. Now days, phased rollouts to opt-in beta testers is standard to try and catch any remaining issues.
Just get normal locks and hire a damn property manager. When I used to work for an ISP in a resort town, our support desk would get tons of calls when power went out (a frequent issue there) or our service went out.
It was always property owners who lived multiple states away who couldn't get their renters access to their property. Always furious. Always without someone local who could let someone on to the property.
Are you going to operate a crank to recharge the battery? I'm not too keen on wiring mains power up to something connected to my door handle.
This is a solved problem: https://en.wikipedia.org/wiki/Isolation_transformer
If the failure mode is potentially "you die", then we're probably better off low tech.
We had a similar debacle in early 2014 with automated updates. The crowdfunded Lockitron used a gear train that required it to back turn in order to unload the mechanism such that a user could turn the lock manually.
A hastily deployed update with insufficient QA would overturn the gear train past the limit of the underlying lock mechanism. At this point the stress would be released in one of three ways; (1) the main worm gear would pop off from the motor shaft, (2) a secondary gear would lose some teeth or, in a few rare cases, (3) the underlying lock would be damaged. We luckily had enough units to quickly ship replacements and rolled up an update that reversed the changes.
As a result, we made an explicit decision not to implement remote OTA firmware update capability for Lockitron Bolt. Instead we require users to be within Bluetooth range of the door so they can immediately verify the device is working as expected after an update. Given we don't want to nag users on a regular basis to update their lock, we opt for very conservative release cycles.
Additionally Lockitron Bolt uses a vastly simplified clutch mechanism that doesn't require any back turn behavior. This means you can always turn the lock manually (by hand or key), even if the device is dead.
LockState makes the mistake, and AirBnB takes collateral damage.