The article makes it sound like the engineers didn't know what was going on, so it could be an actual bug, unless support was just trying to placate her.
Sure, but isn't that still terrifying? It brings me back to something I wonder about with these voice activated devices: you have no idea what it can do. Voice UIs are so utterly opaque, and Amazon(/Google/whoever) pushes new software updates without informing you. So it'll add new commands all the time, and you won't know about it until you accidentally trigger it, making it do... whatever.
But either way, they should surely be confirming this before send, the same way Siri does when I ask it to send a text.
I haven't used a smartphone in years. Looking back on the days when I did use a smartphone, there was always this tiny fear in the back of my head that maybe I'd pocket-dialed someone and they'd be able to hear my conversation. It actually had a subtle chilling effect.
The Hawthorne effect (which discusses how people react to being observed) seems to be especially relevant nowadays as there's a real possibility that we're being recorded at any time. https://en.wikipedia.org/wiki/Hawthorne_effect
It's actually a big theme in 1984 where the mere possibility that you were being listened to via telescreen was enough to keep you in line and influence your behavior.
I could be mistaken but I believe there are confirmations required?
Could it be that the volume was set very low and they inadvertently triggered their way through the entire dialog? I've accidentally set alarms from a few rooms away but this is a whole other level.
POST requests should require secondary verification from the user. GET requests are probably fine to execute.
Some POST requests may only need to let you know that they did it, such as adding a calendar reminder on to YOUR calendar. It should ask if it wants to send a calendar invite to someone else's calendar.
Unless it’s expected behavior that malfunctioned. Antecdotally I’ve found a lot of companies support have almost no leeway for appeasement if the triggering behavior is considered “expected behavior” or a bug based off of it.
But this is Amazon. I guess things have changed for the worse on the customer service front though. This happened not too long ago (2-3 years) - probably around the time they posted a net profit
In terms of the big picture, this was dumb PR wise and this story should have never happened. For less than a $1000, Amazon could have stopped the story if the PR dept were more in sync with customer service
I don’t disagree at all on the PR side. Spent sometime in the contact center tools space and my general experience is if a company allows it exempt higher level employees avoid engaging with customer service directly at pretty much all costs.
Not saying it happened here but thats how you get $10/hour vendors making judgement calls on customer situations that are front page news.
Yes, I also used to work in that space, our software / platform was used by Fortune 500 companies.
My point still stands. This is Amazon and not random company X's customer service. There are very few companies like this that people can name (e.g. Nordstrom, Patagonia, ...) It this used to not happen at Amazon. Customer service used to be empowered to make customers happy, which was what made Amazon legendary. Things have changed. Customer service is now crippled like almost every other mediocre company. This is just one of the symptoms. Maybe Amazon feels that it has enough market share now, so it doesn't matter as much?
I see what you are saying it definitely could be the market share or maybe even just the overall volume (QA at scale).
I guess I just don’t have a long enough track record with them to see them in that grouping of great customer service. They’ve pretty much fought me tooth and nail over price matching and prime shipping that doesnt meet the two guarantee over the last couple of years which is probably biasing my view.
That is not like Amazon. Typically their customer support is almost unreasonably good. I've gotten refunds and perks without even asking for them on support calls.
Well, when you consider that Amazon's devices once interpreted random noises as a "<trigger>, laugh" command (as per their explanation), that a voice command to record and send a message could be created from random conversation is not in the least bit surprising.
I wonder how would you even do that?
It's not a tape recorder, where someone has to physically press a button, there are no moving parts, thus it must be software controlled.
WiFi is always on. WiFi doesn’t disconnect just because you aren’t actively moving data. You’d be pretty annoyed if every time you wanted to send or receive data your device had to reconnect.
There are usually 2 stages, a local processor that can only detect the wake word, and then once that hears the wakeword it begins streaming data to the internet.
It doesn't need to be based on the microphone, it can be based on that second processor. It can be based on network activity (my USB Wifi adapter blinks whenever data is sent/received, even though it's connected 24/7, I don't see why anything else couldn't do that).
We're also discussing theoretical hardware changes, there's no reason it can't have 2 sets of microphones where one is hardwired only to the wake-word processor which has no direct connection to the main processor except some one-way signalling, and the LED is hardwired to the second set.
Your WiFi led is probably blinking pretty constantly, which would freak people out if they were told that blinking means active listening (as opposed to checking it updates or any of dozens of other things that might use the network). It’s probably also software controlled anyway.
The idea that there would somehow be a dedicated wake word microphone is a little ridiculous. Firstly, no one would trust this supposed 1-way connection. Second, it would require a dedicated processor to make the wake word even work, driving up costs. Third, the echo uses an array of microphones so your wake word would either be unreliable or drive costs up further as you duplicate the entire array. Hardly a net win.
The reality is that if you don’t trust amazon to do the right thing you shouldn’t install their listening device in your home. (Likewise for your phone.)
True but the microphone is always on (otherwise how would the device be able to listen for voice commands). Thus what you're describing would effectively just be a power LED rather than a recording LED.
> where someone has to physically press a button, there are no moving parts, thus it must be software controlled.
If the microphone uses an amplifier, you could wire an LED to light when power is supplied to the amp. The indicator light is physically part of the circuit, so its operation cannot be modified by software. There are probably other, better, ways to do it, I'm not an electronics guy.
> Due to watchword detection, the microphone amp would always be on.
I don't think that would be a bad thing; especially if there's a switch to disable the mic. When you turn it off, you'll get reliable feedback to know it's actually off.
I think it's important that these kinds of devices have simple feedback and control mechanisms that can be independently verified and reasoned about. Software is too opaque and too untrustworthy.
I don't disagree, but the light ring is also important to know that it actually heard your command. Having it on all the time would make the UX much worse. Unless it was a separate light.
> I don't disagree, but the light ring is also important to know that it actually heard your command. Having it on all the time would make the UX much worse. Unless it was a separate light.
I see that, I think it would be best as a separate light.
I think product UX has drifted too far towards blank monoliths; I know I wouldn't mind a few more blinkenlights :)
I think is possible though I am not an electrical engineer, you can have one LED on if the microphone has power(so you can't be tricked is off) you can have an LED for the networking device, but now considering that the microphone is always on it won't be as useful,
Just one example, separate memory for storing said data, which when accessed in write mode (current line or write enable low voltage signal for writing to DRAM) activates LED as well. Or, again if DRAM, DQ pins and monitor for voltage high. I'm sure there are a dozen of ways to do it in hardware like that. Maybe someone with more knowledge can chime in.
Doing it via microphone/amp wouldn't do it, since you'd still want to use it without LED on (and not software controlled).
great way to just have companies add an annoying, always on LED, that wastes energy to avoid lawsuits. how about we let the market regulate, as it is doing currently. did amazon respond faster than the government has on surveillance? yes. should companies that don't abuse their power/neglect their software be punished for all eternity because of amazon's actions? i think not.
Do you also get mad for the LED in your TV,router and monitor, that shows the device is in stand by or on? I t would be a small one and you should have an hardware slider for brightness , you could also tape it off
yes. if you want that, you can only buy from a company that does it. if enough people agree, they'll all do it or fail. i don't abide by the 'its such a good idea, everyone must be forced to do it' mentality
> yes. if you want that, you can only buy from a company that does it. if enough people agree, they'll all do it or fail. i don't abide by the 'its such a good idea, everyone must be forced to do it' mentality
We don't live in a world with a healthy enough market or enough competition for that to work.
We have laws for all sorts of things to protect users from their own ignorance. You can't expect every consumer to be 100% aware of the dangers of things they don't fully understand.
What does "recording" mean in this context? The way these devices work is they're always "recording" in a loop on a secondary processor looking for the wake word. If we take this request literally the LED would never be off.
I'm not entirely sure that this would be a bad thing. It would set the expectation in customers' minds that the device is always listening, which is at least somewhat true.
> expectation in customers' minds that the device is always listening
If, even with that expectation, enough people still use "voice assistant" technology for a judge to consider the tech to be "in general public use"[1], the bright line test defined by Kyllo v United States is triggered and the police no longer need a warrant to use the technology (in the abstract - they don['t need to use your hardware) to view "details of a private home that would previously have been unknowable without physical intrusion"[2].
Normalizing the expectation that previously private areas (like the inside of your home) might be recorded and sent to a remote 3rd party will eventually result in everyone losing some of their 4th amendment protection against search and seizure.
No, it would just become a meaningless decorative light if it's on all the time and customers would pay no attention at all.
It's like the misguided Prop 65 signs that are in practically every commercial building in California so instead of warning people about dangerous levels of hazardous chemicals, they get a meaningless sign in every building for hazards that are no worse than if they were walking down the street or sitting at home.
Presumably, it would mean "when anything other than that secondary processor comes on." I.e., when it stops throwing away the buffer containing your speech at a hardware level, and starts instead feeding it through its local parsers and to the cloud in a way that could result in information from your speech being captured.
That would require, though, that it's not buffering the last N seconds of audio to reprocess once that processor wakes up. Do any/all of the modern smart-speaker devices do that? If so, then you'd have to take into account that when you see the light, you've potentially leaked any secrets you said in the last N seconds as well. Less like a reporter coming in and asking to speak to you; more like an eavesdropper coming in and telling you they heard what you were just saying through the door.
As nerds, it makes sense to us to do local wake recognition and only upload a request phrase after that. But in reality, this not the first time[1] we've learned that continuous speech was getting uploaded.
Charitably, it could be a mistake, maybe the previous request never terminated and it just kept streaming. But again, not the first time.
More likely, my opinion, is there's a mountain of cash on the table in the form of private conversation, and AMZ would be foolish to walk away from it. After you talk about "hardwood floors" in your house, you could expect targeted ads, consumer profiling, variable pricing, and third party sales. Why woudln't they do this? Why else are they pushing home devices so hard? They're front and center on the front AMZ page every time you visit, aren't they? Oh and GOOG too, they just haven't been caught yet.
The device is not always visible when you are using it. Amazon Echo has such an LED (at least the one I am familiar with). But it will misinterpret conversations happening in other rooms now and then. It does get confirmations for most things, so the normal scenario for us is this:
Talking loudly in another room.
Alexa: "Do you want to send this message to Jim?"
"Alexa, you suck"
I think we can assume there was a light here. It's almost happened to me before. Alexa disastrously misinterprets an ambient conversation as "start transcribing and send to one of my contacts". During this, the light does blink to indicate it's gobbling up your input. You just wouldn't know to look for it if you didn't even know you'd activated the device.
You mean like the very visible colored LED ring that already exists on the devices? Though I suspect that most people keep their devices tucked away in a corner where they are hard to see -- but that's not amazon's fault -- few people would want the devices if they had a bright flashing strobe light visible anywhere the room.
A confirmation after every request makes the device a lot less useful, I suspect most people do not want the device to confirm commands before it does them.
The trouble is that the LED would be lit 24/7 and become exactly as meaningful as the absence of a lit LED. These devices can't function without constantly recording to listen for trigger words or phrases.
It's like saying that someone buys a car that's powered by an explosive liquid, and then complains when all the gas in the tank explodes at the same time.
I feel like you're being purposefully reductive here. You know they're different right? The entire value proposition (and marketing strategy) of Alexa and related devices is that they are voice activated, so you don't have to physically interact with the device to make a request.
Given that, it would be ridiculous to expect paying customers to switch it off when not in use. The onus on Amazon is to protect their customers' privacy while the device is being used as intended.
If you really want to pick apart the car analogy, maybe this is closer to having automatic start and driving capability on a car, but the car sometimes randomly starts and drives away. Are you really going to blame the customer for that, even though they could disable the feature every time they exit the car?
The car analogies are usually quite poor. However, it depends on your use case. Echo Dot's have a mute button which prevents monitoring, or at least it seems so. I'm not an echo power user, so my needs are definitely different than for intended uses like home automation.
I think for some, at least users like me would like a feature similar to "su". I'd like to confirm that Alexa should start doing stuff if it has not been confirmed for x number of minutes. I think the real problem with these home assistant devices is they are not designed with a confirmation message. This would be annoying in some cases, but surely a balance can be struck.
I'm not trying to be reductive at all about the car analogy. When you turn on a car, you acknowledge that the engine is running. You can also mute an echo if you have the foreknowledge that you will not be doing activities that require voice activation.
I don't know about her but I would be upset not because Amazon recorded private conversation but because they sent it to some random contact on my contact list. I don't want to bother my contacts with my inane ramblings in the privacy of my room.
I would think what if that was not the only time this happened, what if something else was sent to someone else but that person did not did nothing about it, what if the data was sent to some server and logged, what if all it hears is transcoded to text and stored on some dev box logs now.
Summary of vague technical details (which may be all we hear about this):
> an Alexa engineer investigated ... they said 'our engineers went through your logs, and they saw exactly what you told us, they saw exactly what you said happened, and we're sorry.' He apologized like 15 times in a matter of 30 minutes and he said we really appreciate you bringing this to our attention, this is something we need to fix!"
> the engineer did not provide specifics about why it happened, or if it's a widespread issue.
> "He told us that the device just guessed what we were saying" The device did not audibly advise that it was preparing to send the recording, something it’s programmed to do.
I don't. And in this particular scenario, it's an important distinction to make because there's a risk of Amazon using equivocation to deflect responsibility.
If this was intentional, then yes they shouldn't be able to deflect responsibility. However, if it was just a bug, I think it is a bit unfair to vilify either the company or the developers. Bugs happen, and hopefully they can learn what caused this and prevent this class of bugs in the future.
And this is where "software" diverges from "engineering".
Bugs happen in architecture, aircraft, etc. too. the difference is that the actual engineers are paid to have a precautionary approach and spend significant resources to actively prevent bugs from making it into the final product.
In contrast, software is often written to "ship first", be "agile", and "move fast and break things". Yet when it causes problems, they just say "bugs happen", and "it is unfair to vilify them".
Features are not better than reliable security.
And yes, negligence is less bad than malice, but it is still damaging and developers and managers need to be held to account.
If your manager is pressing you to do unsafe crap in too big a hurry, it's your responsibility to push back, and if unsuccessful, leave for saner pastures and make it more difficult for that management to proceed.
Well, yes and no. Bugs, security, reliability, etc are all important things to consider, but they can't be the only focus. Security doesn't matter if the thing you are creating has no features; it would be the same as if it didn't exist at all.
Instead, we must manage risk; the risk of bugs, the risk of security vulnerabilities, etc. Nothing we do is risk free. Even walking across the bathroom floor has SOME risk; we might slip and fall. Does that mean we should just stay in bed all day to avoid any risk?
No, we measure risk by factoring the chance of the bad thing happening and the consequence of the bad thing happening. We then determine how much effort we should spend on that risk, since there are infinite risks and only a finite amount of effort we can expend.
If the consequence of a risk is death, then we should absolutely put a lot of effort into minimizing that risk. If the consequence of a risk is that a private conversation is sent to a contact, we should definitely put a lot of effort into minimizing that risk, but probably not quite as much as you would into something that has the consequence of death.
Even when the risk is death, however, we don't put infinite effort into avoiding it. We choose to cross the street, even when we know there is a risk of death when we do it. We drive cars that have chances of mechanical failures that could cause our death, but we don't bring the car to the shop every day to check for failures.
Things are not so black and white as to say "security is always the most important thing"
I did not say >"security is always the most important thing". I did not suggest that anyone should develop such that >" the thing you are creating has no features". I certainly never suggested that no one get out of bed because they might slip in the bath. These are strawman arguments.
You do not need to lecture me about risk. I've had a career in international downhill ski racing, have won auto racing championships, and enjoyed lots of technical rock climbing, all of which require a high degree of risk assessment, both in extended preparation phases and at split-second time scales. I've also run risk analysis for UAV flight systems
I understand well the difference between smart-crazy and dumb-crazy, and where the pseudo-mathematical risk models like yours break down.
Your 'analysis' to "...measure risk by factoring the chance..." would have fit right in at the meetings where Ford decided to just go ahead with the design of the Pinto/Bobcat because the lawsuits would cost less than the fix -- they wound up killing dozens of people.
Your 'analysis' would have fit right in where the trading algorithms were being designed, which worked fantastically profitably, until they didn't and ended up crashing the global economy in 2007-8.
You cannot simply multiply the cost of the consequences by the expected probability and get an allocation of resources. That is what you do to see if the lottery jackpot is big enough for you to want to buy a $2 ticket this week.
You must instead 1) fully examine the system for potential critical failure points/modes and then 2) allocate WHATEVER resources are necessary to account for preventing those critical failures, then implement those remedies along with the features.
These preventative measures may involve installing redundant systems around the critical points, redesigning the points so they fail in a safe mode (e.g., fail to send the data vs sending it off, shut down vs, explode, etc.), adding check procedures around the potential critical failure, etc.
Note that NONE of these measures involve not implementing the feature. They involve 1) checking for critical failure modes, 2) allocating R&D to develop preventative & fail-safe measures, 3) implementing the measures, 4) testing, and 5) field monitoring.
This is what you do if you are serious about risk.
Ok, I never said that the 'analysis' should be purely based on dollar value. I never even said it should be a mathematical model. You accuse me of a strawman and then turn around and do the same to me.
I was simply pointing out that we can never get to zero risk, and since we can't, we have to weigh risks based on consequences and probability.
> 1) fully examine the system for potential critical failure points/modes
Sure, to the best of your ability. How can you know for certain you have found all potential critical failure points? You can get pretty sure, but never fully sure. We still have industrial accidents, in every single industry in the world.
You also have to define what a 'critical risk' is. I don't think it is an a priori fact that accidentally sending a recording of a conversation to a contact is a 'critical risk'.
So your model isn't even mathematical, it's what, just a SWAG of the combined hazards and odds? That works for linear, small risks.
It absolutely does NOT work for serious risks, e.g., of death, serious injury, massive privacy violation, and other potential life-changing events.
The concept you are clearly avoiding or missing non-linear risk.
You (and amzn_engineer1) are advocating for simply subsuming risk assessment into the ordinary development cycle, and calling it "taking it seriously".
That is fooling yourself.
Taking it seriously is actually making full and serious effort OUT OF THE NORMAL DEVELOPMENT CYCLE for no other purpose than to SEEK and identify potential critical risks.
It is then engineering a variety of in-depth solutions to prevent those critical failure points from ever seeing the light of day. And implementing them. and testing them. And monitoring them.
>> I don't think it is an a priori fact that accidentally sending a recording of a conversation to a contact is a 'critical risk'.
This is an exact example of this sort of failure: 'it's not a priori bad'..., minimize it and streamline it into dev.
I really want to know in what world any sane person would say that it's OK to randomly divulging an intimate conversation to a contact or random recipient -- seriously, who would say that?
I mean sure, most conversations are benign, but some could be utterly life-changing if revealed. and that's OK with you?
It's worth observing that, firstly, aircraft still crash and bridges still collapse, amd secondly, that a culture of blameless analysis has gone a long way towards making them crash and collapse less.
But note the blameless analysis is NOT the same as saying "meh" errors happen.
It is a culture and deliberate practice of allocating resources to seeking and classifying risks, and designing, implementing, and testing engineering and procedural mitigation strategies (vs. development as usual).
Right, or "the current source code says not to do that, but some discrepant, previous, buggy build of the machine code made it into the product you actually bought."
It’s even worse. Sometimes the packages that the DRAM ICs are encapsulated in have minor radioactive elements in and alpha particles emitted can flip bits.
Yeah, pretty much. It's likely that the device heard something like "Alexa, send a note to Enrique" and then recorded and sent the subsequent note to the matching contact that it found. The microfailure is that it either failed to respond to the user in a way that was clear, or did it in a way that the user didn't notice (I dunno, volume all the way down? Output to a bluetooth speaker or headphone jack that wasn't audible?). And that can be fixed.
The macrofailure, though, is that you have a device just guessing at what you want and doing it. That can't be fixed without going back to manually constructed (or at least manually triggered) notifications. Which for "send a note" might be an obvious fix, sure, but there is a lot of gray area in what constitutes "privacy" and Alexa would be pretty useless if you had to affirmatively OK everything it did.
>Output to a bluetooth speaker or headphone jack that wasn't audible?). And that can be fixed.
Not sure how to fix that, if you use the aux out it has no way of knowing what happens on the other end; a confirmation prompt that isn't heard could still pick up an errant "Ok" response. Perhaps require a PIN like the shopping confirmation that would make it much much more unlikely? Or better yet, disable messaging as an option.
I thought the response was pretty disturbing. They have logs of everything you say to Alexa? I guess I assumed the private conversations were not kept beyond some sort of metadata. No thanks.
That response could easily mean that they saw that there was an interaction where a message was recorded and sent without the device getting confirmation beforehand which matched what was reported to them, not necessarily that they kept the content of the conversation and could play it back. That's also a paraphrased conversation reported by a possibly non-technical person and likely passed through a customer service representative, so I'd take any technical details gleaned from it with a grain of salt.
"Always listening" is a fundamentally unsafe design.
Once recordings of private conversations leave the local environment and make it to the cloud, eventually they will leak. It's akin to data collection by law enforcement: once the data exists, eventually it will be abused.
Yes. Assume that all data uploaded to the cloud will eventually be compromised. That is the only safe assumption.
Not even the most responsible companies (e.g. Google) can hold out 100% of the time in the face of determined assault by government. Some of that data is going to leak to three-letter agencies, or similar.
Somewhat less responsible companies (e.g. Amazon) will leak data more often, to a wider range of threat agents.
So then we have to consider how valuable this data is. Random sampling of private conversations within the home? Sometimes innocuous -- but if the wrong moment gets leaked, the consequences are potentially life-shattering.
That seems like an extreme perspective. Kind of like the tech version of abstinence only sex ed. Sure, it's the only 100% safe way, but it's not useful for most people. It doesn't weigh the benefits against the potential cons, or even take into consideration the actually likelihood of data being leaked.
The alternative perspective you're presenting sounds to me utterly cavalier about the prospect of ruining people's lives. It's like Equifax's attitude towards identity theft: it doesn't affect their profitability, so why care?
It's because such blithe dismissal of the damage caused by data gathering is so prevalent in the industry that the likelihood of devastating compromise is so high and the costs borne by the populace are spiraling upwards.
Some data should never be collected. Some data should never even be uploaded.
The problem I have is that you are conflating potential damage and actual damage as the same thing, which is not how you accurately measure risk.
I am honestly confused as to how you interpreted my last comment as "utterly cavalier about the prospect of ruining people's lives", when all I said was that your assumption doesn't take into account the actual probability of data being leaked and it doesn't weigh any of the benefits of data collection against that risk.
Small-likelihood times many-chances times grave-consequences equals a finite but significant number of lives wrecked. A gamble you deem acceptable.
I can only hope that karma visits those who arrogate to themselves the decision to sacrifice a few of their fellow human beings: may they and their loved ones become the sacrifices.
Nevermind that we are already in a world where you can play any song you like from the wireless computer you carry in your pocket. That's not convenient enough.
It’s not “whataboutism” to point out that (almost all of) the people who can’t conprehend why anyone would ever have one of these devices have, in fact, already had one for years.
There's a big difference between a device that occasionally listens and can easily be powered off or muffled and a device that is actively listening at all times and can easily "accidentally" capture your conversations.
OK, but you used the word “tapped.” A smart speaker isn’t supposed to be tapping you any more than a phone is. In both cases, they have to be doing something more than they’re supposed to.
I disagree. To be listening at all times is the very definition of a tap, don’t you think?
And people know this. It’s not like they are being misled. It’s a conscious exchange of privacy for convinence. It’s just that, in my mind, you are getting very little for what you’re giving.
In my mind, you’re giving almost nothing if you already have a cell phone.
I made a different choice, but I totally understand why someone would choose not to have a phone or smart speaker due to privacy concerns. What I don’t get is someone who has a phone but sees smart speakers as crazy.
- You are pretty certain that your phone is listening at all times, even though it's set not to.
- You see no difference between a product that's advertised manly as a listening device and one that also has such a feature, that can theoretically be turned off, and is very useful otherwise.
I don’t think my phone is listening, other than the wake word. Same deal with my smart speakers. I think either one could, if hacked or just updated by the manufacturer with nefarious intent.
I see no difference, privacy-wise, between the two. I don’t care how they’re advertised, I care what they’re capable of.
What’s so bad about having a small processor run a recognition routine on all incoming audio, discarding everything that doesn’t match, and activating the device if it does? Are you just worried about accidental activation?
>What’s so bad about having a small processor run a recognition routine on all incoming audio, discarding everything that doesn’t match, and activating the device if it does?
If that was the case, nothing. But we have enough evidence to believe it's not. While having the device listen, even though it's been explicitly configured not to, is a bit more of a stretch.
>Are you just worried about accidental activation?
That could be an inconvenience too, but I'm more concerned with malicious intent.
What evidence do we have? I’m about 99.9% sure that this story is just accidental activation and bad reporting.
If you’re concerned with malicious intent then I really don’t understand the difference. If the device maker is trustworthy then you’re fine either way. If they’re malicious then you’re screwed either way. A setting for “please don’t listen” isn’t going to make the slightest bit of difference if they’re malicious. M
Trust is proportional to access. I don’t trust any third party with an always-on microphone in my house, especially one that’s networked. Malice aside, I don’t trust their competence.
That’s fair, but again, a smartphone fits that description too and nobody is talking about how you’d have to be crazy to let one in your house.
It seems to me that smartphones are significantly worse. You can unplug your smart speaker or kick it off your WiFi and be confident that it can no longer hear you. Smartphones have batteries and cellular data connections. Also they’re usually with you at all times, even when you’re away from home.
Can you? I'm not a tinfoil hat, but the echo is only listening when you address it. It also has a physical mic kill switch, but you don't really know that either mic is not live at any time.
My guess is they mostly do work that way but bugs and vulnerabilities don't seem like a stretch. (As is to be expected with all software)
I doubt it. Recording audio is pretty cheap. Lots of smartphones can now do continuous “wake word” listening just like smart speakers while on battery power.
"Pretty cheap" is still likely an order of magnitude or more than actual standby, and using a small ring buffer to identify a pre-established wake word is similarly going to be cheap compared to actually streaming audio to disk.
If you actually care, this would be pretty trivial to verify using an oscilloscope to observe the energy draw from the battery while different things are said around your device.
I’ve verified it accidentally by using an intelligent alarm with my phone unplugged. The battery drain for eight hours (which includes analysis that would be unnecessary for just recording) was a few percent.
I like the idea, but wonder if the same features are possible on a stand alone tool. For example, the device does the NLP there on the premise. It exposes a plugin API for apps. The app gets the text translation by registering a callback URI. Then it can do want ever it wants with that. Expose a streaming receiver api for music. Essential get rid of the middle man service.
Possible? Yes. On a Pi? .. Maybe? Probably not with current power I'd guess. Allot goes into the voice training and equivalent operations. There are a few OSS alternatives to build something similar; I don't think anything has a similar power yet though.
There's Mycroft too: https://mycroft.ai/
We have a lot of alternatives but they are unable to get the sort of marketing and hence the attention mainstream devices like Alexa and Google Home do
In fairness ( I'm Mycroft's CEO ) our product isn't on par with Assistant or Alexa....yet. We've had to do a ton of work around wakeword detection, natural language understanding, device management, skills management, user management, credential management and speech synthesis BEFORE we've had a chance to work on the user experience.
That said....we are on track to have a high quality experience when we go to production in Feb 2019 ( Release 19.02 ). As an open source community we'd welcome any help folks have to offer. Contributors don't even need to know how to code, all they need to be able to do is listen to sound samples at https://home.mycroft.ai and tag them as "Hey, Mycroft" or not "Hey, Mycroft"
Thanks to everyone who is helping make Mycroft a reality!
Sure. Have a look at Snips [0], Mycroft [1], Jasper [2] or Adrian [3] which are all open source home assistant devices with voice activation, although some do use Googles NLP cloud APIs for speech processing. Snips seems the most interesting, but looks terrible - a bare PCB on your wall? Really?
It's a little more than just that. Every light in my house is controlled with Alexa which is a very nice and hard to explain until you have tried it. Also my home theater system is all controllable via Alexa as well. This kind of thinking makes no sense to me.
> People are voluntarily paying for their houses to be tapped for the convenience of being able to shout: play me some song.
> I’m so out of touch with this world it’s scary.
I could just as easily say "People are voluntarily paying for their locations to be tracked for the convenience of being able to get live maps on the go" (ie. cell phones, which also are listening to everything you say unless you disable it)
I never thought that voice control would be that useful until I had my daughter... being able to turn on the white noise, turn off the tv, play her music, etc while having my hands full of toddler is very useful.
Yeah, that is kinda my guess, but it doesn't seem to be predictable... sometimes it happens within a few minutes, other times it takes an hour or more.
Heck it helps me sleep -- I'm a light sleeper. I started using the Noisli app every night this past year and my sleep quality improved in noticeable ways.
I also live in a city, so white noise (actually brown noise) masks out tiny noises that interrupt sleep.
Are you arguing that we should just stop creating anything new, since we clearly survived before without it? This comment could literally apply to ANY new thing.
No I'm not, I work on home automation for a living. I am saying the value and added convenience is largely perception and it doesn't. I raised my first kid pre-echo and my second post echo. I think for the most part, raising a kid is easier than we thing, and we are surprised at how well we are doing and attribute that to things around us, much like our old superstitious ancestors.
The TV not running in the first place and singing to one's child isn't "surviving", it's like a lush forest compared to the irradiated wasteland that is "turning off the tv and playing some white noise and/or music".
> This comment could literally apply to ANY new thing.
Yes, and?
It applies to using x-ray machine to fit shoes in shoe shops, which was super fancy schmancy at first, and I'm sure anyone who rejected it solely on the bases of being unnecessary was laughed at by first adopters, too. On the other hand, there's the discovery of washing hands being derided and ignored, when quicker adoption could have saved lives.
Exactly because it can go both ways, you have to look at the actual thing. You can't determine the quality of a thing or the usefulness of a habit by just looking at a calendar and a clock.
Mankind has made countless decisions to trade convenience for some degree of personal freedom. I can live without mobile internet - its just so convenient to have it. Voice control for home appliances is exactly the same.
If I'm home and don't have my phone (or have my hands full, or it's dark, or whatever) I can easily turn lights on or off and without fumbling around for switches or trying to juggle a cat or a child.
I can also control my tv, thermostat, get a news briefing, check my calendar and set reminders.
Yes, it's ultimately a convenience, but so is indoor plumbing and store bought bread.
>Yes, it's ultimately a convenience, but so is indoor plumbing and store bought bread.
I’m sorry, clean water and food are not conviniences. They are among the most basics of human needs. That’s the kind of thing that scares me. That someone would make such a comparison with a straight face.
This is the kind of argument that seems really common and I find very annoying. So 'indoor plumbing' is obviously used to provide us with clean water, but the convenience aspect is the indoor part - you can have clean water from an outdoor communal well, instead. Similarly 'store bought bread' is food, but so is home made bread. I just cannot work out if the commenter was being deliberately obtuse or really misunderstood?
> you can have clean water from an outdoor communal well, instead
Have you ever considered that this is not the case in most of the world? Resources are far from evenly distributed.
It is not by chance that the availability of indoor plumbing correlates to increase in life expectancy. It's nice that I can just open the tap and clean water comes out, but it's also a lot more sanitary than people carrying and storing buckets around.
>I just cannot work out if the commenter was being deliberately obtuse or really misunderstood?
Same here. Comparing the savings of a few steps towards a light switch to clean water and cheap food is intellectual dishonesty at best and severe ignorance at worse.
The whole point is that they were not comparing anything to clean water or cheap food. They were comparing something to water that doesn't need to be moved inside manually and bread that doesn't have to be made at home.
Neither of which is needed to live. Both of which are conveniences.
It's not though, please reread the comment above. Water brought and stored in buckets from an outdoor communal well would not be nearly as clean or as plentiful.
There's a reason we have indoor plumbing and convenience is just one of them. And I can't believe I have to explain this here.
You seem to be intentionally missing the point. Change my earlier examples to remote controls for your TV if you prefer.
Convenience does, absolutely, have strong value and can be meaningful improvement to one's life. Generalized voice control is a major and meaningful convenience - even if it's currently early and might have some problems - because it drastically changes how we can interact with our environments.
Sure, we can short sell the case as "not needing to walk to a light switch", but even then if it's dark, my arms are full, and the path to the switch is littered with children's toys then that's a meaningful improvement to be able to say "Hey google, turn on the living room light".
Consider cases of people with severe MS or who otherwise can't easily walk. Sure, they could hook up a remote control and keep it near them, or get one of those stupid clapper things, but even then that only works for the lights.
A generalized voice assistant gives me control over my house, from anywhere my voice can be heard, and not only executes my actions, but can give me realtime feedback and data. Best of all, this requires no special skills on my part, just a willingness to talk to the damned thing loudly and clearly with simple words.
Can it act as a wire tap? Of course it can. My store bought bread can also be contaminated (right now in my area there is a huge contamination issue with eggs and lettuce), my indoor plumbing can burst and flood my house, so on. It's a new technology, it has kinks and flaws, and we're currently in the early adopter curve of it moving towards the initial disappointment trend.
> Change my earlier examples to remote controls for your TV if you prefer.
My disagreement with you is precisely because you consider them equally interchangeable.
To me, putting those two on the same level is actually offensive.
If data on the increased life expectancy isn’t persuasive enough, I suggest you speak with someone with no access to basic sanitation. Ask them what they think of this comparison.
Because the computational requirements of good speech rec exceed what you would build into a $100 device, and because doing it in the cloud makes training and improving it easier. The “wake word” is local, which is why it is constrained to a few choices.
I suspect we are only a few years away from being able to do a slightly inferior version using local processing, or a private cloud (already possible today when there is a market). I would probably opt for that, but use Alexa in rooms already already untrusted.
Yes to the former, pretty much for the latter - the "training" I recall was less than a minute long and consisted of repeating one or two example sentences, hardly anything that would be an adoption impediment to devices like an Echo. The resulting voice recognition worked, in my experience, better than that on modern Android phones.
I'm pretty skeptical of this. Apple has a pretty good commitment to privacy, one that is costing them on the AI front, and as far as I can tell they still upload Siri audio to the cloud for processing. If this was easy to do 10 years ago, I think Apple would at least offer as an option.
That's not why people I know say Siri sucks: for us it's because it can't look up as much info as Amazon or Google. It understands the query, it just can't do as much with it.
Siri does more processing locally than Alexa does, but does upload to the cloud, and depends on the cloud, both for analyzing the statement and for accomplishing it. It can then execute local actions. The info uploaded to the cloud includes essentially all the relevant data for the query, so it doesn't particularly help with privacy -- it's just not the raw waveform.
But Siri sucks for MANY reasons, and not just technical ones. It does worse processing but also does worse cloud integration, and now vs. alexa, has a far worse ecosystem around it (alexa "skills" are pretty awesome, and trivial to create)
Because it makes the voice recognition vastly better. Have you ever noticed that Google can get words correct in navigational searches, even if it only sounds like the name?
We are a long way from being able to do that on $100 devices.
There is a massive difference between walking a few steps to turn on a light switch (which is only useful when you are in close proximity) vs having a accurate real-time mapping functionality when on the go.
Similarly, there is a massive difference in the privacy implications of listening to every conversation everyone in your home (including guests) is having vs. having your current location known. Wiretapping laws exist for a reason.
Turning off all the light switches that have been left on in a big house though is a meaningful convenience. I think the privacy point is strong enough without having to trash people's use cases.
No trashing it at all, just contrasting it. I do see the convenience (luxury) of home automation, but when you compare that against accurate mobile mapping, location finding and navigation, the difference in usefulness is orders of magnitude. One can literally save your life.
> It's a little more than just that. Every light in my house is controlled with Alexa which is a very nice and hard to explain until you have tried it. Also my home theater system is all controllable via Alexa as well. This kind of thinking makes no sense to me.
There's no reason why your house needs to be "tapped" for either of those use cases. All those things could be accomplished without leaving your home network.
Nobody is going to disagree that it's useful and hard to live without. That's why the big 3 or so tech companies know you'll sell your soul to have it. And yes,I and you should disable location services on your phone when not needed.
Does controlling your home need sending data to an external server thought? Has no one come up with a competing product where all the processing is done in the machine itself?
Actually, yes.
But I am not in a formal relationship, so I can imagine this to be harder to arrange when you live together on a regular schedule.
But I would certainly try to do this in this situation as well.
It is not only privacy btw. it is also just nice to have both persons here in the now and not half distracted because the Smartphone just got new messages. Could be important... but usually never is important enough, to destroy the moment.
So when I go to sleep, alone or not, the mobile is off (without battery) or away. Unless there might be a emergency and I need to be avaiable, but those situations are rare.
I agree completely. Being able to say things out loud and have things happen is cool I guess. However, that doesn't mean I want to give all my data away and it to be stored in some black box on the Internet. I want control of my data.
So when will such devices just be made illegal entirely? It doesn't seem they can be made to be safe for average people if stuff like this keeps happening and companies are unwilling to compensate people.
What if this was a lawyer talking about clients or a doctor talking about patients? Is Amazon (or Google, or Microsoft) willing to deal with such legal liabilities? Is this what move fast and break things looks like?
Edit: To those trying to downvote, remember, the Silicon Valley lives in a bubble, and for the safety of people inside and outside of the bubble, the bubble must be poked every so often. This is one of those times.
I think this tech is just in it's infancy, calling it inherently dangerous or saying it should be banned is silly. I think it will be done properly some time in the future, it's most likely going to be similar to the Windows vs. Linux vs. MacOS battle.
You ever butt dial someone? Should all cell phones be outlawed?
I'd say the Echo needs to have a more difficult activation routine available. Maybe have the option of setting the wake word to "Alexa, can you please" or something similarly long and unlikely to appear in normal conversation.
Also, I wonder if they had their wake word set to "Computer". I did that once, and quickly reverted after so many false activations.
My experience is totally anecdotal and I'm not even an Alexa owner, but I had a Skype conversation with a client that had one them set up in his office and it was constantly triggering, even when the word wasn't used.
So if it though it heard "Alexa, send this message to John: {conversation of ten minutes}" it just did what it thought it was supposed to do. But it's weird they didn't hear any audible confirmation or anything.
I received one of these messages just a week ago. Alexa sent me a message of my friend and his girlfriend having a private conversation. I immediately texted him to ask if he intended to do that and he did not- so weird.
Can you explain a bit more. What was the context of the message? was it an email? What was the subject line and the text? Was the audio an attachment or a link? A link to where...?
"Amazon sent me a message" is so frustratingly vague...
That means that, unlike butt-dialing, you're more likely to accidentally send a message to someone you're actually talking about since the person's name is part of the command. Yikes!
So that we consumers can better understand this potentially huge vulnerability to determine if it's a bug, a "bug-feature", user error, user lying, or Amazon actually doing something evil.
Yea, I wish the girl mentioned what the message started with. Something had to have been misinterpreted as a voice command and a contact name ... and the device UI has to be broken to the point where it doesn't give you any audio acknowledgement or confirmation.
Since Alexa wasn't originally available in my country, I setup a whole separate account for "Alexa LastName" in the US with alexa@mydomain.com as the email address.
Now that Alexa is available in my country, I've considered switching it over to a real account explicitly to be able to use some of these calling and messaging features. Maybe I'll wait till the bugs are worked out.
Do you have any proof of this online or can you provide more specific details? Not that I don't trust you but I don't trust anything I read on the internet without a minor amount of verification.
I'm sure another anecdote from a stranger won't be enough to convince you, but this also happened to a friend of mine just last week. A work contact of his received a voicemail from him which consisted of police sirens and muffled shouting. Of course he panicked thinking something bad had happened. Turns out my friend's Alexa had picked up a police show on the TV in the same room.
What actually happened: Alexa misinterpreted some voice commands and activated a "call" skill. The people involved and local news got very excited and escalated this into a conspiracy story.
Amazon takes customer privacy EXTREMELY seriously. There's no way a team would get the "ok" to build a skill that randomly records private conversations then sends them to a random contact. It also doesn't make any logical sense to build such a skill.
Yes, I might sound biased because I am an engineer at Amazon. This statement is my own and unrelated to Amazon's opinion.
It's pretty clear to me that this was an inadvertent activation of the call skill. The equivalent of butt-dialing someone from your phone. I get random voicemails from people all the time that are clearly recordings of their phone in their pocket.
If that's the case, then you need to provide additional features to reduce the chances of this happening. Longer and more unique wake word options, or more complex and deliberate confirmations before the call is placed. Something that a user can enable if they're worried about this sort of thing.
100% this. Seems like there should at the very least be a product feature that automatically disables any outbound messaging by default (similarly to how you can block purchases without a PIN). This doesn't solve the problem of always listening, but at least prevents this particular situation from inadvertently happening. Additionally you should be able to set an automatic deletion of voice recording after a specified amount of time. I'm sure some of this has to be in the pipeline with GDPR.
The obvious, predictable and serious screwup occurs and people shrug it off. Engineers from Amazon post how it's all a regrettable mistake and it won't happen again. I am not gonna unload on you specifically here, but if you could maybe pass along to whoever is making business and product decisions over there that yeah, maybe 80% are stupid enough to buy this crap, but there are a huge number of people that are extremely creeped out by this and avoid it like the plague. That deafening silence you here from them should not be interpreted as license to push the boundary even further.
If there was an Alexa in my place of residence, I would rip it out of the wall, smash it to bits with a hammer, and fire it out of a cannon into the sun.
Hell no to this crap from my side. I hope there are more incidents like this until people wake up to the fact they are putting telescreens in their homes. You guys can't be trusted!
>If there was an Alexa in my place of residence, I would rip it out of the wall, smash it to bits with a hammer, and fire it out of a cannon into the sun.
You may consider disposing of the bits at Kilauea instead of the sun. It would be almost the same and much more affordable.
Your comment sounds belittling to me. The OP isn't a "conspiracy story", it simply says what happened. Yes, it tells the story from the couple's point of view. Shouldn't it? Sure, they "got excited". Who wouldn't?
I get what you are saying but I would say that Amazon does NOT take privacy extremely seriously or this couldn't have happened. Let me be clear that I'm not saying they don't care at all or they are conspiring with the NSA.
What I mean by the above is that the "call" skill is much different than the "weather" skill. All Alexa has to do is have a confirmation prompt in the "call" skill and this wouldn't have happened. That is what extremely serious looks like. This is exactly the same as the phantom laughter incident from a few months ago. Alexa "heard" someone say 'Alexa laugh' and laughed, but that wasn't the user's intent. It was fixed by moving to 'Sure, I can laugh,' followed by laughter.
Voice UI is very hard and still in its infancy but ability for personal harm (physical or emotional) must be considered in these interfaces. Turning off the lights may not need confirmation but unlocking the doors or turning off the alarm probably should. Sending recordings or answering calls or even calling people should require more hoops or at least allow the user to control the risk/reward.
There is absolutely no reason to believe they are not conspiring with the NSA. They have a huge deal with the CIA, plausible some of these funds are for surveillance capabilities. They would not (and likely would be legally prohibited from) disclosing any relationship they have.
> There is absolutely no reason to believe they are not conspiring with the NSA.
Or, more likely, being made an offer they can not refuse by the NSA. And the rank-in-file engineers may not even know it's happening, all it takes is inserting some diverting code into the pipeline and calling it "QA monitoring" or something. Couple of people in the whole org would know that somebody from some IP connects and downloads these "QA" data periodically, all the rest would be completely ignorant and indignant at the thought. Don't see anything preventing this from happening at Amazon - or anywhere else.
> I get what you are saying but I would say that Amazon does NOT take privacy extremely seriously or this couldn't have happened.
Apple shared personal photos of myself onto the internet without my permission. I could not delete it without getting support to assist, and they could not provide me with a reason why this happened.
Would you say that Apple does not take privacy seriously?
I think you accurately identify the issue with this situation / skill; voice triggers need to weigh convenience (how easy should it be to activate this skill) against it's permissions or potential (how can this skill affect a customer). In this case, I think this was not done properly.
This does not mean Amazon does not take privacy seriously. It is a company of small teams with very few layers of management between an engineer and business decision. The error in judgement of one team does not reflect on all of Amazon.
As you say, Voice UI is still in its infancy and not lacking growing pains. However, because Amazon does take privacy very seriously, after this incident, I'm certain there will be actions taken internally to ensure teams properly weigh the gravity of a skill with its voice trigger (or adjustments made if there is already an existing policy).
If a "call" skill is accidentally triggered, before it is sent to any email addresses, it should tell user "N seconds of voice was recorded and about to send to ....", please said "Send" to send, "Play" to play back the message, etc.
The default must always be voice recording will be auto deleted after 1 minute if no response is heard. It should let user know about that too.
Better yet, let the user pick a confirmation word (kinda like a 'safe word'), or choose one at random from a list of moderately complex but well known words that are unlikely to be uttered in casual conversation.
To send this message to John Smith, say 'artichoke'.
Having a UI that gives the customer no audio confirmation that it got a command to record and send a message is a serious UI failure. This isn't a small thing here. If a bug like this makes it though, how can I trust that device.
I don't think the article is indicating what you say. The report says it's a bug, but it does bother me why Amazon won't comment on the specifics. This should be patched immediately with a full retrospective explaining what happened.
Is your best guess at why this might impact someone's trust in Amazon products really that people would think it was an intentional design choice? (Seems kind of straw-man-y.) If not, then you should generally address the part that will matter to people -- not whether this was on purpose, but what issues caused it and how will they be fixed.
And this is where "software" diverges from "engineering". It's not a conspiracy, it's negligence.
Bugs happen in architecture, aircraft, etc. too. the difference is that the actual engineers are paid to have a precautionary approach -- and spend significant resources -- to actively prevent bugs from making it into the final product.
Amazon and your team has built a great product (I have one and make moderate use of it, have even considered building some skills).
But, you have planted a full-on bugging device in millions of people's homes. Done by a government, this would be cause for war or revolution. This is serious, and you need to treat it much more seriously than you obviously are. Every 'skill' does not require the same minimal levels of security and verification, some, like this one, require much more, or should be forbidden outright until such security can be properly implemented (and yes, this should probably include calls only to pre-configured whitelists, intent confirmation, etc. and to any manager that says "that's too inconvenient to the user", the response is "screw you, it's critical").
You call yourself an "engineer" at least twice, and claim that you take privacy "extremely seriously". The evidence from this incident and others noted in this thread indicates otherwise. Clearly, insufficient resources were allocated to figuring out the potential failure modes of a "call skill", and preventing them.
All due respect, but your team needs more of an engineering approach than you have. This entire "it's gotta ship yesterday" mentality in the software industry used to be just inconvenient. Now it's getting dangerous. Please help stop it.
First, the Tech/Dev managers need to stand up. Instead of saying 'Yes' to every feature request, they need to say 'No', or 'Later'. I've seen too many who just feel that their job is to implement everything as fast as possible, and as close to approximating the sales/mktg/product guy's latest half-baked idea as fast as possible.
This is not easy, especially as the CEO is still typically above the CTO, and can overrule. I had it happen to me, when we were ahead on a scalable version of the product, but they didn't like the timeline. The mgt decision cost almost a year of messed-up myopic development schedule just to roll out apparent features sooner, but ignored the likelihood and eventuality of bugs. Afterwards, when we got back to the scalable highly modular version, we started taking biz from competitors who couldn't scale. I'd say that my mistake was to only give the broader consequences, and not spend time to be able to enumerate in detail the consequences of non-scalable quick program would be. Of course you cannot predict exactly what bugs will happen, but I could probably have done a better job of drawing the scenarios (not sure it would have made a difference, but it might have).
I'd also say that we need to create specific structures and plans to study and quantify risks, as is done in real engineering like aerospace, architecture, etc. Classify those risks into a range of categories, from small bugs to existential for your customers or project.
Different steps need to be taken for each class, and significant part of the planning needs to go into de-risking the project.
I'm in physical vs software development now of carbon fiber type technologies, and I notice that my military customer who are building very cutting-edge stuff often talk of 'de-risking' the project, whereas I don't hear this much from other customers. Seems like an important distinction to take on board.
---
From a user perspective, I noticed after looking at the issue on our own Echo yesterday: the UI is a totally greased slide to hide choice for the user and slide them right into giving permissions for contact list. It seems that effort was made to hide the actual features and functions that will result from giving permission, and obscure the 'Skip' option. So it would be easy to not even notice that your device had these new possibilities. Obviously, I'd recommend taking more time to sell the features and let us make an informed choice. Then even if things go wrong, you'll enjoy some benefit of the doubt in the market and press.
> First, the Tech/Dev managers need to stand up. Instead of saying 'Yes' to every feature request, they need to say 'No', or 'Later'.
Yes, I see this a lot too. I think software managers have more incentive to get new features deployed. I don't think this is a good way to measure their performance as a manager because of the consequences we've already mentioned.
I would also like to see more concepts from physical development implemented in software. At times feels like the wild west out here and too often we ignore the lessons from similar experiences.
I know amazon's got a different motivation matrix than a startup; e.g., Amazon won't die if some feature isn't delivered by the next trade show, but they do have competition from the other majors.
That said, amazon certainly also has the funds available to invest in a parallel risk team. If they're not motivated to do it from the risk to their users, the best argument might be the potential reputation setbacks if stuff like this gets out there & causes problems, bad press, reputation for creepiness, etc.
I know it always seems inevitable that you'll weather the reputation hits from errors and just press on to greater usage/adoption/sales, but it will always seem that way from the inside -- until it doesn't. Google's experience with Glass comes to mind; could have been a fantastic product, but it went just under the tipping point of being creepy, and poof, they're gone. It'll be a generation before anything similar comes back.
I'd hate to see that happen to Echo/Alexa. TBH, it looks like this product has both greater potential, and also greater creepiness potential than Glass ever did.
What is this alleged 'conspiracy' story? An Amazon device recorded a private conversation and transmitted it to someone else, and it did so without the user's knowledge or intent. That actually happened.
How it happened is only relevant to the engineers who build and maintain the thing. I, on the other hand, could not care less how it happened, and the fact that it did happen is reason enough never to buy one of those infernal devices.
>How it happened is only relevant to the engineers who build and maintain the thing. I, on the other hand, could not care less how it happened
I'm sorry, this is just stupid. If you cannot see the distinction between a feature that was intended to spontaneously record audio, and a bug caused by faulty voice-processing, then that's on you. The distinction is pretty critical
Critical to what, exactly? The consequences to the user are the same.
Obviously, intentionally designing a feature that spontaneously records and transmits audio is a problem for many reasons. But the lack of intent does not magically erase the consequences for the people who experience this kind of bug.
And, to be clear, this was not simply a matter of "faulty voice-processing". The fact that this could happen without the user's knowledge is a problem in itself. Clearly, there are inadequate visual and audio cues, and insufficient or nonexistent verification. Those failures are not bugs; they are bad design and engineering.
>There's no way a team would get the "ok" to build a skill that randomly records private conversations then sends them to a random contact.
If you took privacy seriously you would have added an audible prompt to ask for confirmation before sending or similar safeguard.
I mean if you are an engineer then you know that there is a nonzero false positive rate for the device to detect the "call" command.
Knowing this, and not implementing a trivial safeguard sure seems to me like the team was "ok" with building exactly that, a device that: "randomly records private conversations then sends them to a random contact."
> Yes, I might sound biased because I am an engineer at Amazon.
> This statement is my own and unrelated to Amazon's opinion.
Just because you admit your bias doesn't dismiss you from it. Your statement and opinion could very well be influenced by your work surroundings at Amazon. While I do not doubt that Amazon does take security and privacy seriously, I find it hard to believe that they employ proper ethics (based on this and past examples such as the NYT incident).
I know you feel like the story is not being fairly presented, but you are almost certainly not helping Amazon by posting here about it.
Why? Because from a PR perspective, it doesn't matter what Amazon or the engineers intended, all that matters is what actually happened. You said:
> There's no way a team would get the "ok" to build a skill that randomly records private conversations then sends them to a random contact.
But that's exactly what happened. Whether it was a bug or a feature doesn't matter. Imagine if the wing fell off a plane and an airframe engineer came on here and said "hey, there's no way we would get the OK to design a wing that falls off in midair." Yeah, we know. The fact that it happened by accident doesn't make it better (maybe worse, actually).
Now you've got dozens of people responding to you, and BTW a lot of reporters read HN too. Do you really want to see stories like "Amazon Engineer Calls Customers Conspiracy Theorists"?
> There's no way a team would get the "ok" to build a skill that randomly records private conversations then sends them to a random contact.
Of course. But it is entirely plausible that somebody builds a skill that records certain conversations, and somebody builds a skill that sends recording somewhere, and then due to some bugs or coincidences or missing controls to prevent such occurrence, the first skill is activated when it should not be, and the second one is activated when it should not be and with wrong parameters.
So there should be more checks. Like - ask for explicit permission before sending any voice recording out, or ensure that the target where recording is set is validated against whitelist explicitly set by the user, etc.
Actually, I can think of a very good use of the ability to trigger silent phone calls. In the case of home invasion or domestic violence. If Alexa detects what could be signs of distress it can dial an emergency number without alerting that it was triggered. Otherwise, trying to shout "Hey Alexa, dial nine one one" will alert or enrage the perpetrator.
Sorry but there is no real reason to give Amazon the benefit of the doubt on anything related to data and privacy. You guys can feel free to roll your eyes and tell people they're overreacting all they want. I find it revealing. We get it, people are just holding it wrong. Okay.
Look- we all know that bugs will happen. The question is, what will you do in light of that? What bounds will you set up to prevent the system from acting in ways you want to disallow? How will you detect something is awry? I would say that really caring about people's privacy means doing this part seriously, not just a baseline lack of active malice.
I have not experienced that, can you elaborate? They keep spamming me with mails of all there different services, show me what I should buy, etc. No way off even turning these things off...
I shouldn't have called this a "conspiracy story" and I should have given more respect to the parties involved who experienced this. While I think the issue is similar to a "butt dial", the customer felt violated and more precautions should have been taken to prevent this.
What I should have said is this report makes it difficult to understand what actually happened. It seems clear this article favors click-bait quotes that insinuate a "big brother" vibe such as:
> "'unplug your Alexa devices right now,' she said. 'You're being hacked.'"
If she had instead said "Alexa butt-dialed me!", would you still be interested in this report?
Android has per-app microphone permissions that you can disable, if you want. At least in theory, Google apps honor these permissions, and on LineageOS, you can verify that this is the case - while with the Alexa, not only is there no way to audit any of the code running on the device, but it's explicitly marketed as being always listening.
Oh, this is true - although the likelihood and ease of it being exploited as a listening device are much lower than that of a device explicitly designed to do so.
Edit: Not to mention that the entire Alexa is a black box, too.
Second Edit: Not trying to claim that you're an idiot, just provide some perspective.
Or maybe they just value different things than you do?
I have an Echo on my kitchen counter. I know there's a small chance that it may be hacked or have a bug or be deliberately modified to surreptitiously record from my kitchen audio that ends up on some Amazon server somewhere, or in the hands of law enforcement, or the NSA, or hackers, or whatever.
I just don't really care. I like the convenience of the device, there's a lot of scrutiny on the devices and the companies that make them, discouraging deliberate misuse, and I don't really say anything interesting or worth hiding. And if someone is going to deliberately target me, my life is jammed with devices that are probably a much bigger risk surface area.
I would never say anyone else should get one if they feel the privacy risks outweigh the benefits, but I'm not personally concerned.
However, I may be wrong. So change my view: why should I care or be worried?
I feel that while these devices are great in principal, there still needs to be work done on whether the "trigger word" activation is the best way to enable them. Maybe we need longer / more complex trigger words? Maybe there's a different way we can activate them alltogether, avoiding the "always listening" problem?
I'm "involved with tech" (developer for 15+ years) and I have two of them. You might also be horrified to learn that my house contains multiple smartphones, computers, routers, and other devices, all of which are connected to the internet!
I am also a developer,living in countless universes, each enveloped in its own shell, compelled by the wheel of time, like particles of dust blowing about in the sky, and I don't understand what's wrong with you.
I feel creeped out by these home listening devices and I don't own one, but don't our phones already have this capability? You can turn "Ok Google" on on an android phone. I sometimes record audio, and the mic is incredibly good. Is there a substantial difference between our phones and these devices?
EDIT: Just realized the substantial difference is that Google and Amazon own all of these things. They don't control all makes and models of phones.
Not sure about Android phones with "Ok Google" but on iOS, all of Siri's voice processing is on the device. As opposed to Amazon Alexa which sends the data to the cloud for processing.
I got an Echo Dot a month ago, because there was a special offer on Amazon for the Philips Hue system, and it included one free. After reading this and having a discussion with my girlfriend last night about how she needs a backup of something, but we don't have a DVD writer at home (she's not tech savvy and thinks people still back-up stuff on DVDs), and now, the second day my Amazon daily offers is filled with portable DVD writers, it starts to creep me out and the first thing that I want to do when I go home is to unplug it. I know it can be a crazy coincidence and I was never the kind of guy that believed this Facebook is listening to you, but still, what if they actually just listen for some keywords?
It might just be a coincidence and a psychological bias that you noticed it, but given that it would have a very, very clear economic benefit to Amazon if it were true, we're stuck with these facts:
1.) They have an economic incentive to do so
2.) Only they know how their systems work
3.) Their network traffic is encrypted
4.) They face legal risks and user backlash if caught doing so
Given 2+3, you can't be entirely sure that they aren't doing it. If they deny it, your only recourse is to hope that their cost/benefit calculus considers 4.) to be more costly in terms of dollars.
"Okay Google" and "Hey Siri" run locally on the device, but all other transcription is done "in the cloud". Try setting your phone on airplane mode and going "Hey Siri, What time is it?"
It will recognize the Hey Siri, but give an error for everything else. Pretty sure "Okay Google" will behave the same way. Also if you turn on battery save on iOS it disables "Hey Siri".
This. It would be a tremendous battery drain if your phone had the microphone on at all times, with a constant connection to the server, sending all audio over it at all times, just so it can detect when somebody says "hey siri".
It's a custom low-power chip they added on the 6S and later to allow a local tight loop that only listens for that utterance using a hardware-assisted neural net, and only activates the rest of the software stack if it detected it with reasonable confidence.
I have this idea of a system I would like to have in my house. It contains cameras in every room that are constantly watching where people are and relaying the coordinates to a central server. That server makes decisions on if lights should be on or if A/C should be running in that room. But I would never buy this system. I would have to make it myself. I am hopeful that open source software and hardware can produce individual components that I can trust to piece together.
They should feel more comfortable than an equivalent system built by a company that is looking to profit off of your data - and additionally, you can give the guest stronger guarantees that when you say that the system is "off", it actually is.
The chance that a random implementer has a security vulnerability is much higher than that Jeff Bezos is listening to me watch TV. A private system is more vulnerable to target attack and an Amazon system is more vulnerable to mass surveillance.
You are on the right track, but I don't think you are quite right. Amazon like systems are more vulnerable to mass surveillance.
Your vulnerability isn't a targeted attack: you are not valuable enough to be worth the effort to figure out your system. As an attacker on your system I'd have to figure out how to break in, and then how to use the hardware you have. You are more valuable as part of a botnet - attacks that already exist. (if you are a politician then maybe, but that person is also vulnerable to a targeted attack on their amazon system - probably more so because the target is easier to figure out).
And what exactly does this "vulnerability" mean in real terms? Let's be honest. No one cares what conversations are going on in your house _unless_ you're someone specifically targeted. There's little use in mopping up data with no goal.
If mopping up everybody's conversations is cheap enough Russia/Iran/China/(insert your favorite large evil) does. If you happen to run of political office 15 years from now having all your conversations available to analyze will be useful. If they don't like you, you might find some out of context snippet of "private conversation" all over social media killing your campaign. (or alternatively the blackmail threat if you don't X)
That is they will target everybody because they know in a few years that will include somebody who they currently think is a nobody.
Of course as AI gets better and cheaper they may eventually listen to everything to see what who can be targeted automatically for what.
If I'm using a dragnet to grab every conversation and filter it for things I personally care about, your conversation could make you someone I want to specifically target.
As a random example: let's say that I want to kill someone who lives in your neighborhood. I could analyze your conversations, comings and goings to figure out when and how to kill that person, and blame it on you. I could on a continual basis run numbers on everyone emitting data in your neighborhood, until someone had arrived at a point where their friends would testify against them on the basis of conversations with the potential patsy, that patsy had no alibi, and tailor the murder on the basis of the means that the patsy had available to them at the time.
I could also just be trying to figure out if you were a homosexual, or muslim.
A secure language does not protect you from insecure design. I could very easily build a system in Rust with gaping security holes, purposefully or accidentally.
>> Even if you build it, knowing guest may never feel completely comfortable visiting your house.
> They should feel more comfortable than an equivalent system built by a company that is looking to profit off of your data - and additionally, you can give the guest stronger guarantees that when you say that the system is "off", it actually is.
From what I gather, if you need fine-grained location information in a building, you can do this without cameras and just a handful of strategically placed wireless access points.
Less likely to be seen as creepy and such systems already exist.
If you just need 'is a person in this room' or 'how many people are in this room' levels of data, solutions to this problem have existed for decades. No need to over complicate a solved problem.
If you are cleaver you can use the wifi signals as a radar and find people/objects even when they don't have a wifi device on them.
I'm pretty sure all current technology only works with the have a wifi decide on your person. It is significantly easier to do this and for the most part good enough.
Sounds far-fetched to me. If you are cleaver, you are also likely to get stuck in a coffee cup billboard or Eddie Haskell will trick you into insulting your Spanish-speaking friend.
Not only can you detect motion and objects through physical mass using WiFi spectrum with a properly equipped device, you can detect if someone is breathing.
“What's more, this "Time Reversal Machine" technology is essentially just some clever algorithmic work with little burden on the processor, so it can potentially be added to any existing WiFi mesh routers via a firmware update. In other words, security system vendors should take note.”
Such a system would be a huge GDPR nightmare. You'd need consent from every visitor to your house to collect data on them, and also you have to delete it if they ever request you to. Best not to even try.
Interactions between private individuals are out-of-scope.
The following processing is outside the scope of the GDPR:
- any activity outside the scope of EU law (e.g., activities of a Member State in relation to national criminal law);
- any activity performed by Member States when carrying out activities in relation to the common foreign and security policy of the EU;
- any activity performed by a natural person in the course of a purely personal or household activity;
- any processing by the EU itself;
and
- any activities performed by national authorities for the purposes of prevention, investigation, detection or prosecution of criminal offences, or performance of judicial functions.
You don’t need to use cameras for this. A simple speaker and microphone is all you need to make a functioning motion sensor. Just exploiting the doppler effect. And even better, you can do it all outside of the human audible spectrum. And it can work with capturing motion around corners, too, since sound bounces off walls.
I’d recommend something like this in every room, or even IR sensors, over cameras. You don’t want to capture video of your children jerking off.
A lot of home automation stuff seems like something that would be fun to make, but not something I really want to have. I mean, the AC might be a good energy saver (a smarter thermostat) but that's not something I want, more like something that I would buy if it was cheap and practical enough.
To some extent, this is a phenomenon of early technology. It feels full of potential, so you want to play with it but it doesn't really do anything you really need yet. The early web was ki d of like that. We made sites because we wanted to make sites, moreso than because we wanted to have them.
Home automation though... Im kind of skeptical that this goes anywhere useful. The useful examples people think of (eg remote close all the windows and lock the doors) are more about mechanisation than automation.
At my house, we have blinds installed on every window, and every evening I go around the house, pressing 12 switches (2 for every blind, because UX is obviously optional) so it's perfectly dark in the bedroom. In the morning, I go around the house and press those 12 switches again, to let the sunshine in. Now that's a task I would love to automate!
Alas, remote controlling those blinds would be a major hassle since AFAIK I would have to install 6 wifi-enabled devices and tear holes everywhere (not even sure there is a powerline near) - and likely do all the programming myself. Thanks but no thanks.
If it bothers you enough, take a look at ZigBee switches. Not sure I understand how you operate the blinds (since the current switches are apparently not on powerlines?), but you could still make a centralized solution to control them over ZB. There's https://www.home-assistant.io/ if you don't want to program (much), and probably others. And you will smile every evening and every morning for the next few years, thinking of the time when you had to do it manually... not to mention, it's cool. :) You might need to invest something in these devices and controller though.
Yeah it's not really relieving a pain point. I've never thought "O why lord, why must I toil away flipping these light switches as I enter and exit rooms?"
Another pain point with making is that it's really hard to get it integrated with stuff you have bought. I built my own garage door automation with a Particle Photon board. It works great and can do things like text me if I leave the house with the garage door open using the IFTT support from my WiFi router. The problem is that it's really hard to get it integrated with any other control system that the rest of my house uses like my ZWave light switches and Hue lights.
I've been working on a custom UI that sits on top of the Wink Hub API to unify everything, but I'be been stuck with their almost completely undocumented Pubnub event API.
I would check out https://www.home-assistant.io/ for integrating multiple systems together. I use it on a RasPi to integrate a few disparate systems (Amazon, Nest, RadioRa2) and it works very well. There are modules for most existing systems and it's easy to write your own in python.
I probably should try it. I've been writing my own partly as an excuse to learn React. I've already got a Pi with a touchscreen and a 3d printed enclosure setup to run whatever solution I actually end up using.
I know I’m a little late here, but I would look into MQTT as a transport layer for messages across your different devices. It’s super easy to interface with via python or a host of third party services.
https://www.home-assistant.io doesn't have computer vision, but can use motion sensors and the like to tell where people are, and use that to activate/deactivate devices. It's all open source and runs entirely locally.
I actually used to have a solution with home-assistant, find (uses wifi signal fingerprinting for location) and hue. It worked okay with very little work (sometimes 1-5s delay turning lights on/off) but I never cared enough to put more work in to get it better.
I think "find" and similar tools are now much more advanced so it might work better out of the box now.
There's a whole class of tech like this for me. An Alexa/Echo/etc, a fitness tracker with GPS and sleep monitoring, a maps program that learns my routine and integrates with a weather app, and so on.
And ideally? All of it integrated. It actually sounds nice to say "I'm going home", and have Maps say "today that will take 35, should your oven start preheating when you're 20 minutes out?" IoT devices are overrated, but I can absolutely imagine a critical mass of integrated tools being very useful.
But I'm not even slightly willing to do that. It's too much information and too much risk surface. I'd pay a hefty premium to get local-data-only versions of these products, but no one is offering that, and it doesn't look like they're going to start.
Maps currently does do what you're saying - Google knows the time I'm going home and displays current traffic information and estimated time to arrive, at least for me.
Same. But it gets such obvious things wrong. Instead of "traffic to [daycare center] is light" it says "traffic to [some weird company name probably registered in the same building] is light".
Waze also knows where I'm going in advance and most of the time it gets it right, like every Thu evening "are you heading to [evening school] ?". My weekly schedule is exactly the same 99% of the time but sometimes it still makes obvious mistakes. If you need an AI for that it must be a very simple one, yet it still fails.
Google maps also auto saves my parking location most of the time but other times it doesn't, for reasons unknown. Things like that make it hard for me to have faith in future versions of this stuff.
>a maps program that learns my routine and integrates with a weather app
That sounds awesome until the company providing that service starts abusing their knowledge of their location. That abuse doesn't even have to necessarily be malicious in nature either. For example, Google Maps on Android started asking me to rate, review, and/or take photos of my present location if they deemed it a point of interest (certain restaurants, parks, etc). I never opted in to this feature and the only way to disable it that I've figured out is to literally disable all location services on the phone.
I really dislike the idea of Google storing a timestamped record of almost every place I've ever visited. Tt has to beconstantly phoning home in order to deliver the request to document my visit within a minute or two of my arrival and that constant reminder that Big Brother Google is tracking me at every moment is just disturbing on so many levels. Even if they aren't using that data right now, remember the "data is never destroyed" principle of the internet.
On a side note, I would have ditched Android if I didn't need it for work simply because use of the GPS radio is hidden behind the acceptance of enabling Google's Location system and all the invasions of privacy that entails.
I feel the same way. Between that and my overuse of my phone for checking hn, reddit, etc. I am considering trying to make a habit of leaving my phone at home when I leave the house, at least some of the time. It's a shame that I feel that way about such an incredible useful device, but there it is.
Any idiot who breaks into your home would be well positioned to steal your server along with all of your memories and use them for ransom/blackmail/etc.
As opposed to any idiot who breaks into your IoT provider's server?
An encrypted hard disk with the key on a USB stick would be enough, just keep the key somewhere separate and you'll only have to plug it in when there's a power outage.
I like the idea (though I think with enough abstraction, you could have it also replicating itself to "regular" cloud).
The main fight needs to happen at application level, not infrastructure. Cloud services are already mostly transparent and interchangeable. But applications aren't. The problem is, it's the application vendor that owns the code, determines where's going to run, and asks you to send over the data. How it should be working, is that you own the data and determine location of computing, and own or rent code to be run on that data.
IR camera would probably make this simpler than a visible-light camera. You want to know if there are people in the room, you don't really care who they are specifically.
> I have this idea of a system I would like to have in my house. It contains cameras in every room that are constantly watching where people are and relaying the coordinates to a central server. That server makes decisions on if lights should be on or if A/C should be running in that room. But I would never buy this system. I would have to make it myself. I am hopeful that open source software and hardware can produce individual components that I can trust to piece together.
You don't need cameras for that, just motion sensors.
> That server makes decisions on if lights should be on...in that room.
You don't need a server for that, just a motion-sensing switch. They can be totally offline. My office has them to shut off lights automatically when a conference room becomes unoccupied.
That's the main problem with motion activated office lights tbf, especially when you're doing software development for example - not enough motion to keep them on.
Every time! The coffee shop by my old house had motion sensing lights in the washroom, and every damned time I would be plunged into darkness half-way through my visit... Urgh.
Ours does too, but there's no sensors inside the stall... You can wave all you want, there's no way to turn the lights back on without opening the stall door.
My problem with the motion activated office lights (at least the ones at the wework my company is located) is that you can't turn them off. The button appears to turn the lights off and disable the motion sensor for some (short) period of time, then movement turns the lights back on again. Very annoying.
OSHA also requires a minimum light level in the US but I believe it only pertains to hazardous working environments (like on a production floor in a manufacturing facility for example). Either that or the enforcement in white collar environments is so weak that literally no one cares to follow the law which is surprisingly common with reagrd to many workplace regulations.
FLIR for turning off a light in a room seems like a ridiculous amount of overkill. I don't know numbers off the top of my head but I feel like at that point you're using more energy for FLIR than the light itself. Not to mention the cost of a single FLIR sensor is probably pretty high.
One reason you might want a server is to have more expressive power than "if someone literally enters the room, turn on the lights". You might want to turn on the lights and run the AC a little before you predict people will arrive, for example.
> One reason you might want a server is to have more expressive power than "if someone literally enters the room, turn on the lights". You might want to turn on the lights and run the AC a little before you predict people will arrive, for example.
Maybe, but that seems like a lot of work for little gain.
You probably don't actually have that much control over the AC in your rooms unless you have zoned heating and a lot of zones, regardless of how smart your sensors are.
Intent is a very important factor when answering the question "should the lights be on," I don't think you'll be able to predict that. For instance: if movement is detected in the bedroom at 2AM, should the lights come on? The answer is: a very strong maybe.
> Intent is a very important factor when answering the question "should the lights be one," I don't think you'll be able to predict that. For instance: if movement is detected in the bedroom at 2AM, should the lights come on? The answer is: a very strong maybe.
You would be collecting data from many sources in order to predict intent. This is why you need a centralized server.
> You would be collecting data from many sources in order to predict intent. This is why you need a centralized server.
Please explain, exactly, what other sources you would collect data from and how the central server would process it to determine if I want the lights on in the middle of the night.
IMHO, determining intent in this scenario is impossible without...
1. a mind-reading sensor, or...
2. an explicit user signal, such as a button-press or command.
The only realistic option is a user signal, and most of those options obviate a lot of these prediction ideas.
I think there's a lot less practical value to having a "central server" controlling everything than you seem to assume.
> Please explain how, exactly, what other sources you would collect data from and how the central server would process it to determine if I want the lights on.
The more data, the more smarter. In fact, the only difference between a thermostat and the human mind is the number of datas. This is because the Law of Averages predicts that half of all datas will be relevant.
> Please explain how, exactly, what other sources you would collect data from and how the central server would process it to determine if I want the lights on.
Machine learning navel-gazing is the new "throw a start-up at it", so I'm guessing the answer to this is going to be "if we have enough data..."
While I haven't had the urge to get any 'smart speaker', I have a hunch that the current hype* around these devices comes mainly from the novelty of being impressed with voice recognition.
As someone who doesn't have a voice for radio, I much prefer being able to interact through dexterity. Besides, I'm very picky when it comes to which specific track of music is to be played (out of various similarly named pieces of music).
* I don't think take-up is quite as big as the tech world currently makes it out to be. People in my circles don't really have that kind of disposable income and are prioritising other purchases first.
So yeah, Alexa is a lot of extra literal work for little gain.
If you don't have multiple zones you can still effectively have zones by opening/close the registers/vents. This will prevent hot or cold air from entering the room. However, it will also increase duct pressure and that could be somewhat problematic. Either way, "smart vents" are on the market. They're ridiculously overpriced though and one could hack together an alternative for under $50 per vent (source: personal experience). Probably less if they are clever. Of course then there is the problem of running power to the vents or having sufficient battery. But that's an exercise left for the reader.
Your solution is simple. However, it doesn't allow for as much modification as the original poster's idea. With his idea it could be modified in so many ways to add functionality because it wouldn't be limited by the technology. The solution you supplied will eventually be limited by the technology if the original poster wants to add other functions.
Sure, but what is the value gain/opportunity cost between a simple, two hour installation that achieves a majority of the desired outcome with nearly rock-solid stability vs. sinking a massive amount of time into a bespoke and likely fragile system?
> However, it doesn't allow for as much modification as the original poster's idea.
My main point was the original poster's idea was probably focusing on the wrong kind of sensor for what he wanted to do. He could still network a bunch of motion sensors.
Also, I'd dispute the idea that my proposals were less "modifiable." If only because they're far easier to implement and a couple of orders of magnitude cheaper, so it's practical to replace if more capabilities are needed.
There may be a market for something between a standard one-bit low-rez motion sensor and a full color TV camera. Maybe a 16x16 pixel IR sensor with a fisheye lens and a puny CPU that reports an approximate number of people in the area, for HVAC and lighting control, and security.
Reporting an approximate number of people is not a trivial task, even with a real camera. Depending on what exactly you mean with "approximate", of course.
> 0, 1, a few, many. Just enough to tell you how much to crank up the HVAC.
I don't think the number of people in a room will give you meaningful information to tell you "how much to crank up the HVAC." Also, the HVAC systems in most homes aren't capable of even cranking up the HVAC in a particular room.
YAGNI. You can get to a 90% solution today at 10% of the cost and effort. If, later, you want to extend the system you’ll not only have learned a lot about the operations and failure modes of the current system, but hardware purchased later will be cheaper and may support even more functionality.
> The second advisor, a software developer, immediately recognized the danger of such short-sighted thinking. [...] "A toaster that only makes toast will soon be obsolete. If we don't look to the future, we will have to completely redesign the toaster in just a few years."
This is eerily similar to the concept of a 'cookie' seen in the Black Mirror episode, White Christmas.
(Spoiler Alert)
A cookie is a device "that is inserted under the clients head by the brain and kept there for a week, giving it time to accurately replicate the individuals consciousness. It is then removed and installed in a larger, egg shaped device which can be connected to a computer or tablet (to automate their smart house controls as if the house was knew their personal preferences moving from room to room.)"
I had exactly same idea just about the lights but am too lazy to even try to implement it. Plus sensors require power and that requires cables and that's way too much bother.
Low tech alternative solution from my friends? Get LEDs. Never turn off the lights. This way the room you are going to will always be lit.
> "That server makes decisions on if lights should be on [...]"
Why would you even want such a system?
My flat uses the Button-Framework which provides a really convenient UI/UX. The edge between 2 nodes (rooms) has a button (a haptic device to switch between boolean states) at around hand-height that you can press to switch the light on or off. It works quite well (it uses a switch-system technically) and the user decides by her/himself if the light should be on or off and just presses the button after careful reasoning.
Granted, it doesn't use Docker, but it really works well and needs low-maintainance. My model is running since 23 years and I never had any issues - it's even open source.
That's the "Facial recognition fooled by funny-colored glasses" study from Carngie Mellon, where researchers were able to make machine learning algorithms fail disastrously (e.g. mistake a man for Milla Jovovich) with a pair of glasses printed with what looks like a random assortment of colorful pixels, but is in fact a targeted attack specifically designed to trick the algorithm.
This Alexa failure is obviously not a targeted attack; but when your system is exposed to enough data, eventually you will stumble over some input that happens to resemble a targeted attack by pure chance, right? It's equivalent to saying that if you aimed the facial recognition algos in that paper at millions of faces wearing randomly-colored glasses, eventually some of the glasses would be close enough to the targeted-attack glasses to produce the same effect.
Obviously I'm theorizing on almost no data here - I don't know anything about Alexa's voice recognition, maybe it contains no ML at all. But it seems plausible that this might be what happened here - not a bug per se, but the natural and totally expected result of giving an opaque, machine-generated system with a very low failure rate so much input data that the failure rate is significant.
I recently stayed at a house which had an alexa device. In a conversation where I said the words light switch several times Alexa beeped and responded to me each time. I think most of us just believe Amazon when they say "Alexa responds to its name" and don't stop to consider the possible failure modes. We want to believe it can understand our words, when it's really just guessing. Over long enough time it's inevitable that it will misunderstand you and do something you don't want.
Still, I wonder how it heard "record this conversation and send it to someone on my contact list".
731 comments
[ 3.0 ms ] story [ 313 ms ] threadBut either way, they should surely be confirming this before send, the same way Siri does when I ask it to send a text.
I haven't used a smartphone in years. Looking back on the days when I did use a smartphone, there was always this tiny fear in the back of my head that maybe I'd pocket-dialed someone and they'd be able to hear my conversation. It actually had a subtle chilling effect.
The Hawthorne effect (which discusses how people react to being observed) seems to be especially relevant nowadays as there's a real possibility that we're being recorded at any time. https://en.wikipedia.org/wiki/Hawthorne_effect
It's actually a big theme in 1984 where the mere possibility that you were being listened to via telescreen was enough to keep you in line and influence your behavior.
Could it be that the volume was set very low and they inadvertently triggered their way through the entire dialog? I've accidentally set alarms from a few rooms away but this is a whole other level.
This seems dangerous, some smart house could open the doors or stop the heat or stream your entire live on the internet.
Some POST requests may only need to let you know that they did it, such as adding a calendar reminder on to YOUR calendar. It should ask if it wants to send a calendar invite to someone else's calendar.
Based on Amazon's admission and apologies, I'd think they'd be willing to give a lot more than just a refund. Seems like a huge liability for them.
In terms of the big picture, this was dumb PR wise and this story should have never happened. For less than a $1000, Amazon could have stopped the story if the PR dept were more in sync with customer service
Not saying it happened here but thats how you get $10/hour vendors making judgement calls on customer situations that are front page news.
My point still stands. This is Amazon and not random company X's customer service. There are very few companies like this that people can name (e.g. Nordstrom, Patagonia, ...) It this used to not happen at Amazon. Customer service used to be empowered to make customers happy, which was what made Amazon legendary. Things have changed. Customer service is now crippled like almost every other mediocre company. This is just one of the symptoms. Maybe Amazon feels that it has enough market share now, so it doesn't matter as much?
I guess I just don’t have a long enough track record with them to see them in that grouping of great customer service. They’ve pretty much fought me tooth and nail over price matching and prime shipping that doesnt meet the two guarantee over the last couple of years which is probably biasing my view.
Inappropriate, worrying, but not surprising.
Also why could it send data out without any confirmation from the user.
And the LED should be forbidden by law from being software-controlled.
They should also have a physical inline toggle switch to disconnect the microphone(s) entirely, whose state is easy to visually confirm.
WiFi is always on. WiFi doesn’t disconnect just because you aren’t actively moving data. You’d be pretty annoyed if every time you wanted to send or receive data your device had to reconnect.
We're also discussing theoretical hardware changes, there's no reason it can't have 2 sets of microphones where one is hardwired only to the wake-word processor which has no direct connection to the main processor except some one-way signalling, and the LED is hardwired to the second set.
The idea that there would somehow be a dedicated wake word microphone is a little ridiculous. Firstly, no one would trust this supposed 1-way connection. Second, it would require a dedicated processor to make the wake word even work, driving up costs. Third, the echo uses an array of microphones so your wake word would either be unreliable or drive costs up further as you duplicate the entire array. Hardly a net win.
The reality is that if you don’t trust amazon to do the right thing you shouldn’t install their listening device in your home. (Likewise for your phone.)
If the microphone uses an amplifier, you could wire an LED to light when power is supplied to the amp. The indicator light is physically part of the circuit, so its operation cannot be modified by software. There are probably other, better, ways to do it, I'm not an electronics guy.
I don't think that would be a bad thing; especially if there's a switch to disable the mic. When you turn it off, you'll get reliable feedback to know it's actually off.
I think it's important that these kinds of devices have simple feedback and control mechanisms that can be independently verified and reasoned about. Software is too opaque and too untrustworthy.
I see that, I think it would be best as a separate light.
I think product UX has drifted too far towards blank monoliths; I know I wouldn't mind a few more blinkenlights :)
Doing it via microphone/amp wouldn't do it, since you'd still want to use it without LED on (and not software controlled).
We don't live in a world with a healthy enough market or enough competition for that to work.
It has that. They call it the power cable.
If you disconnect the microphone, an echo is pretty useless.
If, even with that expectation, enough people still use "voice assistant" technology for a judge to consider the tech to be "in general public use"[1], the bright line test defined by Kyllo v United States is triggered and the police no longer need a warrant to use the technology (in the abstract - they don['t need to use your hardware) to view "details of a private home that would previously have been unknowable without physical intrusion"[2].
Normalizing the expectation that previously private areas (like the inside of your home) might be recorded and sent to a remote 3rd party will eventually result in everyone losing some of their 4th amendment protection against search and seizure.
[1] http://caselaw.findlaw.com/us-supreme-court/533/27.html
[2] Ibid.
It's like the misguided Prop 65 signs that are in practically every commercial building in California so instead of warning people about dangerous levels of hazardous chemicals, they get a meaningless sign in every building for hazards that are no worse than if they were walking down the street or sitting at home.
That would require, though, that it's not buffering the last N seconds of audio to reprocess once that processor wakes up. Do any/all of the modern smart-speaker devices do that? If so, then you'd have to take into account that when you see the light, you've potentially leaked any secrets you said in the last N seconds as well. Less like a reporter coming in and asking to speak to you; more like an eavesdropper coming in and telling you they heard what you were just saying through the door.
As nerds, it makes sense to us to do local wake recognition and only upload a request phrase after that. But in reality, this not the first time[1] we've learned that continuous speech was getting uploaded.
Charitably, it could be a mistake, maybe the previous request never terminated and it just kept streaming. But again, not the first time.
More likely, my opinion, is there's a mountain of cash on the table in the form of private conversation, and AMZ would be foolish to walk away from it. After you talk about "hardwood floors" in your house, you could expect targeted ads, consumer profiling, variable pricing, and third party sales. Why woudln't they do this? Why else are they pushing home devices so hard? They're front and center on the front AMZ page every time you visit, aren't they? Oh and GOOG too, they just haven't been caught yet.
1. https://www.wired.com/2017/02/murder-case-tests-alexas-devot...
With that said, my Sonos Alexa has a tiny LED but does respond with an audible tone when the keyword is heard.
That is... when it's not randomly laughing https://www.theverge.com/circuitbreaker/2018/3/7/17092334/am...
Talking loudly in another room. Alexa: "Do you want to send this message to Jim?" "Alexa, you suck"
A confirmation after every request makes the device a lot less useful, I suspect most people do not want the device to confirm commands before it does them.
It's like saying that someone buys a car that's powered by an explosive liquid, and then complains when all the gas in the tank explodes at the same time.
Given that, it would be ridiculous to expect paying customers to switch it off when not in use. The onus on Amazon is to protect their customers' privacy while the device is being used as intended.
If you really want to pick apart the car analogy, maybe this is closer to having automatic start and driving capability on a car, but the car sometimes randomly starts and drives away. Are you really going to blame the customer for that, even though they could disable the feature every time they exit the car?
I think for some, at least users like me would like a feature similar to "su". I'd like to confirm that Alexa should start doing stuff if it has not been confirmed for x number of minutes. I think the real problem with these home assistant devices is they are not designed with a confirmation message. This would be annoying in some cases, but surely a balance can be struck.
I'm not trying to be reductive at all about the car analogy. When you turn on a car, you acknowledge that the engine is running. You can also mute an echo if you have the foreknowledge that you will not be doing activities that require voice activation.
But there is a hilarity of, on one hand being privacy centric, and with the other hand, planting a listening device in ones home.
> an Alexa engineer investigated ... they said 'our engineers went through your logs, and they saw exactly what you told us, they saw exactly what you said happened, and we're sorry.' He apologized like 15 times in a matter of 30 minutes and he said we really appreciate you bringing this to our attention, this is something we need to fix!"
> the engineer did not provide specifics about why it happened, or if it's a widespread issue.
> "He told us that the device just guessed what we were saying" The device did not audibly advise that it was preparing to send the recording, something it’s programmed to do.
Apparently it's not programmed to do that. Unless this was a hardware glitch or cosmic-ray event.
Bugs happen in architecture, aircraft, etc. too. the difference is that the actual engineers are paid to have a precautionary approach and spend significant resources to actively prevent bugs from making it into the final product.
In contrast, software is often written to "ship first", be "agile", and "move fast and break things". Yet when it causes problems, they just say "bugs happen", and "it is unfair to vilify them".
Features are not better than reliable security.
And yes, negligence is less bad than malice, but it is still damaging and developers and managers need to be held to account.
If your manager is pressing you to do unsafe crap in too big a hurry, it's your responsibility to push back, and if unsuccessful, leave for saner pastures and make it more difficult for that management to proceed.
Well, yes and no. Bugs, security, reliability, etc are all important things to consider, but they can't be the only focus. Security doesn't matter if the thing you are creating has no features; it would be the same as if it didn't exist at all.
Instead, we must manage risk; the risk of bugs, the risk of security vulnerabilities, etc. Nothing we do is risk free. Even walking across the bathroom floor has SOME risk; we might slip and fall. Does that mean we should just stay in bed all day to avoid any risk?
No, we measure risk by factoring the chance of the bad thing happening and the consequence of the bad thing happening. We then determine how much effort we should spend on that risk, since there are infinite risks and only a finite amount of effort we can expend.
If the consequence of a risk is death, then we should absolutely put a lot of effort into minimizing that risk. If the consequence of a risk is that a private conversation is sent to a contact, we should definitely put a lot of effort into minimizing that risk, but probably not quite as much as you would into something that has the consequence of death.
Even when the risk is death, however, we don't put infinite effort into avoiding it. We choose to cross the street, even when we know there is a risk of death when we do it. We drive cars that have chances of mechanical failures that could cause our death, but we don't bring the car to the shop every day to check for failures.
Things are not so black and white as to say "security is always the most important thing"
You do not need to lecture me about risk. I've had a career in international downhill ski racing, have won auto racing championships, and enjoyed lots of technical rock climbing, all of which require a high degree of risk assessment, both in extended preparation phases and at split-second time scales. I've also run risk analysis for UAV flight systems
I understand well the difference between smart-crazy and dumb-crazy, and where the pseudo-mathematical risk models like yours break down.
Your 'analysis' to "...measure risk by factoring the chance..." would have fit right in at the meetings where Ford decided to just go ahead with the design of the Pinto/Bobcat because the lawsuits would cost less than the fix -- they wound up killing dozens of people.
Your 'analysis' would have fit right in where the trading algorithms were being designed, which worked fantastically profitably, until they didn't and ended up crashing the global economy in 2007-8.
You cannot simply multiply the cost of the consequences by the expected probability and get an allocation of resources. That is what you do to see if the lottery jackpot is big enough for you to want to buy a $2 ticket this week.
You must instead 1) fully examine the system for potential critical failure points/modes and then 2) allocate WHATEVER resources are necessary to account for preventing those critical failures, then implement those remedies along with the features.
These preventative measures may involve installing redundant systems around the critical points, redesigning the points so they fail in a safe mode (e.g., fail to send the data vs sending it off, shut down vs, explode, etc.), adding check procedures around the potential critical failure, etc.
Note that NONE of these measures involve not implementing the feature. They involve 1) checking for critical failure modes, 2) allocating R&D to develop preventative & fail-safe measures, 3) implementing the measures, 4) testing, and 5) field monitoring.
This is what you do if you are serious about risk.
I was simply pointing out that we can never get to zero risk, and since we can't, we have to weigh risks based on consequences and probability.
> 1) fully examine the system for potential critical failure points/modes
Sure, to the best of your ability. How can you know for certain you have found all potential critical failure points? You can get pretty sure, but never fully sure. We still have industrial accidents, in every single industry in the world.
You also have to define what a 'critical risk' is. I don't think it is an a priori fact that accidentally sending a recording of a conversation to a contact is a 'critical risk'.
It absolutely does NOT work for serious risks, e.g., of death, serious injury, massive privacy violation, and other potential life-changing events.
The concept you are clearly avoiding or missing non-linear risk.
You (and amzn_engineer1) are advocating for simply subsuming risk assessment into the ordinary development cycle, and calling it "taking it seriously".
That is fooling yourself.
Taking it seriously is actually making full and serious effort OUT OF THE NORMAL DEVELOPMENT CYCLE for no other purpose than to SEEK and identify potential critical risks.
It is then engineering a variety of in-depth solutions to prevent those critical failure points from ever seeing the light of day. And implementing them. and testing them. And monitoring them.
>> I don't think it is an a priori fact that accidentally sending a recording of a conversation to a contact is a 'critical risk'.
This is an exact example of this sort of failure: 'it's not a priori bad'..., minimize it and streamline it into dev.
I really want to know in what world any sane person would say that it's OK to randomly divulging an intimate conversation to a contact or random recipient -- seriously, who would say that?
I mean sure, most conversations are benign, but some could be utterly life-changing if revealed. and that's OK with you?
But note the blameless analysis is NOT the same as saying "meh" errors happen.
It is a culture and deliberate practice of allocating resources to seeking and classifying risks, and designing, implementing, and testing engineering and procedural mitigation strategies (vs. development as usual).
https://xkcd.com/378/
https://lwn.net/Articles/219983/
Interesting paper: https://media.blackhat.com/bh-us-11/Dinaburg/BH_US_11_Dinabu...
The macrofailure, though, is that you have a device just guessing at what you want and doing it. That can't be fixed without going back to manually constructed (or at least manually triggered) notifications. Which for "send a note" might be an obvious fix, sure, but there is a lot of gray area in what constitutes "privacy" and Alexa would be pretty useless if you had to affirmatively OK everything it did.
Not sure how to fix that, if you use the aux out it has no way of knowing what happens on the other end; a confirmation prompt that isn't heard could still pick up an errant "Ok" response. Perhaps require a PIN like the shopping confirmation that would make it much much more unlikely? Or better yet, disable messaging as an option.
Once recordings of private conversations leave the local environment and make it to the cloud, eventually they will leak. It's akin to data collection by law enforcement: once the data exists, eventually it will be abused.
Not even the most responsible companies (e.g. Google) can hold out 100% of the time in the face of determined assault by government. Some of that data is going to leak to three-letter agencies, or similar.
Somewhat less responsible companies (e.g. Amazon) will leak data more often, to a wider range of threat agents.
So then we have to consider how valuable this data is. Random sampling of private conversations within the home? Sometimes innocuous -- but if the wrong moment gets leaked, the consequences are potentially life-shattering.
It's because such blithe dismissal of the damage caused by data gathering is so prevalent in the industry that the likelihood of devastating compromise is so high and the costs borne by the populace are spiraling upwards.
Some data should never be collected. Some data should never even be uploaded.
I am honestly confused as to how you interpreted my last comment as "utterly cavalier about the prospect of ruining people's lives", when all I said was that your assumption doesn't take into account the actual probability of data being leaked and it doesn't weigh any of the benefits of data collection against that risk.
I can only hope that karma visits those who arrogate to themselves the decision to sacrifice a few of their fellow human beings: may they and their loved ones become the sacrifices.
It will be pretty easy to justify by bringing up bomb threats, but it would mostly be people buying/using drugs.
1. https://www.nytimes.com/2018/05/22/technology/amazon-facial-...
I expect worse to come.
[1] Am not saying anyone deserves this, or that AZON is either evil or beneficent, or anything else other than precisely what was stated.
I’m so out of touch with this world it’s scary.
/s
This is just whataboutism.
We can complain about Amazon in a thread about Amazon without having to tackle a whole different issue.
The whole point of these cylinders is to be listening at all times.
Yes, it’s ultimately trust in the device maker, the compiler, the chip manufacturer, etc, and those can and have been verified to some extent.
But that’s not the point. A phone can listen or not, where you have no doubt with the cylinder. It’s its raison d'etre.
And people know this. It’s not like they are being misled. It’s a conscious exchange of privacy for convinence. It’s just that, in my mind, you are getting very little for what you’re giving.
I made a different choice, but I totally understand why someone would choose not to have a phone or smart speaker due to privacy concerns. What I don’t get is someone who has a phone but sees smart speakers as crazy.
- You are pretty certain that your phone is listening at all times, even though it's set not to.
- You see no difference between a product that's advertised manly as a listening device and one that also has such a feature, that can theoretically be turned off, and is very useful otherwise.
I see no difference, privacy-wise, between the two. I don’t care how they’re advertised, I care what they’re capable of.
I'm sure you can appreciate the difference: press a button to talk to Siri vs Siri is always listening.
The latter seems crazy to me, on a phone, cylinder, laptop or whatever device it's implemented.
If that was the case, nothing. But we have enough evidence to believe it's not. While having the device listen, even though it's been explicitly configured not to, is a bit more of a stretch.
>Are you just worried about accidental activation?
That could be an inconvenience too, but I'm more concerned with malicious intent.
If you’re concerned with malicious intent then I really don’t understand the difference. If the device maker is trustworthy then you’re fine either way. If they’re malicious then you’re screwed either way. A setting for “please don’t listen” isn’t going to make the slightest bit of difference if they’re malicious. M
Nor should anyone I think.
It seems to me that smartphones are significantly worse. You can unplug your smart speaker or kick it off your WiFi and be confident that it can no longer hear you. Smartphones have batteries and cellular data connections. Also they’re usually with you at all times, even when you’re away from home.
“Networked device with microphone records and uploads your voice” might create reactions such as: yeah, what did you expect, how else would it work?
“Networked device with microphone records and uploads your voice even when explicitly told not to” is a lot worse. Company X is lying to us!
My guess is they mostly do work that way but bugs and vulnerabilities don't seem like a stretch. (As is to be expected with all software)
If you actually care, this would be pretty trivial to verify using an oscilloscope to observe the energy draw from the battery while different things are said around your device.
Is this possible of a Pi?
https://snips.ai
Then all that's left to figure out is...
1. Make it work really well across a vast array of voices, accents, and background noise.
2. Make is easy to add new features to, in order to keep up with the competition.
3. Mass produce it at a price people will pay.
4. Have a viable business model.
Do these simple things and you'll be a very, very rich person.
Looks cool, but I haven’t tried. I’m the kind of person that gets up and walks to the light switch.
That said....we are on track to have a high quality experience when we go to production in Feb 2019 ( Release 19.02 ). As an open source community we'd welcome any help folks have to offer. Contributors don't even need to know how to code, all they need to be able to do is listen to sound samples at https://home.mycroft.ai and tag them as "Hey, Mycroft" or not "Hey, Mycroft"
Thanks to everyone who is helping make Mycroft a reality!
0. https://snips.ai/
1. https://mycroft.ai/
2. http://jasperproject.github.io/
3. http://www.theadrianproject.com/
> People are voluntarily paying for their houses to be tapped for the convenience of being able to shout: play me some song. > I’m so out of touch with this world it’s scary.
I could just as easily say "People are voluntarily paying for their locations to be tracked for the convenience of being able to get live maps on the go" (ie. cell phones, which also are listening to everything you say unless you disable it)
“Alexa... play white noise”
“ALEXA... stop”
I also live in a city, so white noise (actually brown noise) masks out tiny noises that interrupt sleep.
clap clap
Clap off
clap clap
Clap on, clap off. The Clapper!
clap clap
> This comment could literally apply to ANY new thing.
Yes, and?
It applies to using x-ray machine to fit shoes in shoe shops, which was super fancy schmancy at first, and I'm sure anyone who rejected it solely on the bases of being unnecessary was laughed at by first adopters, too. On the other hand, there's the discovery of washing hands being derided and ignored, when quicker adoption could have saved lives.
Exactly because it can go both ways, you have to look at the actual thing. You can't determine the quality of a thing or the usefulness of a habit by just looking at a calendar and a clock.
Mankind has made countless decisions to trade convenience for some degree of personal freedom. I can live without mobile internet - its just so convenient to have it. Voice control for home appliances is exactly the same.
I can also control my tv, thermostat, get a news briefing, check my calendar and set reminders.
Yes, it's ultimately a convenience, but so is indoor plumbing and store bought bread.
I’m sorry, clean water and food are not conviniences. They are among the most basics of human needs. That’s the kind of thing that scares me. That someone would make such a comparison with a straight face.
Have you ever considered that this is not the case in most of the world? Resources are far from evenly distributed.
It is not by chance that the availability of indoor plumbing correlates to increase in life expectancy. It's nice that I can just open the tap and clean water comes out, but it's also a lot more sanitary than people carrying and storing buckets around.
>I just cannot work out if the commenter was being deliberately obtuse or really misunderstood?
Same here. Comparing the savings of a few steps towards a light switch to clean water and cheap food is intellectual dishonesty at best and severe ignorance at worse.
Neither of which is needed to live. Both of which are conveniences.
Convenience does, absolutely, have strong value and can be meaningful improvement to one's life. Generalized voice control is a major and meaningful convenience - even if it's currently early and might have some problems - because it drastically changes how we can interact with our environments.
Sure, we can short sell the case as "not needing to walk to a light switch", but even then if it's dark, my arms are full, and the path to the switch is littered with children's toys then that's a meaningful improvement to be able to say "Hey google, turn on the living room light".
Consider cases of people with severe MS or who otherwise can't easily walk. Sure, they could hook up a remote control and keep it near them, or get one of those stupid clapper things, but even then that only works for the lights.
A generalized voice assistant gives me control over my house, from anywhere my voice can be heard, and not only executes my actions, but can give me realtime feedback and data. Best of all, this requires no special skills on my part, just a willingness to talk to the damned thing loudly and clearly with simple words.
Can it act as a wire tap? Of course it can. My store bought bread can also be contaminated (right now in my area there is a huge contamination issue with eggs and lettuce), my indoor plumbing can burst and flood my house, so on. It's a new technology, it has kinks and flaws, and we're currently in the early adopter curve of it moving towards the initial disappointment trend.
My disagreement with you is precisely because you consider them equally interchangeable.
To me, putting those two on the same level is actually offensive.
If data on the increased life expectancy isn’t persuasive enough, I suggest you speak with someone with no access to basic sanitation. Ask them what they think of this comparison.
I suspect we are only a few years away from being able to do a slightly inferior version using local processing, or a private cloud (already possible today when there is a market). I would probably opt for that, but use Alexa in rooms already already untrusted.
I have difficulty believing this. We had good speech recognition on <$1000 devices 10 years ago.
But Siri sucks for MANY reasons, and not just technical ones. It does worse processing but also does worse cloud integration, and now vs. alexa, has a far worse ecosystem around it (alexa "skills" are pretty awesome, and trivial to create)
We are a long way from being able to do that on $100 devices.
Similarly, there is a massive difference in the privacy implications of listening to every conversation everyone in your home (including guests) is having vs. having your current location known. Wiretapping laws exist for a reason.
To be fair, you can have all the maps without the location tracking; the location tracking just tells you where you are on the map.
This is not the world we should be engineering. Actually, I would say, this is not engineering.
There's no reason why your house needs to be "tapped" for either of those use cases. All those things could be accomplished without leaving your home network.
And about tapped .. well, I don't consider any conversation as private, when there is a Smartphone around.
Is there nothing you've ever said about anybody while in the vicinity of a smartphone that you wouldn't have said to their face?
There also other things, like pillow talk etc.
And I simply don't want anyone else to take part in that, whether they are russian, chinese or NSA hackers, or the company by itself.
And I usually tell people in their face, what I think of them.
Perfect example. Do you always ensure there are no smartphones within earshot?
But I would certainly try to do this in this situation as well.
It is not only privacy btw. it is also just nice to have both persons here in the now and not half distracted because the Smartphone just got new messages. Could be important... but usually never is important enough, to destroy the moment.
So when I go to sleep, alone or not, the mobile is off (without battery) or away. Unless there might be a emergency and I need to be avaiable, but those situations are rare.
- people in the sixties: "the government will wiretap your home"
- people now: "hey wiretap, can cats eat pancakes?"
[1] https://imgur.com/gallery/QFjXc
What if this was a lawyer talking about clients or a doctor talking about patients? Is Amazon (or Google, or Microsoft) willing to deal with such legal liabilities? Is this what move fast and break things looks like?
Edit: To those trying to downvote, remember, the Silicon Valley lives in a bubble, and for the safety of people inside and outside of the bubble, the bubble must be poked every so often. This is one of those times.
I'd say the Echo needs to have a more difficult activation routine available. Maybe have the option of setting the wake word to "Alexa, can you please" or something similarly long and unlikely to appear in normal conversation.
Also, I wonder if they had their wake word set to "Computer". I did that once, and quickly reverted after so many false activations.
So if it though it heard "Alexa, send this message to John: {conversation of ten minutes}" it just did what it thought it was supposed to do. But it's weird they didn't hear any audible confirmation or anything.
"Amazon sent me a message" is so frustratingly vague...
i would assume that the profile for even a free amazon account would require a phone number.
Now that Alexa is available in my country, I've considered switching it over to a real account explicitly to be able to use some of these calling and messaging features. Maybe I'll wait till the bugs are worked out.
Amazon takes customer privacy EXTREMELY seriously. There's no way a team would get the "ok" to build a skill that randomly records private conversations then sends them to a random contact. It also doesn't make any logical sense to build such a skill.
Yes, I might sound biased because I am an engineer at Amazon. This statement is my own and unrelated to Amazon's opinion.
If that's the case, then you need to provide additional features to reduce the chances of this happening. Longer and more unique wake word options, or more complex and deliberate confirmations before the call is placed. Something that a user can enable if they're worried about this sort of thing.
If there was an Alexa in my place of residence, I would rip it out of the wall, smash it to bits with a hammer, and fire it out of a cannon into the sun.
Hell no to this crap from my side. I hope there are more incidents like this until people wake up to the fact they are putting telescreens in their homes. You guys can't be trusted!
You may consider disposing of the bits at Kilauea instead of the sun. It would be almost the same and much more affordable.
What I mean by the above is that the "call" skill is much different than the "weather" skill. All Alexa has to do is have a confirmation prompt in the "call" skill and this wouldn't have happened. That is what extremely serious looks like. This is exactly the same as the phantom laughter incident from a few months ago. Alexa "heard" someone say 'Alexa laugh' and laughed, but that wasn't the user's intent. It was fixed by moving to 'Sure, I can laugh,' followed by laughter.
Voice UI is very hard and still in its infancy but ability for personal harm (physical or emotional) must be considered in these interfaces. Turning off the lights may not need confirmation but unlocking the doors or turning off the alarm probably should. Sending recordings or answering calls or even calling people should require more hoops or at least allow the user to control the risk/reward.
There is absolutely no reason to believe they are not conspiring with the NSA. They have a huge deal with the CIA, plausible some of these funds are for surveillance capabilities. They would not (and likely would be legally prohibited from) disclosing any relationship they have.
https://venturebeat.com/2014/03/18/snowden-slams-amazon-for-...
https://www.theatlantic.com/technology/archive/2014/07/the-d...
Or, more likely, being made an offer they can not refuse by the NSA. And the rank-in-file engineers may not even know it's happening, all it takes is inserting some diverting code into the pipeline and calling it "QA monitoring" or something. Couple of people in the whole org would know that somebody from some IP connects and downloads these "QA" data periodically, all the rest would be completely ignorant and indignant at the thought. Don't see anything preventing this from happening at Amazon - or anywhere else.
Apple shared personal photos of myself onto the internet without my permission. I could not delete it without getting support to assist, and they could not provide me with a reason why this happened.
Would you say that Apple does not take privacy seriously?
The issue from the article is systemic. Everyone could easily accidentally be recorded and share that with random people
This does not mean Amazon does not take privacy seriously. It is a company of small teams with very few layers of management between an engineer and business decision. The error in judgement of one team does not reflect on all of Amazon.
As you say, Voice UI is still in its infancy and not lacking growing pains. However, because Amazon does take privacy very seriously, after this incident, I'm certain there will be actions taken internally to ensure teams properly weigh the gravity of a skill with its voice trigger (or adjustments made if there is already an existing policy).
My own thoughts, not Amazon's.
The default must always be voice recording will be auto deleted after 1 minute if no response is heard. It should let user know about that too.
To send this message to John Smith, say 'artichoke'.
I don't think the article is indicating what you say. The report says it's a bug, but it does bother me why Amazon won't comment on the specifics. This should be patched immediately with a full retrospective explaining what happened.
And then builds exactly that...
It's a mistake, but it's a pretty bad one.
Bugs happen in architecture, aircraft, etc. too. the difference is that the actual engineers are paid to have a precautionary approach -- and spend significant resources -- to actively prevent bugs from making it into the final product.
Amazon and your team has built a great product (I have one and make moderate use of it, have even considered building some skills).
But, you have planted a full-on bugging device in millions of people's homes. Done by a government, this would be cause for war or revolution. This is serious, and you need to treat it much more seriously than you obviously are. Every 'skill' does not require the same minimal levels of security and verification, some, like this one, require much more, or should be forbidden outright until such security can be properly implemented (and yes, this should probably include calls only to pre-configured whitelists, intent confirmation, etc. and to any manager that says "that's too inconvenient to the user", the response is "screw you, it's critical").
You call yourself an "engineer" at least twice, and claim that you take privacy "extremely seriously". The evidence from this incident and others noted in this thread indicates otherwise. Clearly, insufficient resources were allocated to figuring out the potential failure modes of a "call skill", and preventing them.
All due respect, but your team needs more of an engineering approach than you have. This entire "it's gotta ship yesterday" mentality in the software industry used to be just inconvenient. Now it's getting dangerous. Please help stop it.
I'm all for doing away with the "it's gotta ship yesterday" mentality, what your thoughts are on how developers can help stop it?
This is not easy, especially as the CEO is still typically above the CTO, and can overrule. I had it happen to me, when we were ahead on a scalable version of the product, but they didn't like the timeline. The mgt decision cost almost a year of messed-up myopic development schedule just to roll out apparent features sooner, but ignored the likelihood and eventuality of bugs. Afterwards, when we got back to the scalable highly modular version, we started taking biz from competitors who couldn't scale. I'd say that my mistake was to only give the broader consequences, and not spend time to be able to enumerate in detail the consequences of non-scalable quick program would be. Of course you cannot predict exactly what bugs will happen, but I could probably have done a better job of drawing the scenarios (not sure it would have made a difference, but it might have).
I'd also say that we need to create specific structures and plans to study and quantify risks, as is done in real engineering like aerospace, architecture, etc. Classify those risks into a range of categories, from small bugs to existential for your customers or project.
Different steps need to be taken for each class, and significant part of the planning needs to go into de-risking the project.
I'm in physical vs software development now of carbon fiber type technologies, and I notice that my military customer who are building very cutting-edge stuff often talk of 'de-risking' the project, whereas I don't hear this much from other customers. Seems like an important distinction to take on board.
--- From a user perspective, I noticed after looking at the issue on our own Echo yesterday: the UI is a totally greased slide to hide choice for the user and slide them right into giving permissions for contact list. It seems that effort was made to hide the actual features and functions that will result from giving permission, and obscure the 'Skip' option. So it would be easy to not even notice that your device had these new possibilities. Obviously, I'd recommend taking more time to sell the features and let us make an informed choice. Then even if things go wrong, you'll enjoy some benefit of the doubt in the market and press.
(edit: add parenthetical)
Yes, I see this a lot too. I think software managers have more incentive to get new features deployed. I don't think this is a good way to measure their performance as a manager because of the consequences we've already mentioned.
I would also like to see more concepts from physical development implemented in software. At times feels like the wild west out here and too often we ignore the lessons from similar experiences.
Thank you for that input btw.
I know amazon's got a different motivation matrix than a startup; e.g., Amazon won't die if some feature isn't delivered by the next trade show, but they do have competition from the other majors.
That said, amazon certainly also has the funds available to invest in a parallel risk team. If they're not motivated to do it from the risk to their users, the best argument might be the potential reputation setbacks if stuff like this gets out there & causes problems, bad press, reputation for creepiness, etc.
I know it always seems inevitable that you'll weather the reputation hits from errors and just press on to greater usage/adoption/sales, but it will always seem that way from the inside -- until it doesn't. Google's experience with Glass comes to mind; could have been a fantastic product, but it went just under the tipping point of being creepy, and poof, they're gone. It'll be a generation before anything similar comes back.
I'd hate to see that happen to Echo/Alexa. TBH, it looks like this product has both greater potential, and also greater creepiness potential than Glass ever did.
How it happened is only relevant to the engineers who build and maintain the thing. I, on the other hand, could not care less how it happened, and the fact that it did happen is reason enough never to buy one of those infernal devices.
I'm sorry, this is just stupid. If you cannot see the distinction between a feature that was intended to spontaneously record audio, and a bug caused by faulty voice-processing, then that's on you. The distinction is pretty critical
Obviously, intentionally designing a feature that spontaneously records and transmits audio is a problem for many reasons. But the lack of intent does not magically erase the consequences for the people who experience this kind of bug.
And, to be clear, this was not simply a matter of "faulty voice-processing". The fact that this could happen without the user's knowledge is a problem in itself. Clearly, there are inadequate visual and audio cues, and insufficient or nonexistent verification. Those failures are not bugs; they are bad design and engineering.
If you took privacy seriously you would have added an audible prompt to ask for confirmation before sending or similar safeguard.
I mean if you are an engineer then you know that there is a nonzero false positive rate for the device to detect the "call" command.
Knowing this, and not implementing a trivial safeguard sure seems to me like the team was "ok" with building exactly that, a device that: "randomly records private conversations then sends them to a random contact."
> This statement is my own and unrelated to Amazon's opinion.
Just because you admit your bias doesn't dismiss you from it. Your statement and opinion could very well be influenced by your work surroundings at Amazon. While I do not doubt that Amazon does take security and privacy seriously, I find it hard to believe that they employ proper ethics (based on this and past examples such as the NYT incident).
Heh, skill.
Why? Because from a PR perspective, it doesn't matter what Amazon or the engineers intended, all that matters is what actually happened. You said:
> There's no way a team would get the "ok" to build a skill that randomly records private conversations then sends them to a random contact.
But that's exactly what happened. Whether it was a bug or a feature doesn't matter. Imagine if the wing fell off a plane and an airframe engineer came on here and said "hey, there's no way we would get the OK to design a wing that falls off in midair." Yeah, we know. The fact that it happened by accident doesn't make it better (maybe worse, actually).
Now you've got dozens of people responding to you, and BTW a lot of reporters read HN too. Do you really want to see stories like "Amazon Engineer Calls Customers Conspiracy Theorists"?
Of course. But it is entirely plausible that somebody builds a skill that records certain conversations, and somebody builds a skill that sends recording somewhere, and then due to some bugs or coincidences or missing controls to prevent such occurrence, the first skill is activated when it should not be, and the second one is activated when it should not be and with wrong parameters.
So there should be more checks. Like - ask for explicit permission before sending any voice recording out, or ensure that the target where recording is set is validated against whitelist explicitly set by the user, etc.
Oh is that all.
Look- we all know that bugs will happen. The question is, what will you do in light of that? What bounds will you set up to prevent the system from acting in ways you want to disallow? How will you detect something is awry? I would say that really caring about people's privacy means doing this part seriously, not just a baseline lack of active malice.
I have not experienced that, can you elaborate? They keep spamming me with mails of all there different services, show me what I should buy, etc. No way off even turning these things off...
What I should have said is this report makes it difficult to understand what actually happened. It seems clear this article favors click-bait quotes that insinuate a "big brother" vibe such as:
> "'unplug your Alexa devices right now,' she said. 'You're being hacked.'"
If she had instead said "Alexa butt-dialed me!", would you still be interested in this report?
Edit: Not to mention that the entire Alexa is a black box, too.
Second Edit: Not trying to claim that you're an idiot, just provide some perspective.
I have an Echo on my kitchen counter. I know there's a small chance that it may be hacked or have a bug or be deliberately modified to surreptitiously record from my kitchen audio that ends up on some Amazon server somewhere, or in the hands of law enforcement, or the NSA, or hackers, or whatever.
I just don't really care. I like the convenience of the device, there's a lot of scrutiny on the devices and the companies that make them, discouraging deliberate misuse, and I don't really say anything interesting or worth hiding. And if someone is going to deliberately target me, my life is jammed with devices that are probably a much bigger risk surface area.
I would never say anyone else should get one if they feel the privacy risks outweigh the benefits, but I'm not personally concerned.
However, I may be wrong. So change my view: why should I care or be worried?
I enjoy living dangerously.
EDIT: Just realized the substantial difference is that Google and Amazon own all of these things. They don't control all makes and models of phones.
1.) They have an economic incentive to do so
2.) Only they know how their systems work
3.) Their network traffic is encrypted
4.) They face legal risks and user backlash if caught doing so
Given 2+3, you can't be entirely sure that they aren't doing it. If they deny it, your only recourse is to hope that their cost/benefit calculus considers 4.) to be more costly in terms of dollars.
It will recognize the Hey Siri, but give an error for everything else. Pretty sure "Okay Google" will behave the same way. Also if you turn on battery save on iOS it disables "Hey Siri".
It's a custom low-power chip they added on the 6S and later to allow a local tight loop that only listens for that utterance using a hardware-assisted neural net, and only activates the rest of the software stack if it detected it with reasonable confidence.
There's a nice blog post about it here from the apple engineers (see the "Two-pass detection" section): https://machinelearning.apple.com/2017/10/01/hey-siri.html
Your vulnerability isn't a targeted attack: you are not valuable enough to be worth the effort to figure out your system. As an attacker on your system I'd have to figure out how to break in, and then how to use the hardware you have. You are more valuable as part of a botnet - attacks that already exist. (if you are a politician then maybe, but that person is also vulnerable to a targeted attack on their amazon system - probably more so because the target is easier to figure out).
That is they will target everybody because they know in a few years that will include somebody who they currently think is a nobody.
Of course as AI gets better and cheaper they may eventually listen to everything to see what who can be targeted automatically for what.
Just working for a company that an agency is surveilling is enough to make you worth targeting.
As a random example: let's say that I want to kill someone who lives in your neighborhood. I could analyze your conversations, comings and goings to figure out when and how to kill that person, and blame it on you. I could on a continual basis run numbers on everyone emitting data in your neighborhood, until someone had arrived at a point where their friends would testify against them on the basis of conversations with the potential patsy, that patsy had no alibi, and tailor the murder on the basis of the means that the patsy had available to them at the time.
I could also just be trying to figure out if you were a homosexual, or muslim.
> They should feel more comfortable than an equivalent system built by a company that is looking to profit off of your data - and additionally, you can give the guest stronger guarantees that when you say that the system is "off", it actually is.
They won't, because they'll be concerned the person that implemented it is a creeper. When regular people think "cameras installed in the room," they think this: https://www.bustle.com/p/airbnb-host-was-caught-with-a-hidde...
Distrust of a person is more more immediate in visceral. Distrust of a company is more distant and esoteric. The former usually trumps the latter.
There are many methods for occupancy tracking without cameras. IR sensors, antennas in the walls, NFC (bleh), proximity sensors.
Less likely to be seen as creepy and such systems already exist.
If you just need 'is a person in this room' or 'how many people are in this room' levels of data, solutions to this problem have existed for decades. No need to over complicate a solved problem.
I'm pretty sure all current technology only works with the have a wifi decide on your person. It is significantly easier to do this and for the most part good enough.
Not only can you detect motion and objects through physical mass using WiFi spectrum with a properly equipped device, you can detect if someone is breathing.
“What's more, this "Time Reversal Machine" technology is essentially just some clever algorithmic work with little burden on the processor, so it can potentially be added to any existing WiFi mesh routers via a firmware update. In other words, security system vendors should take note.”
The following processing is outside the scope of the GDPR:
- any activity outside the scope of EU law (e.g., activities of a Member State in relation to national criminal law);
- any activity performed by Member States when carrying out activities in relation to the common foreign and security policy of the EU;
- any activity performed by a natural person in the course of a purely personal or household activity;
- any processing by the EU itself;
and
- any activities performed by national authorities for the purposes of prevention, investigation, detection or prosecution of criminal offences, or performance of judicial functions.
https://gdpr-info.eu/art-2-gdpr/
I’d recommend something like this in every room, or even IR sensors, over cameras. You don’t want to capture video of your children jerking off.
You could just use an infrared motion sensor. They are dirt cheap and widely used for controlling lights.
The AC unit at work has one, too.
So, all of this already exists. Without spying capabilities, that is.
A lot of home automation stuff seems like something that would be fun to make, but not something I really want to have. I mean, the AC might be a good energy saver (a smarter thermostat) but that's not something I want, more like something that I would buy if it was cheap and practical enough.
To some extent, this is a phenomenon of early technology. It feels full of potential, so you want to play with it but it doesn't really do anything you really need yet. The early web was ki d of like that. We made sites because we wanted to make sites, moreso than because we wanted to have them.
Home automation though... Im kind of skeptical that this goes anywhere useful. The useful examples people think of (eg remote close all the windows and lock the doors) are more about mechanisation than automation.
Alas, remote controlling those blinds would be a major hassle since AFAIK I would have to install 6 wifi-enabled devices and tear holes everywhere (not even sure there is a powerline near) - and likely do all the programming myself. Thanks but no thanks.
I've been working on a custom UI that sits on top of the Wink Hub API to unify everything, but I'be been stuck with their almost completely undocumented Pubnub event API.
Open source, gets you most of the way there.
I think "find" and similar tools are now much more advanced so it might work better out of the box now.
And ideally? All of it integrated. It actually sounds nice to say "I'm going home", and have Maps say "today that will take 35, should your oven start preheating when you're 20 minutes out?" IoT devices are overrated, but I can absolutely imagine a critical mass of integrated tools being very useful.
But I'm not even slightly willing to do that. It's too much information and too much risk surface. I'd pay a hefty premium to get local-data-only versions of these products, but no one is offering that, and it doesn't look like they're going to start.
Not yet preheating the oven though :)
Waze also knows where I'm going in advance and most of the time it gets it right, like every Thu evening "are you heading to [evening school] ?". My weekly schedule is exactly the same 99% of the time but sometimes it still makes obvious mistakes. If you need an AI for that it must be a very simple one, yet it still fails.
Google maps also auto saves my parking location most of the time but other times it doesn't, for reasons unknown. Things like that make it hard for me to have faith in future versions of this stuff.
That sounds awesome until the company providing that service starts abusing their knowledge of their location. That abuse doesn't even have to necessarily be malicious in nature either. For example, Google Maps on Android started asking me to rate, review, and/or take photos of my present location if they deemed it a point of interest (certain restaurants, parks, etc). I never opted in to this feature and the only way to disable it that I've figured out is to literally disable all location services on the phone.
I really dislike the idea of Google storing a timestamped record of almost every place I've ever visited. Tt has to beconstantly phoning home in order to deliver the request to document my visit within a minute or two of my arrival and that constant reminder that Big Brother Google is tracking me at every moment is just disturbing on so many levels. Even if they aren't using that data right now, remember the "data is never destroyed" principle of the internet.
On a side note, I would have ditched Android if I didn't need it for work simply because use of the GPS radio is hidden behind the acceptance of enabling Google's Location system and all the invasions of privacy that entails.
An encrypted hard disk with the key on a USB stick would be enough, just keep the key somewhere separate and you'll only have to plug it in when there's a power outage.
The main fight needs to happen at application level, not infrastructure. Cloud services are already mostly transparent and interchangeable. But applications aren't. The problem is, it's the application vendor that owns the code, determines where's going to run, and asks you to send over the data. How it should be working, is that you own the data and determine location of computing, and own or rent code to be run on that data.
You don't need cameras for that, just motion sensors.
> That server makes decisions on if lights should be on...in that room.
You don't need a server for that, just a motion-sensing switch. They can be totally offline. My office has them to shut off lights automatically when a conference room becomes unoccupied.
https://www.homedepot.com/b/Electrical-Wiring-Devices-Light-...
It can get annoying if you're not moving much, but the key point is: no camera or server required.
> That server makes decisions on...if A/C should be running in that room.
Ecobee thermostats can already do that using motion/temperature sensors. They run fine without an internet connection.
https://www.ecobee.com/room-sensors/
Thank goodness for phone flashlights!
A new dystopian developer-productivity metric.
Edit: typo
By the way, German workplace safety regulations require a certain minimum of light at a workplace.
[0] https://en.wikipedia.org/wiki/Passive_infrared_sensor [1] https://en.wikipedia.org/wiki/Forward_looking_infrared
[0] https://www.adafruit.com/product/3538
Maybe, but that seems like a lot of work for little gain.
You probably don't actually have that much control over the AC in your rooms unless you have zoned heating and a lot of zones, regardless of how smart your sensors are.
Intent is a very important factor when answering the question "should the lights be on," I don't think you'll be able to predict that. For instance: if movement is detected in the bedroom at 2AM, should the lights come on? The answer is: a very strong maybe.
You would be collecting data from many sources in order to predict intent. This is why you need a centralized server.
Please explain, exactly, what other sources you would collect data from and how the central server would process it to determine if I want the lights on in the middle of the night.
IMHO, determining intent in this scenario is impossible without...
1. a mind-reading sensor, or...
2. an explicit user signal, such as a button-press or command.
The only realistic option is a user signal, and most of those options obviate a lot of these prediction ideas.
I think there's a lot less practical value to having a "central server" controlling everything than you seem to assume.
The more data, the more smarter. In fact, the only difference between a thermostat and the human mind is the number of datas. This is because the Law of Averages predicts that half of all datas will be relevant.
There are several important caveats to this statement.
> In fact, the only difference between a thermostat and the human mind is the number of datas.
This is an oversimplification that borders on dogma.
> This is because the Law of Averages predicts that half of all datas will be relevant.
An interesting interpretation and application.
Machine learning navel-gazing is the new "throw a start-up at it", so I'm guessing the answer to this is going to be "if we have enough data..."
That's how I feel about Alexa.
As someone who doesn't have a voice for radio, I much prefer being able to interact through dexterity. Besides, I'm very picky when it comes to which specific track of music is to be played (out of various similarly named pieces of music).
* I don't think take-up is quite as big as the tech world currently makes it out to be. People in my circles don't really have that kind of disposable income and are prioritising other purchases first.
So yeah, Alexa is a lot of extra literal work for little gain.
My main point was the original poster's idea was probably focusing on the wrong kind of sensor for what he wanted to do. He could still network a bunch of motion sensors.
Also, I'd dispute the idea that my proposals were less "modifiable." If only because they're far easier to implement and a couple of orders of magnitude cheaper, so it's practical to replace if more capabilities are needed.
I don't think the number of people in a room will give you meaningful information to tell you "how much to crank up the HVAC." Also, the HVAC systems in most homes aren't capable of even cranking up the HVAC in a particular room.
This is what the "Internet of Things" should be doing, but seldom does.
> The second advisor, a software developer, immediately recognized the danger of such short-sighted thinking. [...] "A toaster that only makes toast will soon be obsolete. If we don't look to the future, we will have to completely redesign the toaster in just a few years."
hackaday.com/blog/ will have some details on how people set up computer home controllers as well as working with OpenCV
A cookie is a device "that is inserted under the clients head by the brain and kept there for a week, giving it time to accurately replicate the individuals consciousness. It is then removed and installed in a larger, egg shaped device which can be connected to a computer or tablet (to automate their smart house controls as if the house was knew their personal preferences moving from room to room.)"
Low tech alternative solution from my friends? Get LEDs. Never turn off the lights. This way the room you are going to will always be lit.
Why would you even want such a system?
My flat uses the Button-Framework which provides a really convenient UI/UX. The edge between 2 nodes (rooms) has a button (a haptic device to switch between boolean states) at around hand-height that you can press to switch the light on or off. It works quite well (it uses a switch-system technically) and the user decides by her/himself if the light should be on or off and just presses the button after careful reasoning.
Granted, it doesn't use Docker, but it really works well and needs low-maintainance. My model is running since 23 years and I never had any issues - it's even open source.
That's the "Facial recognition fooled by funny-colored glasses" study from Carngie Mellon, where researchers were able to make machine learning algorithms fail disastrously (e.g. mistake a man for Milla Jovovich) with a pair of glasses printed with what looks like a random assortment of colorful pixels, but is in fact a targeted attack specifically designed to trick the algorithm.
This Alexa failure is obviously not a targeted attack; but when your system is exposed to enough data, eventually you will stumble over some input that happens to resemble a targeted attack by pure chance, right? It's equivalent to saying that if you aimed the facial recognition algos in that paper at millions of faces wearing randomly-colored glasses, eventually some of the glasses would be close enough to the targeted-attack glasses to produce the same effect.
Obviously I'm theorizing on almost no data here - I don't know anything about Alexa's voice recognition, maybe it contains no ML at all. But it seems plausible that this might be what happened here - not a bug per se, but the natural and totally expected result of giving an opaque, machine-generated system with a very low failure rate so much input data that the failure rate is significant.
Still, I wonder how it heard "record this conversation and send it to someone on my contact list".