Dozens of times in the past few years I've been served relevant ads on some topic instantly after discussing it in person in proximity to a smartphone. I've encountered countless other anecdotal stories from friends. When this happens I feel an acute sense of violation and vulnerability. It's hard to imagine a less sinister explanation for this phenomena.
This seems like something that could be settled via some kind of study, like playing audio to the phone and recording what ads get shown and doing it to a population of phones and social media accounts.
You can't imagine a less sinister explanation other than Facebook or some other entity is secretly recording your voice and the voices of millions of people, in violation of numerous state and federal laws, through their smartphone microphones, uploading it to their servers surreptitiously, performing voice analysis on it to determine who is speaking out of 350,000,000 possible cases (presumably they've already cataloged everyone in the country somehow?), and then use that to serve you more targeted advertising?!
I mean, they did break iOS platform rules and pulled some tricks to keep the app open in the background all of the time by sending an empty audio beffer to the phone for "playback."
True, it seems far-fetched at first. We should keep in mind that a even few short years ago theorizing about the US Government operating a widespread domestic electronic surveillance dragnet was still the realm of delusional conspiracy theorists.
I don't see how that matters. Besides, people have been discussing widespread, domestic electronic surveillance for decades far outside conspiracy fringes. The Clipper chip, as just one example, was a big deal at the time and received widespread, mainstream coverage.
Not sure if it is so far fetched to do some lightweight voice analysis (client side) and store significant terms in the same way that you would store search history.
I can add to that anecdote... I was discussing which podcast app was best for Android with a friend, and in the presence of her Android phone. The next day, she started getting served ads for podcast apps all over the place.
Keep in mind, this was someone who would have been completely new to podcasting and have never searched for podcasts or podcast apps before.
I was using google navigation and someone in the back asked "what do you mean one-size fits all"... then my phone which was only running navigator chimed in "one-size fits all means blah blah blah" was definitely creepy.
Google Maps navigation feature listens for verbal input (although it's supposed to be triggered by the wake word) to do things like add stops and such without touching it
> It's hard to imagine a less sinister explanation for this phenomena.
You just forget all the other signal you've giving. Funnily enough, last week somebody was trying to convince me that Facebook was recording their every word.
"I was telling a friend about a recent trip to Thailand", they said, "and I immediately saw an ad for Thai Tea". I asked if he'd used Facebook when he was in Thailand. He had.
Which is more likely: Facebook is recording every conversation of every user on their app and mini them for ads or the advertiser in question just targeted people who had recently been to Thailand.
interesting. thanks... I wonder if they have something like this for ios devices? seems like they have similar capabilities via the "hey siri" or whatever it is but I am curious if there is a hack to allow it to use that all the time from userland.
IMHO this is irrelevant to what is being discussed here.
> enables devices powered by Snapdragon 800 processors to be “woken up” by a custom voice command (beginning with a custom phrase set by the device OEM like “Hey Snapdragon”, for example)
Because they are always on and have to use minimal power, these chips are used to recognise a very specific term (probably why these companies don't let you customise how you summon the assistant), and cannot do anything like voice transcription, network calls, ... (which would just kill your battery).
Why would audio have to be transcribed locally on battery power? You could do that server-side and/or keep the data stored locally as audio until the device is charging and on WiFi.
Record audio data continuously on disk?
I'm not sure if this is even worth debating, it is a fact that this is not happening (these chips are on most new Android phones, and they do not save audio).
Wait are you trying to say there’s no way to record audio on a smartphone app? And also, why are you calling it “disk”? Smartphones use flash memory for storage.
What minimal power? Phones are being charged at home all the time. Android apps can prevent sleep. Dsp to parse voice is there. There are sdks for all this.
I fully agree with you on that. However, please grant that mobile devices spend a substantial amount of the day plugged in and can draw full power (people mining coins, etc).
If this is happening then it must require some amount of CPU and memory use on the device, which even if hidden from the OS can be detected via power use and heat dissipation. It should be possible to indirectly measure a phone's CPU load (including any hidden processors) by filming it in the infrared and/or by attaching it to power and measuring the wattage consumed from the supply.
If this is happening it must also require some network I/O. Something must be phoning home and reporting the data if the data is to be useful.
Note that there is an intrinsic trade-off between the heat dissipation angle and the bandwidth angle. If the device is doing parsing, speech recognition, AI, etc. locally then it must be consuming more energy and dissipating more heat. If the device does these things remotely it must upload more data. Compressed audio is far more bandwidth intensive than digested meta-data.
This seems like something security researchers can and should look into.
Wouldn't it be easier to reverse engineer their app? It seems easy enough, at least one the Android side. You could use something like baksmali to convert all the dex code to smali, then grep that for calls to AudioRecord/MediaRecorder and figure out what triggers that code.
That's why I'm skeptical that Facebook would do this -- it seems like a terrible business decision given the high chance of getting caught, either by reverse engineering or a whistleblower.
Once happened while remarking on the Fruit Loops shake at Burger King. Opened Facebook minutes later and got a Fruit Loops ad. I assure you I don’t message, post, or in any other way remotely suggest an affinity for Fruit Loops. Commenting on it at Burger King was the definition of a one-off incident.
I guess another possible explanation is some very sophisticated combination of location tracking and Kellogg’s targeting people they know will be primed by that promotional item. And then of course it could always just be an amazing coincidence.
All I know is I revoked microphone permission after that.
EDIT: Something just occurred to me. Facebook doesn’t have to be the one using your microphone to target ads. The ad buyers just have to have a way to match your phone to your ID (which they can easily; there was a site posted here a while back that sells that) and then listen in through ANY one of the apps you have installed. If this were the scheme, Facebook could reasonably get away with saying “WE don’t use your microphone to target ads.”
Out of curiosity, why did you remark on the Fruit Loops shake? Could it be due to an ad campaign that started prior to your remark, and continues after it?
We were going through the drive-thru and saw a sign. It seemed so magnificently revolting, I wondered at the time if it was a red herring to drive word of mouth.
I was once sitting at a bar having a beer, chatting with the bartender (one of my best friends) and the guy sitting next to me, whom I had never met before. I pulled out my phone to show my friend something that had been posted on Facebook and, while browsing Facebook right after that, the app showed the guy sitting right next to me as the first person under "People You May Know". I had not "checked in" or posted any updates while I was there, FWIW. It was at that point that I removed all permissions (i.e. location tracking) that I could from the Facebook app.
More on-topic: I recall reading that the Facebook app would activate the microphone when one was posting a status update, apparently listening for things like a movie playing in the background and such.
After Obama’s comments on the NSA surveillance program back when sh*t was hitting the fan, I’m always keenly parsing these kinds of statements for word play. There’s usually a way out of accountability and liability through evasive rhetoric.
I agree. Really, I don't see why Facebook wouldn't monitor audio like this for targeting ads. If you make money from serving ads, that seems like exactly the kind of thing you would want to do.
Like you, I have gotten pretty good at noticing "weasel words" and such in the government's statements (denials). It is clear that they take their time to make sure they word their statements ever so carefully.
Anecdotally, Facebook battery performance is pretty poor, it's always the app that drains my battery the fastest. I don't think they care about power consumption.
There's a very significant difference between "Facebook's battery impact is pretty shitty because the app is clunky" and "this would mean the average smartphone standby time would be two hours because it would require a phenomenal amount of power."
> Remember the time they kept the app open in the background on iOS devices by sending an empty audio buffer to the phone for playback?
> People started discovering that after iOS introduced the battery/power usage section in the Settings app.
They couldn't do it 24/7, of course, but they could certainly take "samples" at regular intervals or in certain locations or something. From my own experience, it doesn't seem that Facebook has ever been worried about battery life or preserving it.
>Really, I don't see why Facebook wouldn't monitor audio like this for targeting ads.
Because they'd get hit by huge lawsuits and fines from governments all over the world. Europe is especially hungry to fine US companies for privacy issues.
Also it would be terrible for their reputation. Hordes of people would uninstall the app, and even delete their accounts. I would for sure.
It's becoming harder to avoid their apps now that people I need to stay in touch with are on the inevitable Whatsapp that needs voice+other permissions, plus whatever other apps they bring into our lives next. I think it's kinda hopeless.
What about the possibility of geofences detecting your proximity to a Burger King and perpetuating through ad networks to trigger the current ad campaign which happens to be the Fruit Loops shake?
Why would any rational human being search ‘Fruit Loops shake’ on Facebook? Let’s also assume for the sake of argument that I only associate with rational people.
Yep. My immediate read of the denial was, "the legal entity that is Facebook does not itself spy on users in this exact manner".
Once they have a business model that does not revolve around selling data about their users to their customers, claiming they don't spy can be something other than a bad punchline.
> then listen in through ANY one of the apps you have installed
If there was some third party ad library that was installed across multiple apps that streamed audio and/or performed local speech recognition on everything you said don't you think somebody would have a) noticed by now, b) a security researcher figured it out, or c) everyone's battery's would be running red hot all of the time.
Teams at Apple and Google reverse engineer and debug the top 100 apps on their platforms all the time (there are teams dedicated to doing it). Facebook is also not the NSA - people leak stuff from them all the time. You're suggesting that either everyone is complicit in this, or there's a conspiracy so large and technically complex that it's managed to be kept secret all this time with no technical evidence outside of anecdotes.
Presuming Facebook has some sort of totally encompassing omniscience is a little foolhardy in light of e.g. the Russia issues, isn’t it? I tacitly admitted they may at worst suspect but not know that third parties are targeting in this way.
I’m not sure I trust that security researchers can be relied upon to catch everything. There are so many flagrant abuses of user privacy that go unchecked or remarked upon and nevertheless ignored that such an avenue of prevention seems highly unreliable.
Also you’d be surprised at what people are willing to do when you define a certain culture and incentive structure. While you may dismiss it, I find it frankly astounding that the NSA program managed to metastasize into the behemoth it became before it was finally outed to the public. And even a cursory reading of behavioral psychology will help to explain how that can happen.
I also know that Facebook perversely prioritizes metrics above pretty much all else. And it means they’re perfectly willing to indulge in dark patterns if the numbers line up.
Facebook may not be recording audio, but they do pay attention to your cell phone, GPS, and wifi, so they know that you're at a particular fast food place. Then the brand-named shake is probably something they're trying to advertise, maybe even time-limited, so the fact that you're at that place is enough. Past that it's coincidence, the law of large numbers, and confirmation bias - you remember that instance and no others, and while a few people have their own anecdotes to share, HN's readership is large enough that FB could produce that population of anecdotes with a hit rate of "One time out of ten, advertise $restaurant and mention a single totally random item selected off their menu".
The "People you may know" anecdote below is, similarly, some combination of location, wifi sniffing, luck, and confirmation bias.
In my case I was talking about flying with some coworkers and mentioned the entertaining safety videos that New Zealand Air had done. Probably hadn't thought or talked about it for 3 years. Opened up Facebook and had an ad announcing their new video. The microphone permission was turned off on Facebook, Messenger, and Instagram on my phone too, so either Android isn't blocking permissions properly, or they are getting it from somewhere else which is even worse.
You think they need microphone access for that? Froot Loops is in the middle of a massive co-branding campaign with Burger King, and you are surprised that you'd get an ad for Froot Loops while at a Burger King? Occam's razor says no.
As long as it is possible (closed source software and hardware which is unaudited by trusted organizations), then it's going to continue to be assumed to be happening.
I used to dismiss these rumors as paranoia/conspiracy theories in a maybe naive application of Occam's razor. But concrete leaks and tech news of recent years made me a believer. I sure believe Facebook is capable of this, nowadays. The only thing that makes me skeptical is that there should be reproduceable cases someone has documented for concrete evidence. All I ever hear is anecdotes.
The fact that there is no hard evidence makes me very skeptical.
Every anecdote is explainable another way. Someone in the comments said they were having a conversation about storage units and saw an ad for storage units immediately after. Even if they had never searched for storage units -- and can we really be sure? That would be easy to forget -- Facebook might just have extrapolated they they would need a storage unit from related searches, like searching for a moving company.
I would be willing to bet this type of algorithmic extrapolation explains every one of the microphone-ad-conspiracy stories.
> The only thing that makes me skeptical is that there should be reproduceable cases someone has documented for concrete evidence.
I'll throw another layer on the conspiracy pile and say they deactivate that behavior for anyone they notice trying to prove it, similar to Uber's Greyball.
I don't play golf or have any interest in it, but I was talking to my brother about him using golf carts on his farm as vehicles. I kept getting golf cart adverts in Facebook for the next month. Maybe it used location data to suggest we might be interested in the same thing, but it was a bit weird.
I know that smart TVs and the now defunct Kinect have the ability to monitor conversations. Hell I remember when I first used Soundhound! I was flabbergasted, what will they think of next.
But I also know corporations are usually scrupulous about following laws so unless there is something in Facebook's ToS/EULA that mentions doing this I'm skeptical.
> so unless there is something in Facebook's ToS/EULA that mentions doing this I'm skeptical
I don't see why the vague terms in their data policy [1] couldn't include audio (all-caps are my own emphases):
> We collect information FROM or about the computers, PHONES, or other devices where you install or access our Services, depending on the permissions you’ve granted. [...] Here are some EXAMPLES of the information we collect:
> - Device locations, including specific geographic locations, such as through GPS, Bluetooth, or WiFi signals.
If device location is merely an example of information they gather from my phone then why couldn't audio just as easily be another example?
One thing I've wondered was that Facebook began showing the recent photos you'd taken at the top of the app, and prompted you to share them. Are they sending every photo you take to their servers to feed it back to you at the top of your stream? I realize they could implement this feature by displaying local photos without sending them to their servers, but I've never seen confirmation that photos remain on your device only until you post. I uninstalled the app awhile back, so maybe they don't do this anymore, but anybody have any insights into how they handle your photos?
There's an easy way to verify this: run tcpdump and record it happening. The fact that nobody has done this, instead only offering anecdotes, strongly suggests that these are coincidences or from other forms of targeting.
Regardless, it’s easy enough to dump HTTPS traffic (pinned or otherwise) if you have root access to the client, which in the case of Android is not difficult to get.
> Regardless, it’s easy enough to dump HTTPS traffic (pinned or otherwise) if you have root access to the client, which in the case of Android is not difficult to get.
Hm, well okay, would you mind explaining the easy procedure that lets you capture Facebook traffic then? I have tried lots of different methods (Fiddler, Xposed modules, etc.) with root access and have failed. It's no problem in theory, but practice is another matter...
If you have root access to one end of encryption, you necessarily can inspect and save it. The way I would do this is by issuing a certificate for facebook.com that I mark as trusted on the device. This will let you do a man in the middle "attack." But you can probably do this directly on the device: just look for where the encryption is taking place, and intercept it.
> The way I would do this is by issuing a certificate for facebook.com that I mark as trusted on the device.
Half the entire point is "mark it as trusted" doesn't work when the application has already pinned the certificate it's expecting. Have you actually done this yourself at all?
> But you can probably do this directly on the device: just look for where the encryption is taking place, and intercept it.
"Just" intercept it? You mean "just" spend several weeks if not months trying to disassemble/decompile their code, figure out how to inject your own, somehow locate the relevant in-memory data structures for encryption, & reliably patch them at runtime? all while preventing the application from crashing? That's "easy" to you? Have you done any of these things you're suggesting yourself? How often have you done them? and how long have they taken you that you found them "easy"?
I've done all of these things since it's my job. I haven't tried Facebook, because I don't have any confidence in the surveillance hypothesis, but my first guess being inapplicable doesn't change the fact that root access would allow people to prove this is happening, and that hasn't happened.
Furthermore, this could be proven with some fair reliability using correlation only. Is more encrypted data sent when you're speaking? Is more encrypted data sent when the microphone permission is enabled? Does the app access the microphone while sleeping? Nobody has presented anything _close_ to evidence.
> I've done all of these things since it's my job.
It's weird that this is your day job and yet you tell me that I should mark a certificate as "trusted" when we both explicitly acknowledged that the problem was with certificate pinning. You didn't answer this part: how long does it take you to manage to intercept Android HTTPS traffic for a brand-new, never-before-seen application that uses certificate pinning on your day job?
> I haven't tried Facebook, because I don't have any confidence in the surveillance hypothesis
Well then try it with Facebook. If this kind of thing is really your day job then it shouldn't take long, and you'd do everyone a favor by (a) showing that nothing is going on, and (b) teaching people how to do it themselves so that the myth doesn't keep spreading. People would appreciate it.
> Furthermore, this could be proven with some fair reliability using correlation only.
No, it can't. They don't need to be sending raw audio. They could just do some rudimentary speech recognition and send it along with some other routine data.
> Does the app access the microphone while sleeping? Nobody has presented anything _close_ to evidence.
I've personally logged it accessing the microphone when I've been scrolling on my news feed. Though I don't see why you'd believe me anyway.
Technicalities and choice of words are amazing ways to deny what isn't exactly happening here.
For example, why use the word 'recording' when the problem is accessing the microphone at all... Facebook doesn't have to 'record' to stream the audio to their server for audio recognition without 'recording' anything.
Of course all of this is just from the title of the article but the paranoid will pretty much always be able to find wiggle room.
> Here’s how it works: if you choose to turn the feature on, when you write a status update, the app converts any sound into an audio fingerprint on your phone. This fingerprint is sent to our servers to try and match it against our database of audio and TV fingerprints. By design, we do not store fingerprints from your device for any amount of time. And in any event, the fingerprints can’t be reversed into the original audio because they don’t contain enough information.
And my previous comment :[2]
> Notice how they say they do not store sound or recordings. But what if it is converted to text?
Notice below they say that the fingerprint cannot be reversed into the original audio because it doesn't contain enough information.
Text cannot be converted to the ORIGINAL audio but it can be converted to audio.
This may be just me being paranoid, but I do know that legalese is designed exactly to say something without saying everything.
Once it happened to me that Facebook has shown me under "People You May Know" a person that was on the same WiFi as I was. We had zero mutual friends and no known connection prior.
I'm inclined to dismiss it as confirmation bias, but I've had this experience personally - a conversation followed shortly by an extremely specific advertisement on FB shortly thereafter.
I wonder how likely it is that this sort of thing happens without a mic being involved. Consider that Facebook already mines all sorts of Facebook data, as well as data from websites you visit. They much have a pretty good profile of people's interests.
Given that, it's fairly probably that you will be shown ads about things that you are interested in, and it's fairly probably that you'll talk about these topics with friends. Given this and the number of people that use Facebook, it's probably not that unlikely that your experience happens simply by chance fairly frequently.
The odds of something like that happening JUST AFTER you've spoken aloud about it should be small. Too small even for a dozen "random" events, even considering the number of Facebook users.
As suggested in the comments on Reddit, there's another possible explanation for this -- someone else in the conversation does a search based on what you were talking about, clicks on a result that sends tracking data to Facebook, and then the ads are retargeted to you based on a shared public IP address.
The last time I noticed this, it was a conversation between my wife and I, and I (we) poked around in her search history for that reason. Nothing that I could see, and it was within minutes of the conversation.
All that said, I still lean towards confirmation bias.
I worry about facebook and all that and the state of privacy in the modern world, but a lot of these stories sound a lot like confirmation bias to be honest.
As I’m curious, are you using Android or iOS? I can’t prove it, but I find it suspect this happens on iOS, but I know from first hand experience it’s technically possible on Android.
My partner is convinced this happens, she even tried to run experiments trying to say random things and see if they show up. None of the experiments worked out, but I have to admit some of the coincidences are interesting. But not so much advertising, just any content that ends up in the feed. I'm more on the side of confirmation bias.
> "I run ads product at Facebook. We don't - and have never - used your microphone for ads. Just not true," Mr Goldman wrote.
Personally, I am so used to corp speak and word play by major companies that it is difficult to take anything at face value. Therefore, I think the nature of the question asked to Facebook should be different. It should be on what they do rather do they do X. So, here's my question - "What does Facebook use data gathered from user microphones across any devices for?"
I was about to leave an extremely similar comment, but I was going to frame it in terms of the NSA "direct access" story. I got the impression from that story that immediate denials from tech companies, which seemed categorical, turned out to hinge on which magic words were used.
In this case, when I read a sentence like "We don't use your microphone for ads" I end up doing that game where you re-read the sentence over and over and emphasize a different word each time, trying to guess which one is the magic word.
I’m sure they don’t use the microphone for ads. If they use the microphone, they surely use it to feed data into their profile of you which gets used for all sorts of things... like ads. One level of indirection is enough to make this denial technically true.
In a statement issued on June 2nd [2016], Facebook said it "does not use your
phone’s microphone to inform ads or to change what you see in News Feed."
That's pretty fucking clear to me.
Original comment:
If they had said something like "We don't use microphones to gather data to drive personalisation on users profiles" there would be comments here on HN complaining about corporate speak and how they didnt just flat out deny using it for advertising.
Apps get busted for doing dodgy things all the time. The Facebook app has been decompiled and network traffic inspected and studied so often that if this actually was true then it would have been proven by now. Instead, all we have is this weird 'survivorship bias' where one time someone said something out loud there was a tangentially related ad on FB.
Today I was looking up bars in Budapest and found one on Google. I started looking at images, at the only one that loaded so far on the slow internet was of food. "Wow, Google knows me so well. All I care about right now is food and the first image it loads is of food" I said out loud to my friend. Eventually the internet caught up and the rest of the images that loaded were of food. It only showed me food photos (in a bar thats more known for its decoration and alcohol). Nothing more than a coincidence.
And people have a problem with it, and so the companies use Corp speak to try and make it seem like they aren't doing what you asked. The poster higher up asked how they could remove all doubt, and it's really easy. The problem is that they are trying to remove all doubt while continuing to do business as usual
Its only useful to train if they also have the transcript by a human. Eavesdropping conversations and having armies of people transcribing them seems like a very expensive and illegal way to get that data when there are probably millions of available samples, TV shows, etc that have both voice and transcription available already off the shelf.
The 2017 F8 developer conference featured a lot on Machine Vision. Processing images and video for objects, such as for automatic close captioning, is where vast resources are focused. I highly recommend watching a few of the videos, they’re specially aware as well and can infer orientation of obscured things like limbs. Microsoft has real-time audio translation, doing machine transcription at scale is totally feasible.
The OP that I was replying to was insinuating that FB is collecting audio as training data to create AI models like the ones you are talking about. I was pointing out that raw audio is useless to train an AI model for recognizing words, the whole point of training data for AI is that you have an input and a known output (transcription) that you can use to train and test the model with, having just input is useless for training.
That makes sense. So, given that this is data it's users have an obvious interest in keeping private, Facebook could at least inform of this. The only reason I could imagine they aren't doing this is that their voice recognition isn't used for any released products. I think that's a viable theory of what they would be doing with the voice data.
EDIT: User 'crucifiction' makes a good point about needing the transcript to use it for voice recognition. So, who knows.
It doesn't in any sense justify collecting data and being so elusive about their intent. We naturally have reason to get the impression that Facebook just wants the data, will try anything it can to keep collecting it until so much energy has been invested in the PR issues that the collection has an attributable effect on their market.
But maybe they are using the voice data to change the world for the better and bring people together yada yada! :)
Edit: Actually, FB will ask for microphone permission if try to record a video. Based on every other app that records video also prompts for permission to record using the microphone, I'm lead to believe the microphone permission is required on iOS if you want to record video.
FB Messenger does not prompt for Microphone until you record a video or voice memo.
Instagram can record video posts and Messenger can make video calls, both requiring microphone access. In either case the recording is obvious with a red bar for microphone access in the background, like all other iOS apps.
Those apps have voice and video features, so it makes sense that they'd request microphone access. (And iOS lets you disable those specific features if you want.)
iOS also shows an indicator if an app is using the microphone. This is true even for built-in Apple apps like Voice Memos. There's no way to surreptitiously record.
Apparently people think there's a secret backdoor that lets an iPhone app record audio without turning the status bar red. You could deny mic permission and people would still think it's the secret backdoor.
Well no, Apple (in an unprecedented move), granted the Uber app the com.apple.private.allow-explicit-graphics-priority entitlement.
"Apple gave us this permission because early versions of Apple Watch
were unable to adequately handle the level of map rendering in the Uber
app," an Uber representative, Melanie Ensign, told Business Insider.
"Subsequent updates to Apple Watch and our app removed this dependency,
and we're working with Apple to remove the API completely."
Uber was doing other sneaky stuff to read the devices serial number (or other unique identifiers) and evade Apple's countermeasures, but that's seperate from this.
What is the meaningful difference, in the context of tedunangst's post, between an entitlement that allows surreptitious recording of video and a backdoor that allows surreptitious recording of audio?
Uber's entitlement was always 'public'. Facebook doesn't hold any private entitlements that would allow it to record audio in the background without notifying the user.
I'm 95% sure that there will be an entitlement for recording audio without showing the red mic bar (look at the Phone app), but it won't be a public one. There is no way for a third party app to get past app review with a custom entitlement without explicit permission from Apple. I'd be very surprised if it even passed automated review when the app was uploaded.
“Facebook does not use microphone input for ads, user interest assessment or for any purpose other than facilitating a single user initiated transaction.”
My question was NOT "Does Facebook use the phone's microphone to inform ads or change what we see in the News Feed?". It seems you have misunderstood my question. Let me repeat my question again :
"What does Facebook use data gathered from user microphones across any devices for?"
No, it's not a loaded question. It's much more like "list everything you've punched". Maybe with "in my house" appended as an analogy to it being user microphones.
The question does not presume any misuse. It just leaves no room for implications.
I understand what you're asking and you do have to be particularly aware of what /you/ are asking.
That particular question, in context of these allegations, would presuppose that they use microphone data for something other than what the user clearly is aware of, e.g. voice chat.
So an answer like "to provide users the service they need" would undeniably be spun by media to be: Facebook uses microphone to tweak news feed. The longer the answer is, the more likely it can be misunderstood and taken out of context.
You shouldn't expect a public company to answer questions like that, simply because the media will gladly take the answer out of context. You need to phrase it specifically, so they can answer without being misunderstood.
>In a statement issued on June 2nd [2016], Facebook said it "does not use your phone’s microphone to inform ads or to change what you see in News Feed."
>That's pretty fucking clear to me.
Said nobody who understands law, language, or politics. It's absolutely fucking clear to everyone who reads with discrimination what this means. It means they are using your microphone to vacuum up your information. That they added the qualifier "..to inform ads or to change what you see in the News Feed" gives them the verbal and legal wiggle room they need to do absolutely anything they want with your information. For example, they could use your microphone to monitor, record and quantify everything they hear to "adjust a user profile" that they keep on all users. In that case, they could simply claim that the ads you see and the news feed you get are based on your user profile.
>How could they respond to remove all doubt they're listening?
They could very simply. "We do not listen to your microphone or record content, data, or metadata from your microphone in any circumstance". Without qualifiers of any kind.
I just assume that they hire a 3rd party to do it and just incorporate the processed "external data source". Now they don't use the mic, just a text stream purchased from Acoustic Magic Analytics Inc.
Why would they even bother? On the one hand, they don't need to, they already collect an incredible amount of data on everyone. And on the other, if they were to get caught listening, that's a gross violation of so many federal laws that it's possible Facebook wouldn't even exist the day after it was uncovered.
The risk is incredibly high, and the payoff not that great.
What we're seeing is simply that people fail to understand how much data they leak, and how effective Facebook is at putting it all together. They assume it must be magic, listening to their words, listening to their thoughts, but the reality is they don't need to, because we are _voluntarily_ telling them all this info already.
They don't at all deny listening to conversations in this article. They only deny using that information for advertising.
Now about the confirmation bias thing, it seems more likely to me that your credit card history or the store member card purchase history is being mined.
I think the risk versus reward for Facebook and other companies is very high. All things being equal, I think it's more likely than not they will record things, will listen for words you say, and use that. The risk is enormous if it comes out with proof, but let's say they only listen a small percentage of the time, that kind of AI analysis is priceless.
No, you’re right. There is zero accountability for this kind of stuff. No US government agency or US-based business has ever suffered for it. I defy anyone to prove me wrong.
I doubt it will make a difference to most people. The staunch pro-privacy stance really only exists on tech-savvy forums like HN. Go on reddit sometimes and observe how people will say "so what if Google has all my info, it's useful and free". If Zuckerberg calling his users dumb fucks for trusting him didn't get them off the platform, a "small" privacy violation like this won't either.
And the general sentiment appears to be very far from what you claim. I think it's possible that this issue is going to come to a head this time around. I don't know what the outcome will be. Just saying it's confirmation bias (the likely real explanation?) may not be enough to satisfy people.
Assume it was perfectly true at the time. Is that relevant to the current state of Facebook, with 20k employees, public stock, and a 33-year-old Zuckerberg?
On the other hand seeing as this must be done client side then someone should be able to do an investigation and see if Facebook sends voice data to their servers.
And not to mention how many engineers that work on Facebook that could be whistle blowers.
The kind of people who believe these conspiracy theories about Facebook are not the kind of people who believe in scientific methodology. Fuzzy bigfoot syndrome and all.
I'm sure many people have tried to do that and have found nothing. But having found nothing is not a definite proof of such a thing not existing so nobody published their results.
414 comments
[ 2.5 ms ] story [ 198 ms ] threadYou can decompile the app and see what it does.
I mean, really?
So yes, really.
Keep in mind, this was someone who would have been completely new to podcasting and have never searched for podcasts or podcast apps before.
Made both of us highly suspicious.
You just forget all the other signal you've giving. Funnily enough, last week somebody was trying to convince me that Facebook was recording their every word.
"I was telling a friend about a recent trip to Thailand", they said, "and I immediately saw an ad for Thai Tea". I asked if he'd used Facebook when he was in Thailand. He had.
Which is more likely: Facebook is recording every conversation of every user on their app and mini them for ads or the advertiser in question just targeted people who had recently been to Thailand.
https://developer.qualcomm.com/blog/using-qualcomm-hexagon-s...
"Industry wide adoption"
2013
The rest is PR.
Early 2014, phones that used it were:
http://www.freewaresquad.com/2014/02/18/phones-with-snapdrag...
Today that CPU series is everywhere.
> enables devices powered by Snapdragon 800 processors to be “woken up” by a custom voice command (beginning with a custom phrase set by the device OEM like “Hey Snapdragon”, for example)
Because they are always on and have to use minimal power, these chips are used to recognise a very specific term (probably why these companies don't let you customise how you summon the assistant), and cannot do anything like voice transcription, network calls, ... (which would just kill your battery).
If this is happening then it must require some amount of CPU and memory use on the device, which even if hidden from the OS can be detected via power use and heat dissipation. It should be possible to indirectly measure a phone's CPU load (including any hidden processors) by filming it in the infrared and/or by attaching it to power and measuring the wattage consumed from the supply.
If this is happening it must also require some network I/O. Something must be phoning home and reporting the data if the data is to be useful.
Note that there is an intrinsic trade-off between the heat dissipation angle and the bandwidth angle. If the device is doing parsing, speech recognition, AI, etc. locally then it must be consuming more energy and dissipating more heat. If the device does these things remotely it must upload more data. Compressed audio is far more bandwidth intensive than digested meta-data.
This seems like something security researchers can and should look into.
People started discovering that after iOS introduced the battery/power usage section in the Settings app.
That's why I'm skeptical that Facebook would do this -- it seems like a terrible business decision given the high chance of getting caught, either by reverse engineering or a whistleblower.
I guess another possible explanation is some very sophisticated combination of location tracking and Kellogg’s targeting people they know will be primed by that promotional item. And then of course it could always just be an amazing coincidence.
All I know is I revoked microphone permission after that.
EDIT: Something just occurred to me. Facebook doesn’t have to be the one using your microphone to target ads. The ad buyers just have to have a way to match your phone to your ID (which they can easily; there was a site posted here a while back that sells that) and then listen in through ANY one of the apps you have installed. If this were the scheme, Facebook could reasonably get away with saying “WE don’t use your microphone to target ads.”
More on-topic: I recall reading that the Facebook app would activate the microphone when one was posting a status update, apparently listening for things like a movie playing in the background and such.
ETA: Related Snopes articles:
https://www.snopes.com/computer/facebook/facebooklisten.asp
https://www.snopes.com/2016/06/04/professor-claims-facebook-...
Like you, I have gotten pretty good at noticing "weasel words" and such in the government's statements (denials). It is clear that they take their time to make sure they word their statements ever so carefully.
Horrible battery life?
> Remember the time they kept the app open in the background on iOS devices by sending an empty audio buffer to the phone for playback?
> People started discovering that after iOS introduced the battery/power usage section in the Settings app.
They couldn't do it 24/7, of course, but they could certainly take "samples" at regular intervals or in certain locations or something. From my own experience, it doesn't seem that Facebook has ever been worried about battery life or preserving it.
[0]: https://news.ycombinator.com/item?id=15580748
Because they'd get hit by huge lawsuits and fines from governments all over the world. Europe is especially hungry to fine US companies for privacy issues.
Also it would be terrible for their reputation. Hordes of people would uninstall the app, and even delete their accounts. I would for sure.
Anyway, I want a LED on my camera and microphone. And a hardware switch.
Once they have a business model that does not revolve around selling data about their users to their customers, claiming they don't spy can be something other than a bad punchline.
If there was some third party ad library that was installed across multiple apps that streamed audio and/or performed local speech recognition on everything you said don't you think somebody would have a) noticed by now, b) a security researcher figured it out, or c) everyone's battery's would be running red hot all of the time.
Teams at Apple and Google reverse engineer and debug the top 100 apps on their platforms all the time (there are teams dedicated to doing it). Facebook is also not the NSA - people leak stuff from them all the time. You're suggesting that either everyone is complicit in this, or there's a conspiracy so large and technically complex that it's managed to be kept secret all this time with no technical evidence outside of anecdotes.
I’m not sure I trust that security researchers can be relied upon to catch everything. There are so many flagrant abuses of user privacy that go unchecked or remarked upon and nevertheless ignored that such an avenue of prevention seems highly unreliable.
Also you’d be surprised at what people are willing to do when you define a certain culture and incentive structure. While you may dismiss it, I find it frankly astounding that the NSA program managed to metastasize into the behemoth it became before it was finally outed to the public. And even a cursory reading of behavioral psychology will help to explain how that can happen.
I also know that Facebook perversely prioritizes metrics above pretty much all else. And it means they’re perfectly willing to indulge in dark patterns if the numbers line up.
The "People you may know" anecdote below is, similarly, some combination of location, wifi sniffing, luck, and confirmation bias.
which ones do you mean?
The fact that there is no hard evidence makes me very skeptical.
Every anecdote is explainable another way. Someone in the comments said they were having a conversation about storage units and saw an ad for storage units immediately after. Even if they had never searched for storage units -- and can we really be sure? That would be easy to forget -- Facebook might just have extrapolated they they would need a storage unit from related searches, like searching for a moving company.
I would be willing to bet this type of algorithmic extrapolation explains every one of the microphone-ad-conspiracy stories.
I'll throw another layer on the conspiracy pile and say they deactivate that behavior for anyone they notice trying to prove it, similar to Uber's Greyball.
But I also know corporations are usually scrupulous about following laws so unless there is something in Facebook's ToS/EULA that mentions doing this I'm skeptical.
I don't see why the vague terms in their data policy [1] couldn't include audio (all-caps are my own emphases):
> We collect information FROM or about the computers, PHONES, or other devices where you install or access our Services, depending on the permissions you’ve granted. [...] Here are some EXAMPLES of the information we collect:
> - Device locations, including specific geographic locations, such as through GPS, Bluetooth, or WiFi signals.
If device location is merely an example of information they gather from my phone then why couldn't audio just as easily be another example?
[1] https://www.facebook.com/full_data_use_policy
Hm, well okay, would you mind explaining the easy procedure that lets you capture Facebook traffic then? I have tried lots of different methods (Fiddler, Xposed modules, etc.) with root access and have failed. It's no problem in theory, but practice is another matter...
Half the entire point is "mark it as trusted" doesn't work when the application has already pinned the certificate it's expecting. Have you actually done this yourself at all?
> But you can probably do this directly on the device: just look for where the encryption is taking place, and intercept it.
"Just" intercept it? You mean "just" spend several weeks if not months trying to disassemble/decompile their code, figure out how to inject your own, somehow locate the relevant in-memory data structures for encryption, & reliably patch them at runtime? all while preventing the application from crashing? That's "easy" to you? Have you done any of these things you're suggesting yourself? How often have you done them? and how long have they taken you that you found them "easy"?
Furthermore, this could be proven with some fair reliability using correlation only. Is more encrypted data sent when you're speaking? Is more encrypted data sent when the microphone permission is enabled? Does the app access the microphone while sleeping? Nobody has presented anything _close_ to evidence.
It's weird that this is your day job and yet you tell me that I should mark a certificate as "trusted" when we both explicitly acknowledged that the problem was with certificate pinning. You didn't answer this part: how long does it take you to manage to intercept Android HTTPS traffic for a brand-new, never-before-seen application that uses certificate pinning on your day job?
> I haven't tried Facebook, because I don't have any confidence in the surveillance hypothesis
Well then try it with Facebook. If this kind of thing is really your day job then it shouldn't take long, and you'd do everyone a favor by (a) showing that nothing is going on, and (b) teaching people how to do it themselves so that the myth doesn't keep spreading. People would appreciate it.
> Furthermore, this could be proven with some fair reliability using correlation only.
No, it can't. They don't need to be sending raw audio. They could just do some rudimentary speech recognition and send it along with some other routine data.
> Does the app access the microphone while sleeping? Nobody has presented anything _close_ to evidence.
I've personally logged it accessing the microphone when I've been scrolling on my news feed. Though I don't see why you'd believe me anyway.
For example, why use the word 'recording' when the problem is accessing the microphone at all... Facebook doesn't have to 'record' to stream the audio to their server for audio recognition without 'recording' anything.
Of course all of this is just from the title of the article but the paranoid will pretty much always be able to find wiggle room.
> Here’s how it works: if you choose to turn the feature on, when you write a status update, the app converts any sound into an audio fingerprint on your phone. This fingerprint is sent to our servers to try and match it against our database of audio and TV fingerprints. By design, we do not store fingerprints from your device for any amount of time. And in any event, the fingerprints can’t be reversed into the original audio because they don’t contain enough information.
And my previous comment :[2]
> Notice how they say they do not store sound or recordings. But what if it is converted to text?
Notice below they say that the fingerprint cannot be reversed into the original audio because it doesn't contain enough information. Text cannot be converted to the ORIGINAL audio but it can be converted to audio.
This may be just me being paranoid, but I do know that legalese is designed exactly to say something without saying everything.
[1] http://newsroom.fb.com/news/2014/05/a-new-optional-way-to-sh...
[2] https://news.ycombinator.com/user?id=foxfired
If it’s syllable- or word-length fingerprints it’s not much solace that they won’t be able to get the original audio.
Given that, it's fairly probably that you will be shown ads about things that you are interested in, and it's fairly probably that you'll talk about these topics with friends. Given this and the number of people that use Facebook, it's probably not that unlikely that your experience happens simply by chance fairly frequently.
All that said, I still lean towards confirmation bias.
Personally, I am so used to corp speak and word play by major companies that it is difficult to take anything at face value. Therefore, I think the nature of the question asked to Facebook should be different. It should be on what they do rather do they do X. So, here's my question - "What does Facebook use data gathered from user microphones across any devices for?"
In this case, when I read a sentence like "We don't use your microphone for ads" I end up doing that game where you re-read the sentence over and over and emphasize a different word each time, trying to guess which one is the magic word.
Edit: https://www.theverge.com/2016/6/3/11854860/facebook-smartpho...
That's pretty fucking clear to me.Original comment:
If they had said something like "We don't use microphones to gather data to drive personalisation on users profiles" there would be comments here on HN complaining about corporate speak and how they didnt just flat out deny using it for advertising.
Apps get busted for doing dodgy things all the time. The Facebook app has been decompiled and network traffic inspected and studied so often that if this actually was true then it would have been proven by now. Instead, all we have is this weird 'survivorship bias' where one time someone said something out loud there was a tangentially related ad on FB.
Today I was looking up bars in Budapest and found one on Google. I started looking at images, at the only one that loaded so far on the slow internet was of food. "Wow, Google knows me so well. All I care about right now is food and the first image it loads is of food" I said out loud to my friend. Eventually the internet caught up and the rest of the images that loaded were of food. It only showed me food photos (in a bar thats more known for its decoration and alcohol). Nothing more than a coincidence.
EDIT: User 'crucifiction' makes a good point about needing the transcript to use it for voice recognition. So, who knows.
It doesn't in any sense justify collecting data and being so elusive about their intent. We naturally have reason to get the impression that Facebook just wants the data, will try anything it can to keep collecting it until so much energy has been invested in the PR issues that the collection has an attributable effect on their market.
But maybe they are using the voice data to change the world for the better and bring people together yada yada! :)
Amend their privacy policy.
https://imgur.com/a/ifaiT
Edit: Actually, FB will ask for microphone permission if try to record a video. Based on every other app that records video also prompts for permission to record using the microphone, I'm lead to believe the microphone permission is required on iOS if you want to record video.
FB Messenger does not prompt for Microphone until you record a video or voice memo.
I can see "Messenger" and "Instagram" with access to microphone.
iOS also shows an indicator if an app is using the microphone. This is true even for built-in Apple apps like Voice Memos. There's no way to surreptitiously record.
The banner won’t show if the app using the mic is in the foreground.
Inside the app I believe they can record whenever, without the banner appearing.
"What does Facebook use data gathered from user microphones across any devices for?"
The question does not presume any misuse. It just leaves no room for implications.
That particular question, in context of these allegations, would presuppose that they use microphone data for something other than what the user clearly is aware of, e.g. voice chat.
So an answer like "to provide users the service they need" would undeniably be spun by media to be: Facebook uses microphone to tweak news feed. The longer the answer is, the more likely it can be misunderstood and taken out of context.
You shouldn't expect a public company to answer questions like that, simply because the media will gladly take the answer out of context. You need to phrase it specifically, so they can answer without being misunderstood.
>That's pretty fucking clear to me.
Said nobody who understands law, language, or politics. It's absolutely fucking clear to everyone who reads with discrimination what this means. It means they are using your microphone to vacuum up your information. That they added the qualifier "..to inform ads or to change what you see in the News Feed" gives them the verbal and legal wiggle room they need to do absolutely anything they want with your information. For example, they could use your microphone to monitor, record and quantify everything they hear to "adjust a user profile" that they keep on all users. In that case, they could simply claim that the ads you see and the news feed you get are based on your user profile.
>How could they respond to remove all doubt they're listening?
They could very simply. "We do not listen to your microphone or record content, data, or metadata from your microphone in any circumstance". Without qualifiers of any kind.
The risk is incredibly high, and the payoff not that great.
What we're seeing is simply that people fail to understand how much data they leak, and how effective Facebook is at putting it all together. They assume it must be magic, listening to their words, listening to their thoughts, but the reality is they don't need to, because we are _voluntarily_ telling them all this info already.
Somehow this kind of phrasing turns several lights bright on my bullshit detector.
Now about the confirmation bias thing, it seems more likely to me that your credit card history or the store member card purchase history is being mined.
Maybe I'm just having a bad day.
And the general sentiment appears to be very far from what you claim. I think it's possible that this issue is going to come to a head this time around. I don't know what the outcome will be. Just saying it's confirmation bias (the likely real explanation?) may not be enough to satisfy people.
It was back when he was 19, so I think it's fair to cut some slack for that, but the quote is true.
And not to mention how many engineers that work on Facebook that could be whistle blowers.