This has worrying privacy implications. I hope Apple makes a public announcement about this but wouldn’t be surprised if they don’t. I also would expect EFF will get on this shortly.
> Regardless of what Apple’s long term plans are, they’ve sent a very clear signal. In their (very influential) opinion, it is safe to build systems that scan users’ phones for prohibited content.
> That’s the message they’re sending to governments, competing services, China, you.
Is it? That’s just something the tweets have read in.
The message could equally well be ‘We won’t become an easy political target by ignoring a problem something most people care about like child porn, but we are going to build a point solution to that problem, so the public doesn’t force us to bow government surveillance requests.’
It’s easy to nod along with an anti-Apple slogan, but we need to consider what would happen if they didn’t do this.
If Apple thought this kind of dragnet was a losing political fight that tells me they've become too weak to stand up to unreasonable government demands. Where is the company that won the public opinion battle over unlocking a mass shooter's phone?
This isn’t a government demand. This is something the public cares deeply about, and Apple is solving it their own way.
Public opinion is not in favor of giving safe harbor to pedophiles and child pornographers, and I can’t see why anyone would even want Apple to fight that battle.
Not sure where you got that information. I haven't seen any official announcements, so I assumed based on it being US-only and rolled out with no fanfare (except critical press stories citing unnamed sources) that it's something the FBI asked for.
Not sure where you got that - seems like you’ve just made up an explanation that it’s an FBI demand out of whole cloth.
What we do know is that it is a proprietary solution using a proprietary hash which only applies to Apple products and that Apple has always presented themselves as a family friendly company that doesn’t support criminal use cases.
Everything points to this being something Apple thinks needs to be solved before the public asks why they haven’t.
If it was a government demand, we’d presumably see Google responding to it too.
You still haven’t explained why not working to deter pedophiles and child pornographers, is an important battle for them to fight.
They might think it’s a problem they want to solve on their own terms, and that would seem to be what they have done here.
Not solving a problem people care about just because the government also cares about it seems illogical. I think a lot of people like the idea of corporations taking responsibility for the social problems they cause without needing to be forced to do so by the government.
False positives, what if someone can poison the set of hashes, engineered collisions, etc. And what happens when you come up positive - does the local sheriff just get a warrant and SWAT you at that point? Is the detection of a hash prosecutable? Is it enough to get your teeth kicked in, or get you informally labeled a pedo by your local police? On the flip side, since it's running on the client, could actual pedophiles use it to mutate their images until they can evade the hashing algorithm?
False positives are clearly astronomically unlikely. Not a real issue.
Engineered collisions seem unlikely too. Not impossible. Unless there is a straight up cryptographic defect in the hash algorithm, it seems hard to see how engineered collisions could be made to happen at any scale.
At Apple scale, a once in a million issue is going to ruin the lives of 2000 people. A false positive here is not a mild inconvenience. It means police raiding their house, potentially damaging it, seizing all of their technology for months while it is analyzed, and leaving these people highly stressed while they try to put their lives back together.
This isn't some web tech startup where a mistake means someones tshirt got sent to the wrong address. Peoples lives will quite literally be ruined over mistakes here.
Is it a once in a million issue? The collision rate matters. It could easily be much higher and then it wouldn’t matter that it was being used at Apple’s scale.
If this was the kind of hash where flipping one bit of the input completely scrambles the output, the bad guys would just flip one bit of the input to evade it. Obviously a PhotoDna type of hash is going to be be easier to cause a collision with because they're averaging out a ton of the input data. According to Wikipedia the classic way to do it is convert it to monochrome, divide it into a grid, and average the shade of each of the cells. If they're doing that you could probably just pass in that intermediate grid and it would "hash" to the same result as the original picture with no porn present.
We know perceptual hashing and cryptography have incompatible requirements. Think of an image, the same image with 1 pixel changed, and a very different image. A perceptual hash should say 1 and 2 were related and not 3. Cryptographers call that failing a chosen plaintext attack.
Why do you think that? There are plenty of whitepapers on fooling NNs by changing random pixels by a bit, so that the picture is not meaningfully changed for a person, but the computer will label it very differently. Do note that these are not cryptographic hashes because they have to recognize the picture even when compressed differently, cropped a bit, etc.
The hashes will of course be provided by local governments, who have the ultimate authority (because they can forbid Apple to sell there, and Tim Cook never says no to money).
So if I understand correctly, they want to scan all your photos, stored on your private phone, that you paid for, and they want to check if any of the hashes are the same as hashes of child porn?
So... all your hashes will be uploaded to the cloud? How do you prevent them from scanning other stuff (memes, leaked documents, trump-fights-cnn-gif,... to profile the users)?
Or will a huge hash database of child porn hashes be downloaded to the phone?
Honestly, i think it's one more abuse of terrorism/child porn to take away privacy of people, and mark all oposing the law as terrorists/pedos.
...also, as in the thread from the original url, making false positives and spreading them around (think 4chan mass e-mailing stuff) might cause a lot of problems too.
It always starts with child porn, and in a few years the offline Notes app will be phoning home if you write speech criticising the government in China.
This technology inevitably leads to the sueveillance, suppression and murder of activists and journalists. It always starts with protecting the kids or terrorism.
Perceptual hashes like what Apple is using are already used in WeChat to detect memes that critique the CCP.
What happens on local end user devices must be off limits. It is unacceptable that Apple is actively implementing machine learning systems that surveil and snitch on local content.
Which'd be fine if we had one global government with world wide jurisdiction. Or technology choices from companies which couldn't be pressured by governments outside your personal regulation jurisdiction.
I wouldn't hold your breath waiting for those regulations to become law in, say, Chine or Turkey or Saudi Arabia. I'd bet even Israel won't pass them, surely NSO have enough political lobbying swing (and probably also suitable blackmail material on sitting politicians).
They sell software designed to break into people's phones to oppressive regimes all over the planet. They definitely have the capability and requisite lack of ethics to compromise their own politicians; there's no probably about it.
But we don't have a global government, so the next best thing is for individual countries to pass such regulation, which would prevent products violating privacy like this from being offered and sold in those countries.
Think of GDPR, which is essentially each member of the EU saying in unison, "your product/service must comply with these data protection laws, or you can't legally do business with any of our citizens".
Come to think of it, I wonder if this Apple thing would even fly under GDPR?
> Come to think of it, I wonder if this Apple thing would even fly under GDPR?
Possibly? I’m not a lawyer, but if this is about compliance with a legal obligation, and they’re under that category of pressure? I think GDPR would allow that?
Certainly seems more likely allowed than the stuff Facebook complained Apple was preventing them from doing.
>Which'd be fine if we had one global government with world wide jurisdiction.
I wholeheartedly disagree. A world wide goverment would be catastrophic for whistleblowing. Atleast as a whistleblower you can live somewhat safely in a country opposing your own. With a one world government you would have nowhere to run.
And I wouldn't expect them to protect citizen's interests any better than current governments do. Contrary, I think this lack of balance in the world would embolden them further.
> in a few years the offline Notes app will be phoning home if you write speech criticising the government in China.
A totalitarian autocracy like China does not need this technology to search for wrongspeech, sadly. You are of course aware that all Chinese iCloud users get their data stored in a special set of datacenters that Apple actually doesn't control.
The problem is, that this will be done in other, (currently) freer countries. Eg. Reddit has removed a lot of anti-china posts in the last few years. And of course, local leaders will use this to find anti-local-leader stuff on their citizens phones too.
This seems to be done voluntarily by NVIDIA, at least partially. While in Seattle I set up geo-blocking on my LAN as an experiment. Later when I tried to create an NVIDIA account I couldn't because it was attempting to store my PII at nvidia.cn. When I changed the url to nvidia.com everything worked just fine. I've always wondered what non-evil reasons one could use to explain that choice by NVIDIA. Ping was at least 2x longer to .cn.
No, your hashes are not uploaded to the cloud, yes, hashes are downloaded to your phone. Yes, it will be interesting to see if it gets spammed with false positives, although it seems as though that can easily be identified silently to the user.
128 bit hashes are “expect your first collision when each human buys 2305 iPhones, all with one terabyte storage, and then fills them up with photos that are an average of 1MB in file size”
The probability of Apple and all its devices winking out of existence due to quantum fluctuations is not zero. ‘Not zero’ is effectively zero if the number is small enough.
It cannot be a cryptographically secure hash, simply because avoiding detection would then be trivial: change one channel in one pixel by one. Imperceptible change, different cryptographic hash.
This isn't cryptographic though. That would make the entire database absolutely trivial to bypass with tiny imperceptible random changes to the images.
If the details of the "hashing" scheme used is publicized, I imagine it will be near trivial. It's a long-standing problem in computer vision, to find a digital description of an image such that two similar images compare equal or at least similar.
State-of-the-art for this field is deep learning, and a /huge/ problem with the DL approach is that you can generate adversarial examples. So for example, a picture of a teacup that is identified by /most/ networks as a dog. It's particularly damning, because it seems like you don't have to do this for particular deep networks, they get tricked the same way, so to speak.
Indeed, at which point we’ll know if Apple has implemented an obviously broken solution which opens us up to egregious government surveillance, or whether that is all just speculation without a factual basis.
That document you downloaded that is critical of the party will land you and your family in jail. Enjoy your iPhone.
Seriously, folks, we shouldn't celebrate Apple's death grip over their platform. It's dangerous for all of us. The more of you that use it, the more it creates a sort of "anti-herd immunity" towards totalitarian control.
Apple talks "privacy", but jfc they're nothing of the sort. Apple gives zero shits about your privacy. They're staking more ground against Facebook and Google, trying to take their beachheads. You're just a pawn in the game for long term control.
Apple cares just as much for your privacy as they do your "freedom" to run your own (un-taxed) software or repair your devices (for cheaper).
And after Tim Cook is replaced with a new regime, you'll be powerless to stop the further erosion of your liberties. It'll be too late.
But is there a realistically better alternative? Pinephone with a personally audited Linux distro? A jailbroken Android device with a non-stock firmware that you built yourself? A homebuilt RaspberryPi based device? A paper notepad and a film camera and an out of print street map?
Not really, and I'm not going to sway anyone deeply into the ecosystem.
My hope is that those of you that share my viewpoint will call your legislators and demand regulations or a break up. There are forces of good within the DOJ that are putting together an antitrust case against Apple, and the more of us that lend our voices, the louder and more compelling the argument.
The DOJ is really the last lever we have, and that's pretty good measure for the power Apple wields.
An Android device running non-stock is a realistically better scenario. The big problem there is that the state of Android drivers means your hardware options are severely cut down (in practice, to a selection about the size of Apple's - the Pixel line and some assorted others).
With non-stock (assuming not jailbroken but just a totally different operating system) I think (I might be wrong... I should know for sure but I awkwardly don't) you aren't even allowed to use Google Play Services at all?
You are allowed to use Google services. There is even an alternative called microG which is compatible with apps requiring Google serivices but it sends "fake" data to Google.
The best bet is probably a pixel phone with GrapheneOS. (Do note, that copperhead os is a scam and is not to be used)
Gnu/linux phones have nonexistent security, other than being niche (so security by obscurity at most). And also, they are not yet usable as a daily driver for me personally, at least.
Whether or not I am allowed to check that my entrance has no locks whatsoever doesn’t make it harder to open it. And the reverse, even if I don’t know the details of the lock in my door, it will not let others pass through.
What if I have a locksmith verify it for me? Like Apple and Android have been checked by several security researchers and while they absolutely have holes, there is are at least gates that can have them. Sandboxing is the bare minimum an OS should do if it wants to have third party applications installed.
Basically Micay is a legitimate security researcher who created the project and it was later hijacked by the company funding some of it. That company since then try to badmouth Micay at any place they find and is doing shady things on top of the still open source code base. Micay was so professional to destroy the verification key at the time of the forking.
> Pinephone with a personally audited Linux distro?
Even if you don't personally audit it, you still benefit from other people doing it. Especially if the software is reproducible (and many packages are).
Viable alternatives were long gone. I really miss the days of Symbian and Meego, phones that are hackable yet intuitive to use (I.e. Nokia N900, N9).
Realistically now we have Tizen and Jolla OS, which had backings from Samsung but nobody gave two damn about it.
I bet even if any of these vanilla mobile OS gets big enough they’ll get bought by the 3 giants and suffocated to death just like how Microsoft sniped Nokia.
Not the parent commenter, but for me - Samsung are just as morally vacuous as Google, but are way less competent, at least on there software side (their component manufacture seems to be world class in at least some areas).
They'll happily do evil shit, and execute it poorly. Samsung are _way_ more likely to leak the unnecessarily and possibly illegally collected personal data they hoover up than Google are.
> and they want to check if any of the hashes are the same as hashes of child porn?
... without any technical guarantee or auditability that any of the hashes they're alerting on are actually of child porn.
How much would you bet against law enforcement to abuse their ability to use this, and add hashes to find out who's got anti government memes or police committing murder images on their phones?
And that's just in "there land of the free", how much worst will the abuse of this be in countries who, say, bonesaw journalists to pieces while they are alive?
I remember the story where some large gaming company permanently banned someone because they had a file with a hash that matched a "hacking tool". Turns out the hash was for an empty file.
That's the stated purpose, but keep in mind that these databases (NCMEC's in particular, which is used by FB and very likely Apple) contain legal images that are NOT child porn.
Think of it this way, take a regular, legal set of adult pornographic pictures. While legal, we'd still classify this set of pictures as known porn if we were tracking it.
Now the first few images might be the model completely clothed and not even be porn, maybe there's a picture of her lounging around a pool, then another picture of the pool itself. Still its part of a set of pictures that is known porn.
Heck most porn starts off with actors being clothed (so I hear lol).
I agree, I would add that people have generated legal images that match the hashes.
So I want to ask what happens if you have a photo that is falsely identified as one in question and then an automated mechanism flags you and reports you to the FBI without you even knowing. Can they access your phone at that point to investigate? Would they come to your office and ask about it? Would that be enough evidence to request a wiretap or warrant? Would they alert your neighbors? How do you clear your name after that happens?
edits: yes, the hash database is downloaded to the phone and matches are checked on your phone.
Another point is that these photos used to generate the fingerprints are really legal black holes that the public is not allowed to inspect I assume. No one wants to be involved in looking at them, no one wants to be known as someone who looks at them. It could even be legally dangerous requesting to find out what has been put into the image database I assume.
>I would add that people have generated legal images that match the hashes.
That seems like a realistic attack. Since the hash list is public (has to be for client side scanning), you could likely set your computer to grind out a matching image hash but of some meme which you then distribute.
One thing to note is these are not typical cryptographic hashes because they have to be able to find recompressed/cropped/edited versions as well. Perhaps a hash is not an accurate way to describe it.
There have been a number of cases where people have found ways to trick CV programs in to seeing something that no human would ever see. If you were sufficiently malicious I imagine it would be possible to do with this system as well.
At least one of these two things must be true: either Apple is going to upload hashes of every image on your device to someone else's server, or the database of hashes will be available somehow to your device.
Replying to my own question since I can’t edit anymore: it turns out “perceptual hashing,” which I didn’t know much about, has exactly this property, that small changes in the input result in small changes in the output.
No need to upload every hash or download a huge database with very hash. If I were building this system, I'd make a bloom-filter of hashes. This means O(1) space and time checking of a hash match, with a risk of false positives. I'd only send hashes to check against a full database.
A malware will definitely be created, almost immediately, that will download files that are intentionally made to match CP - either for the purposes of extortion or just watching the world burn.
I'm usually sticking my neck out in defence of more government access to private media than most on HN because of the need to stop CP, but this plan is so naive, and so incredibly irresponsible, that I can't see how anyone with any idea of how easy it would be to manipulate would ever stand behind it.
Signal famously implemented, or at least claimed to implement, a rather similar-sounding feature as a countermeasure against the Cellebrite forensics tool:
If they told that, people would (try to) remove it. The whole point is that you can't know which (if any) of the hundreds of thousands of files on your device it is. So they aren't telling. Could be they have (or at least claim to have) written their system so it chooses files at random; I think that's what I would do (or claim to have done).
If you can recreate a file so it’s hash matches known CP then that file is CP my dude. The probability of just two hashes accidentally colliding is approximately: 4.3*10-60
Even if you do a content aware hash where you break the file into chunks and hash each chunk, you still wouldn’t be able to magically recreate the hash of a CP file without also producing part of the CP.
The Twitter thread this whole HN thread is about shows just how to make collisions on that hash. So any image can be manipulated to trigger a match, even if that image isn’t CP.
It's the weights from the middle of a neural network that they're calling a "hash" because it encodes and generates an image it has classified as bad. Experts have trouble rationalizing about what weights mean in a neural network. This is going to end badly.
If this was a hash then it would be as the parent describes, this is at best a very fuzzy match on an image to take into account blurring/flipping/colour shifting.
It's vastly more likely that innocent people will be implicated for fuzzy matches on innocuous photos of their own children in shorts/swimming clothes than it is to catch abusers.
The other thing is, when you have nothing to hide you won't take efforts to hide it - meaning you'll upload all of your (completely normal) photos to iCloud without thinking about it again.
The monsters making these images know what they're doing is wrong, so they'll likely take efforts to scramble or further encrypt the data before uploading.
tldr; it's far likelier that this dragnet will only even apply to innocent people, than it is to catch predators.
All this said, I'm still in support of Apple taking steps in this direction, but it needs far more protections put in place to prevent false positives than this solution allows. A single false accusation by this system, even if retracted later and rectified, would destroy an entire family's lives (and could well cause suicides).
Look what happened in the Post Office case in the UK as an example of how these things can go wrong - scores of people went to prison for years for crimes they didn't commit because of a simple software bug.
> The monsters making these images know what they're doing is wrong, so they'll likely take efforts to scramble or further encrypt the data before uploading.
The ones that make national news from big busts do, because the ones that don't get caught much sooner and only make local news, because Google and other parties are have automatic CSAM identification online already (server side, not client side, AFAIK), and are sending hits to Homeland Security.
No-no-no. It's not your phone. If it was your phone - you would have a root access to it. It's their phone. And it's their photos. They just don't like when there's something illegal on their photos, so they will scan it, just in case.
I'm fairly sure the ToS says Apple can do whatever it wants whenever it wants as long as it isn't blatantly illegal. Recall when they decided everyone needed a Bono album on their phone without consent.
Apple has been scanning your photos locally for quite some time now. They've been detecting faces, making collections, finding pets, etc. This is just another step in what they have already been doing.
Processing data locally and maintaining information locally is wholly different from reading (supposedly encrypted) information from your device and reporting it back.
Maybe the part in every TOS that says the company can change the TOS at any time without warning and you should regularly check the TOS page and stop using the service if you saw a change and didn't like it?
I understand the hesitation here, but fundamentally this is like trying to close pandoras box. If something is technically possible to do AND governments demand it be done, it will be done. If not by Apple, by someone else.
Rather than complain about it, I am interested in what alternative solutions exist, or how concerns regarding privacy and abuse of this system could be mitigated.
It’s not trying to close Pandora’s box. It’s liability limitation. They’re effectively saying that if you want to distribute CP don’t do it on an iPhone.
They're basically saying, that they're watching all your multimedia, to see if maybe you're distributing child porn. This is like doing rectal exams on everyone, every day, because someone might be hiding drugs there.
I don't understand this argument at all. Look at the Clipper Chip debacle in the 90s. It was technically feasible and the government very much wanted to do it. And the reason they didn't is push back from the public, saying this is a bad idea that can easily be misused, even if it does make some law enforcement things easier. I don't see how this is any different.
Sacrificing the privacy of the many to help catch a relatively small amount of (admittedly some of the worst possible) criminals, while simultaneously enabling yet more effective surveillance and oppression by those inclined governments (of which there are plenty) is a pretty terrible idea.
Eliminating the 4th amendment or mandating clear walls sure would make the cops' job easier. But no one thinks that's even a remotely good idea.
This argument falls on it's head when confronted by reality. Either you have a trustworthy government already that will respect your rights in a slow-rolling fashion that shifts as the dialogue within the courts evolves or you already have a government that doesn't care about you and your laws/desires at all and which will do what it wants anyway. Unless you're in the ladder there's no reason to be so hostile to empowering technologies, especially when they're being used to fight some of the most heinous types of crime.
Yes, it probably could. And Apple users are helping them to normalize constant surveillance of private information. This is a read permission on the content of you phone.
There is literature about how obedience enables totalitarianism. Tons of it. Should be part of any schools curriculum in my opinion.
That tweet thread is saying it will scan for hashes client side and upload the result, circumventing E2E encryption, but then says theyre just going to do it on your icloud backups because they dont have E2E encryption, so which is it?
It doesn't say that. Today Apple servers scan uploaded photos. Tomorrow Apple phones will scan uploaded photos. The next day Apple phones could scan not uploaded photos.
The next day Apple phones could periodically listen to the environment via the internal microphone. The next day Apple phones could take and upload photos of the environment by itself.
Since Snowden I use my phone in minimalistic way. Phone calls. Minimal texting. No games. Banking apps if necessary.
Treat your phones as an enemy. Use real computers with VPN and software like Little Snitch when online. Use cameras for photography and video.
The benefits of this approach are immense. I have long attention span. I don't have fear of missing out.
If governments wan't the future to be painted by tracing and surveillance mediated towards people trough big tech - lets make it mandatory by law. And since big tech will reap benefits from the big data they must provide phones for free.
:)
>Treat your phones as an enemy. Use real computers with VPN and software like Little Snitch when online.
I'm assuming your "real computer" is a mac (since little snitch is mac only). What makes you think apple won't do the same for macos? Also, while you have greater control with a "real computer", you also have less privacy from the apps themselves, since they're unsandboxed and have full access to your system.
Your advice might be sound with the proper operating system choice, but the fact that you made such a glaring error in your initial comment makes it hard to take you seriously. It also brings into question whether you actually have a good understanding of privacy/security, or are just LARPing.
Error? Seriously? Did you make a wrong assumption that I care? About karma?
I share my experience and my point of view. Nothing more, nothing less. And the most important point is not technical.
The most important point is personal habits. To overcome smartphone addiction. We put our lives, thoughts and photos on closed technology and have false expectations that someone will care for our security or wellbeing.
I get that, but on the other hand if someone says "if you care about privacy, you should use an e2e messenger like whatsapp", then I'll have serious doubts about whether you're actually knowledgeable or just spouting buzzwords.
If you're treating your phone as hostile why would you skip gaming apps but use banking ones? That seems backwards if you're assuming your mobile is the weak point.
Sorry, I misunderstand your question. I just don't like mobile games, and mobile banking is acceptable use case for me at the moment. But pc banking is obviously a better choice. The general idea of treating your phone as a problem has deep personal benefits. It started for me with realization (years ago) that I am an addict for "dopamine" hits and this "thing" in my pocket has direct influence on my mental performance.
>It started for me with realization (years ago) that I am an addict for "dopamine" hits and this "thing" in my pocket has direct influence on my mental performance.
Sounds like this is more about "checking your phone less", than "improving security/privacy". This is evident elsewhere in your advice. eg. "phones as an enemy", but no advice about killswitches for microphone? Or some sort of mitigation against GPS/mobile networking tracking?
I don't agree with some (most?) of the parent posters comments in this threa.
But I feel there's a valid argument to be made that if your adversary is the sort of people who'd be feeding Apple image hashes to find people, you're probably be wise to carry a regular phone on which you do boring norm-core sorts of things.
A phone you use to take pictures of cats and pay your rent using banking apps and call your parents - while not using it to communicate with your dealer or your anarchist collective or your friendly investigative journalist.
In the EU the PSD2 directive obliged banks to provide strong authentication for customers login process and various operations on the account incl. payments ofc. Most of the time mobile applications are being used in the result - for either login confirm or as software OTP generators (biometric verification is also supported); the lists of printed codes are rather obsolete now and some banks may actually charge your extra for sending you text messages with such codes. I know there are hardware security tokens but in all these years I haven't seen anyone using such here.
So, it's rather hard to avoid banking apps.
Also, the PSD2 directive implements the duty of providing API infrastructure for third-parties. [1]
There still exist banks that provide you with an RSA token. If a bank does not give you the option, how can one (sorry) "of the right segment" have business with it? You look at the service provider, you see all kinds of bad signals, you hire it anyway: this is a big part of what is destroying us!
Restraining myself to write something very strong about phone security and general user expectancy and duly expectancy (low) - let us stress again the legal side: how do you prove to a bank that, in case of theft from the account, your device was safe? People who see their money stolen then have controversies with the bank about responsibility.
BTW: PSD2 has been, in many parts, a huge nightmare. Furthermore, healthy parts of it for some reason have not been implemented.
Yes, I am a dumb person obviously. Thanks for your invaluable input. Dude.
But my use case is not to hide or remove digital exhaust.
Creating habit of limited usage is more important and realistic. Funny part is that as a side-effect I don't cary my smartphone around so much. I have separate GPS system in my cars and dumb phone for emergency.
The indian government has recently introduced new laws that give them power to dictate terms over many online platforms and broaden their surveillance powers over online social media and messenger platforms. One of the laws dealing with messenger platforms requires the platform to track shared content, especially "origin of content" (first originator) of any content that is shared through their network. (Facebook / WhatsApp has already gone to the court to challenge this, as it claims that they would need to break end-to-end encryption for this and it thus violates indian privacy laws).
Apple's iMessage platform has more than 25 million users, and thus should come under the ambit of this law. But strangely, the indian government seems to have given them an "exception" .... and now we know why.
Also, if they send perceptual hashes to your device - it's possible images could be generated back from those hashes. These aren't cryptographic hashes, so I doubt they are very good one-way functions.
Another thought - notice that they say "if too many appear". This may mean that the hashes don't store many bits of information (and would not be reversible) and that false positives are likely - ie, one image is not enough to decide you have a bad actor - you need more.
But at Apple's scale, statistically, some law-abiding users would likely get snagged with totally innocent images.
It's also just plain absurd. Hundreds of pictures of my own children at the beach in their bathing suits? No problem. Hundreds of photos of other peoples' children in bathing suits? Big problem. Of course, the algorithm is powerless to tell the difference.
Ah, I guess unsurprisingly the twitter thread is light on details. I saw "perceptual hash" which I usually interpret to mean some kind of feature-based semantic hash that is not as sensitive to small edits. Even if it isn't currently used, the door is open for it to be implemented in the future.
In cryptography creating a one-way function is not a problem. The only thing required for that is loosing information, which is trivial. For example taking the first n bytes of a file is a one-way hash function (for most files). So reversing the hashes is most definitely not a problem.
Creating collisions could be though, eg. brute forcing a normal picture by modifying random pixels by a bit into matching an illegal content’s hash is a possibility.
For context, I deeply hate the abuse of children and I've worked on a contract before that landed 12 human traffickers in custody that were smuggling sex slaves across boarders. I didn't need to know details about the victims in question, but it's understood that they're often teenagers or children.
So my initial reaction when reading this Twitter thread was "let's get these bastards" but on serious reflection I think that impulse is wrong. Unshared data shouldn't be subject to search. Once it's shared, I can make several cases for an automated scan, but a cloud backup of personal media should be kept private. Our control of our own privacy matters. Not for the slippery slope argument or for the false positive argument, but for its own sake. We shouldn't be assuming the worst of people without cause or warrant.
That said, even though I feel this way a not-small-enough part of me will be pleased if it is deployed because I want these people arrested. It's the same way I feel when terrorists get captured even if intelligence services bent or broke the rules. I can be happy at the outcome without being happy at the methods, and I can feel queasy about my own internal, conflicted feelings throughout it all.
Since you worked on an actual contract catching these sorts of people you are perhaps in a unique position to answer the question: will this sort of blanket surveillance technique in general but also in iOS specifically - actually work to help catch them?
I have direct knowledge of examples of where individuals were arrested and convicted of sharing CP online and they were identified because a previous employer I worked for used PhotoDNA analysis on all user uploaded images. So yeah, this type of thing can catch bad people. I’m still not convinced Apple doing this is a good thing, especially on private media content without a warrant, even though the technology can help catch criminals.
now im afraid, i have two young children < 5 years old.
i have occasionally took pictures of them naked with some bumps on the skin or mosquito bite and sent them to my wife over whatsapp to look at and decide do we need to send them to doctor, do i have to fear now that i will be marked as distributing CP.
It’s not just you. I have pictures of my kids playing in the bath. No genitals are in shot and it’s just kids innocently playing with bubbles. The photos aren’t even shared but they’d still get scanned by this tool.
This kind of thing isn’t even unusual either. I know my parents have pictures of myself and my siblings playing in the bath (obviously taken on film rather than digital photography) and I know friends have pictures of their kids too.
While the difference between innocent images and something explicit easy for a human to identify, I’m not sure I’d trust AI to understand that nuance.
> While the difference between innocent images and something explicit easy for a human to identify, I’m not sure I’d trust AI to understand that nuance.
In this case it’s not AI that’s understanding the nuance, it’s authorities that identify the exact pictures they want to track and then this tool lets them identify what phones/accounts have that photo (or presumably took it). If ‘AI’ is used here it is to detect if one photo contains all/part of another photo, rather than to determine if the photo is abusive or not.
Although there is a legitimate slippery slope argument to be had here.
Is there some way of verifying that the fingerprints in this database will never match sensitive documents on their way from a whistleblower to journalists, or anything else that isn't strictly illegal? How will this tech be repurposed over time once it's in place?
You seem to be suggesting that the AI will go directly from scanning your photos for incriminating fingerprints to reporting you to journalists.
I have to assume humans are involved at some point before journalists are notified. The false-positive will be cleared up and no reputations sullied (except perhaps the reputation of using AI to scan for digital fingerprints).
The other way around. If the database of fingerprints is unauditable, and especially if the database varies from country to country, then it would be very easy to add fingerprints for classified documents, or photos documenting known war crimes, or even just copyrighted stuff to close the so-called analog hole.
Documents could also be engineered to trigger false positives, making it difficult or impossible for a corporate whistleblower to photograph incriminating evidence to deliver to the authorities.
So, if the rumors are true and every iPhone will check every photo against an opaque database of perceptual fingerprints, what safeguards exist (beyond "trust us" from the database keepers) to prevent abuse of the feature to suppress evidence and control the flow of information, and which organizations or governments will have control over the contents of the database? As always, who watches the watchers?
>The false-positive will be cleared up and no reputations sullied...
This is dangerously naive. The US justice system alone will hound people on goosed up charges and try to get people to accept a plea deal and write a bogus confession. Parallel construction. Additionally if you can't audit the database (I'd bet very few people can, including your senator) how do you know a hash of something not CP wasn't inserted into the database. This entire system screams ready for govt overreach. It's worse than normal since there'll be no public evidence when it's abused.
That you even have to consider sexual interpretations of your BABY'S GENITALS is an affront to me. I have pictures of my baby completely naked, because it is, and I stress this, A BABY. They play naked all the time, it's completely normal.
Yeah that’s a fair point. The only reason I was careful was just in case those photos got leaked and taken out of context. Which is a bloody depressing thing to consider when innocently taking pictures of your own family :(
> no court is going to indict you because you have baby pictures on your phone
Maybe, maybe not. Bad luck is possible with anything involving police, prosecutors, judges, and juries. Need justification for that point of view? Just look at the number of people who were convicted and spent time in jail who truly were innocent. That doesn't even touch on the possible repercussions that can happen from just being questioned/arrested and later let go.
Don't immediately take affront, take the best possible interpretation of the parent comment. This is about automatic scanning of people's photo libraries in the context of searching for child pornography, presumably through some kind of ML. It seems to me that the concern of the commenter is that if there are photos of their child's genitals that they'll be questioned about creating child pornography, not that they're squeamish about photographing their child's genitals. This happened in 1995 in the UK: https://www.independent.co.uk/news/julia-somerville-defends-...
> While the difference between innocent images and something explicit easy for a human to identify, I’m not sure I’d trust AI to understand that nuance.
I recall a story several years ago where someone was getting the film developed at a local drugstore, and the employee reported them for CP because of bath photos. This was definitely a thing before computers with normal every day humans.
I don't have knowledge of how Apple is using this, but based on what I know about how it's used at Google this would be flagging previously reviewed images. That wouldn't include your family photos, but are generally hash-type matches of images circulating online. The images would need to depict actual abuse of a child to be CSAM.
You would only be flagged if the photos (' hashes) were added as part of some investigation right? So you only have to fear for your criminal record in the event that an actual criminal gets ahold of your indecent (in their hands!) photographs. In which eventuality you might be (relatively to not, but still leaked) glad they'd been discovered and arrested etc. assuming your good name could be cleared.
Just playing devil's advocate, my gut (and I think even considered) reaction is in alignment with surely just about the whole tech industry: it's over-reach (if they're not public images).
Look at all the recent findings that have come to light regarding ShotSpotter law enforcement abuse [1] These systems, along with other image and object recognition projects are rife for false positives, bias, and garbage-in-garbage-out. They should in no way be considered trustworthy for criminal accusations let alone arrest.
As mentioned in the twitter thread, how does image hashing & recognition tools such as PhotoDNA handle adversarial attacks?[2][3]
Just as being banned from one social media platform for bad behavior pushes people to a different social media platform, this might very well push the exactly wrong sort of people from iOS to Android.
If Android then implements something similar, they have the option to simply run different software, as Android lets you run whatever you want so long as you sign the wavier.
"You're using Android?! What do you have to hide?"
-- Apple ad in 2030, possibly
I'm the person you're responding to, and I think so? My contract was on data that wasn't surveilled, it was willingly supplied in bad faith. Fake names, etc. And there was cause / outside evidence to look into it. I can't really go into more details than that, but it wasn't for an intelligence agency. It was for another party that wanted to hand something over to the police after they found out what was happening.
I see. I was responding to you, yes. And in this case I was more curious about your opinion - based on your previous knowledge - on the viability of Apple’s technology here, rather than the specific details of your work.
In my (uninformed) opinion - this looks like more of a bad faith move on Apples part that will maybe catch some bad actors but will be a net harmful result for apple’s users and society, as expressed in the Twitter thread.
Others who responded here though also seem to think it’ll be a viable technique.
The NCMEC database that Apple is likely using to match hashes, contains countless non-CSAM pictures that are entirely legal not only in the U.S. but globally.
This should be reason enough for you to not support the idea.
From day 1, it's matching legal images and phoning home about them. Increasing the scope of scanning is barely a slippery slope, they're already beyond the stated scope of the database.
So are we talking about images which are not actually CSAM but a reasonable person would consider to be CSAM if they encountered them? Or is it just the README.txt for whatever popular bittorrent client?
A reasonable person would call it a normal picture if they encountered it in isolation. Like I said, content which is not even borderline are included.
By both a legal and moralistic standard they're not CSAM. Not even nearly.
Of course, this is a minority of the content in the database. But even 1 such image is gross neglect of stated purpose in my book.
I am reminded of an infamous couch and pool that are notorious for appearing in many adult productions... Possibly stock footage of a production room or repeating prop intended for being subsampled for so that multiple or repeated works by the same person or group can be flagged. I recall a person of interest was arrested after of all things posting a completely benign YouTube tutorial video. My thought at the time was likely a prop match to the environment or some such within the video. The method is definitely doable. Partitioned out to every consumer device with unflinching acceptance? Yeahhhh.
Remember, these databases are essentially signature DB's, and there is no guarantee that all hashes are just doing a naive match on the entire file, or that all scans performedare fundamentally the same.
This is why I reject outright the legitimacy of any Client-based CSAM scanners. In a closed source environment, it's yet another blob, therefore an arbitrary code execution vector.
I'm sorry, but in my calculus, I'm not willing to buy into that, even for CSAM. It won't stay just filesystems. It won't stay just hash matching. The fact there's so much secrecy around ways and means implies there's likely dynamicity in what they are looking for, and with the permissions and sensors on a phone that many apps already ask for, my not one inch instincts are sadly firmly engaged with no signs of letting up.
I'm totally behind the fight. I'm not an idiot though, and I know what the road to hell is paved with. Law Enforcement and anti-CSAM agencies are cut a lot of slack, and enjoy a lot of unquestioning acceptance by the populace. In my book, this warrants more scrutiny, and caution not less. The rash of inconvenient people being rather frequently called out as having CSAM found on hard drives in media with no additional context indicates the CSAM definition is being wielded in a manner that produces a great degree of political convenience.
Law enforcement care because they want to get pinged whenever someone shares imagery of interest to an investigation, irrespective of its legality.
A person who creates CSAM likely doesn't just create CSAM all the time, right? Those innocuous pictures get lumped together with illegal content and make it into the database.
The database is a mess, basically. Of course it is. It's gigantic beyond your wildest estimates.
But why am I downloading such things to begin with? Not only do these sound like very boring photos, given their providence I don’t understand this realistic pathway to get onto my phone.
This isn't CSAM or illegal, nor would it ever end up in a database. Speaking generally, content has to be sexualized or have a sexual purpose to be illegal. Simple nudity does not count inherently.
That’s not entirely true. If a police officer finds you in possession of a quantity of CP, especially of multiple different children, you’ll at least be brought in for questioning if not arrested/tried/convicted, whether the images were sexualized or not.
> nor would it ever end up in a database
That’s a bold blanket statement coming from someone who correctly argued that NCMEC’s database has issues (as in I know your previous claim is true because I’ve seen false positives for completely innocent images, both legally and morally). That said, with the amount of photos accidentally shared online (or hacked), to say that GP’s scenario can not ever end up in a database seems a bit off the mark. It’s very unlikely as sibling commenter said, but still possible.
That's why I said it's not inherently illegal. Of course, if you have a folder called "porn" that is full of naked children it modifies the context and therefore the classification. But, if it's in a folder called "Beach Holiday 2019", it's not illegal nor really morally a problem. I'm dramatically over-simplifying of course. "It depends" all the way down.
>That’s a bold blanket statement
You're right, I shouldn't have been so broad. It's possible but unlikely, especially if it's not shared on social media.
It reinforces my original point however, because I can easily see a case where there's a totally voluntary nudist family who posts to social media getting caught up in a damaging investigation because of this. If their pictures end up in the possession of unsavory people and gets lumped into NCMEC's database then it's entirely possible they get flagged dozens or hundreds of times and get referred to police. Edge case, but a family is still destroyed over it. Some wrongfully accused people have their names tarnished permanently.
This kind of policy will lead to innocent people getting dragged through the mud. For that reason alone, this is a bad idea.
> But, if it's in a folder called "Beach Holiday 2019", it's not illegal nor really morally a problem.
With all due respect, please please stop making broad blanket statements like this. I'm far from a LEO/lawyer, yet I can think of at least a dozen ways a folder named that could be illegal and/or immoral.
> This kind of policy will lead to innocent people getting dragged through the mud. For that reason alone, this is a bad idea.
I believe both you and the other poster, but I still haven't seen anyone give an example of a false positive match they've observed. Was it an actual image of a person? Were they clothed? etc.
It's very concerning if the fuzzy hash is too fuzzy, but I'm curious to know just how fuzzy it is.
Unless I'm missing something, those are just theoretical examples of how one could potentially deliberately try to find hash collisions, using a different, simpler perceptual hash function: https://twitter.com/matthew_d_green/status/14230842449522892...
So, it's theoretical, it's a different algorithm, and it's a case where someone is specifically trying to find collisions via machine learning. (Perhaps by "reversing" the hash back to something similar to the original content.)
The two above posters claim that they saw cases where there was a false positive match from the actual official CSAM hash algorithm on some benign files that happened to be on a hard drive; not something deliberately crafted to collide with any hashes.
You're not missing something, but you're not likely to get real examples because as I understand it the algorithm and database are private, the posters above are just guardedly commenting with (claimed) insider knowledge, they're not likely to want to leak examples (and not just that it's private, but with the supposed contents.. Would you really want to be the one saying 'but it isn't, look'? Would you trust someone who did, and follow such a link to see for yourself?)
To be clear, I definitely didn't want examples in terms of links to the actual content. Just a general description. Like, was a beach ball misclassified as a heinous crime, or was it perfectly legal consensual porn with adults that was misclassified, or was it something that even a human could potentially mistake for CSAM. Or something else entirely.
I understand it seems like they don't want to give examples, perhaps due to professional or legal reasons, and I can respect that. But I also think that information is very important if they're trying to argue a side of the debate.
> I understand it seems like they don't want to give examples, perhaps due to professional or legal reasons, and I can respect that.
In my case, it’s been 7 years so I’m not confident enough of my memory to give a detail description of each false positive. All I can say is that the photos that were false positive that included people were either very obviously fully clothed and doing something normal, or the photo was of something completely innocuous all together (I seem to remember an example of the latter was the Windows XP green field stock desktop wallpaper, but I’m not positive on that).
> Was it an actual image of a person? Were they clothed?
Some of the false positives were of people, others weren’t. It’s not that the hashing function itself was problematic, but that the database of hashes had hashes which weren’t of CP content, as the chance of a collision was way lower than the false positive rate (my guess is it was “data entry” type mistakes by NCMEC, but I have no proof to back up that theory). I made it a point to never personally see any content which matched against NCMEC’s database until it was deemed “safe” as I didn’t want anything to do with it (both from a disgusted perspective and also from a legal risk perspective), but I had coworkers who had to investigate every match and I felt so bad for them.
In the case of PhotoDNA, the hash is conceptually similar to an MD5 or a SHA1 hash of the file. The difference between PhotoDNA and your normal hash functions is that it’s not an exact hash of the raw bytes, but rather more like the “visual representation” of the image. When we were doing the initial implementation / rollout (I think late 2013ish), I did a bunch of testing to see how much I could vary a test image and have the hash be the same as I was curious. Resizes or crops (unless drastic) would almost always come back within the fuzziness window we were using. Overlaying some text or a basic shape (like a frame) would also often match. I then used photoshop to tweak color/contrast/white balance/brightness/etc and that’s where it started getting hit or miss.
Since this is using a db of known images. I doubt that would be an issue. I believe the idea here is that once police raid an illegal site, they collect all of the images in a db and then want to know a list of every person who had these images saved.
But it said they use a "perceptual hash" - so it's not just looking for 1:1, byte-for-byte copies of specific photos, it's doing some kind of fuzzy matching.
This has me pretty worried - once someone has been tarred with this particular brush, it sticks.
You can’t do a byte-for-byte hash on images because a slight resize or minor edit will dramatically change the hash, without really modifying the image in a meaningful way.
But image hashes are “perceptual” in the sense that the hash changes proportionally with the image. This is how reverse image searching works, and why it works so well.
Sure, I get how it works, but I feel like false positives are inevitable with this approach. That wouldn't necessarily be an issue under normal police circumstances where they have a warrant and a real person reviews things, but it feels really dangerous here. As I mentioned, any accusations along these lines have a habit of sticking, regardless of reality - indeed, irrational FUD around the Big Three (terrorism, paedophilia and organised crime) is the only reason Apple are getting a pass for this.
There is also a number of flagged pictures to reach before an individual is actually classified as a "positive" match.
It is claimed that the chance of being a false-positive for a positive match is one out of a trillion.
> Apple says this process is more privacy mindful than scanning files in the cloud as NeuralHash only searches for known and not new child abuse imagery. Apple said that there is a one in one trillion chance of a false positive.
To be fair the Twitter thread says (emphasis mine) "These tools will allow Apple to scan your iPhone photos for photos that match a specific perceptual hash, and report them to Apple servers if too many appear."
I don't know what the cutoff is, but it doesn't sound like they believe that possession of a single photo in the database is inherently illegal. That doesn't mean this is overall a good idea. It simply weakens your specific argument about occasional false positives.
The database seems legally murky. First of all, who would want to actually manually verify that there aren't any images in it that shouldn't be? If the public can even request to see it, which I doubt, would you be added to a watch list of potentially dangerous people or destroy your own reputation? Who adds images to it and where do they get those images from?
My point is that we have no way to verify the database wouldn't be abused or mistaken and a lot of that rests on the fact that CSAM is not something people want to have to encounter, ever.
Wait, so two American companies, Apple and NCMEC, are working together to install spyware on all Apple devices world-wide, with no government involvement?
Not quite, they're an NGO and the CyberTipline which it operates (the database Apple will use) was established by S.1738 [PROTECT our Children Act of 2008], and they get an appropriation from Congress to run that and work with law enforcement.
It's kind of like the PCAOB... private 501.3(c) with congressional oversight and funding.
I think the strategy is that the organization is able to do more for helping children internationally if they're not seen as part of the Justice department and the executive, which after the debacle with CBP and "kids in cages", was probably the right call.
It’s a database of hashes, not images, though, right? I would argue the hashes absolutely should be public, just as any law should be public (and yes, I am aware of some outrageously brazen exceptions to even that).
Anyone should be able to scan their own library against the database for false positives. “But predators could do this too and then delete anything that matches!” some might say, but in a society founded on the presumption of innocence, that risk is a conscious trade-off we make.
yes, it is a database of hashes but I don't know if the hashes are public information per se although I am sure copies of it are floating around. But I am referring to the images that the hashes are generated from. There is no verification of these that I know of. No one would want to do that and if you did you might be breaking the law.
The law requires companies like Google and Apple to report when they find CSAM and afiact they would generate hashes and add to this database if new material is found.
I don't know if there is any oversight in this. It's all done behind closed doors so you just have to trust that the people creating the hashes aren't doing anything nefarious or mistaken and that's a separate point apart from what others have said on here that you should be able to trust your devices you own to not be informants against you.
NCMEC is an private organization created by the U.S. Government, funded by the U.S. Government, operates with no constitutional scrutiny, operates with no oversight / accountability, could be prodded by the U.S. Government, and they tell you to "trust them".
> Unshared data shouldn't be subject to search. Once it's shared, I can make several cases for an automated scan, but a cloud backup of personal media should be kept private. Our control of our own privacy matters. Not for the slippery slope argument or for the false positive argument, but for its own sake. We shouldn't be assuming the worst of people without cause or warrant.
I have a much simpler rule: Your device should never willingly* betray you.
*With a warrant, police can attempt to plant a bug, but your device should not help them do so.
I don't think this rule makes any sense, because it just abstracts all the argument into the word "betray".
The vast majority of iPhone users won't consider it a betrayal that they can't send images of child abuse, any more than they consider it a betrayal that it doesn't come jailbroken.
The victims of child abuse depicted in these images may well have considered it a betrayal by Apple that they allowed their privacy to be so flagrantly violated on their devices up until now.
> The vast majority of iPhone users won't consider it a betrayal that they can't send images of child abuse
Probably neither would child abusers, since as soon as they send an image of child abuse, they're much more likely to be caught than if it had stayed on their phone.
> a betrayal by Apple that they allowed their privacy to be so flagrantly violated on their devices up until now.
"Their" devices? Once Apple sells an iPhone, it no longer belongs to Apple. Taking "betrayal" to mean "didn't plant backdoors on other people's computers to catch your abusers" is stretching that word far beyond reason.
I don't think you read your ancestor post carefully enough. I at least don't see any room for ambiguity.
The rule is that your (note the emphasis) device won't ever willingly betray you. There's nothing here that implicates the majority in any way. Simply, your own device should never work against you.
This actually sounds like a great rule to prevent this kind of authoritarian scope creep.
“If you want a vision of the future, imagine a boot stamping on a human face - forever.” I always think about this Orwell quote and think it’s up to us to try to fight for what is good, but we were too busy doom-scrolling on Twitter to do anything about it.
I'm not at all conflicted about this. Obviously CSAM is bad and should be stopped, but it is inevitable this new feature will become a means for governments to attack material of far more debatable 'harm'.
This scanning doesn't prevent the actual abuse and all this surveillance doesn't get to the root of the problem but can be misused by authoritarian governments.
It's a pandoras box.
You wouldn't allow the regular search of your home in real life.
If you truly want to "protect the children" you should have no issue for the police to visit and inspect your, and all of your neighbors houses. Every few days. Unannounced, of course. And if you were to resist, you MUST be a pedophile who is actively abusing children in their basement.
I'm actually more OK with unannounced inspections of my basement (within reason) than with some government agents reading through my files all the time.
Having known many victims of sexual violence and trafficking, I feel for the folks that honestly want that particular kind of crime to stop. Humans can be complete scum. Most folks in this community may think they know how low we can go, but you are likely being optimistic.
That said, law enforcement has a nasty habit of having a rather "binary" worldview. People are either cops, or uncaught criminals. ..and they wonder why they have so much trouble making non-cop friends (DISCLAIMER: I know a number of cops).
With that worldview, it can be quite easy to "blur the line" between child sex traffickers, and parking ticket violators. I remember reading a The Register article, about how anti-terrorism statutes are being abused by local town councils to do things like find zoning violations (for example, pools with no CO).
Misapplied laws can be much worse than letting some criminals go. This could easily become a nightmare, if we cede too much to AI.
And that isn't even talking about totalitarian regimes, run by people of the same ilk as child sex traffickers (only wearing Gucci, and living in palaces).
”Any proposal must be viewed as follows. Do not pay overly much attention to the benefits that might be delivered were the law in question to be properly enforced, rather one needs to consider the harm done by the improper enforcement of this particular piece of legislation, whatever it might be.”
> People are either cops, or uncaught criminals. ..and they wonder why they have so much trouble making non-cop friends (DISCLAIMER: I know a number of cops).
Ehh let's not make a habit of asserting anecdote as fact, please. Saying you know cops is like saying you know black people and that somehow it affords you some privilege others do not possess.
> is like saying you know black people and that somehow it affords you some privilege others do not possess.
Of course it does. Interacting with black people (or any race) affords you insight into their life experiences, struggles, worldview etc...
Of course sociological discourse is highly subjective but this attitude on HN that anecdotal data has no value whatsoever is silly. Do you seriously expect every fact of every people to be published in some infallible academic journal?
Based on a fairly cursory examination, the "N-word" ban is largely enforced by white people and using it is likely to get them more riled up than black folk.
So simultaneously I can imagine how black people wouldn't care and why that doesn't matter. The word isn't banned because blacks have delicate eardrums, it is banned because white people are showing respect.
I'm not. I am very unambiguously against this and I think if word gets out Apple could have a real problem.
I would like to think I am against child porn as any well-adjusted adult. That does not mean I wish for all my files to be scanned without my consent or even knowledge for compliance, for submission to who knows where matching to who knows what reporting to, well, who knows.
That's crossing a line. You are now reading my private files, interpreting them, and doing something based on that interpretation. That is surveillance.
Well, it's a lot like everything. No one wants abusers, murderers, and others out and about. But then, we can't search everyone's homes all of the time for dead bodies, or other crimes.
We would all be better off without these things happening, and anyone would want less of it to happen.
Since they are only searching for _known_ abusive content, by definition they can only detect data that has been shared, which I think is the important point here.
It's quite easy to extrapolate this and in a few steps end up in a boring dystopia.
First it's iPhone photos, then it's all iCloud files, that spills into Macs using iCloud, then it's client side reporting of local Mac files, and somewhere along all other Apple hardware I've filled my home with have received equivalent updates and are phoning home to verify that I don't have files or whatever data they can see or hear that some unknown authority has decided should be reported.
What is the utopian perspective of this which counterbalances the risks for this to be a path worth taking?
> What is the utopian perspective of this which counterbalances the risks for this to be a path worth taking?
Apple takes care of everything for you, and they have your best interests at heart. You will be safe, secure, private and seamlessly integrated with your beautiful devices, so you can more efficiently consume.
What's not to like about a world where child crime, terrorism, abuse, radical/harmful content and misinformation can be spotted in inception and at the source and effectively quarantined?
No one here has a problem with the worst criminals being taken out. The problem is the scope creep that always comes after.
In 2021 and 2020 we saw people being arrested for planning/promoting anti lockdown protests. Not for actually participating but for simply posting about it. The scope of what "harmful content" is is infinite. You might agree that police do need to take action against these people but surely you can see how the scope creeped from literal terrorists and pedophiles to edgy facebook mums and how that could move even further to simple criticisms of the government or religion.
It's difficult to say how we draw the line to make sure horrible crimes go punished while still protecting reasonable privacy and freedom. I'm guessing apples justification here is that they are not sending your photos to police but simply checking them against known bad hashes and if you are not a pedophile, there will be no matches and none of your data will have been exposed.
Thank you, it clearly shows that the German government cannot be trusted to do the right thing.
And the underlying desire for having this information will no doubt prolong the Corona restrictions longer than necessary, which is certainly not in the interest of German citizens.
In Germany police went around and even looked at contact tracing lists (on paper) in restaurants [1]. Even while politicians still stated publicly that these lists were only used (or to be used) for contact tracing.
Also the partly state sponsored luca app (check in in locations, festivals, restaurants, concerts) that is privately developed (and riddled with security holes) is already in discussion to use the data on the people to better target them for concert tickets and the like [2].
So we see this data is already in abuse by the state and also by state sponsored private entities.
I believe, that this data, once collected, will only be (ab)used further in the future. In my experience it will be as with all data caches - somebody wants to create additional value from it.
(Note that one mostly-united political party controls 89.2% of Singapore's legislature seats, and can pass any laws or amend its constitution to their liking.)
We also saw HN shadow banning entire IP CIDR blocks because they didn’t like argument against fleeting CDC guidance that was put forth or the Chinese lab origin theory in 2020.
You can’t register from these CIDR blocks. If you had an account before the comments would just end up in a black hole.
Dang can explain.
I mean, Apple isn't too far from the Mac thing you mention. Since Catalina running an executable on macOS phones home and checks for valid signatures on their servers.
> "What is the utopian perspective of this which counterbalances the risks for this to be a path worth taking?"
Basically victims of rape don't want imagery of their rape freely distributed as pornography. They consider that a violation of their rights.
It's interesting how many users in this thread are instinctively siding with the offenders in this, and not the victims. Presumably because they made it through their own childhoods without having imagery of their own abuse shared online.
This is a situation where different people's privacy is in conflict. What's infantile is claiming sole ownership of privacy advocacy while so-whating the worst privacy violation imaginable, from the victims' perspective.
That's an interesting point. However, I'm not sure victim privacy is the reason for CSAM regulations. Rather, it's reducing the creation of CSAM by discouraging its exchange. For example, suppose instead of deleting/reporting the images, Apple would detect and modify the images with Deepfake so the victim is no longer identifiable. That would protect the victim's privacy but wouldn't reduce the creation or exchange. The fact that such a proposal is ridiculous suggests that privacy isn't the reason for regulation and that reducing creation and exchange is.
There is an utterly perverse incentive to consider as well.
If the median shelf-life of abuse evidence is shortened, in that the item in question can no longer be forwarded/viewed/stored/..., what does that imply in a world where the demand remains relatively stable?
I despise the abusers for what they do, and the ecosystem they enable. But I also remember first having this argument more than ten years ago. If you, as a member of law enforcement or a child wellbeing charity, only flag the awful content but do not do anything else about it, you are - in my mind - guilty of criminal neglect. The ability to add an entry to a database is nothing more than going, "at least nobody else will see that in the future". That does NOTHING to prevent the creation of more such material, and thus implicitly endorses the ongoing abuse and crimes against children.
Every one of these images and videos is a piece of evidence. Of a horrifying crime committed against a child or children.
The two sides proposed by your argument are only logically valid opposites if you can logically/mathematically guarantee that this technology will only ever be used for detecting photos depicting blatant and obvious sex abuse. Since you cannot, the entire argument is void. I'm not siding with abusers, I simply want arbitrary spies staying the hell away from my computers.
I feel it's a little disingenuous to describe millions of innocent people being surveilled as "the offenders" because there are a handful of actual offenders among them.
There's a small number of victims, a small number of offenders (but much more than "a handful"), and hundreds of millions of other users. This change is in the direct interest of victims, direct opposition to offenders.
Most normal people probably support the measures in solidarity with group 1, HN generally doesn't.
...And direct opposition to those hundreds of millions of other users. Trying to fit this to a victims vs. offenders model is a deliberate attempt to turn those hundreds of millions of other users into uninvolved bystanders. They have been pushed out by the lack of space in the model for them and their right to not have their door kicked down based on the results of an algorithm and database they can't audit, which are susceptible to targeted adversarial attacks and authoritarian interference respectively.
If drunk driving laws were enforced by mandating a breathalyzer in every car and nobody really knew how the breathalyzer worked and also it maybe doubled as an instrument for the government to catch you doing fifteen other things then I might consider that a fair comparison.
But yes, there's a lot of drive-by engagement in this thread, thank you for at least engaging with it directly.
Having private devices randomly snooped for forbidden materials is fine, okay. So why limit this to phones?
There are kidnapped children being locked inside homes. If you don't open your doors and accept weekly full home inspections, I think it's safe to say you support offenders and hate victims if you oppose this. I mean, we're all against people kidnapping and abusing children.
There's a small number of victims, a small number of offenders (but much more than "a handful"), and hundreds of millions of other home owners. This change is in the direct interest of victims, direct opposition to offenders.
It is a violation of their rights. But we have a justice system set up which makes distributing such images knowingly (and committing such acts with the intent to distribute those images) a crime.
It's also incredibly likely this could be used to send people you don't like to prison by sending their phone innocuous images that look like wrong images to an AI.
It's also also incredibly likely this will evolve into scanning for more than just abuse photos, especially in the hands of governments around the world.
You are actually creating a false dichotomy here. There are more sides to this. And you are creating (as said a false) black and white image here.
I strongly believe that nobody wants to further victimize people by publicly showing images of their abuse.
And I believe very strongly that putting hundreds of millions of people under blanket general suspicion is a dangerous first step.
Imagine if every bank had to search all documents in safe deposit boxes to see if people had committed tax evasion (or stored other illegal things like blood diamonds obtained with child labor). That would be an equivalent in the physical world.
Now add to this, as discussed elsewhere here, that the database in question contains not only BIlder of victims, but also perfectly legal images. This can lead to people "winning" a house search because they have perfectly legal data stored in their cloud.
Furthermore, this means that a single country's understanding of the law is applied to a global user community. From a purely legal point of view, this is an interesting problem.
And yes: I would like to see effective measures to make the dissemination of such material more difficult. At the same time, however, I see it as difficult to use a tool for this purpose that is not subject to any control by the rule of law and cannot be checked if the worst comes to the worst.
Using your bank analogy for a second: banks already do report on activity to authorities who can then identify people to investigate based on patterns. I've heard that large transactions (>10k) or near-sized ones are flagged.
A great deal of skepticism is being given to the NCMEC database in these comments, which I'm surprised by as from what information I have I think this is being exaggerated. At the same time we have no idea whether Apple would even be using that database or another one that they may have created themselves.
> I've heard that large transactions (>10k) or near-sized ones are flagged.
Thi sis transmission of funds and there are laws regulating the monitoring of those.
I used bank vaults were you put things into the vaults without the bank often times knowing what is in there. If they knew, they would need to report to authorities.
So Apple doing this scan would be the bank opening all vaults, scanning the contents and reporting things to the IRS (I think this is the tax thing in the US if I am not mistaken - in Germany it would be the Finanzamt).
I'm a little bit confused here and hope maybe some of you can clear this up.
My parents took lots of photos of me as a baby/small child. Say lying naked on a blanket or a naked 2yr old me in a kiddie pool in the summer in our backyard. Those are private photos and because it was the 1970s those were just taken with a normal non-digital camera. They were OBVIOUSLY never shared with others, especially outside immediate family.
Transform that into the 2020s and today these type of pictures would be taken with your iPhone. Would they now be classified as child pornography even though they weren't meant to be shared with anyone nor were they ever shared with anyone? Just your typical proud parent photo of your toddler.
Sounds a bit like a slippery slope, but maybe I am misunderstanding the gravity here. I'm specifically highlighting private "consumption" (parent taking picture of their child who happens to be naked as 1yr olds tend to be sometimes) vs "distribution" (parent or even a nefarious actor taking picture of a child and sharing it with third parties). I 100% want to eliminate child pornography. No discussion. But how do we prevent "false positives" with this?
So far. Many websites already use NN trained to detect any nudity. It is only a matter of time before it lands on all consumer computing devices. The noose will keep on tightening because people keep debating instead of protesting.
Nudity detectors are decades old. People use them when they want to block any nudity. Making iCloud servers scan for nudity would have been simpler than making this new system. And pornography is easier to access than ever for most people.
As with all horribly-ill-defined laws, it depends how the judge is feeling that day and their interpretation of the accused's intent. If the case can be made that the images arouse inappropriate gratification, they can be deemed illegal.
If that sounds absurd - most laws are like that. For better or worse, there's a human who interprets the law, not a computer. It's unfortunate Apple is choosing to elect a computer as the judge here, for exactly concerns like yours.
I believe there is a large database of known child pornography.
Unless someone has been distributing photos of your kids as child porn (which would probably be good to know) it's unlikely any of your photos will match the hashes of the photos in that database.
I'm not sure that's how it works, but that's what I've gathered from the other comments on this post.
I was under the impression that one of the reasons why these tools aren’t available for public download is because the hashes and system can be used to design defeat mechanisms? Doesn’t this mean that someone who has an image and a jail broken device can just watch the system, identify how the photo is detected, and modify it so that it doesn’t trip the filter?
PhotoDNA and systems like it are really interesting, but it seems like clientside scanning is a dangerous decision, not just from the privacy perspective. It seems like giving a CSAM detector and hashes to people is a really risky idea, even if it’s perfect and it does what it says it does without violating privacy.
If the algorithm and the blocklists leaked, then not only it would be possible to develop tools that reliably modify CSAM to avoid detection, but also generate new innocent-looking images that are caught by the filter. That could be used to overwhelm law enforcement with false positives and also weaponized for SWAT-ing.
Fortunately, it seems that matching is split between client-side and server-side, so extraction of the database from the device will not easily enable generation of matching images.
It is lovely when your "own" device is working against you to catch if you are in possession of illegal numbers https://en.wikipedia.org/wiki/Illegal_number.
And surely we can trust Apple that it will only be used for this kind of content instead of for example government leaks.
I just assumed the entire FAANG group scanned user content for CP already. I mean, EU recently passed a vote that extended the permission for companies to be able to scan and report user content for this purpose without it being considered a privacy violation [1] (first introduced in 2020). And I recall MS[2]/Google[3] being open about this practice way in the past.
Personally I somehow doubt that MS/Google weren't scanning private content (aka not shared) for this type of material. But can't have transparency with these behemoths.
how the fuck am i supposed to know if that image i downloaded from some random subreddit is of a girl who is 17.98 years old? how long until we just use a NN to identify images of children automatically? she looks pretty young so i guess you will get disemboweled alive in prison? what is stopping someone from planting an image on your phone or a physical picture somewhere on your property? im so tired of this fucking dogma around child porn. you can always identify the presence of dogma by the accompanying vacuum of logic that follows in its wake. a teenage girl can go to jail for distributing pictures that she took of herself. do i even need to say more?
Well this really debunks my common phrase “Apple is a Privacy company, not a Security company”
I can’t say I’m surprised they are implementing this (if true), under the radar. I can’t imagine a correct way or platform for Apple to share this rollout publicly. I’m sure nothing will come of this, press will ignore the story, and we all go back to our iPhones
This is really a pointless comment - yes all companies are ultimately there to make money, but that does not mean always blindly doing whatever makes the most money in the short term. Clearly apple sees value in marketing themselves as privacy friendly.
Apple only claim to care about privacy because they couldn't manage to compete with Google on running ad-serving cloud service. Since they couldn't sell ads in meaningful number, they figured they might as well brag about it.
But Apple iCloud for ex doesn't intrude on your privacy any more or less than Google Photos.
I really have to wonder why Apple chose to do this.
As far as I know, this kind of scanning is not legally mandated. So, either they think that this will truly make the world a better place and are doing it out of some sense of moral responsibility, or they've been pressured into it as part of a sweetheart deal on E2E ("we won't push for crypto backdoors if you'll just scan your users' phones for us"). Either way it doesn't thrill me as a customer that my device is wasting CPU cycles and battery life under the presumption that I might possess data my current jurisdiction deems illegal.
For all the acclaim privacy-forward measures like GDPR get here, I'm surprised there isn't more outright repudiation of this frankly Orwellian situation.
If you like this, I have some other innovations that you may be interested in:
* A car that automatically pulls over when a police cruiser attempts to intercept you
* A front door that unlocks when a cop knocks
* A camera that uses AI to detect and prevent the photography of minors, police, and critical infrastructure
* A Smart TV that counts the number of people in your living room to ensure you aren't performing an unauthorized public broadcast of copyrighted content
Surely, at least one of those sounds ridiculous to you. As well-intentioned as this scanning may be, it violates a core principle of privacy and human autonomy. Your own device should not betray you. As technologists, just because we can do something doesn't mean we should.
> The problem with allowing this is that you’re paving the way for future tyrants to use it against us.
It's funny how everybody talk about the future. This is happenning now. Remember how a certain german guy took the power some 90 years ago ? He was elected.
Not your point, but: Technically he was Austrian by birth, stateless sind since 1925 and tried at least 7 times to get German citizenship before elected in 1932.
I trust we live in a different world today, but it is concerning to read about mandated vaccines to get a job and mandated contact-tracing in countries such as Germany.
Apps weren't mandated, restaurants and such are required to request you check in, but people didn't always fill them or otherwise filled them with junk (especially after they were abused)
People nowadays voluntarily carry tracking devices. This will not stop getting worse until that behavior is denormalized.
The power to be gained from abusing it is beyond irresistable. Expecting those in power to not abuse it is like expecting a heroin junkie to be a good pharmacist.
> People nowadays voluntarily carry tracking devices
on the strict premise that tracking is exceptional fair use from a law enforcement agency. Do not mix up the voluntary accolites of Zuck and people who want tools. (The use of 'tools' in the sentence is an originally unintended pun. I will keep the term 'accolites', though tempted to replace it for the pun.)
By enforcing the copyright laws that exist already to protect content creators from being counterfeited and ripped off by illegal scammers and bootleggers. Are you saying you don't agree with the concept of copyrights?
How does showing Lion King at your kids birthday party qualify for any of that?
I know I know, Disneys inability to collect money for every pair of eyes watching their movies during VHS times literally made it one of the poorest companies on the world. /s
Lots of people don't support copyright. It's a concept that should be abolished in it's current form. Right to attribution is OK. Ability to restrict other forms of use is not OK.
Does that mean if someone spends time and effort writing a book or painting a picture that I can resell it for less than them without their permission?
Why would any fan of someones work buy a knock-off copy? We can safely assume they are a fan, otherwise why would even buy it?
Also, painting is a physical object. It's a one of a kind.
EDIT: your objection and many many other questions like that are nicely argued against here: https://www.youtube.com/watch?v=mhBpI13dxkI
(The Surprising History of Copyright talk by Karl Fogel)
Yes. Definitely. Absolute staggering majority of people writing books don't sell even one copy. And my bet is - they know what are the chances.
EDIT1:
More than that. People don't live of royalties. Publishers do. Can people create without being paid upfront by publishers? Obviously yes. They can be paid upfront in kickstarter-like arrangement. They can be paid via donations later. They can even be paid via "retroactive funding of public goods" [0]. Or they can be never paid - just as they are today.
Also, imagine how much good could come of long-standing IP properties owned by Disney if not for them sitting on it. Vide disaster that is current management of Star Wars.
Do you know how many stories cannot be published because people sit on the IP? There are unpublished "movies" that only exist to maintain exclusive licensing deals that outright prohibit studios from sitting on the IP, letting the original authors work rot until it is forgotten.
Or we can talk about the mess you get with too many groups involved? Do you know who has the rights to Westwoods Dune? Westwood itself had a limited time license for games based on the movie Dune, which also had a license based on the Dune books. So you would have to deal with three license owners to make a Game involving the Ordos faction. Have fun convincing EA, whoever owns the movie rights and the original authors family that you can make a new game worth their signature on a new licensing agreement (you could probably manage if you have a small country to sell).
I live in a major city and I’m a fan of none of these things. I’m only one data point, but yours is a sweeping and inaccurate generalization that “cities are frighteningly unsafe”.
Maybe one trusts Apple more than <insert politician>, but they cannot so easily elect away Apple.
Apple has generally locked things down successfully.
There was concern about phone's being grabbed - street robberies. Regardless of whether you believe their were sweeping generalizations - apple ended up creating more power for themselves with their activation lock system. If they don't want to let you sell your phone to someone - they can block use of your phone. But folks trust them to operate the system reasonably, and so far so good there.
They have locked down their app store very tightly for a variety of reasons including supposedly for security. Users have accepted that.
Apple's competitors (google photos etc) generally are directly scannable in the cloud by google et al. Facebook and others routinely scan users photos. Youtube scans their videos etc. My guess is apple will explain why they are doing it and users are going to be happy.
And yes, users are linking things like ring doorbells and home security video cameras together or registering them so that the police state can use them.
The technology is there, now we only need the motivation.
If politicians decides that they want it now, they can simply orchestrate a media campaign and have it. The next time an "outrageous" act of crime is conducted, they can make sure that it stays at the media attention and be portrayed as "If we don't act now very bed things will happen", then slide in their solution.
* Cars can automatically pull over by installing a cheap cut fuel cut switch that can be activated by short range radio. In many places people are used to add devices for toll collection anyway. People are also used to pay for regulatory inspections on their vehicles.
* For the old cars, simply connect an NFC reader that unlocks the central lock system of a car by a master key. For the new cars, simply make manufacturers add a police master key.
* Commercial drones are already stopping their users from flying over forbidden areas, simply extend that to smartphones. Smartphones have enough power and sensors to identify forbidden locations and persons. Add NFC kill switch, meaning the police can send a signal to lock down cameras.
* There were reports of Smart TVs that record all the time, simply mandate it to all manufacturers and enforce automated inspection of the recordings.
Uneven application of the law seems crucial to keep the system functioning and technology can erode that. Many simple laws, if enforced thoroughly and without prejudice, would become absolutely draconian. It is not even possible for a human to know all the laws we are meant to follow at all times, yet computers can.
Apple devices already betray their "owners", and they've been doing it for a long time.
You can't repair them.
You can't run your own software.
You can't use a better, more compliant web browser.
Businesses have to pay a 30% tax.
Businesses are forced to use login with Apple and forfeit a customer relationship.
Businesses have to dance to appease Apple. Their software gets banned, randomly flagged, or unapproved for deployment, sometimes completely on a whim.
Soon, more iDevices and Apple Pay will lead to further entrenchment. Just like in the movie Demolition Man, everything will eventually be Apple. Your car, your movies, your music, your elected officials.
The slippery slope argument is that the use of this method on private files, i.e. not shared with others except for the service provider can legitimise the expansion of such scamming scopes.
While this argument can and have indeed happened in other instances, this is akin to saying that we should not give anyone any powers to do anything because it is a slippery slope that they can use it to do bad things.
What then sets out the difference between what a slippery slope and a non-slippery one is? Checks and balances and the history of USA have shown that this is indeed what can reign in the worst instincts of any single entity. History of course have also shown when these failed and these should serve not as a reason to reject the idea of checks and balances but as acknowledging it's imperfection and think of ways to mitigate it.
Sure. And as I mentioned, there will be screw ups along the way. As with any new capability/tech be it nuclear power or recombinant DNA or ability to locate CP, there can be legitimate uses that we can rally behind and ways for them to be abused.
Checks and balances are never a done deal. If we reject checks and balances and as a result reject new tech because of abuse potential, how then should we as a civilisation advance?
756 comments
[ 2.9 ms ] story [ 352 ms ] thread> Regardless of what Apple’s long term plans are, they’ve sent a very clear signal. In their (very influential) opinion, it is safe to build systems that scan users’ phones for prohibited content.
> That’s the message they’re sending to governments, competing services, China, you.
The message could equally well be ‘We won’t become an easy political target by ignoring a problem something most people care about like child porn, but we are going to build a point solution to that problem, so the public doesn’t force us to bow government surveillance requests.’
It’s easy to nod along with an anti-Apple slogan, but we need to consider what would happen if they didn’t do this.
Public opinion is not in favor of giving safe harbor to pedophiles and child pornographers, and I can’t see why anyone would even want Apple to fight that battle.
What we do know is that it is a proprietary solution using a proprietary hash which only applies to Apple products and that Apple has always presented themselves as a family friendly company that doesn’t support criminal use cases.
Everything points to this being something Apple thinks needs to be solved before the public asks why they haven’t.
If it was a government demand, we’d presumably see Google responding to it too.
You still haven’t explained why not working to deter pedophiles and child pornographers, is an important battle for them to fight.
They might think it’s a problem they want to solve on their own terms, and that would seem to be what they have done here.
Not solving a problem people care about just because the government also cares about it seems illogical. I think a lot of people like the idea of corporations taking responsibility for the social problems they cause without needing to be forced to do so by the government.
Engineered collisions seem unlikely too. Not impossible. Unless there is a straight up cryptographic defect in the hash algorithm, it seems hard to see how engineered collisions could be made to happen at any scale.
This isn't some web tech startup where a mistake means someones tshirt got sent to the wrong address. Peoples lives will quite literally be ruined over mistakes here.
Is it a once in a million issue? The collision rate matters. It could easily be much higher and then it wouldn’t matter that it was being used at Apple’s scale.
What is the actual attack you are imagining?
Like gay porn, 'save Khashoggi' meme, or a photo from documentary about missing Uighurs.
It's hard to imagine how this could be misused, right?
E.g. how the hashes are computed, where they come from, and what happens when a positive match is detected.
Until we have a clear understanding of these things, the rest is just speculation.
Money citizens of those countries want to give him.
So... all your hashes will be uploaded to the cloud? How do you prevent them from scanning other stuff (memes, leaked documents, trump-fights-cnn-gif,... to profile the users)?
Or will a huge hash database of child porn hashes be downloaded to the phone?
Honestly, i think it's one more abuse of terrorism/child porn to take away privacy of people, and mark all oposing the law as terrorists/pedos.
...also, as in the thread from the original url, making false positives and spreading them around (think 4chan mass e-mailing stuff) might cause a lot of problems too.
That isn't how I interpret "client-side".
The privacy implications are far more subtle.
It always starts with child porn, and in a few years the offline Notes app will be phoning home if you write speech criticising the government in China.
This technology inevitably leads to the sueveillance, suppression and murder of activists and journalists. It always starts with protecting the kids or terrorism.
Perceptual hashes like what Apple is using are already used in WeChat to detect memes that critique the CCP.
What happens on local end user devices must be off limits. It is unacceptable that Apple is actively implementing machine learning systems that surveil and snitch on local content.
I wouldn't hold your breath waiting for those regulations to become law in, say, Chine or Turkey or Saudi Arabia. I'd bet even Israel won't pass them, surely NSO have enough political lobbying swing (and probably also suitable blackmail material on sitting politicians).
Think of GDPR, which is essentially each member of the EU saying in unison, "your product/service must comply with these data protection laws, or you can't legally do business with any of our citizens".
Come to think of it, I wonder if this Apple thing would even fly under GDPR?
Possibly? I’m not a lawyer, but if this is about compliance with a legal obligation, and they’re under that category of pressure? I think GDPR would allow that?
Certainly seems more likely allowed than the stuff Facebook complained Apple was preventing them from doing.
I wholeheartedly disagree. A world wide goverment would be catastrophic for whistleblowing. Atleast as a whistleblower you can live somewhat safely in a country opposing your own. With a one world government you would have nowhere to run.
And I wouldn't expect them to protect citizen's interests any better than current governments do. Contrary, I think this lack of balance in the world would embolden them further.
The OP still addresses the inaccurate statement (presented in the form of a question for plausible deniability).
A totalitarian autocracy like China does not need this technology to search for wrongspeech, sadly. You are of course aware that all Chinese iCloud users get their data stored in a special set of datacenters that Apple actually doesn't control.
https://en.wikipedia.org/wiki/Birthday_attack
http://www.wolframalpha.com/input/?i=2%5E64%20%2F%20%288e9%2...
It's a perceptual hash.
State-of-the-art for this field is deep learning, and a /huge/ problem with the DL approach is that you can generate adversarial examples. So for example, a picture of a teacup that is identified by /most/ networks as a dog. It's particularly damning, because it seems like you don't have to do this for particular deep networks, they get tricked the same way, so to speak.
Seriously, folks, we shouldn't celebrate Apple's death grip over their platform. It's dangerous for all of us. The more of you that use it, the more it creates a sort of "anti-herd immunity" towards totalitarian control.
Apple talks "privacy", but jfc they're nothing of the sort. Apple gives zero shits about your privacy. They're staking more ground against Facebook and Google, trying to take their beachheads. You're just a pawn in the game for long term control.
Apple cares just as much for your privacy as they do your "freedom" to run your own (un-taxed) software or repair your devices (for cheaper).
And after Tim Cook is replaced with a new regime, you'll be powerless to stop the further erosion of your liberties. It'll be too late.
Stop. Using. Apple.
But is there a realistically better alternative? Pinephone with a personally audited Linux distro? A jailbroken Android device with a non-stock firmware that you built yourself? A homebuilt RaspberryPi based device? A paper notepad and a film camera and an out of print street map?
My hope is that those of you that share my viewpoint will call your legislators and demand regulations or a break up. There are forces of good within the DOJ that are putting together an antitrust case against Apple, and the more of us that lend our voices, the louder and more compelling the argument.
The DOJ is really the last lever we have, and that's pretty good measure for the power Apple wields.
no idea what the situation is now, but i wouldn't consider a phone that doesn't get those patches (project mainline) on time to be a serious option.
Gnu/linux phones have nonexistent security, other than being niche (so security by obscurity at most). And also, they are not yet usable as a daily driver for me personally, at least.
https://puri.sm/posts/defending-against-spyware-like-pegasus...
https://source.puri.sm/Librem5/community-wiki/-/wikis/Freque...
Whether or not I am allowed to check that my entrance has no locks whatsoever doesn’t make it harder to open it. And the reverse, even if I don’t know the details of the lock in my door, it will not let others pass through.
You absolutely can not make that assertion without being able to verify the lock.
For a fact we know that ios has strong sandboxing, secure bootchain and apps are revokably verified.
Not sure about that. No source code. Also, Pegasus.
Can you expand on this point a little bit?
Basically Micay is a legitimate security researcher who created the project and it was later hijacked by the company funding some of it. That company since then try to badmouth Micay at any place they find and is doing shady things on top of the still open source code base. Micay was so professional to destroy the verification key at the time of the forking.
Even if you don't personally audit it, you still benefit from other people doing it. Especially if the software is reproducible (and many packages are).
Realistically now we have Tizen and Jolla OS, which had backings from Samsung but nobody gave two damn about it.
I bet even if any of these vanilla mobile OS gets big enough they’ll get bought by the 3 giants and suffocated to death just like how Microsoft sniped Nokia.
Were there any recent examples where they failed in those?
They'll happily do evil shit, and execute it poorly. Samsung are _way_ more likely to leak the unnecessarily and possibly illegally collected personal data they hoover up than Google are.
... without any technical guarantee or auditability that any of the hashes they're alerting on are actually of child porn.
How much would you bet against law enforcement to abuse their ability to use this, and add hashes to find out who's got anti government memes or police committing murder images on their phones?
And that's just in "there land of the free", how much worst will the abuse of this be in countries who, say, bonesaw journalists to pieces while they are alive?
This will end badly for humanity.
Now the first few images might be the model completely clothed and not even be porn, maybe there's a picture of her lounging around a pool, then another picture of the pool itself. Still its part of a set of pictures that is known porn.
Heck most porn starts off with actors being clothed (so I hear lol).
So I want to ask what happens if you have a photo that is falsely identified as one in question and then an automated mechanism flags you and reports you to the FBI without you even knowing. Can they access your phone at that point to investigate? Would they come to your office and ask about it? Would that be enough evidence to request a wiretap or warrant? Would they alert your neighbors? How do you clear your name after that happens?
edits: yes, the hash database is downloaded to the phone and matches are checked on your phone.
Another point is that these photos used to generate the fingerprints are really legal black holes that the public is not allowed to inspect I assume. No one wants to be involved in looking at them, no one wants to be known as someone who looks at them. It could even be legally dangerous requesting to find out what has been put into the image database I assume.
That seems like a realistic attack. Since the hash list is public (has to be for client side scanning), you could likely set your computer to grind out a matching image hash but of some meme which you then distribute.
There have been a number of cases where people have found ways to trick CV programs in to seeing something that no human would ever see. If you were sufficiently malicious I imagine it would be possible to do with this system as well.
https://news.ycombinator.com/item?id=28097683
A malware will definitely be created, almost immediately, that will download files that are intentionally made to match CP - either for the purposes of extortion or just watching the world burn.
I'm usually sticking my neck out in defence of more government access to private media than most on HN because of the need to stop CP, but this plan is so naive, and so incredibly irresponsible, that I can't see how anyone with any idea of how easy it would be to manipulate would ever stand behind it.
https://signal.org/blog/cellebrite-vulnerabilities/
Even if you do a content aware hash where you break the file into chunks and hash each chunk, you still wouldn’t be able to magically recreate the hash of a CP file without also producing part of the CP.
It's the weights from the middle of a neural network that they're calling a "hash" because it encodes and generates an image it has classified as bad. Experts have trouble rationalizing about what weights mean in a neural network. This is going to end badly.
If this was a hash then it would be as the parent describes, this is at best a very fuzzy match on an image to take into account blurring/flipping/colour shifting.
It's vastly more likely that innocent people will be implicated for fuzzy matches on innocuous photos of their own children in shorts/swimming clothes than it is to catch abusers.
The other thing is, when you have nothing to hide you won't take efforts to hide it - meaning you'll upload all of your (completely normal) photos to iCloud without thinking about it again.
The monsters making these images know what they're doing is wrong, so they'll likely take efforts to scramble or further encrypt the data before uploading.
tldr; it's far likelier that this dragnet will only even apply to innocent people, than it is to catch predators.
All this said, I'm still in support of Apple taking steps in this direction, but it needs far more protections put in place to prevent false positives than this solution allows. A single false accusation by this system, even if retracted later and rectified, would destroy an entire family's lives (and could well cause suicides).
Look what happened in the Post Office case in the UK as an example of how these things can go wrong - scores of people went to prison for years for crimes they didn't commit because of a simple software bug.
The ones that make national news from big busts do, because the ones that don't get caught much sooner and only make local news, because Google and other parties are have automatic CSAM identification online already (server side, not client side, AFAIK), and are sending hits to Homeland Security.
No, it'll be done on-device.
> How do you prevent them from scanning other stuff (memes, leaked documents, trump-fights-cnn-gif
Nothing. Given that it's only done on their closed-source messaging platform though, nothing is preventing them from reading your messages already.
But yes, it could potentially be used to detect images that the current political party doesn't like.
Rather than complain about it, I am interested in what alternative solutions exist, or how concerns regarding privacy and abuse of this system could be mitigated.
Sacrificing the privacy of the many to help catch a relatively small amount of (admittedly some of the worst possible) criminals, while simultaneously enabling yet more effective surveillance and oppression by those inclined governments (of which there are plenty) is a pretty terrible idea.
Eliminating the 4th amendment or mandating clear walls sure would make the cops' job easier. But no one thinks that's even a remotely good idea.
There is literature about how obedience enables totalitarianism. Tons of it. Should be part of any schools curriculum in my opinion.
Yet.
Remember SOPA that the internet killed?
stop trading freedom for security.
All?
iOS downloads CSAM scanning code from Apple and runs it locally—hence, client-side tool.
Treat your phones as an enemy. Use real computers with VPN and software like Little Snitch when online. Use cameras for photography and video.
The benefits of this approach are immense. I have long attention span. I don't have fear of missing out.
If governments wan't the future to be painted by tracing and surveillance mediated towards people trough big tech - lets make it mandatory by law. And since big tech will reap benefits from the big data they must provide phones for free. :)
I'm assuming your "real computer" is a mac (since little snitch is mac only). What makes you think apple won't do the same for macos? Also, while you have greater control with a "real computer", you also have less privacy from the apps themselves, since they're unsandboxed and have full access to your system.
On a serious note: If you want privacy/security as a semi-professional use: No mobile phone. https://www.qubes-os.org/ with https://www.qubes-os.org/doc/certified-hardware/
Sounds like this is more about "checking your phone less", than "improving security/privacy". This is evident elsewhere in your advice. eg. "phones as an enemy", but no advice about killswitches for microphone? Or some sort of mitigation against GPS/mobile networking tracking?
But I feel there's a valid argument to be made that if your adversary is the sort of people who'd be feeding Apple image hashes to find people, you're probably be wise to carry a regular phone on which you do boring norm-core sorts of things.
A phone you use to take pictures of cats and pay your rent using banking apps and call your parents - while not using it to communicate with your dealer or your anarchist collective or your friendly investigative journalist.
So, it's rather hard to avoid banking apps.
Also, the PSD2 directive implements the duty of providing API infrastructure for third-parties. [1]
https://www.ecb.europa.eu/paym/intro/mip-online/2018/html/18...
Restraining myself to write something very strong about phone security and general user expectancy and duly expectancy (low) - let us stress again the legal side: how do you prove to a bank that, in case of theft from the account, your device was safe? People who see their money stolen then have controversies with the bank about responsibility.
BTW: PSD2 has been, in many parts, a huge nightmare. Furthermore, healthy parts of it for some reason have not been implemented.
Dude, you carry it around with you, with its radio enabled. You're just fooling yourself.
The indian government has recently introduced new laws that give them power to dictate terms over many online platforms and broaden their surveillance powers over online social media and messenger platforms. One of the laws dealing with messenger platforms requires the platform to track shared content, especially "origin of content" (first originator) of any content that is shared through their network. (Facebook / WhatsApp has already gone to the court to challenge this, as it claims that they would need to break end-to-end encryption for this and it thus violates indian privacy laws).
Apple's iMessage platform has more than 25 million users, and thus should come under the ambit of this law. But strangely, the indian government seems to have given them an "exception" .... and now we know why.
Another thought - notice that they say "if too many appear". This may mean that the hashes don't store many bits of information (and would not be reversible) and that false positives are likely - ie, one image is not enough to decide you have a bad actor - you need more.
But at Apple's scale, statistically, some law-abiding users would likely get snagged with totally innocent images.
Just a bad idea all around.
Creating collisions could be though, eg. brute forcing a normal picture by modifying random pixels by a bit into matching an illegal content’s hash is a possibility.
For context, I deeply hate the abuse of children and I've worked on a contract before that landed 12 human traffickers in custody that were smuggling sex slaves across boarders. I didn't need to know details about the victims in question, but it's understood that they're often teenagers or children.
So my initial reaction when reading this Twitter thread was "let's get these bastards" but on serious reflection I think that impulse is wrong. Unshared data shouldn't be subject to search. Once it's shared, I can make several cases for an automated scan, but a cloud backup of personal media should be kept private. Our control of our own privacy matters. Not for the slippery slope argument or for the false positive argument, but for its own sake. We shouldn't be assuming the worst of people without cause or warrant.
That said, even though I feel this way a not-small-enough part of me will be pleased if it is deployed because I want these people arrested. It's the same way I feel when terrorists get captured even if intelligence services bent or broke the rules. I can be happy at the outcome without being happy at the methods, and I can feel queasy about my own internal, conflicted feelings throughout it all.
This kind of thing isn’t even unusual either. I know my parents have pictures of myself and my siblings playing in the bath (obviously taken on film rather than digital photography) and I know friends have pictures of their kids too.
While the difference between innocent images and something explicit easy for a human to identify, I’m not sure I’d trust AI to understand that nuance.
In this case it’s not AI that’s understanding the nuance, it’s authorities that identify the exact pictures they want to track and then this tool lets them identify what phones/accounts have that photo (or presumably took it). If ‘AI’ is used here it is to detect if one photo contains all/part of another photo, rather than to determine if the photo is abusive or not.
Although there is a legitimate slippery slope argument to be had here.
I have to assume humans are involved at some point before journalists are notified. The false-positive will be cleared up and no reputations sullied (except perhaps the reputation of using AI to scan for digital fingerprints).
Documents could also be engineered to trigger false positives, making it difficult or impossible for a corporate whistleblower to photograph incriminating evidence to deliver to the authorities.
So, if the rumors are true and every iPhone will check every photo against an opaque database of perceptual fingerprints, what safeguards exist (beyond "trust us" from the database keepers) to prevent abuse of the feature to suppress evidence and control the flow of information, and which organizations or governments will have control over the contents of the database? As always, who watches the watchers?
This is dangerously naive. The US justice system alone will hound people on goosed up charges and try to get people to accept a plea deal and write a bogus confession. Parallel construction. Additionally if you can't audit the database (I'd bet very few people can, including your senator) how do you know a hash of something not CP wasn't inserted into the database. This entire system screams ready for govt overreach. It's worse than normal since there'll be no public evidence when it's abused.
That you even have to consider sexual interpretations of your BABY'S GENITALS is an affront to me. I have pictures of my baby completely naked, because it is, and I stress this, A BABY. They play naked all the time, it's completely normal.
As that seems unlikely, I guess CSAM just uses a constantly updated database of known hashes for matching.
I say let children be free, no court is going to indict you because you have baby pictures on your phone.
Maybe, maybe not. Bad luck is possible with anything involving police, prosecutors, judges, and juries. Need justification for that point of view? Just look at the number of people who were convicted and spent time in jail who truly were innocent. That doesn't even touch on the possible repercussions that can happen from just being questioned/arrested and later let go.
I recall a story several years ago where someone was getting the film developed at a local drugstore, and the employee reported them for CP because of bath photos. This was definitely a thing before computers with normal every day humans.
Your intent when producing the image was irrelevant.
Just playing devil's advocate, my gut (and I think even considered) reaction is in alignment with surely just about the whole tech industry: it's over-reach (if they're not public images).
After all the courts he ended up with 13 years in prison, where he is likely going to die.
As mentioned in the twitter thread, how does image hashing & recognition tools such as PhotoDNA handle adversarial attacks?[2][3]
[1] https://news.ycombinator.com/item?id=27959755
[2] https://news.mit.edu/2019/why-did-my-classifier-mistake-turt...
[3] https://towardsdatascience.com/black-box-attacks-on-perceptu...
If Android then implements something similar, they have the option to simply run different software, as Android lets you run whatever you want so long as you sign the wavier.
"You're using Android?! What do you have to hide?" -- Apple ad in 2030, possibly
In my (uninformed) opinion - this looks like more of a bad faith move on Apples part that will maybe catch some bad actors but will be a net harmful result for apple’s users and society, as expressed in the Twitter thread.
Others who responded here though also seem to think it’ll be a viable technique.
This should be reason enough for you to not support the idea.
From day 1, it's matching legal images and phoning home about them. Increasing the scope of scanning is barely a slippery slope, they're already beyond the stated scope of the database.
The thing I can publicly say is that the database is not strictly for illegal or even borderline imagery.
NCMEC try to keep the contents, processes and access to the database under wraps for obvious reasons.
By both a legal and moralistic standard they're not CSAM. Not even nearly.
Of course, this is a minority of the content in the database. But even 1 such image is gross neglect of stated purpose in my book.
Why would I have any content in my phone that would be in that database?
- A kitchen with nobody in frame.
- A couch with nobody in frame.
- Outdoor scenery with nobody in frame.
- A bathroom with nobody in frame.
Is it hard to believe you wouldn't download something like this without knowing where it came from?
I'm not talking about borderline stuff. I'm talking about content that has not even a hint of pornography or illegality.
Remember, these databases are essentially signature DB's, and there is no guarantee that all hashes are just doing a naive match on the entire file, or that all scans performedare fundamentally the same.
This is why I reject outright the legitimacy of any Client-based CSAM scanners. In a closed source environment, it's yet another blob, therefore an arbitrary code execution vector.
I'm sorry, but in my calculus, I'm not willing to buy into that, even for CSAM. It won't stay just filesystems. It won't stay just hash matching. The fact there's so much secrecy around ways and means implies there's likely dynamicity in what they are looking for, and with the permissions and sensors on a phone that many apps already ask for, my not one inch instincts are sadly firmly engaged with no signs of letting up.
I'm totally behind the fight. I'm not an idiot though, and I know what the road to hell is paved with. Law Enforcement and anti-CSAM agencies are cut a lot of slack, and enjoy a lot of unquestioning acceptance by the populace. In my book, this warrants more scrutiny, and caution not less. The rash of inconvenient people being rather frequently called out as having CSAM found on hard drives in media with no additional context indicates the CSAM definition is being wielded in a manner that produces a great degree of political convenience.
Again, more scrutiny, not less.
In principle, the OS environment could be made independently auditable - keeping undeleteable signed logs.
A person who creates CSAM likely doesn't just create CSAM all the time, right? Those innocuous pictures get lumped together with illegal content and make it into the database.
The database is a mess, basically. Of course it is. It's gigantic beyond your wildest estimates.
That’s not entirely true. If a police officer finds you in possession of a quantity of CP, especially of multiple different children, you’ll at least be brought in for questioning if not arrested/tried/convicted, whether the images were sexualized or not.
> nor would it ever end up in a database
That’s a bold blanket statement coming from someone who correctly argued that NCMEC’s database has issues (as in I know your previous claim is true because I’ve seen false positives for completely innocent images, both legally and morally). That said, with the amount of photos accidentally shared online (or hacked), to say that GP’s scenario can not ever end up in a database seems a bit off the mark. It’s very unlikely as sibling commenter said, but still possible.
That's why I said it's not inherently illegal. Of course, if you have a folder called "porn" that is full of naked children it modifies the context and therefore the classification. But, if it's in a folder called "Beach Holiday 2019", it's not illegal nor really morally a problem. I'm dramatically over-simplifying of course. "It depends" all the way down.
>That’s a bold blanket statement
You're right, I shouldn't have been so broad. It's possible but unlikely, especially if it's not shared on social media.
It reinforces my original point however, because I can easily see a case where there's a totally voluntary nudist family who posts to social media getting caught up in a damaging investigation because of this. If their pictures end up in the possession of unsavory people and gets lumped into NCMEC's database then it's entirely possible they get flagged dozens or hundreds of times and get referred to police. Edge case, but a family is still destroyed over it. Some wrongfully accused people have their names tarnished permanently.
This kind of policy will lead to innocent people getting dragged through the mud. For that reason alone, this is a bad idea.
With all due respect, please please stop making broad blanket statements like this. I'm far from a LEO/lawyer, yet I can think of at least a dozen ways a folder named that could be illegal and/or immoral.
> This kind of policy will lead to innocent people getting dragged through the mud. For that reason alone, this is a bad idea.
Completely agree.
It's very concerning if the fuzzy hash is too fuzzy, but I'm curious to know just how fuzzy it is.
So, it's theoretical, it's a different algorithm, and it's a case where someone is specifically trying to find collisions via machine learning. (Perhaps by "reversing" the hash back to something similar to the original content.)
The two above posters claim that they saw cases where there was a false positive match from the actual official CSAM hash algorithm on some benign files that happened to be on a hard drive; not something deliberately crafted to collide with any hashes.
I understand it seems like they don't want to give examples, perhaps due to professional or legal reasons, and I can respect that. But I also think that information is very important if they're trying to argue a side of the debate.
I gave that above in a sibling thread.
> I understand it seems like they don't want to give examples, perhaps due to professional or legal reasons, and I can respect that.
In my case, it’s been 7 years so I’m not confident enough of my memory to give a detail description of each false positive. All I can say is that the photos that were false positive that included people were either very obviously fully clothed and doing something normal, or the photo was of something completely innocuous all together (I seem to remember an example of the latter was the Windows XP green field stock desktop wallpaper, but I’m not positive on that).
Some of the false positives were of people, others weren’t. It’s not that the hashing function itself was problematic, but that the database of hashes had hashes which weren’t of CP content, as the chance of a collision was way lower than the false positive rate (my guess is it was “data entry” type mistakes by NCMEC, but I have no proof to back up that theory). I made it a point to never personally see any content which matched against NCMEC’s database until it was deemed “safe” as I didn’t want anything to do with it (both from a disgusted perspective and also from a legal risk perspective), but I had coworkers who had to investigate every match and I felt so bad for them.
In the case of PhotoDNA, the hash is conceptually similar to an MD5 or a SHA1 hash of the file. The difference between PhotoDNA and your normal hash functions is that it’s not an exact hash of the raw bytes, but rather more like the “visual representation” of the image. When we were doing the initial implementation / rollout (I think late 2013ish), I did a bunch of testing to see how much I could vary a test image and have the hash be the same as I was curious. Resizes or crops (unless drastic) would almost always come back within the fuzziness window we were using. Overlaying some text or a basic shape (like a frame) would also often match. I then used photoshop to tweak color/contrast/white balance/brightness/etc and that’s where it started getting hit or miss.
This has me pretty worried - once someone has been tarred with this particular brush, it sticks.
But image hashes are “perceptual” in the sense that the hash changes proportionally with the image. This is how reverse image searching works, and why it works so well.
It is claimed that the chance of being a false-positive for a positive match is one out of a trillion.
> Apple says this process is more privacy mindful than scanning files in the cloud as NeuralHash only searches for known and not new child abuse imagery. Apple said that there is a one in one trillion chance of a false positive.
https://techcrunch.com/2021/08/05/apple-icloud-photos-scanni...
I don't know what the cutoff is, but it doesn't sound like they believe that possession of a single photo in the database is inherently illegal. That doesn't mean this is overall a good idea. It simply weakens your specific argument about occasional false positives.
My point is that we have no way to verify the database wouldn't be abused or mistaken and a lot of that rests on the fact that CSAM is not something people want to have to encounter, ever.
It's kind of like the PCAOB... private 501.3(c) with congressional oversight and funding.
I think the strategy is that the organization is able to do more for helping children internationally if they're not seen as part of the Justice department and the executive, which after the debacle with CBP and "kids in cages", was probably the right call.
Anyone should be able to scan their own library against the database for false positives. “But predators could do this too and then delete anything that matches!” some might say, but in a society founded on the presumption of innocence, that risk is a conscious trade-off we make.
The law requires companies like Google and Apple to report when they find CSAM and afiact they would generate hashes and add to this database if new material is found.
I don't know if there is any oversight in this. It's all done behind closed doors so you just have to trust that the people creating the hashes aren't doing anything nefarious or mistaken and that's a separate point apart from what others have said on here that you should be able to trust your devices you own to not be informants against you.
I have a much simpler rule: Your device should never willingly* betray you.
*With a warrant, police can attempt to plant a bug, but your device should not help them do so.
The vast majority of iPhone users won't consider it a betrayal that they can't send images of child abuse, any more than they consider it a betrayal that it doesn't come jailbroken.
The victims of child abuse depicted in these images may well have considered it a betrayal by Apple that they allowed their privacy to be so flagrantly violated on their devices up until now.
Probably neither would child abusers, since as soon as they send an image of child abuse, they're much more likely to be caught than if it had stayed on their phone.
> a betrayal by Apple that they allowed their privacy to be so flagrantly violated on their devices up until now.
"Their" devices? Once Apple sells an iPhone, it no longer belongs to Apple. Taking "betrayal" to mean "didn't plant backdoors on other people's computers to catch your abusers" is stretching that word far beyond reason.
The rule is that your (note the emphasis) device won't ever willingly betray you. There's nothing here that implicates the majority in any way. Simply, your own device should never work against you.
This actually sounds like a great rule to prevent this kind of authoritarian scope creep.
- Yea yeah
- Great. Give me access to every part of your life so i know you're not a terrorist.
It's a pandoras box. You wouldn't allow the regular search of your home in real life.
That said, law enforcement has a nasty habit of having a rather "binary" worldview. People are either cops, or uncaught criminals. ..and they wonder why they have so much trouble making non-cop friends (DISCLAIMER: I know a number of cops).
With that worldview, it can be quite easy to "blur the line" between child sex traffickers, and parking ticket violators. I remember reading a The Register article, about how anti-terrorism statutes are being abused by local town councils to do things like find zoning violations (for example, pools with no CO).
Misapplied laws can be much worse than letting some criminals go. This could easily become a nightmare, if we cede too much to AI.
And that isn't even talking about totalitarian regimes, run by people of the same ilk as child sex traffickers (only wearing Gucci, and living in palaces).
”Any proposal must be viewed as follows. Do not pay overly much attention to the benefits that might be delivered were the law in question to be properly enforced, rather one needs to consider the harm done by the improper enforcement of this particular piece of legislation, whatever it might be.”
-Lyndon B. Johnson
Ehh let's not make a habit of asserting anecdote as fact, please. Saying you know cops is like saying you know black people and that somehow it affords you some privilege others do not possess.
This is a weak and ad-hom argument.
Of course it does. Interacting with black people (or any race) affords you insight into their life experiences, struggles, worldview etc...
Of course sociological discourse is highly subjective but this attitude on HN that anecdotal data has no value whatsoever is silly. Do you seriously expect every fact of every people to be published in some infallible academic journal?
Just knowing someone of a particular demographic doesn't mean you're entitled to generalize about them. That's not how discourse works.
So simultaneously I can imagine how black people wouldn't care and why that doesn't matter. The word isn't banned because blacks have delicate eardrums, it is banned because white people are showing respect.
I probably could have done without the second sentence, but the first stands.
Have a nice day.
I'm not. I am very unambiguously against this and I think if word gets out Apple could have a real problem.
I would like to think I am against child porn as any well-adjusted adult. That does not mean I wish for all my files to be scanned without my consent or even knowledge for compliance, for submission to who knows where matching to who knows what reporting to, well, who knows.
That's crossing a line. You are now reading my private files, interpreting them, and doing something based on that interpretation. That is surveillance.
I am very not OK with this.
"Think of the children" is how they force changes that would otherwise be unconscionable.
They've done it with encryption and anonymity for years. Now they're doing it with the hardware in your pocket.
We would all be better off without these things happening, and anyone would want less of it to happen.
Since they are only searching for _known_ abusive content, by definition they can only detect data that has been shared, which I think is the important point here.
First it's iPhone photos, then it's all iCloud files, that spills into Macs using iCloud, then it's client side reporting of local Mac files, and somewhere along all other Apple hardware I've filled my home with have received equivalent updates and are phoning home to verify that I don't have files or whatever data they can see or hear that some unknown authority has decided should be reported.
What is the utopian perspective of this which counterbalances the risks for this to be a path worth taking?
Apple takes care of everything for you, and they have your best interests at heart. You will be safe, secure, private and seamlessly integrated with your beautiful devices, so you can more efficiently consume.
What's not to like about a world where child crime, terrorism, abuse, radical/harmful content and misinformation can be spotted in inception and at the source and effectively quarantined?
In 2021 and 2020 we saw people being arrested for planning/promoting anti lockdown protests. Not for actually participating but for simply posting about it. The scope of what "harmful content" is is infinite. You might agree that police do need to take action against these people but surely you can see how the scope creeped from literal terrorists and pedophiles to edgy facebook mums and how that could move even further to simple criticisms of the government or religion.
It's difficult to say how we draw the line to make sure horrible crimes go punished while still protecting reasonable privacy and freedom. I'm guessing apples justification here is that they are not sending your photos to police but simply checking them against known bad hashes and if you are not a pedophile, there will be no matches and none of your data will have been exposed.
In Germany police requested contact tracing lists from restaurants in investigations.
And the underlying desire for having this information will no doubt prolong the Corona restrictions longer than necessary, which is certainly not in the interest of German citizens.
Oh dear, one thing we could definitely state that German government is absolutely could be trusted to do the right thing
Australia, WA, border pass data: https://www.abc.net.au/news/2021-06-17/g2g-app-data-accessed...
Australia, VIC, check-in data: https://www.theage.com.au/politics/victoria/police-sought-ac...
Australia, QLD, check-in data: https://www.abc.net.au/news/2021-06-29/queensland-coronaviru...
[1]: https://www.golem.de/news/hamburg-polizei-nutzt-corona-konta...
Also the partly state sponsored luca app (check in in locations, festivals, restaurants, concerts) that is privately developed (and riddled with security holes) is already in discussion to use the data on the people to better target them for concert tickets and the like [2].
[2]: https://www.ccc.de/de/updates/2021/luca-app-ccc-fordert-bund...
So we see this data is already in abuse by the state and also by state sponsored private entities.
I believe, that this data, once collected, will only be (ab)used further in the future. In my experience it will be as with all data caches - somebody wants to create additional value from it.
https://fortune.com/2021/02/01/singapore-covid-data-tracetog...
https://www.straitstimes.com/singapore/politics/police-can-a...
https://www.straitstimes.com/singapore/proposed-restrictions...
https://www.straitstimes.com/singapore/politics/bill-limitin...
(Note that one mostly-united political party controls 89.2% of Singapore's legislature seats, and can pass any laws or amend its constitution to their liking.)
Please stop creating accounts to break HN's rules with.
Basically victims of rape don't want imagery of their rape freely distributed as pornography. They consider that a violation of their rights.
It's interesting how many users in this thread are instinctively siding with the offenders in this, and not the victims. Presumably because they made it through their own childhoods without having imagery of their own abuse shared online.
That is infantile. Painting people advocating privacy as siding with offenders is highly insulting.
If the median shelf-life of abuse evidence is shortened, in that the item in question can no longer be forwarded/viewed/stored/..., what does that imply in a world where the demand remains relatively stable?
I despise the abusers for what they do, and the ecosystem they enable. But I also remember first having this argument more than ten years ago. If you, as a member of law enforcement or a child wellbeing charity, only flag the awful content but do not do anything else about it, you are - in my mind - guilty of criminal neglect. The ability to add an entry to a database is nothing more than going, "at least nobody else will see that in the future". That does NOTHING to prevent the creation of more such material, and thus implicitly endorses the ongoing abuse and crimes against children.
Every one of these images and videos is a piece of evidence. Of a horrifying crime committed against a child or children.
There's a small number of victims, a small number of offenders (but much more than "a handful"), and hundreds of millions of other users. This change is in the direct interest of victims, direct opposition to offenders.
Most normal people probably support the measures in solidarity with group 1, HN generally doesn't.
It's a restriction on their liberty and privacy that they willingly support because of the overall positive effects.
Anyway I'll duck out of this now the driveby downvotes annoy me.
But yes, there's a lot of drive-by engagement in this thread, thank you for at least engaging with it directly.
There are kidnapped children being locked inside homes. If you don't open your doors and accept weekly full home inspections, I think it's safe to say you support offenders and hate victims if you oppose this. I mean, we're all against people kidnapping and abusing children.
There's a small number of victims, a small number of offenders (but much more than "a handful"), and hundreds of millions of other home owners. This change is in the direct interest of victims, direct opposition to offenders.
It's also incredibly likely this could be used to send people you don't like to prison by sending their phone innocuous images that look like wrong images to an AI.
It's also also incredibly likely this will evolve into scanning for more than just abuse photos, especially in the hands of governments around the world.
I strongly believe that nobody wants to further victimize people by publicly showing images of their abuse.
And I believe very strongly that putting hundreds of millions of people under blanket general suspicion is a dangerous first step.
Imagine if every bank had to search all documents in safe deposit boxes to see if people had committed tax evasion (or stored other illegal things like blood diamonds obtained with child labor). That would be an equivalent in the physical world.
Now add to this, as discussed elsewhere here, that the database in question contains not only BIlder of victims, but also perfectly legal images. This can lead to people "winning" a house search because they have perfectly legal data stored in their cloud.
Furthermore, this means that a single country's understanding of the law is applied to a global user community. From a purely legal point of view, this is an interesting problem.
And yes: I would like to see effective measures to make the dissemination of such material more difficult. At the same time, however, I see it as difficult to use a tool for this purpose that is not subject to any control by the rule of law and cannot be checked if the worst comes to the worst.
A great deal of skepticism is being given to the NCMEC database in these comments, which I'm surprised by as from what information I have I think this is being exaggerated. At the same time we have no idea whether Apple would even be using that database or another one that they may have created themselves.
Thi sis transmission of funds and there are laws regulating the monitoring of those.
I used bank vaults were you put things into the vaults without the bank often times knowing what is in there. If they knew, they would need to report to authorities.
So Apple doing this scan would be the bank opening all vaults, scanning the contents and reporting things to the IRS (I think this is the tax thing in the US if I am not mistaken - in Germany it would be the Finanzamt).
It's only boring until we get another Hitler or equivalent.
You will make Apple tons of money.
My parents took lots of photos of me as a baby/small child. Say lying naked on a blanket or a naked 2yr old me in a kiddie pool in the summer in our backyard. Those are private photos and because it was the 1970s those were just taken with a normal non-digital camera. They were OBVIOUSLY never shared with others, especially outside immediate family.
Transform that into the 2020s and today these type of pictures would be taken with your iPhone. Would they now be classified as child pornography even though they weren't meant to be shared with anyone nor were they ever shared with anyone? Just your typical proud parent photo of your toddler.
Sounds a bit like a slippery slope, but maybe I am misunderstanding the gravity here. I'm specifically highlighting private "consumption" (parent taking picture of their child who happens to be naked as 1yr olds tend to be sometimes) vs "distribution" (parent or even a nefarious actor taking picture of a child and sharing it with third parties). I 100% want to eliminate child pornography. No discussion. But how do we prevent "false positives" with this?
If that sounds absurd - most laws are like that. For better or worse, there's a human who interprets the law, not a computer. It's unfortunate Apple is choosing to elect a computer as the judge here, for exactly concerns like yours.
Unless someone has been distributing photos of your kids as child porn (which would probably be good to know) it's unlikely any of your photos will match the hashes of the photos in that database.
I'm not sure that's how it works, but that's what I've gathered from the other comments on this post.
PhotoDNA and systems like it are really interesting, but it seems like clientside scanning is a dangerous decision, not just from the privacy perspective. It seems like giving a CSAM detector and hashes to people is a really risky idea, even if it’s perfect and it does what it says it does without violating privacy.
If the algorithm and the blocklists leaked, then not only it would be possible to develop tools that reliably modify CSAM to avoid detection, but also generate new innocent-looking images that are caught by the filter. That could be used to overwhelm law enforcement with false positives and also weaponized for SWAT-ing.
https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
- (Check) fragile
- (Check) best monopoly app store
- (Check) high price
- (Check) phones the police in case of a sha collision
Personally I somehow doubt that MS/Google weren't scanning private content (aka not shared) for this type of material. But can't have transparency with these behemoths.
[1] https://www.euronews.com/2021/07/07/eu-adopts-temporary-rule...
[2] https://www.theverge.com/2014/8/7/5977827/microsoft-tips-off...
[3] https://www.theverge.com/2014/8/5/5970141/how-google-scans-y...
I can’t say I’m surprised they are implementing this (if true), under the radar. I can’t imagine a correct way or platform for Apple to share this rollout publicly. I’m sure nothing will come of this, press will ignore the story, and we all go back to our iPhones
But Apple iCloud for ex doesn't intrude on your privacy any more or less than Google Photos.
As far as I know, this kind of scanning is not legally mandated. So, either they think that this will truly make the world a better place and are doing it out of some sense of moral responsibility, or they've been pressured into it as part of a sweetheart deal on E2E ("we won't push for crypto backdoors if you'll just scan your users' phones for us"). Either way it doesn't thrill me as a customer that my device is wasting CPU cycles and battery life under the presumption that I might possess data my current jurisdiction deems illegal.
For all the acclaim privacy-forward measures like GDPR get here, I'm surprised there isn't more outright repudiation of this frankly Orwellian situation.
https://fightthenewdrug.org/apple-fights-child-porn-by-scann...
* A car that automatically pulls over when a police cruiser attempts to intercept you
* A front door that unlocks when a cop knocks
* A camera that uses AI to detect and prevent the photography of minors, police, and critical infrastructure
* A Smart TV that counts the number of people in your living room to ensure you aren't performing an unauthorized public broadcast of copyrighted content
Surely, at least one of those sounds ridiculous to you. As well-intentioned as this scanning may be, it violates a core principle of privacy and human autonomy. Your own device should not betray you. As technologists, just because we can do something doesn't mean we should.
The UK and Israel allowed cops to monitor cell phone locations to crack down on unlawful gatherings in private homes.
https://www.theguardian.com/world/2020/mar/17/israel-to-trac...
The problem with allowing this is that you’re paving the way for future tyrants to use it against us.
It's funny how everybody talk about the future. This is happenning now. Remember how a certain german guy took the power some 90 years ago ? He was elected.
People nowadays voluntarily carry tracking devices. This will not stop getting worse until that behavior is denormalized.
The power to be gained from abusing it is beyond irresistable. Expecting those in power to not abuse it is like expecting a heroin junkie to be a good pharmacist.
on the strict premise that tracking is exceptional fair use from a law enforcement agency. Do not mix up the voluntary accolites of Zuck and people who want tools. (The use of 'tools' in the sentence is an originally unintended pun. I will keep the term 'accolites', though tempted to replace it for the pun.)
I know I know, Disneys inability to collect money for every pair of eyes watching their movies during VHS times literally made it one of the poorest companies on the world. /s
Also, painting is a physical object. It's a one of a kind.
EDIT: your objection and many many other questions like that are nicely argued against here: https://www.youtube.com/watch?v=mhBpI13dxkI (The Surprising History of Copyright talk by Karl Fogel)
EDIT1:
More than that. People don't live of royalties. Publishers do. Can people create without being paid upfront by publishers? Obviously yes. They can be paid upfront in kickstarter-like arrangement. They can be paid via donations later. They can even be paid via "retroactive funding of public goods" [0]. Or they can be never paid - just as they are today.
[0] - https://medium.com/ethereum-optimism/retroactive-public-good...
EDIT2:
Also, imagine how much good could come of long-standing IP properties owned by Disney if not for them sitting on it. Vide disaster that is current management of Star Wars.
Or we can talk about the mess you get with too many groups involved? Do you know who has the rights to Westwoods Dune? Westwood itself had a limited time license for games based on the movie Dune, which also had a license based on the Dune books. So you would have to deal with three license owners to make a Game involving the Ordos faction. Have fun convincing EA, whoever owns the movie rights and the original authors family that you can make a new game worth their signature on a new licensing agreement (you could probably manage if you have a small country to sell).
People WILL be a fan of all of this if the alternative is lots of robbery / car theft etc etc.
We see this globally. If the state is not offering security now - they will accept incredible craziness for security.
One note - in most cases folks trust Apple MORE than they would for example the Trump administration. Food for thought.
Maybe one trusts Apple more than <insert politician>, but they cannot so easily elect away Apple.
There was concern about phone's being grabbed - street robberies. Regardless of whether you believe their were sweeping generalizations - apple ended up creating more power for themselves with their activation lock system. If they don't want to let you sell your phone to someone - they can block use of your phone. But folks trust them to operate the system reasonably, and so far so good there.
They have locked down their app store very tightly for a variety of reasons including supposedly for security. Users have accepted that.
Apple's competitors (google photos etc) generally are directly scannable in the cloud by google et al. Facebook and others routinely scan users photos. Youtube scans their videos etc. My guess is apple will explain why they are doing it and users are going to be happy.
And yes, users are linking things like ring doorbells and home security video cameras together or registering them so that the police state can use them.
The technology is there, now we only need the motivation. If politicians decides that they want it now, they can simply orchestrate a media campaign and have it. The next time an "outrageous" act of crime is conducted, they can make sure that it stays at the media attention and be portrayed as "If we don't act now very bed things will happen", then slide in their solution.
* Cars can automatically pull over by installing a cheap cut fuel cut switch that can be activated by short range radio. In many places people are used to add devices for toll collection anyway. People are also used to pay for regulatory inspections on their vehicles.
* For the old cars, simply connect an NFC reader that unlocks the central lock system of a car by a master key. For the new cars, simply make manufacturers add a police master key.
* Commercial drones are already stopping their users from flying over forbidden areas, simply extend that to smartphones. Smartphones have enough power and sensors to identify forbidden locations and persons. Add NFC kill switch, meaning the police can send a signal to lock down cameras.
* There were reports of Smart TVs that record all the time, simply mandate it to all manufacturers and enforce automated inspection of the recordings.
Apple devices already betray their "owners", and they've been doing it for a long time.
You can't repair them.
You can't run your own software.
You can't use a better, more compliant web browser.
Businesses have to pay a 30% tax.
Businesses are forced to use login with Apple and forfeit a customer relationship.
Businesses have to dance to appease Apple. Their software gets banned, randomly flagged, or unapproved for deployment, sometimes completely on a whim.
Soon, more iDevices and Apple Pay will lead to further entrenchment. Just like in the movie Demolition Man, everything will eventually be Apple. Your car, your movies, your music, your elected officials.
While this argument can and have indeed happened in other instances, this is akin to saying that we should not give anyone any powers to do anything because it is a slippery slope that they can use it to do bad things.
What then sets out the difference between what a slippery slope and a non-slippery one is? Checks and balances and the history of USA have shown that this is indeed what can reign in the worst instincts of any single entity. History of course have also shown when these failed and these should serve not as a reason to reject the idea of checks and balances but as acknowledging it's imperfection and think of ways to mitigate it.
Two instances: 1) Post 9/11 Patriot Law 2) McCarthy era: https://www.e-ir.info/2011/11/03/the-extraordinary-injustice...
Checks and balances are never a done deal. If we reject checks and balances and as a result reject new tech because of abuse potential, how then should we as a civilisation advance?