Except the difference is nudity is natural and sometimes instrumental - for washing, bathroom use, and medical care for example - and denying basic healthcare to your children is not, unless you plan to be instrumental in your child's demise and that comes natural to you.
Reading back now, I realize my comment was indeed just dumb and borne from not properly reading the OP's comment and thinking of the anti-vaxxer movement in general, not the actually rational aversion to untested ones during the Covid crisis.
I would have editing a disclaimer on it but it's too late.
COVID vaccines are not basic child healthcare, heck they are not even approved for children under 12 and in many European countries not for children at all
also not sure how necessary is to take nude photos of children unless you are doctor and yes I am parent with 2 kids
taking care about your health and not willing to risk long term effects of unproven stuff someone is putting in your body seems perfectly natural to me
Excuse me, sweetie, but "think of the trans" is a very important movement. It prevents suicides, helps bring gender equality, raises awareness, and creates tech jobs for trans activists to assist in stopping online transphobia.
I mean, who's to say that my hash is actually what they think it is...
...and if it's not uploaded to iCloud for someone to manually review what the image is, is that going to be enough to get a search warrant for my phone?
"AUSA SOMEONE: But, Your Honor, The Defendant's phone has an image that matched a hash of a CSAM picture!"
"DEFENDANT: I'm not going to give you my phone's password."
"THE COURT: Bonk: Stay in jail for contempt."
I understand why they're doing it. I don't agree with it. At all, and I say that as a survivor of childhood sexual abuse.
EDIT: It appears they are using perceptual hashes, which are likely much more prone to collisions. The following math does NOT apply to the tech Apple is using.
>> I mean, who's to say that my hash is actually what they think it is...
Let's do the math. assume they are using a UUID for the hash. "For example, the number of random version-4 UUIDs which need to be generated in order to have a 50% probability of at least one collision is 2.71 quintillion" [1]
"The odds of This number is equivalent to generating 1 billion UUIDs per second for about 85 years." And that's for just 1 collision between two random photos. Considering that the pool of bad hashes is not going to be 2.7 quintillion but a fraction of that, this is more than adequate.
And of course they could use a longer checksum if that's the concern. (Not to take a stance on the parent issue, just to explain that hash's are probably a higher standard of proof than DNA matches)
Hell, what if you were browsing a site like Facebook or Imgur or Reddit, saw an abuse image, reported it, but it still lives in your phone's browser cache?
> "The odds of This number is equivalent to generating 1 billion UUIDs per second for about 85 years."
If we're talking about random files, sure.
What if the files are intentionally created to match the desired hash? The malicious actor doing this could be a private party intending to disrupt your life, but it could also be law enforcement as a means of gaining access to your device when it would otherwise require a warrant.
They could come up with a dozen other ways apart from swatting too, but swatting still happens and we generally agree that it's a problem where actions should be taken. Political debates over whether that means the police should act in a less militarised fashion, or harsher sentences for culprits are ongoing, but cutting down on avenues for life ruining abuse is generally supported.
SWATing is actually the absolute cheapest option - if you want to ruin someone's week from across the nation (or across the ocean), your best bet is a burner phone calling the person's local 911 with a threat that warrants a SWAT team response (I am of course not advocating for anyone to do this, just explaining the process malicious actors currently use to trigger these). Outside of that, you'd need to pay a lot of money for something like a PI or obfuscated malware to steal PII or credentials from the victim.
I hope you don't ever run afoul of an internet mob from somewhere like 4chan! They might SWAT you while DDOSing your IP and doxing you then emailing your iphone dozens of photos that collide with CSAM perceptual hashes.
LEO have never, ever planted a baggie of weed in someone's car. Likewise, LEO would never, ever ensure that a picture that matched a known phash gets on a target's phone.
Once someone develops the tool for that, it WILL become available in the wild. Then, any script kid will be able to "swat" you in a even more insidious way.
Or what if you're a researcher who has downloaded a collection of these assets? Wouldn't you be a very high signal hit, beyond even actual perpetrators of crime? Would you need the protection of an umbrella organization to do research? Or register with the government perhaps?
This is digital biowarfare material of the worst kind and should be handled as such.
Any researcher downloading this material to anything but lab machines should probably receive a visit from the relevant authorities to make sure they understand the seriousness of it.
This is a function that is created to be robust against modifications to the image.
A perfectly robust function would - in my opinion - be one that could detect even a recreation of the original setup.
I don't know where on this scale the function law enforcement use is, but if it is any good I struggle to see how it cannot detect similar legal imagesm
I understand the danger of the false positive but I suspect someone with actual child pornography on their device would trigger a dozen or more matches. Clever software would ignore a single match.
Nope, federal law states they can hold you for 18 months. So long as they can claim to already know what's on the device (which they will, because Apple pinged them about it), they can hold you for 18 months.
Alright, after further investigation it looks like the Supreme Court has so far declined to rule on this issue so it depends on the state. It sounds like most states supreme courts have ruled that you can't be forced to unlock a device, but some, like pennsylvania say that if the state can prove you know the password they can force you to unlock the device.
The 100 mile zone is clearly fucked up, but I haven't seen any cases of it being used to force anyone to give up their password. Even with a warrant US citizens do not have to give up their password. The only exception I've seen is a case where the defendant shared some info on their phone but refused to share everything. The judge ruled the defendant waived his fifth amendment rights in that case. I think it's clear that that was the wrong decision, but the solution is clear and easily done, don't give cops any information.
This is how the police gets warrantless searches "for drugs" - having identified the subject, call for a dog and have it "alert" on the subject (or the vehicle belonging to the subject), after which there's a probable cause for a search and everything that is found can be used against the subject. The fact that there was no drugs is irrelevant.
Now let's translate it to digital world and eliminate the non-scalable components - and you get AI "alerting" on the content of your account and it's a probable cause for searching you.
But the best part is of course the Apple doesn't even need a probable cause to search you - they aren't the police, so they are free to search anything at their pleasure and then report to the police. And this report, by itself, would be the probable cause, of course.
Whose to say what is abuse? Are Caesar Milan tactics on a child abuse? This is outright dangerous beyond belief. Nobody is gonna wanna be a parent if you have to worry about your kid being mad at you and then posts something like that.
IIRC there was a teenage girl that lied about her father molesting her back in the 2000s. No evidence, just pure hearsay. Dude was locked up for like 10 years. Now imagine that on a greater scale with several videos and photos taken out of context.
That would be false testimony, not hearsay. Hearsay ("John Smith told me he heard MeinBlut kidnapped the Lindbergh baby") is not allowed. Witnesses lying about what they saw and or participated in is allowed, although it can be lies.
Sorry, and as for "what is abuse", these are looking for exploited children. There are a lot of questions here, the morality of mass surveillance, the risk factors due to hash collisions (intentional or otherwise), etc, the one thing I doubt is an issue is a fuzzy line on "is this abuse"
I don't think society can function without harsh penalties for child abuse, and I think thats a widely held belief everyone agrees with.
I do think your comment is conflicting though. The fuzziness of 'is this abuse' is real because collisions can occur, mass surveillance never goes in reverse, etc.
Doesn't seem to be working very well from here; risk of penalty isn't going to stop most of these people, more likely make their game even more exciting.
Of course they have to be prevented from hurting people, to the best of our ability to do so, but punishment isn't going anywhere.
Huh, I specifically said "concerns about collisions" is a real concern. I said that "mass surveillance" is a real concern.
What is not a real concern is fuzziness in the question "is this image of child abuse". And I really don't want to continue a discussion where you try to come up with gray areas. A computer may not be able to tell the difference, but humans can.
Scan for child abuse today; scan for wrong-think tomorrow. I'm liking the idea of Graphene OS more and more each day... or just going phone-free entirely.
We will not be able to have nice things until we get better privacy laws or come up with novel ways to prevent unwarranted snooping. It's sad that it's come to this.
I really don't see how this is going to end well, there could be perfectly innocent photos on someone's phone of their own children doing perfectly normal things that kids do. Like a kid running butt naked around the house, or a photo of something like a rash that is sent to a nurse friend for advice on what it is etc.
I'm all for protecting children from being abused, but how are they going to filter what is normal and what is abuse without human intervention? And at that point isn't that a vast invasion of privacy to the perfectly innocent? It's different if it's online because that puts it in the public realm or on devices/servers owned by Apple, etc.
Plain file hashing for this kind of thing has not been used for years (or even decades), Microsoft came out with Photo DNA along time ago, and I am sure now days they have something ML powered to do it.
I am not sure the accuracy of these systems but even if they are pretty accurate there is something off putting about this trend.
This very much feels like a Guilty until proven innocent type of program... "if you have nothing to hide you have nothing to fear"...
But this software doesn’t make arrests? A comparison would be police asking for what cellphones were in an area. There should be a lot of due process that happens before anything comes of this. I imagine police won’t look into one photo, they would want to match more than one.
I agree though, the scary part might be how to clear your name after an overreaction occurs, similar to the effect swatting has on innocents or other kinds of false accusations. But those would happen even without such a system - we’ve heard about such stories before. “Man held for child porn charges, but it turned out they were innocent,” - it doesn’t take this kind of system to have police look over your phone for evidence and find a different crime than expected.
That all said… child porn is really, really terrible and I’m actually in favour of this. As a society, we can decide on what’s acceptable. As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes. I do agree though, all these systems might make it harder to catch perpetrators if it pushes them offline. But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I suppose the fundamental problem is that having more evidence doesn’t necessarily lead to justice if it paints a false picture.
I see it as an unwarranted search, which makes me feel like the state views me as inherently guilty until innocence is proven by sifting through my photos.
It’s not the state, it’s Apple. And they are making the decision that they do not want to put out a product that makes it easy for CP to be shared or viewed. As a huge player in the market this is great.
This also limits the possible avenues for future exposure - if one has less ways to distribute or get CP or the overall amount of it is reduced, they may never get into it in the first place and the market for it will shrink. Thereby reducing the amount produced over time.
According to the article, Apple is responding to government and law enforcement pressure to implement the scanning. The state is just laundering its desires through private corporations; there must be a word for this type of system.
Apple will be scanning the photos on your phone and cross-referencing them against photoDNA for a government database. You don't see how that's not a huge potential for abuse right there? The government decides the images. And you best believe they're not going to release the images publicly to verify what's in the db, for [many] obvious reasons.
This is a horrible idea, a massive regression in Apple (and therefore the broader industry's) stance on security/privacy, and the fact that you are dutifully swallowing the "child porn" justification is so incredibly concerning to me.
The article says Apple will have your phone generate safety vouchers for images you upload into iCloud, and when enough of those vouchers are for photos that match a government-supplied blacklist, they will access the photos you’ve given them. How is this worse than what Google Photos and Facebook already do?
You think Facebook and Google Photos should be the goal for privacy standards? You and I are practically in different universes then, philosophically.
You're also being naive by swallowing the "it's only iCloud" justification. There is a clear intent to progress to scanning all local photos. After all, if photos must be match those of a government child abuse database, and any matches trigger manual review, what would an innocent person possibly have to hide? By that logic it'd be absurd to not scan just because it's only locally. After all, you're only catching bad guys! This is about protecting the children after all! /s
I oppose government-supplied blacklists. Doubly so when we can't even see what's in the database, but I would oppose them either way.
What advice would you give Apple if they’re faced with a legitimate government asking it to comply with the laws they have instituted in the society Apple wants to do business?
I don’t appreciate you calling me naive. There isn’t clear intent to scanning all local photos. In fact, it seems like the system was deliberately designed to only work with photos that are being uploaded to iCloud.
> What advice would you give Apple if they’re faced with a legitimate government asking it to comply with the laws they have instituted in the society Apple wants to do business?
When someone is trying to rob you, generally the best thing to do is to hand over your wallet. If a law passes you should obey it for your own sake. That's an entirely orthogonal question to whether we as the users should be outraged or not.
Also while obeying an injust law, a company has a responsibility to legally challenge it in court with their resources, if for no other reason than to avoid the profit loss of having to comply with invasive regulations
Furthermore, it's not clear that this is the direct result of some federal law. It reads more like pro-actively appeasing their federal overlords.
> What advice would you give Apple if they’re faced with a legitimate government asking it to comply with the laws they have instituted in the society Apple wants to do business?
It is good advice not to carry around huge amounts of money on your wallet. Someone might want it and have the power to get it.
Same with megacorps constructing and bragging about tools for oppression:
Better not do it.
Otherwise just be prepared that China will demand that you also upload their hashes of extremely bad photos - and one or two guys they want to get hold of.
It is kind of a Manhattan project for digital privacy - it is meant to bring a quick end to a actual real problem we have - but it has the potential to become a shadow over society for decades if not centuries to come.
> But this software doesn’t make arrests? A comparison would be police asking for what cellphones were in an area. There should be a lot of due process that happens before anything comes of this. I imagine police won’t look into one photo, they would want to match more than one.
I doubt they need, or want to match more than one. They'll use the reasoning that it only catches already-known child pornography, and that that person might have a whole cache of novel child pornography. Just look at the drug wars and how many people get arrested for trivial quantities of marijuana. And we hate child pornography way more than drugs.
> As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes.
The judicial system has a presumption of innocence. No one else does, or has any requirement to. The court of public opinion will ruin your life regardless of whether it's true or not.
That would be far less concerning if active cases were sealed, and only published if the defendant was ruled guilty.
> That all said… child porn is really, really terrible and I’m actually in favour of this.
I have a slight disagreement with this. The production of child pornography is obviously abhorrent. This system does nothing to stop that though, because it can only flag already-known child pornography.
This system stops the consumption of already produced and documented child pornography. While it's far outside my own interests, I can see some validity to the argument that consumption is of far less concern because it doesn't directly involve any abuse. The abuse has already happened, and no amount of tracking down pedophiles is going to reverse that. There are second order effects, like potentially reducing the demand for novel child pornography, though. Which leads me to this:
> But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I would imagine this is far more likely to affect consumers than producers. In the drug market, stop and frisk is far more likely to net you users than producers or dealers. For one, there are more users than dealers/producers. For two, dealers, but particularly producers, are typically far more cautious than users.
That's a long-winded way of saying again that this doesn't seem likely to do much in the way of stopping people from abusing children.
But to directly answer your question: doing nothing is absolutely a valid option here. We do that for lots of bad things that happen when the costs of the solution are unbearable. It's what we're currently doing for famine in other parts of the world, which almost surely impacts more children than child pornography could even dream of doing. It's what we do for cars, which again, almost certainly impacts more children than child pornography does.
We wait until we can find a method with benefits commensurate to its costs. A 2016 DoJ report said that since 2002, 10,500 victims have been located and identified by law enforcement. That's 750 per year. There are 74 million children under 18 in the US, so that's 0.001% of children per year. 10,500 people is only 200 people higher than died from "accidental suffocation and strangling in bed" over the same period according to the CDC. The scale of the problem is the same, so should we go ahead and let Amazon send pings to the police if your Alexa thinks your suffocating in bed too?
So are you in Favor of installing Camera's in every room in your home, that you do not control, and that the police and review anytime?
If in the US Do you like the concept behind 4th amendment? being free from unreasonable searches unless probable cause can be articulated that you have committed a crime?
It is very scary how effect the "think of the children" narrative is to get people to willing give up their liberty
What will happen is that criminals who actually had bad intent will move off of the platform and the ones who get the brunt of the blame are the innocent who had no ill intent.
The process is described above, but it’s very hard to “innocently” end up with one of those images that they are looking for from the database.
And the way it’s being done (hashes), a collision is highly unlikely. If it does occur it doesn’t mean it’s similar in nature (e.g. innocent picture of own child in bath). The hash isn’t looking at the image content in the sense of “what’s in the picture”, just the bits of the file. So it’s highly, highly unlikely, even if a collision occurs, that the collision would be an image that happens to be another child innocently bathing.
Presumably the verification that these false matches are not problematic is manual?
That's not good enough given how our media currently works. Imagine articles published when information about such a check leaks that say "celebrity X's phone checked by police for suspected CSAM." While that is the truth ("suspected") no one cares about that nuance and such a person would get cancelled very quickly, even if there was no evidence of wrongdoing.
What happens if someone spams CP via iMessage to their enemies?
Additionally, this isn't just a hash of the file but a perceptual hash on the image content. So e.g. changing a single bit in the image would create a different cryptographic hash, but generally not a different perceptual hash.
> but it’s very hard to “innocently” end up with one of those images that they are looking for from the database.
Are you sure, given how many iPhone takeover/jailbreak bugs regularly exist, including the recent 0-click iMessage bug? Would you like to dare a hacking group, domestic or foreign, to land a single verboten image on your iPhone?
> And the way it’s being done (hashes), a collision is highly unlikely.
The limited details so far are suggestive that its using perceptual hashes, which are more susceptible to collision-engineering/false-positives than cryptographically-secure hashes.
Furthermore, "highly unlikely" is an understatement. Using SHA256 there are 2^256 hashes.
That's 115792089237316195423570985008687907853269984665640564039457584007913129639936. Probability of collision (two different inputs generating the same hash) is infinitesimal.
> The process is described above, but it’s very hard to “innocently” end up with one of those images that they are looking for from the database.
Actually, it's very easy to end up with an image that has a similar perceptual hash to an illegal image.
They are not doing MD5 hashing, they're taking perceptual hashes and then using something like the hamming distance or Levenshtein distance to make a fuzzy match with hashes from illegal images.
I've built products using these methods, and it is incredibly easy to make a fuzzy match based on perceptual hashes from two images that have nothing to do with each other.
Presuming that is what they're doing, it won't stay that way for long.
It's absurdly easy to change the hash of a file. In the case of something like JPEG, you don't even have to change the file itself, you can just change the metadata. Apple could presumably only hash the image itself, but again, all you have to do is make tiny, imperceptible to humans changes to the image and the hash is totally different.
Long story short, this is either nearly pointless and privacy invasive, or it's about to get drastically more invasive to be effective.
I actually do want a feature where I will get a notification on my phone if someone nearby has snagged a picture of me or my kids. A creepy shopper at Costco once took a pic of my 2-year-old. They pretended to be working up a conversation with my little kid meanwhile they had their phone at hip-level to snag a shot. I even confronted them about it. The Costco employees couldn't do a thing. Phones with cameras are useful but have also become a nuisance.
Edit: It makes sense that such a notification system would be impossible to implement. Perhaps, registered offenders should not only have their location placed on a map but their phones have certain privileges disabled or monitored.
I am sure, presumbly in a sane world, computer scanning is only the early stage of the process, human will be invovled to make final judgements and decisions.
I imagine the biggest catch of this will be teens taking inappropriate photos of themselves because teens often do things like this in their awkward years when sexuality and body consciousness kicks in. With this we'll be seeing warrants for those images and a big fuss made on this most suicidal age group who thought they were doing something completely private. This isn't good.
> perfectly innocent photos on someone's phone of their own children
They are using a library of human verified images to compare the hashes. Probably similar to PhotoDNA[0]. There can be false positives, but AFAIK the algorithms are not trying to identify naked children, but comparing 2 similar image hashes.
Images are submitted by over 1400 organizations[0] like the FBI and NCMEC, although the inner-workings of the database aren’t public and I can’t find a source or confirmation of every image actually being human-verified. The closest is how the "Child Victim Identification Program" helps identify victims in these images.
Of all of the discussion, my brain has only keyed in on the child porn aspect. It just now clicked that if a missing kid is in a photo (even if not porn), it could help ID the missing kid. Some runaway at the age of 13 gets photographed in a group of other kids just doing teenager things.
So even if not running hashes against known porn, it could still be doing facial recognition. Wonder what kind of implications that would have for people in witness protection or battered spouses and other legitimately "disappeared" people.
“Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse.”
The conventional wisdom for people integrating PhotoDNA is that false positives are vanishingly rare. If you run it against your data and you get a hit then you should call the FBI.
It would be better if the false positive rate was known to be nonzero.
And what, in this context, is "your data?" Is a photo on an iPhone Apple's data?
I'd argue that the thing you should do if you get a hit is to delete your data, unless you want to spend enormous amounts of time and money defending yourself from prosecution by people who believe that false positives are vanishingly rare.
I'm talking from the perspective of a service-owner who handles user-generated content. You typically set up PhotoDNA on two parts of your infrastructure. You scan client-side and refuse uploads to prevent the images from getting on your servers in the first place and then you scan on the backend to catch anything that slipped through. You do the first part so you don't have to call the FBI.
I really don't care about the semantics of data ownership. If Apple wants to scan photos you upload to iCloud so they don't run into a scandal years down the road that "iCloud is being used to distribute CP" then that's their call.
And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
In the real world the system delete the data, deletes the associated user, bans the user and no human will do anything about it. Most likely there isn't even a report made to a human it just goes under ToS violation or something so no one inside the companies aks questions.
No FBI is called. Imagine the financial damage this would do if specialist come over and try to collect evidence and the company is forced to cooperated and work for them, for free ofc.
Only the big tech does this. They have whole teams for this and automated system and what not because surely the FBI wont walk into googled data center ever to collect evidence. But all the smaller companies they cant afford to do this and the detection system is actually to protect them form going bankrupt because they accidentally stored some illegal data from a user.
>And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
Yes such laws exist. So they use filter software from third parties. They intentionally offer some generic filter that includes "unwanted content" and doesn't specify what it is when it find something so the companies can wave away the liability as they have no way to know if the image was valuable evidence of a crime or just some LiveLeak gore. They can justify that no human watches it because that's to expensive.
It's still bad. I used to scrape pictures from many legal pornography sites. I still have hundreds of thousands of pictures. Since many are user uploaded, I imagine a small number could be real CP.
Thank God, I don't use Apple.
(I should probably delete them, since I've basically never used them... although they could be useful for one of those end-of-world Raspberry Pi builds: https://back7.co/home/raspberry-pi-quick-kit-one Edit: the non-bad ones, in case this gets misconstrued)
No offence dennis but I gotta be a bit blunt here.
If you're not trolling then you're being a fucking idiot, you could land serious prison time for having a shit ton of child porn on you. If you live in a country where it's illegal delete literally all of it of the hard drive, run DBAN on it, smash it to pieces with a hammer and then set the hard drive on fucking fire.
Literally zero courts are gonna believe "It's just a dataset for if I want to train an AI on it later XD". You will get horsefucked in court
cat /dev/urandom and it also has CP, oh noes! quick delete everything.
Hoarding porn is kind of useless nowadays, especially without any kind of selection and categorization, but maybe someone does it because it's their hobby (or obsession), who knows.
Also if a lot of people has a bunch of porn images scraped from forums it's quite possible that eventually some of that might become illegal as the laws change here or there around the world. Should they proactively delete everything? Sure, they could. To be safe.
But this kind of safety is absolutely the bad kind on the long term. (Though of course it's possible to advocate for reform while not storing content with uncertain legality.)
Dude, this is all great in theory but in a court of law you'll get completely horsefucked if you have child porn on your hard drive. The judge and jury won't care about /dev/random or a fall to reform
You would think that in our "great" and "fair" legal system simply scrapping a legal site would not be enough to ruin your life. Especially if no harm was done and no intention of harm was there.
Believe me, as someone who runs a website I completely agree with you. I just also know the courts will not be understanding under any circumstance.
I don't allow image uploads at all now out of fear I'll back up my site and accidentally download CP. It's ridiculous, all it takes is one dumbfuck pedophile uploading a single file ( completely out of my control ) and I'll get thrown in prison for 10 years
Yeah if you ran the giant box of photos in my mom's closet through an AI you'd totally get hits from photos of baby me being bathed in the sink, baby me running around the back yard naked, etc. This can't end well.
If this even comes close to doing any “recognition” on local content and isn’t just (as the article says) a simple lookup of known images through hashes, then that would be a massive scandal. So I’m giving them the benefit of the doubt until it’s clarified, as the only confirmed way it works is the image hash lookup.
There are words like “neural” and “trained” here, but no clear sign that there is actually any kind of image recognition that would risk false positives other than hash collisions.
>The system that scans cloud drives for illegal images was created by Microsoft and Dartmouth College and donated to NCMEC. The organization creates signatures of the worst known images of child pornography, approximately 16,000 files at present. These file signatures are given to service providers who then try to match them to user files in order to prevent further distribution of the images themselves, a Microsoft spokesperson told NBC News. (Microsoft implemented image-matching technology in its own services, such as Bing and SkyDrive.)
Again, that is not the same as scanning a local machine for it. That is all things that are on Microsoft's own servers or in their search results. Microsoft, Google, Cloudflare all have features like this. It's not the same as scanning local machines.
Or when an internet troll gets your email address and/or phone number and starts spamming you with child porn pictures, which your phone immediately identifies and notifies authorities.
> I really don't see how this is going to end well, there could be perfectly innocent photos on someone's phone of their own children doing perfectly normal things that kids do. Like a kid running butt naked around the house, or a photo of something like a rash that is sent to a nurse friend for advice on what it is etc.
These are still illegal in the US right now right? It's been a major criticism of these laws that they can hurt people accidentally.
Don't worry, AI is very good at these things and never makes mistakes. It is so advanced it even understands nuance and complex communication mechanisms like sarcasm in text, even if it's only a single sentence to contrast! It will be able to know exactly the context of each image and it will affect only the sickos. We're also working on watching your face through the camera to see your true expression when viewing them. Oh, but if a few innocent people do happen get caught up in it and lives ruined, it's totally justifiable because this type of activity is the absolute worst. And false positives are exceeeeedingly rare. We'll apologize and adjust our algorithms. Sue us? How dare you? We're really trying to wipe out sickos. Are you a sicko or something? You must be. Only sicko's would disagree with protecting children.
How about we turn the tables and have the complainers suggest a solution. Because every single time an approach to targeted child porn takedowns has been suggested, such as datacenter raids which would not affect as many people, someone is screaming about their privacy.
Child abusers evolve and are very happy if law enforcement doesn't.
So this means it is checking if you are sharing known CP images? That does seem to be much less invasive and problematic as there is likely no good reason to be sharing these images.
Yes, that's exactly it. It uses a database compiled by NGOs and specialized firms comprising of file hashes matching child porn. These lists are handled by humans.
Fuzzy means that it takes compression and the like into account, because even if just one pixel out of 20 thousand is different, the hash is different too. Fuzzy hash still recognizes it as the same image, so using an algorithm to alter the color etc. won't work.
That's also true for the no-fly list and the Terrorist Screening Database,[1] yet those are full of false positives. And unlike those lists, CSAM databases cannot be independently verified. To do so would require having the original images, which is illegal.
The no-fly list and the terrorist screening database aren't used in a court of law. The Confrontation Clause of the Sixth Amendment guarantees you access to all the evidence presented against you. You also don't need the original images to defend yourself, though apparently CP can be presented to a (traumatized) jury [0].
So if you're charged on the basis of a fuzzy hash matching, you'd subpoena Apple for the photo in your backup that matched, present it to the court (since it doesn't actually matter if it's CP or not to be admissible), and you win the case.
It’s not just checking files you are sharing, it’s also scanning files that exist on your device. The worry, or slippery slope argument, is that it’s one step away from scanning your device for other types of content, like memes critical of the government or just general wrongthink.
There actually are some edge cases even for matching against image blacklists. Google has experience with hitting them because it's used this type of image simhash for years (for shared cloud files at least).
The definition of child porn varies around the world. These systems use the US definition. This is not entirely what you might expect. For example, in the USA the courts have decided that cartoons can be child porn even though no actual children are in the picture. Most of the world does not agree with this, meaning an image can be CP in one place but not another. Is Apple going to enforce the US definitions or the ones where the user actually lives?
In the USA, photos an under-age person takes of themselves can also be considered CP.
What counts as a "child" for sexual purposes also varies around the world. Some countries have a lower age of consent than other places. In some parts of the world the age of consent and the age at which a child stops being a child for CP purposes are different, meaning that a teenager can have sex legally but if they take a photo of themselves doing it, they are trafficking in CP.
Finally, what is actually on these image blacklists? Hardly anyone actually knows because of the third rail nature of CP. Tech firms are often delivered image hashes, not even the images themselves, by third party 'charities' of various kinds and tech workers are - for obvious reasons - not normally given access to the actual pixels. Additionally, appeals from users are invariably ignored because people say "legal issues, it's complicated" and so everyone clams up. If FPs occur there is no way to resolve it and the people who see your appeal, if there even is one, won't be willing to actually look at the image to find out what it was.
It should be obvious how much potential for abuse this hands the people who actually manage these CP databases. Literally any image can be made verboten immediately, without any recourse, and basically nobody will ever find out including the people who shut down the affected users.
> So this means it is checking if you are sharing known CP images?
No. NeuralMatch was “trained” using 200,000 CP images. “Neural“ is likely a reference to the perceptual matching that it uses. It is not a bit for bit match.
Perceptual matching is a technique used for categorizing images based on characteristics and content.
The algorithm will scan your library containing new information and compare it to what it understands as CP.
Microsoft has been using PhotoDNA [0] to scan OneDrive content for quite some time. The news is that a device manufacturer is doing it on your device with local data. There's some MS research to use machine learning on metadata to identify offending material [1].
The specific subject of child abuse is irrelevant in my commentary. It was a commentary on the general category of AI, used all over, for many things, and more and more every day, but nice try.
Edit: You edited out what I was referring to as I was replying.
Being worried about privacy, establishing a precedent for scanning my data against a government database, and the risk of false positives with such an insanely emotionally charged crime is more than mere complaining.
The onus should not be on me to justify why this shouldn't be done. This is something new and it is perfectly fine to argue against it without needing to provide an alternative.
That being said, my solution is to continue to follow the process that law enforcement is currently using.
This all boils down to the classic "Liberty vs. Security" dilemma. While I obviously want to see more prosecutions for people who commit these crimes, a value judgement must be made about what it takes to achieve that.
> Child abusers evolve and are very happy if law enforcement doesn't.
Yes. And now they will evolve by developing a simple system to modify pixels in images when they copy and transmit that will easily defeat this hashing system. The only effect this will have is that moral panickers like you will have got everybody's privacy invaded over your moral panic of the day.
I have a hard time believing that this algorithm will be able to resist simple image manipulations while still being sensitive enough to avoid false positives.
I have similar qualms with this, because this is increasing the number of photos scanned by 2-3 orders of magnitude, and the number of false positives presumably also increases correspondingly.
Just saying "fuzzy hash" doesn't begin to explain how this would work. There are an infinite variety of algorithms along with arbitrary tolerances configurable. "fuzzy hash" just isn't helpful no matter how many times you repeat it.
I was a kid in the 80s and teenager in the 90s. My favorite thing during that time was pirating video games. A game would come out, and it was cracked, usually within hours and often before the game was even released to be sold. That's when "zero day" had a different meaning. All the "warez" ftp sites had a section for 0 day warez. The cryptologists and math brains would come up with new protection methods to protect their IP from being copied. Spend millions, probably billions for all of these projects. Yet, some kid in their basement with a commodore 64 was always able to crack them. Sometimes it would take longer. There were a few that took years, but once figured out, unlocked hundreds of titles previously secured.
This is, and always is, a game of cat and mouse. Law enforcement is always catching up. They are the cryptologists here. They are never ahead, always behind, because they don't know the new protections peddlers are using until they have been in use and later discovered.
No matter what vector you plug, they will use another, and the game continues (sick game). Maybe divide the image into 32 different quadrants and rearrange them, then put them back in the correct order when viewing through a specific image viewer. I'm sure that would bypass whatever detections they've come up in their fuzzy fingerprinting with as the entire image is now different. By the time they catch someone using this, they'll have already moved on to something different, as they always do.
I will never be ok with warrantless searches of my personal property, no matter the reason or justification or subject, and no matter who it is done by (government or private company). And I say that as a survivor of some pretty horrific shit as a kid to the point I fucking tremble with absolute rage when thinking about it 35+ years later. I would be banned from everything for life if I were to honestly state what I would do with these types of people. The movie "Saw" is tame in comparison. I have no compassion or sympathy for these sickos. But when reading world history, I can absolutely see the importance of "innocent until proven guilty" and Blackstone's Ratio "It is better that ten guilty persons escape than that one innocent suffer." Most of human history was the opposite, and it was brutal and full of literal witch hunts. Are we progressing as a species, or regressing in terms of human rights when it comes to technology?
You're spot on about the 0-day comparison. As always, something will arise to let people with the motivation hide illegal images. The problem is how it will be used against everyday citizens, who don't have sophisticated tools and maybe just share images of Hong Kong freedom protests, or books, or anything anti-totalitarian. The emotional appeals about this being about child abuse are absurd on their face because of how easily those people will hide. It's a good thing that some people are able to see through that as a ploy. We shouldn't have to go there and prove our bonafide hatred of abusers every time we justify our right to secure encryption or freedom from surveillance. Doing so almost validates the government's position. Just like in China you would have to say "of COURSE I hate the democracy protesters! I just think..." No, you shouldn't have to take a deep emotional dive into a history of abuse to justify your human right to privacy.
> And now they will evolve by developing a simple system to modify pixels in images when they copy and transmit that will easily defeat this hashing system
They don't use simple file hashes to match images, but perceptual hashes. That way they can find modified derivatives of a source image. The problem with this approach, though, is that this is ripe for false positives. Two completely unrelated images can have similar hashes.
could you use multiple perceptual hash functions with different salts, so that collisions would be less likely while allowing derivatives to be detected?
They aren't just matching exact hash hits, but are using a metric like the hamming distance between hashes to determine if one image is the same as , or a derivative of, another. The data structures that allow for efficient lookups rely on that metric, or another metric, for matching.
That reduces to just inventing a “fancier” single hash function. This adds no value or security in cryptography; it just makes things slower. I expect the same is true of perceptual hashes.
It makes sense for website owners to scan what's being uploaded to their servers, but that is totally different from scanning what's stored locally on people's devices.
There is zero AI classification involved, it’s a lookup vs hashes of known child porn images. You would need a hash collision, misidentified photo in the dataset, or legitimate use case to end up with a false positive.
There won’t be any collisions. It’s not a bit-for-bit Comparison. The algorithm uses machine learning to categorize images using perceptual match.
There will be many false positives, they will be reviewed by people. When there’s more than a few false positives, you will be investigated by the FBI.
The PhotoDNA platform doesn’t offer the participating organization raw photos, just hashes. The AI part is stitching photos with matching edges together, incase someone cropped them into different parts.
Again, Apple nor any company, have access to the source data, just hashes.
I hope it's hash-based and not AI. Training that NN would be one of the worst jobs I could imagine, you're adding commercial value to child porn and all the misery it entailed.
They trained a neural net to perturb images. Not classify them.[1]
They did add a classifier to iMessage. But it's designed to prevent children seeing any sexually explicit images.[2] There wouldn't be a reason to train it on images of children specifically.
They are using AI techniques as part of perceptual hashing. In other words it’s trying to match files with specific images rather than abstract concepts.
I’m pretty sure you don’t know what you’re talking about. Perceptual matching is not image matching.
Perceptual matching is used to sort categories of images. A quick DuckDuckGo will turn up many results. No stretch of the imagination will turn this into a bit for a bit comparison. This is a machine learning algorithm used to categorize images. https://www.ibm.com/blogs/research/2019/10/learning-implicit...
Perceptual hashing has a very specific meaning. It’s for verifying one file is directly related to a different file not an overall classification.
“Perceptual hashing is the use of an algorithm that produces a snippet or fingerprint of various forms of multimedia.[1][2] A perceptual hash is a type of locality-sensitive hash, which is analogous if features of the multimedia are similar.”
The goal is to verify a black and white copy of an image is identical to a colored original. Search algorithms want a similar thing so they can validate an image contains a blue car. However, a perceptual hashing algorithm must differentiate between different images containing a blue car while matching a photoshopped copy of the same image.
If they're using perceptual hashes, which they almost certainly are at least some of the time, then there's a huge space for false positives for each perceptual hash.
Ok. Hashed-based image rec. Where are the hashes? Is the comparison done in my phone against a downloaded database, or is a hash of every image on my phone uploaded for comparison on a server? Could this program then be expanded to other classes of images? Animal abuse imagery? Terrorism? Hashes of known bomb-making instructions? How about people sharing illegal pdfs and text files? Will the
MPAA be allowed to submit hashes of their copyrighted material? In short: What other warrantless government inspections of my files will Apple allow?
If apple really cares about security then those images in iCloud should be stored in an encrypted form that makes hash comparisons impossible. If they are hashing them then they have access to them in plaintext. If I used iCloud I would be more worried about the wider implications that creates.
iCloud needs to allow people to restore backups on a new iPhone after losing their old one.
You can setup secure encrypted backups, but the customer losing the key means losing the back so that’s not what consumer focused companies are going to do. In other words any backup service that doesn’t have big warnings that losing your key loses your backup means they can read your data.
"The mud puddle test: You don’t have to dig through Apple’s ToS to determine how they store their encryption keys. There’s a much simpler approach that I call the ‘mud puddle test’"
I was answering this: “Is the comparison done in my phone against a downloaded database, or is a hash of every image on my phone uploaded for comparison on a server?”
To be clear each image, the image’s NeuralHash, and a visual derivative are uploaded to iPhoto. This allows for the inspection of the NeuralHash algorithm used which I actually prefer.
From that same page: "Apple further transforms this database into an unreadable set of hashes that is securely stored on users’ devices". The phone is downloading the hash database.
It’s perceptual hashing not AI classification. “NeuralHash is a perceptual hashing function that maps images to numbers.”
The perceptual hashing is based on AI techniques. “The system computes these hashes by using an embedding network to produce image descriptors and then converting those descriptors to integers using a Hyperplane LSH (Locality Sensitivity Hashing) process.”
The difference is AI classification is based on defining something as say a “Cat” and then the AI spits some association with how cat like the image is. This extracts features from an image then compares lists of features to specific images.
It sure sounds a lot like AI to me. Saying its just hashes seems misleading to me.
From the PDF:
"The system generates NeuralHash in two steps. First, an image is passed into a convolutional neural
network to generate an N-dimensional, floating-point descriptor. Second, the descriptor is passed through a hashing scheme to convert the N floating-point numbers to M bits. Here, M is much smaller than the number of bits needed to represent the N floating-point numbers. NeuralHash achieves this level of compression and preserves sufficient information about the image so that matches and lookups on image sets are still successful, and the compression meets the storage and transmission requirements.
The neural network that generates the descriptor is trained through a self-supervised training scheme. Images are perturbed with transformations that keep them perceptually identical to the original, creating an original/perturbed pair. The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair. Similarly, the network is also taught to generate descriptors that are farther away from one another for an original/distractor pair. A distractor is any image that is not considered identical to the original. "
It’s closely related to AI, but it’s not image classification. The important thing is your trying to match specific files to hashes rather than features to an image so the output doesn’t need to be human readable.
Image classification on the other hand cares about if the image contains say a stop sign or a trash can. That’s useful for self driving cars etc.
Aka classification you might want to match two different bands playing the same song as identical. Where perception hashing would want them to be classified differently.
They need to stop calling it an iPhone. It's a govPhone. It now belongs to the government with "In-built Citizen Scanning" as free civic service. Thank you for serving the nation!
What I don't understand is how is it even constitutional in the US. Isn't it a warrant-less search? It's not like scanning content on a cloud. The cloud owns the device on which you store your data. But you own your mobile.
You're correct, but I would love to see a TOS that gives Apple permission to exercise cruel and unusual punishment for violating their terms. The eighth amendment only applies to the government, after all! /s?
Not that I am qualified to hold a legal debate, but as far as I know, if Apple implements this under the pressure from the state, and that it reports what it findings to the state, it is effectively an agent of the state and it should apply.
As far as I’m aware, the laws which require cloud service operators to scan for CSAM have been on the books for a long time, and none of the usual orgs have raised a constitutional challenge to them. I’m not a legal expert either, though.
But that's the big difference, cloud providers are scanning the data they hold on their device. Your iphone is not Apple's device. Cloud providers have a liability if they knowingly store illegal content, the data is stored on their property, Apple does not, it is your property. It's like if Ford was going to scan what you store in the trunk of your car.
The headline is misleading. The rumored change only impacts photos your device is uploading to iCloud, and it seems like Apple servers are a necessary part of the protocol.
It’s been a long time since I’ve set up my phone, so I don’t remember the defaults. If I recall correctly, photo stream (last ~100? photos) was on by default but not the entire photo library.
"Your iphone is not Apple's device". Well, is it not ? When you "jailbreak" it, who has the key of the cage ? How many network exchange does your phone with Apple do every day, every hour, every second ?
Oh, the irony. Apple users will finally demand that their phone be "their" device and won't say that they are perfectly happy with Apple making decisions for them because they absolutely trust Apple
The Constitution applies only to the government. A user's relationship with Apple is not in any way governed by the Constitution, just like how Twitter banning people who violate the terms of service is not a violation of the first amendment to the Constitution.
Well that is under the false assumption of you own your mobile.
Which isn't true for Apple. Not too sure about Android. But everything you do on your Apple Devices, including its Hardware, Software and its Services, works more like Apple lend it to you for a price.
And as other mention, only the government and not a private company.
Yes, not no to mention the gulf that exist between american culture of what is innocent and acceptable and the rest of the world.
If we apply the american paranoid standards to the rest of the world, i am pretty sure we will have plenty of false positives.
We really shouldn't be moving the conversation window into the technicalities just yet without discussing if this is even ok or reasonable in the first place.
Apparently, there's a Threshold Secret Sharing (TSI) technology in place to eliminate false positive. Besides, NeuralHash will basically match your photos against a database provided by NCMEC.
The key item in the article is that if enough content is marked contentious apple will automatically decrypt your phone for you and disable your encryption. So apple's not merely helping law enforcement find contentious content, they are enabling a back door that makes phone encryption essentially swiss cheese.
In 5 years: Apple and Google to scan your phones for politically contentious content and automatically remove it for you/report it to the authorities so you can get the proper help you need to understand what you should believe.
I've loved my apple phones for many years now, but it seems this is the end of my relationship with apple. I will be using an open source phone next.
Hopefully they are using phash to match the phone's images to known child abuse image hashes, because if they are using ML, I can only imagine what a false positive will do to someone's life.
Phashes can have collisions, and the false positive rate increases exponentially when you use a fuzzy metric like hamming distance to find derivatives of a source image.
This could backfire in a huge way. I think a lot of people, even EFF/privacy focused people are fine with a private company doing CSAM detection (or anything really) on uploaded user content since it is actually voluntary.
Now that CSAM detection is being deployed to user devices including those previously sold without it, there's a motivation to circumvent/block it. Unless Apple is doing CSAM scanning 100% inside a 100% secure processor, eventually someone will figure out the specifics of the current perceptual hash algorithm. There are already methods to make perceptual hash collisions. All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless.
Sure eventually they will improve the model/dataset but it just starts a cat & mouse game between Apple/Governments & honest citizens/users which actually benefits the people committing these crimes.
> All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless.
And even better: someone could create false-positives and just send them to people, or put them on websites. While you're watching funny kittens tumbling around, your iphone is calling the cops on you because it thinks you're watching child porn.
As in an activist might distribute them far and wide to sabotage the system, yeah.
If it's just a lot of people who get targeted individually, a moral panic might just wipe concerns away, saying "I guess there were more pedophiles than I thought" and "if they hadn't done anything, Apple wouldn't have found them".
But even if the courts strike it down, you'll still have all the trouble that comes with such accusations: the police searching your house, your employer, family, friends and neighbors learning that you're accused of possessing child pornography. Good luck getting back to a normal life after some forensic specialist confirms that it was a hash collision and the judge throws out the case.
Pardon my ignorance on the subject, but is it feasible to create a false-positive hash match? Isn't a hash directly correlated to the file content? So if the file content is changed at all, the hash should be completely different, should it not?
(1) You're assuming it's a "traditional" hash, as opposed to a perceptual hash. The former is purely based off file contents and thus any transformations applied on an image will lead to a new pseudorandom hash. By contrast, perceptual hashes are made to be able to still return a positive when a photo is resized, accumulated artifacts etc. This proposal is to use phashes, not hashes
(2) You have a subtle misunderstanding of how a traditional cryptographic hash function works. Such a function maps an infinite number of possible inputs to a finite number of possible outputs. To your point, flipping even a single bit will basically give you a completely new pseudorandom hash. However, given the infinite inputs and finite outputs, for every output (every hash) there is a countably infinite number of possible inputs that would hash to that value. Thus it's entirely possible to change the file contents and end up with the same hash, although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
irrc a cryptosecure hash is computationally infeasible to generate a new sample that has the same hash.
that is if you know hash (or reasonably small, ie only a percent of the total bit space), it's improbable bordering on impossible to generate a false positive .
> although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
Repeating that this process doesn't appear to be using cryptographic hash functions in the first place, so this conversation isn't relevant, but:
Only in the sense that "we don't know of a way to generate a new sample that has the same hash, yet". Unless we can prove P!=NP, we can't even prove that there exist hash functions that are actually infeasible to generate a sample with the same hash - so far we are no where close to knowing whether P!=NP or P=NP.
Even if we assume P!=NP, that doesn't mean that it is infeasible to generate collisions for our current cryptographic hash functions. They're not backed by some mathematical proof of correctness assuming <assumptions>, just the fact that so far no one has publicly figured out a way to break them, and the people who spend time trying think that their design is one that is unlikely to be broken in the near future.
There's no great reason to think that private groups could not have an attack against current hash functions [1], and there's definitely no reason to think that they won't find one in the future.
[1] There sort of is for SHA2, and that's that bitcoin has effectively created a giant bounty for breaking it.
Even better -- someone could remotely exploit their adversary with malware like Pegasus, implant actual CP on the phone, hide malware traces, and then the Apple's system would alert the authorities. Who would believe the protests of innocence when there is the evidence on the phone, and the infraction is so horrific? A Perfect crime.
The article states that the scanning is implemented as some kind of mutual operation between device and server when the device tries to upload photos into Apple’s cloud services. Presumably this is better than solely scanning photos on the server, because a client has even less control over what types of scanning Apple deploys there.
WhatsApp automatically saves photos sent to the phone to the Photo Album. Does this mean they will get a knock on their door if someone sent a bad image?
> Apple’s neuralMatch algorithm will continuously scan photos that are stored on a US user’s iPhone and have also been uploaded to its iCloud back-up system. Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse.
It's weird and invasive that they'd search users devices, but this makes it sound like they're just searching for files with certain hashes, which (to me) is fairly uninteresting. What I don't understand is why this is called "neuralMatch". Is it some sort of perceptual hashing (versus cryptographic hashing)? What exactly makes it "neural"?
If neuralMatch is to Google's reverse image search as Apple Maps was to Google Maps when it launched, this is going to have so many false positives.
> According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities
Is this separate? Or is the article just confusing iCloud storage for on-device storage? Regardless, there's some very lazy journalism going on here.
The name is certainly concerning. It's one thing to check for specific hashes. It's another entirely to do approximate hashing. Locality sensitive hashing even in its best implementations has tons of false positives.
It's also worrying that eg; someone could stealthily airdrop or iMessage a suspect photo to frame someone.
As any of the lawyers here can attest, the legal definitions here open to huge debates. Some material is unambiguous, other images reside extensive grey areas. The legality of material can even change based on context. Something that is perfectly legal in one area can be absolutely illegal in another. Example: If I had to list the most iconic photographs in world history, one of them is of a totally nude child (Vietnam). That isn't an illegal image, but try explaining that to a computer. At the moment these nuances are handled by cops, prosecutors and in extreme cases the courts. Is Apple going to put a marker down on a particular universal definition? Will that definition become the de facto world standard?
Material that is deemed illegal in jurisdiction A but not in B. Material for which someone was once convicted but if found on someone else's device would not be illegal. Material isn't added to these databases after sober legal deliberation. Police agencies can add basically anything they see fit.
This is a system for just one jurisdiction though? Is there talk about doing this outside the US? For multiple jurisdictions you’d need different sets of known hashes.
Imagine if Microsoft does this with Windows, lots of people would get arrested rightly so.
But like some people say this can be dangerous because evidence can be planted on your device. I remember reading articles how through torrent clients files could be planted but I never actually saw it in the wild.
In a world where all police were incorruptible and all laws were just, maybe I'd be ok with this. But since we live in this world, I feel like it's only a matter of time until the "think of the children" technology is coopted to scan for other "bad" images.
I wonder how long it'll take for China to compel Apple to add winnie_the_pooh.jpg to the set of naughty images for users in China and HK? And like we saw when bing altered the search results for "tank man" even in the countries like the US, at that point we'd be one configuration error away from those hashes leaking into the US dataset.
Privacy was only a talking point to get them some advantage over the Eye of Sauron (Google). I have no doubt that they were ready to jettison their privacy protection marketing angle at a moment's notice. You can't have access to people's data in the cloud and on their phones, and not use it in certain ways. To do so would be to leave money, and the good will of the government, on the table.
> Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities.
Haven't Apple previously claimed that their encryption works such that Apple themselves can't decrypt it no matter what? (IIRC this came up during the San Bernadino attack when the FBI requested that Apple decrypt/unlock a suspect's iPhone and Apple said they couldn't.) Is this not a change? Maybe I'm confusing iCloud encryption vs. iPhone encryption and the former doesn't have any of those guarantees?
It’s 2035, a major terrorist attack occurs. Apple scans all phones for location and images near the area over the last 12 months, sends the list over since the government declared Marshall law.
You are now suspicious until proven not suspicious. The real terrorists don’t even use iPhones. But here we are.
Edit:
Can any legal-heads explain how this does not constitute unlawful search? Apple is not the government, but once they hand the information over, isn’t the government indirectly committing unlawful search? Don’t you need a warrant for this?
____________
Second Edit:
Wanted to reply to another comment but being rate limited:
As of June 2016, the Terrorist Watch List [No-Fly-List] is estimated to contain over 2,484,442 records, consisting of 1,877,133 individual identities.
The number of Muslims in America:
A 2017 study estimated that 3.35 million Muslims were living in the United States, about 1.1 percent of the total U.S. population.
> ...ongoing demands from governments, law enforcement agencies and child safety campaigners for more assistance in criminal investigations, including terrorism and child pornography.
I get why they're making these demands, but this is in the US. This would clearly be an unreasonable search by Fourth Amendment standards if it were done by the government, so I'm not sure why government agencies think they can make demands that they, themelves can't legally execute of a private company. To be clear, it's legal for Apple to do this, and legal for the government to ask them to. The government is just laundering its door-to-door search.
It's also indicative of the unhealthy relationship these giant semi-monopolistic companies have with the government(s), that they implement the government's supposed will out of the fear of anti-trust actions/legislation, without any need for the government to pass legislation. Same goes for the demands to curb "misinformation" of various kinds, which basically constitutes government mandated speech regulation, even though the standing jurisprudence clearly forbids it.
389 comments
[ 2.1 ms ] story [ 289 ms ] threadAnyone not buying into these proven 1-2 years old technologies is crazy anti-vaxxer.
Same reasoning can be used with anyone opposing scanning your private images, after all you have nothing to hide.
carefulness regarding new medical procedures is also completely natural.
We aren't robots and we need to cut one another some slack. Different persons being mire or less risk averse is even a strength for society.
I would have editing a disclaimer on it but it's too late.
> and denying basic healthcare to your children is not, unless you plan to be instrumental in your child's demise and that comes natural to you.
You wrote that in reply to someone clearly not talking about vaccines generally but the new vaccines specifically.
also not sure how necessary is to take nude photos of children unless you are doctor and yes I am parent with 2 kids
taking care about your health and not willing to risk long term effects of unproven stuff someone is putting in your body seems perfectly natural to me
I mean, who's to say that my hash is actually what they think it is...
...and if it's not uploaded to iCloud for someone to manually review what the image is, is that going to be enough to get a search warrant for my phone?
"AUSA SOMEONE: But, Your Honor, The Defendant's phone has an image that matched a hash of a CSAM picture!"
"DEFENDANT: I'm not going to give you my phone's password."
"THE COURT: Bonk: Stay in jail for contempt."
I understand why they're doing it. I don't agree with it. At all, and I say that as a survivor of childhood sexual abuse.
>> I mean, who's to say that my hash is actually what they think it is...
Let's do the math. assume they are using a UUID for the hash. "For example, the number of random version-4 UUIDs which need to be generated in order to have a 50% probability of at least one collision is 2.71 quintillion" [1]
"The odds of This number is equivalent to generating 1 billion UUIDs per second for about 85 years." And that's for just 1 collision between two random photos. Considering that the pool of bad hashes is not going to be 2.7 quintillion but a fraction of that, this is more than adequate.
And of course they could use a longer checksum if that's the concern. (Not to take a stance on the parent issue, just to explain that hash's are probably a higher standard of proof than DNA matches)
1. https://en.wikipedia.org/wiki/Universally_unique_identifier#...
Then they could send you the new image and fuck up your life.
If we're talking about random files, sure.
What if the files are intentionally created to match the desired hash? The malicious actor doing this could be a private party intending to disrupt your life, but it could also be law enforcement as a means of gaining access to your device when it would otherwise require a warrant.
This seems a little alarmist to me.
So why would we open up new ones?
This is digital biowarfare material of the worst kind and should be handled as such.
Any researcher downloading this material to anything but lab machines should probably receive a visit from the relevant authorities to make sure they understand the seriousness of it.
That's called a second-preimage attack and is similarly infeasible to achieve against a cryptographically secure hash function.
There are good arguments to be made for why this is a bad idea, but hash collisions aren't one of them.
This is a function that is created to be robust against modifications to the image.
A perfectly robust function would - in my opinion - be one that could detect even a recreation of the original setup.
I don't know where on this scale the function law enforcement use is, but if it is any good I struggle to see how it cannot detect similar legal imagesm
Nope, federal law states they can hold you for 18 months. So long as they can claim to already know what's on the device (which they will, because Apple pinged them about it), they can hold you for 18 months.
Definitely a pretty thorny situation.
https://www.aclu.org/other/constitution-100-mile-border-zone
- Add the hash of a non-pornographic image to the database
- Using a burner email address, email the non-pornographic image to the target's Gmail address. The target wouldn't think anything of it.
- The innocent image would trigger a CP alert, giving law enforcement the pretense it needs to access the account
Now let's translate it to digital world and eliminate the non-scalable components - and you get AI "alerting" on the content of your account and it's a probable cause for searching you.
But the best part is of course the Apple doesn't even need a probable cause to search you - they aren't the police, so they are free to search anything at their pleasure and then report to the police. And this report, by itself, would be the probable cause, of course.
IIRC there was a teenage girl that lied about her father molesting her back in the 2000s. No evidence, just pure hearsay. Dude was locked up for like 10 years. Now imagine that on a greater scale with several videos and photos taken out of context.
Sorry, and as for "what is abuse", these are looking for exploited children. There are a lot of questions here, the morality of mass surveillance, the risk factors due to hash collisions (intentional or otherwise), etc, the one thing I doubt is an issue is a fuzzy line on "is this abuse"
I do think your comment is conflicting though. The fuzziness of 'is this abuse' is real because collisions can occur, mass surveillance never goes in reverse, etc.
Of course they have to be prevented from hurting people, to the best of our ability to do so, but punishment isn't going anywhere.
What is not a real concern is fuzziness in the question "is this image of child abuse". And I really don't want to continue a discussion where you try to come up with gray areas. A computer may not be able to tell the difference, but humans can.
https://quotepark.com/quotes/1911588-immortal-technique-the-...
I'm all for protecting children from being abused, but how are they going to filter what is normal and what is abuse without human intervention? And at that point isn't that a vast invasion of privacy to the perfectly innocent? It's different if it's online because that puts it in the public realm or on devices/servers owned by Apple, etc.
I am not sure the accuracy of these systems but even if they are pretty accurate there is something off putting about this trend.
This very much feels like a Guilty until proven innocent type of program... "if you have nothing to hide you have nothing to fear"...
Neither of those normally work out really well.
But this software doesn’t make arrests? A comparison would be police asking for what cellphones were in an area. There should be a lot of due process that happens before anything comes of this. I imagine police won’t look into one photo, they would want to match more than one.
I agree though, the scary part might be how to clear your name after an overreaction occurs, similar to the effect swatting has on innocents or other kinds of false accusations. But those would happen even without such a system - we’ve heard about such stories before. “Man held for child porn charges, but it turned out they were innocent,” - it doesn’t take this kind of system to have police look over your phone for evidence and find a different crime than expected.
That all said… child porn is really, really terrible and I’m actually in favour of this. As a society, we can decide on what’s acceptable. As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes. I do agree though, all these systems might make it harder to catch perpetrators if it pushes them offline. But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I suppose the fundamental problem is that having more evidence doesn’t necessarily lead to justice if it paints a false picture.
This also limits the possible avenues for future exposure - if one has less ways to distribute or get CP or the overall amount of it is reduced, they may never get into it in the first place and the market for it will shrink. Thereby reducing the amount produced over time.
This is a horrible idea, a massive regression in Apple (and therefore the broader industry's) stance on security/privacy, and the fact that you are dutifully swallowing the "child porn" justification is so incredibly concerning to me.
You're also being naive by swallowing the "it's only iCloud" justification. There is a clear intent to progress to scanning all local photos. After all, if photos must be match those of a government child abuse database, and any matches trigger manual review, what would an innocent person possibly have to hide? By that logic it'd be absurd to not scan just because it's only locally. After all, you're only catching bad guys! This is about protecting the children after all! /s
I oppose government-supplied blacklists. Doubly so when we can't even see what's in the database, but I would oppose them either way.
I don’t appreciate you calling me naive. There isn’t clear intent to scanning all local photos. In fact, it seems like the system was deliberately designed to only work with photos that are being uploaded to iCloud.
When someone is trying to rob you, generally the best thing to do is to hand over your wallet. If a law passes you should obey it for your own sake. That's an entirely orthogonal question to whether we as the users should be outraged or not.
Also while obeying an injust law, a company has a responsibility to legally challenge it in court with their resources, if for no other reason than to avoid the profit loss of having to comply with invasive regulations
Furthermore, it's not clear that this is the direct result of some federal law. It reads more like pro-actively appeasing their federal overlords.
It is good advice not to carry around huge amounts of money on your wallet. Someone might want it and have the power to get it.
Same with megacorps constructing and bragging about tools for oppression:
Better not do it.
Otherwise just be prepared that China will demand that you also upload their hashes of extremely bad photos - and one or two guys they want to get hold of.
It is kind of a Manhattan project for digital privacy - it is meant to bring a quick end to a actual real problem we have - but it has the potential to become a shadow over society for decades if not centuries to come.
I doubt they need, or want to match more than one. They'll use the reasoning that it only catches already-known child pornography, and that that person might have a whole cache of novel child pornography. Just look at the drug wars and how many people get arrested for trivial quantities of marijuana. And we hate child pornography way more than drugs.
> As long as there’s due process and a presumption of innocence, I’m all in favour of making it harder for criminals to get away with crimes.
The judicial system has a presumption of innocence. No one else does, or has any requirement to. The court of public opinion will ruin your life regardless of whether it's true or not.
That would be far less concerning if active cases were sealed, and only published if the defendant was ruled guilty.
> That all said… child porn is really, really terrible and I’m actually in favour of this.
I have a slight disagreement with this. The production of child pornography is obviously abhorrent. This system does nothing to stop that though, because it can only flag already-known child pornography.
This system stops the consumption of already produced and documented child pornography. While it's far outside my own interests, I can see some validity to the argument that consumption is of far less concern because it doesn't directly involve any abuse. The abuse has already happened, and no amount of tracking down pedophiles is going to reverse that. There are second order effects, like potentially reducing the demand for novel child pornography, though. Which leads me to this:
> But what other options do we have to stop these sort of crimes when the kids involved can’t or won’t speak up?
I would imagine this is far more likely to affect consumers than producers. In the drug market, stop and frisk is far more likely to net you users than producers or dealers. For one, there are more users than dealers/producers. For two, dealers, but particularly producers, are typically far more cautious than users.
That's a long-winded way of saying again that this doesn't seem likely to do much in the way of stopping people from abusing children.
But to directly answer your question: doing nothing is absolutely a valid option here. We do that for lots of bad things that happen when the costs of the solution are unbearable. It's what we're currently doing for famine in other parts of the world, which almost surely impacts more children than child pornography could even dream of doing. It's what we do for cars, which again, almost certainly impacts more children than child pornography does.
We wait until we can find a method with benefits commensurate to its costs. A 2016 DoJ report said that since 2002, 10,500 victims have been located and identified by law enforcement. That's 750 per year. There are 74 million children under 18 in the US, so that's 0.001% of children per year. 10,500 people is only 200 people higher than died from "accidental suffocation and strangling in bed" over the same period according to the CDC. The scale of the problem is the same, so should we go ahead and let Amazon send pings to the police if your Alexa thinks your suffocating in bed too?
If in the US Do you like the concept behind 4th amendment? being free from unreasonable searches unless probable cause can be articulated that you have committed a crime?
It is very scary how effect the "think of the children" narrative is to get people to willing give up their liberty
And the way it’s being done (hashes), a collision is highly unlikely. If it does occur it doesn’t mean it’s similar in nature (e.g. innocent picture of own child in bath). The hash isn’t looking at the image content in the sense of “what’s in the picture”, just the bits of the file. So it’s highly, highly unlikely, even if a collision occurs, that the collision would be an image that happens to be another child innocently bathing.
That's not good enough given how our media currently works. Imagine articles published when information about such a check leaks that say "celebrity X's phone checked by police for suspected CSAM." While that is the truth ("suspected") no one cares about that nuance and such a person would get cancelled very quickly, even if there was no evidence of wrongdoing.
Additionally, this isn't just a hash of the file but a perceptual hash on the image content. So e.g. changing a single bit in the image would create a different cryptographic hash, but generally not a different perceptual hash.
Are you sure, given how many iPhone takeover/jailbreak bugs regularly exist, including the recent 0-click iMessage bug? Would you like to dare a hacking group, domestic or foreign, to land a single verboten image on your iPhone?
> And the way it’s being done (hashes), a collision is highly unlikely.
The limited details so far are suggestive that its using perceptual hashes, which are more susceptible to collision-engineering/false-positives than cryptographically-secure hashes.
That's 115792089237316195423570985008687907853269984665640564039457584007913129639936. Probability of collision (two different inputs generating the same hash) is infinitesimal.
Actually, it's very easy to end up with an image that has a similar perceptual hash to an illegal image.
They are not doing MD5 hashing, they're taking perceptual hashes and then using something like the hamming distance or Levenshtein distance to make a fuzzy match with hashes from illegal images.
I've built products using these methods, and it is incredibly easy to make a fuzzy match based on perceptual hashes from two images that have nothing to do with each other.
It's absurdly easy to change the hash of a file. In the case of something like JPEG, you don't even have to change the file itself, you can just change the metadata. Apple could presumably only hash the image itself, but again, all you have to do is make tiny, imperceptible to humans changes to the image and the hash is totally different.
Long story short, this is either nearly pointless and privacy invasive, or it's about to get drastically more invasive to be effective.
Of course, there's no telling what images will be in the database when this inevitably expands beyond CP.
Edit: It makes sense that such a notification system would be impossible to implement. Perhaps, registered offenders should not only have their location placed on a map but their phones have certain privileges disabled or monitored.
The same does not go for random creeps wanting to take close up pictures of a child.
The creep behavior is the problem. The photograph is a symptom of it.
The list ballooned to over a million names, then required an ACLU lawsuit until humans became involved and a review process was initiated.
As long as the majority of Americans dont feel burdened by something, dont expect anything logical to happen.
They are using a library of human verified images to compare the hashes. Probably similar to PhotoDNA[0]. There can be false positives, but AFAIK the algorithms are not trying to identify naked children, but comparing 2 similar image hashes.
[0] - https://en.wikipedia.org/wiki/PhotoDNA
I didn't get that from the article, could you point to a source on this?
0: https://www.missingkids.org/theissues/csam#:~:text=Electroni... (ctrl-f "1,400")
So even if not running hashes against known porn, it could still be doing facial recognition. Wonder what kind of implications that would have for people in witness protection or battered spouses and other legitimately "disappeared" people.
“Users’ photos, converted into a string of numbers through a process known as “hashing”, will be compared with those on a database of known images of child sexual abuse.”
Also from the article.
‘Trained’ being important.
And what, in this context, is "your data?" Is a photo on an iPhone Apple's data?
I'd argue that the thing you should do if you get a hit is to delete your data, unless you want to spend enormous amounts of time and money defending yourself from prosecution by people who believe that false positives are vanishingly rare.
I really don't care about the semantics of data ownership. If Apple wants to scan photos you upload to iCloud so they don't run into a scandal years down the road that "iCloud is being used to distribute CP" then that's their call.
And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
No FBI is called. Imagine the financial damage this would do if specialist come over and try to collect evidence and the company is forced to cooperated and work for them, for free ofc. Only the big tech does this. They have whole teams for this and automated system and what not because surely the FBI wont walk into googled data center ever to collect evidence. But all the smaller companies they cant afford to do this and the detection system is actually to protect them form going bankrupt because they accidentally stored some illegal data from a user.
>And to the letter of the law you can't just delete the data if you found CP that one of your customers uploaded without running afoul of mandatory reporting laws.
Yes such laws exist. So they use filter software from third parties. They intentionally offer some generic filter that includes "unwanted content" and doesn't specify what it is when it find something so the companies can wave away the liability as they have no way to know if the image was valuable evidence of a crime or just some LiveLeak gore. They can justify that no human watches it because that's to expensive.
Thank God, I don't use Apple.
(I should probably delete them, since I've basically never used them... although they could be useful for one of those end-of-world Raspberry Pi builds: https://back7.co/home/raspberry-pi-quick-kit-one Edit: the non-bad ones, in case this gets misconstrued)
Edit: I've deleted my archive.
I mean, that is actually illegal. The problem here is the possession of child pornography.
If you're not trolling then you're being a fucking idiot, you could land serious prison time for having a shit ton of child porn on you. If you live in a country where it's illegal delete literally all of it of the hard drive, run DBAN on it, smash it to pieces with a hammer and then set the hard drive on fucking fire.
Literally zero courts are gonna believe "It's just a dataset for if I want to train an AI on it later XD". You will get horsefucked in court
But, yeah, I've deleted them now.
Hoarding porn is kind of useless nowadays, especially without any kind of selection and categorization, but maybe someone does it because it's their hobby (or obsession), who knows.
Also if a lot of people has a bunch of porn images scraped from forums it's quite possible that eventually some of that might become illegal as the laws change here or there around the world. Should they proactively delete everything? Sure, they could. To be safe.
But this kind of safety is absolutely the bad kind on the long term. (Though of course it's possible to advocate for reform while not storing content with uncertain legality.)
I don't allow image uploads at all now out of fear I'll back up my site and accidentally download CP. It's ridiculous, all it takes is one dumbfuck pedophile uploading a single file ( completely out of my control ) and I'll get thrown in prison for 10 years
There are words like “neural” and “trained” here, but no clear sign that there is actually any kind of image recognition that would risk false positives other than hash collisions.
Thought it was buck naked — but that works too. :-)
It's going to be used in China against enemies of the state.
Then it will be used domestically for the same.
Apple is a horrific company that you should stop using.
>The system that scans cloud drives for illegal images was created by Microsoft and Dartmouth College and donated to NCMEC. The organization creates signatures of the worst known images of child pornography, approximately 16,000 files at present. These file signatures are given to service providers who then try to match them to user files in order to prevent further distribution of the images themselves, a Microsoft spokesperson told NBC News. (Microsoft implemented image-matching technology in its own services, such as Bing and SkyDrive.)
https://www.nbcnews.com/technolog/your-cloud-drive-really-pr...
There was another round of outrage when the Windows 10 TOS extended this to local storage.
It also has the use case of scanning for DRM violations. (ripped mp3 files or movies)
These are still illegal in the US right now right? It's been a major criticism of these laws that they can hurt people accidentally.
It's fuzzy hash, not ""AI"", based. Cloudflare uses it too, and last time i checked the web is still functional.
https://support.cloudflare.com/hc/en-us/articles/36004610611...
How about we turn the tables and have the complainers suggest a solution. Because every single time an approach to targeted child porn takedowns has been suggested, such as datacenter raids which would not affect as many people, someone is screaming about their privacy.
Child abusers evolve and are very happy if law enforcement doesn't.
Fuzzy means that it takes compression and the like into account, because even if just one pixel out of 20 thousand is different, the hash is different too. Fuzzy hash still recognizes it as the same image, so using an algorithm to alter the color etc. won't work.
That's also true for the no-fly list and the Terrorist Screening Database,[1] yet those are full of false positives. And unlike those lists, CSAM databases cannot be independently verified. To do so would require having the original images, which is illegal.
1. https://en.wikipedia.org/wiki/Terrorist_Screening_Database
So if you're charged on the basis of a fuzzy hash matching, you'd subpoena Apple for the photo in your backup that matched, present it to the court (since it doesn't actually matter if it's CP or not to be admissible), and you win the case.
0. https://www.johntfloyd.com/the-difficulty-with-criminal-evid...
The definition of child porn varies around the world. These systems use the US definition. This is not entirely what you might expect. For example, in the USA the courts have decided that cartoons can be child porn even though no actual children are in the picture. Most of the world does not agree with this, meaning an image can be CP in one place but not another. Is Apple going to enforce the US definitions or the ones where the user actually lives?
In the USA, photos an under-age person takes of themselves can also be considered CP.
What counts as a "child" for sexual purposes also varies around the world. Some countries have a lower age of consent than other places. In some parts of the world the age of consent and the age at which a child stops being a child for CP purposes are different, meaning that a teenager can have sex legally but if they take a photo of themselves doing it, they are trafficking in CP.
Finally, what is actually on these image blacklists? Hardly anyone actually knows because of the third rail nature of CP. Tech firms are often delivered image hashes, not even the images themselves, by third party 'charities' of various kinds and tech workers are - for obvious reasons - not normally given access to the actual pixels. Additionally, appeals from users are invariably ignored because people say "legal issues, it's complicated" and so everyone clams up. If FPs occur there is no way to resolve it and the people who see your appeal, if there even is one, won't be willing to actually look at the image to find out what it was.
It should be obvious how much potential for abuse this hands the people who actually manage these CP databases. Literally any image can be made verboten immediately, without any recourse, and basically nobody will ever find out including the people who shut down the affected users.
No. NeuralMatch was “trained” using 200,000 CP images. “Neural“ is likely a reference to the perceptual matching that it uses. It is not a bit for bit match.
Perceptual matching is a technique used for categorizing images based on characteristics and content.
The algorithm will scan your library containing new information and compare it to what it understands as CP.
[0] https://en.wikipedia.org/wiki/PhotoDNA
[1] https://arxiv.org/pdf/2010.02387.pdf
The specific subject of child abuse is irrelevant in my commentary. It was a commentary on the general category of AI, used all over, for many things, and more and more every day, but nice try.
Edit: You edited out what I was referring to as I was replying.
Being worried about privacy, establishing a precedent for scanning my data against a government database, and the risk of false positives with such an insanely emotionally charged crime is more than mere complaining.
The onus should not be on me to justify why this shouldn't be done. This is something new and it is perfectly fine to argue against it without needing to provide an alternative.
That being said, my solution is to continue to follow the process that law enforcement is currently using.
If you knew how they approach this you wouldn't be satisfied either
Yes. And now they will evolve by developing a simple system to modify pixels in images when they copy and transmit that will easily defeat this hashing system. The only effect this will have is that moral panickers like you will have got everybody's privacy invaded over your moral panic of the day.
This is, and always is, a game of cat and mouse. Law enforcement is always catching up. They are the cryptologists here. They are never ahead, always behind, because they don't know the new protections peddlers are using until they have been in use and later discovered.
No matter what vector you plug, they will use another, and the game continues (sick game). Maybe divide the image into 32 different quadrants and rearrange them, then put them back in the correct order when viewing through a specific image viewer. I'm sure that would bypass whatever detections they've come up in their fuzzy fingerprinting with as the entire image is now different. By the time they catch someone using this, they'll have already moved on to something different, as they always do.
I will never be ok with warrantless searches of my personal property, no matter the reason or justification or subject, and no matter who it is done by (government or private company). And I say that as a survivor of some pretty horrific shit as a kid to the point I fucking tremble with absolute rage when thinking about it 35+ years later. I would be banned from everything for life if I were to honestly state what I would do with these types of people. The movie "Saw" is tame in comparison. I have no compassion or sympathy for these sickos. But when reading world history, I can absolutely see the importance of "innocent until proven guilty" and Blackstone's Ratio "It is better that ten guilty persons escape than that one innocent suffer." Most of human history was the opposite, and it was brutal and full of literal witch hunts. Are we progressing as a species, or regressing in terms of human rights when it comes to technology?
They don't use simple file hashes to match images, but perceptual hashes. That way they can find modified derivatives of a source image. The problem with this approach, though, is that this is ripe for false positives. Two completely unrelated images can have similar hashes.
If they're using fuzzy matching with perceptual hashes, then the space that false positives can exist in for each perceptual hash is huge.
People can argue against this approach for privacy reasons, but I think the false positive argument is a relatively weak one.
There will be many false positives, they will be reviewed by people. When there’s more than a few false positives, you will be investigated by the FBI.
Again, Apple nor any company, have access to the source data, just hashes.
They did add a classifier to iMessage. But it's designed to prevent children seeing any sexually explicit images.[2] There wouldn't be a reason to train it on images of children specifically.
[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
[2] https://www.eff.org/deeplinks/2021/08/apples-plan-think-diff...
Perceptual matching is used to sort categories of images. A quick DuckDuckGo will turn up many results. No stretch of the imagination will turn this into a bit for a bit comparison. This is a machine learning algorithm used to categorize images. https://www.ibm.com/blogs/research/2019/10/learning-implicit...
Matthew Green is tweeting about this: https://twitter.com/matthew_d_green/status/14230711866160005... and mentions that it is "preceptial hashing”
9 to 5 Mac article, in which they restate that it is not a classical bit-by-bit hash:
https://9to5mac.com/2021/08/05/scanning-for-child-abuse-imag...
“Perceptual hashing is the use of an algorithm that produces a snippet or fingerprint of various forms of multimedia.[1][2] A perceptual hash is a type of locality-sensitive hash, which is analogous if features of the multimedia are similar.”
https://en.wikipedia.org/wiki/Perceptual_hashing
The goal is to verify a black and white copy of an image is identical to a colored original. Search algorithms want a similar thing so they can validate an image contains a blue car. However, a perceptual hashing algorithm must differentiate between different images containing a blue car while matching a photoshopped copy of the same image.
I would hope most privacy conscious people disable iCloud, but that’s another story.
You can setup secure encrypted backups, but the customer losing the key means losing the back so that’s not what consumer focused companies are going to do. In other words any backup service that doesn’t have big warnings that losing your key loses your backup means they can read your data.
"The mud puddle test: You don’t have to dig through Apple’s ToS to determine how they store their encryption keys. There’s a much simpler approach that I call the ‘mud puddle test’"
[1] https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
To be clear each image, the image’s NeuralHash, and a visual derivative are uploaded to iPhoto. This allows for the inspection of the NeuralHash algorithm used which I actually prefer.
The phone isn’t downloading the hash database.
Citation: https://www.apple.com/child-safety/pdf/CSAM_Detection_Techni...
The perceptual hashing is based on AI techniques. “The system computes these hashes by using an embedding network to produce image descriptors and then converting those descriptors to integers using a Hyperplane LSH (Locality Sensitivity Hashing) process.”
The difference is AI classification is based on defining something as say a “Cat” and then the AI spits some association with how cat like the image is. This extracts features from an image then compares lists of features to specific images.
From the PDF:
"The system generates NeuralHash in two steps. First, an image is passed into a convolutional neural network to generate an N-dimensional, floating-point descriptor. Second, the descriptor is passed through a hashing scheme to convert the N floating-point numbers to M bits. Here, M is much smaller than the number of bits needed to represent the N floating-point numbers. NeuralHash achieves this level of compression and preserves sufficient information about the image so that matches and lookups on image sets are still successful, and the compression meets the storage and transmission requirements.
The neural network that generates the descriptor is trained through a self-supervised training scheme. Images are perturbed with transformations that keep them perceptually identical to the original, creating an original/perturbed pair. The neural network is taught to generate descriptors that are close to one another for the original/perturbed pair. Similarly, the network is also taught to generate descriptors that are farther away from one another for an original/distractor pair. A distractor is any image that is not considered identical to the original. "
Image classification on the other hand cares about if the image contains say a stop sign or a trash can. That’s useful for self driving cars etc.
Aka classification you might want to match two different bands playing the same song as identical. Where perception hashing would want them to be classified differently.
It is a perceptual Hash of the images characteristics and perceived continent by the algorithm, that’s the “neural” in neuralmatch.
Apple, for yours has been building-in machine learning dedicated chips into their builds so this shouldn’t affect battery life.
Matthew Green is tweeting about this: https://twitter.com/matthew_d_green/status/14230711866160005... and mentions that it is "preceptial hasing"
9 to 5 Mac article, in which they restate that it is not a classical bit-by-bit hash:
https://9to5mac.com/2021/08/05/scanning-for-child-abuse-imag...
Which isn't true for Apple. Not too sure about Android. But everything you do on your Apple Devices, including its Hardware, Software and its Services, works more like Apple lend it to you for a price.
And as other mention, only the government and not a private company.
How would that match a hash list of known CP imagery though?
In 5 years: Apple and Google to scan your phones for politically contentious content and automatically remove it for you/report it to the authorities so you can get the proper help you need to understand what you should believe.
I've loved my apple phones for many years now, but it seems this is the end of my relationship with apple. I will be using an open source phone next.
This topic already made it to the current frontpage.
Now that CSAM detection is being deployed to user devices including those previously sold without it, there's a motivation to circumvent/block it. Unless Apple is doing CSAM scanning 100% inside a 100% secure processor, eventually someone will figure out the specifics of the current perceptual hash algorithm. There are already methods to make perceptual hash collisions. All it takes is one security researcher to publish a tool that creates millions of false-positives for the current dataset/model used by Apple & others to be useless.
Sure eventually they will improve the model/dataset but it just starts a cat & mouse game between Apple/Governments & honest citizens/users which actually benefits the people committing these crimes.
And even better: someone could create false-positives and just send them to people, or put them on websites. While you're watching funny kittens tumbling around, your iphone is calling the cops on you because it thinks you're watching child porn.
If it's just a lot of people who get targeted individually, a moral panic might just wipe concerns away, saying "I guess there were more pedophiles than I thought" and "if they hadn't done anything, Apple wouldn't have found them".
But even if the courts strike it down, you'll still have all the trouble that comes with such accusations: the police searching your house, your employer, family, friends and neighbors learning that you're accused of possessing child pornography. Good luck getting back to a normal life after some forensic specialist confirms that it was a hash collision and the judge throws out the case.
(1) You're assuming it's a "traditional" hash, as opposed to a perceptual hash. The former is purely based off file contents and thus any transformations applied on an image will lead to a new pseudorandom hash. By contrast, perceptual hashes are made to be able to still return a positive when a photo is resized, accumulated artifacts etc. This proposal is to use phashes, not hashes
(2) You have a subtle misunderstanding of how a traditional cryptographic hash function works. Such a function maps an infinite number of possible inputs to a finite number of possible outputs. To your point, flipping even a single bit will basically give you a completely new pseudorandom hash. However, given the infinite inputs and finite outputs, for every output (every hash) there is a countably infinite number of possible inputs that would hash to that value. Thus it's entirely possible to change the file contents and end up with the same hash, although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
that is if you know hash (or reasonably small, ie only a percent of the total bit space), it's improbable bordering on impossible to generate a false positive .
> although (to your point) without doing any magic to intentionally cause a collision, effectively you will never get one because of just how many damn buckets there are
Only in the sense that "we don't know of a way to generate a new sample that has the same hash, yet". Unless we can prove P!=NP, we can't even prove that there exist hash functions that are actually infeasible to generate a sample with the same hash - so far we are no where close to knowing whether P!=NP or P=NP.
Even if we assume P!=NP, that doesn't mean that it is infeasible to generate collisions for our current cryptographic hash functions. They're not backed by some mathematical proof of correctness assuming <assumptions>, just the fact that so far no one has publicly figured out a way to break them, and the people who spend time trying think that their design is one that is unlikely to be broken in the near future.
We have managed to find collisions against hash functions that we previously considered cryptographically secure, that were designed with the same sort of standard in mind: E.g. md5 (https://en.wikipedia.org/wiki/MD5#Collision_vulnerabilities) and sha1 (https://en.wikipedia.org/wiki/SHA-1#Birthday-Near-Collision_...).
There's no great reason to think that private groups could not have an attack against current hash functions [1], and there's definitely no reason to think that they won't find one in the future.
[1] There sort of is for SHA2, and that's that bitcoin has effectively created a giant bounty for breaking it.
It's weird and invasive that they'd search users devices, but this makes it sound like they're just searching for files with certain hashes, which (to me) is fairly uninteresting. What I don't understand is why this is called "neuralMatch". Is it some sort of perceptual hashing (versus cryptographic hashing)? What exactly makes it "neural"?
If neuralMatch is to Google's reverse image search as Apple Maps was to Google Maps when it launched, this is going to have so many false positives.
> According to people briefed on the plans, every photo uploaded to iCloud in the US will be given a “safety voucher” saying whether it is suspect or not. Once a certain number of photos are marked as suspect, Apple will enable all the suspect photos to be decrypted and, if apparently illegal, passed on to the relevant authorities
Is this separate? Or is the article just confusing iCloud storage for on-device storage? Regardless, there's some very lazy journalism going on here.
It's also worrying that eg; someone could stealthily airdrop or iMessage a suspect photo to frame someone.
But like some people say this can be dangerous because evidence can be planted on your device. I remember reading articles how through torrent clients files could be planted but I never actually saw it in the wild.
I wonder how long it'll take for China to compel Apple to add winnie_the_pooh.jpg to the set of naughty images for users in China and HK? And like we saw when bing altered the search results for "tank man" even in the countries like the US, at that point we'd be one configuration error away from those hashes leaking into the US dataset.
All iCloud Data in China are already within their control. The whole DataCenter are done with "partnership" agreement.
HK user have their Data in US but I believe there could be a temporary server in HK DC before it is sent to US.
https://theintercept.com/2018/06/25/att-internet-nsa-spy-hub...
That would be Steve Job's Apple. Tim Cook's Apple will think they are doing it for the greater good.
The road to hell is paved with good intentions
Haven't Apple previously claimed that their encryption works such that Apple themselves can't decrypt it no matter what? (IIRC this came up during the San Bernadino attack when the FBI requested that Apple decrypt/unlock a suspect's iPhone and Apple said they couldn't.) Is this not a change? Maybe I'm confusing iCloud encryption vs. iPhone encryption and the former doesn't have any of those guarantees?
Apple has root on every iPhone. They can do whatever they want by pushing whatever code they like to any iphone.
You are now suspicious until proven not suspicious. The real terrorists don’t even use iPhones. But here we are.
Edit:
Can any legal-heads explain how this does not constitute unlawful search? Apple is not the government, but once they hand the information over, isn’t the government indirectly committing unlawful search? Don’t you need a warrant for this? ____________
Second Edit:
Wanted to reply to another comment but being rate limited:
As of June 2016, the Terrorist Watch List [No-Fly-List] is estimated to contain over 2,484,442 records, consisting of 1,877,133 individual identities.
The number of Muslims in America:
A 2017 study estimated that 3.35 million Muslims were living in the United States, about 1.1 percent of the total U.S. population.
Lol. Jesus Christ, you do the math.
No. That's the beauty of the third-party doctrine. Your rights disappear in a puff of twisted logic because you agreed to the fine print.
I get why they're making these demands, but this is in the US. This would clearly be an unreasonable search by Fourth Amendment standards if it were done by the government, so I'm not sure why government agencies think they can make demands that they, themelves can't legally execute of a private company. To be clear, it's legal for Apple to do this, and legal for the government to ask them to. The government is just laundering its door-to-door search.