80 comments

[ 3.8 ms ] story [ 137 ms ] thread
I am surprised their system for catching bad content is only user reports/flags.

While revenge porn and stuff is bad, there are a number of known child related incidents.

So if a private group of scumbags, are posting stuff to a private group, noone will know unless someone within reports.

Who needs the darkweb when u can have a private Facebook group?

Edit: as per replies below, they do use image recognition aswell for I guess known content lists

They also use image recognition to detect nudity and previously classified child porn.
No, they use Magic Number among a bunch of various tricks. You can't really call it image recognition, more like 'data-hash comparison.' To train true image recognition would require a LARGE subset of stored illegal pornography.
Why? Just use two separate datasets, one of images of adults and children, to learn age, the other of porn and non-porn, to learn what's "nudity" or "sex".
Knowledge transfer training is really hard.

If you can do this (with different categories, for instance, determining what is nudity, then determining black people, to automatically classify porn with black actors automatically), it would be quite impressive. Results have nearly 5% error on CIFAR-10, which is full of clearly distinct categories - planes versus boats.

> it would be quite impressive

Not only impressive, but likely very profitable.

> Results have nearly 5% error on CIFAR-10

Any idea what the error rate is for human classification on CIFAR-10 images?

Can you explain (just a broad overview) why that wouldn't work? I mean, it's not like black porn actors are categorically different from black people in general... in particular, if you have an algorithm that can locate the face, you could just pass the face of the porn actor to the other algorithm to determine age/race/whatever...
I mean if companies like Google train ML models on millions/billions of user's private data without looking at it, then surely it would be possible to train this in an ethical manner for ethical purposes. And via some Gov collaboration on the legal side
You are confusing image recognition with classification. They use recognition, similar to reverse image search services. Not really 'data-hash comparison', but rather 'image-hash', since it may see through re-compression and other minor modifications.
Defining "bad content" is difficult. I don't think we're nearly advanced enough avoid human intervention. Using user reports works well.

Here's an example: if you go on YouTube you can find harmless videos of parents bathing their babies, and I doubt anyone would call that child pornography. Same goes for something like a birth video, even though it shows a vagina. How do you detect the differences?

I think it'd be an improvement if Facebook were more open with their policies and allowed nudity, even if it required flagging content as mature. To get around this people end up placing tiny black bars in any of the "offensive" parts. But how does that seriously change anything?

I'd say it's pretty reasonable to request users flag content. Detecting it programmatically is currently impossible, so what other choices do you have if you want to allow users to upload content? Should only multi-billion dollar companies be able to have user-uploaded content? How would you handle this issue if you were building a small media sharing site?

> "54,000 potential cases... and 33 of the cases reviewed involved children."

That's 0.06% of reviewed cases, which is frankly less than I would have thought. I suspect that child abusers are a more savvy lot than the average chucklehead who does something like this.

I am sure there is a ton of stuff going on in the developing world were nobody is used to reporting anything.

Edit: Happy to see someone is leaking this stuff given how many flip flops we had to sit threw before someone admitted fake news was a thing.

I'd imagine that the low hanging fruits are picked off by law enforcement pretty quickly.
While others drag on for decades with establishment involvement - the U.K. being the current but not first example.
"savvy" = not using Facebook.
Yea, someone tried doing this whole 'sextortion' thing to me on FB.

I laughed at them. I used to work in porn, you think you're showing people something they haven't seen (let alone probably want to see again since I 'retired' [AKA got married?])

Sad that shameless people such as myself seem to be immune to this kind of thing. Perhaps everyone should have a career in porn just to eliminate this issue.

Give me a break.

>and the challenges for moderators, who say they are overwhelmed by the volume of work, which means they often have “just 10 seconds” to make a decision.

Maybe Facebook should just hire more moderators? I know they'd love to totally relegate this to some awesome NLP algorithm but we're not there yet. Facebook makes more than enough money to bring on some more employees (and clean up their guidelines).

EDIT: oops, had two articles from the guardian on FB up and referenced the wrong one.

"Maybe Facebook should just hire more moderators? I know they'd love to totally relegate this to some awesome NLP algorithm but we're not there yet. Facebook makes more than enough money to bring on some more employees (and clean up their guidelines)."

There are systems that can detect porn pretty reliably. Helping moderators spot any porn more quickly will also let them spot revenge porn more quickly. Any correct or false positives can be fed back into detection systems. It might still take more moderators, though. Facebook needs more anyway given the size of user base and money available for moderation.

> Maybe Facebook should just hire more moderators

They are doing exactly that this year

It's disappointing, but not super surprising.

Shit like this is why I'd never want to operate a site with a large user base that can freely upload content. Sure you could probably afford the servers for awhile with minimal monetization, but could you pay for even a half-dozen policy experts to moderate the content to a point where you don't send your largest potential audiences running screaming for the hills?

I had a female friend last month whose ex-bf created a fake account and proceeded to set the profile and public photos and videos of them (having sex). He then went ahead and started sending friend requests to her entire family and friends. I felt so bad for her - it was awful enough to make her even have suicide thoughts.

What really surprised me was how horrible the FB system to report stuff like this is. It is basically non-existant. If this thing takes 1 month to be reviewed, that is 1 (eternal) month where the damage can be irreversible.

what is worse is that FB has posts about how they are "fighting" this without having any instructions or obvious process defined to report this. So you "report inappropriate content" and the report goes to a black hole while the perpetrator continues with their shenanigans. They dont even need to create a new acct, because they can keep going without the offending account getting blocked.

Something else is that the non-profit foundations that exist to fight this (the ones that will send legal take-down notices) are US-centric. This means that if you are from a latin american country and you face this (as was this particular case), you are screwed.

Yes obviously the lesson is not to take pics like that in the first place, but sadly a lot of the victims may not be that computer savvy or mature enough to realize what a mistake those things can be.

This isn't about FB solving the problem, it's about FB announcing the existence of their own internal bureaucracies. That is, they won't solve the problem, but they'll create an email address specifically for the problem.
This guy is in the videos as well, right? Could she somehow strike back harder and demolish his reputation? Tell his friends and family, and his employer, what he did to her? Who the hell would send a video of themselves having sex with someone to that person's family?
Unfortunately, in a lot of cultures and environments today (where culture is not only your global, national, or regional culture, but also the culture that you choose to surround yourself with), this kind of thing is much more devastating for women than for men.
I'm not saying that the existence of porn is supposed to hurt his reputation. I'm saying the fact that he would violate somebody in this way should hurt his reputation. The porn is just evidence of his doing this. I'm in no way trying to diminish this lady's situation. On the contrary, it makes me angry and I would want to see this guy see some serious consequences for it.
> This guy is in the videos as well, right?

I suppose that would depend on how well the guy knows about digital editing of video.

...for that matter, if you wanted to go the opposite way, and you had the right software (and a few other skills) - one could seriously put that guy in a very compromising situation by more digital editing...

You were downvoted... but you make a point that I (as a woman) would have no problems following.

I would call / message / text his mom. His sister. His boss. His dad. The new girlfriend.

Not sure what the norm is, but I reported a fake account using my wife's picture (just a single, regular portrait) and it was removed within 5 minutes.

A month would be ridiculous. I wonder if that happens often?

I report spam & fake accounts all the time and get responses back before the next day.
I regularly get friend requests from scantily clad teenagers in a city I no longer live in, whose profiles are full of links to porn sites. I report them, and a few weeks later get back a message saying 'we reviewed your report and found nothing objectionable'. There isn't even a report category that seems accurate - I usually pick 'fake account'.
There must be some real dullards working on the mod team.
> Yes obviously the lesson is not to take pics like that in the first place, but sadly a lot of the victims may not be that computer savvy or mature enough to realize what a mistake those things can be.

In a lot of cases pics like that are not even taken voluntarily but unknowingly.

could you elaborate on that? Honestly curious.

EDIT: about the issue, not the materials in question!

I presume the parent means where one of the participants sets up a hidden camera - for whatever reason (revenge, blackmail, self-gratification, monetary gain, etc).
Yes, something like that. Or when the victim is wasted or deliberately drugged.
So The Guardian puts this article up just as May's government is promising to tighten government restrictions on social media. That was convenient.
The Guardian isn't a right-wing paper, it's left of center, and I can't remember when it last endorsed the tories. You'd be far more likely to see that kind of thing from The Telegraph or The Daily Mail.

There's another explanation much closer to home which doesn't involve Theresa May: The Guardian, like many other newspapers, is under intense pressure from Facebook which is eroding their direct traffic in favor of traffic coming from FB. They literally compete against each other for clicks and ad traffic.

Sure, but the timing is suspicious. The editors at The Guardian aren't above giving a Tory proposal a little push if it's in their interest. Besides, Labour is proposing essentially the same thing.
Some problems run deeper than their individual manifestations such as these for which one may propose techie solutions -- here, sex ed is the only permanent solution. It's time to find ways to explain to boys that sex is an ordinary way to have fun and/or connect romantically between adult people, rather than a source or arena of power or status as it is endlessly portrayed in media.
Sex and power/status are inextricably linked for much of society.
Society implies culture thus construct whereas this feature is observed in much (all?) of animal kingdom as well.
To boys? really?

Have you ever read the cover of Cosmo or seen clothing with blatant cleavage?

The power/control dynamic extends in all directions & I have no doubt that some of this revenge porn extends in the other direction..

They should have a verification process for accounts beyond verifying an email address. Verify identity, it's way too easy for people to be impersonated, there's a whole TV show dedicated to this...
You need a phone number to register a new Facebook account...
They're free with Google Voice
I don't think Google Voice will work for that nor will a Trello number. I saw someone complaining on HN that their landline didn't even work.
So here's my thing: Imagine a world where facebook didn't exist. Would it have been impossible for this guy to get their photos and videos, post them on the internet?

I guess facebook does make it easier to email things to family and friends - but it only does so with the user's consent - you have the option to make your friends list private.

It seems to me like this guy is a real creep, and he should certainly be punished to the full extent of the law. But your friend also seems to have made a poor decision in allowing someone who is a creep to take very personal photos and videos of herself which could cause enormous humiliation.

The law should certainly step in to prevent stuff like this. But I question how much private companies should have to step in to protect people from their own bad decisions.

Yes facebook should certainly take effort to protect it's community standards and prevent porn and revenge porn as a part of it's duty to the community - but does facebook also have an affirmative duty to protect people from outcomes that are a result of their own poor life decisions? Does it owe an affirmative duty to your friend specifically to get rid of all revenge photos about her no matter whom she and how shares her nude or sexual photos? Or is there a duty on your friend to protect herself and exercise discretion in who she shares such content with?

Isn't there a duty on your friend to exercise discretion as to whom she sleeps with and whom she allows to take discreet photos and videos with. When she took these photos and videos, she knew there was a possibility that they may be used negatively, so does it negate her responsibility and become completely facebooks problem when those photos are abused by those who misuse their platform?

I mean imagine a case where this guy found her bosses and family's phone number and texted the image to her family? Does that then become AT&T's problem to go in there and clean up those images and moderate them?

> But your friend also seems to have made a poor decision in allowing someone who is a creep to take very personal photos and videos of herself which could cause enormous humiliation.

Yeah, this is just classic victim blaming nonsense.

Sure, it's good advice to avoid recording anything you don't want eventually published—this goes for conversations as much as it does 'personal' pictures and videos.

But that's entirely irrelevant here and sharing 'personal' pictures with your significant other is normal.

(comment deleted)
She dated the wrong guy and trusted him too much. That's fine, lots of women make that mistake and she shouldn't be shamed or blamed for that. I never said she should be. All I am saying is that it's not facebooks responsibility to fix that mistake for her.
She didn't make a mistake. This is entirely the guy's fault, not hers.
The parent's central point isn't that the the victim is to blame here. It's that this sort of harassment is very much possible even without facebook existing.

Being a victim does not make one immune to criticism. Agreeing to produce material that you don't want to be publicized does carry a risk. Not talking about this risk and shoving it under the carpet isn't going to help solve the problem. It's very much relevant here, as that is one key reason that enabled this whole drama. It may be increasingly normal for people to produce such material, but as shown by the article it's also increasingly common that this material is abused. Being more careful with production and storage of such material is a perfectly valid strategy to avoid this type of abuse. It shouldn't be ignored just because it can also be viewed as a mistake by a victim.

(comment deleted)
I didn't downvote you, but I think your frame of reference here is awful. Victim blaming in these cases really understates the harm caused by these actions, and the lack of agency that the victims have in preventing them.

If the same boyfriend had stabbed her, would it be her fault for dumping someone who might stab her later? Of course not. The victim has a legal (and ethical) right to take whatever photos she deems appropriate, and then control their dissemination later.

FB isn't responsible for every embarrassing thing on their platform, but they are responsible for responding to "Report of a crime" combined with a nudity filter flag ASAP. If OP is true that it can take over a month, that's not acceptable.

She shouldn't have to make her friends list private to not be the victim of harassment. In this case I don't think it would matter anyway, as he likely knew the names of her family.

Your texting example isn't exactly 1:1 because we don't think of those items as being uploaded publicly to AT&T's servers, although keep in mind if that happened she could still roll into a police station and get him charged with harassment, and more serious crimes in many US states that now have specific statues against revenge porn. If AT&T was hosting the texts on a server, then yes they should be responsible for illegal activity hosted on their platform. (I'm not sure how US/EU common carrier regulations apply to texting, so they may not be).

(comment deleted)
(comment deleted)
I don't think it's really wrong to consider what responsibility a victim has in their fate. If you know that touching a kettle with a barehand may burn you, you are to blame when the kettle burns you when you touch it! You can blame the kettle, as children do, but it's not helpful, so we teach and learn not to touch it. The kettle doesn't have agency, but I suspect that for many who 'victim blame' they see crime, disasters etc as forces of nature that don't really make sense to blame as we can't readily change them, but can predict them. Few blame babies for the mishaps they encounter because they do not know better. At some point, this changes. We can and do try to control our environment to prevent these incidences where we can, but in many ways it is wiser to prepare ourselves. Don't pave the road, wear shoes...etc etc.

  Revenge porn is a well known tactic. I'd be surprised if she didn't pause to consider the ramifications when the media was made. As per your example, I think it depends on what she knew apriori... if she could reasonably assume that he was a psychopath who may murder and she dated him and broke up with him anyway, we'd probably give her some share of the blame. From another example, If you walk through a war zone knowing it's a war zone, maybe it's stupid to blame the soldier who shoots you?
I want to live in a world with more "paved roads" and "seatbelt measures", even if it makes Facebook's job harder.
I wanted to reply to the original response which is now dead so I can't respond to it, but I think it fits here as well:

The point would be the same if he had taken those photos/videos without her knowledge. What'd be the suggestion then? Don't ever have sex in a room you didn't search thoroughly, and if you have to go to the bathroom, ask your partner to leave the room first, lock the door, have them wait while you're in the bathroom, and then go back in?

> I mean imagine a case where this guy found her bosses and family's phone number and texted the image to her family? Does that then become AT&T's problem to go in there and clean up those images and moderate them?

Facebook wants to see my ID after 9 years of membership and 100+ RL friends, just because some random clowns who sure as fuck didn't show their ID probably don't like my political views, and how good I'm at expressing them. So something like that is their problem, some random people clicking on "fake profile" or whatever and then they just HAVE my papers, and pronto. But a pretty clear cut case of revenge porn, not touching that with a barge pole? Heh. Stuff like that makes it clear Facebook isn't an utility like mail or roads, and isn't even trying. And you can't be half pregnant with this. The "life decisions" people make are kind of orthogonal to that.

Not sure kettle is the right analogy here. It's too simplistic. As a product Facebook is vastly more complex beast, and the argument is much more naunced. Many feel compelled to use Facebook. 99.99% didn't read the Terms and Conditions (and BTW that's by design) and aren't aware, for example, what Facebook is allowed to do with their content. Most aren't aware that it's not in Facebooks interest to make it easy for you to report revenge porn; it's costly for them requiring manual review and against their general principle that all your content is theirs and will be stored forever. And that's just for starters
Some of your questions come very close to blaming the victim, don't they?

What the real shame here to me is that nude photos/videos are something to be ashamed of or embarrassed about in the first place. Because from that premise, now you need to be careful about what you allow to be filmed and by whom. It is especially sad that women apparently need too worry about this a lot more than men.

>Because from that premise, now you need to be careful about what you allow to be filmed and by whom.

Yes, exactly!! You do need to be careful about what you allow to be filmed and by whom. That is the substance of my comment. I agree that this is unfortunate. Maybe in some wonderful utopia to come we won't have to worry about such things, but in this world and under current circumstances, unfortunately we have to take steps to protect ourselves from the possible evil deeds of other men and women.

You still shouldn't blame the victim. The problem here is the perpetrator. He probably would have found some other way to embarrass his girfriend even if she hadn't allowed the pictures/videos.
You're kind of forgetting that a lot of these kind of recordings are not taken by the victims knowledge. And that they can literally destroy someone's life in the future - especially if they're entering a public profession like teaching.
> What the real shame here to me is that nude photos/videos are something to be ashamed of or embarrassed about in the first place.

If it were simply nude photos or videos, then I could wholly agree with you. But it's not like this in all cases.

In many cases, these photos and videos are much more explicit; and while one shouldn't be ashamed of such videos, I can see the embarrassment aspect of them - there are some things that should remain in the bedroom between the participants unless they all agree to its public release.

Furthermore, these instances aren't just to friends and family, but usually employers as well. It could mean a loss of income or worse for a person. If it goes to people or leaders of their religious community, it could cause some devastating repercussions, depending on that community.

> I guess facebook does make it easier to email things to family and friends - but it only does so with the user's consent - you have the option to make your friends list private.

An ex-boyfriend wouldn't know the name of his ex-girlfriend's parents? Wouldn't be able to recognize them in the profile photos?

> But your friend also seems to have made a poor decision in allowing someone who is a creep to take very personal photos and videos of herself which could cause enormous humiliation.

He's likely only a creep in hindsight. And many things you share with people have the potential to cause humiliation. Opening yourself up to others is dangerous - but necessary.

> So here's my thing: Imagine a world where facebook didn't exist. Would it have been impossible for this guy to get their photos and videos, post them on the internet?

Uhm, yes. There are a lot of venues on the internet flooding with this type of content.

Facebook disabling 14,000 accounts in a single month because of that kind of terrible behavior does show this is a serious problem. But it's still a very rare problem. Given Facebook has 1.19 billion monthly users, that's on the order of 1 in 100,000 accounts.
That seems high to me. We should strive for this to be so rare that it's news-worthy when it happens at all.
Well, it's really just an extension of gossip, blame, scandal and shaming that has been around forever. There would need to be a major social change first. Otherwise, even the perfect tech solution would just force this bullshit to move somewhere else.
Somebody check my math:

Facebook has 1.13 billion dailies, and 1,800 potential daily revenge porn incidents: 1 in 627,777. Involving children, about 1 per day.. 1 in 1.13 billion?

Is flooded the right word to describe that? It actually seems pretty low to me.