Ask HN: How do I keep child porn out of my site?
A year or two ago I started an image sharing site that's been modestly successful in terms of traffic (a blessing). No money or fame, but it's nice to see movement.
I try to filter user uploads to at least classify the sexual stuff (80% of it) as nsfw and feature good stuff on the homepage. This is excruciatingly time consuming with 6000 galleries posted per day, but I suffer through it as I can.
Sadly, I've noticed a huge amount of extremely taboo photos on the site. From rape and bdsm, which I can kind of tolerate, all the way to extreme child porn. The latter is extremely disturbing.
Amazingly, these people post this openly.
I never see the press talking about the nsfw side of Youtube, Tumblr, Reddit, Imgur, and others. How do those sites deal with this problem? What kind of content filtering systems do they use to keep the visible parts of the site clean? How many interns are flagging photos all day long? Is it wise to allow these pages to be indexed? What's my legal burden under Safe Harbor?
And.. more importantly.. how does the organic traffic in the nsfw sections play into the strategy of these huge user-generated content sites.
NB. I've attempted to build user profiles and a kind of self-moderation system, akin to how Reddit flagging works, but my users seem to be mostly interested in "one thing," and no community-focused members have emerged so far. I still have hope, but need a solution that I can use now.
194 comments
[ 2.3 ms ] story [ 255 ms ] threadIf you aren't comfortable reporting it, what is your banning process like currently?
Also, post on your site that you report child porno. That should scare people off.
Now, if someone is uploading an image of their 16-year-old girlfriend taking a nude picture of themselves, that's still illegal but more of an ethical gray area, in which case I could understand hesitation at reporting to authorities. But based on OP's description, it's the real bad stuff, so reporting seems to be a no-brainer.
> if someone is uploading an image of their 16-year-old girlfriend
That isn't a grey area, that is wrong unless you have consent[0]. My point is that it is hard to tell what is and isn't child porn(hence this thread in general) and that an 18 y/o could look younger. What I think is possible is that someone could upload photos of THEMSELVES when they are either in the US or even in a country where that is legal and it is non-obvious what the age is.
Child Porn is fucking disgusting, I am not here to defend it. I am just pointing out how hard it is to identify it and that some people upload their own photos online. If you had a database that magically let you know if an image was illegal then of course I would say take it down.
edit: [0]I meant this generally, not that it would be OK to do this if she was 16, but if your gf was of legal age and you posted pictures of her or the two of you together.
Let's say they didn't have consent. It's wrong in the sense of it being "revenge porn", but should it really be considered as child pornography in the way it's classically viewed?
It's a gray area in that regard.
If someone is uploading things to his website which he can clearly identify as child porn without further analysis (if the child depicted is under 13, it's probably pretty easy to tell, unfortunately), I don't think he should hesitate to report it to law enforcement.
And if law enforcement investigates and learns it's not actually child porn... then they almost definitely won't charge anyone so long as no other crimes were committed. There are no downsides to reporting it and many downsides to keeping it to yourself.
Whether or not this is moral is irrelevant if it's not scalable. How about automated flagging of images for moderator review, and thinking about ways to reward or remunerate the moderators?
CV age detection should really be a thing, if it isn't already. Maybe there's an opportunity there if it's not already done, or way too hard?
I think this falls into the category of impossible. It is a hard enough problem for the human eye/brain. see https://en.wikipedia.org/wiki/Traci_Lords
http://www.missingkids.com/
Reporting to them is the same as reporting it to the FBI.
They have various APIs you can use, but you probably need to contact them to make sure you meet the requirements.
Anyway, here's your API https://developer-westus.microsoftmoderator.com/docs/service...
Guess you can be trendy AND help child rape victims.
That being said, if it is obvious that the image is CP and you have the IP address of someone who uploaded it, I couldn't imagine not passing on that information. Also, I would put a public disclaimer on the site indicating your policy.
edit: before people start flaming me as a rape apologist I want to clarify that notifying the FBI and giving away your users public information is a strong action. It is warranted in the case of CP (one of the few things I personally think) but that, as this thread is indicative of, it is difficult to identify IRL.
As the victim of child molestation; this absolutely needs to be reported. Saying anything else is sympathy for the lowest filth in existence.
is there any incentive to participate in your community?
With moderators you feed the "power tripper".
With karma you feed people obsessed with points.
This is a bit complicated: what if you had some sort of capcha that required users to classify images as nsfw/sfw/illegal?
I do support Twitter login which is fairly common, and have had a few hundred users sign up (out of like >10mil uniques), but I wouldn't say that they've exhibited overly engaged behavior after doing so. They're basically the same from a stats perspective from what I've observed.
I'm afraid to make user logins compulsory, especially considering the kinds of knuckleheads that are on my site.
How do I get from "eh, have fun as a guest" to "everyone's got an id" without destroying my stats?
I also question the assumption that taking steps toward accountability and community will hurt the numbers. Maybe in the short term, but if your site becomes a cesspool it will eventually drive your real users away anyway. If you figure out how to sustainably handle the filth, it may be possible to attract a much larger audience who would otherwise have been repulsed. Yik Yak is an example of that strategy.
If the user normally encounters photos on the site by requesting them (e.g., by entering a search query or browsing a friend's album) rather than having random photos thrown in their face (like HotOrNot.com), I would think you could run into some very upset users (and possibly legal problems) if you are throwing random photos that might contain disturbing images in their faces. I mean, if you go to a website intending to browse photos your friend took of his boat and the site throws up some random child porn on your screen, you'd be pretty annoyed, right?
prompt a capcha when someone attempts to `upload` a file
https://pbs.twimg.com/media/CA4S4lFWEAEJUUB.png:large
[0] https://www.crowdflower.com/type-content-moderation
For the future: Asking legal questions without stating your jurisdiction is... not helpful. :)
Yes, big sites employ a lot of people to clean content. I remember reading an article about poor people in $third_world_country that do this all day long.
I'd rather keep my money inside US borders, but it ain't cheap, and staring at the sickening contents of the internet's collective wet dream ain't exactly a high-profile career path for young folks.
"Hemanshu Nigam, the former chief security officer of MySpace who now runs online safety consultancy SSP Blue, estimates that the number of content moderators scrubbing the world’s social media sites, mobile apps, and cloud storage services runs to “well over 100,000”—that is, about twice the total head count of Google and nearly 14 times that of Facebook."
http://www.missingkids.com/Contact
"You didn't ask for it or seek it out did you? Someone else uploaded it to your server and you don't want it? Report it to us, then delete it once we've collected our evidence."
That's probably representative of the average agent's disposition, but make sure your ass is covered first.
This seems very ominous.
It does. Likewise with the casual questioning of guilt and suggestion that his/her answers will simply be taken at face value.
In any case, self-reporting does not absolve one of suspicion; else it'd be easy to get away with pretty much anything.
This, when it seems pretty obvious that they'd have to do some investigation of the host, if only to rule out his/her degree of involvement.
And I think you're also right that they'd do some investigation, but I imagine the investigation would be over very quickly. This "people are uploading filth to my website" situation isn't so uncommon like it once was (back in my day, sonny!!).
The chance that that individual is trying to buddy up to the FBI in order to escape detection is more of a Hollywood fantasy than how real life would play out. These agents are human beings too, and they know that if someone's coming to them to ask for help, they're all on the same page.
You're most likely correct. I'd add that even with the agents being human beings, there may still be some protocol that they are compelled to follow in vetting the host. I'd want to know that going in.
That's still not to say that this would end badly for the host. It's just that, given that the tone of the response is correct (even if shortened) , there is clearly more involved. And, when you have a representative of an agency with pretty broad powers, deep resources, and potential mandates soft-pedaling what's at stake, it can have a pretty ominous feel to it.
I didn't imagine that the FBI would literally have a 30-second conversation with someone who claimed to have child porn in their possession, with no actual follow-up or action steps discussed.
But I can see how that would seem problematic if you did take it literally.
I know the parent commenter said they would speak with a lawyer, I just wouldn't take comfort in a casual remark by an FBI agent.
In the US, you're likely to be left alone with all the associated costs. Help the cops all right, but if I do their work for them, I don't want to bear all the costs.
How about you donate all your time, money and resources stopping child abuse.
In any case if you create a platform (you possibly profit from) that is used to distribute child pornography you are faced with restrictions that the rest of the public understandbly isn't.
I see no reason OP shouldn't be legally compelled to cooperate in an investigation in at least the same way a witness can be.
That is exactly what you implied by mocking the fact that the OP even asked the question.
> I see no reason OP shouldn't be legally compelled to cooperate in an investigation in at least the same way a witness can be.
This is irrelevant to the question of whether the OP should look out for personal interests. You implied that one should not be so selfish as to even ask the questions about personal risks and costs. You mocked the very idea that he might ask such a question.
Before you make such a callous and judgmental comment, you should really think about what you would yourself sacrifice to cooperate with law enforcement. If you had, you would not have been so myopic. You would't question the seriousness of the question, even if you still thought that cooperation was necessary.
Volunteering to help stop child abuse and being compelled to participate in an investigation of unknown depth, breadth, duration, and resource burden (time, money, etc.) to you are two completely different things.
If you've never been involved in litigation or other legal situation wherein you couldn't just stop the process whenever you chose to, it might be more difficult to imagine the stress and costs involved, as well as the loss of control over one's own life.
It's nice to think that it's worth it at any cost (and at the sacrifice of one's other life responsibilities). But, of course, given that one can volunteer to make such a sacrifice without waiting to be compelled by a police investigation, then anyone who has not already chosen to do so might be wise to consider whether it's really a manner in which they can afford to help.
(EDIT: conciseness)
Also, the abuse already happened, you are only stopping the dumber CP collectors from sharing images of it.
Especially at a scale where you need automated systems to deal with the problem, law enforcement will inevitably notice sooner or later. I can't help but fell that it's not going to go over well with them (and it shouldn't), if they notice you deleted that content and possibly destroyed evidence in the process.
Technology companies and law enforcement have cooperated on this issue for a long time very successfully. They have experienced people working on nothing but this kind of thing and you're not going to deal with some local low level idiot that barely manages to deal with noise complaints. There is no reason to be paranoid and to believe they are going to act stupid.
- shut it down right now
- take a 'wait and see' approach. Then one day during the course of a bigger investigation they find that your server is hosting CP. Also, you apparently knew about it and didn't do anything (admittedly proving this will be quite difficult and unlikely, but still). In that case, they'll come down on you like a hammer.
Better be proactive. And if you're paranoid about the feds jailing an infrastructure provider who actively came to them asking for help (do you have any examples of where this happened? even just an investigation?), then all you have left is option 1.
NCMEC has protocols around how to report the images/video, and how to delete it on your end.
I would highly recommend against calling the FBI. You should work with NCMEC, as they have experience working with this stuff and their CyberTipline is one of the major ways that Congress has mandated that online service providers should report this stuff. Plus talking to law enforcement employed by the federal government has a host of risks associated with it:
https://en.wikipedia.org/wiki/Making_false_statements
Someone brought it to my attention that Bing's cache is full of CP, after the offending websites are taken down, Bing keeps the images for a long time. The Rapidshare sites are also full of it and they password protect RAR files so admins cannot peak into it. It is a major problem that has no solution for it yet. People run Wordpress blogs and spambots leave comments that link to CP sites.
This has become a hot topic issue because that Jared guy from Subway had a manager of his foundation that was found with CP, and they raided Jared's computers and found more evidence.
My ethics and morals won't allow me to look at porn, but it is a big industry. There are all kinds of porn out there. The CP is the worst of it, and a lot of children are trafficked as sex slaves for it. They grow up with a criminal record and sex offender record, and by the time they expunge the record they are in their 40s and can't find work. I was contacted by a woman who was in that situation on Github during the Opal CoC debates. She is trying to get out of her situation by programming and cannot find work because of it.
This CP stuff ruins the lives of the children who suffer abuses for it. Once they grow up they have a hard time in life trying to make ends meet. Some have serious psychological problems that are hard to treat and deal with.
I remember that in some cases the website is found responsible for the content that users post on their websites. Laws in your nation may vary on that. If you find illegal content you should remove it, least you be found liable for it. Make sure to report the IP address of the poster to the government or a non government agency that handles it.
A funny idea I had was to reverse the whole system - feed UGC content of a site that's supposed to be SFW into a porn site which is definitely NSFW, one that has lots of thumbnails. The ones that don't get any clicks to enlarge probably aren't porn and can pass the test :)
See also: SFW porn - where other images are super imposed on top of real porn. It's hilarious, but is it really SFW? You decide. http://www.reddit.com/r/sfwporn
Hopefully this can help you.
(Disclosure: I work at Microsoft but not on PhotoDNA.)
edit: I should clarify not child porn, but personal photos such as instagram and facebook which are private/semi-private and then are posted to public forums.
Really though it's not too hard to whip something up yourself. I did it for a bunch of those 'booru' sites (roughly 3 million images) like this:
- Find image hashing library (I used https://github.com/JohannesBuchner/imagehash but there's a nice series of articles here http://www.hackerfactor.com/blog/?/archives/432-Looks-Like-I... if you want to implement your own)
- Build a database of image hashes using said library
- Use an algorithm that allows you to lookup hashes by distance. In the case of hamming distance (used by many image hashes) you can just throw them in MySQL. You could also use any of the nearest neighbours search algorithms like k Nearest Neighbours or locality sensitive hashing (you'd want one of these for larger datasets)
https://expattmagazine.wordpress.com/2015/05/17/the-perils-o... http://www.wired.com/2014/10/content-moderation/ http://www.telegraph.co.uk/technology/facebook/9118778/The-d...
I am curious how one goes about developing a service like this without having to see child porn itself. Is there a database somewhere with known hashes? I'm assuming there would have to be along with a way to generate hashes yourself so you could test as I couldn't imagine running automated unit testing with real child porn.
I'm happy that Microsoft is providing this as a free service. It's going to be a lot less painful for me to use it than to figure out how to run my own (or in this case, figure out how to even get it).
Somebody please just give me access to the PhotoDNA code, the hashes, and a little funding. I'll make an API anybody can use for this. It's ridiculous how hard it is to do this. It's still easier for people to get spam IP lists than to see if CP is being uploaded to their servers. You can't just have it available for Facebook and Google or it doesn't work, you need to make it available to everybody in an easy, simple way.
Seriously, if you are connected with this at all or want to fund this work please email me, I am more than happy to work on improving this: kyle@neocities.org.
https://www.linkedin.com/pub/john-scarrow/6/2b8/354
What are they afraid of? That "pedophile hackers" will be able to reverse the hashes and get the images?
Authenticating access to the service is desirable for many obvious reasons.
Help me out with the obviousness please? Are those reasons more important than deleting child pornography from the web?
There are a lot more legitimate uses for a public repository of CP hashes along with free software for verifying them locally. Not only entrepreneurs and online community operators who don't want the stuff on display, but also users of poorly moderated online communities who don't want the stuff in their browser caches.
One way it to make contact with the police and get permission to list the names of the police agencies that are allowed to inspect the site via backdoor etc. Of course this might enrage some?? So some sort of middle ground might be to quietly approach the police for advice
[1] http://caffe.berkeleyvision.org/
The Laborers Who Keep Dick Pics and Beheadings Out of Your Facebook Feed http://www.wired.com/2014/10/content-moderation/
Anyway, I think your best bet is to outsource this kind of work to the sort of company described in the article. It seems to be a regrettable necessity for any sizable user-generated content site.
Also, of course, please try and get in touch with the relevant authorities mentioned in other comments and assist their efforts in tracking users who try distributing that kind of... content.
https://www.law.cornell.edu/uscode/text/18/2258A
http://www.ncsl.org/research/telecommunications-and-informat...
Google "safe image search" has the additional help of searching the content of the page the image is used. You might be able to do the same, up to some limit, by checking the http referer header field to know where requests are coming from. You could scan the referer's page for some keywords. This might give you a better idea of the context where the image is used. Note that this might be tricky, since you probably don't want traffic coming out of your server to some child porn site.
That said, those are just some ideas. Youtube has a good community that flags videos, but also an army of reviewer that look at the flagged content.
http://mobile.nytimes.com/2010/07/19/technology/19screen.htm...
Another way to look at it would be to try to manually select some images as "front page worthy", instead of trying to filter the bad stuff.
The fine for not complying started off fairly low, but has been increased in subsequent legislation. In my experience though, NCMEC is mostly just interested in getting regular reports uploaded to their system. I met with them once, and they have a rough sense for how many reports should be sent over for a site of a certain activity/traffic level, and if the number of repots is zero... then they know you're not in compliance.
Their reporting interface is beyond awful though. Maybe they've improved it in recent years; when I last saw it, everything had to be uploaded and reported manually.
I'd recommend sending new photos to a queue, and only letting them get indexed after someone from mturk has marked them as acceptable.
If the workers find something illegal, seek counsel and do as the counsel advises.
You're asking them to moderate unknown material from new users you know nothing about. That's something that every volunteer moderator from Reddit does.
The PhotoDNA API looks absolutely brilliant. That is one reason why I love Bill Gates he always (or a lot of the time) gets involved with projects that truly help people.
Don't think of it as a problem, it is an opportunity to help a child or a parent that may not know a relative, teacher, or stranger is hurting their child.
Many times these crimes are committed by loved ones and the children are not abducted, they are lured / tricked by people near and dear to them.
I would also announce to uploaders that you will be processing their images through the service. Including their ip address, etc.
That might help with just the volume of bad images.
But I do have some questions:
1) considering the headache - maybe shut down site for maintenance while implementing service.
2) process all images currently
3) consider that some images will inevitably be not in the database but be CP.
... therefore, at what point do you decide to just shut down the service.
It would suck to have to spend a lot on a lawyer to stay out of jail.
Its nice to pretend that the FBI would understand that you are an innocent victim - but what happens when those images end up on your machine (browser cache) and a fed prosecutor sees things differently?
But I'm wondering how much messaging you present indicating that illegal content is forbidden and will be prosecuted?
dHash is fairly simple to implement, and you might even be able to offload the hash checking at the database level. Comparing dHash's is just a matter of AND'ing the two hashes and counting the number of bits.
Obviously as the sample size increases so will the computation time. You could help the process by prioritising checks against new accounts, certain IP ranges (if you're seeing more or less content of a certain type from different countries or VPN providers) or if an account has a history of uploads in the past.
Its a horrible problem to have. Best of luck!
Edit: Other advantages: 1) you never risk viewing stuff that you can't unsee; and 2) you outsource content review to concerned users and other third parties.
But then what? Look at positives, and decide? Or just forward all positives to LEA, let them decide, and nuke what they indicate? LEA probably wouldn't like that.
Or don't look at it until a certain number of users flag it but still run it through the PhotoDNA. Now I am curious how imgur handles this problem.
I am curious how this gets handled. Having been goatse'd a few times with CP, I cannot imagine reviewing that crap on a regular basis.
> They say “the lawyers” tell them they can’t edit out an obscenity or remove a rude or abusive post without bringing massive legal liability upon themselves [...] That’s not true, and hasn’t been true since 1996.