Ask HN: What do you use for content moderation of UGC?
My co-founder and I started a static site hosting platform. From our past experience in user-generated content, we knew we needed moderation. So, we wrote a very simple script that checks for common phishing attempts and alerts us.
However, this simple script does not catch some of the more advance phishing sites and it doesn't catch other types of content that we can't support on our platform like porn.
Curious if anyone has tips and tricks for content moderation? We're still going to be manually reviewing sites because we haven't reached a scale that makes that impossible. But automation is nice.
45 comments
[ 2.5 ms ] story [ 110 ms ] threadThat makes approximately all business ideas to host user generated content non-viable. The conflict is dynamic and you are the Maginot Line...except that any breach of laws creates a potential attack by state enforcement agencies too.
To put it another way, ASCII files and a teletype were enough to see pictures of naked ladies. Good luck.
for any site hosting user-generated content, how efficiently you can run your moderation system is essentially your unique selling point - LLM is the baseline.
Google Gemini 1.5 Flash 8B charges $0.04/million input tokens and $0.15/million output tokens.
If a piece of content that needs to be moderated is 1,000 tokens (that's pretty long!) and you expect a 10 token return it will cost you 0.0039 cents - that's not a dollar amount, that's less than a 250th of a single cent.
So 1 cent will moderate 250 items of content. $1 will moderate 25,000 items of content.
LLMs are dirt cheap. You can play around with pricing across different models using my calculator here: https://tools.simonwillison.net/llm-prices
The alternative is paying a person…
A phisher may insert text for an LLM with a disclaimer that's only an educational example of what not to do, or that they're the PayPal CEO authorizing this page.
I don't know if they're using a LLM specifically, as opposed to say computer vision models or some other method.
Pricing: https://azure.microsoft.com/en-us/pricing/details/cognitive-...
AWS has something similar: https://aws.amazon.com/rekognition/content-moderation/
https://platform.openai.com/docs/guides/moderation
Wouldn't a solid set of processes to handle content complaints and knowing who your customers are in case the hosting country's law enforcement has a case suffice?
Or are you having some free tier where users can anonymously upload stuff?
In the latter case — a free place to stash megabytes — you'll need to detect password protected archives in addition to unencrypted content. Get ready for a perpetual game of whack-a-mole though.
[1]: https://news.ycombinator.com/item?id=42780265 [2]: https://oxcheck.com/safe-api
I'm aware of eg NSFWJS, which is a tensorflowJS model [1]. Is there anything else that, say, can also do violence/gore detection?
[1] https://github.com/infinitered/nsfwjs
One problem is legal liability around training and testing. If you use a large foundation vision model, you could describe your moderation criteria in text and cross your fingers, but those aren't going to be running client-side. If you want a compact, efficient model to embed into an app, you need to train it.. but with what? There's a dilemma here where it's easier to build filters for legal things because you can't legally use the illegal things to train a filter for the illegal things! Systems like NSFWJS work because run of the mill porn is legal in most jurisdictions, but an equivalent "CSAMJS" filter would be a legal nightmare to produce.
That being said, it might be possible to get to "better than nothing" with transfer learning, by training a model to detect both "there are children here" as well as "this is pornography". I have no idea what the success rate would be, but there have been some pretty impressive generalization results in recent years with classification models.
Spam filters on the recipient side are already viable; there's no advantage to having them on the sending side where they can be more easily bypassed. If the recipient does intend to receive illegal content, then they will simply arrange to use a communication method which does not have scanning on the sender's side. Whether or not encryption is used is immaterial.
What about group chats? Again, if the group doesn't want to receive illegal content then they can use a bot account for spam detection. End-to-end encryption doesn't change the situation at all.
I'm weary (if not wary) of discussion on this topic because it always comes down to the idea that developers of communication methods have a responsibility to read and potentially censor everything that is sent by that method. Yet that was never the usual expectation for snail mail; indeed many countries have laws against reading other people's letters. It was also never the expectation for the Internet back when it was being designed. So by all means discuss the relative merits of censorship, but the existence of encryption is not a valid argument one way or the other!
I believe the ask here is for receive-side moderation, similar to an ad blocker. If the messages are encrypted, the platform can't moderate them. Consumers would do their own content moderation locally and discard/hide/clickthrough/etc content they don't want to see, without actually removing it from the platform.
Eg I'm sure the video of the UHC CEO getting shot made the rounds in some group chats, and that would be hard for some people to see. These kinds of filters would let those users hide the video on their messenger without impacting anyone else.
We’ll look into making it easy to support a self-hosted model.
We have many happy clients that moderate UGC with Cleanspeak, including gaming and customer service applications. You can read more about the approach in the docs[1] or the blog[2]. Here's a blog post[3] that talks about root word list extrapolation, which is one of the approaches.
Cleanspeak is not the cheapest option and there's some integration work via API required, but if you are looking for performant, scalable, flexible moderation, it's worth an eval.
0: https://cleanspeak.com/
1: https://cleanspeak.com/docs/3.x/tech/
2: https://webflow.cleanspeak.com/blog
3: https://cleanspeak.com/blog/root-word-list-advantage-content...
[1]: https://git.sr.ht/~jamesponddotco/bonk
[2]: Because I host in Germany.
You are likely aware by now but "bonking" someone is also anglosphere slang for exactly what you're trying to detect.
Moderation is a vast topic - there are different services that focus on different areas: such as, text, images, CSAM, etc. Traditionally you treat each problem area differently.
Within each area, you, as an operator, need to define the level of sensitivity for the category of offense (policies).
Some policies seem more clear cut (eg image: porn) while others seem more difficult to define precisely (eg text: bullying or child grooming).
In my experience, text moderation is more complex and presents a lot of risks.
There are different approaches for text moderation.
Keyword based matching services like Cleanspeak, TwoHat, etc. are baseline level useful but limiting because assessing a keyword requires context. A word can be miscategorized and results in false positive or false negative with this approach, which may impact your operation at scale; or UX if a platform requires more of a real-time experience.
LLM is theoretically well suited for taking context into account for text moderation; however they are also pricier and may require furthering fine tuning or self-hosting for cost savings.
CSAM as a problem area presents the highest risks though may be more clear cut. There are dedicated image services and regulatory bodies that focus on this area (for automating reporting to local law enforcement).
Finally, EU (DSA) also requires social media companies adhere to self report on moderation actions. EU also requires companies to provide pathways for users to own and delete their data (GDPR).
Edit: FIXED typos; ADDED a note on CSAM and DSA & GDPR