Show HN: A free, AI-powered service for automating hate speech moderation
I built ModerateHatespeech, an initiative that working on better understanding + building solutions to combat hate speech in online platform. We have our flagship API, which is completely free and ML-powered, and gives a lot more actionable/better results than pretty much any other similar platform out there.
We've been able to leverage our partnerships with many communities to get a lot of good data + feedback, and bring our system to a lot of users (we process ~200k comments a day right now).
We've also done a lot of work to better understand biases/potential abuse-cases of our API, which you can read about on our site (trying to avoid too many links getting caught in spam filters).
I would definitely love to hear any thoughts/feedback! Here's a link to information about our project/API: https://moderatehatespeech.com/
21 comments
[ 4.6 ms ] story [ 65.3 ms ] threadSome moderation platforms I've worked with are too slow for messaging, especially if users are in different countries.
Do you have plans for image moderation?
We generally aim for <400ms in the continental US. The backend is hosted in us-central-1 on GCP. Obviously, we're always trying to expand, though we do rely on grant funding as a non profit so that comes into play too.
> Some moderation platforms I've worked with are too slow for messaging, especially if users are in different countries.
Typically it works pretty well on the message platforms we're on -- we've got a Chrome extension for Twitter + discord integrations we're testing right now.
> Do you have plans for image moderation?
We do! Though that's further down the line.
> "She is a he" => { "class": normal, "confidence": 1 }
> "He will never be a woman" => { "class": normal, "confidence": 0.999 }
But it does seem to identify 2nd-person targeted transphobia:
> "You will never be a woman" => { "class": flag, "confidence": 0.996 }
Don’t flame me I’m just pointing it out.
The first are of course, transphobic given the clear intent, but it there is definitely some ambiguity w/o context. Ie, if something just posted, "He will never be a woman" there's definitely many contexts where something like that is not malicious. Ie, if someone else is correcting an accidental mis-gender, they might say, "No, she is actually a he" which isn't malicious.
Since the state of our API (and literally every other similar system out there, lol) can't capture the context of the conversation -- what "He" is referring to, the overall topic of the conversations, etc -- we basically have to make the best calls given the available context...which unfortunately does create false negatives in certain cases (and false positives in others).
But, context is something we're working on, because its definitely a big issue, and as you can see, plays a very prominent role in informing better moderation decisions!
As:
“any unreasonably rude or hateful content, constituting as, "threats, extreme obscenity, insults, and identity-based hate"”
{ "class": normal "confidence": 0.955 }
"Democrats are bigots"
{ "class": flag "confidence": 0.575 }