In my experience, yes. Try discussing in an impassioned manner something of importance to the human race, or of life and death importance to the human you are speaking to, and watch how quickly people will change the subject to whining about your "attitude" or fixating on some curse word you used, rather than the important subject at hand.
Makes sense. It's not like Google or any other company training AI models are hiring professional linguists or psychologists to investigate the true meaning behind each of the billions of internet posts they've scraped, labeled and trained their models on. They're throwing pennies at workers in the developing world to label as much data as they can as fast as they can.
It's also likely that there's a significant lack of context to the data points, not just because the posts are divorced from their parent content, but because of a culture and language divide between the labeler and the author of the data they're labeling, as well.
Yeah, I think this is part of the problem. Is large-scale, low-quality data good? Sometimes it is (depending on the tradeoff), but from a model performance perspective, it's often more effective to get smaller amounts of higher-quality data instead.
Hopefully people also don't need to be at the level of a professional linguist to label messages like "this is fucking awesome" correctly!
And great point on context. For example, the GoEmotions dataset didn't present labelers with the actual post or subreddit the message came from -- just the text itself. That makes it really difficult to label something like "his traps hide the fucking sun"! But once you see the comment in its original context https://www.reddit.com/r/nattyorjuice/comments/aee3wx/olympi..., and know that it's in the /r/nattyorjuice bodybuilding subreddit, it's much easier to realize that this is talking about someone's large muscles.
Even with proper labelling done by people who have lots of time to dig into each message, I don't believe we'd ever get a reasonable model. People can trivially hide the true meaning of sentences. The worst actually-consciously-racist accounts on Twitter will not have a single thing to report. You can find some insinuation or fragments where you know exactly what it all adds up to, then click report, get asked to choose messages to report and... yeah, not a single one of them is toxic in the literal sense.
Not in the cases I describe. The ones I mean are not just crap that's left after moderation. They are cleverly maintained to continuously fly under the radar.
It depends on what the purpose of content moderation is.
Good if you want to accurately identify abusive behavior and protect users from harm? No.
Good enough if you want to find the most blatant examples of name-calling and insults to appease regulators and trigger-happy lawyers by appearing to use "state of the art technology"? Sure.
> It's not like Google or any other company training AI models are hiring professional linguists or psychologists to investigate
I had a housemate once who was an American linguistics grad and spent a year applying semantic labels for Google. I know he worked on a team, although I don't know what they did since I didn't work at a Google at the time.
Yeah, Google and a couple other companies often hire "Analytical Linguists" as their labelers, or to help write their guidelines and manage labeling projects.
Although -- and I say this having done a lot of my graduate coursework in linguistics -- I don't think having a linguistics background is particularly needed (unless you're doing specialized annotation, like creating syntax trees or tagging phonemes in Praat), outside of you being more likely to enjoy thinking about the nuances of language.
If I could filter out the overuse of profanity shown in this article, I would. "Fuck yeah!!! That bad bitch is totally the shit!!" gets caught by a profanity filter. No great loss, IMO.
If you get normal not-always-online not-gen-Z people to evaluate these messages and label them as Good or Bad then you will get results like this. If I got any member of my family over the age of 30 to evaluate these messages, they'd label them all as offensive.
But these are all subjective opinions – I could just as easily say that it _is_ a surprise to me that it's flagged as offensive, or that unnecessarily using profanity isn't toxic or impolite. And I feel like that's where this post is gesturing toward – not a judgment on what should or should not be considered toxic, but just a reminder to be intentional when writing definitions and sourcing training data.
Chiming in to say you are dearly mistaken, you turkey-squid uncle buntler.
I, on the other hand, am a fucking amazing shit OG bitch.
But yeah this stuff is absolutely dangerous, opinions like these are destroying culture. It won't be long until the threads these automatic tools are moderating, WONT BE WORTH HAVING.
I first heard "the shit" meaning "good, cool, amazing" in 1996, I think. Never knew the origin, I heard it from a Californian in Arizona, and not really from anyone else since.
>If I got any member of my family over the age of 30 to evaluate these messages, they'd label them all as offensive.
How sure are you on that? I'm 31 and there's definitely no one in my peer group who would find those examples offensive. Hell, only 1/2 parents might be offended by that language, and my mom has been working around young people enough for the past decade that she might have softened up since I lived with her too.
The issue is more "Offensive" depends very much on context - not a black and white state.
There are contexts that quote would not be offensive to anyone (more casual settings with a group that trusts each other), but also contexts where everyone would find it offensive, no matter their age or generation.
Which is why its futile and counterproductive to have content-based mass censorship. Offensiveness is completely subjective. Give everyone a mute button and a blacklist for them to block out any posts with whatever whatever words they are too fragile to see and let people decide for themselves. On large social media platforms it would be far better if we went back to the old system of banning spammers and people posting explicitly illegal content and letting everything else sort itself out.
The point the article is making is not that the messages are "Good" or "Bad", it's that an automatic rating system that intends to allow some usages of profanity rates messages with the type of profanity that it purports to allow as very bad, towards the end of the scale that it uses.
Where are you from, and what's your cultural background? I'm from Melbourne, and my parents are from Adelaide (possibly the most puritan city in Australia).
Not a single person I know would take offense to this.
My South African wife's family, however, is a completely different story. Her mother's side (English) would be horrified by this, whereas her dad's side (Afrikaner) wouldn't be upset in the least.
Point is, it's geography and upbringing far more than age that predicts puritanism. If anything, the young people from your area are regressing to the global mean.
One of the problems with real world machine learning is that engineers often treat models as pure black boxes to be optimized, ignoring the datasets behind them. I've often worked with ML engineers who can't give you any examples of false positives they want their models to fix!
Perhaps this is okay when your datasets are high-quality and representative of the real world, but they're usually not. For example, many toxicity and hate speech datasets mistakenly flag texts like "this is fucking awesome!" as toxic, even though they're actually quite positive -- because NLP datasets are often labeled by non-fluent speakers who pattern match on profanity.
(So is 99% accuracy or 99% precision actually a good thing? Not if your test sets are inaccurate as well!)
Many of the new, massive scale language models use the Perspective API to measure their safety. But we've noticed a number of Perspective API mistakes on texts containing positive profanity, so this post was an attempt to explain the problem and quantify it.
> In the end you can be profane and very positive.
I've noticed a lot of humans can't detect sarcasm. And that isn't correlated with traditional smarts either. For AI it's a hopeless task.
> You can also have strike language while being incredibly vile and negative.
I'm reminds of a group of people I know. They love love using genteel language to throw vile insults at each other. Using profane language is a automatic foul.
You can also write things which sound violent (when read literally) but are actually innocent.
My wife's cousin recently had a baby, and posted a photo on Facebook. My wife commented (something like) "She's so cute I'll have to kidnap her". Facebook locked her account for 24 hours for making a "violent threat". I am sure the mother wasn't threatened by the remark in the slightest. It just added to my wife's anger at Facebook for repeatedly giving her "warnings" over trivial or innocent things.
Not online, in person, but one of the teachers at our son's school sometimes "threatens" to "steal him" from us – her remarks don't worry us, because we know she would never actually do that, it is just a colloquial way of expressing affection.
To Facebook they are “friends” and have many mutual “friends” as well. You’d think they could take that into account in evaluating the nature of the remark, but it does not appear they have done so.
You cant conclude "this is not a threat" from "being friends and having mutual friends". Most of violence by large is between people who know each other.
If you know that A and B are “friends with many mutual friends”, and A posts baby photos, and B comments “She's so cute I'll have to kidnap her", that’s enough information to conclude that the remark is more likely to be experienced as innocent than threatening.
My wife is quite convinced - and I think she is probably right - that nobody reported her comment, some automatic system at Facebook picked it up purely on the basis of the keyword “kidnap”.
If a male relative or friend had made the same remark, would my wife’s cousin have felt threatened? My wife tells me she would not have been bothered by such a remark, in that context, from one of her male relatives or friends.
And, if someone was really bothered by it, they could report it. But, it doesn’t seem like that is what has happened here, it appears to just be the work of some automated system. I’m not saying that if the mother reports the remark Facebook shouldn’t take the report seriously - cultures differ after all, in some cultures those kind of remarks are normal in that context, in others they would seem bizarre and offensive - but it shouldn’t just assume the remark is threatening because it contains the word “kidnap”
To be clear: I 100% believe there was no intent of threat and no person read it as such in your situation. I agree it was likely automated something. It is way more likely then report, because if you had report eager people on facebook, you would know.
I was more interested in hypothetical "can we conclude it is ok from friend status on Facebook without actually knowing involve people" question.
Facebook removes the offending posts in this case, yes? If someone on my friend's list says something offensive on one of my posts, I think I'd rather see it and address it myself rather than have Facebook sweep it away.
Which is to say: Often it's not reasonable to act without first knowing if the recipient actually took offence or felt threatened, before even considering anything else, because you can't know. A model aimed at judging utterances needs to also take into account "how close is this group? could this be in-group language?" and it needs sufficient context to judge that. Even then it'll have huge potential for error.
Incidentally, a lot of this wouldn't be nearly as much of an issue if we could trust that these kinds of tools would be used appropriately. E.g. instead of banning things, even just asking a "are you sure this will be taken the right way?" or similar is a whole lot more benign. The dating app Bumble uses a filter that checks for possibly explicit pictures, for example, but instead of blocking them it shows you a short message asking if you're sure the recipient wants to receive what you're sending, and lets you choose. Depending on use there might well be several levels of severity in messaging justified, from just a mild "are you sure?" marker to forcing you to explicitly acknowledge that you're on notice that you're responsible for the content.
Violence is one thing, but I think harassment and threats in public forums may not usually be between people who "are friends and have mutual friends".
In that case, if it's a genuine threat, then freezing the account isn't the right action, notifying local law enforcement is.
Beyond this, by hiding the comment/post, the mother is now unaware of the threat and Facebook didn't contact local law enforcement in fact worse than doing nothing at all.
> Facebook locked her account for 24 hours for making a "violent threat".
This is due to years of old media digging through Facebook posts to write stories like "Child kidnapped after threat on Facebook and Facebook DID NOTHING". This led to more and more calls for FB to "do something" to "keep people safe online". So they started to run all posts through the AI prescreener and well, these are the results.
But hey, they get to at least say that they take down X billions of posts before any other human sees it.
One of my childhood friends was told by his parents they'd dock his pocket money if he kept insulting his brother. So he started using the names of cheeses. Worked just as well at insulting his brother, and created a quandary for his parents who couldn't well dock his pocket money for saying "camembert" to his brother.
Now consider a large group of people upset about censorship and motivated to adjust their language to circumvent it...
It’s not really a quandary though is it? The friend is still insulting his brother. The parents and other people worried about good behaviour aren’t stupid and can apply the spirit of the rule. The problem (at scale) as ever is trying to fix a social issue with a barely working bit of tech.
The quandary is that it made it apparent that trying to block terms just creates a moving target. It demonstrated that completely innocuous terms could be weaponized by context, and it was not an issue that could be solved by blocking specific terms.
The specific words don't matter, as any reaction would just shift the insult to another term. Ban cheeses? Start saying "I love you" ("mom, he told me he loves me! Tell him to stop it!"...). Or call him awesome. If anything, I think the brother missed a trick by not going straight to compliments - few things can be more cutting.
Given the right context it'd be clear to the brother what the intent was no matter the term, but increasingly impossible for the parents to distinguish "legitimate" communications from the insults. For that matter, no words are necessary to achieve that once both sides know. Just a look is enough.
We don't know how to fix this social issue with human censors, so it's much worse than just the state of the tech - it's an issue that's basically unsolvable without addressing why the speaker is intending to insult (if they are) and why the receiver takes offence (if they are).
The best you can do is address a small proportion of the most blatant cases (and that might well be ok and enough if you can do it with few enough false positives, but getting the false positives to an acceptable level is in itself a momentous challenge).
Of course we know how to fix this kids aren’t magic and neither are adults. Explain the rules, explain why the behaviour they are exhibiting is unacceptable and punish them accordingly. I don’t know if you have kids but the one being insulted is capable of explaining things to the parents who then adjudicate. The same is largely true for communities as well. You get the community based on the norms you allow to establish and defend. Part of that is being involved enough to understand what’s happening.
There of course isn’t a perfect solution and likely never will be.
I do have kids, and I find your solution naive exactly because neither kids nor adults are magic.
The problem is that unless you observe the behaviour yourself as it happens, with substantial context, the idea that you can reliably know whether the behaviour was unacceptable is fundamentally flawed.
This is exactly what the choice of innocuous terms is for; to create plausible deniability. E.g. the word "camembert" thrown at someone out of the blue might be obvious because it stands out. But that's not the story you're going to get from the person who has used it as an insult. You're going to get a story about how they just talked about what they like to eat, or something like that.
You might see through that, but what quickly happens - what indeed happened with the brothers in my example - was that the person making the insults quickly calibrates to know exactly what results in a reaction and what creates enough plausible deniability that the person deciding if this is sanctionable is unable to determine with any reliability whether what they're hearing genuinely is an insult or the "target" being oversensitive and misinterpreting things. I was bullied at school at times, and saw this first hand as a target myself - the bullies quickly learned which ways they could word themselves to make the insults clear to me while creating enough plausible deniability even when there was no conflicting claims about what exactly they'd said.
Point being being that humans even with context and in the position to interrogate the people involved can't solve this with any kind of reliability. Then the notion that an AI at current state of the art without access to context or the ability to interrogate those involved doesn't stand a chance.
> The problem is that unless you observe the behaviour yourself as it happens, with substantial context, the idea that you can reliably know whether the behaviour was unacceptable is fundamentally flawed.
Yes but we're not trying to be perfect, we have to live and deal with this grey area.
> Point being being that humans even with context and in the position to interrogate the people involved can't solve this with any kind of reliability.
I disagree here, I think we can quite reliably do this a great amount of the time. There are absolutely bad actors who will get away with things in the short term but because context builds up over time it never lasts. And once you are aware they're trying to be subversive its much easier to spot earlier. It's not an attack with legs.
> Then the notion that an AI at current state of the art without access to context or the ability to interrogate those involved doesn't stand a chance.
I absolutely agree here.
And I'm sorry the people in charge were rubbish at protecting you from bullying.
Even if totally 'accurate' to the dataset, there's the issue that a lot of 'toxicity' and offence is completely culturally based. Take 'cunt' for example. Used in casual, informal conversation here, but deemed incredibly offensive by some Americans. Or, more broadly, 'thumbs up' could mean "Good job" or "Stick it up your arse".
As teens, some of my friends and I had a "game" where we would find new innocent phrases to describe sexual acts, taking a shit, and other vulgar/gross/unmentionable stuff. Yeah, stupid teenager humor, but we had a lot of fun nonetheless. We really didn't have a points system, but you got bonus rep if you'd be daring enough to utter the phrases you came up in the presence of adults and get away without a scolding - which resulted in one of the dad's kinda joining our game after he figured us out.
A few weeks ago someone posted a guide to interviewing as a data scientist, and while it touches on all sorts of algorithms and statistical relations, the number one thing you want out of a data scientist is respect for the source data. Second is understanding what you want to do. Modelling comes third.
> "this is fucking awesome!" as toxic, even though they're actually quite positive
And this touches on the question of what we want to do. It might not be a negative sentiment, but it might still be offensive. On the other hand, there are no offensive words, and only "positive" sentences in many sarcastic utterances.
And there are a lot of sentences that can be perceived as insensitive, that are not meant as such. GCP Grey has a video on the words "Indian" vs "Native American", and apparently it's very complicated which word you can or should use.
One thing I came across is that 99% precision/recall/F1 score might not be enough. Especially in large sample sets "only" accurately identifying 99% leaves you with too many wrong classifications. Eg 1.000.000 samples -> 10.000 false classifications.
No idea if there is a different metric that somehow (again) takes the number of sample into account.
Interesting stuff. I doinked around with this a while back when working on a 'hot take oracle' - basically a search box that finds a strongly-opinionated tweet about something (https://hottakeoracle.herokuapp.com/).
You can see that my model is basically just filtering for profanity as an indicator of "strong emotion", which makes sense. But it's interesting that postive profanity seems to be such a thorny problem, at least for Perspective.
Former Jigsawyer here. I think this article is pretty fair to Perspective given that it was never meant to be used in a fully automated way, just as a first pass to help forum moderators.
It's very difficult when you blur the lines of code and ethics, as real world ethical judgements aren't necessarily consistent or well defined in a way which is easily translatable, even by a large ML model. Jigsaw is a great example of this -- right across the aisle (pre-pandemic) from Perspective is a team fighting internet censorship. Obviously Perspective's "censorship" is different in quality from the Great Firewall, but it shows the hairiness of the problems.
All this is to say the people at Jigsaw are some of the most brilliant people I've ever met and I'm glad they're out there working on difficult problems.
Yeah, we love what Jigsaw's building! This is all with the hope of improvement and collaboration.
We deal with these hairy problems a lot too. Even before ML proper, getting the definitions right is very tricky. Should a comment that's polite and positive on its own, but supportive of a toxic parent post ("I love Nazis!" -> "I agree!"), be treated as toxic? Is "toxic counterspeech" equally toxic? What about a comedian making fun of an actor's nose, and does it depend on whether the joke is to that actor directly vs. merely referencing them in the third person? etc.
For these reasons, I actually personally like the fact that the Jigsaw annotation guidelines are very high level (as opposed to long and prescriptive) -- it lets the data capture the spectrum of "human preferences" on its own (at least, it does when you can trust that the annotators are able and trying to do a good job).
> I think this article is pretty fair to Perspective given that it was never meant to be used in a fully automated way, just as a first pass to help forum moderators.
Given how eager Google itself is to automate all your interaction with them (and how hard it is to get access to anything resembling actual human judgment), did they really think it wouldn't be used that way?
Or did they just not care?
Also, I would have hoped the project's close ties to the US State Department would have worried some of you brilliant people a little more.
The close ties to the state department made me uncomfortable while I was there, but I never saw anything that looked like they had any say in anything we did. But I never worked on Perspective.
Something alluded to here is that many of the Languages models use US English. Many terms that are offensive in the US, may not be offensive at all in the UK. e.g. "Fag" in the UK is frequently used to refer to cigarettes. "Can I bum a fag?" literally means "Can I have one of your cigarettes please?".
Similarly something that might be a cat call such as "Get your baps out" (shows us your breasts), could also be used by a baker since a "bap" is a type of bread roll in a slightly cheeky advert as most people are aware of the pun.
How are you going to train an AI to know the context that the person might be talking about bread instead of a woman?
Has anyone realised yet that almost all of this folly? I suppose not when there is money to be made.
> For starters, you can't unless you actually include the context in the training set.
It's worse than that. The context doesn't just have to be in the training set. It also has to be available on the other end, when you're using the model to make a determination. And it isn't.
Which is why even human censors can't get it right. The parties communicating can, and almost always do, have external shared state that The Decider doesn't. Which gives the same words different meaning.
Imagine hearing an inside joke you're on the outside of and then being asked to adjudicate whether it was offensive.
Sure, but you do have extra context you can make available on both ends, but that I'm assuming most don't even try to include.
E.g. if judging tweets for example you'd presumably do a lot better if you evaluated the tweets in context of followers and in context of past tweets - both recent and the accounts history. E.g. an account that posts white supremacy tweets regularly is likely to mean something entirely different if RT'ing a BLM tweet with "that is fucking awesome" than what someone with #BLM in their profile is likely to mean.
Part of the problem is that they're throwing away a huge amount of the state they do in fact have.
But I absolutely agree it's in general an unsolvable issue - I pointed to a story from my childhood elsewhere showing how trivially people cause problems for "The Decider": A childhood friend being forced not to call his brother names and switching to using names of cheeses.
Is "Edam" an insult or a food preference? You can't know without context.
And it needs to be very local context, because humans very quickly pick up on when you just start using a word to mean something else, so it doesn't even need to be any shared external state about the word, just a shared understanding of where the receiver might expect an insult coupled with an unexpected response that will then easily get labelled an insult. That unexpected term might well in itself be positive if the receiver expects criticism. "Awesome" and "I love it!" are perfectly good insults when the other party has just told you something where the appropriate response would be negative, for example.
> E.g. if judging tweets for example you'd presumably do a lot better if you evaluated the tweets in context of followers and in context of past tweets - both recent and the accounts history.
Must never use followers. Or anything else the account holder has no control over. Otherwise you'll end up with the same kind of SEO problems where competitors and foreign governments create fake accounts to follow or interact with disfavored ones and destroy their algorithmic reputation.
Guilt by association in general is malice. Someone who regularly interacts with white supremacists might be a white supremacist -- and maybe 90% of them are -- but it could also be someone criticizing, mocking or debunking them. And if it gets out that the algorithm is penalizing people for doing that, they'll stop. Which is very bad.
Even using who an account follows incentivizes echo chambers, more than they already are.
I specifically follow a few locals that I would often notice upset with the same politicians that I was, but for diametrically different reasons. Even though I disagree with them on most everything, I find value in having a few of those voices on my feed visibly attached to a consistent person. It helps me to resist seeing the "other side" as just an impersonal sea of voices.
So we hope that customers eventually stop using their services, hurting their bottom line.
I've been following some drama in chat/user interaction in modern games. People are getting banned in Forza Horizon 5 for funny decals. One guy got an 8,000 year ban for a Kim Jong-un KFC paint job (https://i2-prod.dailystar.co.uk/incoming/article25686363.ece...).
EA is banning entire accounts - access to ALL games on an account - for swearing in Apex Legends. All of the major studios are in the process of severely restricting and even removing chat in their games.
The latest Battlefield 2042 game doesn't even have voice chat, and they removed the global scoreboard altogether because they didn't want poorly performing players to know how poorly they were performing.
The list goes on and on. To me, this is all a serious reduction in the enjoyment I have in games. I grew up playing Counter Strike, and s**-talking and competing was a big part of the experience. I just won't play games where I'm not allowed to interact with other players.
The market will solve that. People will stop giving crazy companies money when their products become "not fun". See: Battlefield 2042 failing pretty hard.
That being said: I specifically only buy entertainment products that do NOT require me to listen to some 14-year-old describing sexual activity with my female family members in great detail because his nonexistent skills make me keep winning. Different strokes for different blokes.
I couldn't agree more. I've been watching this shift in acceptable standards in America move from intent to perception with considerable consternation. It seems that an absence of malintent in language is no longer sufficient defense. If one perceives the language as offensive, it is. This is an absurd standard because with a sufficient audience size, someone will find something offensive. This standard is a de facto end to all discussion. It only takes one hyper sensitive audience member (legitimate or troll) to shut down all discussion.
This is not a sustainable strategy for society. It is of course impossible for algorithms to parse perceived intent by the lowest common denominator, and attempting to do so is nothing more than defensive legal posturing.
Because "intent" rule was easy to abused and was in fact abused often. Directly by people seeking or enjoying to be "toxic" (in loose meaning). All you have to do is to pretend little bit you are clueless again and again, despite being told again and again.
And to shut up targets - "he/she does not mean that", "he/she is just insecure", "you know how he/she is". If you cant prove intent, you had no grounds to even talk about that that person is doing to you again and again.
I don't contend that. I'm arguing that the cost of the move to a perception based standard is far higher than the former. I'm arguing that it is, in effect, completely unenforceable, arbitrary, and destructive to not just individuals, but society as a whole.
Free speech is not without cost, but it has been successfully utilised to everyone's benefit for hundreds of years. I can only surmise that those advocating for this have never read about or understood the importance of the Enlightenment. They don't understand the horror of living under a social regime where the powerful control language and effectively thought. They don't understand why the ability to risk offense is critical for democracy, science, and social progress. Eliminating free speech will harm the most vulnerable in society; not the most powerful.
I can only surmise that those advocating for this have never read about or understood the importance of the Enlightenment. They don't understand the horror of living under a social regime where the powerful control language and effectively thought. They don't understand why the ability to risk offense is critical for democracy, science, and social progress. Eliminating free speech will harm the most vulnerable in society; not the most powerful.
You are certainly correct - unfortunately the people burning the books are not the ones reading or writing them (even if they pretend to do so, with advanced credentials or padded resumes).
> Free speech is not without cost, but it has been successfully utilised to everyone's benefit for hundreds of years.
American history itself had tons and tons of situation where people risked a lot for saying things - including death. All within the scope you talk about. The time span you talk about include runup to civil war, civil war, reconstruction - all of which were full of political violence.
This also when dueling was the thing. Where men of means were expected to shoot each other over words. If you did not, you was effectively over in that town. And the line between "duel worthy" offense and not could be very thin.
> Eliminating free speech will harm the most vulnerable in society; not the most powerful.
Moreover, we are not talking here about legal standard at all. Neither was legal standard EVER. You could legally seek to insult and offend people as much as you wanted. Nobody cared about intent. This is about when people say "you are jerk".
There is threat to free speech, especially with new proposed laws, but it has zero to do with intent vs effect.
I don't know that you can take dueling at face-value; dueling got to the point where you were expected to show up to the duel, but actually shooting someone dead was classless and would have made you a social pariah.
>American history itself had tons and tons of situation where people risked a lot for saying things - including death. All within the scope you talk about. The time span you talk about include runup to civil war, civil war, reconstruction - all of which were full of political violence.
Now do European history pre-enlightement.
It was way, way worse. Shit talk the church and the inquisition slows up. Complain about the cut of your grain that the mill takes and the local lord threatens you lest other peasants start complaining.
The acceptance of freedom of speech as a general principal has greatly increased the civility of civilization.
This is a tough problem, because we are also experiencing people who take offence in a toxic manner.
E.g. people who object to a statement because it might possibly be offensive to another group, despite members of that other group being present and not being offended. Clearly they are doing this out of the same enjoyment of other people's discomfort that you describe.
I think some people do go a little far, but I don't think everyone's doing this out of some sadistic desire to watch people squirm. More likely they're just a bit too earnest and as a result try to correct others and overcompensate. Most people here have been an annoying college-aged kid at some point with an axe to grind on some specific point, we should remember that.
That said, it's interesting this has come up during a discussion about how the intent behind what someone said is often either misunderstood or discarded.
I agree, but there seems to be no similar leeway for people who have said offensive things with no intent to be offensive. Any speech that could be construed as offensive is presumed to be malicious. So it seems fair to assume that taking offence needlessly is also malicious.
There's another aspect too: mysterious third-party echo chambers deciding what's now offensive. The black/whitelist thing is a good example; very often the topic is introduced as if the issue is resolved "we got together and had a discussion, and yes it is offensive" and suddenly everyone not present in that conversation has to accept this verdict.
Other times, you get the "well it's so very easy to make this small change, it's practically rude (or suspicious) not to" - a completely wrong sentiment for many reasons, but easy to make you look difficult for arguing about.
I agree, but the fact that different sensibilities can have very strong/opposite opinions about a situation proves that bots are unsuited to regulate human behavior as a "neutral" judgement, which was the original point :)
> More likely they're just a bit too earnest and as a result try to correct others and overcompensate.
You are assuming way too much good faith. For every one that does it in earnest there's probably 10 grifters just taking advantage of the situation for their own personal benefit:
The real reason seems to me like virtue-signalling. They don't care whether they're personally offended, as that would require burning social "points" by rocking the boat but potentially getting no outcome out of it (nobody else might agree). It's "safer" to just suck it up than sacrifice social points especially when you're on the lowest ranks.
They also don't know nor care whether the protected class they are worried about is actually going to be offended. Again, rocking the boat and burning social capital for little personal benefit.
This leaves the argument of earning social capital by virtue signaling among like-minded peers. Pick a common issue you already know is popular within your target "market" (racism, sexism, "diversity & inclusion", etc) and raise the problem. This doesn't risk any social capital, after all, people like you won't disagree (for the same reasons) and people that might be on the fence or would like to add nuance wouldn't want to risk their social capital in fear of being labelled a racist/etc. It also transcends the ranks - if anything, a higher-ranking person in the company would risk much more by speaking out than someone on your rank or lower.
This eventually results in a positive feedback loop where everything will become offensive given enough time and those who disagree either comply or get pushed out. That's how you get bullshit like GitHub's "main" branch which helps nobody while continuing to sell services to ICE which you could argue actually hurts minorities.
>taking advantage of the situation for their own personal benefit:
And therein lies the problem. A subset of people get a benefit from engaging in this behavior. So of course they do it at every opportunity.
You get what you incentivize.
Until the outcome is so frequently a net negative (however slight) in so many cases that the behavior is marginalized to a negligibly small number of people/settings the behavior will persist in enough volume to be worth caring about.
The problem is that a large part of the tech industry nowadays relies on "engagement" aka monetizing user attention for the benefit of advertisers, thus extremely vulnerable to public opinion (the advertisers themselves would engage in this behavior, thus companies that supply them must also conform to it). The entire industry is on thin ice already so it makes sense for everyone to play it safe even though it just reinforces the feedback loop even more.
This is why this behavior is rarely seen in other industries that make the bulk of their money on a tangible output, both because the output is less fungible/replaceable as well as those industries' employees typically having less time to waste engaging in bullshit.
I'm just going by conversations I've had with people in person and online, and assuming that this is broadly reflective of society overall. Don't get me wrong, I am quite sure that there are people who are cancelling people for sadistic pleasure and get a kick out of torturing someone and use hot-button issues to do so. However I cannot imagine this type of person outnumbering those with good-faith concerns at a 10:1 ratio.
We're probably not going to see eye to eye on this. Honestly it sounds like you've been on the receiving end of this, and I can imagine that being a very frustrating experience so I do sympathise if that is the case. But really the world you're painting is one where you have to walk on eggshells in every conversation because a mob is waiting to pounce if you used the word "retard" as a verb or called someone "Miss" instead of "Ms" or something like that. Perhaps that might happen in the most overly-earnest, try-hard, super-woke group where everyone uses custom, ever-changing neopronouns and you're expected to remember and use every one of them. But I think those sort of people are busy enough with their own internal squabbles to be going out bothering HNers.
Which is fair enough, and I'd agree with that. But why do we then assume that everyone who says anything that could be construed as offensive is definitely malicious and needs to be cancelled?
Intent is always difficult and often impossible to judge.
There is no happy medium here, particularly since those seeking to offend and to wolfwhistle to signal their allegiance to other people who hate your race (for example) will try to couch their terms to hide within the shadows of plausible deniability unless they feel completely safe.
It's more a question of how many false positives and false negatives your culture is prepared to tolerate.
Of course, offense is just as difficult and impossible to judge as intent. In both cases, one can simply ask the individual, "Were you offended?" or "What was your intent?" But biased or motivated itself reporting makes each unreliable.
I certainly agree with your final conclusion: it depends on what society is willing to tolerate in terms of false positives and in terms of false negatives. And particularly, the balance between those two kinds of errors.
Anyone can say they’re offended without any cost to themselves. It’s ‘cheap’, and the outcome is often very ‘expensive’ for the offender. You could even make the case that virtue is derived from being offended. This disproportionality invites bad faith. I’ve stopped caring about people’s offense.
Anyone can say they didn't mean it that way/they were joking without any cost to themselves as well.
Not to say that there isn't anything wrong with how people have been increasingly incensed in the last few decades, but that's basically what it was like in the 90s and before. Someone was like, "whoa dude what you said wasn't cool" was met with "it was just a joke".
The happy medium, such as it is, is to not try to police “toxicity” in the first place.
If you want to be a global publisher of your opinions, you’re going to receive a global response, from people across the political and social gamut — the larger your audience, and the more political/ideological your speech, the more substantial a response.
My dictionary’s definition of “Faustian” has this usage example:
> Modern celebrities enter into a Faustian pact with the general public.
What people seem to want is the ability to enforce a one-sided bargain with the public, under which their own toxicity, opinions, and any public affirmations should never be censored, as long as their opinions are the politically correct ones.
>The happy medium, such as it is, is to not try to police “toxicity” in the first place.
So, give hard core racists the benefit of the doubt when they wolf whistle each other?
Can't see any historical precedent for that going disastrously wrong.
>What people seem to want is ... their own toxicity, opinions, and any public affirmations should never be censored
Some people are hypocrites. News at 11.
Some people also want anti-han Chinese racism to be treated equivalently to anti-Uyghur racism because they choose to be blind to the inherent power differential.
I remember reading a story of the Hutus and the Tutsis. The genocide began with the radio - shows that very deliberate but cautious (& deniable) dehumanization of the other that ramped up over time. Nobody policed these deliberately aggravated ethnic tensions. By the time they were consistently obvious it was a little on the late side.
On the other hand, I've been falsely accused of being a racist before and it undeniably hurt my feelings a little bit.
Standards based on perception sounds ugly at first, but fundamentally it's an expression that you don't have an automatic entitlement to communicate with me. Should a big external entity like government or companies control that? No. Bu I'm totally fine for accepting complete responsibility for filtering my communications intake. I just wish that ...
- all social media implemented simple word filters. A word is on YOUR list, the post or chat line doesn't make it to your page/chat. It'd be awesome if these lists were exportable, syncable, and tradeable online.
And I wish the AI revolution would benefit me personally instead of companies.
I would like an AI that censors on my behalf only if and when I want it to, and a personalized AI that I can train to flag accounts/people that I don't want to interact with based on language. I would also like some mechanism to indicate how much is getting caught in the AI-generated filters and allow me to stop or modify its behavior. Once the AI is doing mostly what I want, I would like to be able to share it. I don't know how that would work.
- Group owners should have similar mechanisms for all posts in a group.
> I've been watching this shift in acceptable standards in America move from intent to perception with considerable consternation.
I think that both intent and perception are important, and they are both judged in the more modern understanding of "toxicity". Malintent is a direct "strike". But, lack of malintent doesn't mean that perceived offense should be entirely ignored.
Let's imagine a scenario: as in the example above, an English person says "can I burn a fag" and an American person believes they are casually thinking about hurting a gay man. The English person meant no harm; but, the American person is not at fault for not understanding this. They can complain, and the English person should explain their intent. The American person should learn and remember the meaning of this expression in British English, and the English person will need to remember that this expression may sound offensive to American listeners, and avoid it in such scenarios.
Basically, the important aspect is that a misunderstanding is not always the fault of the listener (which the intent-only model suggests: you took offense, but I didn't mean that, so you're silly for feeling offended). It is of course also not only the fault of the speaker (which the perception-only model would suggest: I felt offended by what you said, so it doesn't matter what you meant, you owe me an apology or more). Both parties are involved in a miscommunication, if indeed there was one.
> [...] an English person says "can I burn a fag" and an American person believes they are casually thinking about hurting a gay man.
FYI, it's combination of difficult-to-discern keming with the default font on HN, and your unfamiliarity with the phrase, but it's actually "BUM A FAG" and not "BURN A FAG".
It's interesting to see so many people thought it was "burn" --- perhaps the reference to cigarettes primed the "mental autocorrect" in the wrong direction.
> Let's imagine a scenario: as in the example above, an English person says "can I burn a fag" and an American person believes they are casually thinking about hurting a gay man. The English person meant no harm; but, the American person is not at fault for not understanding this.
If you call someone out for "offensive speech" and your offense rests on your own misunderstanding, it is absolutely your fault.
The major issue with this model is that the offended person has options that are a bit less hostile and assuming of bad intent than "calling somebody out".
Unfortunately, we seem to be in a place where bad intent is taken for granted and merits an aggressive reaction rather than placing any responsibility on the offended party to be civil and open-minded in their response to be perceiving offense.
In the case of "burning a fag", our hypothetical American has the option of inquiring about the meaning of an unfamiliar idiom rather than jumping straight to calling out the speaker.
Yes, but not necessarily your problem. I’m British. I once asked for a glass of water in a restaurant in Virginia, and while I thought the waitress’s response was a little peculiar (no reply, just turned around and walked away), I was astounded when the police showed up five minutes later. Apparently she’d misheard me, and thought “just a glass of water, please” was a threat. It was never revealed to me what the threat I had apparently made was, but it cost me several extremely stressful hours convincing the local PD that there had been a misunderstanding - so while it was her fault, it was my problem. I’ve puzzled for years over what she heard.
As a white guy wearing a suit, I got out of there without being charged. If I weren’t… I can imagine it could have gone very differently.
> If someone who's not a crackpot says to you "can I burn a gay", even though that seems unambiguous, they've probably just misspoken or are joking.
Well, for one, this only applies if you already know that person is not a crackpot. Secondly, burning someone would indeed be an extreme example, but not all (accidental) offenses are so obviously monstrous. If someone said "I hate fags" (intending to say they don't like cigarettes), this would be more likely to be someone's actual homophobic opinion, depending on culture.
But, again, most people aren't running about spouting horrible stuff, and neither are people trying to catch them out either.
It's all just a bit weird to me. If someone said to me "I hate fags", I'd either ask them to clarify, or change the topic. If they were persistent, then yeah, get rid. It's not a problem that requires instantaneous resolution.
Yep you are absolutely right. People routinely get offended by anything that they don’t agree with, and often interpret whatever is said using the worst possible out of context straw man interpretation. I personally get offended when PC advocates whip themselves into a frenzy over the slightest misunderstanding. So I guess I should blame those PC advocates for harming me?
Faggots are also a food in the UK. It was a poor man's dish, using offals. Not so popopular anymore but you can still
find it in some stores (and probably at local butchers in the Midlands and Wales)
HN marked this comment as dead. Looks like HN has some of the same problem we are talking about here
Before I noticed this comment, I made one of my own, making the same point - and it seems HN has marked it as dead for much the same reason: https://news.ycombinator.com/item?id=30069899
‘Tis ironic. (edit: thank you to whoever vouched my comment, it isn't dead any more)
While there's a touch of irony, it the HN system of letting users see dead comments and vouch for them is also a good example of how this kind of filtering can be done in a far less brutal way than outright blocks. If we could trust these filters to only be used for relatively soft touch blocks like this, then false positives wouldn't be such an issue.
The same argument can be made for machine translation. For an algorithm to be able to successfully translate an entirely new idiom, expression or metaphor, it has to be aware of the real world and context it came from. Until there is some level of AGI that can observe and interpret the world on its own, translation and toxicity detection will be limited to examples included in training data.
Nonce. Means paedophile in the UK, apparently [0]. I was brought up in the UK and only discovered this recently (the bad way). So not even the whole of the UK, but only parts of it.
Obviously also a technical term used all the time in its technical sense with no problem.
Context and culture is way more important than the actual words used when trying to determine the meaning of a statement.
Supposedly the term originates in a UK prison sign N.O.N.C.E. used around the turn of the century, meaning "not on normal courtyard exercise".
"Obviously also a technical term used all the time in its technical sense with no problem."
Or sometimes not ... a blog I'm involved with recently had a WordPress server problem which caused commenters to receive a message like "Illegal operation: bad nonce". The readers are non technical and mostly British. Some had to be calmed down a bit!
Maybe it's an age thing? Everyone my age knows what nonce means but mostly because it was so amusingly satirized by Brass Eye in the late 90s at the height of a media induced paedophilia scare. Chris Morris got politicians and celebrities to say stupid stuff on camera by telling them it was for charity literally called Nonce Sense:
"Did you know that a child under the influence of a paedophile may smell like hammers? That's right - I'm talking Nonce Sense"
The program was a sensation at the time but probably only within a certain age group. I can't imagine that sort of satire would have appealed to older people.
it could be age-related. Though I was in the UK in the late 90's and don't remember this at all. But then, I discovered the internet and stopped watching TV completely around then too.
> "Fag" in the UK is frequently used to refer to cigarettes.
Another example: faggots are a traditional meatball dish from parts of England and Wales-the term has nothing to do with anyone’s sexuality. And yet, American social media companies have repeatedly blocked British users for posting about the dish.
There is also a British pudding called “spotted dick” - in this context, “dick” is a dialectical term for “pudding”, no known connection with genitalia.
A friend of my wife recently had her Facebook account restricted for making a post about cooking jerk chicken for dinner. Facebook claimed she was using “abusive language”
I'm not sure if it's an example (surely it's used in AmE too?) but in a spectacularly dumb one, I felt, Amazon recently discarded (completely! no possibility to edit it!) my lengthy & detailed & photograph-laden review, because I commented on the quality of the 'knob'.
Didn't even accept it in the first place. I would have reworded it happily of course - but it really annoyed me and put me off ever bothering to review anything that it just completely destroyed it; I could start again from scratch but all my effort was wasted so just abandoned it at that point.
Br.Eng. has had plenty of experience evolving under censorship, it's where our great tradition of innuendo comes from. I'm thinking particularly of Polari (Roma-influenced gay slang), and how Kenneth Williams got an entire comedy show packed with man-on-man innuendo to be broadcast regularly on national radio in the middle of the afternoon (Round the Horne). At a time when homosexuality was illegal.
Or try explaining panto to Americans. Or even Allo Allo, a show which would .. definitely not be made nowadays.
I've seen false positives referred to as the "Scunthorpe problem". Someone on the internet has a nice map of filthy-sounding real UK place names ...
The absurdity is so high that to lower the amount of hate-speech Facebook started banning the word "hate" (and translations).
This isn't documented anywhere, but I find it hilarious and dystopian that posts that contain the world "hate" in the description or in the comments get blocked more easily, or soft-banned (they don't show up in anyone's feed), even if the word is used in harmless contexts...
Let's be clear about this: the purpose of these models is to cut costs, not to accurately gauge what was said.
Many business models based on user generated content wouldn't be possible if the businesses had to pay minimum wage to people for moderating that content. Using an AI model, no matter how broken, allows them to seem more concerned than if they were just relying on an old-school word filter without doing any actual due diligence.
> Has anyone realised yet that almost all of this folly? I suppose not when there is money to be made.
Yes, of course everyone realizes that no filter actually works in practice. But they are not built to work, they are built to give the smallest impression of working. Tumblr doesn't need their site to be absolutely clear of nudity to attract investors, they only need to give the impression that it's absolutely clear of nudity and that they are trying to keep it so.
Context, and the inability to actually have any understanding is the entire problem with what is called "AI" these days.
Sure "AI" can do a lot of impressive first-level pattern matching, and that can be the basis of many useful outputs.
But ANYTHING that requires actually understanding the context, whether it is existence of new obstacles for a 'self-driving' vehicle, understanding of actual meaning for a language model, or anything else, is a complete and utter failure.
Despite appearances, while some of it is genuinely useful, we're really no further than fancy parlor tricks. Crack even the next level of contextual understanding, and it will be an astonishing leap.
That language aspect aside (it's difficult enough to wrap around), the abject and consistent failure around contextual cues is abysmal. Sarcasm, deflection and other aspects of talking patterns in relationships (e.g. I have a good friend and we consistently threaten each other with ever more Rube Goldbergian forms of cartoonish violence -such as offering to trebuchet them into a wall of shattered glass coated in guano- when one of us says something fleetingly dumb, as a running gag. FB has flagged this more than once, and it's dumb AF).
Content management done well is expensive, and requires humans with superior grokking skills. No place cares enough to try.
We don't need computers judging our speech. It will never work properly, ever. If tech companies want to police "toxicity" then they should use their vast wealth to hire fluent native speakers to do it.
Anyone selling AI-based sentiment analysis is a grifter.
There's really no elegant or generalized way of analyzing things that has acceptable fidelity because semantic rules are so contextual and conditional.
The same words can change context depending on what kind of building you're in or what time of day it is.
Even common phrases like "I'm running over" could mean I'm coming to see you or I'm taking longer than I expected; completely unrelated things.
Flow analysis may give you something, but content analysis is basically impossible unless there's also context analysis.
But since antagonizing trolls aren't usually using profanity or necessarily a different vocabulary of words but they are engaging in patterns, that kind of engagement style analysis may be all that's actually needed
Not to mention typos, bad grammar, etc. that perhaps even mean someone actually said the bad thing, when 'clearly' (to a human reader) that's not what was meant, and it's fine.
Right. I think the bot issues will always be with us. I'm just suggesting that there is a huge class of problems ML can solve but because of 'tail risk' (bad behaviour at extreme X values), ML solns can fail entirely.
We need to develop a way to work with ML st most queries can be answered by ML but there is an exception pattern that defaults to a manual, prescriptive process, or whatever.
Also we need APIs to allow detection of that state.
I can relate. Recently learned while talking to some folks from Spain, that they use the word "puta" a lot, and it is used to express feelings not meant as a rude insult, as they explained.
There are some differences in German, too. For example "wixen/wichsen" is an old word that means to wipe/shine your shoes and is still in active use in this sense in Switzerland as well as Austria, however it lost its appeal in Germany, because it is now primarily with a different meaning. The Wix company took this different understanding of its brand name to an ad: https://www.youtube.com/watch?v=IddnMutPgTI
Since we have an IT background here, same goes for "Mongo", like in MongoDB. Mongo is considered making fun of handicapped people in Germany.
Former Fraport AG changed its brand name because it was abbreviated FAG - Flughafen AG and found it difficult to expand business with that brand name.
No bad actors, if you ask me, only different context. List could go on and on...
That reminds me of a time where we had the substring "prd" in our Azure Storage Accounts. The MSFT profanity filter thought it meant fart in Czech, where we used it as a moniker for production. Ever since then the accounts were named with the substring "prod".
Reminds me of AoE chat filters, which aren't satisfied with censoring specific words, but literally just any words that resemble it. Which one looks more offensive to you?
"Hold on, my dock is nearly up"
"Hold on, my ** is nearly up"
Censorship can actually make things MORE offensive.
(other censored words include "but" and "come")
On some (techhical) bullettin boards you couldn't talk about Matsushita CD/DVD drives, but luckily enough Panasonic was fine.
Still, all in all, automatically replacing a (supposed) swear/offensive word with some asterisks makes less overall damage than banning someone's account (temporarily or for longer periods or forever).
It is simply intimidating when you have to think twice or thrice before saying something for fear of being automatically banned (without possibility of explanation/recourse).
Yes, it's pretty surprising that MongoDB has gotten by without change when even "master repository" had to go. What's seen as offensive online is very US-centric, I knew that, but I didn't know mongo wasn't a well-known slur in the US.
People with Down's syndrome used to be called "mongoloids" because they were thought to have Asian-looking eyes (epicanthic folds), so calling someone "mongo" or "mong" is basically an extra racist way of calling someone mentally retarded.
Even as a sheltered American, I know “mongoloid” is not “politically correct”. Having an important ruby library by the same name makes me uncomfortable.
It had to go? Certainly not gone from git itself, Github or most other Git hosting services.
If someone creates a bug report on your project about virtue-signalling word changes and you give it credibility, it doesn't mean the original word went away from the lexicon.
I would say they are not even that, not by a long shot, since they are unable to evaluate context. It is more probable that content is offensive when vulgar language is present, but that doesn't have to be the case.
Delegating content control to an AI (that doesn't qualify for anything intelligent) is not a working solution.
> as a first-pass filter, leaving final judgments to human decision makers — marking all profanity as toxic can make perfect sense
You would need humans to look at profanity constantly.
> Our mission involves creating a safer Internet, but we don’t want to miss out on our favorite content because of AI flaws in the meantime.
There is a limited AIs that do create content, but a profanity filter always does the exact opposite.
Leaving final judgement to human decision makers, like every other company does their user support, right... Ok that was sarcasm - I expect the bigger the company the smaller the chance to ever be able to reach a human to discuss reversal of the illogical ban you just received. AI will rule also this world, purely for stocks price reasons.
Making your training data into a cultural standard is just imperialism, but that's the goal here righ?
If you don't comply to the standards of US toxic positivity you should be excluded to not hinder the add sales.
That's another aspect of it. Though I think it really starts at the very root of the problem: bad data.
Instead of putting money into gathering quality data, tech giants instead chose to use cheap labour (here: India) with insufficient skills. You get what you pay for.
>Making your training data into a cultural standard is just imperialism, but that's the goal here righ?
yep. The cultural imperialism is an open secret. But I've completely lost any ability to tell the difference between people pretending not to know, and those actually not noticing.
On the other hand and from my recent experience, my 2ç:
I recently started playing counter strike source again online, just for 10 minutes of fun at first (to see if it would still tick with me). I randomly picked up a server and the ambiance was cheerful and nice. I noticed the rules said "no profanity, have fun" and indeed people were mostly polite.
I tried another server at random a bit later and there was more insults, along with a lot of taunting.
I switched back to the first server and have been regularly playing an hour or two every three days and there is a difference with other servers. Some random people coming and throwing insults, even mild ones like "fuck you" or "you son of a bitch awp" get insta ban and it makes the whole session a much better experience. Maybe it's a safe place but playing with polite people is more enjoyable to me now than playing with insult gatlings.
Language is political. There are many meanings to words, depending on context but I do think it's not innocent to swear in front of people or to use swear words to look cool. These are still swear words and insults and their first original use is to provoke or taunt or display aggression. Even if it's only used for "this is album is the shit !", it's still a (childish) provocation. Reminds me of the brogrammer fad.
FWIW: I get regularly owned on this server and I am at the bottom of the ranks but it's still more fun and enjoyable than other servers I tried where I can reach the top but... it's not a nice place. I think online servers are like bars.
Side-note: I was pleasantly surprised to see that "gg" is still thrown around after rounds :). It's way better than "git gud" that came later and that I find horribly toxic.
I thought it could be interpreted that way but I wanted to point out that I didn't expect a CS:S server to be a safe place (I wasn't really sure if my experience fitted the safe place definition.). Chalk it up the barrier language ^^.
Do the moderators engage when someone says "Dude, that was fucking sick!"?
I'm just saying, I'm not surprised that a moderated server is more, uh, moderated, compared to an unmoderated server. I'm not sure the individual word choices are the determining factor compared to having someone paying attention and removing people with undesired behavior.
> Do the moderators engage when someone says "Dude, that was fucking sick!"?
Hard to tell because of the different language. They wouldn't engage someone saying something close to that but people wouldn't express their admiration or congratulations with swear words anyway (because it's not the style of the house ?). Of course they'd be using the community's lingo though to express sick moves but these aren't insulting/taunting/agressive to begin with.
> I'm not sure the individual word choices are the determining factor compared to having someone paying attention and removing people with undesired behavior.
I think too. For the comparison I should try a CS:GO or valorant or pubg or fortnite next week-end to see the difference.
I worked a bit with an "intent detection" library and boy, was it unhelpful. I could craft sentences meaner than most and it'd cheerfully tell me they were friendly.
In a similar vein, there are popular "AI Mental Health" apps I've gotten to straight up instruct me to end my own life with some trivial conversation.
The perceived goal is “detect toxicity”, but let’s unwind this goal a bit.
Is it the lofty “make people be nice to each other”?
Well, the paradox is that being nice is possible with the strongest choice of words, while being very harsh can sound most fluffy bunnies on the surface. In fact, in human relationships there are degrees of mutual familiarity where being exceedingly polite and not “insulting” your counterparty would be perceived as negative—where insults are not taken at face value, but rather as signifiers of friendliness (there’s a line, of course).
Shall we unwind the perceived goal differently?
Of course, the platform’s actual customers are the advertisers (we are talking about a hypothetical platform, but where is it really different?), and by being free to the user it participates in a very limited oligopoly of big social so no, it doesn’t really care about what anyone really meant or intended, and it definitely isn’t going to hire real humans who’d make an effort at grasping the context of the conversation.
The real objective is for the platform to not have problems with law enforcement when one user complains about another user for being naughty, discriminatory, threatening, etc.—and, of course, we shouldn’t expect anything more from toxicity detectors geared towards that goal. As long as it’s not egregious enough that users leave en masse to a competitor (which can hardly exist, no honest business could reasonably compete with “free”) the platform wouldn’t care since users only matter to advertising revenue as cattle in numbers.
The waste of resources on detecting what is decidedly "toxic" on internet forums is insanity. If you want to police communities online hire moderators, if your platform is so big you ""can't"" have moderators moderate then you are not in a position to be policing the platform. If you want to build puritanical devices to spam your moderators into doing a human review than that is your business but the use of machine learning for any sort of proactive policing is going to be a parody that will result in a sterile environment and/or a lot of bitter users.
There's still big money in steering opinion on your platform the way that you want it. "Toxic opinion detection" is somewhere between a euphemism and a side-benefit.
I did research on this topic in a cyber safety research project. We focused on cyber bullying specifically but encountered the issue of non-toxic profanity as well. We used neural representation learning as well as feature methods and indeed, common profanity words are weak predictors in the better models. Still we found instances of non-toxic profanity being classified as bullying.
An immediate solution is to apply multitask methods to your target dataset and include the one proposed in OP. It's always good to have more resources like this, even though SurgeHQ overstates the size of their resources by large margin in their copy. The 1000 post instances of their dataset is far from "the largest": I have several aggression, toxicity and bullying human-annotated datasets right here with over 100k instances.
223 comments
[ 3.2 ms ] story [ 292 ms ] threadIt's also likely that there's a significant lack of context to the data points, not just because the posts are divorced from their parent content, but because of a culture and language divide between the labeler and the author of the data they're labeling, as well.
Hopefully people also don't need to be at the level of a professional linguist to label messages like "this is fucking awesome" correctly!
And great point on context. For example, the GoEmotions dataset didn't present labelers with the actual post or subreddit the message came from -- just the text itself. That makes it really difficult to label something like "his traps hide the fucking sun"! But once you see the comment in its original context https://www.reddit.com/r/nattyorjuice/comments/aee3wx/olympi..., and know that it's in the /r/nattyorjuice bodybuilding subreddit, it's much easier to realize that this is talking about someone's large muscles.
That's survivorship bias.
It depends on what the purpose of content moderation is.
Good if you want to accurately identify abusive behavior and protect users from harm? No.
Good enough if you want to find the most blatant examples of name-calling and insults to appease regulators and trigger-happy lawyers by appearing to use "state of the art technology"? Sure.
I had a housemate once who was an American linguistics grad and spent a year applying semantic labels for Google. I know he worked on a team, although I don't know what they did since I didn't work at a Google at the time.
Although -- and I say this having done a lot of my graduate coursework in linguistics -- I don't think having a linguistics background is particularly needed (unless you're doing specialized annotation, like creating syntax trees or tagging phonemes in Praat), outside of you being more likely to enjoy thinking about the nuances of language.
If you get normal not-always-online not-gen-Z people to evaluate these messages and label them as Good or Bad then you will get results like this. If I got any member of my family over the age of 30 to evaluate these messages, they'd label them all as offensive.
It's not impolite, it's just very informal.
I, on the other hand, am a fucking amazing shit OG bitch.
But yeah this stuff is absolutely dangerous, opinions like these are destroying culture. It won't be long until the threads these automatic tools are moderating, WONT BE WORTH HAVING.
'Nuff said.
How sure are you on that? I'm 31 and there's definitely no one in my peer group who would find those examples offensive. Hell, only 1/2 parents might be offended by that language, and my mom has been working around young people enough for the past decade that she might have softened up since I lived with her too.
There are contexts that quote would not be offensive to anyone (more casual settings with a group that trusts each other), but also contexts where everyone would find it offensive, no matter their age or generation.
Agreed. As an example, your usage of "black" and "white" in this manner could be offensive to someone in some context.
Not a single person I know would take offense to this.
My South African wife's family, however, is a completely different story. Her mother's side (English) would be horrified by this, whereas her dad's side (Afrikaner) wouldn't be upset in the least.
Point is, it's geography and upbringing far more than age that predicts puritanism. If anything, the young people from your area are regressing to the global mean.
Not everything is the Disney Channel and not everybody shares your views on "morality"
I actually think that too saccharine language is trying to hide something, so immoral.
Perhaps this is okay when your datasets are high-quality and representative of the real world, but they're usually not. For example, many toxicity and hate speech datasets mistakenly flag texts like "this is fucking awesome!" as toxic, even though they're actually quite positive -- because NLP datasets are often labeled by non-fluent speakers who pattern match on profanity.
(So is 99% accuracy or 99% precision actually a good thing? Not if your test sets are inaccurate as well!)
Many of the new, massive scale language models use the Perspective API to measure their safety. But we've noticed a number of Perspective API mistakes on texts containing positive profanity, so this post was an attempt to explain the problem and quantify it.
In the end you can be profane and very positive. You can also have strike language while being incredibly vile and negative.
I've noticed a lot of humans can't detect sarcasm. And that isn't correlated with traditional smarts either. For AI it's a hopeless task.
> You can also have strike language while being incredibly vile and negative.
I'm reminds of a group of people I know. They love love using genteel language to throw vile insults at each other. Using profane language is a automatic foul.
My wife's cousin recently had a baby, and posted a photo on Facebook. My wife commented (something like) "She's so cute I'll have to kidnap her". Facebook locked her account for 24 hours for making a "violent threat". I am sure the mother wasn't threatened by the remark in the slightest. It just added to my wife's anger at Facebook for repeatedly giving her "warnings" over trivial or innocent things.
Not online, in person, but one of the teachers at our son's school sometimes "threatens" to "steal him" from us – her remarks don't worry us, because we know she would never actually do that, it is just a colloquial way of expressing affection.
I don’t think the mother would be as accepting then, so it also matters who posts the comment to who
If a male relative or friend had made the same remark, would my wife’s cousin have felt threatened? My wife tells me she would not have been bothered by such a remark, in that context, from one of her male relatives or friends.
And, if someone was really bothered by it, they could report it. But, it doesn’t seem like that is what has happened here, it appears to just be the work of some automated system. I’m not saying that if the mother reports the remark Facebook shouldn’t take the report seriously - cultures differ after all, in some cultures those kind of remarks are normal in that context, in others they would seem bizarre and offensive - but it shouldn’t just assume the remark is threatening because it contains the word “kidnap”
I was more interested in hypothetical "can we conclude it is ok from friend status on Facebook without actually knowing involve people" question.
Which is to say: Often it's not reasonable to act without first knowing if the recipient actually took offence or felt threatened, before even considering anything else, because you can't know. A model aimed at judging utterances needs to also take into account "how close is this group? could this be in-group language?" and it needs sufficient context to judge that. Even then it'll have huge potential for error.
Incidentally, a lot of this wouldn't be nearly as much of an issue if we could trust that these kinds of tools would be used appropriately. E.g. instead of banning things, even just asking a "are you sure this will be taken the right way?" or similar is a whole lot more benign. The dating app Bumble uses a filter that checks for possibly explicit pictures, for example, but instead of blocking them it shows you a short message asking if you're sure the recipient wants to receive what you're sending, and lets you choose. Depending on use there might well be several levels of severity in messaging justified, from just a mild "are you sure?" marker to forcing you to explicitly acknowledge that you're on notice that you're responsible for the content.
Beyond this, by hiding the comment/post, the mother is now unaware of the threat and Facebook didn't contact local law enforcement in fact worse than doing nothing at all.
This is due to years of old media digging through Facebook posts to write stories like "Child kidnapped after threat on Facebook and Facebook DID NOTHING". This led to more and more calls for FB to "do something" to "keep people safe online". So they started to run all posts through the AI prescreener and well, these are the results.
But hey, they get to at least say that they take down X billions of posts before any other human sees it.
Now consider a large group of people upset about censorship and motivated to adjust their language to circumvent it...
The specific words don't matter, as any reaction would just shift the insult to another term. Ban cheeses? Start saying "I love you" ("mom, he told me he loves me! Tell him to stop it!"...). Or call him awesome. If anything, I think the brother missed a trick by not going straight to compliments - few things can be more cutting.
Given the right context it'd be clear to the brother what the intent was no matter the term, but increasingly impossible for the parents to distinguish "legitimate" communications from the insults. For that matter, no words are necessary to achieve that once both sides know. Just a look is enough.
We don't know how to fix this social issue with human censors, so it's much worse than just the state of the tech - it's an issue that's basically unsolvable without addressing why the speaker is intending to insult (if they are) and why the receiver takes offence (if they are).
The best you can do is address a small proportion of the most blatant cases (and that might well be ok and enough if you can do it with few enough false positives, but getting the false positives to an acceptable level is in itself a momentous challenge).
There of course isn’t a perfect solution and likely never will be.
The problem is that unless you observe the behaviour yourself as it happens, with substantial context, the idea that you can reliably know whether the behaviour was unacceptable is fundamentally flawed.
This is exactly what the choice of innocuous terms is for; to create plausible deniability. E.g. the word "camembert" thrown at someone out of the blue might be obvious because it stands out. But that's not the story you're going to get from the person who has used it as an insult. You're going to get a story about how they just talked about what they like to eat, or something like that.
You might see through that, but what quickly happens - what indeed happened with the brothers in my example - was that the person making the insults quickly calibrates to know exactly what results in a reaction and what creates enough plausible deniability that the person deciding if this is sanctionable is unable to determine with any reliability whether what they're hearing genuinely is an insult or the "target" being oversensitive and misinterpreting things. I was bullied at school at times, and saw this first hand as a target myself - the bullies quickly learned which ways they could word themselves to make the insults clear to me while creating enough plausible deniability even when there was no conflicting claims about what exactly they'd said.
Point being being that humans even with context and in the position to interrogate the people involved can't solve this with any kind of reliability. Then the notion that an AI at current state of the art without access to context or the ability to interrogate those involved doesn't stand a chance.
Yes but we're not trying to be perfect, we have to live and deal with this grey area.
> Point being being that humans even with context and in the position to interrogate the people involved can't solve this with any kind of reliability.
I disagree here, I think we can quite reliably do this a great amount of the time. There are absolutely bad actors who will get away with things in the short term but because context builds up over time it never lasts. And once you are aware they're trying to be subversive its much easier to spot earlier. It's not an attack with legs.
> Then the notion that an AI at current state of the art without access to context or the ability to interrogate those involved doesn't stand a chance.
I absolutely agree here.
And I'm sorry the people in charge were rubbish at protecting you from bullying.
For example, to pick a bit on the Google Emotions dataset again, it's difficult to label this message...
“Also Republicanism is a belief system. It’s taught and handed down like religion. Conservative talk radio is its evangelism.”
...unless you're familiar with US politics. Hence why it was labeled as APPROVAL by the non-US annotators, even though it's criticizing Republicans.
"I'm going to sleep with your father and then give him a son he actually loves."
Rated not toxic. All this API is going to do is promote a renaissance of polite burns.
> "this is fucking awesome!" as toxic, even though they're actually quite positive
And this touches on the question of what we want to do. It might not be a negative sentiment, but it might still be offensive. On the other hand, there are no offensive words, and only "positive" sentences in many sarcastic utterances.
And there are a lot of sentences that can be perceived as insensitive, that are not meant as such. GCP Grey has a video on the words "Indian" vs "Native American", and apparently it's very complicated which word you can or should use.
https://www.cgpgrey.com/blog/indian-or-native-american-reser...
No idea if there is a different metric that somehow (again) takes the number of sample into account.
You can see that my model is basically just filtering for profanity as an indicator of "strong emotion", which makes sense. But it's interesting that postive profanity seems to be such a thorny problem, at least for Perspective.
[1] https://twitter.com/celesteperez___/status/13508599618452070...
In general I've found the results to be much better for subjects that a lot of people are currently tweeting about.
It's very difficult when you blur the lines of code and ethics, as real world ethical judgements aren't necessarily consistent or well defined in a way which is easily translatable, even by a large ML model. Jigsaw is a great example of this -- right across the aisle (pre-pandemic) from Perspective is a team fighting internet censorship. Obviously Perspective's "censorship" is different in quality from the Great Firewall, but it shows the hairiness of the problems.
All this is to say the people at Jigsaw are some of the most brilliant people I've ever met and I'm glad they're out there working on difficult problems.
We deal with these hairy problems a lot too. Even before ML proper, getting the definitions right is very tricky. Should a comment that's polite and positive on its own, but supportive of a toxic parent post ("I love Nazis!" -> "I agree!"), be treated as toxic? Is "toxic counterspeech" equally toxic? What about a comedian making fun of an actor's nose, and does it depend on whether the joke is to that actor directly vs. merely referencing them in the third person? etc.
For these reasons, I actually personally like the fact that the Jigsaw annotation guidelines are very high level (as opposed to long and prescriptive) -- it lets the data capture the spectrum of "human preferences" on its own (at least, it does when you can trust that the annotators are able and trying to do a good job).
Given how eager Google itself is to automate all your interaction with them (and how hard it is to get access to anything resembling actual human judgment), did they really think it wouldn't be used that way?
Or did they just not care?
Also, I would have hoped the project's close ties to the US State Department would have worried some of you brilliant people a little more.
Similarly something that might be a cat call such as "Get your baps out" (shows us your breasts), could also be used by a baker since a "bap" is a type of bread roll in a slightly cheeky advert as most people are aware of the pun.
How are you going to train an AI to know the context that the person might be talking about bread instead of a woman?
Has anyone realised yet that almost all of this folly? I suppose not when there is money to be made.
For starters, you can't unless you actually include the context in the training set.
Then again, a lot of humans won't successfully manage that either...
It's worse than that. The context doesn't just have to be in the training set. It also has to be available on the other end, when you're using the model to make a determination. And it isn't.
Which is why even human censors can't get it right. The parties communicating can, and almost always do, have external shared state that The Decider doesn't. Which gives the same words different meaning.
Imagine hearing an inside joke you're on the outside of and then being asked to adjudicate whether it was offensive.
E.g. if judging tweets for example you'd presumably do a lot better if you evaluated the tweets in context of followers and in context of past tweets - both recent and the accounts history. E.g. an account that posts white supremacy tweets regularly is likely to mean something entirely different if RT'ing a BLM tweet with "that is fucking awesome" than what someone with #BLM in their profile is likely to mean.
Part of the problem is that they're throwing away a huge amount of the state they do in fact have.
But I absolutely agree it's in general an unsolvable issue - I pointed to a story from my childhood elsewhere showing how trivially people cause problems for "The Decider": A childhood friend being forced not to call his brother names and switching to using names of cheeses.
Is "Edam" an insult or a food preference? You can't know without context.
And it needs to be very local context, because humans very quickly pick up on when you just start using a word to mean something else, so it doesn't even need to be any shared external state about the word, just a shared understanding of where the receiver might expect an insult coupled with an unexpected response that will then easily get labelled an insult. That unexpected term might well in itself be positive if the receiver expects criticism. "Awesome" and "I love it!" are perfectly good insults when the other party has just told you something where the appropriate response would be negative, for example.
Must never use followers. Or anything else the account holder has no control over. Otherwise you'll end up with the same kind of SEO problems where competitors and foreign governments create fake accounts to follow or interact with disfavored ones and destroy their algorithmic reputation.
Guilt by association in general is malice. Someone who regularly interacts with white supremacists might be a white supremacist -- and maybe 90% of them are -- but it could also be someone criticizing, mocking or debunking them. And if it gets out that the algorithm is penalizing people for doing that, they'll stop. Which is very bad.
I specifically follow a few locals that I would often notice upset with the same politicians that I was, but for diametrically different reasons. Even though I disagree with them on most everything, I find value in having a few of those voices on my feed visibly attached to a consistent person. It helps me to resist seeing the "other side" as just an impersonal sea of voices.
100%. It's crazy how often context is missing from datasets! We wrote a separate blog post recently about this too: https://www.surgehq.ai/blog/why-context-aware-datasets-are-c...
I've been following some drama in chat/user interaction in modern games. People are getting banned in Forza Horizon 5 for funny decals. One guy got an 8,000 year ban for a Kim Jong-un KFC paint job (https://i2-prod.dailystar.co.uk/incoming/article25686363.ece...).
EA is banning entire accounts - access to ALL games on an account - for swearing in Apex Legends. All of the major studios are in the process of severely restricting and even removing chat in their games.
The latest Battlefield 2042 game doesn't even have voice chat, and they removed the global scoreboard altogether because they didn't want poorly performing players to know how poorly they were performing.
The list goes on and on. To me, this is all a serious reduction in the enjoyment I have in games. I grew up playing Counter Strike, and s**-talking and competing was a big part of the experience. I just won't play games where I'm not allowed to interact with other players.
That being said: I specifically only buy entertainment products that do NOT require me to listen to some 14-year-old describing sexual activity with my female family members in great detail because his nonexistent skills make me keep winning. Different strokes for different blokes.
At some point games companies were bound to notice the ways voice chat limited their market and remove it.
Stop taking features away from the rest of us
there are enough trigger happy people with less understanding than an AI, all eager to kick enough rages to wear down a saint…
This guy did: https://youtu.be/3-son3EJTrU
Humans are inventing double-entendres to create the ambiguity.
This is not a sustainable strategy for society. It is of course impossible for algorithms to parse perceived intent by the lowest common denominator, and attempting to do so is nothing more than defensive legal posturing.
And to shut up targets - "he/she does not mean that", "he/she is just insecure", "you know how he/she is". If you cant prove intent, you had no grounds to even talk about that that person is doing to you again and again.
Free speech is not without cost, but it has been successfully utilised to everyone's benefit for hundreds of years. I can only surmise that those advocating for this have never read about or understood the importance of the Enlightenment. They don't understand the horror of living under a social regime where the powerful control language and effectively thought. They don't understand why the ability to risk offense is critical for democracy, science, and social progress. Eliminating free speech will harm the most vulnerable in society; not the most powerful.
You are certainly correct - unfortunately the people burning the books are not the ones reading or writing them (even if they pretend to do so, with advanced credentials or padded resumes).
American history itself had tons and tons of situation where people risked a lot for saying things - including death. All within the scope you talk about. The time span you talk about include runup to civil war, civil war, reconstruction - all of which were full of political violence.
This also when dueling was the thing. Where men of means were expected to shoot each other over words. If you did not, you was effectively over in that town. And the line between "duel worthy" offense and not could be very thin.
> Eliminating free speech will harm the most vulnerable in society; not the most powerful.
Moreover, we are not talking here about legal standard at all. Neither was legal standard EVER. You could legally seek to insult and offend people as much as you wanted. Nobody cared about intent. This is about when people say "you are jerk".
There is threat to free speech, especially with new proposed laws, but it has zero to do with intent vs effect.
You would think we would learn from these examples and make offensive speech more acceptable, rather than less.
I'm not sure how you conflated legality with social acceptance. My arguments center around the later.
Now do European history pre-enlightement.
It was way, way worse. Shit talk the church and the inquisition slows up. Complain about the cut of your grain that the mill takes and the local lord threatens you lest other peasants start complaining.
The acceptance of freedom of speech as a general principal has greatly increased the civility of civilization.
E.g. people who object to a statement because it might possibly be offensive to another group, despite members of that other group being present and not being offended. Clearly they are doing this out of the same enjoyment of other people's discomfort that you describe.
That said, it's interesting this has come up during a discussion about how the intent behind what someone said is often either misunderstood or discarded.
Other times, you get the "well it's so very easy to make this small change, it's practically rude (or suspicious) not to" - a completely wrong sentiment for many reasons, but easy to make you look difficult for arguing about.
You are assuming way too much good faith. For every one that does it in earnest there's probably 10 grifters just taking advantage of the situation for their own personal benefit:
The real reason seems to me like virtue-signalling. They don't care whether they're personally offended, as that would require burning social "points" by rocking the boat but potentially getting no outcome out of it (nobody else might agree). It's "safer" to just suck it up than sacrifice social points especially when you're on the lowest ranks.
They also don't know nor care whether the protected class they are worried about is actually going to be offended. Again, rocking the boat and burning social capital for little personal benefit.
This leaves the argument of earning social capital by virtue signaling among like-minded peers. Pick a common issue you already know is popular within your target "market" (racism, sexism, "diversity & inclusion", etc) and raise the problem. This doesn't risk any social capital, after all, people like you won't disagree (for the same reasons) and people that might be on the fence or would like to add nuance wouldn't want to risk their social capital in fear of being labelled a racist/etc. It also transcends the ranks - if anything, a higher-ranking person in the company would risk much more by speaking out than someone on your rank or lower.
This eventually results in a positive feedback loop where everything will become offensive given enough time and those who disagree either comply or get pushed out. That's how you get bullshit like GitHub's "main" branch which helps nobody while continuing to sell services to ICE which you could argue actually hurts minorities.
And therein lies the problem. A subset of people get a benefit from engaging in this behavior. So of course they do it at every opportunity.
You get what you incentivize.
Until the outcome is so frequently a net negative (however slight) in so many cases that the behavior is marginalized to a negligibly small number of people/settings the behavior will persist in enough volume to be worth caring about.
The problem is that a large part of the tech industry nowadays relies on "engagement" aka monetizing user attention for the benefit of advertisers, thus extremely vulnerable to public opinion (the advertisers themselves would engage in this behavior, thus companies that supply them must also conform to it). The entire industry is on thin ice already so it makes sense for everyone to play it safe even though it just reinforces the feedback loop even more.
This is why this behavior is rarely seen in other industries that make the bulk of their money on a tangible output, both because the output is less fungible/replaceable as well as those industries' employees typically having less time to waste engaging in bullshit.
I'm just going by conversations I've had with people in person and online, and assuming that this is broadly reflective of society overall. Don't get me wrong, I am quite sure that there are people who are cancelling people for sadistic pleasure and get a kick out of torturing someone and use hot-button issues to do so. However I cannot imagine this type of person outnumbering those with good-faith concerns at a 10:1 ratio.
There is no happy medium here, particularly since those seeking to offend and to wolfwhistle to signal their allegiance to other people who hate your race (for example) will try to couch their terms to hide within the shadows of plausible deniability unless they feel completely safe.
It's more a question of how many false positives and false negatives your culture is prepared to tolerate.
I certainly agree with your final conclusion: it depends on what society is willing to tolerate in terms of false positives and in terms of false negatives. And particularly, the balance between those two kinds of errors.
Not to say that there isn't anything wrong with how people have been increasingly incensed in the last few decades, but that's basically what it was like in the 90s and before. Someone was like, "whoa dude what you said wasn't cool" was met with "it was just a joke".
Your reasoning cuts both ways I think.
If you want to be a global publisher of your opinions, you’re going to receive a global response, from people across the political and social gamut — the larger your audience, and the more political/ideological your speech, the more substantial a response.
My dictionary’s definition of “Faustian” has this usage example:
> Modern celebrities enter into a Faustian pact with the general public.
What people seem to want is the ability to enforce a one-sided bargain with the public, under which their own toxicity, opinions, and any public affirmations should never be censored, as long as their opinions are the politically correct ones.
So, give hard core racists the benefit of the doubt when they wolf whistle each other?
Can't see any historical precedent for that going disastrously wrong.
>What people seem to want is ... their own toxicity, opinions, and any public affirmations should never be censored
Some people are hypocrites. News at 11.
Some people also want anti-han Chinese racism to be treated equivalently to anti-Uyghur racism because they choose to be blind to the inherent power differential.
I remember reading a story of the Hutus and the Tutsis. The genocide began with the radio - shows that very deliberate but cautious (& deniable) dehumanization of the other that ramped up over time. Nobody policed these deliberately aggravated ethnic tensions. By the time they were consistently obvious it was a little on the late side.
On the other hand, I've been falsely accused of being a racist before and it undeniably hurt my feelings a little bit.
- all social media implemented simple word filters. A word is on YOUR list, the post or chat line doesn't make it to your page/chat. It'd be awesome if these lists were exportable, syncable, and tradeable online.
And I wish the AI revolution would benefit me personally instead of companies.
I would like an AI that censors on my behalf only if and when I want it to, and a personalized AI that I can train to flag accounts/people that I don't want to interact with based on language. I would also like some mechanism to indicate how much is getting caught in the AI-generated filters and allow me to stop or modify its behavior. Once the AI is doing mostly what I want, I would like to be able to share it. I don't know how that would work.
- Group owners should have similar mechanisms for all posts in a group.
I think that both intent and perception are important, and they are both judged in the more modern understanding of "toxicity". Malintent is a direct "strike". But, lack of malintent doesn't mean that perceived offense should be entirely ignored.
Let's imagine a scenario: as in the example above, an English person says "can I burn a fag" and an American person believes they are casually thinking about hurting a gay man. The English person meant no harm; but, the American person is not at fault for not understanding this. They can complain, and the English person should explain their intent. The American person should learn and remember the meaning of this expression in British English, and the English person will need to remember that this expression may sound offensive to American listeners, and avoid it in such scenarios.
Basically, the important aspect is that a misunderstanding is not always the fault of the listener (which the intent-only model suggests: you took offense, but I didn't mean that, so you're silly for feeling offended). It is of course also not only the fault of the speaker (which the perception-only model would suggest: I felt offended by what you said, so it doesn't matter what you meant, you owe me an apology or more). Both parties are involved in a miscommunication, if indeed there was one.
FYI, it's combination of difficult-to-discern keming with the default font on HN, and your unfamiliarity with the phrase, but it's actually "BUM A FAG" and not "BURN A FAG".
Incidentally, did you intentionally write keming instead of kerning?
If you call someone out for "offensive speech" and your offense rests on your own misunderstanding, it is absolutely your fault.
Unfortunately, we seem to be in a place where bad intent is taken for granted and merits an aggressive reaction rather than placing any responsibility on the offended party to be civil and open-minded in their response to be perceiving offense.
In the case of "burning a fag", our hypothetical American has the option of inquiring about the meaning of an unfamiliar idiom rather than jumping straight to calling out the speaker.
bum is not burn.
As a white guy wearing a suit, I got out of there without being charged. If I weren’t… I can imagine it could have gone very differently.
Also, “is was part”? So much for being clear.
Part of being an adult is understanding that generally, people are not 4chan trolls.
If someone who's not a crackpot says to you "can I burn a gay", even though that seems unambiguous, they've probably just misspoken or are joking.
In real life almost everyone seems to get this, online it's like everything is a strawman.
Well, for one, this only applies if you already know that person is not a crackpot. Secondly, burning someone would indeed be an extreme example, but not all (accidental) offenses are so obviously monstrous. If someone said "I hate fags" (intending to say they don't like cigarettes), this would be more likely to be someone's actual homophobic opinion, depending on culture.
It's all just a bit weird to me. If someone said to me "I hate fags", I'd either ask them to clarify, or change the topic. If they were persistent, then yeah, get rid. It's not a problem that requires instantaneous resolution.
I don't really want to take the discussion in this direction, but note that almost all of that "shift" comes from one side of the political spectrum.
https://groceries.aldi.co.uk/en-GB/p-mr-brains-6-pork-faggot...
Before I noticed this comment, I made one of my own, making the same point - and it seems HN has marked it as dead for much the same reason: https://news.ycombinator.com/item?id=30069899
‘Tis ironic. (edit: thank you to whoever vouched my comment, it isn't dead any more)
Obviously also a technical term used all the time in its technical sense with no problem.
Context and culture is way more important than the actual words used when trying to determine the meaning of a statement.
https://www.urbandictionary.com/define.php?term=nonce
"Obviously also a technical term used all the time in its technical sense with no problem."
Or sometimes not ... a blog I'm involved with recently had a WordPress server problem which caused commenters to receive a message like "Illegal operation: bad nonce". The readers are non technical and mostly British. Some had to be calmed down a bit!
Maybe it's an age thing? Everyone my age knows what nonce means but mostly because it was so amusingly satirized by Brass Eye in the late 90s at the height of a media induced paedophilia scare. Chris Morris got politicians and celebrities to say stupid stuff on camera by telling them it was for charity literally called Nonce Sense:
"Did you know that a child under the influence of a paedophile may smell like hammers? That's right - I'm talking Nonce Sense"
https://thesundae.net/2021/01/21/talking-nonce-sense-in-prai...
The program was a sensation at the time but probably only within a certain age group. I can't imagine that sort of satire would have appealed to older people.
There have been some unfortunate mistakes from companies not aware of this. https://www.dailymail.co.uk/news/article-9760601/Cryptocurre...
Another example: faggots are a traditional meatball dish from parts of England and Wales-the term has nothing to do with anyone’s sexuality. And yet, American social media companies have repeatedly blocked British users for posting about the dish.
There is also a British pudding called “spotted dick” - in this context, “dick” is a dialectical term for “pudding”, no known connection with genitalia.
A friend of my wife recently had her Facebook account restricted for making a post about cooking jerk chicken for dinner. Facebook claimed she was using “abusive language”
[1]: https://news.sky.com/story/spotted-dick-renamed-spotted-rich...
Or try explaining panto to Americans. Or even Allo Allo, a show which would .. definitely not be made nowadays.
I've seen false positives referred to as the "Scunthorpe problem". Someone on the internet has a nice map of filthy-sounding real UK place names ...
There's a reason Essex University had to register sx.ac.uk as an alternative domain name.
This isn't documented anywhere, but I find it hilarious and dystopian that posts that contain the world "hate" in the description or in the comments get blocked more easily, or soft-banned (they don't show up in anyone's feed), even if the word is used in harmless contexts...
Many business models based on user generated content wouldn't be possible if the businesses had to pay minimum wage to people for moderating that content. Using an AI model, no matter how broken, allows them to seem more concerned than if they were just relying on an old-school word filter without doing any actual due diligence.
Yes, of course everyone realizes that no filter actually works in practice. But they are not built to work, they are built to give the smallest impression of working. Tumblr doesn't need their site to be absolutely clear of nudity to attract investors, they only need to give the impression that it's absolutely clear of nudity and that they are trying to keep it so.
Sure "AI" can do a lot of impressive first-level pattern matching, and that can be the basis of many useful outputs.
But ANYTHING that requires actually understanding the context, whether it is existence of new obstacles for a 'self-driving' vehicle, understanding of actual meaning for a language model, or anything else, is a complete and utter failure.
Despite appearances, while some of it is genuinely useful, we're really no further than fancy parlor tricks. Crack even the next level of contextual understanding, and it will be an astonishing leap.
Content management done well is expensive, and requires humans with superior grokking skills. No place cares enough to try.
Anyone selling AI-based sentiment analysis is a grifter.
we need to get rid of black people
we need to get rid of black people poverty
we need to get rid of black people below poverty level
we need to get rid of black people hurdles keeping them below poverty level
the gist is that without unravelling a sentence full context, a lot of verbage can refer to a lot of different action.
focusing on profanity is the low anging fruit, so to say.
The same words can change context depending on what kind of building you're in or what time of day it is.
Even common phrases like "I'm running over" could mean I'm coming to see you or I'm taking longer than I expected; completely unrelated things.
Flow analysis may give you something, but content analysis is basically impossible unless there's also context analysis.
But since antagonizing trolls aren't usually using profanity or necessarily a different vocabulary of words but they are engaging in patterns, that kind of engagement style analysis may be all that's actually needed
We need to come up with UI patterns and flows that reflect this otherwise ML solns will continue to disappoint.
I'm not convinced this solves anything or even helps - UI patterns can easily be abused by bots and just shift the problem into a different direction.
We need to develop a way to work with ML st most queries can be answered by ML but there is an exception pattern that defaults to a manual, prescriptive process, or whatever.
Also we need APIs to allow detection of that state.
There are some differences in German, too. For example "wixen/wichsen" is an old word that means to wipe/shine your shoes and is still in active use in this sense in Switzerland as well as Austria, however it lost its appeal in Germany, because it is now primarily with a different meaning. The Wix company took this different understanding of its brand name to an ad: https://www.youtube.com/watch?v=IddnMutPgTI
Since we have an IT background here, same goes for "Mongo", like in MongoDB. Mongo is considered making fun of handicapped people in Germany.
Former Fraport AG changed its brand name because it was abbreviated FAG - Flughafen AG and found it difficult to expand business with that brand name.
No bad actors, if you ask me, only different context. List could go on and on...
"Hold on, my dock is nearly up" "Hold on, my ** is nearly up"
Censorship can actually make things MORE offensive. (other censored words include "but" and "come")
Still, all in all, automatically replacing a (supposed) swear/offensive word with some asterisks makes less overall damage than banning someone's account (temporarily or for longer periods or forever).
It is simply intimidating when you have to think twice or thrice before saying something for fear of being automatically banned (without possibility of explanation/recourse).
People with Down's syndrome used to be called "mongoloids" because they were thought to have Asian-looking eyes (epicanthic folds), so calling someone "mongo" or "mong" is basically an extra racist way of calling someone mentally retarded.
It had to go? Certainly not gone from git itself, Github or most other Git hosting services.
If someone creates a bug report on your project about virtue-signalling word changes and you give it credibility, it doesn't mean the original word went away from the lexicon.
Delegating content control to an AI (that doesn't qualify for anything intelligent) is not a working solution.
> as a first-pass filter, leaving final judgments to human decision makers — marking all profanity as toxic can make perfect sense
You would need humans to look at profanity constantly.
> Our mission involves creating a safer Internet, but we don’t want to miss out on our favorite content because of AI flaws in the meantime.
There is a limited AIs that do create content, but a profanity filter always does the exact opposite.
Instead of putting money into gathering quality data, tech giants instead chose to use cheap labour (here: India) with insufficient skills. You get what you pay for.
yep. The cultural imperialism is an open secret. But I've completely lost any ability to tell the difference between people pretending not to know, and those actually not noticing.
I'm guessing it's only a matter of time until this comes to toxicity models, so they can look up who said it, and to who it was said.
I recently started playing counter strike source again online, just for 10 minutes of fun at first (to see if it would still tick with me). I randomly picked up a server and the ambiance was cheerful and nice. I noticed the rules said "no profanity, have fun" and indeed people were mostly polite.
I tried another server at random a bit later and there was more insults, along with a lot of taunting.
I switched back to the first server and have been regularly playing an hour or two every three days and there is a difference with other servers. Some random people coming and throwing insults, even mild ones like "fuck you" or "you son of a bitch awp" get insta ban and it makes the whole session a much better experience. Maybe it's a safe place but playing with polite people is more enjoyable to me now than playing with insult gatlings.
Language is political. There are many meanings to words, depending on context but I do think it's not innocent to swear in front of people or to use swear words to look cool. These are still swear words and insults and their first original use is to provoke or taunt or display aggression. Even if it's only used for "this is album is the shit !", it's still a (childish) provocation. Reminds me of the brogrammer fad.
FWIW: I get regularly owned on this server and I am at the bottom of the ranks but it's still more fun and enjoyable than other servers I tried where I can reach the top but... it's not a nice place. I think online servers are like bars.
Side-note: I was pleasantly surprised to see that "gg" is still thrown around after rounds :). It's way better than "git gud" that came later and that I find horribly toxic.
You say that like having a place be safe is inherently negative. But your post is a textbook example of why people want safe places!
I'm just saying, I'm not surprised that a moderated server is more, uh, moderated, compared to an unmoderated server. I'm not sure the individual word choices are the determining factor compared to having someone paying attention and removing people with undesired behavior.
Hard to tell because of the different language. They wouldn't engage someone saying something close to that but people wouldn't express their admiration or congratulations with swear words anyway (because it's not the style of the house ?). Of course they'd be using the community's lingo though to express sick moves but these aren't insulting/taunting/agressive to begin with.
> I'm not sure the individual word choices are the determining factor compared to having someone paying attention and removing people with undesired behavior.
I think too. For the comparison I should try a CS:GO or valorant or pubg or fortnite next week-end to see the difference.
In a similar vein, there are popular "AI Mental Health" apps I've gotten to straight up instruct me to end my own life with some trivial conversation.
EDIT: Here's one, though I don't think it's ML. https://text2data.com/Demo
> It would be really nice if you'd end your own life :) Everyone would be happy.
> This document is: positive (+0.62)
For https://monkeylearn.com/sentiment-analysis-online/:
> Positive 84.1%
For http://text-processing.com/demo/sentiment/:
> Pos 0.7, Neg 0.3
For https://aidemos.microsoft.com/text-analytics:
> 100% positive
For https://komprehend.io/sentiment-analysis
> Positive
- unpleasant discourse on online platforms is blamed on the platforms
- the platforms can't moderate this manually (nor objectively), so they look for a tool
- a tool can't possibly do anything useful, but it satisfies the "something must be done" media demand
- the tool will hurt conversation, and people, and potentially eventually threaten the platforms it runs on
- but those problems feel smaller than the demand that "something must be done"
Is it the lofty “make people be nice to each other”?
Well, the paradox is that being nice is possible with the strongest choice of words, while being very harsh can sound most fluffy bunnies on the surface. In fact, in human relationships there are degrees of mutual familiarity where being exceedingly polite and not “insulting” your counterparty would be perceived as negative—where insults are not taken at face value, but rather as signifiers of friendliness (there’s a line, of course).
Shall we unwind the perceived goal differently?
Of course, the platform’s actual customers are the advertisers (we are talking about a hypothetical platform, but where is it really different?), and by being free to the user it participates in a very limited oligopoly of big social so no, it doesn’t really care about what anyone really meant or intended, and it definitely isn’t going to hire real humans who’d make an effort at grasping the context of the conversation.
The real objective is for the platform to not have problems with law enforcement when one user complains about another user for being naughty, discriminatory, threatening, etc.—and, of course, we shouldn’t expect anything more from toxicity detectors geared towards that goal. As long as it’s not egregious enough that users leave en masse to a competitor (which can hardly exist, no honest business could reasonably compete with “free”) the platform wouldn’t care since users only matter to advertising revenue as cattle in numbers.
An immediate solution is to apply multitask methods to your target dataset and include the one proposed in OP. It's always good to have more resources like this, even though SurgeHQ overstates the size of their resources by large margin in their copy. The 1000 post instances of their dataset is far from "the largest": I have several aggression, toxicity and bullying human-annotated datasets right here with over 100k instances.