Path in the leak is:
safety-ml\offensive-usernames\data_pull\sql\bad.sql
Highlights:
CREATE OR REPLACE FUNCTION is_tragedy (VARCHAR) RETURNS BOOLEAN STABLE AS $$
SELECT replace($1,'_','') LIKE '%george%floyd%'
$$ LANGUAGE SQL;
CREATE OR REPLACE FUNCTION is_derogatory (VARCHAR) RETURNS BOOLEAN STABLE AS $$
SELECT replace($1,'_','') LIKE '%retard%'
$$ LANGUAGE SQL;
Now we know we can make as many goergefloyd accounts as we want! *devious grin* *chuckles to self*
My understanding from talking to {current,former} {Amazon,Twitch} employees is that Twitch has retained a decent amount of engineering independence. For better or worse, it's unlikely that some rando at Amazon ended up with this particular PHP file on their desk.
Truth is, Amazon is the parent organization of Twitch, so calling Twitch employees Amazon employees as well is perfectly fine. Just like calling GitHub employees Microsoft employees.
So yes, Amazon employees have ended up with this particular file on their desk.
I have a hard time believing this was / is the real version used. It doesn't seem broad enough. More likely it was a kind of smoketest that made sure that a more automated keyword checker was working.
It also mostly checks for English naughty words and not much else. People can have fun in lots of other languages, so it would seem this is a small sample.
Woah, 20+ years later and I literally had no idea until now. Learn something new every day, but somehow this one is shocking because of how obvious it is and how long it took :)
After looking through it quickly it seems to do the same as most profanity checks with Dutch: it doesn't block "kut" (a crude word for female genitalia) but it does block "kunt" (third person form of "can")
Ah, that's right—need to have had an account for four days and... to have clicked the 'Edit' button at least once?
That's what the linked page says (I'm not familiar with Dutch Wikipedia specifics), but that seems like such a strange statistic to track instead of minimum edit count.
So weird that I looked up how to configure MediaWiki autoconfirm requirements. They probably mean a minimum of one edit. Auto-confirmation considers only age and edit count, and I don't see why clicking 'Edit' is so meaningful a condition to warrant developing an extension.
I can't find the article now but the developer who wrote these scripts said they were a singular effort from years ago before security was taken over by a more formal development team.
Training an ML model to "learn" a rules engine strikes me as an incredibly bad practice. It'd make more sense to just have an actual corpus of labeled data.
Would be unsurprised if some poor engineer got assigned the project, realized it was an untractable mess of scunthorpe, and decided to check some boxes and move on to some ticket of higher value.
This seems like the kind of thing that would be horribly specced and be a user story along the lines of "the user must not be allowed to make an inappropriate username."
The engineer would write something for every test case the product manager complained about, anything else computationally easy, and call it a day.
I once had to implement an audit logging system. What was supposed to be logged? "Important actions." Nobody on the team could define it. We just logged every database write along with the username responsible and called it a day. Nobody ever followed up or inspected it.
Bad, old memory: "Every transaction must have a line written to the printer"
Tracking down "why is the antique system suddenly slow". Power went out, system came back up fine, everything but the one ancient but vital app is fine. Dig, dig dig, there's this old dot matrix printer in another room (because it used to be loud and annoying) that no on has fed or looked at in years.
It finally died with that outage, and it not accepting data was the problem. It had cheerfully printed the ribbon through, then fed out the rest of the box of paper it had, and that might've been several years before i saw it.
The roller the paper was supposed to ride had been eroded. The metal rods the print head rode on had a perceptible bump at the ends of the normal stroke.
The fix was a little dongle for the printer port that held the appropriate "i'm alive" lines up. hardware /dev/null. I'm thinking it was 25 pin rs232 because I remember a lot of cussing over it.
Dot matrix printers still make up a lot of the flight manifest printers at airport gates. Listen for them right before they close the doors. They print off the list of passengers checked in as boarded
I want to say it was probably installed in '85 and I saw it in '94; but i wouldn't swear to those dates.
I'm pretty sure it was the only dot matrix printer with a serial port i ever saw. Even daisy wheels were parallel port by the time this went in; but they had a like 50ft cable to move it to the other room. Someone worked hard and paid large to set that up originally.
In a previous life, all of our code was scanned for "vulnerabilities". One of the issues they looked for was if passwords were being stored in local variables. Initially, LOTS of people would do something like:
$Username=<username>
$Password=<password>
Connection.string=($Username, $Password)
The parser would flag this - Password was being stored to a variable! So we just changed our code:
This looks like legit, no-nonsense gets-the-job-done code that gets updated every time some jerk find a new way to be a jerk to others. It isn't great, but at least not over-engineered, and I'm not sure if Twitch account sign-up volumes and abuse are at the point where they should staff a project to do this more robustly / scalably
Exactly what I think. This is the kind of code that doesn't have a platonic ideal, it has to get updated with time and experience and reports. There is no "non-hacky" way to do this, you just have to look at the reports that are coming in and keep adding rules that are relevant.
Well there are some significantly less hacky / more scalable ways to do this. This looks at the edge of maintainability, but if the lists were 10x as long and people constantly stepped on each other's toes causing outages while making updates, a bit of project investment probably wouldn't be a bad idea.
There are portions of my codebase that are intentionally "dumb" code. They contain cascading rules controlling what UI elements are visible that can be challenging to reason about. So I wrote it so simple that anyone can read it.
I see the same here. It's not clever, but no one has any doubt what words are being checked.
Seems plausible to me. One of the streamers on Twitch had an actual wooden board on the wall he lasered subscriber names onto, and some of the regulars had fun finding lewd usernames to gift subs to. There were quite a lot of them out there. It was kind of a running joke how much Twitch let through the cracks.
looks like it is limited to English only. There is whole world of non-English offense out there. Like using English characters to make national offensive words as well as using national characters to make English offensive words.
Should optimize checks with a de-obfuscation function (attempt to expand non-AZ back to AZ, even if that then shoots permutations at banned word-runs).
It should probably also look more like a spam scoring system, where really obvious stuff is hard-trashed but borderline things are flagged for review / discussion.
I am also very disturbed that, as with most censorship, 'obviously bad' things such as terrorism/etc are co-mingled with 'is adult' as a negative check.
It seems reasonable for Twitch to have validated 'safe for minors' areas where names are filtered. Generic areas, where things are in the gray area and unchecked. Adult Only areas, where swears, profanity, maybe even some of the hateful things are allowed. Informed consumer choice.
> CREATE OR REPLACE FUNCTION is_hateful (VARCHAR) RETURNS BOOLEAN STABLE AS $$
>...
>OR replace($1,'_','') SIMILAR TO '%(kill|keel|hang|burn)%(black|bl4ck|black|jew|trans|gay|african|afrikan|minorit|asian|nig|n1g)%'
Typical Twitch (D-CA). All (D-CA) companies really. Under the law (in the USA), all races, and sexes are equally protected. On big tech platforms it's only one slice of demographics that harassment and hatred is tolerated toward, even encouraged. Do they actually think this is helping? Singling out one demographic for abuse? Do they think this is progress? That people won't notice? That it will never backfire?
If you liked this chaos, you'd love my 15+ years of cobbled-together efforts at limiting forum spam and the like. I suddenly don't feel quite so alone in the myriad efforts needed to tackle this sort of thing.
It's an endless arms race. I spend a lot of time studying unsavory people and a great deal of effort goes into the development of new dogwhistles that are designed to either provoke or connect with peers while maintaining deniability. You might like this paper on the evolutionary dynamics of covert social signaling: https://www.nature.com/articles/s41598-018-22926-1
I can really identify with that. My efforts are against spam and trolls; at least the spam is predictable and easier to take a sledgehammer to! Those grey-area trolls are such a miserable part of online content and moderation.
If they're covert and only members of the group using them know what they mean, then why fight it? Seems like a helpful way to communicate about things that others might not like to hear. The paper you linked said that too - "Such signals may allow coordination and enhanced cooperation while also avoiding the alienation or hostile reactions of individuals with different preferences.".
This happens all the time in cartoons that appeal to children but also contain subtle adult jokes so everyone can enjoy them on different levels.
Because it encourages more of the same, and it's rarely completely covert. Often and increasingly, it's racial dogwhistling and just devolves every thread to wasteful, antagonistic and unproductive conversation. You might've heard the line/story about accommodating nazis in your bar.
At an online company I worked at we had various word filters for our forums, but new stuff was always popping up and getting through.
What worked in the end was having any newly created thread send a message containing the post title & body to a slack channel specifically for monitoring the forums. Employees and our forum moderators were in there, and any bad threads were nearly instantly deleted. Eventually the spammers mostly gave up. Hard to beat a dozen human brains :)
I get alerted to every new thread, and when I'm online my response rate is also very fast. When the message count was under 1,000/week, I used to get an email for every single post too. But if other moderators aren't around and I'm offline, I am out of luck. Shadow-banning can be effective. I also give regulars the ability to sin-bin any post which removes it from view and leaves it for me to check.
Having a dozen reviewers, especially if spread across timezones, would be a dream!
CREATE OR REPLACE FUNCTION is_blasphemy (VARCHAR) RETURNS BOOLEAN STABLE AS $$
SELECT replace($1,'_','') SIMILAR TO '%p(o|0)rc(o|0)di(o|0)%'
OR replace($1,'_','') SIMILAR TO '%p(o|0)rc(o|0)mad(o|0)nna%'
$$ LANGUAGE SQL;
c) "Porco dio saranno mica i testimoni di Geova? No eh diocan digli che i signori sono fuori, non ho tempo per stargli dietro."
("Fuck, they can't be Jehovah's witnesses, can they? Tell them we're out, we don't have time for their shit.")
is not even remotely complete, in italy we have two regions dedicated to the creation of blasphemies so advanced in ingenuity that two telephone books in regexp would not be enough to stop them
This is pretty much the same problem as Email spam. There needs to be a service/collaborative project to filter these, instead of each app hacking up an ad-hoc way of doing it.
We had to do this for a link shortening system (to make sure random base64 didn't contain profanity). It was a pretty fun problem. Not just the implementation, but doing the math to make sure it didn't make our shortened links easily enumerable. The implementation wasn't too bad, but we set up logging initially to spit out any random strings it decided to block. I demo'd this in front of the whole company and live tailed the logs and the first one that popped up during the demo was a big ole F bomb. It made for an excellent demo.
Fascinating! I guess an easy solution is to inject non alpha characters into any generated string. I imagine a constraint was that you wanted them to be easy to type?
SMS is the biggest constraint. Unicode characters trigger lower segment char limits (effectively doubling the cost of a 71 char text message). And also it's important that the links can be clicked on a smartphone. So url-safe base64 (some shorteners use base62). And numbers can be N4u6hty too, so you gotta catch those cases.
The hardest problem with the implementation was that with a long list you can't just search for a few dozen inappropriate words (like the Twitch implementation). It would be very expensive to do hundreds or even thousands of checks against every inappropriate word.
The solution we came to was to truncate all the inappropriate words to either 3 or 4 letters and store them in a big set. We then take our generated strings, which are usually 11 characters, and break them up into all possible substrings of lengths 3 and 4. For example, 1a2b3c4d5e6 would be broken down into 1a2 a2b 2b3 b3c 3c4 c4d 4d5 5e6 1a2b a2b3 2b3c b3c4 3c4d c4d5 4d5e d5e6. An 11 character string would always have 16 such substrings. We then check all 16 against the banned set. 16 lookups into a set is pretty cheap and as we have expanded the word set over time (e.g. add a new language) our performance hasn't changed.
One drawback to our approach is that we do have false positives but we did the math and our space was still large enough, the cost of generating a new one was pretty low, and customers never see it so it's just not a big deal to throw out false positives.
I agree, I only wanted to point out that they can I they want to. I didn't say they have to
Edit: I'm being downvoted so I want to explain - the internet is a huge mishmash of different cultures and all I wanted to say is that it is allowed to swear because I though that maybe, in their local one, it is not and they think it's universal
It’s so laughable that we care about whether a generated string contains some temporally relevant profanity. We truly are still barbarians, and will be viewed as such by history.
Well, given that we have a more than 5000 years old habit of looking for omens in random data to divine the future (whether that random data is scattered bones, laid out animal entrails, tea leaves, coffee grounds, tarot cards or so many others), it's unfortunate but not surprising.
It highly depends on what the purpose of the string is: if it can be clearly seen by the user, has to be read, or worse typed, then it's not just a random string in a database.
If you generate an identifier for an important client that contains "knobhead," they won't think it's a randomly generated string, but that someone at your company is deliberately insulting them.
When it comes to censoring randomly generated strings, I like simply to omit vowels from the alphabet. Usually I'll omit some of the more obvious lookalikes too, e.g. [1 0 v].
It's a simple solution. Sure, it is still possible for something to slip through that looks similar to something bad. But the potential to strongly offend is greatly reduced.
Yeah there was an article I read a while back about a company looking to prevent the use of 'naughty' words in randomly-generated strings used as event IDs. Apparently someone with some pull had seen a message with an offensive word. Some management committee spent a long time trying to figure out how to solve the problem including proposing keeping a list of bad words, and then worrying about what should be in it and who would maintain it. At some point an engineer got a chance to speak and said something like "just use base-31 and omit vowels". The story as I remember it didn't mention the use of v or l33t-speak, but they were randomly generated, not maliciously constructed, values.
Hashids does this (avoid bad words) if anyone is curious to see an implementation
> algorithm tries to avoid generating most common English curse words by never placing the following letters (and their uppercase equivalents) next to each other:
Oh yes, there's plenty of ways to avoid curse words, but each one has a cost, and if the system needs to generate ids very quickly, any scheme that works too hard could be a bottleneck. The naive way of just randomly throwing together letters of the alphabet will eventually generate a forbidden word. Any steps taken to reduce the probability should understand the time/space tradeoffs.
That's amusing, but I think it also highlights the effectiveness of the strategy. WNKR is excusable and defensible. WANK would not be.
Edit: But I'll concede that when your outputs are only four characters long and end users will actively interact with them (write them down, type them again later, etc.), additional safeguards might be appropriate. Or simply omit all alphas and use only numerics.
> Or simply omit all alphas and use only numerics.
You're still not out of the park with numerics - people with 1313 or 6660 or 4444 or something will complain a lot. The possibility of a 666 in some new biometric government IDs in my country rose a massive stink from church...
> When Beijing lost its bid to stage the 2000 Olympic Games, it was speculated that the reason China did not pursue a bid for the following 2004 Games was due to the unpopularity of the number 4 in China. Instead, the city waited another four years, and would eventually host the 2008 Olympic Games, the number eight being a lucky number in Chinese culture.
In Germany, where you can request number plate combinations (as long as they are free and follow a few roles), 666 is a pretty common combination amongst young drivers.
> What’s the issue with 4444
4 is pronounced similar to "death" in sino-japanese languages and dialects.
Also, as you have the muncipality-shortcut at the beginning, and then two user-definable letters, you freqently see "rude" combinations, and noone bats an eye.
BIT-CH, MON-GO, ANA-L, DIL-DO. (And those are just the combinations that are understandable in english which I saw when driving). I'll never forget the face of the guy at the Zulassungsstelle when I was there with a friend who wanted COC-K-6969. He got it it, btw. When I tried some years later, AC-DC-666 sadly was already taken.
it spells ACAB if you match each number with the letter in the alphabet at this index (I realized that after seeing a bunch of 1312 tags around where I live)
Yeah. An important, long-lived ID that will stick with an individual for their entire life, and that they may want to commit to memory. That seems like a good time to take a hypersensitive approach and adopt some kind of filter.
My girlfriend got a new bank account and when she received her account number it contained 666. She asked for a different number and they changed it without charge.
> Or simply omit all alphas and use only numerics.
That works pretty well until you realize that some numerical combinations are common neo-nazi codes and may lead to ... unfortunate associations. The ADL lists a few of those^1, but the list is by far not comprehensive, codes actually differ based on locality, and accidental combinatory collision in a 10-character space than it is in an alphanumerical 36-character space.
> to make sure random base64 didn't contain profanity
I would have said "why bother" until this happened to us.
A customer rang us up in a fury because some demo/ random data that we generated happened to have the word "penis" in it. They were convinced we must have put it there because we thought he was a cock. It was very difficult to defuse the situation.
I just recently saw a randomly generated ID of ours in production that starts with "doggy". Thankfully "doggy" is pretty innocent, but it really made me think "wow what if it was something bad". Unfortunate that that exact scenario seems to have happened to you already.
A guy at my school(in Poland) got in serious trouble because of his hoodie with the huge FCUK logo on it. It took actually showing the principal that this is a legit company[0] and not just a play on "FUCK".
I mean, it was still a play on the word, just being done by a legit company. Not that legit means mature.
I was doing a student event when I saw someone wear "K1SS MY 4RSE". I told him "What an obnoxious hoodie.". He meekly said "I thought it said 'Kiss my force'.". I later saw him in a corner praying. I should've asked him what Allah would've thought about him talking to him wearing that hoodie.
We had a similar situation with a random name generator that just picked first names and last names at random. One result of this was 'Gaylord Dickinson', which sounds like it could only possibly have been made up as a homophobic joke, but which was just the random combination of two quite common first and last names.
Recently on HN someone thought that reddit.com/imgur directing to a post on /r/Drugs meant something when it's just a randomly generated ID. All 5 letter word that I tried worked because there's been so many posts.
> I would have said "why bother" until this happened to us.
Aah, the good ole "one customer is unhappy, let's waste a week of time on this" approach to IT management. Takes guts to tell such customers "here's your refund, now piss off", but it is the right thing to do.
How statistically likely is this? Can't you just regenerate until no you have a suitable short URL. Aside from performance, this is as random as you can be. Or generate the characters one by one and backtrack, this requires less random days. Or regenerate the unwanted substring.
Our main concern was whether we needed to increase the size to 26 to account for the loss of keys. After doing the math, a 25 digit random string has a ~5% chance of containing one of 150 three or four character inappropriate substrings. That 5% loss isn't that big of a deal. But we had to figure out the math as part of due diligence before shipping.
Hashids (https://hashids.org/#how-does-it-work) have a pretty clever trick for this. They’re able to encode multiple IDs to a single obfuscated hash, which works by reserving some characters from the alphabet to use as a separator between each encoded value. That guarantees that whatever characters you choose to be separators are never next to each other in the output. By default their separators are (lower + upper case) “c, s, f, h, u, i, t”
In my help desk days I took a call from an irate person ranting about how we were telling her to "get a male sex change". Eventually I figured out she had become upset with "msexchange" showing up in the address!
Before Stack Overflow there was Experts Exchange. Their URL was of course those two words, all lowercase and mashed together into one... (Can't recall if they later inserted an underscore in between them?)
Your best bet atm is to just look through reddit/hn comments/posts people make as they find stuff. The leak's too big for one person/team to quickly find all spicy stuff.
Reminds me of the guy that streamed a talking banana on Twitch, where viewers could make it say things. People submitted variations of the n-word and got him banned, and after trying to filter out all character combinations he could think of he wrote a phonetic filter. That apparently worked much better than trying to think of every permutation of characters that sounds like bad words.
Ellis Island in New York harbour (AFAIK) used to be the main immigration center for people arriving by ship from Europe, and is famous (among other things, I assume) for having originated some weirdly spelled names when the immigration officials who registered the newly arrived got it wrong.
Don't know if the GP meant that they had soundex, or that they invented it, but in any case it seems they would have needed it.
Anything exposed to the open internet devolves into porn or racism or both unless active effort is made to prevent it. I'm reminded of https://en.wikipedia.org/wiki/Tay_(bot)
quite curious as to how Instagram managed to avoid this, despite launching with no moderation, (just Mike and Kevin), yet all you needed was an email to use the service...
They certainly have moderation now, and it's a constant exercise in boundary-pushing. I'd be interested in a "history of Instagram moderation", that would be a great piece of anthropology.
Was the no-links policy there from the start? I think that would have helped a lot. As it is, you're allowed basically one outbound link from your profile, so there are link-expander services which people use to link to more things.
The list included mike hawk (phonetically similar to 'my...'), so they are interested in phonetics, apparently, even for these usernames. The banana streamer has their stuff set up better than twitch, then.
But note: for the most part this isn't using regexs, and to the extent it does, it seems largely intended to make the maintainers' lives easier by avoiding having to represent (and maintain) all the permutations they are trying to match for.
What's sad though is that they're doing many, many passes through the pattern matcher, rather than just building a single big DFA from the whole list of patterns they want to match, which gets traversed in one pass.
Could Postgres convert the function into a DFA using the JIT optimizer (based on LLVM)? That might delve into sufficiently smart compiler territory but recognizing a bunch of OR'ed string matches seems on the easier end of optimization passes.
Yup, that's definitely sufficiently smart compiler territory. If you want to write an optimization pass that handles that, go for it, but you won't find one already there.
I feel like you could make an interesting game out of this. Given these rules, find the best "false negative," a realistic and inoffensive, but banned username. My best so far are "brownie_gurl" and "Megasthenes."
My wife used to name her RPG characters “Isis” after a cat we used to have. Used to have to explain to vets that we named her Isis years before Bush and Cheney created Isis by starting their illegitimate war in Iraq.
347 comments
[ 2.9 ms ] story [ 317 ms ] threadHighlights:
Now we know we can make as many goergefloyd accounts as we want! *devious grin* *chuckles to self*A hand coded if statement
Looks like Twitch doesn't like weed.
- should be in its own application with its own rules engine so you dont accidentally whack a bunch of userames
- I would have done in the past and cringe when people ask me to update it.
So yes, Amazon employees have ended up with this particular file on their desk.
It does remind me of the XKEYSCORE (Snowden leaks) that used keywords to bubble up potential threats from emails etc https://www.businessinsider.com/nsa-prism-keywords-for-domes... .
It used to be "if I search for this term, am I accidentally going to wind up getting goatse or something?" The good old days.
Now it's "if I search for this term, is the FBI going to kick my door in?"
you need the domain to make the goat-sex joke work
https://nl.wikipedia.org/wiki/Speciaal:Filter/10
That's what the linked page says (I'm not familiar with Dutch Wikipedia specifics), but that seems like such a strange statistic to track instead of minimum edit count.
So weird that I looked up how to configure MediaWiki autoconfirm requirements. They probably mean a minimum of one edit. Auto-confirmation considers only age and edit count, and I don't see why clicking 'Edit' is so meaningful a condition to warrant developing an extension.
https://www.mediawiki.org/wiki/Manual:Autoconfirmed_users
Searching about extensions did lead to discovering an obscure feature: there's now a built-in URL shortener: https://w.wiki/Q8
Edit: interestingly, the single-character ones seem to have been pre-planned: https://w.wiki/e, https://w.wiki/E, https://w.wiki/4
In the Dutch WP? Because otherwise I should qualify.
https://meta.wikimedia.org/wiki/Help:Unified_login
You can list your local accounts at
https://meta.wikimedia.org/wiki/Special:CentralAuth
The engineer would write something for every test case the product manager complained about, anything else computationally easy, and call it a day.
I once had to implement an audit logging system. What was supposed to be logged? "Important actions." Nobody on the team could define it. We just logged every database write along with the username responsible and called it a day. Nobody ever followed up or inspected it.
Same deal. Both exist mostly for compliance.
Tracking down "why is the antique system suddenly slow". Power went out, system came back up fine, everything but the one ancient but vital app is fine. Dig, dig dig, there's this old dot matrix printer in another room (because it used to be loud and annoying) that no on has fed or looked at in years.
It finally died with that outage, and it not accepting data was the problem. It had cheerfully printed the ribbon through, then fed out the rest of the box of paper it had, and that might've been several years before i saw it.
The roller the paper was supposed to ride had been eroded. The metal rods the print head rode on had a perceptible bump at the ends of the normal stroke.
The fix was a little dongle for the printer port that held the appropriate "i'm alive" lines up. hardware /dev/null. I'm thinking it was 25 pin rs232 because I remember a lot of cussing over it.
That kind of thing is probably fairly common in the industry.
I'm pretty sure it was the only dot matrix printer with a serial port i ever saw. Even daisy wheels were parallel port by the time this went in; but they had a like 50ft cable to move it to the other room. Someone worked hard and paid large to set that up originally.
It's not a list of words used by the NSA or any spies. https://attrition.org/misc/keywords.html
$Username=<username>
$Password=<password>
Connection.string=($Username, $Password)
The parser would flag this - Password was being stored to a variable! So we just changed our code:
$pw=<Password>. Problem solved!
https://www.youtube.com/watch?v=bJ5ppf0po3k
He made a filter that did analysis of the pronunciation of the messages that were being sent. His breakdown of it starts at around 13:30
I see the same here. It's not clever, but no one has any doubt what words are being checked.
This must be more nuanced. Maybe, it's "additional algo processing when a word is hit", eg another layer before "involve human".
Should optimize checks with a de-obfuscation function (attempt to expand non-AZ back to AZ, even if that then shoots permutations at banned word-runs).
It should probably also look more like a spam scoring system, where really obvious stuff is hard-trashed but borderline things are flagged for review / discussion.
I am also very disturbed that, as with most censorship, 'obviously bad' things such as terrorism/etc are co-mingled with 'is adult' as a negative check.
It seems reasonable for Twitch to have validated 'safe for minors' areas where names are filtered. Generic areas, where things are in the gray area and unchecked. Adult Only areas, where swears, profanity, maybe even some of the hateful things are allowed. Informed consumer choice.
Protected classes:
black, jewish, trans, gay, african, asian, minorities
Non-protected classes:
white
Typical Twitch (D-CA). All (D-CA) companies really. Under the law (in the USA), all races, and sexes are equally protected. On big tech platforms it's only one slice of demographics that harassment and hatred is tolerated toward, even encouraged. Do they actually think this is helping? Singling out one demographic for abuse? Do they think this is progress? That people won't notice? That it will never backfire?
This happens all the time in cartoons that appeal to children but also contain subtle adult jokes so everyone can enjoy them on different levels.
What worked in the end was having any newly created thread send a message containing the post title & body to a slack channel specifically for monitoring the forums. Employees and our forum moderators were in there, and any bad threads were nearly instantly deleted. Eventually the spammers mostly gave up. Hard to beat a dozen human brains :)
Having a dozen reviewers, especially if spread across timezones, would be a dream!
I have a hard time believing this silly mess is an actual component of anything.
c) "Porco dio saranno mica i testimoni di Geova? No eh diocan digli che i signori sono fuori, non ho tempo per stargli dietro." ("Fuck, they can't be Jehovah's witnesses, can they? Tell them we're out, we don't have time for their shit.")
The hardest problem with the implementation was that with a long list you can't just search for a few dozen inappropriate words (like the Twitch implementation). It would be very expensive to do hundreds or even thousands of checks against every inappropriate word.
The solution we came to was to truncate all the inappropriate words to either 3 or 4 letters and store them in a big set. We then take our generated strings, which are usually 11 characters, and break them up into all possible substrings of lengths 3 and 4. For example, 1a2b3c4d5e6 would be broken down into 1a2 a2b 2b3 b3c 3c4 c4d 4d5 5e6 1a2b a2b3 2b3c b3c4 3c4d c4d5 4d5e d5e6. An 11 character string would always have 16 such substrings. We then check all 16 against the banned set. 16 lookups into a set is pretty cheap and as we have expanded the word set over time (e.g. add a new language) our performance hasn't changed.
One drawback to our approach is that we do have false positives but we did the math and our space was still large enough, the cost of generating a new one was pretty low, and customers never see it so it's just not a big deal to throw out false positives.
Edit: I'm being downvoted so I want to explain - the internet is a huge mishmash of different cultures and all I wanted to say is that it is allowed to swear because I though that maybe, in their local one, it is not and they think it's universal
If the string is a url, imagine sending https://somesite/wanker to your client, when it actually could also be https://somesite/ay3ugd
It's random, I swear!
And this, ladies and gentlemen, is what it would show BEFORE the filter... but after (runs the code again, and prays it works) ... NO PROFANITY!
"Had we not done this work, that link would have been sent out to one of our users." was very well received.
It's a simple solution. Sure, it is still possible for something to slip through that looks similar to something bad. But the potential to strongly offend is greatly reduced.
Also note that if you're too naive about checking for 'naughty' words, you get https://en.wikipedia.org/wiki/Scunthorpe_problem
What is 'v' in this context?
Edit: thanks for the answers. It makes sense now.
the combo "cv" could then become problematic.
> algorithm tries to avoid generating most common English curse words by never placing the following letters (and their uppercase equivalents) next to each other:
> c, s, f, h, u, i, t
https://hashids.org/#how-does-it-work
E: ah it was already mentioned later on, hadn't got that deep into the comments yet!
Edit: But I'll concede that when your outputs are only four characters long and end users will actively interact with them (write them down, type them again later, etc.), additional safeguards might be appropriate. Or simply omit all alphas and use only numerics.
You're still not out of the park with numerics - people with 1313 or 6660 or 4444 or something will complain a lot. The possibility of a 666 in some new biometric government IDs in my country rose a massive stink from church...
Also, have a feelin you meant to do 1312. What’s the issue with 4444, though?
https://en.wikipedia.org/wiki/Tetraphobia
> When Beijing lost its bid to stage the 2000 Olympic Games, it was speculated that the reason China did not pursue a bid for the following 2004 Games was due to the unpopularity of the number 4 in China. Instead, the city waited another four years, and would eventually host the 2008 Olympic Games, the number eight being a lucky number in Chinese culture.
Thought this was particularly interesting.
> What’s the issue with 4444
4 is pronounced similar to "death" in sino-japanese languages and dialects.
Yeah. An important, long-lived ID that will stick with an individual for their entire life, and that they may want to commit to memory. That seems like a good time to take a hypersensitive approach and adopt some kind of filter.
That works pretty well until you realize that some numerical combinations are common neo-nazi codes and may lead to ... unfortunate associations. The ADL lists a few of those^1, but the list is by far not comprehensive, codes actually differ based on locality, and accidental combinatory collision in a 10-character space than it is in an alphanumerical 36-character space.
[1] https://www.adl.org/education/references/hate-symbols/88
I would have said "why bother" until this happened to us.
A customer rang us up in a fury because some demo/ random data that we generated happened to have the word "penis" in it. They were convinced we must have put it there because we thought he was a cock. It was very difficult to defuse the situation.
[0] https://en.wikipedia.org/wiki/French_Connection_(clothing)
I was doing a student event when I saw someone wear "K1SS MY 4RSE". I told him "What an obnoxious hoodie.". He meekly said "I thought it said 'Kiss my force'.". I later saw him in a corner praying. I should've asked him what Allah would've thought about him talking to him wearing that hoodie.
https://news.ycombinator.com/item?id=28676096
https://imgur.com/gallery/xXtEL
Aah, the good ole "one customer is unhappy, let's waste a week of time on this" approach to IT management. Takes guts to tell such customers "here's your refund, now piss off", but it is the right thing to do.
Our main concern was whether we needed to increase the size to 26 to account for the loss of keys. After doing the math, a 25 digit random string has a ~5% chance of containing one of 150 three or four character inappropriate substrings. That 5% loss isn't that big of a deal. But we had to figure out the math as part of due diligence before shipping.
It worked surprisingly well when we used it.
CREATE OR REPLACE FUNCTION is_hateful (VARCHAR) RETURNS BOOLEAN STABLE AS $$ SELECT ... OR replace($1,'_','') LIKE '%aggin%'
https://youtu.be/bJ5ppf0po3k?t=715
It's not that big really, half of it is the plinth to be honest.
Don't know if the GP meant that they had soundex, or that they invented it, but in any case it seems they would have needed it.
Of all the ways I expected a talking banana to backfire, I didn't expect this one. Thanks for sharing
Was the no-links policy there from the start? I think that would have helped a lot. As it is, you're allowed basically one outbound link from your profile, so there are link-expander services which people use to link to more things.
That sums up the WoW Classic (and I'm sure many other gaming communities) a little too perfectly.
Would regex be really much faster than checking it against a 1000 or more bad word list?
Also bad word list can easily get updated by moderators as well, I really can’t understand the logic behind using so much regex.
But note: for the most part this isn't using regexs, and to the extent it does, it seems largely intended to make the maintainers' lives easier by avoiding having to represent (and maintain) all the permutations they are trying to match for.
What's sad though is that they're doing many, many passes through the pattern matcher, rather than just building a single big DFA from the whole list of patterns they want to match, which gets traversed in one pass.
Meanwhile, I HOPE I'M NOT "POSTING TOO FAST," ASSHOLES. IT'S ONLY BEEN A COUPLE OF DAYS.
Looks like "Baggins" is banned.
Poor Bilbo...
Poor Mike.
The story we know was written by him. I wonder what the trolls would say, or the dead dragon, or the town that his actions helped destroy?
Is he a hero, or, just the guy who wrote it all down?
And just when something really important happens, he throws a powerful weapon(the one right) at his nephew and goes away to retire!
A life lead with riches (gold from the trolls), a ring granting him extremely long life and health, and yup.. off he goes, first sign of real trouble.
Poor Bilbo indeed!
I have some fun emails whose subject line is "here are your ISIS family log-in details"
and this was right around the time the terrorist group was frequently in the news
The Iraq War began in 2003.
Bush's administration ended in 2009.
ISIS gained power in 2014, 5 years into Obama's administration.