I highly doubt that. These reported posts might contain private data or highly shocking content; putting that on a public platform, especially associated with the name of Facebook, is very unlikely to happen.
It still would open up a gore/scare platform (think LiveLeak), since only the bad content is reported. Sure, most might be tame, but it wouldn't take long for gore seekers to come.
> Is that not what the person who originally posted the content being moderated is doing?
The person that posted the content does not expect a random person from the internet to see a video of his children because it was accidentally flagged by an AI as child porn. :)
>Can you crowdsource the moderation task to double check what the ai is flagging?
This would help with false positives, which is a noble goal. But it would not help with false negatives, which is imo a very disastrous consequence in this specific scenario.
Something that isn't CP got marked as CP? Sure, a moderator can manually doublecheck that flag and override if needed. Something that is CP didn't get marked as CP? You got a problem on your hands, because there is no way to manually doublecheck that, since that would mean manually checking every single post marked as "no issues here", which gets us back to square one of manual moderation of all posts.
Not even mentioning the exposure of FB users to CP that it would bring, which would mentally ruin quite a lot of people and bring very negative press to FB on the level of severity that we haven't seen before.
Speaking as someone who has to deal with supervised learning models, human beings falsely tag positive examples as negative all the time. Merely having "average" human beings tag examples doesn't guarantee perfect recall.
Or to restate, sites that rely purely on manual moderation of content that breaks rules still has content that breaks rules leak through all the time.
Nope. That's what the people in charge of our social media companies keep trying, over and over, and it doesn't work at all. AI-moderation is like the Ring is to Golem for them. They can't accept that "cheap and fair" moderation isn't currently possible.
> Can you crowdsource the moderation task
This is the clear answer, imo, and it isn't obvious to me why these C-level execs avoid it. If I had to make a guess, maybe they think engaging with millions of their users about moderation would open a can of worms.
When the goal is to prevent users from seeing this content, crowd-sourcing moderation defeats the purpose.
Users don't want the back-stop to be "a critical mass of community members object;" for content like this, they want to see zero of it, ever, and will choose to use another site that satisfies that hard constraint if FB cannot.
Many want that, but very few (probably nobody at all, by this point) expect it.
> When the goal is to prevent users from
> seeing this content, crowd-sourcing moderation
> defeats the purpose.
It's not actually all-or-nothing, and if it were, it would defeat it no less frequently than paid moderation already does. People tolerated toxic internet content before social media not because they never encountered any, but because they encountered much less.
No system can ever stop 100% of users from seeing 100% of the material they find objectionable. Every user has their own idea of what material is beyond the pale, and even if we invent AI that is competent at blocking content, and capable of tailoring its filters per-user, it will still fail some of the time, because users are people, and people's tastes change over time.
Aside from that, the way Youtube, Reddit, Twitter et al, currently operate, a user who reports objectionable content typically waits days, weeks, or years (or forever) for the company to take action. If you give a user a little real agency, it goes a long way to mitigate their displeasure over occasional exposure to unwanted content.
If I had to come up with a system, it would be some sort of reputation hierarchy, with a small number of paid employees at the top. Each level is responsible for auditing the levels under them.
> What real agency does the system you propose offer?
The ability for millions of users to moderate themselves. So... fast response time, by opinionated, emotionally-invested moderators, as opposed to the status quo: slow response time by paid burn-outs following a flow-chart.
> if your system requires users to view beheading videos regularly
I'm pretty sure that content would be reported faster than anything else. To be fair, another part of the issue is ban-evasion using multiple accounts, and that indeed requires additional measures to handle. Sadly, there's a disincentive to dealing with fake accounts, because trolls count as "sign-ups" too.
> people will just migrate to a site that doesn't require that.
Like Youtube and Facebook still occassionally do. Well, most users haven't abandoned them yet, so there goes that theory :)
I could swear there was also a Christchurch copy-cat attack in Europe, but I can't remember sufficient details about it to find an article. Perhaps that one wasn't streamed on FB.
I don't have encyclopedic knowledge of Facebook atrocities (I closed my account nearly a decade ago), but without vetting content before it goes live, I don't see how they will entirely prevent these videos from reaching some users.
I don't know that all is realistic. It's the desired unattainable goal. Almost all is the current status quo. If a service like FB were to adopt a volunteer moderator model, that track record would crash by definition because the moderators would be seeing the garbage.
(That's before we factor in unintended consequences such as the risk that a critical mass of moderators decide the garbage is signal and start passing it. It's more risk than FB wants to take on for a problem they already solve via paid employees).
I also made a comment before I saw yours about creating a machine learning based system to automate this process, I think it must be fairly easy given the amount of data already available.
They should recruit people on Reddit for this. A large number of them seem perfectly fine with this type of content.
There are dedicated communities to all these things.
I do not mean this as a hit against Reddit users. It is just that there are small pockets of people who are psychologically capable of tolerating this or even enjoy it. It seems more ethical to hire people from /r/watchpeopledie.
Then they should run sidebar ads on the gore sites. The point is that there appears to be a subset of the population that is not traumatized by this stuff. We may as well give them an opportunity to put that difference to use.
Edit: and saying things like "well clearly these people are already traumatized if they're seeking this out, it's not good for them, we shouldn't give these jobs to them" seems laughably paternalistic when the alternative is to traumatize more people by making them look at it instead.
^this. I am not arguing one way or another, because I don't see strong evidence making me lean predominantly one way. But imo, it is at least just as possible that they are seeking out that kind of content voluntarily because they are already traumatized in the first place, so this is a coping mechanism.
Maybe it helps dealing with that trauma. Maybe it makes the trauma worse. Maybe there was no trauma to begin with, and the person was seeking out that content simply due to morbid curiosity, but then it evolved into something else. Maybe there was no trauma to begin with and no trauma appeared after the exposure. Who knows.
But just because there are communities on reddit with people seeking that kind of material and not going on crazy violent sprees doesn't mean that it doesn't negatively affect them. It could be the case, but we should probably look into that first before jumping to "obvious solution" claims.
tl;dr: we gotta do more research and weigh pros and cons before simply declaring "we have an obvious solution, let's just hire all those gore content enthusiasts to moderate violent content, because they already do it on their own and don't seem to be going crazy in obvious ways due to it".
Meh, if that’s the worst PR they can get they’d be massively ahead of where they are now. Especially considering how massively the press narrative has turned against them overall.
At some point that stops being a factor. If the haters are going to hate no matter what, you actually get impunity because your actual actions stop making a difference either way.
There were pockets where the video of the Jordanian pilot being burned to death by ISIS was popular. Same with the raw video of the New Zealand shooting.
Most importantly, there's a big difference between seeing someone being tortured to death and frequently seeing people being tortured to death every weekday.
Are moderators really meant to watch the whole video in cases like that? It seems to me like they could stop once the content of the video became apparent. Even just a fraction of a second of that sort of content could be scaring, but I think probably less scaring than watching several minutes of it...
My basis for thinking this is I've seen videos like that before and clicked away quickly. It's disturbing, but I think I'm okay.
I'd wonder if frequents in these subreddits are typically professional and healthy people to have on a team, though (honestly no idea, never been).
It's a strange conundrum because ideally you want good people who are comfortable watching things good people rarely want to watch.
Don't have much experience with this demographic -- maybe there are more sane, emotionally healthy people watching snuff films for fun than I instinctively expect.
As long as they have a pulse and can sit in the chair long enough to do the job they'll likely be fine. Content moderation farms, call centers and the like tend to optimize their internal processes such that they don't need to restrict their hiring to "professional and healthy people". There are plenty of people who are insane and unhealthy that can hack this sort of work.
Do you want "insane and unhealthy" people who voluntarily look at horrifying stuff deciding whether that stuff should be allowed on the platform, though?
It'd be like having a jury pool made up of serial killers.
>Do you want "insane and unhealthy" people who voluntarily look at horrifying stuff deciding whether that stuff should be allowed on the platform, though?
These kinds of bottom of the barrel places of employment tend to have their policies and procedures tuned to make sure employees do things the way the company wants them done. That's just the nature of doing an unskilled, non-laborious job for a BigCo. There will be automatic spot checks, audits, reviews, etc. It is no different than the cube farm on the next floor where they review images of documents that AI systems have flagged as possibly fraudulent. Such systems are likely in place already for their current moderators.
> These kinds of bottom of the barrel places of employment tend to have their policies and procedures tuned to make sure employees do things the way the company wants them done.
Yes, but their hiring pool tends to be "everyone".
Those procedures might not work as well if your hiring pool is explicitly confined to "the people who watch ISIS torture videos for fun". I suspect standard policies and procedures include "fire the people who get off on this stuff".
This is usually a tempting fate question but what is the worst that could happen?
They have little power to abuse and obvious accountability. If they block kitten and baby pics and allow people being tortured to death they'll get fired quickly after the inevitable shitstorm.
Insane and unhealthy or creepy doesn't neccessarily mean dangerous. Even if they got twisted pleasure from it they would be gross but not inflicting actual harm. It isn't like people are being brutally murdered to give them kicks - they aren't a source of demand.
I remember talking to an FBI agent who had to watch a lot of terrible stuff. He said that at first it was very distressing and sickening, but after awhile he became desensitized to it and could eat lunch with no issues while continuing to watch it. I wonder if some people have the psychological profile that allows them to become desensitized while other people never can. Though, it probably isn't good that people have to become desensitized.
For me it was more like watching nature documentaries, morbid curiosity and all that. There wasn't really anything I'd consider snuff on that subreddit, it was natural death, murder, and accidents, all of which are part of natural life.
I found most of the videos less offensive than extremely graphic horror movies, which ironically I have less stomach for. If anything that subreddit made me generally more aware of my surroundings and safer in my daily life.
A snuff film is defined by most as at least having been made for the purpose of entertainment of others. Some include the proviso that it has to have been made for profit, and yet others demand that it has at least been sold. But regardless, not even the first level of these criteria have ever been fulfulled.
That article was last updated 2006 and does not take into account ISIS or Mexican drug cartel videos, which arguably meet your first definition and many would consider snuff.
One could hardly make the argument that those videos were made to entertain anybody for their enjoyment. Terrorists make videos to terrorize, i.e. scare, people, not for people to enjoy.
Making the argument that is at least partially made for the entertainment of other terrorists doesn't strike me as outlandish. Personally I wouldn't make that argument though.
Regardless my point was just that in my experience many would consider them 'snuff' even if it doesn't meet the exact technical definition.
> my point was just that in my experience many would consider them 'snuff' even if it doesn't meet the exact technical definition.
And those people would simply be wrong.
I mean, let’s back up a bit. Why is it important whether or not “snuff” films exist? Because many people want to believe the common urban legends about it – i.e. that there exists an (be it ever so small) underground production and market for it, even though, by all accounts, this does not exist, and has never existed. Believing such urban legends helps people feel justifiably horrified and helps validate their cynicism; basically the same reason why many people believe many other urban legends. However, no movies fitting that particular model have ever been found to exist.
If anyone wants to to be picky about the definition and claim that some movie or other fits the definition of “snuff”, or is “considered” to fit even though it doesn’t(!), this is only because they want to be able to say “see, snuff films exist!”, and thereby validate their preexisting belief in the urban legend. But this is a motte-and-bailey fallacy; it tries to defend the urban legend (which is hard) by trying to defend a technical definition of a word to be slightly wider than it really is (which is comparitively easier). Luckily, even though it is easier to defend, this latter position is still demonstrably wrong.
Those people weren’t killed for the purpose of entertainment. The initial videos were made by terrorists, to terrorize, not to entertain. Remixes aren’t “snuff” films, otherwise any compilation video on /r/watchpeopledie would qualify.
Thanks to Wikileaks publishing the Dutroux Dossier in 2008, we know hundreds of snuff films were recovered from the notorious pedo-rapist Marc Dutroux.
Just because the police keep those films as sealed evidence that are never published does NOT mean snuff films are an Urban Legend. I can see why the police always keep snuff films secret. Imagine what would happen if snuff films were uploaded all over the Internet? There would be mobs of angry villagers armed with torches and pitchforks descending on jails and prisons to lynch pedos. It would make harassment from QANONs look like school yard bullying.
Snuff films are, fortunately, still only a subject of urban legends, since those films you mention do not meet the definition. Were those films made in order to entertain other people than the person who made them? Were the films ever distributed to any of those people? Was anyone ever actually killed in any of these movies? If the answer to any one of those three questions is “no”, then the movies, horrible as they may be, were not “snuff” films, as commonly defined. From what I can see in your reference, the first two of those things are decidedly not true, and the reference is unclear regarding the third.
(The films were, according to your reference, made only for blackmail purposes, and therefore certainly not made for enjoyment of any viewer, nor were they ever distributed to other people for their enjoyment. The reference does claim that people were killed in a specific place which was also filmed, but does not, what I can see, explicitly state that any murders were actually filmed, other than incorrectly calling all the films “snuff” films.)
I think that some of you should take a look at news websites about the drug and kidnapping cartels down in Latin America... They regularly post their own snuff films online and I can guarantee you that they're not only very real but also extraordinarily brutal. No urban legend about it.
> It is just that there are small pockets of people who are psychologically capable of tolerating this or even enjoy it.
I wouldn't assume "watches this stuff" and "is psychologically capable of tolerating this stuff" are one and the same. Those people might be actively harming themselves by watching the content. That's one thing when they're volunteering to do so (though that's an interesting conversation in and of itself) but it's quite another when Facebook is paying them to do it.
Honestly, this post speaks a lot of truth, at least in my experience. I use to be subbed to /r/watchpeopledie, really just out of morbid curiosity.
One day I had an anxiety attack because I had a dream that I had a heart attack and suddenly I had a deep fear of randomly dying.
All the random deaths on that sub started to really get to me, I even started to get depressed. It became harder and harder to watch content on it. Eventually I unsubbed from it and gradually bounced back to normal.
I don't really regret it because it gave me a new perspective on life and how valuable it is. But I would be lying if I said it didn't fuck me up in a way.
I think the best counter to becoming desensitized or depressed is having an over arching sense of mission and identifying your work with a cause for good that is greater than yourself.
take ER doctors. they daily witness human trauma far worse than viewing horror show videos and images. do you ever hear about ER docs who become depressed or mentally ill from being exposed to so much horrific human suffering? Doctors have a personal sense of doing good in the world through healing. They witness so much trauma that they become so desensitized that non doctors often criticize them for seeming so detached and for treating human bodies like mechanistic processes. But doctors have to be somewhat detached in order to do their jobs well.
It's not specifically about ER doctors. Anecdotally, I'm a lawyer, and they warned us in law school that lawyers have abnormally high rates of substance abuse and depression, and constant jokes to the contrary not withstanding, most lawyers believe they do good. Emotionally hard jobs are hard, even if they are indisputably good.
> do you ever hear about ER docs who become depressed or mentally ill from being exposed to so much horrific human suffering?
Yes you do! But I would imagine it is at least somewhat offset by the fact that they are doing a lot of good: they are saving people from death while also witnessing death. I think a content moderator would struggle to feel such a strong sense of purpose.
These niche demographics are typically into it for the wrong reasons though, which includes fantasizing about the material (murder, rape, child pornography, etc.).
But are people like that people you want as your co-workers?
Serious question, I don't mean this facetiously. I think is an extremely complex question and I wonder if introducing someone who is openly comfortable with gore is dangerous to the psychological health of the company.
For example, at a previous company a co-worker publicly shared, without a trigger warning and in great detail, a very gorey thing he enjoyed watching to relax. Lots of people were extremely disturbed by this -- not disturbed by him, per se, but disturbed that he shared this without any kind of warning. I don't even resent him despite myself being pretty disturbed because what one considers normal is subjective, and if you regularly relax to this content you probably don't realise that this might not be anyone else's cup of tea. But it really makes me wonder -- what if the guy who sits next to me starts telling me that ISIS beheadings are relaxing to him?
Well, I probably wouldn't invite him to my game night, but if it's his job to watch ISIS videos, it seems like he'd be well-suited, no?
It seems like if you acknowledge that:
a) Terrible job exists
b) Terrible job is necessary
You should also allow for a person to do terrible job without thinking that they must be a terrible person by extension. Otherwise you continue the well-trodden history of certain professions like hide-makers, executioners, or coroners being ostracized from society for doing the job we told them to do.
I'm sorry if I implied that I think that people who would enjoy this work are bad people. I certainly don't feel this way; I mean, I myself have worked on a platform that plenty of Nazis use and i don't consider myself a Nazi, so it would be hypocritical of me to say as such.
I agree that I wouldn't personally spend time with said person, but my question was less about personal social responsibilities outside of work. My example I pointed out happened at work, during normal work hours, in a social space where people have inherently less control over who they surround themselves with. This is the part of it all that is fuzzy to me, and I don't have a clear answer for.
Certainly in our own, private social spaces we are free to choose whomever we socialize with, but at work that option is less available. The example I gave happened during work hours over work communication, so there is always going to be social overlap that you can't simply opt out of (the age old 'what is culture fit' question applies here, and every co-worker contributes to company culture in some way).
And while a lot of content moderators do focus purely on clicking thru things, a lot of other ones still regularly interface with the company. I used to work at Discord, where content moderation of this level (and traumatic nature) was a constant concern. As engineers you may be staffed to work on a tool to help with moderation. Or you may be working on some fancy AI to help sort and tag unsafe content. And this is only from the product engineering perspective; content moderators will have to work with customer success to create content policy, or enforce bans for unsavory behavior. So I don't think they really are siloable away from the rest of the company, and I'm not even certain that that is a humane way to treat them.
There are not hordes of people on Reddit watching intensely disturbing child pornography. The gore stuff maybe (although r/wpd was banned), but still I think any normal person, even the person who was occasionally on r/wpd, would be extremely traumatized by watching anything involving children. Which to be honest is probably a lot of the content discussed here, as sick as it is.
I spent my youth on 4chan. I saw everything that's mentioned in the article and worse.
Desensitization is certainly a thing. The first few times seeing those things were shocking and disturbing. Then it got exciting. Then it got boring. I don't actively look for that content, but never having left 4chan, I still sometimes stumble upon it and it causes no big reaction. I think I could totally do that content moderation job, I have to admit some morbid curiosity about what I would stumble upon exists in me. The only content that never stopped to disturb me was violence against animals.
Talking with my "normie" friends about this I realized I am the odd one out. I wonder just how much consuming that content in my youth changed me.
I can't help but wonder what the numbers look like on this content. One of the things Facebook should have enough data to know is how prevalent this sort of thing is.
Is deeply offensive content generated by a handful of users frequently, or many users less often? What's the volume look like?
Agreed. I would definitely love to see some sort of a basic data analysis blog post on this topic from FB, similar to how match.com used to publish data analysis posts on their blog back in the day.
I don't understand what's done with the contractors in the US after the successful lawsuit, surely all those jobs were just moved offshore? There's no mention of this at all in the article....
I'm not sure if you are being sarcastic so forgive me if this is off-base, but there are REAL people who are dealing with trauma from moderating content so you don't need to "test it out" and validate.
The issue is as real as western countries dumping garbage in eastern countries just because they can pay them off.
The local governments won't acknowledge this as problem because of the money flowing in and a lot of people's livelihood is dependent on this. But we should call it what it is, exploitation.
No, western countries are NOT dumping garbage. They are buying the service "waste disposal", they are NOT responsible for the supplier of that service just dumping the garage.
I think in regards to India, the issue is a little more complex, and intertwined with social mores. This one sentence from the article is really important:
> According to Dr. K. Jyothirmayi, a Hyderabad-based psychiatrist, stigmas, such as the perceived impact of a mental health diagnosis on one’s marital prospects, often prevent young Indians from seeking treatment.
From growing up in India, my view is that meditation and the general state of the mind has been taken seriously for, well, centuries. But move the focus towards anything with 'psych' or 'mental' in the title and people in India will shy away from it even if it is affordable or free. "Seeing a mental health counselor" is a recipe for considerable social stigma in India [1], not an empowering practice like it's perceived as in the United States.
The solution to this is hard, and in my opinion not something that Facebook or their Indian subcontractors can accomplish.
-----------
[1] I'm talking "All your relatives will refer to you as the crazy person" or "Your girlfriend's parents will call off the engagement" levels of stigma, not just "I don't feel comfortable talking about it in a bar" levels. Like a lot of other things, families with higher levels of income and education are sometimes an exception.
The awkward ethical question is would they be better off without "exploitation"? Part of the reason companies are even willing to go through the overhead is because the costs are lower.
Real life has always been filled with messed up situations where the punchline is "... and the best/worst part is it qualified as an improvement!".
I recall a statistic from about 10 years ago that computer forensic investigators in law enforcement burn out after two years due to the trauma of the images they are exposed to.
There's likely some confounding factors. Pressing "go" on the overpriced software tools and then entering into evidence what you find is the lowest level of work in that field so the churn is going to naturally be very high as people move up or out. The pay also isn't that great.
No, that is not the issue. Rather the issue is that even the hardest stuff on Facebook isn't remotely comparable to stuff of actual criminals, and the effort is wildly different:
- Facebook: it's violating rules? Delete, next.
- Forensic IT on a multi TB disk full with child porn: document every photo, what it shows, extract identifiable faces to cross reference with other content (to check for recurring places and victims), and the process is even more gory for video content. You have to watch every second or the defense can attempt "you didn't watch the video in full where the perp gives the victim an ice cream at the end" or whatever else. The amount of time you spend with documenting a single photo or video is many orders of magnitude worse than FB content mods.
Radical idea: if your service requires subjecting lots and lots of workers to scarring images, the correct solution is for that service not to exist.
Facebook doesn’t have to let people post material to their site with very low barriers to entry. Granted they have to if they want to continue to be what they are and to make tons of money... but maybe they shouldn’t continue to be what they are and make tons of money if this kind of abuse is necessarily coupled to those outcomes.
“Well this sucks but we can’t keep operating if we don’t do it”. Well... sure but the solution is right there in that statement. Don’t keep operating.
1) that seems basically fine, but also 2) non-commercial efforts run by hobbyists and gated from the general public ought to be Ok. If you want to run a PHPBB site and subject yourself to harmful garbage by letting randos write to your server, well, go nuts.
[edit] thought experiment: how many of the people making tons of money off Facebook—c-suite, major shareholders—would find some other way to make money if continuing to make Facebook mega bucks meant they had to do this 5 days a week? What would we think of any of them who chose “bring on the trauma, I want those sweet greenbacks” and kept it up for years?
I think there’s a difference that depends on scale and privacy. If your network is private (or something more akin to p2p / messaging) this is much less of an issue, especially if you have strong blocking controls. Users moderate themselves.
I think the problems that Facebook, Youtube, etc are seeing stem from the fact that the content is much more public, thus in much more need of moderation.
It's probably a bit controversial to say this, but haven't large sites had to deal with these problems for years by the time the likes of Facebook/Twitter/YouTube/Instagram/whatever came around?
It's obviously horrible doing this as a job and it's obviously a bit more common on Facebook than say, a large old school internet forum, but... at least in this case people are paid to do this. Reddit mods and old school forum mods have to deal with this stuff for free.
Right but the difference is the sheer amount of money involved and the politicization that comes with that. If Facebook was worth 100m nobody would care. If facebook only has 10m users nobody would care. It's the scale that completely breaks the legal and political frameworks the our society leans on.
I'd say another difference is the percentage of your attention is being devoted to this kind of thing. Presumably reddit or phpbb mods aren't spending 40 hr/wk looking at gore.
Wouldn't that suggest then that the "ideal" way to do it is a very part time job then with high volume? Like say 5 hours a week. It would be a scheduling and logistical nightmare of course but it would give more "dispersion time" for trauma.
I'd be interested to see if the tooling around moderation could also be improved. For example, could the images start heavily blurred, with a circle of clarity that opens when you click. Or something like that, which would harm throughput, but be more humane.
tbh I've been a moderator on various old school forums for over a decade, and I've never had to deal with anyone posting any gore at all. I think they're usually too tight knit and reputation based for anyone to feel like they could get away with it. it's not like people posting gore on facebook are doing it to mess with the moderators, that's just a side effect
From my time observing volunteer groups, it's also about leverage and force multipliers.
I can go door to door asking people to change something, and dozens of my volunteers will feel useful but overwhelmed by how much work there is to do in a city of a million people. Or I can go to city council meetings, and get more leverage there. Or if I can convince one high profile citizen to participate, others will follow. Or I can get into contact with the mayor, state congressional members, the governor, federal congressional members.
It's possible that if I convince the correct five people that they will take half of this burden off of my hands. You'd be crazy not to reach for a lever like that. They are a juicy target, and not just from a punitive standpoint. Of course when you are rebuffed repeatedly by a potential ally, then it's a very human (if questionable) response to seek out some sort of punishment.
Do you know what community organizing is? How movements accomplish change? Everything does not have some easy optimization method, only in silicon valley does everyone think it does.
edit: What I mean to say is, people have been organizing against FB, AMZN, and tech in general for the past few years now. The last is reactionary but opposition to FB and AMZN has meaningful academia, ngo/gov, and general support.
I worked with a group that got 100 volunteers at each event who did the work while the regular members provided coordination and resources. I am trying to do something similar on my own now and learning in a very visceral way just how important spreading that load out is.
But... that group wouldn't be able to do nearly what it has if it weren't for funding from foundations and the department of neighborhoods, and resources on permanent loan from another public sector organization. You might argue it couldn't exist at all.
I think I would say that both are necessary but insufficient.
I think the point is that, if I as an activist am aiming to maximize the number of content moderators treated well, the highest leverage way to push for that is getting people mad at Facebook. There may be a preexisting system of unpaid content moderators who are supported even less, but that's much harder for anyone to address.
Another factor is the working conditions at Cognizant. The difference between Reddit moderators and Cognizant employees is that the former are allowed to have bathroom breaks and individual agency. The latter not only have to deal with fairly oppressive workplace policies, but they also rely on this work to live.
I think it's fair to say the sheer quantity has changed since Facebook first came around (FB predates smartphones for one!)
I also think part of the objection here is around off-shoring. It's similar to other complaints about off-shored jobs: we're farming out unpleasant work to people that get paid less and have much less support (e.g. mental health).
> I also think part of the objection here is around off-shoring. It's similar to other complaints about off-shored jobs: we're farming out unpleasant work to people that get paid less and have much less support (e.g. mental health).
That's an interesting point, and has me wondering something.
If a site like Facebook opened up a volunteer moderator scheme where people could sign up to filter out stuff that broke their rules, would it make things better or worse on a moral level?
What about a US/Europe based agency that did this thing onshore as a contract deal?
As for the volunteer side, it's a really interesting debate. Others in this thread have pointed out Reddit as an example of this in practise. I think the sheer amount of money that Facebook has means that relying on volunteers to (potentially) inflict emotional damage upon themselves wouldn't be a great look.
I'd consider volunteering to moderate a particular geographical area, when I was a student Facebook constantly took down the student confessions page I eventually ended up helping to run for the vaguest reasons, while letting actual hate posters run amok. I know I would say that, but we would genuinely get taken down for jokes that wouldn't even make sense out of the tiny university I attended. It was immensely frustrating having to build up the userbase again and again because the moderators of Facebook just weren't/aren't terribly good at their jobs. I'd happily have taken on actual moderation duties rather than had to constantly appeal and get a blank, inhuman "declined" message.
I think they tend to take an "if in doubt, remove it" approach which basically allows anyone who doesn't like a niche page for whatever reason to have it taken down through sheer attrition, if you report it enough it'll probably go down eventually even for stupid reasons. I actually ended up writing an app to cut Facebook out of our system altogether (it was kind of similar to Yik Yak with a much better moderation system and better anonymity protection - I really can't be arsed to deal with a GDPR lawsuit...) but we couldn't launch it in part because it took me ages to develop as I've since graduated and work as a full-time developer and because coronavirus meant our launch event got cancelled. I might take it live when students go back this year, but I'm concerned that with nothing much going on it'll fizzle and die because there's nothing interesting to post about. It'll all depend on how the Welsh government lift the coronavirus restrictions.
YouTube tried doing something like this with YouTube Heroes and got tons of backlash for it. Then YouTube tried automating everything away and I'm guessing now people wish they had implemented something like YouTube Heroes instead.
For a different reason. The community was generally upset about Heroes for the potential of large groups to mass flag videos they disagreed with politically, something that was already a growing problem at the time.
Disclosure: I work for but don't represent Google. This is my own opinion.
I suspect there is a certain interest group that has emerged and grown around this subject that has been elevating and promoting this issue in the public's mind. It would not even surprise me one bit that especially overseas contractors, because there are likely fewer penalties/prohibitions, are paying people to flood, e.g., fb with things like beheading videos in order to produce a cause for their services. It may be cynical, but like I said, it would not surprise me one bit.
Very few forums had even a fraction of the number of members as Facebook has users and the amount of content is increased again. Also the type of content they're required to moderate has changed most forums before facebook were largely text and video is a big part of facebook. It's easy to see how video and photo content is more traumatizing than the largely text content of forums.
>at least in this case people are paid to do this.
I've done commercial moderation before and I would sooner go jobless than go back to exposing myself to the kinds of things people have to when doing this professionally. It's the kind of things you wish there was a brain cleaning solution for, but there isn't so it's stuck there basically forever (though I suspect the alcoholism rate is higher among people who have to handle this kind of thing than in the general population).
The commercial moderation team I was on actually ended up separating out a task force for handling particularly gnarly content so that people had a way of opting out of being exposed to it (in our case we were responding to user reports rather than randomly stumbling on things, so manually filtering reports was feasible), which I think is the responsible approach since there are plenty of other moderation tasks to handle and some people are more or less sensitive to this content.
'Drug-induced amnesia' to reduce long term memory generation is a thing and is sometimes used before traumatic surgical procedures. It needs to be taken in advance and is probably not very healthy. Plus I would have serious ethical concerns with jobs that would encourage that.
I've had procedures where those drugs are used. They're benzodiazepines, and thus powerfully sedating. You are in no sense able to work while doped to the eyeballs with Versed.
This seems like something specific to the individual. Maybe this isn't the job for you, but there's probably enough people "made of different stuff" that could handle it? I feel like there's a subtext in this discussion that "nobody can handle it" or "you have to be a monster to work this job" but that's just not true.
Just out of college in the mid-90s, I worked for a few months at a small local software company in the last throws of death. As a last grasp at survival, all the programmers switched to doing cold call sales. With phones.
I would stare at mutilated children all day long rather than go back to cold calling. I was miserable and I hated every moment of every day until the company (happily, for me) closed up. Yet I know people who somehow think it's perfectly normal to dial random numbers and try to pitch them something. They're made of different stuff.
I've talked to people who make the same claim about computer programming. They'd rather starve.
Facebook has 10,000 moderators. I'm pretty sure there are at least 10k people in the world who are psychologically fit for content policing. It's doesn't have to be for everyone.
It doesn't seem to be specific to an individual when there are class actions lawsuits and many people reporting to suffer from it.
There exist jobs, like slaughterhouse knockers, which are linked to causing PTSD and increased criminal behavior. It's not far fetched to believe these moderator jobs do too.
Yeah, and they made an interesting point about how these problems are compounded by the lack of mental health awareness and the difficulties of defending mental health trauma in a court of law in countries like India and the United States. I didn't think of that before.
Before I accept this is a serious problem, I'd need to see some statistics. The article is just a bunch of anecdotes. If this job is hazardous, quantify it?
It wasn't that long ago HN was abuzz about large numbers of suicides at Foxconn... until someone realized that the suicide rate was actually slightly below the community average. I simply don't trust arguments without statistics anymore.
I get what you're saying, but you would have to look for traits and develop processes to make an environment that people could do well in.
Big companies like Facebook (although they are not alone) don't do that. That costs money. They use disposable workers, like on-shore or off-shore contractors, and do industrial-style quotas/piecework methods to maximize productivity.
When you hire like that, you'll have a normal distribution of people. On the left there will be pre-broken people who get pleasure from watching decapitation videos and nasty shit, on the right you'll have people who are traumatized and broken by the work, and in the middle you have people who will cope for some period of time, and burn out.
Is this not the same type of thinking that’s convinced people who do all sorts of dirty but necessary jobs are “real sickos”? Jobs like homicide/sex crime investigation, fixing the sewers, exterminating pests, and so on. I would probably get PTSD if I had to investigate bleeding anuses digitally but my doctor seems to manage and I doubt she’s a sociopath.
My point is, some people simply aren’t traumatized by photos/videos, either via talent or training, and because of this they can process it enough to categorize it and keep their emotional/social/intellectual distance. That doesn’t make them sociopaths and it certainly doesn’t make them bad people.
You seem to be confusing any exposure with constant exposure, and also type of gross/messed up. Nothing you listed compares to content moderation in severity and volume.
Fixing a sewer is gross. Extermination is gross, not traumatic, unless you have extreme empathy for small rodents and insects. Doctors deal with very few remotely bleeding anuses on the order of one a week/month maybe versus one a minute. None of these should be called real sickos, but none would be expected to develop PTSD either.
A significantly bigger portion of the population would be traumatized and develop PTSD by the level of exposure of content moderation. While not nearly all are sociopaths, they 100% will be represented higher even without factoring in selection bias. Hiring a department of them would not be inconceivable, putting the validity of that aside.
You also make the jump from sociopath to bad person, which many I'm sure would have some qualms with as a categorical.
PTSD is real but lots of people still go to war and not everyone comes back a basketcase. Most don't, in fact.
No, I've never had a job watching pictures of mutilated children all day. But like a lot of naturally curious people, I've searched out and watched bloody videos of beheadings and car accidents and whatnot. It wasn't pleasant, but it wasn't really traumatic either. If I didn't already have a much more lucrative career, I don't think I'd suffer too much as a Facebook censor.
Does that make me a sociopath? My wife and son don't think so, nor do my close friends. I'm not sure if I should feel a little offended here.
On the other hand, I may still have PTSD from those days of cold calling as a junior software engineer 25 years ago. I hate phones. Maybe people are just wired differently.
MySpace had to deal with this. They also had content moderators who were kept in a separate room at HQ who had to view all manner of flagged videos/photos. The turnover was really high even though at the time, MySpace had the kind of cool factor that developers would pick up women at bars with the promise of taking them into the offices after hours (which is a whole other weirdness and the corporate higher ups had to ban the practice after a drunk couple were injured after falling off a table one Saturday night). I don't know that there was anything acting at this kind of scale before MySpace.
Facebook was a zoo in its early years. Basically a frat house. Google was also a complete boys club and full of misconduct. But you have to be a complete nerd to take your date to your office, even if it’s a nice office.
It really wasn't a nice office. My wife worked there from 2005–2008 and I'd occasionally pick her up after work. I think the big thing was just that it was MySpace and at the time, it was the epitome of cool. They even had a record labelhttps://en.wikipedia.org/wiki/MySpace_Records Unlike Facebook, MySpace was heavily tied into the entertainment industry, especially music, but to a lesser extent film and stand-up comedy which probably contributed to the cool factor. The engineering culture was pretty heavily bro-grammer in a way I've never seen anywhere else.
I have moderated several forums over the course of 15+ years, and I have never come across this kind of content. But then again, all of them were very specialized in their topics. I still am convinced that we would be better off if we went back to small forums for people sharing common interests rather than having a huge platform. People can police their small group of peers much more easily. It works.
Communities grow - I think this kind of strategy only works on a small scale. If it was adopted en masse, it becomes self defeating. Suggesting everyone change groups the moment their current group reaches a certain size ignores the greater point, which is that these large groups are going to exist/emerge regardless.
It's probably a bit controversial to say this, but haven't large sites had to deal with these problems for years by the time the likes of Facebook/Twitter/YouTube/Instagram/whatever came around?
I don't think at the same level. Social media usage went from 10% of the population to 79% from 2005 to 2015 and was even lower a few years before this. So current social media companies operate at unprecedented scales. Moreover, the smaller percentage of people usage social media in earlier times tended to be "sophisticated users" who didn't necessarily expect their feed to be systematically filtered.
Depends on the language. In my experience Vietnamese -> English is horrid and comes out as mostly garbage. Interestingly, the reverse has been better from what my Vietnamese friends tell me.
Post gore, rape and torture in places where people don't want to see that and you should lose the right to have internet access. Criminalize this behavior.
The US has laws with punishments that make it illegal for people to use computers or to use certain online services already. Just do more of this, but broaden the categories to include posting visually horrifying content that's known to cause PTSD to places where people don't expect to see that.
"Just do more of this, but broaden the categories to include posting…" and here you are already thrusting yourself down the slippery slope. You went from "gore, rape, and torture", all of which inherently not easy to prove in and of themselves, but in just a matter of minutes you've already "broadened the categories" to "visually horrifying" (based on who's opinion?) and which is "known to cause PTSD" (known and determined by whom? Not even to mention that PTSD is a professional construct), but also including having to know where people do and do not expect to see something that is something based on the above mentioned ambiguous determinations.
It did not take you long at all to just vault yourself down the full on dictator slope. Next up, let me guess, you want to prevent people's access to the internet if they say things you don't agree with because "word are violence" and then "micro-aggressions are violence", simply because you have the control to abuse. Unfortunately that is already the situation right now, but it does not need to be made worse, because it will only thrust us all down a path of overplaying the unrelenting abuse that I can assure you people will not like the inevitable consequences of one bit.
Let me also remind you that the US has the supreme law of the land that supersedes all other laws, the First Amendment to the Constitution that states that no law may abridge the freedom of speech, regardless of whether you like it or not.
Yes. A huge fraction of modern culture takes place online, through Instagram or Youtube or Facebook or Twitter. I think most people would rather go bankrupt than be barred from all Internet access.
A lot of people will disagree with you on principle (they view internet access as a "right" rather than a privilege), but I think it's an interesting idea that's worth discussing, if only because we've had similar ideas in the past, with other technologies, and arrived upon similar conclusions.
We as a society have decided that negligent actors can do enough harm to themselves or others in operating an automobile on publicly-owned roads, that we compel individuals to seek license from the government to drive vehicles on those roads. The license is a revocable priveilege that has to be earned by ongoing demonstrations of competency.
One could imagine a parallel system for "web privileges", although it sounds dystopian at first blush, in contrast to the unprecedented freedoms we enjoy on the web, today.
The tricky thing is that it's relatively hard to precisely track identity and intentional authorship on the internet. It's also easy to imagine somebody losing their internet privileges, because some botnet hijacked their printer and used it to spread X-rated content. In the analogy, it's hard to disambiguate between drivers and vehicles.
Maybe we'd require some biometric login system, provided by the government, to use a web browser. It's hard to imagine implementing any such system in a way that doesn't make the attack surface bigger, but harder problems have been solved.
I can't imagine how "commercial web licenses" would work, though. That's an interesting space for thought.
> The tricky thing is that it's relatively hard to precisely track identity and intentional authorship on the internet. It's also easy to imagine somebody losing their internet privileges, because some botnet hijacked their printer and used it to spread X-rated content. In the analogy, it's hard to disambiguate between drivers and vehicles.
This basically makes the entire plan unfeasible. Given the current general insecurity it would be entirely too trivial to somehow fuck up and have a botnet get you banned permanently from the "roads".
However, this could work if internet is only available using your digital ID. However, that is seriously actually dystopian and would brutally murder all of the privacy people have and in certain cases also the people.
Yep, that's why I suggested the biometric ID. But I agree, it's dystopian.
I think such a system would encourage the development of two parallel webs: the above-ground, requires-ID-for-every-http-request web (Facebook, Amazon), and a somewhat shadier, sometimes-or-never-requires-ID-except-in-exceptional-circumstances internet.
Kind of like how bars and grocery stores are supposed to verify your age, but many don't. Or kind of like how there's large swathes of the country where people drive around without driver's licenses or insurance, and it isn't an issue unless they end up in an accident.
The "shadier" internet will be slowly killed. Without any doubt. It would just be more convenient for website hosts, to people in power in general.
Especially so under oppressive regimes, I think China has already displayed how internet would end up if we went that route. Restrictive, dangerous for dissidents, without any real privacy.
> ... digital ID. However, that is seriously actually dystopian and would brutally murder all of the privacy people have
Do you really need exactly the same level of privacy for all the cases though? How about having a corner of the internet which is an ultimate walled garden - with access restricted to someone with validated government IDs and everyone's activity tracked and transparent to everyone else? This will work great for many cases - kids can have access to only this corner, plain vanilla entertainment[1] like Netflix or Disney or Sports, money related stuff[2] like banking / money transfer / shopping, etc etc. We can do those activities with relative peace of mind because everyone accessing this corner would be physically trackable and hence nefarious actors will have the risk of instant exposure. In that sense, it will be very similar to the physical world (or even stricter because of enhanced security).
For a lot of other legit reasons like fighting oppressive governments or hacking things to push boundaries or uncensored forums, we can of course have rest of the internet.
There will be some nuances just like we do in real world, like:
[1] different ratings for entertainment, like R-rated vs PG vs G-rated
[2] protecting sensitive info like bank account details or tax returns
Do you think that the free internet won't be slowly strangled until there's none? People will find so many reasons to kill privacy, "think of the kids", "think of the crime", "think of the spies". So on and on, we've seen it happen, I think China and other oppressive regimes have displayed that already.
> Do you think that the free internet won't be slowly strangled until there's none?
In places like China, maybe! In places like the US, definitely not. After all, US Supreme Court literally allowed neo-nazis to march in public defending their freedom of speech[1].
> People will find so many reasons to kill privacy
They might, but they will always have to play catch up with the technical solutions (cryptocurrencies, strong encryption) etc. As long the fundamental protocol remains open, you can recreate your own stack.
> A lot of people will disagree with you on principle (they view internet access as a "right" rather than a privilege)
Well, sure. But that's why the concept of "criminalizing" is key. Criminal behavior curtails your rights. It's on this premise that the concept of prisons is legal and makes sense.
IMO rather than forbid internet access, an idea to explore is treating the internet as some sort of public area with similar jurisdiction. Thus, if you send an unsolicited dick pic it'd qualify as indecent exposure which is punishable by law.
I think this makes more sense because regulating internet access is hard. You can get public wifi pretty much at any airport or coffee-shop. What doesn't quite make sense is that showing your dick to someone without them wanting to see it can get you a fine or even jail time if you do it in person, but in a photo, video or call online is not punishable.
> Post gore, rape and torture in places where people don't want to see that and you should lose the right to have internet access. Criminalize this behavior.
It's probably illegal just about everywhere to post pictures of rape and torture.
You may want to reword your comment, unless you meant for it to imply that it should be legal to post gore, rape, and torture for people who want to view those.
Edge cases are hard. If you record law enforcement acting inappropriately, can you post that? Will you be less likely to do so if you feel you may be banned for doing so?
> Post gore, rape and torture in places where people don't want to see that and you should lose the right to have internet access. Criminalize this behavior.
May I ask you how old are you? I don't mean to insult you or anything like that but lately I am witnessing more and more people who demand the ultimate punishment for every misbehaviour and I wonder of it is a generation thing.
I'm surprised there hasn't been a revenge movie where one of these moderators use their access to find out where these people live and start hunting them down like some sort of batman-like person. I could see a movie like this being a means of making the more general population aware of the terrible stuff content moderators have to watch.
This is totally not what you are talking about, but the closest thing I have seen to that is the Netflix docuseries Don't Fuck With Cats. It hits on a lot of the same topics. The primary differences are that the hunters are Facebook users and not moderators and they are hunting the person to get justice and not violent revenge.
I can't even barely read this article before becoming extremely sad for these people and the individuals (especially children) who were abused in these videos. I hope everyone involve gets the healing they need.
It's not the flat-earther and somesuch conspirary communities that worry me. It's the people who share and get enjoyment from this kind of content, and the networks which enable it.
In my opinion this is an argument in favour of shutting Facebook (and every other large social network) down. This sort of work is damaging and abusive, and nobody should have to endure it just so we can have social networks.
I understand there are some jobs in the world that need to deal with dark stuff (like law enforcement), but social networks just aren’t worth the human cost.
What you’re proposing is not just shutting down social networks, it’s shutting down any website that involves user content, anything that allows photo/video upload, comments, or any kind of user interaction. That’s impossible.
You point out that public safety jobs are view more “worth it,” and certainly they are, but that logic brings up the question of who judges what job is worth undergoing trauma.
In other words, is a subway or freight train driver’s job “worth it,” if they have to see someone commit suicide on the tracks? What about crime scene cleanup companies? Funeral services? Bus drivers? Truck drivers? Nobody’s going to agree on where to draw the line in the sand.
A more realistic solution might be to make comprehensive support systems, mental health resources, and treatment a legally mandated, completely free service provided to any employee that works in these kinds of fields.
Finally, I think there are most certainly people out there who are not as sensitive and affected by this content who would be candidates for these kinds of roles. Perhaps there’s a way to test for that sensitivity before the real job starts.
I’m actually not proposing shutting down any website that allows uploaded content, just large / public sites that require this sort of moderation. Not every site gets this stuff uploaded to it. The more private the network, the less need for this kind of company-led moderation.
As far as “worth it” goes, some people have to be exposed to it so long as we have law enforcement (but I’m certainly open to alternatives here). I’m not sure the train operator is a fair comparison, because seeing a suicide is an exceptional circumstance in their job, it’s not the norm. The content moderators, however, are sadly expected to be exposed to traumatizing content as part of their job description — it’s essentially the point of their job.
There are plenty of kinds of work we deem as hazardous to people’s health, and thus are either banned or regulated. I’m not sure if there’s a healthy way to expose people in these moderator jobs to the traumatizing content they face. It just doesn’t seem worth the tradeoff to endanger them like this.
Think like a legislator. How do you write this regulation?
> [shut down] just large / public sites that require this sort of moderation
Let’s say I start a restaurant review website that allows comments and photos to be uploaded. It does modest business for a while, I now have 50 employees. I’m following the law because my site isn’t big enough to violate this “no user content for big prominent websites” law.
Soon, it becomes big, like a major competitor to Yelp, and I’ve got 1,000 employees. But suddenly, this new law kicks in that says that I have to stop accepting uploads because my site is too high profile. Now, I lay everyone off and go out of business.
This just isn’t a workable solution, at least not in the particular way you’re proposing it be constructed.
And really, you’re asking the second largest advertiser on the web (Facebook), a Fortune 50 company, to just pack up its bags and shut down.
It’s not like I love Facebook or anything, but I’m sure their 45,000 employees wouldn’t be happy about that.
Why not just require social networks to verify identity and verify users are 18+ before allowing them to make an account? Furthermore, maybe even introduce a direct, easy-to-understand, hard-to-misuse law, enforceable locally (e.g. in the US it be a state law) that says committing fraud to join a social network is a misdemeanor with a $1,000 fine. Facebook can then report fraudsters to local police.
People would be less inclined to post this type of content to Facebook if it their account was very connected to their real identity, if they were 18+, and if there was meaningful punishment for making fake accounts or accounts with stolen identities.
This is not just a facebook problem though. Taking your proposal to its logical conclusion suggests that the internet should have no user created content on it - or am I missing some possible middle ground?
How on earth does the size make any difference to the supposed problem? Even if we somehow had a stable and viable size capping system to whatever nebulous concept of "too big" there would still be people exposed to the same content. Not to mention in the US said restrictions would get a "haha no" from the courts on First Ammendment grounds against arbitrary limitations on speech by source.
Every time you use Facebook, see an advertisement there, or click it, every time you share content on Facebook and get others to engage with that platform, you are contributing to a platform that is directly responsible for human psychological harm, in many different ways.
Same for Twitter, and for Reddit, and Instagram... and probably TikTok.
I don’t believe the use of these platforms can be considered ethical.
Does this logic also apply to commenting on hacker news? Or is it specifically the platforms you mentioned?
What specifically is the reason you consider those particular platforms to be unethical, and what is the solution.
It's not enough to say "social media bad". You can say that about anything including the internet as a whole. We need the reasons it's bad and solutions.
> What specifically is the reason you consider those particular platforms to be unethical, and what is the solution.
These platforms are so large that they require hundreds of thousands of moderators in order to remove vile, illegal content, causing trauma for a vast underclass of foreign workers.
Here’s a few solutions:
Limit the number of people that a person can be friends with to 150, the Dunbar number.
Restrict the posting of photos to those which have been identified as having you or your friends in the picture, or no humans at all. Require permission from all human participants to post.
Is it just that you're worried about the outsourcing of content moderation? Because other than than the problems are the same. Someone will always need to moderate the content.
Size is not the determining factor. It's a single factor that highly influences the value in posting damaging content. It's clear from the rise of influencers on YT, IG, Twitter, etc -- as well as the prolific use of these platforms by terrorists and other malefactors -- that being able to reach many people with a single post is a significant driver of people to post content.
My concern is not primarily with the outsourcing of moderation, but with the type of moderation that is required. There are ways to limit the kind of content that people post. Making things less sharable is one way of doing that. Creating barriers to entry is another. My list of suggestions encompasses both. Of course, these are two suggestions which are antithetical to the ad-riddle, growth-driven social network model, so there is no way they would ever be implemented.
I should add another solution to my wish-list: remove ads entirely.
> My concern is not primarily with the outsourcing of moderation, but with the type of moderation that is required.
But the problem doesn't go away when you shrink the size of the social media site. The same people will still try to post disturbing content. In fact, you're giving them more places to post that content, so potentially more people will have to moderate it.
> There are ways to limit the kind of content that people post. Making things less sharable is one way of doing that. Creating barriers to entry is another. My list of suggestions encompasses both.
Your suggestions can be implemented on smaller websites as well as larger ones.
It seems like your issue is with the ease of access to social media. Thus your solution is to limit access. However, I don't think there's a reasonable implementation of your idea that could work. Nor do I think any kind of social media is going to want to go that route. It doesn't matter if the site is funded by ads or is trying to grow. The point of these sites is to share content. People don't want to limit that.
Also, breaking large social media sites into smaller ones makes it much more difficult to deal with troublesome individuals. Right now, it only takes a few bans before you're blocked from most mainstream sites. If we had, say, 10× the number of sites, that's 10× the number of bans required to get these individuals out of the system and 10× the number of moderators who had to look at their content before banning them.
I'm not saying there's no solution here, but I think people are misidentifying the problem. We all want simple answers and simple solutions even when there aren't any.
> But the problem doesn't go away when you shrink the size of the social media site. The same people will still try to post disturbing content. In fact, you're giving them more places to post that content, so potentially more people will have to moderate it.
I’m not sure I follow. A site like Facebook already has mechanisms for detecting multiple accounts, and in any case, multiple accounts don’t mean that there are more “places” to post. It’s still Facebook.
I don’t propose breaking Facebook into smaller websites. I propose limiting the reach of a single individual on those sites. Basically, normalizing the localness of social media to be more meaningful.
As well, I think Facebook and other platforms need to reckon with the consent when it comes to posting images of people, in general. That’s why I suggested limiting the posts to only those of friends who consent.
And again, you run into the problem where nobody wants their website to work like that. In fact, limiting the person's audience really only works with facebook's system where people post content specifically to their friends. Other sites like Hacker News, Reddit, youtube, and various other forums aren't even designed with the concept of friends that are the sole consumers of your content. You're specifically posting in a public space that everyone can see. That's not a social media thing. That's just an internet thing where most things are visible to everyone.
And again, people still post horrible content to small groups just like they post it in large ones. You've divided the problem up, but you haven't really solved anything. Someone has to moderate the content.
>And again, you run into the problem where nobody wants their website to work like that.
Cancer patients don't want chemo, but it's better than dying, some might say.
>Other sites like Hacker News, Reddit, youtube, and various other forums aren't even designed with the concept of friends that are the sole consumers of your content.
Right, which is why those sites (besides HackerNews) would require slightly different solutions.
Reddit: Limit subreddits to 500 participants at most. Eliminate the number next to upvotes. Verify accounts. Limit posting.
YouTube: Not sure. This one is video-forward, probably the most difficult problem in terms of bad content. Definitely remove the algorithm for targeting people based on interest, though.
>That's not a social media thing. That's just an internet thing where most things are visible to everyone.
This is not any feature inherent to the Web, it's a function of sites that purposefully link together and allow people to rapidly post information. Pre-social media, to get your idea out you had to build a website. There was friction. The earl y web had little moderation because you really had to go searching for bad stuff.
>And again, people still post horrible content to small groups just like they post it in large ones. You've divided the problem up, but you haven't really solved anything. Someone has to moderate the content.
Dividing the problem up is a strategy that I propose lessens the impact to both the users (because content can't spread as fast) and the moderators (because there will be less content to moderate).
Elimination of advertisers and the implementation of cost (as a form of friction), I think, would also go a long way. Cost per post would be ideal.
> Cancer patients don't want chemo, but it's better than dying, some might say.
What good is chemo if your patients refuse it?
> Reddit: Limit subreddits to 500 participants at most. Eliminate the number next to upvotes.
This doesn't stop people from posting disturbing content, and it destroys reddit's entire model. This solution doesn't work.
> Verify accounts.
In what way? Do you want to abolish anonymity? Because I can tell you right now that most of us aren't interested in such a solution.
> Limit posting.
In what way? Throttling posting speed? Sure if you just lessen the amount of content overall there will also be less disturbing content... but there's still disturbing content that needs to be moderated.
> This is not any feature inherent to the Web, it's a function of sites that purposefully link together and allow people to rapidly post information.
The default state of the internet is publicly viewable information. Thus it's an inherent feature that you can access almost anything out there.
> Pre-social media
As far as I'm aware, there was no such time. There have almost always been bulletin boards, forums, chat rooms, image/file sharing sites, and other forms of social media.
> there will be less content to moderate
Dividing up the content doesn't reduce the amount of it that needs to be moderated any more than cutting a cake reduces the amount of cake.
> Elimination of advertisers and the implementation of cost (as a form of friction), I think, would also go a long way. Cost per post would be ideal.
This just shuts people out according to income bracket. It reduces the total population, but I doubt it helps reduce the fraction of that population that are interested in posting and sharing disturbing content. It's just throwing the baby out with the bath water.
HackerNews is a limited-focus link aggregation and comment platform. It doesn't require the kind of moderation that larger scale, broad-focus social media platforms do.
I don't envy the work that Dan and Scott have to do in the slightest, but I don't think they'll end up with PTSD from it. At least, that's what I gathered when I read "The Lonely Work of Moderating Hacker News"[0], especially this description of it: "Pressed to describe Hacker News, they do so by means of extravagant, sometimes tender metaphors: the site is a “social ecosystem,” a “hall of mirrors,” a “public park or garden,” a “fractal tree."
social media is a mirror--you see what you want in it. for every horrifying or hyperviolent act or outrageous scandal, you can find the most joyous and beautiful depiction.
have you ever seen foxes laughing? they sound like babies. without social media, i never would have known that, nor experienced one minute of happiness while watching it.
I think this is pushing it a bit. By similar logic, paying taxes to, or simply being a member of the US (or almost any country) is contributing to a platform that is directly responsible for human harm.
>paying taxes to, or simply being a member of the US (or almost any country) is contributing to a platform that is directly responsible for human harm.
I agree with that. But I also assert that people have an ethical obligation to opt-out of systems that cause harm when / if they are able, or at least advocate to change them.
I don't have the ability to change Facebook. But I have the ability to opt out of it, though my actions on the Internet, and the ability to advocate for others to do the same. So, here I am.
What I am wondering, and this is probably a dumb question, is why this has not been automatized? Can't content moderation be done with modern and strong machine learning based systems? There must be plenty of training data on this, and just like a spam filter which does not require humans in most cases, this should also be automatable. Why is it not?
Yes, one would hope so. My guess is that we're still in the data-collection phase. It's probably a tough problem because you want the number of false-negatives to be extremely small.
Not dumb, but I do think it's not thought-through. You're proposing a simple solution to what is a hugely complex problem, and throwing ML at it just isn't going to work. To get "plenty of training data," a human still has to classify all of that, leading to the problem of viewing that much unpalatable content by a human. You also have to train your network, which requires humans to verify the accuracy of training, hence viewing the content again.
If it were as easy as text-based spam filtering, this wouldn't even be a discussion.
I'm guessing you don't actually understand the complexity involved.
How does it differentiate between normal ranting and hate speech? How is hate speech classified in the US vs Saudi Arabia? How does it tell the difference between someone asking a child innocent questions and asking them sexually-related questions on camera? Does the algorithm get trained to flag videos about depression that might lead to suicide, or does it say they're supporting getting help FOR depression and leave the video?
What about subjects that aren't already in the corpus of "flag these naughty things"? You still have to get a human to look at those; most likely a data scientist who knows what the algorithm is doing and what needs to be done to correct the training. Machine learning as-is will not get us there, so in the meantime the only option is moderation by people described in the article. It can be outsourced elsewhere, but it's just shifting the responsibility to a different subset of people.
Certainly there is more complexity than spam filters, and cultural differences, subtle nuances which are hard to train a system to auto-classify; things that need to be generalized to new cases that arises- that may be wrongly classified in the beginning, etc. Nevertheless such systems can be built from simple cases to more difficult ones step by step and gradually reduce the demand on humans to perform these tasks. Based on some other comments on this thread, I realize FB is already using these methods mostly in languages with plenty of data, and someone also in this thread has made an interesting comment about using machine translation to apply these systems to other languages. I think this will be less of a problem in the near future. (edited)
What everyone else said, but also the presence of false positives means there'll be appeals. So you have a choice or ignoring the appeals and being accused of censoring people, or processing them - and that tends to need a person.
The issue is that this is adversarial in nature. Certainly there are ML systems catching content before moderators.. however since it is adversarial people are pivoting to circumvent the automated systems.
It has been. Most of these sites catch a ton of images and video automatically when they are similar enough to prior known content.
That doesn't matter from a staff point of view though. You have a queue to work through. You'll be putting in an 8 hour day dealing with the stuff the system doesn't catch. The automation just means they don't need as much staff.
I am just hoping that those currently uncovered cases will be "milder or more nuanced" cases that will be less damaging to the psyche of human moderators and will, once labeled correctly, improve the coverage rate of automated moderators.
It's not automatable at all, for example Facebook's guidelines change by the day and sometimes by the hour, and many cases are not able to be analyzed by machine as the error rate is too high.
Not to mention the fact, that we'd be subjecting AIs to the worse that humanity has to offer. This is the kind of stuff that creates Ultrons or Skynets who decide that humanity is trash and that it ought to be wiped out. Lol.
It's a really hard problem to automate. You can scan photos against a list of hashes to sort out known bad / illegal content, but new content is generated every single day. Also YouTube has increased the amount of auto-moderation but that leads to many legitimate videos being taken down, with little recourse for those legitimate users. Some examples:
It also sounds (from other articles) like people are seeing specific clips multiple times. Surely a bit of semi-AI filtering should be able to blacklist the regular stuff
Perhaps it's a solution to use brainwave headsets. So whenever the viewer would get too much negative stimulation, the video would stop and the viewer could take a break.
That was good article and I think something really needs to be done to make content moderation more humane.
This quote from the article
“It gets to a point where you can eat your lunch while watching a video of someone dying. … But at the end of the day, you still have to be human.”
reminded me of a similar article published by WIRED six years ago [1].
Eight years after the fact, Jake Swearingen can still recall the video that made him quit. He was 24 years old and between jobs in the Bay Area when he got a gig as a moderator for a then-new startup called VideoEgg. Three days in, a video of an apparent beheading came across his queue.
“Oh fuck! I've got a beheading!” he blurted out. A slightly older colleague in a black hoodie casually turned around in his chair. “Oh,” he said, “which one?” At that moment Swearingen decided he did not want to become a connoisseur of beheading videos. “I didn't want to look back and say I became so blasé to watching people have these really horrible things happen to them that I'm ironic or jokey about it.”
It would be great if there were a way to make content moderation more humane, but perhaps this is like saying that it would be great if we could make hospital emergency room work more humane. It is unavoidably traumatic, and ER staff are known to detach and use dark humor as a coping mechanism. Content moderation is a similarly noble and difficult profession.
Most of their workload is bullshit. People with colds and sore throats, depressing people using ER as primary care, assholes using 911 to score Medicaid cab vouchers.
And at any time, any number of people can show up with any kind of personal tragedies from stokes to various traumas. My sister quit after a 13 year old bled out from a gsw, and she walked out of the room and got kicked in the head by an prisoner who had been stabbed after he bit the ears off of three other prisoners, and broke the arm of a guard.
I would imagine that cases should follow a bell curve - sure, the exceptionally good cases where you feel you saved someone's life are rare, but so are the terrible cases that haunt you at night; and most would fall somewhere in the middle.
That said, I would also wager that the curve can be skewed towards one side or the other depending on the location, timing of your shift, or any other factors.
Having worked as a psych tech and later as an EMT in a major public hospital in Houston, you’d be surprised about how not-well paid many allied health professons are. And the janitors get to clean up the blood and hear the exact same screams or being on the receiving end of drug fueled rage attacks — and they aren’t getting paid much. If you sign up to work in what is effectively a battlefield hospital, you are exposed to the vile underbelly of society on a daily basis. And how does a union matter? It doesn’t shield you from getting attacked or vomited in by violent drunks or attacked by hysterical family members.
Let’s say a content moderator got paid $200k per year, that doesn’t change the images they see. This isn’t about money as much as it’s about the requirements and preparation required for the job.
I think that you are wrong in general, but that parallel is right on point.
The root of the problem, which is rarely mentioned in these discussions, is that many people enjoying comfortable living conditions believe themselves to be “pure-blooded” and unable to deal with “unclean” things because of the possible “traumas”. (What is an ideal person free of “traumas”, anyway? A vegetable stoned on drugs?) You can easily see here the resemblance of a old time countess complaining about her “nerves” while her serfs do all the dirty work. Is there a natural law that says a poor grunt is bound to clean the junk from what you see on the website? Do it yourself.
The same happens with medical, law enforcement, and other professions. They are considered “special” jobs for “special” people (which is not true, the one who defines it like that is “special”). Then, any worries and doubts are neutralized by convenient “but it's their profession, they chose it, they are paid for it, etc.” This creates a stupid rift between “us” and “them”, and affects how both sides reason about themselves.
The resulting imbalance is routinely abused. Social networks are competing in presenting a rose-colored walled fantasy to their human cattle to increase yield — everyone can see they use “The Matrix” as a business plan. This creates a lot of shitty jobs where people waste their emotional capacity to make others pleased, not unlike in prostitution. On the other hand, media is hooking the softened consumer with the sensationalist outrage bait about maniacs, wars, corruption, presumed orgies, etc. To makes things worse, the limits to all of that are set not by some moral standards or concern about someone's well-being, as it is often implied, but by advertisement contracts. The bureaucratic machine decides what is “suitable” for the visitors. For example, this makes most of the talk around “Free The Nipple” moot: there is no undercover agents of patriarchy to battle, it's advertisers who bring money and judge what is unwanted “sexual content” (some might say it's a perfect example of capitalist oppression — cue “L'Internationale”).
Even this discussion is full of overreacting people. No, the video on your screen is not as “horrible, horrible” as experiencing something firsthand, nor the pain of the tortured can be compared to the pain of the random witnesses on the internet (if we don't count their own wild fantasies). If it happens, then it's no good to pretend it's not. Who knows, it may even make you think. All in all, humans don't show a lot of originality in what they produce, so the number of things you can expect to see is quite limited. I can only conclude by reiterating that knowledge is power, ignorance is bliss, and there's a lot of people who'd like to benefit from others' ignorance in many ways.
The case of law enforcement in US is unusual in that claims to being "special" come largely from the officers themselves - that's the whole point of the sheep/wolf/sheepdog theory that much law enforcement training is centered around these days.
I still maintain expecting companies to do this on their own is a category error.
Companies think moderation is a cost center. Even if the moderators do the best possible job, they aren't adding value, only removing liabilities. The company is the wrong entity we should be asking for solutions to this problem. They are going to minimize costs and shift liabilities to 3rd parties (which will in turn hire independent contractors and play shell games in case they are sued).
We have to demand adversarial outside actors (government, labor unions, lawyers, non-profits, mental health professionals, etc) to start shaping the direction of the solution.
Modest proposal: Find people who enjoy horrific content, and have them rate it. Then black-hole whatever they rate as most extreme.
I was is a dark place after 9/11, both from the trauma of that, and some personal issues. So I spent considerable time on Rotten, Something Awful, and worse. Based on that experience, many would likely do content moderation for free.
That sounds like a great idea, not only to just black hole the content, but to have people who are less prone to have adverse side effects from watching such material be moderators.
Thanks for sharing that part about being in that kind of place, I have a feeling many could relate.
Maybe not enjoy, but there are certainly people who are much less affected, or not affected at all, by it.
These media stories always focus on those who picked the wrong job for their caracter, and then make a big drama of it.
I mean, there are people cutting open other humans daily. We call them surgeons. I would puke right there and could never do that. Different people have different abilities. Simple.
Yes, I get it. Those are not nice people. Few of us would want to be friends with them. Or even work in the same building. So have them work from home, or for a standalone subcontractor.
I also get the downside of employing them. But people need to look at that stuff, even if software does most of the work. So it might as well be people who can tolerate it, and perhaps even enjoy it.
If you don’t recognize it, “rest of world” is a ridiculous corporate term commonly used in global business operations. It’s a catch-all phrase that means, basically, “everyone else.” And it generally represents billions of people outside of the Western world. We know that their stories matter. The term “rest of world” is a symptom of a larger problem: a Western-centric worldview that leaves innumerable insights, opportunities and complexity out of the conversation.
267 comments
[ 2.5 ms ] story [ 293 ms ] threadCan you crowdsource the moderation task to double check what the ai is flagging?
I highly doubt that. These reported posts might contain private data or highly shocking content; putting that on a public platform, especially associated with the name of Facebook, is very unlikely to happen.
Is that not what the person who originally posted the content being moderated is doing?
The person that posted the content does not expect a random person from the internet to see a video of his children because it was accidentally flagged by an AI as child porn. :)
"Facebook sent me child porn" is probably not a headline Facebook wants out there.
https://time.com/5739688/facebook-hate-speech-languages/
This would help with false positives, which is a noble goal. But it would not help with false negatives, which is imo a very disastrous consequence in this specific scenario.
Something that isn't CP got marked as CP? Sure, a moderator can manually doublecheck that flag and override if needed. Something that is CP didn't get marked as CP? You got a problem on your hands, because there is no way to manually doublecheck that, since that would mean manually checking every single post marked as "no issues here", which gets us back to square one of manual moderation of all posts.
Not even mentioning the exposure of FB users to CP that it would bring, which would mentally ruin quite a lot of people and bring very negative press to FB on the level of severity that we haven't seen before.
Or to restate, sites that rely purely on manual moderation of content that breaks rules still has content that breaks rules leak through all the time.
Users don't want the back-stop to be "a critical mass of community members object;" for content like this, they want to see zero of it, ever, and will choose to use another site that satisfies that hard constraint if FB cannot.
No system can ever stop 100% of users from seeing 100% of the material they find objectionable. Every user has their own idea of what material is beyond the pale, and even if we invent AI that is competent at blocking content, and capable of tailoring its filters per-user, it will still fail some of the time, because users are people, and people's tastes change over time.
Aside from that, the way Youtube, Reddit, Twitter et al, currently operate, a user who reports objectionable content typically waits days, weeks, or years (or forever) for the company to take action. If you give a user a little real agency, it goes a long way to mitigate their displeasure over occasional exposure to unwanted content.
Meanwhile, if your system requires users to view beheading videos regularly, people will just migrate to a site that doesn't require that.
Precisely. How often? Close to never.
https://www.bangkokpost.com/thailand/general/1853804/mass-sh...
I could swear there was also a Christchurch copy-cat attack in Europe, but I can't remember sufficient details about it to find an article. Perhaps that one wasn't streamed on FB.
I don't have encyclopedic knowledge of Facebook atrocities (I closed my account nearly a decade ago), but without vetting content before it goes live, I don't see how they will entirely prevent these videos from reaching some users.
(That's before we factor in unintended consequences such as the risk that a critical mass of moderators decide the garbage is signal and start passing it. It's more risk than FB wants to take on for a problem they already solve via paid employees).
There are dedicated communities to all these things.
I do not mean this as a hit against Reddit users. It is just that there are small pockets of people who are psychologically capable of tolerating this or even enjoy it. It seems more ethical to hire people from /r/watchpeopledie.
Edit: and saying things like "well clearly these people are already traumatized if they're seeking this out, it's not good for them, we shouldn't give these jobs to them" seems laughably paternalistic when the alternative is to traumatize more people by making them look at it instead.
That they seek it out doesn't mean they're not messed up.
Maybe it helps dealing with that trauma. Maybe it makes the trauma worse. Maybe there was no trauma to begin with, and the person was seeking out that content simply due to morbid curiosity, but then it evolved into something else. Maybe there was no trauma to begin with and no trauma appeared after the exposure. Who knows.
But just because there are communities on reddit with people seeking that kind of material and not going on crazy violent sprees doesn't mean that it doesn't negatively affect them. It could be the case, but we should probably look into that first before jumping to "obvious solution" claims.
tl;dr: we gotta do more research and weigh pros and cons before simply declaring "we have an obvious solution, let's just hire all those gore content enthusiasts to moderate violent content, because they already do it on their own and don't seem to be going crazy in obvious ways due to it".
People come to their choices with all manner of existing problems and conflicts, and they don't always do things in their own best interest.
Human are so diverse.
Is is really hard to believe that there are people that are not negatively affected by this ?
That would do wonders for the Facebook brand.
Look at how much trouble they got in when it was revealed they were using Onavo (even before acquisition).
Obviously not everyone, but they are there.
My basis for thinking this is I've seen videos like that before and clicked away quickly. It's disturbing, but I think I'm okay.
It's a strange conundrum because ideally you want good people who are comfortable watching things good people rarely want to watch.
Don't have much experience with this demographic -- maybe there are more sane, emotionally healthy people watching snuff films for fun than I instinctively expect.
It'd be like having a jury pool made up of serial killers.
These kinds of bottom of the barrel places of employment tend to have their policies and procedures tuned to make sure employees do things the way the company wants them done. That's just the nature of doing an unskilled, non-laborious job for a BigCo. There will be automatic spot checks, audits, reviews, etc. It is no different than the cube farm on the next floor where they review images of documents that AI systems have flagged as possibly fraudulent. Such systems are likely in place already for their current moderators.
Yes, but their hiring pool tends to be "everyone".
Those procedures might not work as well if your hiring pool is explicitly confined to "the people who watch ISIS torture videos for fun". I suspect standard policies and procedures include "fire the people who get off on this stuff".
They have little power to abuse and obvious accountability. If they block kitten and baby pics and allow people being tortured to death they'll get fired quickly after the inevitable shitstorm.
Insane and unhealthy or creepy doesn't neccessarily mean dangerous. Even if they got twisted pleasure from it they would be gross but not inflicting actual harm. It isn't like people are being brutally murdered to give them kicks - they aren't a source of demand.
I found most of the videos less offensive than extremely graphic horror movies, which ironically I have less stomach for. If anything that subreddit made me generally more aware of my surroundings and safer in my daily life.
Probably not, since there are no documented instances of any snuff films ever found.
https://www.snopes.com/fact-check/a-pinch-of-snuff/
Regardless my point was just that in my experience many would consider them 'snuff' even if it doesn't meet the exact technical definition.
And those people would simply be wrong.
I mean, let’s back up a bit. Why is it important whether or not “snuff” films exist? Because many people want to believe the common urban legends about it – i.e. that there exists an (be it ever so small) underground production and market for it, even though, by all accounts, this does not exist, and has never existed. Believing such urban legends helps people feel justifiably horrified and helps validate their cynicism; basically the same reason why many people believe many other urban legends. However, no movies fitting that particular model have ever been found to exist.
If anyone wants to to be picky about the definition and claim that some movie or other fits the definition of “snuff”, or is “considered” to fit even though it doesn’t(!), this is only because they want to be able to say “see, snuff films exist!”, and thereby validate their preexisting belief in the urban legend. But this is a motte-and-bailey fallacy; it tries to defend the urban legend (which is hard) by trying to defend a technical definition of a word to be slightly wider than it really is (which is comparitively easier). Luckily, even though it is easier to defend, this latter position is still demonstrably wrong.
Thanks to Wikileaks publishing the Dutroux Dossier in 2008, we know hundreds of snuff films were recovered from the notorious pedo-rapist Marc Dutroux.
Just because the police keep those films as sealed evidence that are never published does NOT mean snuff films are an Urban Legend. I can see why the police always keep snuff films secret. Imagine what would happen if snuff films were uploaded all over the Internet? There would be mobs of angry villagers armed with torches and pitchforks descending on jails and prisons to lynch pedos. It would make harassment from QANONs look like school yard bullying.
(The films were, according to your reference, made only for blackmail purposes, and therefore certainly not made for enjoyment of any viewer, nor were they ever distributed to other people for their enjoyment. The reference does claim that people were killed in a specific place which was also filmed, but does not, what I can see, explicitly state that any murders were actually filmed, other than incorrectly calling all the films “snuff” films.)
For example, instead of showing one person torturing another person, a DL filter could turn the scene into two teddy bears torturing eachother.
I wouldn't assume "watches this stuff" and "is psychologically capable of tolerating this stuff" are one and the same. Those people might be actively harming themselves by watching the content. That's one thing when they're volunteering to do so (though that's an interesting conversation in and of itself) but it's quite another when Facebook is paying them to do it.
One day I had an anxiety attack because I had a dream that I had a heart attack and suddenly I had a deep fear of randomly dying.
All the random deaths on that sub started to really get to me, I even started to get depressed. It became harder and harder to watch content on it. Eventually I unsubbed from it and gradually bounced back to normal.
I don't really regret it because it gave me a new perspective on life and how valuable it is. But I would be lying if I said it didn't fuck me up in a way.
take ER doctors. they daily witness human trauma far worse than viewing horror show videos and images. do you ever hear about ER docs who become depressed or mentally ill from being exposed to so much horrific human suffering? Doctors have a personal sense of doing good in the world through healing. They witness so much trauma that they become so desensitized that non doctors often criticize them for seeming so detached and for treating human bodies like mechanistic processes. But doctors have to be somewhat detached in order to do their jobs well.
It's not specifically about ER doctors. Anecdotally, I'm a lawyer, and they warned us in law school that lawyers have abnormally high rates of substance abuse and depression, and constant jokes to the contrary not withstanding, most lawyers believe they do good. Emotionally hard jobs are hard, even if they are indisputably good.
Yes you do! But I would imagine it is at least somewhat offset by the fact that they are doing a lot of good: they are saving people from death while also witnessing death. I think a content moderator would struggle to feel such a strong sense of purpose.
Serious question, I don't mean this facetiously. I think is an extremely complex question and I wonder if introducing someone who is openly comfortable with gore is dangerous to the psychological health of the company.
For example, at a previous company a co-worker publicly shared, without a trigger warning and in great detail, a very gorey thing he enjoyed watching to relax. Lots of people were extremely disturbed by this -- not disturbed by him, per se, but disturbed that he shared this without any kind of warning. I don't even resent him despite myself being pretty disturbed because what one considers normal is subjective, and if you regularly relax to this content you probably don't realise that this might not be anyone else's cup of tea. But it really makes me wonder -- what if the guy who sits next to me starts telling me that ISIS beheadings are relaxing to him?
It seems like if you acknowledge that: a) Terrible job exists b) Terrible job is necessary
You should also allow for a person to do terrible job without thinking that they must be a terrible person by extension. Otherwise you continue the well-trodden history of certain professions like hide-makers, executioners, or coroners being ostracized from society for doing the job we told them to do.
I agree that I wouldn't personally spend time with said person, but my question was less about personal social responsibilities outside of work. My example I pointed out happened at work, during normal work hours, in a social space where people have inherently less control over who they surround themselves with. This is the part of it all that is fuzzy to me, and I don't have a clear answer for.
And while a lot of content moderators do focus purely on clicking thru things, a lot of other ones still regularly interface with the company. I used to work at Discord, where content moderation of this level (and traumatic nature) was a constant concern. As engineers you may be staffed to work on a tool to help with moderation. Or you may be working on some fancy AI to help sort and tag unsafe content. And this is only from the product engineering perspective; content moderators will have to work with customer success to create content policy, or enforce bans for unsavory behavior. So I don't think they really are siloable away from the rest of the company, and I'm not even certain that that is a humane way to treat them.
The entire point of this is to find those who deviate enough from normal to not be as impacted.
Desensitization is certainly a thing. The first few times seeing those things were shocking and disturbing. Then it got exciting. Then it got boring. I don't actively look for that content, but never having left 4chan, I still sometimes stumble upon it and it causes no big reaction. I think I could totally do that content moderation job, I have to admit some morbid curiosity about what I would stumble upon exists in me. The only content that never stopped to disturb me was violence against animals.
Talking with my "normie" friends about this I realized I am the odd one out. I wonder just how much consuming that content in my youth changed me.
Is deeply offensive content generated by a handful of users frequently, or many users less often? What's the volume look like?
I want to test if it can cause me any trauma
The local governments won't acknowledge this as problem because of the money flowing in and a lot of people's livelihood is dependent on this. But we should call it what it is, exploitation.
> According to Dr. K. Jyothirmayi, a Hyderabad-based psychiatrist, stigmas, such as the perceived impact of a mental health diagnosis on one’s marital prospects, often prevent young Indians from seeking treatment.
From growing up in India, my view is that meditation and the general state of the mind has been taken seriously for, well, centuries. But move the focus towards anything with 'psych' or 'mental' in the title and people in India will shy away from it even if it is affordable or free. "Seeing a mental health counselor" is a recipe for considerable social stigma in India [1], not an empowering practice like it's perceived as in the United States.
The solution to this is hard, and in my opinion not something that Facebook or their Indian subcontractors can accomplish.
-----------
[1] I'm talking "All your relatives will refer to you as the crazy person" or "Your girlfriend's parents will call off the engagement" levels of stigma, not just "I don't feel comfortable talking about it in a bar" levels. Like a lot of other things, families with higher levels of income and education are sometimes an exception.
Real life has always been filled with messed up situations where the punchline is "... and the best/worst part is it qualified as an improvement!".
- Facebook: it's violating rules? Delete, next.
- Forensic IT on a multi TB disk full with child porn: document every photo, what it shows, extract identifiable faces to cross reference with other content (to check for recurring places and victims), and the process is even more gory for video content. You have to watch every second or the defense can attempt "you didn't watch the video in full where the perp gives the victim an ice cream at the end" or whatever else. The amount of time you spend with documenting a single photo or video is many orders of magnitude worse than FB content mods.
Facebook doesn’t have to let people post material to their site with very low barriers to entry. Granted they have to if they want to continue to be what they are and to make tons of money... but maybe they shouldn’t continue to be what they are and make tons of money if this kind of abuse is necessarily coupled to those outcomes.
“Well this sucks but we can’t keep operating if we don’t do it”. Well... sure but the solution is right there in that statement. Don’t keep operating.
[edit] thought experiment: how many of the people making tons of money off Facebook—c-suite, major shareholders—would find some other way to make money if continuing to make Facebook mega bucks meant they had to do this 5 days a week? What would we think of any of them who chose “bring on the trauma, I want those sweet greenbacks” and kept it up for years?
I think the problems that Facebook, Youtube, etc are seeing stem from the fact that the content is much more public, thus in much more need of moderation.
It's obviously horrible doing this as a job and it's obviously a bit more common on Facebook than say, a large old school internet forum, but... at least in this case people are paid to do this. Reddit mods and old school forum mods have to deal with this stuff for free.
I can go door to door asking people to change something, and dozens of my volunteers will feel useful but overwhelmed by how much work there is to do in a city of a million people. Or I can go to city council meetings, and get more leverage there. Or if I can convince one high profile citizen to participate, others will follow. Or I can get into contact with the mayor, state congressional members, the governor, federal congressional members.
It's possible that if I convince the correct five people that they will take half of this burden off of my hands. You'd be crazy not to reach for a lever like that. They are a juicy target, and not just from a punitive standpoint. Of course when you are rebuffed repeatedly by a potential ally, then it's a very human (if questionable) response to seek out some sort of punishment.
edit: What I mean to say is, people have been organizing against FB, AMZN, and tech in general for the past few years now. The last is reactionary but opposition to FB and AMZN has meaningful academia, ngo/gov, and general support.
But... that group wouldn't be able to do nearly what it has if it weren't for funding from foundations and the department of neighborhoods, and resources on permanent loan from another public sector organization. You might argue it couldn't exist at all.
I think I would say that both are necessary but insufficient.
edit: https://qz.com/1704143/the-antitrust-case-against-facebook/
I also think part of the objection here is around off-shoring. It's similar to other complaints about off-shored jobs: we're farming out unpleasant work to people that get paid less and have much less support (e.g. mental health).
That's an interesting point, and has me wondering something.
If a site like Facebook opened up a volunteer moderator scheme where people could sign up to filter out stuff that broke their rules, would it make things better or worse on a moral level?
What about a US/Europe based agency that did this thing onshore as a contract deal?
https://www.theverge.com/2019/2/25/18229714/cognizant-facebo...
As for the volunteer side, it's a really interesting debate. Others in this thread have pointed out Reddit as an example of this in practise. I think the sheer amount of money that Facebook has means that relying on volunteers to (potentially) inflict emotional damage upon themselves wouldn't be a great look.
I think they tend to take an "if in doubt, remove it" approach which basically allows anyone who doesn't like a niche page for whatever reason to have it taken down through sheer attrition, if you report it enough it'll probably go down eventually even for stupid reasons. I actually ended up writing an app to cut Facebook out of our system altogether (it was kind of similar to Yik Yak with a much better moderation system and better anonymity protection - I really can't be arsed to deal with a GDPR lawsuit...) but we couldn't launch it in part because it took me ages to develop as I've since graduated and work as a full-time developer and because coronavirus meant our launch event got cancelled. I might take it live when students go back this year, but I'm concerned that with nothing much going on it'll fizzle and die because there's nothing interesting to post about. It'll all depend on how the Welsh government lift the coronavirus restrictions.
Disclosure: I work for but don't represent Google. This is my own opinion.
I've done commercial moderation before and I would sooner go jobless than go back to exposing myself to the kinds of things people have to when doing this professionally. It's the kind of things you wish there was a brain cleaning solution for, but there isn't so it's stuck there basically forever (though I suspect the alcoholism rate is higher among people who have to handle this kind of thing than in the general population).
The commercial moderation team I was on actually ended up separating out a task force for handling particularly gnarly content so that people had a way of opting out of being exposed to it (in our case we were responding to user reports rather than randomly stumbling on things, so manually filtering reports was feasible), which I think is the responsible approach since there are plenty of other moderation tasks to handle and some people are more or less sensitive to this content.
'Drug-induced amnesia' to reduce long term memory generation is a thing and is sometimes used before traumatic surgical procedures. It needs to be taken in advance and is probably not very healthy. Plus I would have serious ethical concerns with jobs that would encourage that.
Just out of college in the mid-90s, I worked for a few months at a small local software company in the last throws of death. As a last grasp at survival, all the programmers switched to doing cold call sales. With phones.
I would stare at mutilated children all day long rather than go back to cold calling. I was miserable and I hated every moment of every day until the company (happily, for me) closed up. Yet I know people who somehow think it's perfectly normal to dial random numbers and try to pitch them something. They're made of different stuff.
I've talked to people who make the same claim about computer programming. They'd rather starve.
Facebook has 10,000 moderators. I'm pretty sure there are at least 10k people in the world who are psychologically fit for content policing. It's doesn't have to be for everyone.
There exist jobs, like slaughterhouse knockers, which are linked to causing PTSD and increased criminal behavior. It's not far fetched to believe these moderator jobs do too.
It wasn't that long ago HN was abuzz about large numbers of suicides at Foxconn... until someone realized that the suicide rate was actually slightly below the community average. I simply don't trust arguments without statistics anymore.
Big companies like Facebook (although they are not alone) don't do that. That costs money. They use disposable workers, like on-shore or off-shore contractors, and do industrial-style quotas/piecework methods to maximize productivity.
When you hire like that, you'll have a normal distribution of people. On the left there will be pre-broken people who get pleasure from watching decapitation videos and nasty shit, on the right you'll have people who are traumatized and broken by the work, and in the middle you have people who will cope for some period of time, and burn out.
I don't think many people are immune to PTSD, but I suppose sociopaths need jobs too. But I'm not sure you want to hire a department of sociopaths.
Why not, if they do the work well?
And anyway, they'd be working from home, these days.
My point is, some people simply aren’t traumatized by photos/videos, either via talent or training, and because of this they can process it enough to categorize it and keep their emotional/social/intellectual distance. That doesn’t make them sociopaths and it certainly doesn’t make them bad people.
A bleeding anus is just a thing that happens to people. It's a bit gross but I don't know why anyone should be traumatized by it.
The kind of content that these moderators have to deal with goes well beyond the mere abject.
Fixing a sewer is gross. Extermination is gross, not traumatic, unless you have extreme empathy for small rodents and insects. Doctors deal with very few remotely bleeding anuses on the order of one a week/month maybe versus one a minute. None of these should be called real sickos, but none would be expected to develop PTSD either.
A significantly bigger portion of the population would be traumatized and develop PTSD by the level of exposure of content moderation. While not nearly all are sociopaths, they 100% will be represented higher even without factoring in selection bias. Hiring a department of them would not be inconceivable, putting the validity of that aside.
You also make the jump from sociopath to bad person, which many I'm sure would have some qualms with as a categorical.
PTSD is real but lots of people still go to war and not everyone comes back a basketcase. Most don't, in fact.
No, I've never had a job watching pictures of mutilated children all day. But like a lot of naturally curious people, I've searched out and watched bloody videos of beheadings and car accidents and whatnot. It wasn't pleasant, but it wasn't really traumatic either. If I didn't already have a much more lucrative career, I don't think I'd suffer too much as a Facebook censor.
Does that make me a sociopath? My wife and son don't think so, nor do my close friends. I'm not sure if I should feel a little offended here.
On the other hand, I may still have PTSD from those days of cold calling as a junior software engineer 25 years ago. I hate phones. Maybe people are just wired differently.
It sounds too simple to work, but worth a shot.
I don't think at the same level. Social media usage went from 10% of the population to 79% from 2005 to 2015 and was even lower a few years before this. So current social media companies operate at unprecedented scales. Moreover, the smaller percentage of people usage social media in earlier times tended to be "sophisticated users" who didn't necessarily expect their feed to be systematically filtered.
https://www.pewresearch.org/internet/2015/10/08/social-netwo...
Anyone know? Is it like 10%? 80%?
There must be, by now, a whole lot of data on whats getting flagged region/language wise.
Maybe we need a Cloudflare for Content.
It did not take you long at all to just vault yourself down the full on dictator slope. Next up, let me guess, you want to prevent people's access to the internet if they say things you don't agree with because "word are violence" and then "micro-aggressions are violence", simply because you have the control to abuse. Unfortunately that is already the situation right now, but it does not need to be made worse, because it will only thrust us all down a path of overplaying the unrelenting abuse that I can assure you people will not like the inevitable consequences of one bit.
Let me also remind you that the US has the supreme law of the land that supersedes all other laws, the First Amendment to the Constitution that states that no law may abridge the freedom of speech, regardless of whether you like it or not.
- mom works as a freelancer and the kid uses her computer to post bad content. Does she lose her job?
- someone trolls his mate for lolz.
Someone leaves a hundred dollar bill on the desk and his mate spends it for lolz.
Are these significantly different (if no less unjust)?
You or someone who used your access posts something flagged as inappropriate? Banished to text-only penalty box.
Posting inappropriate ascii art? Banished to emoji character penalty box.
Repeat offender? Longer/permanent prohibition.
We as a society have decided that negligent actors can do enough harm to themselves or others in operating an automobile on publicly-owned roads, that we compel individuals to seek license from the government to drive vehicles on those roads. The license is a revocable priveilege that has to be earned by ongoing demonstrations of competency.
One could imagine a parallel system for "web privileges", although it sounds dystopian at first blush, in contrast to the unprecedented freedoms we enjoy on the web, today.
The tricky thing is that it's relatively hard to precisely track identity and intentional authorship on the internet. It's also easy to imagine somebody losing their internet privileges, because some botnet hijacked their printer and used it to spread X-rated content. In the analogy, it's hard to disambiguate between drivers and vehicles.
Maybe we'd require some biometric login system, provided by the government, to use a web browser. It's hard to imagine implementing any such system in a way that doesn't make the attack surface bigger, but harder problems have been solved.
I can't imagine how "commercial web licenses" would work, though. That's an interesting space for thought.
This basically makes the entire plan unfeasible. Given the current general insecurity it would be entirely too trivial to somehow fuck up and have a botnet get you banned permanently from the "roads".
However, this could work if internet is only available using your digital ID. However, that is seriously actually dystopian and would brutally murder all of the privacy people have and in certain cases also the people.
I think such a system would encourage the development of two parallel webs: the above-ground, requires-ID-for-every-http-request web (Facebook, Amazon), and a somewhat shadier, sometimes-or-never-requires-ID-except-in-exceptional-circumstances internet.
Kind of like how bars and grocery stores are supposed to verify your age, but many don't. Or kind of like how there's large swathes of the country where people drive around without driver's licenses or insurance, and it isn't an issue unless they end up in an accident.
Especially so under oppressive regimes, I think China has already displayed how internet would end up if we went that route. Restrictive, dangerous for dissidents, without any real privacy.
Do you really need exactly the same level of privacy for all the cases though? How about having a corner of the internet which is an ultimate walled garden - with access restricted to someone with validated government IDs and everyone's activity tracked and transparent to everyone else? This will work great for many cases - kids can have access to only this corner, plain vanilla entertainment[1] like Netflix or Disney or Sports, money related stuff[2] like banking / money transfer / shopping, etc etc. We can do those activities with relative peace of mind because everyone accessing this corner would be physically trackable and hence nefarious actors will have the risk of instant exposure. In that sense, it will be very similar to the physical world (or even stricter because of enhanced security).
For a lot of other legit reasons like fighting oppressive governments or hacking things to push boundaries or uncensored forums, we can of course have rest of the internet.
There will be some nuances just like we do in real world, like:
[1] different ratings for entertainment, like R-rated vs PG vs G-rated
[2] protecting sensitive info like bank account details or tax returns
In places like China, maybe! In places like the US, definitely not. After all, US Supreme Court literally allowed neo-nazis to march in public defending their freedom of speech[1].
> People will find so many reasons to kill privacy
They might, but they will always have to play catch up with the technical solutions (cryptocurrencies, strong encryption) etc. As long the fundamental protocol remains open, you can recreate your own stack.
[1] https://en.wikipedia.org/wiki/National_Socialist_Party_of_Am...
Well, sure. But that's why the concept of "criminalizing" is key. Criminal behavior curtails your rights. It's on this premise that the concept of prisons is legal and makes sense.
IMO rather than forbid internet access, an idea to explore is treating the internet as some sort of public area with similar jurisdiction. Thus, if you send an unsolicited dick pic it'd qualify as indecent exposure which is punishable by law.
I think this makes more sense because regulating internet access is hard. You can get public wifi pretty much at any airport or coffee-shop. What doesn't quite make sense is that showing your dick to someone without them wanting to see it can get you a fine or even jail time if you do it in person, but in a photo, video or call online is not punishable.
It's probably illegal just about everywhere to post pictures of rape and torture.
You may want to reword your comment, unless you meant for it to imply that it should be legal to post gore, rape, and torture for people who want to view those.
May I ask you how old are you? I don't mean to insult you or anything like that but lately I am witnessing more and more people who demand the ultimate punishment for every misbehaviour and I wonder of it is a generation thing.
I did not like the movie much to be honest.
It's not the flat-earther and somesuch conspirary communities that worry me. It's the people who share and get enjoyment from this kind of content, and the networks which enable it.
I understand there are some jobs in the world that need to deal with dark stuff (like law enforcement), but social networks just aren’t worth the human cost.
What you’re proposing is not just shutting down social networks, it’s shutting down any website that involves user content, anything that allows photo/video upload, comments, or any kind of user interaction. That’s impossible.
You point out that public safety jobs are view more “worth it,” and certainly they are, but that logic brings up the question of who judges what job is worth undergoing trauma.
In other words, is a subway or freight train driver’s job “worth it,” if they have to see someone commit suicide on the tracks? What about crime scene cleanup companies? Funeral services? Bus drivers? Truck drivers? Nobody’s going to agree on where to draw the line in the sand.
A more realistic solution might be to make comprehensive support systems, mental health resources, and treatment a legally mandated, completely free service provided to any employee that works in these kinds of fields.
Finally, I think there are most certainly people out there who are not as sensitive and affected by this content who would be candidates for these kinds of roles. Perhaps there’s a way to test for that sensitivity before the real job starts.
As far as “worth it” goes, some people have to be exposed to it so long as we have law enforcement (but I’m certainly open to alternatives here). I’m not sure the train operator is a fair comparison, because seeing a suicide is an exceptional circumstance in their job, it’s not the norm. The content moderators, however, are sadly expected to be exposed to traumatizing content as part of their job description — it’s essentially the point of their job.
There are plenty of kinds of work we deem as hazardous to people’s health, and thus are either banned or regulated. I’m not sure if there’s a healthy way to expose people in these moderator jobs to the traumatizing content they face. It just doesn’t seem worth the tradeoff to endanger them like this.
> [shut down] just large / public sites that require this sort of moderation
Let’s say I start a restaurant review website that allows comments and photos to be uploaded. It does modest business for a while, I now have 50 employees. I’m following the law because my site isn’t big enough to violate this “no user content for big prominent websites” law.
Soon, it becomes big, like a major competitor to Yelp, and I’ve got 1,000 employees. But suddenly, this new law kicks in that says that I have to stop accepting uploads because my site is too high profile. Now, I lay everyone off and go out of business.
This just isn’t a workable solution, at least not in the particular way you’re proposing it be constructed.
And really, you’re asking the second largest advertiser on the web (Facebook), a Fortune 50 company, to just pack up its bags and shut down.
It’s not like I love Facebook or anything, but I’m sure their 45,000 employees wouldn’t be happy about that.
People would be less inclined to post this type of content to Facebook if it their account was very connected to their real identity, if they were 18+, and if there was meaningful punishment for making fake accounts or accounts with stolen identities.
I wonder if prosecuting people who post horrible stuff would have a deterrence effect, though.
Code Orange! Breathing mush with eyes !!!!!!!
Same for Twitter, and for Reddit, and Instagram... and probably TikTok.
I don’t believe the use of these platforms can be considered ethical.
What specifically is the reason you consider those particular platforms to be unethical, and what is the solution.
It's not enough to say "social media bad". You can say that about anything including the internet as a whole. We need the reasons it's bad and solutions.
These platforms are so large that they require hundreds of thousands of moderators in order to remove vile, illegal content, causing trauma for a vast underclass of foreign workers.
Here’s a few solutions:
Limit the number of people that a person can be friends with to 150, the Dunbar number.
Restrict the posting of photos to those which have been identified as having you or your friends in the picture, or no humans at all. Require permission from all human participants to post.
Charge a fee for access.
Is it just that you're worried about the outsourcing of content moderation? Because other than than the problems are the same. Someone will always need to moderate the content.
My concern is not primarily with the outsourcing of moderation, but with the type of moderation that is required. There are ways to limit the kind of content that people post. Making things less sharable is one way of doing that. Creating barriers to entry is another. My list of suggestions encompasses both. Of course, these are two suggestions which are antithetical to the ad-riddle, growth-driven social network model, so there is no way they would ever be implemented.
I should add another solution to my wish-list: remove ads entirely.
But the problem doesn't go away when you shrink the size of the social media site. The same people will still try to post disturbing content. In fact, you're giving them more places to post that content, so potentially more people will have to moderate it.
> There are ways to limit the kind of content that people post. Making things less sharable is one way of doing that. Creating barriers to entry is another. My list of suggestions encompasses both.
Your suggestions can be implemented on smaller websites as well as larger ones.
It seems like your issue is with the ease of access to social media. Thus your solution is to limit access. However, I don't think there's a reasonable implementation of your idea that could work. Nor do I think any kind of social media is going to want to go that route. It doesn't matter if the site is funded by ads or is trying to grow. The point of these sites is to share content. People don't want to limit that.
Also, breaking large social media sites into smaller ones makes it much more difficult to deal with troublesome individuals. Right now, it only takes a few bans before you're blocked from most mainstream sites. If we had, say, 10× the number of sites, that's 10× the number of bans required to get these individuals out of the system and 10× the number of moderators who had to look at their content before banning them.
I'm not saying there's no solution here, but I think people are misidentifying the problem. We all want simple answers and simple solutions even when there aren't any.
I’m not sure I follow. A site like Facebook already has mechanisms for detecting multiple accounts, and in any case, multiple accounts don’t mean that there are more “places” to post. It’s still Facebook.
I don’t propose breaking Facebook into smaller websites. I propose limiting the reach of a single individual on those sites. Basically, normalizing the localness of social media to be more meaningful.
As well, I think Facebook and other platforms need to reckon with the consent when it comes to posting images of people, in general. That’s why I suggested limiting the posts to only those of friends who consent.
I agree that no one wants this.
And again, people still post horrible content to small groups just like they post it in large ones. You've divided the problem up, but you haven't really solved anything. Someone has to moderate the content.
Cancer patients don't want chemo, but it's better than dying, some might say.
>Other sites like Hacker News, Reddit, youtube, and various other forums aren't even designed with the concept of friends that are the sole consumers of your content.
Right, which is why those sites (besides HackerNews) would require slightly different solutions.
Reddit: Limit subreddits to 500 participants at most. Eliminate the number next to upvotes. Verify accounts. Limit posting.
YouTube: Not sure. This one is video-forward, probably the most difficult problem in terms of bad content. Definitely remove the algorithm for targeting people based on interest, though.
>That's not a social media thing. That's just an internet thing where most things are visible to everyone.
This is not any feature inherent to the Web, it's a function of sites that purposefully link together and allow people to rapidly post information. Pre-social media, to get your idea out you had to build a website. There was friction. The earl y web had little moderation because you really had to go searching for bad stuff.
>And again, people still post horrible content to small groups just like they post it in large ones. You've divided the problem up, but you haven't really solved anything. Someone has to moderate the content.
Dividing the problem up is a strategy that I propose lessens the impact to both the users (because content can't spread as fast) and the moderators (because there will be less content to moderate).
Elimination of advertisers and the implementation of cost (as a form of friction), I think, would also go a long way. Cost per post would be ideal.
What good is chemo if your patients refuse it?
> Reddit: Limit subreddits to 500 participants at most. Eliminate the number next to upvotes.
This doesn't stop people from posting disturbing content, and it destroys reddit's entire model. This solution doesn't work.
> Verify accounts.
In what way? Do you want to abolish anonymity? Because I can tell you right now that most of us aren't interested in such a solution.
> Limit posting.
In what way? Throttling posting speed? Sure if you just lessen the amount of content overall there will also be less disturbing content... but there's still disturbing content that needs to be moderated.
> This is not any feature inherent to the Web, it's a function of sites that purposefully link together and allow people to rapidly post information.
The default state of the internet is publicly viewable information. Thus it's an inherent feature that you can access almost anything out there.
> Pre-social media
As far as I'm aware, there was no such time. There have almost always been bulletin boards, forums, chat rooms, image/file sharing sites, and other forms of social media.
> there will be less content to moderate
Dividing up the content doesn't reduce the amount of it that needs to be moderated any more than cutting a cake reduces the amount of cake.
> Elimination of advertisers and the implementation of cost (as a form of friction), I think, would also go a long way. Cost per post would be ideal.
This just shuts people out according to income bracket. It reduces the total population, but I doubt it helps reduce the fraction of that population that are interested in posting and sharing disturbing content. It's just throwing the baby out with the bath water.
I don't envy the work that Dan and Scott have to do in the slightest, but I don't think they'll end up with PTSD from it. At least, that's what I gathered when I read "The Lonely Work of Moderating Hacker News"[0], especially this description of it: "Pressed to describe Hacker News, they do so by means of extravagant, sometimes tender metaphors: the site is a “social ecosystem,” a “hall of mirrors,” a “public park or garden,” a “fractal tree."
[0]http://archive.is/bbzan
have you ever seen foxes laughing? they sound like babies. without social media, i never would have known that, nor experienced one minute of happiness while watching it.
https://twitter.com/mollyfprince/status/1277601602866761729
I agree with that. But I also assert that people have an ethical obligation to opt-out of systems that cause harm when / if they are able, or at least advocate to change them.
I don't have the ability to change Facebook. But I have the ability to opt out of it, though my actions on the Internet, and the ability to advocate for others to do the same. So, here I am.
Not dumb, but I do think it's not thought-through. You're proposing a simple solution to what is a hugely complex problem, and throwing ML at it just isn't going to work. To get "plenty of training data," a human still has to classify all of that, leading to the problem of viewing that much unpalatable content by a human. You also have to train your network, which requires humans to verify the accuracy of training, hence viewing the content again.
If it were as easy as text-based spam filtering, this wouldn't even be a discussion.
How does it differentiate between normal ranting and hate speech? How is hate speech classified in the US vs Saudi Arabia? How does it tell the difference between someone asking a child innocent questions and asking them sexually-related questions on camera? Does the algorithm get trained to flag videos about depression that might lead to suicide, or does it say they're supporting getting help FOR depression and leave the video?
What about subjects that aren't already in the corpus of "flag these naughty things"? You still have to get a human to look at those; most likely a data scientist who knows what the algorithm is doing and what needs to be done to correct the training. Machine learning as-is will not get us there, so in the meantime the only option is moderation by people described in the article. It can be outsourced elsewhere, but it's just shifting the responsibility to a different subset of people.
That doesn't matter from a staff point of view though. You have a queue to work through. You'll be putting in an 8 hour day dealing with the stuff the system doesn't catch. The automation just means they don't need as much staff.
You're still going to get false positives and false negatives that need human review, and at a scale of Facebook, that's a lot of humans.
https://news.ycombinator.com/item?id=22487403
https://news.ycombinator.com/item?id=5068626
https://news.ycombinator.com/item?id=18742640
https://news.ycombinator.com/item?id=19065859
This quote from the article
“It gets to a point where you can eat your lunch while watching a video of someone dying. … But at the end of the day, you still have to be human.”
reminded me of a similar article published by WIRED six years ago [1].
Eight years after the fact, Jake Swearingen can still recall the video that made him quit. He was 24 years old and between jobs in the Bay Area when he got a gig as a moderator for a then-new startup called VideoEgg. Three days in, a video of an apparent beheading came across his queue.
“Oh fuck! I've got a beheading!” he blurted out. A slightly older colleague in a black hoodie casually turned around in his chair. “Oh,” he said, “which one?” At that moment Swearingen decided he did not want to become a connoisseur of beheading videos. “I didn't want to look back and say I became so blasé to watching people have these really horrible things happen to them that I'm ironic or jokey about it.”
[1] https://www.wired.com/2014/10/content-moderation/
Most of their workload is bullshit. People with colds and sore throats, depressing people using ER as primary care, assholes using 911 to score Medicaid cab vouchers.
And at any time, any number of people can show up with any kind of personal tragedies from stokes to various traumas. My sister quit after a 13 year old bled out from a gsw, and she walked out of the room and got kicked in the head by an prisoner who had been stabbed after he bit the ears off of three other prisoners, and broke the arm of a guard.
Bite a 2nd inmate's ear...shame on him.
Bite a 3rd inmate's ear...that's fricking badass.
That said, I would also wager that the curve can be skewed towards one side or the other depending on the location, timing of your shift, or any other factors.
Let’s say a content moderator got paid $200k per year, that doesn’t change the images they see. This isn’t about money as much as it’s about the requirements and preparation required for the job.
The root of the problem, which is rarely mentioned in these discussions, is that many people enjoying comfortable living conditions believe themselves to be “pure-blooded” and unable to deal with “unclean” things because of the possible “traumas”. (What is an ideal person free of “traumas”, anyway? A vegetable stoned on drugs?) You can easily see here the resemblance of a old time countess complaining about her “nerves” while her serfs do all the dirty work. Is there a natural law that says a poor grunt is bound to clean the junk from what you see on the website? Do it yourself.
The same happens with medical, law enforcement, and other professions. They are considered “special” jobs for “special” people (which is not true, the one who defines it like that is “special”). Then, any worries and doubts are neutralized by convenient “but it's their profession, they chose it, they are paid for it, etc.” This creates a stupid rift between “us” and “them”, and affects how both sides reason about themselves.
The resulting imbalance is routinely abused. Social networks are competing in presenting a rose-colored walled fantasy to their human cattle to increase yield — everyone can see they use “The Matrix” as a business plan. This creates a lot of shitty jobs where people waste their emotional capacity to make others pleased, not unlike in prostitution. On the other hand, media is hooking the softened consumer with the sensationalist outrage bait about maniacs, wars, corruption, presumed orgies, etc. To makes things worse, the limits to all of that are set not by some moral standards or concern about someone's well-being, as it is often implied, but by advertisement contracts. The bureaucratic machine decides what is “suitable” for the visitors. For example, this makes most of the talk around “Free The Nipple” moot: there is no undercover agents of patriarchy to battle, it's advertisers who bring money and judge what is unwanted “sexual content” (some might say it's a perfect example of capitalist oppression — cue “L'Internationale”).
Even this discussion is full of overreacting people. No, the video on your screen is not as “horrible, horrible” as experiencing something firsthand, nor the pain of the tortured can be compared to the pain of the random witnesses on the internet (if we don't count their own wild fantasies). If it happens, then it's no good to pretend it's not. Who knows, it may even make you think. All in all, humans don't show a lot of originality in what they produce, so the number of things you can expect to see is quite limited. I can only conclude by reiterating that knowledge is power, ignorance is bliss, and there's a lot of people who'd like to benefit from others' ignorance in many ways.
(Note how much the "thin blue line" imagery is used in conjunction with this - that's how you know who's the target audience.)
It all ultimately comes from Dave Grossman, who's one of the most worshipped figures in modern American policing: https://www.policeone.com/police-products/training-products/...
This guy now runs the "Grossman Academy", which offers police training. It's popular enough that they have train-your-entire-PD discounts.
Oh I'm sure FB could think of a few ways.
A decent compensation structure might be a good place to start.
I still maintain expecting companies to do this on their own is a category error.
Companies think moderation is a cost center. Even if the moderators do the best possible job, they aren't adding value, only removing liabilities. The company is the wrong entity we should be asking for solutions to this problem. They are going to minimize costs and shift liabilities to 3rd parties (which will in turn hire independent contractors and play shell games in case they are sued).
We have to demand adversarial outside actors (government, labor unions, lawyers, non-profits, mental health professionals, etc) to start shaping the direction of the solution.
Modest proposal: Find people who enjoy horrific content, and have them rate it. Then black-hole whatever they rate as most extreme.
I was is a dark place after 9/11, both from the trauma of that, and some personal issues. So I spent considerable time on Rotten, Something Awful, and worse. Based on that experience, many would likely do content moderation for free.
Thanks for sharing that part about being in that kind of place, I have a feeling many could relate.
These media stories always focus on those who picked the wrong job for their caracter, and then make a big drama of it.
I mean, there are people cutting open other humans daily. We call them surgeons. I would puke right there and could never do that. Different people have different abilities. Simple.
Yup - these are exactly the kind of people you want to rub shoulders, and stock your company's ranks with.
I also get the downside of employing them. But people need to look at that stuff, even if software does most of the work. So it might as well be people who can tolerate it, and perhaps even enjoy it.
If you don’t recognize it, “rest of world” is a ridiculous corporate term commonly used in global business operations. It’s a catch-all phrase that means, basically, “everyone else.” And it generally represents billions of people outside of the Western world. We know that their stories matter. The term “rest of world” is a symptom of a larger problem: a Western-centric worldview that leaves innumerable insights, opportunities and complexity out of the conversation.