The title is biased, blaming Google for mistreating people and implying that Google's AI isn't smart, but the OP is worth reading, because it gives readers a sense of the labor and cost involved in providing AI models with human feedback, the HF in RLHF, to ensure they behave in ways acceptable to human beings, more aligned with human expectations, values, and preferences.
The title seems kinda misleading, this is from the article (GlobalLogic is the company contracted by Google):
"AI raters at GlobalLogic are paid more than their data-labeling counterparts in Africa and South America, with wages starting at $16 an hour for generalist raters and $21 an hour for super raters, according to workers. Some are simply thankful to have a gig as the US job market sours, but others say that trying to make Google’s AI products better has come at a personal cost."
"Google" posted a job opening. They applied for and took the job, agreeing to posted pay and conditions. End of the story. It's not up to the Guardian to decide
"Google said in a statement: “Quality raters are employed by our suppliers and are temporarily assigned to provide external feedback on our products. Their ratings are one of many aggregated data points that help us measure how well our systems are working, but do not directly impact our algorithms or models.” GlobalLogic declined to comment for this story." (emphasis mine)
How is this not a straight up lie? For this to be true they would have to throw away labeled training data.
When they switch to aligning with algorithms instead of humans we'll get another story about how terrible it was that they removed the jobs that were terrible when they existed.
This doesn't sound as bad to me as the Facebook moderator job or even a call center job, but it does sound pretty tedious.
Something I'd be interested to understand is how widespread this practice is. Are all of the LLMs trained using human labor that is sometimes exposed to extreme content?
There are a whole lot of organizations training competent LLMs these days in addition to the big three (OpenAI, Google, Anthropic).
What about Mistral and Moonshot and Qwen and DeepSeek and Meta and Microsoft (Phi) and Hugging Face and Ai2 and MBZUAI? Do they all have their own (potentially outsourced) teams of human labelers?
I always look out for notes about this in model cards and papers but it's pretty rare to see any transparency about how this is done.
From my shallow understanding, it seems that human training is involved heavily in the post-training/fine-tuning stage, after the base model has been solidified already.
In that case, how is the notion of truthiness (what the model accepts as right or wrong) affected during this stage , that is affected by human beings vs. it being sealed into the basic model itself, that is truthiness being deduced by the method / part of its world model.
Congratulations, you just described most jobs. And many backbreaking laborers make about the same or less, even in the U.S., not to mention the rest of the world.
I'm a contractor for one of these companies. It pays okay ($45+/hour) if you can pass qualifications for your area of expertise but the work isn't steady and communication is non-existent. The coding qualifications I did were difficult FAANG algorithm analysis questions. The work has definitely gotten harder over the last year and often says we need to come up with Masters/PhD level work or problems that someone with 5+ years of experience in a field would have difficulty solving. I wish I had a regular job but I live in rural North Carolina and remote work is hard to come by.
Their work doesn’t seem that bad. This article tries really hard to portray that a simple freelance desk job is somehow literally exploitation or something.
Lots of people would do anything to get such work.
> “At first they told [me]: ‘Don’t worry about time – it’s quality versus quantity,’” she said.
> But before long, she was pulled up for taking too much time to complete her tasks. “I was trying to get things right and really understand and learn it, [but] was getting hounded by leaders [asking], ‘Why aren’t you getting this done? You’ve been working on this for an hour.’”
And:
> Dinika said he’s seen this pattern time and again where safety is only prioritized until it slows the race for market dominance. Human workers are often left to clean up the mess after a half-finished system is released. “Speed eclipses ethics,” he said. “The AI safety promise collapses the moment safety threatens profit.”
Finally:
> One work day, her task was to enter details on chemotherapy options for bladder cancer, which haunted her because she wasn’t an expert on the subject.
It's strange that the Guardian mentions OpenAI's "O3" model and not GPT-5.
Maybe they think o3 is SOTA still, but they should at least name it correctly, in lowercase as OpenAI does.
Diminishing returns is an ugly business. And thats obviously where we are at. The end not the beginning of LLM "innovation".
Any technology that creates "sysiphian" tasks, is not worth anyones time. That includes LLMs, and "Big Data". The "herculean effort" that never ends is the proof in the pudding. The tech doesnt work.
Its like using machine learning for self driving instead of having an actual working algorythm. Your bust.
29 comments
[ 2.6 ms ] story [ 47.2 ms ] thread"AI raters at GlobalLogic are paid more than their data-labeling counterparts in Africa and South America, with wages starting at $16 an hour for generalist raters and $21 an hour for super raters, according to workers. Some are simply thankful to have a gig as the US job market sours, but others say that trying to make Google’s AI products better has come at a personal cost."
How this industry managed to not grasp that meaning exists entirely separate from words is altogether bizarre.
How is this not a straight up lie? For this to be true they would have to throw away labeled training data.
This doesn't sound as bad to me as the Facebook moderator job or even a call center job, but it does sound pretty tedious.
That’s sort of what I expect the Guardian’s UK online non-sub readers to make.
Perhaps GlobalLogic should open a subsidiary in the UK?
There are a whole lot of organizations training competent LLMs these days in addition to the big three (OpenAI, Google, Anthropic).
What about Mistral and Moonshot and Qwen and DeepSeek and Meta and Microsoft (Phi) and Hugging Face and Ai2 and MBZUAI? Do they all have their own (potentially outsourced) teams of human labelers?
I always look out for notes about this in model cards and papers but it's pretty rare to see any transparency about how this is done.
[1] https://www.mturk.com/
[2] https://tinyurl.com/4r2p39v3
In that case, how is the notion of truthiness (what the model accepts as right or wrong) affected during this stage , that is affected by human beings vs. it being sealed into the basic model itself, that is truthiness being deduced by the method / part of its world model.
Congratulations, you just described most jobs. And many backbreaking laborers make about the same or less, even in the U.S., not to mention the rest of the world.
Lots of people would do anything to get such work.
> “At first they told [me]: ‘Don’t worry about time – it’s quality versus quantity,’” she said.
> But before long, she was pulled up for taking too much time to complete her tasks. “I was trying to get things right and really understand and learn it, [but] was getting hounded by leaders [asking], ‘Why aren’t you getting this done? You’ve been working on this for an hour.’”
And:
> Dinika said he’s seen this pattern time and again where safety is only prioritized until it slows the race for market dominance. Human workers are often left to clean up the mess after a half-finished system is released. “Speed eclipses ethics,” he said. “The AI safety promise collapses the moment safety threatens profit.”
Finally:
> One work day, her task was to enter details on chemotherapy options for bladder cancer, which haunted her because she wasn’t an expert on the subject.
RLHF providers:
1. Surge. $1b+ revenue bootstrapped. DataAnnotation is the worker-side (you might've seen their ads), also TaskUp and Gethybrid.
2. Scale. The most well known. Remotasks and Outlier are the worker-side
3. Invisible. Started as a kind of managed VA service.
4. Mercor. Started mostly as a way to hire remote devs I think.
5. Handshake AI. Handshake is a college hiring network. This is a spinout
6. Pareto
7. Prolific
8. Toloka
9. Turing
10. Sepal AI. The team is ex-Turing
11. Datacurve. Coding data.
12. Snorkel. Started as a software platform for data labeling. Offers some data as a service now.
13. Micro1. Also started as a way to hire remote contractor devs
[1]: https://x.com/chrisbarber/status/1965096585555272072
Any technology that creates "sysiphian" tasks, is not worth anyones time. That includes LLMs, and "Big Data". The "herculean effort" that never ends is the proof in the pudding. The tech doesnt work.
Its like using machine learning for self driving instead of having an actual working algorythm. Your bust.