As AMT's largest requester for the past 10 years, this news was relayed to requesters at the same time as respondents. It's also worth noting that our lead contact, the Sr Program Manager at AWS leading AMT, transitioned to Amazon Bedrock and SageMaker Model Evaluations a ~2-3 years ago.. Leaving behind essential zero team managing the project after the migrating over the stored value accounts to native AWS billing.
It’s kind of crazy they’re shutting this down just when this Service probably has the most possibilities ever. You have an agent with Multiple people doing actual physical tasks in the real world seems like something that could be really powerful.
The concept behind it isn't going away. There are plenty of companies that hire people in bulk to do manual data labeling, transcription, RLHF, moderation and lots more. It's just that they are now catering to large AI companies, not regular people looking to get some repeatitive work done (since that can now mostly be done by AI).
In spite of the name I think the overwhelming majority of the tasks were things that could be done by LLMs and they're probably getting flooded by people using bots to do exactly that. Even before the age of LLMs Mechanical Turk data was pretty bad because you'd have a bunch of people racing to answer questions as quickly as possible to get their $0.25 or whatever. So it was essentially a test of 'can you input random answers as quickly as possible while paying enough attention to notice the attention question that says to mark d.'
Any remotely open platform for this sort of stuff is going to wrecked by LLMs. But making some sort of high trust, high verification market is going to end up sending prices high enough that it becomes an unattractive proposition for use. And even in that case it's just going to be a cat and mouse game of people figuring out how to game the system enough to get trusted before handing it off to the LLM.
Seems like whole thing is destined to fail. Buyers of services are there to exploit and pay minimum they can get away with. And other side is ready to cheat if they can to get most out of it.
And same dynamic will apply to most cases. Unless you actually bring it in house with strict oversight. Which is thing to avoid originally...
There's a lot of platforms that are more actively developed and maintained, for example Prolific (which also offers discounts on service fees for academia).
Mechanical Turk had a good run, but not surprised it's shutting down. I'm sure the platform was getting flooded with people doing task arbitrage and using lots of AI anyway.
I believe the issue is that this can no longer be a horizontal play. MTurk was mostly for unskilled tasks...the kind AI can do well enough that it isn't worth the cost differential to verify it or keep farmed to humans. The "trust but verify" AI output is now the kind that requires domain expertise. This is what most full stack AI companies are bringing to industries.
Curious if this kind of work will come around again one day or was just a moment in time. If it does I'm sure it will be specifically about generating training data.
It is weird because the last time I've heard about MTurk was about developing countries being rather reliant on it for doing AI grunt work. If am not totally wrong this must mean that the data work has moved to other services.
I think the bigger issue is that a lot of the demand for labelling training data is now in highly-specialised fields (i.e. things like medical imaging), and mechanical turk's focus was on the generalist problems
I used it before AI coding and it was getting rough. Lots of US interviews with proxy Chinese or Pakistani workers. Lots of bullshit “I’ve done that; I can do this” and instantly apparent that this was untrue. Just the outright lies… whew.
I haven’t touched it since AI coding.
It’s a bad contractor market now. IDK what I would do if I needed a contractor.
A lot of what's been discussed in this thread is what we're tackling at Humwork (YC P26).
We're an MCP/API that connects AI agents to verified domain experts in real time (30s–3 min). Experts are vetted upfront by an AI interviewer that assesses and grades them, then get a mobile notification when a task matches their expertise and chat with the agent directly.
Soon we'll be verifying credentials for doctors, lawyers, CPAs etc on our platform for tasks where people are seeking credentialed experts to sign off and verify ai output.
To the best of my knowledge, that isn't true anymore, and nowadays you'd only get hired to do RLHF if you have particular skills beyond what can be achieved by just running other models against it.
MTurk likely can't fight with cheaters now. It would be useful to give some tasks to humans, if there were an economical way to ensure humans will do them.
I made a couple thousand dollars off it squeezing in tasks here and there between meetings. Amazon Payments were challenging to redeem as I recall. I did more or less write the bulk of someone’s doctoral thesis. Kept giving me a dollar to summarize the findings of various psychology papers. The most memorable was the one where people were put in a room and someone sprayed “liquid ass” on the wall. The subjects given no explanation experienced higher levels of anxiety than those that were told there was a sewage leak being repaired. One of the strangest dollars I ever made. My PS4 and game collection was spectacular.
So, I have a story to share about Mechanical Turk that you might find interesting. I’ve shared it a couple of times before on Twitter and Bluesky, but I’ll share it again.
Long story short: Mechanical Turk saved my bacon.
Back in 2005, I was working a job at a small-town newspaper in a town I’d never lived before. I didn’t know anyone beyond the staff (as I was working layout rather than as a reporter), and I had gotten interested in the idea of doing Mechanical Turk for a few extra bucks. I thought there might be a formative scene of interested people doing this, so I started working on a blog for it. I briefly collaborated on it with another guy who put it on forum software because he didn’t know how to use, like Drupal.
That blog was called Turking.com, and it seemed like it was going well for a bit. We even got a mention on the AWS website. But after about two or three months, it was clear the initial excitement around the idea (and the initial work) had died down. (The work picked up later, but in clearly different ways. I don’t think folks were really “excited” about it after that point.)
The site had started to die out in part because my iBook suffered a catastrophic GPU failure and me, being a broke small-town newspaper employee, did not have the money to replace it. Plus, the community just hadn’t emerged like we had expected.
But then I got an email out of the blue: Someone wanted to buy my domain, which I owned outright, but they didn’t want to say who. I got contacted by a broker, and the exchange took place over escrow.
(The domain most assuredly was bought by Amazon and is managed by MarkMonitor.)
I didn’t get a ton of money from the deal, but I did get enough to pay for a new laptop. I had some regrets about selling it (in part because I originally built the site with someone else), but I was in a bit of a dire financial situation which that proved to be the starting point for getting myself out of.
That domain purchase was notably more than I ever made from clicking and classifying random pictures, that said.
I explained the situation after the fact, and they were understanding. But certainly I admit that if I could do it again I probably would have clued them in sooner. The site had slowed down by the point this happened, so I’d describe it as a little more of a fire sale.
I learned a lot from that situation that I took to future sites.
> Long story short: Mechanical Turk saved my bacon.
Nothing wrong with your story, but that summary is terrible. You had a domain for a blog about a subject, and someone bought it later. That’s it. Mechanical Turk is inconsequential, as is the subject of the blog, the story would have been exactly the same if your blog had been about turkey sandwiches and Burger King offered to buy it.
I honestly don’t see what you don’t understand about “Nothing wrong with your story”. It’s literally the start of the first sentence. It seems pretty clear to me the mild criticism is directed solely at the summary (you know, the quoted part), not the story.
Ironic that you’re accusing someone else of “missing the point”, considering.
The point of the story was that I built a thing about Mechanical Turk to promote its money-making capabilities, and ironically made more money from it indirectly than I ever did directly. Considering that point was implied by my framing, you did in fact miss it.
I’m not saying that you have to like a story, but your framing felt unfair, which is why I responded that way.
Had some absolutely bizarre results from their attempt to integrate mturk with Bedrock's 'ground truth' thing as of a few months ago. Threw simple mnist digits recognition at it, see what the quality, timing, and cost was. Figured mnist digits was at this point trivial. Spent $10 and the accuracy was marginally better than guessing, completely unusable results. Was completely baffled, people were publishing peer reviewed research based on exclusively mturk results.
Surprised Amazon managed to fumble mechanical turk at the same time Mercor/Scale and all these other companies started hiring humans to do data labeling tasks.
I used it to have people transcribe my dad’s handwritten letters and journals. It was touching to get notes from a “Turk” saying how much she enjoyed his travels and following the cast of characters in his life!
The only time I ever used it (as a turk?) was when Jim Gray went missing at sea and satellite imagery of vast regions of the pacific were fed through Mechanical Turk. Seems a trivial problem now, but was not then, and needed humans to take a look.
Absent visibility into AWS P&Ls--even notwithstanding AI--Mechanical Turk has been a real outlier at AWS for a number of years now, so not really surprising.
I was just thinking about the good old days combining human judgements on mturk for classifying high-value forum threads. Since replaced by using judgements from multiple LLMs.
The only use I can think of it these days is for social scientists who need to gather judgements from bona fide humans.
This will be devastating for a lot of workers in low-infrastructure countries.
83 comments
[ 0.18 ms ] story [ 23.1 ms ] threadAny remotely open platform for this sort of stuff is going to wrecked by LLMs. But making some sort of high trust, high verification market is going to end up sending prices high enough that it becomes an unattractive proposition for use. And even in that case it's just going to be a cat and mouse game of people figuring out how to game the system enough to get trusted before handing it off to the LLM.
And same dynamic will apply to most cases. Unless you actually bring it in house with strict oversight. Which is thing to avoid originally...
I believe the issue is that this can no longer be a horizontal play. MTurk was mostly for unskilled tasks...the kind AI can do well enough that it isn't worth the cost differential to verify it or keep farmed to humans. The "trust but verify" AI output is now the kind that requires domain expertise. This is what most full stack AI companies are bringing to industries.
Curious if this kind of work will come around again one day or was just a moment in time. If it does I'm sure it will be specifically about generating training data.
Oh, I remember UpWork.
I haven’t touched it since AI coding.
It’s a bad contractor market now. IDK what I would do if I needed a contractor.
And everyone eager to explore new World, do sh¡t and learn along the way. Not compensate for IRL social isolation like nowadays.
We're an MCP/API that connects AI agents to verified domain experts in real time (30s–3 min). Experts are vetted upfront by an AI interviewer that assesses and grades them, then get a mobile notification when a task matches their expertise and chat with the agent directly.
Soon we'll be verifying credentials for doctors, lawyers, CPAs etc on our platform for tasks where people are seeking credentialed experts to sign off and verify ai output.
They are terribly monotonous tasks.
I can see why it's shutting down if its still the same thing.
Lllms could probably do everything there without rotting out minds for basically pennies.
MTurk likely can't fight with cheaters now. It would be useful to give some tasks to humans, if there were an economical way to ensure humans will do them.
Long story short: Mechanical Turk saved my bacon.
Back in 2005, I was working a job at a small-town newspaper in a town I’d never lived before. I didn’t know anyone beyond the staff (as I was working layout rather than as a reporter), and I had gotten interested in the idea of doing Mechanical Turk for a few extra bucks. I thought there might be a formative scene of interested people doing this, so I started working on a blog for it. I briefly collaborated on it with another guy who put it on forum software because he didn’t know how to use, like Drupal.
That blog was called Turking.com, and it seemed like it was going well for a bit. We even got a mention on the AWS website. But after about two or three months, it was clear the initial excitement around the idea (and the initial work) had died down. (The work picked up later, but in clearly different ways. I don’t think folks were really “excited” about it after that point.)
If you want to get an idea of it, there was one capture on the Wayback Machine: https://web.archive.org/web/20051124231722/http://www.turkin...
The site had started to die out in part because my iBook suffered a catastrophic GPU failure and me, being a broke small-town newspaper employee, did not have the money to replace it. Plus, the community just hadn’t emerged like we had expected.
But then I got an email out of the blue: Someone wanted to buy my domain, which I owned outright, but they didn’t want to say who. I got contacted by a broker, and the exchange took place over escrow.
(The domain most assuredly was bought by Amazon and is managed by MarkMonitor.)
I didn’t get a ton of money from the deal, but I did get enough to pay for a new laptop. I had some regrets about selling it (in part because I originally built the site with someone else), but I was in a bit of a dire financial situation which that proved to be the starting point for getting myself out of.
That domain purchase was notably more than I ever made from clicking and classifying random pictures, that said.
I learned a lot from that situation that I took to future sites.
Nothing wrong with your story, but that summary is terrible. You had a domain for a blog about a subject, and someone bought it later. That’s it. Mechanical Turk is inconsequential, as is the subject of the blog, the story would have been exactly the same if your blog had been about turkey sandwiches and Burger King offered to buy it.
Ironic that you’re accusing someone else of “missing the point”, considering.
I’m not saying that you have to like a story, but your framing felt unfair, which is why I responded that way.
It is written in German, which I know a little, but her handwriting was too difficult for me. So I searched around and found this site:
And it managed to extract the text! Anyway, it turns out the wheat harvest was very good in 1937 and thank you for the letters and newspapers.It appears now we can get along with just a single “artificial.”
The only use I can think of it these days is for social scientists who need to gather judgements from bona fide humans.
This will be devastating for a lot of workers in low-infrastructure countries.
https://arstechnica.com/information-technology/2017/11/expen...