That makes sense. $1 is 40 Indian Rupees, and $50 (the prize money) is 2339 Indian Rupees at current rates. Neither amount is an insignificant amount of money in India.
Yes. Neither amount is insignificant. But I still wonder, why only India?
People from Pakistan, China and even Sri Lanka , Singapore are in +/- 3 hrs. timezones.
The timing of the post would be crucial to understand this.
FYI, $1 is insignificant in Singapore. Average household monthly income probably around 4000 USD (personal estimate, I live there, didn't look for official data)
It shouldn't just irk you as an Indian, it irks me as a dutch person as well.
If a service finds people from a certain country good enough to be 'minions' how come it finds them not good enough to be 'bosses' ?
This is actually a really nice opportunity to give amazon a run for their money and to set up a competing service that operates world-wide on both the give and the take side.
There are employment laws in the UK which mean a Mechanical Turk-like system wouldn't be able to have jobs listed for such low amounts - I think it comes under the minimum wage laws.
Because of this and similar laws in other countries, Amazon have kept job-posting MT accounts limited to the USA at the moment, which doesn't have the restriction.
I don't know enough about Dutch or Indian law to know whether a similar system could be set up within them, but there are problems with rolling something like this out globally.
Minimum wage laws only apply to a situation where there is an employer-employee relationship between the two parties, this is much more like a free-lance situation.
Andy Baio's project took place over the course of 3 days; it's not explicitly stated how long this task took to complete though given how high the pay was (relatively) I could see this job filling up in an hour.
(Edit: or 3 hours - pay seems to be a large factor, I wonder if by paying less one gets more people who do the task out of boredom rather than necessity and how that correlates to quality of results.)
It could have something to do with the time the job was created. As the author mentioned in a comment, after 3 hours the job kinda died. So If he posted it in peak hours for India, than it's not surprising that most of the respondents were Indian, while the fellow from your link may have posted it in peek hours for a more "white" timezone.
What we really need I supposed is one of these every hour for 24 hours. Then we'll get a better idea of the demographics of MTurk.
Yeah, it is. There were a lot of bad submissions. I'm just experimenting with Mechanical Turk right now. Initial impressions. . . 1) a lot of fraud (about half sent in pictures of their cat or something) 2) most people are from india 3) Most of your replies come in the first 3 hours or so. After that it wains. You have to resubmit after this 4) There were repeat submissions. Three of them looked like they are from the same room, same camera. But people claimed different locations.
Mechnical Turk is kind of interesting. It's one of those things that I'd really like try out, but have no immediate use for. I'm sure I'm not the only one. (In fact, most of the cool projects I've seen using MTurk are of the Gee Whiz variety)
Mechanical Turk is a fascinating tool, maybe this is not the most interesting use, but it still make hacker drool. When you can't have AI, you use AAI (Artificial AI).
You can use it to train pattern recognition. It sure beats doing the work myself.
I think he paid them all $1 for the task, with the added bonus of possibly getting $50 if they win. Why is that not nice? They got paid, and entered a contest. $1 for a HIT on Mechanical Turk is pretty good, especially considering the amount of work they needed to do here (which is not much).
$1 seems to be pretty great for this task.
I just read some HIT descriptions, and there were A WHOLE LOT of "write 200 words for $0.03" jobs. Which really sucks if you ask me.
The funny thing is, every single one of those participants thought it was worth their time. They looked for something to earn money, and found this. They clearly thought the time/money tradeoff was sufficient.
If you can earn a buck doing what a lot of (generally wealthier, admittedly) people do for free, why not?
Cool experiment. Thinking of using Turk for photo approvals on my site, but the accuracy issue leaves me slightly hesitant. Has anyone had experience doing this?
From what I read the standard technique seems to be let two people review everything. When there's a discrepancy you can add more manpower (and take the majority view or so) or look at it yourself.
What I did was to have up to five person work on a hit. Many simple hits are done at 0.05$ so it's still not too expansive.
For text submissions, you can enforce validation rules (simple and regex). So you don't have to manually reject non-conforming work. You can also have the Turker fill in their work on a webpage that you control, so validation can be much more advanced. I never went all the way but sometimes I think about it.
Hey ashishk, sorry for using HN to contact you, but I can't seem to find your contact info anywhere.
Our company provides very affordable and accurate photo approval services. Photos are manually approved and we can handle anywhere from 10-1,000,000 photos per day. If you're interested, feel free to email me (email address is in my profile).
This makes me curious about Amazon's vetting process for requests.... taken one step further, I can imagine all sorts of ridiculous things to ask people to do.
What if somebody asked people to write something profane? Or political? Or perhaps deface the image of an icon or religious leader? Once you move past written messages, you open up all sorts of avenues. Crowd-sourced politics? Crowd-sourced porn? Crowd-sourced violence? Etc. At what point does Amazon need to filter these requests?
When it's illegal? There's a huge difference between crowd sourcing violence and writing something political (which a lot of people already get payed to do), ones organized crime the other is not.
I don't see any problem as long as they're not asking them to break any laws. Ultimately it's up to the parties involved to obey the law though. If someone puts an ad for a hitman in the help wanted section of the newspaper, should the newspaper be held responsible?
While I don't doubt that everyone has their price... I think you would find it cost-prohibitive to pay enough people to do something like that to have any sort of real-world impact.
People are not going to go out of their way, or risk jail time, for $1.00/task. You'd need to pony up a lot more money (like, I'm guessing, 100's of thousands of dollars) to get enough people to violate basic principles on a large enough scale for this to create some sort of shift.
The one fascinating angle I could think of is crowdsourcing something that is harmless and legal if done once in isolation but becomes harmful and dangerous when done on a huge scale. My brain is totally failing me here on a coming up with a good example but it seems conceivable (and diabolical!)
One papercut is a nuisance 100 million is another thing altogether.
Something with stock manipulation could work. Buy a ton of options in some company, like Palm for instance. Then on the release of some hot new gadget, like the Pre, pay a ton of people to wait outside the doors of the vendors. Initial buzz will be that Palm is going to strike it big. Cash in.
However, there would be a record of the entire thing in Amazon's system, not to mention that a huge audience of potential Mechanical Turk workers would see the offer - it seems likely that at least one would be suspicious and try to notify an authority.
What if they told people to stand in line at the store next to Palm? Since there's a finite amount of space, they'll naturally overflow. To an outside observer, it looks like all these folks are standing in line for Palm - particularly if you do the same thing at 1000 locations nationwide, and let journalists connect the dots.
It's also far less traceable. Instead of paying 50,000 people to stand in line outside of Palm, you're paying 50 people each to stand in line outside of 1000 different local businesses, whose only connection is that they happen to be next to a Palm store. A just-as-plausible explanation is that you're organizing a "buy local!" protest that day.
You could also pay people on Amazon Mechanical Turk to pay turkers to stand in line, making things even harder to track. Now instead of one guy paying 50 people each to stand in front of 1000 different locations, you have 50 guys, each one of which is paying people to stand outside Subway, or McDonalds, or Verizon, or the corner bookstore, or whatever happens to be next to that Palm store.
If it were in a movie you'd say 'no way', but to see it in real life... I'd hate to be that guard, being pepper-sprayed is not nice, but otherwise this 'heist' has me with ambiguous feelings towards the robber.
That naively assumes that you jump up then fall. They could simply climb up steps and drop down, thus resulting in a net movement. Over time we would move into moon collision orbit.
If they were to do so, beyond merely making people agree to terms of service that prohibit such things, they would very likely open themselves up to a lot more legal liability. If they did claim to review and filter projects, and they missed one, they could then be liable for whatever horrible things happened as a result of someone performing the task. I'm pretty sure being a neutral exchange, where both buyer and seller beware (though each has processes for remedying lack of performance or lack of payment), is a safer model for Amazon.
On top of the seeming additional liability Amazon would be inflicting on themselves, I can't see any upside to Amazon going to extra lengths. It would anger project creators to have to wait for human approval for every project, it would likely reduce the number of jobs underway at any given time (making it less profitable for them and less lucrative a market for service providers), and it would increase the expense of running the market at human scale rather than computer scale (even if they somehow figured out how to use Mechanical Turk itself to provide the vetting). In short, vetting jobs would be a recipe for a fiery death for Mechanical Turk.
There is a request up now looking for someone to take a picture of themselves dressed as a terrorist. Many others ask people to fill out a quick survey or enter in an email for lead generation/reward websites.
Would you do that for a dollar? I might if I knew where I had a spare piece of paper and a Sharpie, but I don't.
The payment for most of the tasks on Mechanical Turk and other things like it is just not worth the time for almost any task I've seen. Which is really sad, because it'd be awesome if it were worth it.
I would if I just happened to come across the posting and just happened to have paper and a sharpie and a digital camera next to me.
It's interesting to read all the comments that say "$1?!?! You're ripping off these people! Why would anyone ever do anything on Mechanical Turk?" How many hours do y'all spend on Hacker News? How long did it take you to write those comments? How much were you paid for it?
In strictly economic terms, Turkers are getting ripped off a lot less than news.YCers are. After all, $1 is infinitely more than $0. We labor for free so that Paul Graham can have a viable online community and pick of the top posters for his seed capital firm.
This isn't really labor, first of all. For most of us it's both a leisure and educational activity that we participate in not for PG's benefit, but for our own benefit by learning and socialization. Making a sign that says "Hi Lili" and taking a picture of it is much less fruitful, even if it nets me $1 in money. The things I learn on HN net me much more money in the long run.
Except perhaps the ad-hoc (non-rigorous) studies that have been done before asking people why they turk. Someone mentioned last years submission of such a study further up the thread.
Of course you may be being philosophical about it. In which case, no we have no real way of knowing if they even think.
The taking a picture task to could be though of as fun and something different compared to a lot of the jobs on there. So in this case someones motivation might not having anything to do with the dollar on offer.
I hope I'm not sounding offensive. But what is the author trying to prove here? It's a well known fact that majority of the people on mechanical turk come from developing / under developed countries.
No one has accused you of being on the offensive side, I don't think, and I wasn't arguing with you. I just didn't see any indication that this project had much to do with the location of participants. So, I just wasn't sure what you were getting at with your "But what is the author trying to prove here?" query, since it seemed to me that the guy was just trying out Mechanical Turk and documenting the results, without any social or political commentary that I could perceive. It doesn't seem like he was trying to prove anything.
I'm surprised that this made it to the top of HN. I did something similar for my friend's birthday last year, but I never considered it particularly newsworthy. Another fun project was proving to our product manager that he was completely wrong (about listing overdue timesheet dates in reverse chronological order) by submitting the question to Mechanical Turk and getting instant user feedback.
Does the popularity of this link mean that there is enough interest in Mechanical Turk to justify the creation of web apps around it?
This is similar to a short survey I ran the other day: I wanted some stats about how many people used wedding gift registries, so I put an ad on Facebook (targeted at married females in NZ) that said "Answer 4 questions about your wedding and be in to win $50".
70% of people who clicked did the survey, which is a really high "conversion" rate -- I suspect the $50 prize draw really helps. I spent about $0.25 per click, so all up 200 survey results cost me about $120.
Yes -- though not yet, because they (weirdly) haven't replied to my "you've won the $50 survey prize" email. Either they thought it was spam, or they typed in their email address wrong. I'm guessing the later, and will move onto the next randomly-selected participant in a couple of days.
Facebook used to have a really useful (and inexpensive) Polls feature that allowed you to get something like 1000 or 5000 responses from your demographic to a multiple choice question. They pulled it for some reason though.
http://www.washingtonpost.com/wp-dyn/content/article/2009/01...
The disproporion is caused by an obvious fact that women usually study stuff like political studies, international studies, women studies, literature and such. Anything hard like engineering or CS, law, economics and you will see very few women in the classroom.
122 comments
[ 3.0 ms ] story [ 186 ms ] threadhttp://www.thestever.net/viewer.php?num=33
http://www.thestever.net/viewer.php?num=34
http://www.thestever.net/viewer.php?num=35
http://www.thestever.net/viewer.php?num=10 http://www.thestever.net/viewer.php?num=14 http://www.thestever.net/viewer.php?num=45 http://www.thestever.net/viewer.php?num=42 http://www.thestever.net/viewer.php?num=41 http://www.thestever.net/viewer.php?num=32 http://www.thestever.net/viewer.php?num=31 http://www.thestever.net/viewer.php?num=30 http://www.thestever.net/viewer.php?num=29
If you're getting $1 for each picture of a person holding a sign you submit, you might as well have everyone else in the room take a turn.
Yes. Neither amount is insignificant. But I still wonder, why only India? People from Pakistan, China and even Sri Lanka , Singapore are in +/- 3 hrs. timezones.
The timing of the post would be crucial to understand this.
I wanted to highlight - the time as a factor for other countries. Timezone.
If a service finds people from a certain country good enough to be 'minions' how come it finds them not good enough to be 'bosses' ?
This is actually a really nice opportunity to give amazon a run for their money and to set up a competing service that operates world-wide on both the give and the take side.
Because of this and similar laws in other countries, Amazon have kept job-posting MT accounts limited to the USA at the moment, which doesn't have the restriction.
I don't know enough about Dutch or Indian law to know whether a similar system could be set up within them, but there are problems with rolling something like this out globally.
http://waxy.org/2008/11/the_faces_of_mechanical_turk/
Interesting to see how the demographic has shifted in that time period.
Regarding the bit about crowd-sourcing ideas; Andrew Hyde used Turk to solicit startup-ideas: http://andrewhy.de/using-mechanical-turk-for-ideas/
(Edit: or 3 hours - pay seems to be a large factor, I wonder if by paying less one gets more people who do the task out of boredom rather than necessity and how that correlates to quality of results.)
What we really need I supposed is one of these every hour for 24 hours. Then we'll get a better idea of the demographics of MTurk.
wanes
Can someone please tell me why this submission is interesting to HN? (Obviously it is, I just can't figure out why.)
Also, people like to look at pictures.
Or, try soliciting feedback from 100 people on the clarity/ease of your company's sign-up process, introductory video, etc.
Many of the comments will be worthless, but you'd be surprised at the quality you get from some people, even at rates as low as $0.05-0.10.
You can use it to train pattern recognition. It sure beats doing the work myself.
That's making other people pay with their time for your own lack of specificness.
Blows my mind how 4 out of 50 (8%) did not even complete the stated requirements.
Also crazy how all many of these people are Indian.
Don't be surprised when people don't take your requirement that seriously.
If you can earn a buck doing what a lot of (generally wealthier, admittedly) people do for free, why not?
For text submissions, you can enforce validation rules (simple and regex). So you don't have to manually reject non-conforming work. You can also have the Turker fill in their work on a webpage that you control, so validation can be much more advanced. I never went all the way but sometimes I think about it.
Our company provides very affordable and accurate photo approval services. Photos are manually approved and we can handle anywhere from 10-1,000,000 photos per day. If you're interested, feel free to email me (email address is in my profile).
I am writing her now.
After all, for someone to get anywhere doing ill with MTurk the request must be seen by people.
I do think it is very interesting to see the faces behind Mechanical Turk.
So?
People are not going to go out of their way, or risk jail time, for $1.00/task. You'd need to pony up a lot more money (like, I'm guessing, 100's of thousands of dollars) to get enough people to violate basic principles on a large enough scale for this to create some sort of shift.
One papercut is a nuisance 100 million is another thing altogether.
It's also far less traceable. Instead of paying 50,000 people to stand in line outside of Palm, you're paying 50 people each to stand in line outside of 1000 different local businesses, whose only connection is that they happen to be next to a Palm store. A just-as-plausible explanation is that you're organizing a "buy local!" protest that day.
You could also pay people on Amazon Mechanical Turk to pay turkers to stand in line, making things even harder to track. Now instead of one guy paying 50 people each to stand in front of 1000 different locations, you have 50 guys, each one of which is paying people to stand outside Subway, or McDonalds, or Verizon, or the corner bookstore, or whatever happens to be next to that Palm store.
http://news.cnet.com/bank-robber-hires-decoys-on-craigslist-...
Perhaps considerations of aiding and abetting prostitution, kidnapping and drug-dealing would enter if the job requests were entirely unvetted?
Uh, and I think India has labor laws too.
On top of the seeming additional liability Amazon would be inflicting on themselves, I can't see any upside to Amazon going to extra lengths. It would anger project creators to have to wait for human approval for every project, it would likely reduce the number of jobs underway at any given time (making it less profitable for them and less lucrative a market for service providers), and it would increase the expense of running the market at human scale rather than computer scale (even if they somehow figured out how to use Mechanical Turk itself to provide the vetting). In short, vetting jobs would be a recipe for a fiery death for Mechanical Turk.
The payment for most of the tasks on Mechanical Turk and other things like it is just not worth the time for almost any task I've seen. Which is really sad, because it'd be awesome if it were worth it.
While that may be true for you, there are plenty of places across the globe where $1 (with a chance of $50) is not an insignificant amount.
It's interesting to read all the comments that say "$1?!?! You're ripping off these people! Why would anyone ever do anything on Mechanical Turk?" How many hours do y'all spend on Hacker News? How long did it take you to write those comments? How much were you paid for it?
In strictly economic terms, Turkers are getting ripped off a lot less than news.YCers are. After all, $1 is infinitely more than $0. We labor for free so that Paul Graham can have a viable online community and pick of the top posters for his seed capital firm.
Of course you may be being philosophical about it. In which case, no we have no real way of knowing if they even think.
I'm pretty sure that might be above the US minimum wage.
http://www.thestever.net/viewer.php?num=33 http://www.thestever.net/viewer.php?num=34 http://www.thestever.net/viewer.php?num=35
As a bonus, he got some nice smiles out of the process ^_^.
Does the popularity of this link mean that there is enough interest in Mechanical Turk to justify the creation of web apps around it?
http://www.thestever.net/viewer.php?num=27
70% of people who clicked did the survey, which is a really high "conversion" rate -- I suspect the $50 prize draw really helps. I spent about $0.25 per click, so all up 200 survey results cost me about $120.
Oh, maybe not...