You probably shouldn't incorperate a scientific study into your model of the world unless you've also memorized the methodology that it's attached to and relies on.
According to Duncan Watts in his book "Everything is Obvious, once you know it" it's quite widespread to use the Turkers for that kind of research.
It's described as a very cost-effective method compared to reaching out the subject directly and trying to convince them to gather somewhere for the study.
Also, it's described as game-changing when it comes to studies that must have a large base and made previously impossible(due to practical reasons) studies possible.
When I graduated from undergrad, Mechanical Turk was just starting to make inroads in social psych research. One of my friends was actually one of the first students to manage a survey for the social psych professor using MTurk. It's been promoted relatively heavily at conferences since.
It's been discussed here [0] as well, but it's important to highlight that prior to MTurk social psychology studies were (and actually probably still are to a great extent) widely susceptible to the "college sophomore problem" wherein a sample meant to provide conclusions that would generalize to the wider population of the human race was composed almost exclusively of college age children. This meant that a large number of conclusions in social psychology have historically been based on non-representative samples. Methodologically, MTurk can be seen as a major improvement over the status quo in this regard.
Data contamination has always been an issue with MTurk. Bots that submit junk data in hopes that you just auto approve the entire data set. You definitely need a secondary validating question to validate the turkers authenticity.
What does he mean by GPS? Is he just getting location info from the IP or using the browser geolocation API? I’ve run my fair share of surveys on mTurk and FigureEight. The best way to prevent scams are with honeypots. Always have a question or two deeper in the survey that seem quick and simple but require engagement. Be sure to change these questions often.
Yeah, it's standard practice for mturk experiments to have a catch trial or two and to exclude responses that answer incorrectly. All of the experiments I've seen in my field do this.
This emphasis on GPS is the strangest part to me. True GPS coordinates are not directly available, AFAIK, in most circumstances pertaining to mturk. Services might use IP address and other data to make a guess. Maybe those with more experience with geolocation can chime in.
Using MTurk for personal surveys and expecting to get good and meaningful results from it is just all kinds of wrong. I do hope there isn't "serious" research that's being done this way.
This article [1] was linked from a Wired article earlier that touched on the same question. To editorialize, undergraduate students are not considered in it to be a much better sample than properly run MTurkers.
You would be surprised how many serious studies you will find mturk as a data source of you dig into them. It's usually buried and sometimes I think they likely omit that little fact all together.
The answer to so many similar classification questions is to apply the simplest of case-built classifiers to detect red flags and then write a regex to exclude new classes of bot data.
I’m the CTO at an MTurk competitor (Prolific.ac). We’ve written up some thoughts on this below. It’s unlikely that these accounts are ‘true’ bots but rather low effort workers hiding location with a VPN and providing pseudo random responses.
We’ve done a lot of work on improving data quality on Prolific and filtering out these kind of accounts and heavily invest in making sure our pool are honest, attentive and engaged.
If anyone’s concerned about data quality in online samples please get feel free to get in touch.
No. All questions are optional and typically studies only filter on 1-3.
We realise there’s a too many there by default and we’re working on staggering the demographics more.
Seconded. Have run thousands of participants over eight years. Ya there's crap, but that's true whenever you ask any random sample of people about anything.
have run many mturk studies. it amazes me how many researchers think you can get away with paying $0.50/hour. most mturkers take pride in their work just like anyone else.
I don't get it. Why are mechanical turk outcomes being founded upon political ideology as a concrete, objective data point?
There's something in this process of deriving authenticity that seems... flimsy?
I mean, substitute "Skeletor" and "Hordak" as stand-ins for "KKK" and "Nazi Party", and "Eternia" for "America".
When you make a video game out of clicking questionnaire responses, and seal it in an inconsequential vacuum, why would anyone expect conclusive results that "Skeletor" and "Hordak" should be regarded objectively as detestable? There's no real consequence to saying "blah", "yay" or "boo" and I don't think I'd expect success if I were to try and ground my hypothesis on simple, obtuse good/bad/indifferent political opinions, to prove whether response data is "authentic" or not.
This feels like a weaker conclusion that gauging lie detector readings.
You can safely ground response data on facts like:
The sky is blue.
Wood comes from plants.
Stars exist in outer space.
But, you can't really demand predictable results to bind on opinions.
Is Mr. Rogers nice?
Are leather jackets good?
Do old cars go fast?
Whether evil people started answering, or ambivalent people started shrugging at the questions doesn't provide a conclusive assessment of whether a system is producing useful data. I feel like this is a garbage-in-garbage-out scenario, no?
I don't get it. Why are mechanical turk outcomes being founded upon political ideology as a concrete, objective data point?
There's something in this process of deriving authenticity that seems... flimsy?
I mean, substitute "Skeletor" and "Hordak" as stand-ins for "KKK" and "Nazi Party", and "Eternia" for "America".
When you make a video game out of clicking questionnaire responses, and seal it in an inconsequential vacuum, why would anyone expect conclusive results that "Skeletor" and "Hordak" should be regarded objectively as detestable? There's no real consequence to saying "blah", "yay" or "boo" and I don't think I'd expect success if I were to try and ground my hypothesis on simple, obtuse good/bad/indifferent political opinions, to prove whether response data is "authentic" or not.
This feels like a weaker conclusion than gauging lie detector readings.
You can safely ground response data on facts like:
The sky is blue.
Wood comes from plants.
Stars exist in outer space.
But, you can't really demand predictable results to bind on opinions.
Is Mr. Rogers nice?
Are leather jackets good?
Do old cars go fast?
Whether evil people started answering, or ambivalent people started shrugging at the questions doesn't provide a conclusive assessment of whether a system is producing useful data. I feel like this is a garbage-in-garbage-out scenario, no?
30 comments
[ 2.8 ms ] story [ 76.6 ms ] threadIt's described as a very cost-effective method compared to reaching out the subject directly and trying to convince them to gather somewhere for the study.
Also, it's described as game-changing when it comes to studies that must have a large base and made previously impossible(due to practical reasons) studies possible.
It's been discussed here [0] as well, but it's important to highlight that prior to MTurk social psychology studies were (and actually probably still are to a great extent) widely susceptible to the "college sophomore problem" wherein a sample meant to provide conclusions that would generalize to the wider population of the human race was composed almost exclusively of college age children. This meant that a large number of conclusions in social psychology have historically been based on non-representative samples. Methodologically, MTurk can be seen as a major improvement over the status quo in this regard.
[0] https://news.ycombinator.com/item?id=17789595
[1] http://journal.sjdm.org/10/10630a/jdm10630a.html
https://en.wikipedia.org/wiki/K-means_clustering
https://en.wikipedia.org/wiki/Naive_Bayes_classifier
https://blog.prolific.ac/bots-and-data-quality-on-crowdsourc... https://blog.prolific.ac/how-to-improve-your-data-quality/
We’ve done a lot of work on improving data quality on Prolific and filtering out these kind of accounts and heavily invest in making sure our pool are honest, attentive and engaged.
If anyone’s concerned about data quality in online samples please get feel free to get in touch.
If anyone is having trouble, feel free to reach out.
Source: Working with MTurk since 2013, including a 3 month internship in the belly of the beast. Starting a related PhD this fall.
Are MTurkers a random sample?
There's something in this process of deriving authenticity that seems... flimsy?
I mean, substitute "Skeletor" and "Hordak" as stand-ins for "KKK" and "Nazi Party", and "Eternia" for "America".
When you make a video game out of clicking questionnaire responses, and seal it in an inconsequential vacuum, why would anyone expect conclusive results that "Skeletor" and "Hordak" should be regarded objectively as detestable? There's no real consequence to saying "blah", "yay" or "boo" and I don't think I'd expect success if I were to try and ground my hypothesis on simple, obtuse good/bad/indifferent political opinions, to prove whether response data is "authentic" or not.
This feels like a weaker conclusion that gauging lie detector readings.
You can safely ground response data on facts like:
But, you can't really demand predictable results to bind on opinions. Whether evil people started answering, or ambivalent people started shrugging at the questions doesn't provide a conclusive assessment of whether a system is producing useful data. I feel like this is a garbage-in-garbage-out scenario, no?There's something in this process of deriving authenticity that seems... flimsy?
I mean, substitute "Skeletor" and "Hordak" as stand-ins for "KKK" and "Nazi Party", and "Eternia" for "America".
When you make a video game out of clicking questionnaire responses, and seal it in an inconsequential vacuum, why would anyone expect conclusive results that "Skeletor" and "Hordak" should be regarded objectively as detestable? There's no real consequence to saying "blah", "yay" or "boo" and I don't think I'd expect success if I were to try and ground my hypothesis on simple, obtuse good/bad/indifferent political opinions, to prove whether response data is "authentic" or not.
This feels like a weaker conclusion than gauging lie detector readings.
You can safely ground response data on facts like:
But, you can't really demand predictable results to bind on opinions. Whether evil people started answering, or ambivalent people started shrugging at the questions doesn't provide a conclusive assessment of whether a system is producing useful data. I feel like this is a garbage-in-garbage-out scenario, no?