I think its a brilliant concept. I think there is a huge market for mundane jobs and no specialized place to go for. Guru, elance, odesk are all for project work. Amazon turks is too spread out. They actually have found out a niche, now it is just going to be about implementing it. I will definitely be giving these guys a try.
CrowdFlower will price the task based on the amount of time it takes to perform the assignment per unit (the user does the sample task). The startup will break the task into units that can be performed by a single person and price the task accordingly.
How do they break tasks down into individual units? That seems like a tough thing to automate, so they may have a bottleneck there when they try to scale. On the other hand, they could have an upsell opportunity if they can sell people by handling more complex tasks that a "virtual project manager" would break down for them.
I dont think it will be too difficult to automate for mundane jobs. They will need a database to calibrate and then use that calibration to automate the process. There will always be deviations but the deviations can improve with more datapoints.
IMHO a lot of tasks that they talked about seem to be be repetitive, large-scale tasks that already can be broken into small pieces. For instance the sentiment analysis demo (each tweet is a task) or image moderation (each image is a task).
There was an explanation of this during the TC50 presentation and in the Q&A afterward.
From my simple understanding, this is a management application layered on top of the Mechanical Turk platform which is able to provide more robust/advanced management and analytics in terms of tracking the quality and speed of repeat turkers, as well as being able to integrate with other commoditized/outsourced labor systems. Due to being able to maintain more state concerning tasks and turkers, they are able to provide valuable intelligence about how trustworthy and accurate the results are.
It was very interesting during the Q&A when Bradley Horowitz asked the company rep if he thought that Amazon might simply do this themselves - but the presenter stated firmly that they had a pretty good personal relationship with the people at Amazon and they took pride on the turk service as really being a black box type of platform.
Regardless, the main problem this business faces (which was handily pointed out by the experts) is the issue of gaining traction - how do you make big businesses aware of this as a possible solution to their labor issues? Even at MySpace where they pay 300 people to moderate photos for obscenity they have not realized they could much more effectively and cheaply be leveraging MT for this task.
MySpace can't use MT for the task because every picture sent to MT would be a potential "OMG PORN ON MYSPACE FILM AT 11" story. They have to keep their screeners in-house to deal with potential publicity issues. Trust me, the MySpace people (as would any moderated community) would love to use MT for this stuff; moderation is no one's idea of a core competency.
Content moderation is one of the things that CrowdFlower does really well -- and yes, you can have workers search for adult content (At least on Mechanical Turk. I'm not sure about our other labor pools. In any case, you can definitely use CrowdFlower for filtering out adult content.)
You'll probably want to make sure your job description mentions that, though!
(I'm a front-end developer for CrowdFlower. I don't represent the company's views. Except in regards to bacon. I represent those.)
It's built on top of mturk - with tools for facilitating the use of mturk in conjunction with other worker networks to post actions/surveys, receive, organize & report on results.
Their quality control could eliminate the quasi-quality control that I now perform with MTurk. I set up a task like "get bio from webpage" and send it to 3 unique workers. I then run the results through a script to check whether they agree. If they do, accept all (pick random one). If they don't agree hand check (!) and possibly reject one of the jobs.
This is the sort of thing that CrowdFlower does automatically, except it takes the quality control aspect to a new level. It takes into account the history of the worker (were they accurate workers in the past? etc.) and also handles automatically rejecting workers who fail too many gold questions.
Speaking of, that's one of the cool bits about CrowdFlower, too, is that you can define "gold" questions. These are questions to which you know the answer. Workers will see these as if they were any other task. We then give them feedback about how well they're doing. If they keep missing the gold questions, then we mark all of their work for your job as "tainted" and their answers don't go into the aggregate calculations, which tell you what the users agreed on the most.
So, for instance, in a job where we pull in tweets about, say, #kanyewest, and ask workers to judge the sentiment of the tweet (positive, negative, etc.), we could take some obvious ones, like "Kanye West is a big jerk!" and mark it as gold with an answer of "Negative". Anyone who says "Positive" is probably yanking your chain. So we trust them less.
Of course, gold questions don't work too well for subjective questions, like writing short paragraphs.
I hope this gives you an idea of what CrowdFlower does in terms of Quality Control. I'm only the front-end guy, so I'm not too knowledgeable about all the math behind the scenes. All I know is that there's lots of it. :)
(I'm a front-end developer for CrowdFlower. I don't represent the company's views. Except in regards to bacon. I represent those.)
Sounds like a very good refinement of the reputation score in MTurk. For example, I usually only request workers with 95%+ acceptance rates.
The "Gold questions" sound like another refinement of the tests you can give Turks: if you have an image processing task, you can require that a worker pass a series of tests and effectively become certified before attempting your job. Are you guys trying to replace that system or complement it?
17 comments
[ 4.4 ms ] story [ 55.7 ms ] thread(I'm a front-end developer for CrowdFlower. I don't represent the company's views. Except in regards to bacon. I represent those.)
No I meant, I have him calling people & asking questions (& putting it into a spreadsheet) - Crowdflower can handle this, right?
How do they break tasks down into individual units? That seems like a tough thing to automate, so they may have a bottleneck there when they try to scale. On the other hand, they could have an upsell opportunity if they can sell people by handling more complex tasks that a "virtual project manager" would break down for them.
From my simple understanding, this is a management application layered on top of the Mechanical Turk platform which is able to provide more robust/advanced management and analytics in terms of tracking the quality and speed of repeat turkers, as well as being able to integrate with other commoditized/outsourced labor systems. Due to being able to maintain more state concerning tasks and turkers, they are able to provide valuable intelligence about how trustworthy and accurate the results are.
It was very interesting during the Q&A when Bradley Horowitz asked the company rep if he thought that Amazon might simply do this themselves - but the presenter stated firmly that they had a pretty good personal relationship with the people at Amazon and they took pride on the turk service as really being a black box type of platform.
Regardless, the main problem this business faces (which was handily pointed out by the experts) is the issue of gaining traction - how do you make big businesses aware of this as a possible solution to their labor issues? Even at MySpace where they pay 300 people to moderate photos for obscenity they have not realized they could much more effectively and cheaply be leveraging MT for this task.
As a side note, this sounds like a potential hole in the market.
You'll probably want to make sure your job description mentions that, though!
(I'm a front-end developer for CrowdFlower. I don't represent the company's views. Except in regards to bacon. I represent those.)
http://crowdflower.com/
Their quality control could eliminate the quasi-quality control that I now perform with MTurk. I set up a task like "get bio from webpage" and send it to 3 unique workers. I then run the results through a script to check whether they agree. If they do, accept all (pick random one). If they don't agree hand check (!) and possibly reject one of the jobs.
Speaking of, that's one of the cool bits about CrowdFlower, too, is that you can define "gold" questions. These are questions to which you know the answer. Workers will see these as if they were any other task. We then give them feedback about how well they're doing. If they keep missing the gold questions, then we mark all of their work for your job as "tainted" and their answers don't go into the aggregate calculations, which tell you what the users agreed on the most.
So, for instance, in a job where we pull in tweets about, say, #kanyewest, and ask workers to judge the sentiment of the tweet (positive, negative, etc.), we could take some obvious ones, like "Kanye West is a big jerk!" and mark it as gold with an answer of "Negative". Anyone who says "Positive" is probably yanking your chain. So we trust them less.
Of course, gold questions don't work too well for subjective questions, like writing short paragraphs.
I hope this gives you an idea of what CrowdFlower does in terms of Quality Control. I'm only the front-end guy, so I'm not too knowledgeable about all the math behind the scenes. All I know is that there's lots of it. :)
(I'm a front-end developer for CrowdFlower. I don't represent the company's views. Except in regards to bacon. I represent those.)
The "Gold questions" sound like another refinement of the tests you can give Turks: if you have an image processing task, you can require that a worker pass a series of tests and effectively become certified before attempting your job. Are you guys trying to replace that system or complement it?