Neat that you can look at and download the free projects.
One of the things I'd look for in a product like this is easily doing rater reliability & interrater reliability.
Optimally I'd love to see a project also allow for easy semi-supervised labeling. I don't see an API for grabbing data points for a model you are training to label and put them back.
Hey, Thanks for checking us out and the inputs. API access is getting ready to launch as I am typing this.
Agree that rater reliability is a very important thing which will matter here, ultimately it converts to dataset quality. Over the period of time, we would show dataset quality in some form using feedbacks from dataset users.
Hey, If its a free plan, data stays open to public. But for paid [for larger & enterprise datasets] plans, DataTurks doesn't have any ownership of data, and we don't keep any copies.
The "How does it work" section doesn't explain how it works.
> You and your team can now easily collaborate to build ML datasets super quick. Send email invite to anyone to help label your datasets, your team, friends, colleagues or external labelers. Pre-built support for more than a dozen data annotation use cases.
Do you guys know if there is any open-source project doing this kind of thing (UI for image labeling, NLP tagging, classification)? I think I've seen something like this before.
It's not terribly feature complete (and it's not stable), but it's functional and pretty fast for basic bounding box annotation. My aim was to minimise clicks compared tools like labelimg, and also to have a database that I could work on with changes being stored immediately. Other tools seem to forget where you left off. I also didn't want a dependency on Python or the cloud.
It runs on a sqlite database, so presumably in a shared environment you could actually use it real-time collaboratively. I was doing some labelling for a project with my girlfriend and we simply took turns with the db and images a shared folder in Dropbox.
Why do so few of these systems have integration with something like Mechanical Turk? Labeling your own data is a good way to get started, but often its a much better to just pay some people to annotate the data.
Hey, Thanks for checking us out, it makes a lot of sense to have integration with something like MTurk, so we are talking to some of such service providers, who provide expertise in kind of task we currently support. Very soon we will come with a good news in this area. :-)
Does anyone have a good tool for pixel-level annotation? I was thinking an iPad app with pencil support might be nice, but open to whatever is out there.
We used Sloth for segmentation labelling (polygons instead of pixels). Direct pixel labelling is very expensive, so usually you can save lots of time by simplifying the problem - one of them being polygon labelling.
Although Sloth is a gtk app it was the best option available year ago - much better than free web based options available back then. We packaged it into single windows exe that we were distributing to labellers in a zip alongside data to label. Labellers would send back zip + annotated data in json format custom to Sloth. Zips landed in s3 folder and we had some scripts doing analytics on that (e.g. Labellers reliability). It was very easy to extend and add new functions to sloth. Although it seems low tech comparing to web labelling it took us surprisingly lots of work to beat this workflow with web based purpose-built workflow.
17 comments
[ 2.9 ms ] story [ 34.1 ms ] threadOne of the things I'd look for in a product like this is easily doing rater reliability & interrater reliability.
Optimally I'd love to see a project also allow for easy semi-supervised labeling. I don't see an API for grabbing data points for a model you are training to label and put them back.
Do they keep copies?
That's important because in ML, data is everything.
> You and your team can now easily collaborate to build ML datasets super quick. Send email invite to anyone to help label your datasets, your team, friends, colleagues or external labelers. Pre-built support for more than a dozen data annotation use cases.
Here's one I wrote for my own use: https://github.com/jveitchmichaelis/deeplabel
It's not terribly feature complete (and it's not stable), but it's functional and pretty fast for basic bounding box annotation. My aim was to minimise clicks compared tools like labelimg, and also to have a database that I could work on with changes being stored immediately. Other tools seem to forget where you left off. I also didn't want a dependency on Python or the cloud.
It runs on a sqlite database, so presumably in a shared environment you could actually use it real-time collaboratively. I was doing some labelling for a project with my girlfriend and we simply took turns with the db and images a shared folder in Dropbox.
Yes Figure Eight (formerly CrowdFlower) does a ton of this kinda thing.
https://m.youtube.com/watch?v=wxi2dInWDnI
Disclosure: I work at Labelbox.
Although Sloth is a gtk app it was the best option available year ago - much better than free web based options available back then. We packaged it into single windows exe that we were distributing to labellers in a zip alongside data to label. Labellers would send back zip + annotated data in json format custom to Sloth. Zips landed in s3 folder and we had some scripts doing analytics on that (e.g. Labellers reliability). It was very easy to extend and add new functions to sloth. Although it seems low tech comparing to web labelling it took us surprisingly lots of work to beat this workflow with web based purpose-built workflow.