Thats cool, but still it needs a lot of work/features. I tried removing the background using the purse picture in the pictorial, and it wasn't as easy as it was made out to be and I wasn't able to achieve the same results. Maybe I'm missing something, but the fixed brush size makes it pretty useless at this point.
Also, one of our usability headaches is that people draw OUTSIDE of the lines - any pixel marked green WILL BE IN THE RESULT.
Now, that's not how people actually use it at first - they "just scribble" (which our example might exacerbate), and then accidentally go outside the actual lines => not good.
Will need to figure out a way to make that more clear.
I copied the marks in the example and besides needing to tell it to include the cream color under the strap, it worked exactly as shown on the homepage, so maybe just add that extra red there in the example.
Unfortunately it is freezing up on me using an image with a gradient at about 90% of the initialization.
Could be related to the image color profile. Noticed the same issue when I uploaded a jpeg image with a CMYK color profile, with RGB images it worked normally. I've had the same trouble when using Java's ImageIO to read images. There were some workarounds, but I didn't find a robust one. Ended up using a preprocessor to normalize images before reading them with Java. Would love to know if someone knows a better solution.
Tried removing a background from a stock car photo. It involved quite a bit of precise mouse wiggling and the end result was still pretty sloppy looking. I assume there are cases where this works well, but it would probably help clarifying what those are before inviting people to try the tool out.
Cars can be very tricky - the lines need to be very sharp, so there's no tolerance for blur. I used to mask and recolour car bodywork in Photoshop as a freelance job, and hand-drawn spline paths with the pen tool were the only acceptable solution for professional use.
Regarding the segmentation phase, we were not satisfied with any existing algorithms because we wanted the tool to be as real-time as possible (for up to 4 megapixel images), so we rolled our own segmentation algorithm that has some similarities to GrabCuts (not GraphCuts).
But the binary segmentation is really only a small part of the technical challenge. You also need to figure out how to anti-alias the boundary, and remove the background halo. Simply feathering doesn't do the job unless you also erode the foreground first, which we wanted to avoid. So we developed a custom sub-pixel background-removal and anti-aliasing algorithm.
I think you guys did a great job. This is a very tough problem (speaking from experience) particularly depending on the complexity of the image. One problem we had to solve along the way for our startup is similar, but focused only on fashion related images. Quite tricky particularly if you want something automatic. Anyway cheers! :)
Really neat. I was really itching to use CTRL+Z though.
I screwed up the first image I tried to clip. Maybe have a walkthrough of the features when the user first uses it? I didnt notice the help button until it was too late. I thought if I closed out of the editor and clicked on the image again, it would let me start over but it just takes you back to the editor. It would be nice if there was a way to redo editing the image from the start.
We only auto-show it during that time. The hope is that it triggers you to click the button if you need it, but lets you ignore it if you already know the drill...
(all these wonderful tradeoffs between "first visit" vs "repeat visit" - and what if it was missed the first time, blabla... )
Yeah, same thing for me. The action was too quick for me to notice that it was intentional. I would just leave the help window open on first time use until the user decides to close it. Very nicely done site though!
You can get a smaller brush by zooming in using the mouse wheel. That way the brush is always an appropriate size (small when you are zoomed in for detail work, and large when you are zoomed out for the broad strokes).
Just re-deployed the backend. It should reconnect automatically, but that's not been tested to 100%. Reloading the page should resolve it if it doesn't auto-recover.
(some memory woes on the backend as well, so restarting is having some hickups. Well, well)
We are having trouble connecting to the server. The following tasks have been unable to complete:
Connecting to worker "Unable to connect to the worker." Waiting"
I first thought that there is still the same problem but then I saw that the web socket worker is using 9006. Our company network is blocking unusual ports. Any chance to move that to 80, 443 or 8080?
Adding a "step-back" button (even if it was only a few states), and the ability to increase or decrease brush size would greatly enhance this. Great start though, and something like this is much needed for a lot of people who may not know an image editor, or don't want to open it for something simple.
I got a bunch of "Failed to get a worker assigned: undefined" for a bit, but once that stopped happening and I figured out the zoom, it was very impressive: http://i.imgur.com/WT4bH5M.jpg
Pros: Doesn't require the fg/bg coloring step needed here, just figures out your FG without intervention.
Cons: Have to register for an account to access it currently, but can use FB or Google to register
Pretty neat! http://imgur.com/p1B9Hng But sharing the shift key as the hotkey for erase & panning made Windows's "Sticky Keys" dialog appear a couple times.
Ah. Good point. We recently changed the shift key from "hold to pan" to toggle between pan / erase. Should probably change it to another key now that the operation is different.
I actually wrote my honours thesis on this topic. In academia it's called Alpha Matting/Image Matting.
There's a great website http://alphamatting.com/ that provides a benchmark dataset and a comparison of a lot of influential methods.
I applaud your work in making a real time and interactive matting algorithm! One day it might be also as accurate as state of the art methods. You guys should write a paper about your method.
Thanks for the link! Real time is a pretty big constraint, and the input for those results is a much more complete labeling than we expect (only 22 to 44 pixels around the ground truth boundary are left uncertain). But I'll definitely read the paper and hopefully we can gain some insights from it!
Have a look at KNN Matting http://ihome.ust.hk/~dli/projects/knn/. Their MATLAB implementation uses a similar style to yours but is not as real time. They take single mouse clicks and spread them over a much larger window/bounding box. Only takes a few clicks on foreground/background to get excellent results.
Actually Shared Matting is far more applicable than KNN Matting. It's designed for real time use and has a post processing step which will help you deal with fine detail like hair.
You should read Jue Wang's 2007 survey paper which groups and puts all the techniques in context, then a handful of the top performing papers from recent years on alphamatting.com
A bit off topic, but I'm working on a video interviewing startup and one of the things we'd looove to offer is (semi-)automatic Chrome Keying / green screening to show just a person's torso and face.
Might (one of) you know what's the state of the art in research on this topic as well?
Bookmark manager, speed, GUI configurability. It's pretty much the difference between a yacht and a tricycle.
I won't even let Chrome on my sytem directly (i.e. outside of a VM), because I'm tired of it downloading updates with some super needy background service, keeping 100s of megabytes around for no good reason, and no way to turn it off. At least Mozilla products ask. For that reason alone Opera and Firefox play in an entirely different league than Chrome or Safari. Fuck features, let me see your heart... Firefox I use and love, it's just not my main browser because only Opera has bookmark management worth calling that.
This is really slick. I work at a newspaper where we do a lot of cutouts and I've looked at several automatic tools to save people time, and I think this is the best I have seen in terms of balancing ease of use with accuracy.
The only thing I might suggest is that for photos with softer edges, you might offer some degree of "feathering" on the edges of the cut out. I don't know how hard this is with your algorithm, but it seems like it might be an easy fix.
Congrats! I had a similar idea a while ago so I really appreciate that somebody accomplished it in so high level! I already signed up and already waiting for the post alpha version :)
Did you guys consider in the future to add multiple object segmentation? That looks like a natural next step. Keep it up the good work! :)
Very cool! I've seen this technique before in research papers and their videos, but not as actual software.
The one suggestion I would make is to follow Photoshop's shortcuts. Space-Drag to move the image, [ or ] to change the brush size, scroll wheel to zoom (already there), and letters to select the various brushes.
Worked really well for me. One issue though: after doing a few of these and returning the main page, my uploaded images look broken. Anyway, great job!
147 comments
[ 4.1 ms ] story [ 188 ms ] thread(Disclaimer: I'm one of the devs)
Now, that's not how people actually use it at first - they "just scribble" (which our example might exacerbate), and then accidentally go outside the actual lines => not good.
Will need to figure out a way to make that more clear.
Unfortunately it is freezing up on me using an image with a gradient at about 90% of the initialization.
I think I stuck a race condition in there somewhere and it's getting exposed for the first time.
Please try uploading the image again :-/
There are many ways to solve such a problem. Another one beside GrabCub would be e.g. Watershed.
https://en.wikipedia.org/wiki/GrabCut
http://en.wikipedia.org/wiki/Watershed_%28image_processing%2...
So many variations available via e.g. OpenCV.
Regarding the segmentation phase, we were not satisfied with any existing algorithms because we wanted the tool to be as real-time as possible (for up to 4 megapixel images), so we rolled our own segmentation algorithm that has some similarities to GrabCuts (not GraphCuts).
But the binary segmentation is really only a small part of the technical challenge. You also need to figure out how to anti-alias the boundary, and remove the background halo. Simply feathering doesn't do the job unless you also erode the foreground first, which we wanted to avoid. So we developed a custom sub-pixel background-removal and anti-aliasing algorithm.
http://szeliski.org/Book/drafts/SzeliskiBook_20100903_draft....
I screwed up the first image I tried to clip. Maybe have a walkthrough of the features when the user first uses it? I didnt notice the help button until it was too late. I thought if I closed out of the editor and clicked on the image again, it would let me start over but it just takes you back to the editor. It would be nice if there was a way to redo editing the image from the start.
The help should have shown during the upload / init!?
(all these wonderful tradeoffs between "first visit" vs "repeat visit" - and what if it was missed the first time, blabla... )
The second time I loaded an image, it did the same thing, and only then did I actually click Help.
My second attempt turned out much better. http://i.imgur.com/kCthvpZ.png
(some memory woes on the backend as well, so restarting is having some hickups. Well, well)
"Network Error
We are having trouble connecting to the server. The following tasks have been unable to complete: Connecting to worker "Unable to connect to the worker." Waiting"
This is occurring after I drag and drop an image
http://ec2-23-20-184-45.compute-1.amazonaws.com:9006/api/web....
Just deployed a smaller brush size (30px instead of 40px when zoomed out).
It should get picked up when you reload the page (though default asset caching might hold it for another hour. Well, well)
My wife will love this for creating transparencies (she does serigraphy).
Pros: Doesn't require the fg/bg coloring step needed here, just figures out your FG without intervention. Cons: Have to register for an account to access it currently, but can use FB or Google to register
There's a great website http://alphamatting.com/ that provides a benchmark dataset and a comparison of a lot of influential methods.
I applaud your work in making a real time and interactive matting algorithm! One day it might be also as accurate as state of the art methods. You guys should write a paper about your method.
Thanks for the link! Real time is a pretty big constraint, and the input for those results is a much more complete labeling than we expect (only 22 to 44 pixels around the ground truth boundary are left uncertain). But I'll definitely read the paper and hopefully we can gain some insights from it!
http://www.inf.ufrgs.br/~eslgastal/SharedMatting/
http://www.juew.org/publication/mattingSurvey.pdf
Might (one of) you know what's the state of the art in research on this topic as well?
(the detection script should be accurate, so if they add them it should "just work")
I won't even let Chrome on my sytem directly (i.e. outside of a VM), because I'm tired of it downloading updates with some super needy background service, keeping 100s of megabytes around for no good reason, and no way to turn it off. At least Mozilla products ask. For that reason alone Opera and Firefox play in an entirely different league than Chrome or Safari. Fuck features, let me see your heart... Firefox I use and love, it's just not my main browser because only Opera has bookmark management worth calling that.
The only thing I might suggest is that for photos with softer edges, you might offer some degree of "feathering" on the edges of the cut out. I don't know how hard this is with your algorithm, but it seems like it might be an easy fix.
Did you guys consider in the future to add multiple object segmentation? That looks like a natural next step. Keep it up the good work! :)
The one suggestion I would make is to follow Photoshop's shortcuts. Space-Drag to move the image, [ or ] to change the brush size, scroll wheel to zoom (already there), and letters to select the various brushes.