Impressive stuff, I especially like the style transfer that can be done by using the global features of one image and the local features of another (Fig. 7)
What I find somewhat annoying is that whilst they show some examples from their validation set, and a couple of examples of the model failures. They don't appear to show a random selection of cases from their validation set.
The results that I see seem quite good, but I also find the lack of any objective benchmarks or performance measures to be one flaw in the paper. This might explain why they published in a graphics conference (SIGGRAPH) rather than an ML or computer vision conference (which would probably demand benchmarks).
This is amazing to me. My major was Digital Imaging Technology in 2005 and I remember doing this by hand in photoshop wondering if one day there would be a button for it.
I remember doing web animation back in 2000, and having to ink each penciled frame by hand, vectorizing the inked drawing and then coloring it in Flash. Now you can train a CNN to vector pencil drawings. Probably train a CNN to color everything too!
While I can see a lot of animation tasks being eliminated, I would like to imagine that they would hold onto some artists to spend some of that saved time embellishing and strengthening the quality of the final work.
But most of the heavy lifting will now be done with the press of a button. It's like Clarke said, "Any sufficiently advanced technology is indistinguishable from magic."
As with most deep learning papers, though, the results aren't independently reproducible. The dataset is private, the source is closed, and it hasn't been turned into a product. The magic inking button is still a ways off yet.
They're not required to release code and a dataset. If you can get away with doing less why not? Everyone in every discipline does this. Finance, home improvement, retail, consulting...
I'm really asking, because I've downloaded the project and skimmed the paper, but haven't had time to vet these assumptions. On the face of it, it seems everything is provided, but you've vetted it further and learned that isn't true?
I would really like to see approaches like these applied to movie scenes. Especially how differences in single colorized frames depicting the same scene could be handled.
They (I believe, although it might be the other guys) have a video up on YouTube demonstrating it on a Vietnam movie trailer. It looks appalling: aside from the usual averaged colors and saturation, the coloring is highly unstable and changes from second to second.
Found a video [1] colorized with the approach by these guys with the unstable coloring.
There are some other videos [2][3] colorized using differing approaches, that don't seem to have as much color unstability though the color in general appears more off.
Could of course also just be a property of the underlying source.
It seems like adding the colour would be the hard part. Once you had that, normalising the colour or smoothing it out across hundreds of similar images could be achieved by a second software algorithm.
Certainly colour correction, hue, contrast, brightness, would be easier to process than adding colour in the first place.
Tried with some historical photos and my own BW images. It's missing the global image prior for most of the images except vegetation, it has a hard time even with people. For local features I've seen similar problems, my guess is that they trained it on a not large enough dataset, the spectacular samples are from over-training. While the idea looks promising the current implementation is far from general.
I'm surprised by how apt Lua is for these kind of algorithms. From the architecture diagram I expected to be hit by a large blob of code but found that most of things are taken care by the language/framework itself!
Can someone please put the file colornet.t7 on the torrent network or a high-volume service somewhere? I'm probably not the only having a hard time downloading that file.
26 comments
[ 4.4 ms ] story [ 68.3 ms ] threadWhat I find somewhat annoying is that whilst they show some examples from their validation set, and a couple of examples of the model failures. They don't appear to show a random selection of cases from their validation set.
Another model previously posted on HN, with (IMO) worse results than these two models: http://tinyclouds.org/colorize/
I remember doing web animation back in 2000, and having to ink each penciled frame by hand, vectorizing the inked drawing and then coloring it in Flash. Now you can train a CNN to vector pencil drawings. Probably train a CNN to color everything too!
While I can see a lot of animation tasks being eliminated, I would like to imagine that they would hold onto some artists to spend some of that saved time embellishing and strengthening the quality of the final work.
But most of the heavy lifting will now be done with the press of a button. It's like Clarke said, "Any sufficiently advanced technology is indistinguishable from magic."
Magical times.
https://github.com/satoshiiizuka/siggraph2016_colorization
And is this not the dataset:
http://places.csail.mit.edu
I'm really asking, because I've downloaded the project and skimmed the paper, but haven't had time to vet these assumptions. On the face of it, it seems everything is provided, but you've vetted it further and learned that isn't true?
anyone know why both papers share this seemingly arbitrary diagram style?
edit.. ahh, it's the same person
CNN = Convolutional Neural Networks in this context.
Found a video [1] colorized with the approach by these guys with the unstable coloring.
There are some other videos [2][3] colorized using differing approaches, that don't seem to have as much color unstability though the color in general appears more off.
Could of course also just be a property of the underlying source.
[1] https://www.youtube.com/watch?v=__kcHbzSNC4 [2] https://www.youtube.com/watch?v=_MJU8VK2PI4 [3] https://www.youtube.com/watch?v=qQSViqdd0tU
Certainly colour correction, hue, contrast, brightness, would be easier to process than adding colour in the first place.
My Grandfather & Brothers: http://adam.gs/v/IMG_0090.jpg http://adam.gs/v/IMG_0090.color.jpg
My Grandfather, My mother and my Aunt http://adam.gs/v/IMG_4629.jpg http://adam.gs/v/IMG_4629.color.jpg
My grandfather and my grandmother: http://adam.gs/v/IMG_6868.jpg http://adam.gs/v/IMG_6868.color.jpg
From my perspective, these are decent results considering what they have to work with, I think it did a very good job.