Probably as a research project to see if it's worth using in HD conversions of old games, such as the Age of Empires remake that's dropping tomorrow. The algorithm seems to work well on standalone sprites, but kinda merges in a gooey way when two sprites are next to each other. The super mario example from the previous discussion shows this slightly: http://files.jjcm.org/upscaling.mp4
Very curious to see this as it progresses however.
It looks like the transformation in being applied to the whole image at once in your example. If each sprite was vectorized individually, you wouldn't see the gooeyness. If you are a game developer, this change should be easy enough.
I'm still impressed with the super mario example, I can't personally see anything negative about it and would enjoy playing the game with that "art style."
It's not perfect but seeing the alternatives it is a huge step in the right direction.
I don't think it was about videogames, more likely upscaling algorithm would have saved a TON of bandwidth for videos.
a a matter of fact some of the most recents algo available for video deinterlacing like nnedi3 are perfect for that, and of course also work for pixel art, and I think they blow away the example showed in their gallery - https://www.youtube.com/watch?v=0691zsXWbhA
I wonder if now, in the tensorflow era, a better automatic approach using generative nn could be devised to reach beyond 8x
Filters have achcieved similar results, but not quite with the same technique!
Or, this algorithm wasn’t intended for real-time processing and creates an adjacency graph of similar pixels. Once it’s done that, it will try to fit/optimize splines over the regions. Emulator filters like hqx determine each pixel’s “upscaled shape” by the colours of its neighbours, usually indexed with a table.
Could you take this a step further? For example, a human knows this thing isn't supposed to be lumpy. What would it take to make this look more like it was interpreted by a human?
Super resolution with deep neural networks has gotten some decent results that incorporate knowledge of what the image is. However, it's probably not worth it. Running this algorithm on a single image is much simpler than finding thousands of pixel art -> image pairs and training a superresolution network.
waifu2x (http://waifu2x.udp.jp/) is another often-posted superresolution approach, focusing on upscaling low-res anime images instead of extremely tiny pixel art. It uses a totally different learning-based approach - however, their approach would not work with pixel art because it depends on having a set of high resolution ground truth images, and downsampling them to generate training data pairs. No such high-resolution examples exist for pixel art.
waifu2x seems to be based on Deep Convolutional Neural Networks, while the article looks like a pure handwritten algorithm and at least in this case the algorithm looks like a winner..
I am wondering whether in the future we may forget about writing algorithms altogether and instead rely on ANNs to do tasks that could have done better using hand written algorithms?
Image super-resolution is an ill-posed inverse problem: many possible high-resolution images would be reduced to the same low-resolution image. However, some of the possibilities are more realistic than others. Consider a grey pixel in a photograph from real life. It's more likely that it was downsampled from a 2x2 block of grey pixels than from a 2x2 black-and-white checkerboard pattern. We apply such knowledge by adding regularization to the problem, or using a prior distribution in a Bayesian formulation.
Deep learning is very good at memorizing those priors by looking at real world data. I think, in problems where the prior is important and complicated, data-driven approaches have a big advantage over hand-engineered approaches. That does not mean they will take over every kind of problem.
I think hq4x has a slight edge because it doesn't go all the way trying to make everything smooth, still preserving some pixelation. There's actually an analog to this in denoising: if you denoise too heavily you end up losing detail. Smart denoising manages to preserve edges, giving a better result. In this case, pixels are the same as noise, undesirable, but containing useful information.
for both directions I would say a competent artist.
Clever algorithms get you some of the way, but you will still need someone with good sense to edit the result to get something usable, which is going to cost you in either time or money.
Downsizing isn't so bad now because of higher DPI displays (e.g. font hinting is obsolete now) but previously designers would need to make individual icons in a number of sizes removing elements as necessary.
It seems pretty trivial but I could be way wrong as this is in no way my area of expertise...but- pixelization filters in photoshop and other image editing softwares have been around for a very long time.
Even logically it is an easy leap as they're(the software/filters) most likely choosing the most common color in a given area and then converting that area to a square of that color..
Just pixelating an image doesn't really give high quality pixel art. No examples handy but I'm confident you'd see this if you tried to reverse the images in the post with the approach you outlined.
Their method change too much on eyes and facial expressions in general. Also that broad sword is really blunt.
When comparing only the 4x results hq4x wins half the time. On the other half hq4x is a close second and generally looks alright. However in some cases the vectorized ones are pretty bad.
Like translating a phrase into a foreign language and back repeatedly, I inagine you’d end up with something fun and impressionistic. I’d like to see it too.
The best examples, those with more divergent results, are those with a lot of subordinate clauses, which get mixed-up and moved from different parts of the sentence. You can select complex sentences from the list below.
I remember back in like 2010, someone uploaded and ripped a YouTube video 1000 times and then edited the natural degradation of the compression. Went from perfectly watchable video to horrible static-y mess.
Theoretically, it should be possible to recompress the video back into the compressed file with no loss, provided the same codec is used, but it would be far more expensive computationally.
Bizarrely, the pixelated stuff in the images you linked look way better. You couldn't use the depixelized whale or dolphin or whatever (our link) in a game, it looks awful. Some of the full comparison is even worse:
But you can use literally any of the pixel art'ed images from your link - they all look truly fantastic and production-ready! They're ready to ship now.
I always thought it was sad that it didn't get as much attention. I think it could also interesting to use in thumbnail generation based on regular photos, for example.
Going to ask in that reddit thread if the author is already aware of this algorithm. Might be able to take some cool new ideas from it if they don't! :)
Huh, that's actually quite interesting - but for an entirely different purpose.
My better half enjoys a bit of cross stitch. For a cheapo little gift I've played with outputting custom cross-stitch patterns based on pixellating images and reducing the number of colors to something more manageable, but it's often a little janky in GiMP and requires a bunch of manual tweaking - for anything remotely detailed it quickly becomes more hassle than it's worth.
This was my first thought as well. I've gotten decent at using photoshop to do this with generous use of cutouts and indexed color modes to custom forced palettes, but it's a very laborious process.
Playing with this, it's very effective for solid-color graphics as it quickly settles in on a nice limited palette. The line-work also does a reasonable job, limiting the manual fixing to areas where lines converge and so forth.
I'm working on writing up a post for /r/crossstitch comparing some different methods, so check over there in a while if you're interested.
Gotta say, I really like those PhotoZoom 4 results, aside from some of the fringing artifacts. They seem to preserve a lot more of the detail, whereas the presented approach focuses more on eliminating noise.
Every time I see this paper I'm always disappointed that there doesn't seem to exist a usable (open source or otherwise) implementation of it anywhere.
I want to go through the research paper but am unable to load or download it. If anyone has managed to do the impossible please share a drive link. BTW which journal is this published in?
95 comments
[ 2.7 ms ] story [ 140 ms ] threadI wonder why Microsoft payed him to do develop this kind of algorithm.
Very curious to see this as it progresses however.
It's not perfect but seeing the alternatives it is a huge step in the right direction.
a a matter of fact some of the most recents algo available for video deinterlacing like nnedi3 are perfect for that, and of course also work for pixel art, and I think they blow away the example showed in their gallery - https://www.youtube.com/watch?v=0691zsXWbhA
I wonder if now, in the tensorflow era, a better automatic approach using generative nn could be devised to reach beyond 8x
Or, this algorithm wasn’t intended for real-time processing and creates an adjacency graph of similar pixels. Once it’s done that, it will try to fit/optimize splines over the regions. Emulator filters like hqx determine each pixel’s “upscaled shape” by the colours of its neighbours, usually indexed with a table.
can't vouch for it yet, but will be giving it a try later.
A shame, I'm really keen to spruce up some old 50x50px forum icons from where I used to hang around back in the early naughties.
Reminds me of some old newgrounds flash animations and games
How about application icons?
I am wondering whether in the future we may forget about writing algorithms altogether and instead rely on ANNs to do tasks that could have done better using hand written algorithms?
Image super-resolution is an ill-posed inverse problem: many possible high-resolution images would be reduced to the same low-resolution image. However, some of the possibilities are more realistic than others. Consider a grey pixel in a photograph from real life. It's more likely that it was downsampled from a 2x2 block of grey pixels than from a 2x2 black-and-white checkerboard pattern. We apply such knowledge by adding regularization to the problem, or using a prior distribution in a Bayesian formulation.
Deep learning is very good at memorizing those priors by looking at real world data. I think, in problems where the prior is important and complicated, data-driven approaches have a big advantage over hand-engineered approaches. That does not mean they will take over every kind of problem.
hq4x and the #xBRZ algorithms are amazing and fast and look nearly identical to these results.
http://johanneskopf.de/publications/pixelart/supplementary/i...
This method still wins out though.. but interesting never the less.
(just kidding)
Clever algorithms get you some of the way, but you will still need someone with good sense to edit the result to get something usable, which is going to cost you in either time or money.
Downsizing isn't so bad now because of higher DPI displays (e.g. font hinting is obsolete now) but previously designers would need to make individual icons in a number of sizes removing elements as necessary.
Even logically it is an easy leap as they're(the software/filters) most likely choosing the most common color in a given area and then converting that area to a square of that color..
http://johanneskopf.de/publications/pixelart/supplementary/m...
When comparing only the 4x results hq4x wins half the time. On the other half hq4x is a close second and generally looks alright. However in some cases the vectorized ones are pretty bad.
http://pixelatorapp.com/
https://www.reddit.com/r/gamedev/comments/7ylvt9/pixelator_a...
http://www.translationparty.com/bizarrely-the-pixelated-stuf...
> Bizarrely, the pixelated stuff in the images you linked look way better.
To
> But then I remembered I look way better in the morning light.
In one step. Lovely! This sounds like poetry.
This exact game features in the plot of Philip K. Dick’s 1969 novel, Galactic Pot-Healer.
Here it is! https://www.youtube.com/watch?v=icruGcSsPp0
This may sound crazy, but I wonder if there's a way to "automate" this deconstruction with ffmpeg...
ffmpeg -i input.mp4 -strict -2 -crf 51 output.mp4 && rm input.mp4 && mv output.mp4 input.mp4
For better results, run the above command ten times:
for ((n=0;n<10;n++)); do ffmpeg -i input.mp4 -strict -2 -crf 51 output.mp4 && rm input.mp4 && mv output.mp4 input.mp4; done
> What happens if you make a copy of a copy of a copy (and so on) of a VHS tape? This experiment shows how the quality degrades with every generation.
> The copying was done using two PAL VCRs in SP mode.
> (The video is Fading like a flower by Roxette)
https://youtu.be/mES3CHEnVyI
http://johanneskopf.de/publications/pixelart/supplementary/m...
Mario looks horrid.
But you can use literally any of the pixel art'ed images from your link - they all look truly fantastic and production-ready! They're ready to ship now.
"Content-Adaptive Image Downscaling" http://johanneskopf.de/publications/downscaling/
I always thought it was sad that it didn't get as much attention. I think it could also interesting to use in thumbnail generation based on regular photos, for example.
Going to ask in that reddit thread if the author is already aware of this algorithm. Might be able to take some cool new ideas from it if they don't! :)
My better half enjoys a bit of cross stitch. For a cheapo little gift I've played with outputting custom cross-stitch patterns based on pixellating images and reducing the number of colors to something more manageable, but it's often a little janky in GiMP and requires a bunch of manual tweaking - for anything remotely detailed it quickly becomes more hassle than it's worth.
This, on the other hand, might just do the trick!
Playing with this, it's very effective for solid-color graphics as it quickly settles in on a nice limited palette. The line-work also does a reasonable job, limiting the manual fixing to areas where lines converge and so forth.
I'm working on writing up a post for /r/crossstitch comparing some different methods, so check over there in a while if you're interested.