HN Explorer

Show HN: I made a new AI colorizer (palette.fm)

472 points by emilwallner ↗ HN
Hi HN, I’m Emil, the maker behind Palette. I’ve been tinkering with AI and colorization for about five years. This is my latest colorization model. It’s a text-based AI colorizer, so you can edit the colorizations with natural language. To make it easier to use, I also automatically create captions and generate filters.

Let me know what you think.

You can see some of my results on my reddit page: https://www.reddit.com/user/emilwallner/?sort=top

121 comments

[ 0.25 ms ] story [ 169 ms ] thread
[deleted]
(comment deleted)
WrtCdEvrydy
Wow, any way to run this locally?

I love this technology but it would feel kinda terrible to upload a whole bunch of stuff to your site and exploit your generosity.

emilwallner
not for now. i might make a local version in the future, but for now, enjoy!
seanw444
Yeah agreed. I feel a bit icky running computationally-expensive stuff like this on someone else's dime. Give us a crypto address or something to donate to! Preferably not PayPal, though, if my say mattered.
aix1
Quite an incredible compilation of photos on your reddit page, thanks for sharing.
MitPitt
This is great. I'd also love to have the option to manually draw guiding colors over parts of the image, for even more control.
waltbosz
Very cool.

Your results look much better than the washed out AI colorization that I've seen in the past.

I think you could charge money for this service.

A suggestion for something fun to do / marketing tool: recolorize this video frame-by-frame https://www.youtube.com/watch?v=hZ1OgQL9_Cw

Feature request: colorize B&W comic books. I really want to create a full color book of Calvin & Hobbes comics. (Not for publication)

tanvach
Nice UI, and the approach works quite well (occasional wrong choice of palette compared to ground truth). It’s kind of fun to turn my photos into B/W and run through the model to try and guess the output.

Do you have a privacy policy for the uploaded photos? I’m not keen on uploading anything important without knowing how it’s stored or will be used in the future.

emilwallner
cheers! i'm working on a proper privacy policy. I don't store any images that are uploaded. i use google analytics and mixpanel to store user interactions.
nakedgremlin
Thank you for providing pre-set examples on your site. That's one of the major barriers on these types of tools to "test" especially when you're not sure where/how/if the images are public.
shubhamjain
Awesome job! This makes me a bit envious. I am hobbyist colorizer [1] who did it the old school way (with tools like Affinity Photo and lot of manual work). I tried de-oldify and other tools but I justified my work thinking how horrible they were but pallette.fm is way too good. Not sure if I would find the motivation to restore old photos anymore. Glad that this was just a small hobby for me and I had just started learning the ropes.

But I would be dead scared if I was a professional[2] who did this full-time. Is this what AI taking your job feels like?

[1]: https://shubhamjain.co/experiments/

[2]: https://www.youtube.com/watch?v=vubuBrcAwtY

emilwallner
ty! i'm glad you enjoy the tool. As for whether or not this tool could replace a professional colorist, I think it depends on the specific project. For some projects, Palette could do a great job of automatically colorizing photos, and for others, a professional colorist would still be necessary to get the best results. Especially when the projects require historical accuracy or a high aesthetic standard. it also makes colorization more accessible, which leads to more opportunities to refine results manually or say print the results.
Valgrim
Even if the results are not 100% historically perfect, 99% of the hard job is done. Skin tones, plants, skies, water... they all look incredibly good, and the object segmentation is almost perfect too.

The tool seems to struggle with fabrics, but that part is by far the easiest to fix with a traditional photo editor.

Congrats man. You made my mom happy this evening. Please keep a free tier on your tool.

throwaway14356
its fast and looks absolutely stunning but if you've done enough hand work you can still spot flaws (that are easy to correct by hand). The easiest to see is probably the use of black in parts of the face that are not dark shadows. Its slightly under-colored, it had to draw the line some place. It looks like black smudge's left and right of the nose. Then again, few/no one would notice without looking for it.

edit: if a human made these this comment would not exist. It makes A fun angle of AI, it can take criticism and do useful things with it. Many humans fail that challange.

Klaster_1
Ten years ago at my first job I used to colorize photos professionally, seeing how this tedious process has been automated away brings me the same joy as when you automate a routine with a shell script.
throwaway290
Could it have been trained on work of people like you?
JoeAltmaier
I love the way it finds features and labels the picture. It thinks I am a meteorologist!
brontosaurusrex
All I can say is: very very nice (Testing on some 3d renders and some bw photos). p.s. Is output image width limited to 1920px?
emilwallner
ty!! yes, 1920x1920. i'm working on an unlimited resolution option, but it will take a few weeks.
writeslowly
I tried it on the photo linked below and it makes the string instruments look like they're made of brass. Not trying to bash it since I thought it was pretty impressive overall, but I'm curious about what leads to this type of failure.

https://commons.wikimedia.org/wiki/File:Jazzing_orchestra_19...

emilwallner
thanks for the feedback! it's made out of two models, one model creates a caption and the second model takes the caption and the black and white image and colorizes it. if you click on the edit button you can see the text that generated that colorization. if the text is incorrect, you can edit the text and recolorize it. this often leads to a better result, however, some cases are still hard, especially damaged photos.
cercatrova
I assume you're using CLIP or BLIP for the text generation of the model, and then img2img or something like that for the colorization. What model are you using for the latter colorization?
learndeeply
> Vivid Natural — A slide of jam, jazzman, hook, band, pipe, and coil. Small contrasting details in natural colors. vivid natural.

This one generates the string instruments with the correct colors.

FL33TW00D
Excellent work - enjoyed the rick roll. Would you mind sharing how you've done the GPU deployment?
emilwallner
lol, thanks! onnx, docker, and fastAPI on CPUs with AWS fargate. although i'm switching to GPUs in a month or two, so if you have any suggestions let me know.
punkspider
I've also been searching for GPU solutions for potential small projects like this, and so far banana.dev and runpod.io seem promising.
kurtreed
I haven't tried any other colorizer tools but this one works very well for me. Thank you.
callumprentice
Thank you so much Emil - I have some old photos B/W of my dad who is no longer with us and have wanted to colorize them somehow for years. The results from just dropping the photos on your page - no tweaking or whatnot - are incredible and have me in tears.

Amazing work.

nathell
Comments like this are what make me come back to HN. I hope the memories are kept alive.
callumprentice
Thank you so much. My daughter is away at camp - I can't wait to show her tomorrow too when she gets back.
loceng
I can't remember the name but there's a Reddit subreddit where people can post photos to request colorization, so that may be an option if you're curious what some passionate volunteers might do for you.
BizarroLand
Just a quick heads up, when I ran the file and then opened the result in irfanview, I get a warning that the file is a JPG file with an incorrect extension.

You may want to check your encoding settings to make sure everything gels together.

Otherwise, great job! This is pretty nice stuff and way better than I could do on my own!

shisisms
This is incredibly well done! Congrats. It’s such a solid layer on top of recent developments and offers instant value.

Wish you much success!!

nextaccountic
Can I paint some colors as a starting point (like img2img) and let it finish the rest?

What about a conversation, like, "the dress isn't blue, it should be orange", doing on top of previous prompt?

emilwallner
not yet, these are great suggestions. it's always a dilemma to add features to mitigate the performance of a weak model, instead of making a better model. most of the problems go away with a better language and colorization model, and many model-specific features are made in vain
nextaccountic
I think there are two uses for an AI colorizer. One is to generate a color image that looks great, another is to generate an image that accurately reflects the true color of things.

A better AI model helps a lot with the first goal, but help only so far with the second one. Truth to be told, there is a lot of contextual color information in black and white photos that an AI model can exploit; but nothing beats someone that knows, for sure, the color of the dress of someone in the photo.

I mean, take a look at https://www.reddit.com/r/ColorizedHistory/ - some of those color artists do a lot of research to know the exact shade of green of the military uniform of some country in the 19th century, and things like that, just to have an accurate reference.

So I think that the ability of directing the color output (either by rejecting a color textually, or by painting over the figure with a starting point - even if maybe I'm not painting with the exact tone or texture but a rough color that should help the AI to figure out the details) is essential for a colorization product, even if the model is flawless!

nextaccountic
Actually, thinking of /r/colorizedhistory, here's a comparison of a professional colorization and palette.fm https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/pa... vs https://www.reddit.com/r/ColorizedHistory/comments/y5mfqu/pa...

My concern here isn't that the professional photo has higher quality (in this case it has, but give it some time - months or years - and maybe technology will catch up). It's that sometimes we already know the right color, while the AI must always guess

w_for_wumbo
So you can kind of do that, the first step creates a description, and the second step colorizes using the description. So you can modify the text for various degrees of success and specify 'orange dress within the text'
learndeeply
Can you summarize the approach you took on creating the model? Really curious!
kmnc
This works amazingly well, just putting in some of my favorite black and white scenes from movies... wow! Now I need some kind of tool that can do this on every frame of a black and white movie. I cannot wait to be able to watch some old classics fully colorized.
zxcvbn4038
Wow! I am impressed. I uploaded a WWII photo of airmen posing in front of their plane and got an amazing result. I’m not certain if WWII era life vests were bright yellow, that seems out of place, but it’s a good guess if not.
karamanolev
Found some that are yellow, not sure how bright we're talking - https://www.ima-usa.com/products/original-u-s-wwii-usaaf-mae...
surfsvammel
This is magic… How the heck does it know that the walls of my home gym is light green?!?

Impressive!

draugadrotten
I laughed out loud reading your ABOUT page. Thank you!
theden
Wow I put a few b+w photos, initially shot in colour so I can compare how it did, and it more or less nailed it—great work. They were shot on a iphone, so I wonder if newer digitally-shot photos have more data for the app to parse, or if it's generally easier compared to film (and whether it uses any EXIF data)
poulpy123
I expect the training to be done mostly on modern digital cameras