It first uses a simple CNN to get one of the 100 classes. It then uses that class to retrieve images using CLIP. The original CLIP Space that helped for inspiration is https://huggingface.co/spaces/vivien/clip.
Oof. That 100 class limit is less practical than just asking the user for arbitrary text input and using CLIP as a zero-shot classifier. You would have gotten better responses if the limitations and motivations were explained on the page.
Thanks for the feedback! This was just a demo I shared in Twitter and didn't expect it to be shared so widely by others. We're training a new CNN for more classes and we might switch to another dataset for the image retrieval part. It's really a very small pet project :D
I’m trying to draw the silence of lamb poster with Jodie foster but it keeps returning pictures of bearded men. Lol. Let me know if anyone is successful with that poster.
Doesn’t work when I try to use it, always returns the same thing, and it seems like everyone else is having the same problem. Sounds cool though and I’ll be excited about the day we progress enough where something like this works! GANs are very cool.
From what I gather this does object detection on the picture and then from those detected items compares to movie stills that have been similarly labelled. On paper sounds like it could work but in practice .. results are a bit underwhelming when I tried
It doesn't seem to work for me - you draw a shape and then it returns some completely unrelated images. You then pick one of those images and try to draw it as closely as possible and then it returns a completely unrelated image again.
I thought the weird thing was that with that second step, if I drew a broom exactly for example, and it guessed “broom” correctly, it returns images with no brooms in them at all?
First of all, most results are anime. I tried a few easy-to-draw, pretty popular symbols like Harry Potter's forehead scar, The Incredibles logo but got nothing close to them. It's a very cool idea but still needs a lot of work.
Looks like it's just guessing the object and then finding images tagged with that object. I was hoping for some sort of scene detection to find specific scenes from films.
33 comments
[ 4.4 ms ] story [ 98.5 ms ] threadNice idea, but I was never able to find the movie I was thinking of. Perhaps I'm too bad drawing.
It first uses a simple CNN to get one of the 100 classes. It then uses that class to retrieve images using CLIP. The original CLIP Space that helped for inspiration is https://huggingface.co/spaces/vivien/clip.
I drew a closed fist, it says moon / alarm clock.
Was hoping for some AI visual processing thing and was doing the mental gymnastics of how to store all that data.