I had to share this one, I sent a screenshot of the Yahoo homepage from a while ago (yeah, I had that hanging around...) with the main image being of Donald Trump.
The caption guess was "I am not really confident, but I think it's a television screen and he seems ."
I uploaded a photograph of a bunch of snow piled on top of a round table[0], which looks a lot like a marshmallow to the human eye. But it came back with "I am not really confident, but I think it looks like a polar bear lying in the snow." Not terrible :)
It's only a matter of time before a repeat of Microsoft's last AI experiment (Tay), when the Internet teaches CaptionBot all of the positions in the Kama Sutra.
Imagine this tech matures and it can be incorporated with bodycams for police, when confronting a subject with objects in their hands it may be able to confidently estimate the probability of being a firearm or not, with better predictability than the police/people.
While we're here, let's go the full way and set up a proveable and public way to train a robocop, and I'd trust that more than a human cop. The awkward moment when AIs have more brains than cops (at least under the US system).
Likely before we get to robocops the "robocops" will be integrated into people who have proven risky by either previously known behavior in addition to social signals.
So Jane truant with convictions of petty theft or battery gets off with probation if she agrees to embed her own personal "robocop". Yes invasion of privacy, etc. But the alternative for her would be time in the pen, for example. So in this case people can become their own robocops who turn the host in to authorities if certain conditions are met (engages in previously restricted activities).
I think this is more likely than a roving robotic cop which looks out for misdeeds.
A couple of days ago I think there was a post about Google doing a lot of development and research around creating systems that understand / categorize / comment / recognize images.
One thing I took away from reading about it is that Google has billions of images to train it with from all their different ventures.
Does Microsoft have access to anywhere near the same numbers of pictures?
The Chinese company has structured a deal with Getty to take over licensing outside of China.
However, having access to world class photography is great, but the images that (probably) will be the most interesting for Microsoft to recognize will be selfies, and other "crowd" created amateur photography and possibly memes.
I personally would think it to be cool to see if the bot could traverse the Getty collection and see if it could recognize the photographer of an image it had not seen before. Why yes, this is Leibovitz.
It is almost as smart as a child. I uploaded a picture of my Notre-Dame vacation photo, and the caption was "A person standing in front of a church"... which is close to my sons "mommy standing in front of that church we went to"
Yep, on the stuff it recognizes. It recognized a picture I tested taken from the Grand Tetons as "a lake with a mountain in the background", which was quite correct, but also kind of generic.
On the other hand, it described a picture of Grand Prismatic Springs in Yellowstone as "a train with smoke coming out of the water." Which also is kind of like the crazy things kids sometimes say when they see something new.
First of all congratulations on a) the science (built on the shoulder of giants...) b) the accessibility / interface and service
Wondering if you plan to open up a caption API of any sort? Can definitely use something like this. If you desire the training feedback, then that could be added as well as part of the API. I'd be willing to do that for some images. So if you do add a training feedback API, please make it optional.
It would be great if you added a text box under the stars so I can (optionally) tell you why I didn't give you five stars.
For example, I uploaded a picture of my daughter as an infant, and it said, "This is a baby on a bed and he's :D" which I gave four stars to because it said he instead of she. But honestly you really have no way of knowing that. :)
Thank you very much for putting your work out there for people to have fun with (and criticise!).
CaptionBot seems to have a bit of trouble with simple two-colour outline drawings. In one case I saw it even get the colour wrong ("red and white" for a black and red image).
Is that something you would have expected?
Also- I notice it doesn't do very well with character recognition either. Is that surprising?
Have you considered an 'abstract' type of version? As in, rather than simply describing the image as the caption, take the information that would be used, fill it in a MAD-LIBS[1] style setup, and see how that turns out? Maybe it's just me but a surrealist CaptionBot could be pretty fun.
169 comments
[ 2.8 ms ] story [ 234 ms ] threadand I got this result "I am not really confident, but I think it's a couple of glass vases with flowers on top of a surfboard."
[1] http://vignette3.wikia.nocookie.net/lovecraft/images/9/95/5b...
:)
https://upload.wikimedia.org/wikipedia/commons/thumb/9/90/Pe...
And it came back with, "I'm not really confident, but I think it's a person on a surf board in a skate park."
http://imgur.com/WWHQu5i
MS has been providing the net with fun activities for a few months now. The other night the celebrity look alike site blew up on twitter and instagram
The caption guess was "I am not really confident, but I think it's a television screen and he seems ."
He didn't mistake the football player with a rugby player, a cricket player or else. And +1 for the emoji
Wonder if it was the ball logo on the player's shirt, the greater popularity of football , or something else that helped it distinguish.
[0] http://imgur.com/nKRK5hc
I uploaded a photo of the planet Saturn and it guessed that it was dish. It got the shape right.
"In a development that surprised even us, after an influx of /b/ and Something Awful Goons, the AI decided to shut itself off."
Edit: recognizes Stalin though http://i.imgur.com/9W6wqUd.png
In fact, I assume this is a crowd sourced training for the tech..
Kind of disappointing, but at the same time I understand that this task is not trivial at all.
>I am not really confident, but I think it's a man is smiling for the camera and they seem . I am 99% sure that's Pope Benedict XVI
Source Image: http://memesvault.com/wp-content/uploads/Wat-Meme-Old-Lady-0...
Needless to say, my errant habits of trying to break stuff shine through once again.
>I am not really confident, but I think it's a man holding a camera.
Source Image: http://www.cinemablend.com/images/news_img/79237/pulp_fictio...
While we're here, let's go the full way and set up a proveable and public way to train a robocop, and I'd trust that more than a human cop. The awkward moment when AIs have more brains than cops (at least under the US system).
So Jane truant with convictions of petty theft or battery gets off with probation if she agrees to embed her own personal "robocop". Yes invasion of privacy, etc. But the alternative for her would be time in the pen, for example. So in this case people can become their own robocops who turn the host in to authorities if certain conditions are met (engages in previously restricted activities).
I think this is more likely than a roving robotic cop which looks out for misdeeds.
A couple of days ago I think there was a post about Google doing a lot of development and research around creating systems that understand / categorize / comment / recognize images.
One thing I took away from reading about it is that Google has billions of images to train it with from all their different ventures.
Does Microsoft have access to anywhere near the same numbers of pictures?
Bill Gates sold it earlier this year to a Chinese Company http://www.ft.com/intl/cms/s/0/d6fbcb88-c126-11e5-846f-79b0e... (paywall),
The Chinese company has structured a deal with Getty to take over licensing outside of China.
However, having access to world class photography is great, but the images that (probably) will be the most interesting for Microsoft to recognize will be selfies, and other "crowd" created amateur photography and possibly memes.
I personally would think it to be cool to see if the bot could traverse the Getty collection and see if it could recognize the photographer of an image it had not seen before. Why yes, this is Leibovitz.
On the other hand, it described a picture of Grand Prismatic Springs in Yellowstone as "a train with smoke coming out of the water." Which also is kind of like the crazy things kids sometimes say when they see something new.
http://imgur.com/JBlpJvQ
I get the same "about as smart as a very young child" vibe from CaptionBot; the intersection of pleasantly silly and rather impressive.
(Photo: Rover P6 on the start line with a Nissan R35 GTR, guessed as "I think it's a police car parked in front of a truck.")
http://i.imgur.com/tc5rz9s.png
http://imgur.com/KAmCF2m
http://imgur.com/GSpanVe
Wondering if you plan to open up a caption API of any sort? Can definitely use something like this. If you desire the training feedback, then that could be added as well as part of the API. I'd be willing to do that for some images. So if you do add a training feedback API, please make it optional.
For example, I uploaded a picture of my daughter as an infant, and it said, "This is a baby on a bed and he's :D" which I gave four stars to because it said he instead of she. But honestly you really have no way of knowing that. :)
It seems from the comments that that is one of the most represented misclassifications.
CaptionBot seems to have a bit of trouble with simple two-colour outline drawings. In one case I saw it even get the colour wrong ("red and white" for a black and red image).
Is that something you would have expected?
Also- I notice it doesn't do very well with character recognition either. Is that surprising?
[1] https://en.wikipedia.org/wiki/Mad_Libs