People might be interested in this video by @kcimc , where Kyle runs the pretrained model forward in real time on his laptop while walking around the streets of Amsterdam
Something people don't fully appreciate about neural networks is that their performance is quite a strong function of their training data. In this case the training data is taken from the MS COCO dataset (http://mscoco.org/explore/). That's why, for example, when Kyle points the camera at himself the model says something along the lines of "man with a suit and tie" - there is a very strong correlation between that kind of an image in the data, and the presence of a suit and tie. With such a strong correlation the model doesn't have a chance to tease the two concepts apart. A similar problem would come up with an ImageNet model, where a similar image might be classified as "seatbelt", because there is no Person class there, and shots of people in that pose usually come from the seatbelt class. It happens to be the most similar concept in the data it has seen. Another example is if you pointed the model at trees it might hallucinate a giraffe, since the two are strongly correlated in the data. Or when Kyle points the camera at the ground I'm fully expecting it to say relatively random things, because I know that those kinds of images are very rare in the training data.
In other words, a lot of the "mistakes" are limitations of training data and its variety rather than something to do with the model itself, and it's easier to recognize this if you're familiar with the training data and its classes and distribution.
I don't think that the seatbelt example is overfitting. It's a consequence of their not being the "correct" available label. In many ways I guess the seatbelt would be the best option.
Sure, I agree the seatbelt example comes from not having enough labeled classes. Feels a little bit more like the realm of semi-supervised learning, but I'd love to learn about interesting approaches in that case too :)
It's not exactly overfitting. It's more that the model is seeing an image that is out of sample, but our approach _forces_ it to say something, so it will do its best but usually fail.
One simple low-hanging-fruit approach would be to include a large repository of additional data (e.g. all of ImageNet) and label it all as a "garbage" class. This way the model could at least learn to distinguish the kinds of images in its training data from the universe of images, and this could be used as one proxy of confidence.
Another simple proxy is to look at the probability of the generated sample, since usually the model tends to assign more diffuse probabilities in more uncertain cases. But this is also not a very clean approach for various reasons.
Another, and probably most appealing, approach would be something along the lines of Bayesian Neural Networks, ensembles, or approximations with dropout, where the disagreement between the predictions of all submodels can be used.
X. Zhang and Y. LeCun had an article [1] about using this technique for regularization very recently. We aren't talking specifically about overfitting in this case, but rather a lack of diversity / size in the training data, however it seems like this kind of thing might help for a variety of tasks.
One more idea would be to have a (non-differentiable / REINFORCE?) penalty based on sentence likelihood using doc2vec or skip-thoughts to avoid the "blah and a blah on a blah" type errors that seem to be common in captioning.
One more would be to use TV / YouTube captions, but that data is extremely noisy - even more the COCO captions, unfortunately!
I get excited whenever we can improve performance with manipulations to the training sets such as data transformation tricks (inducing variances like rotations, translations, whitening [1] etc), labeling tricks (such as those like LeCun or [2]), or including information/learnings/regularizations from external corpus like doc2vec.
It feels like getting something for free :) Of course there is a limit to how much signal you can extract from a noisy dataset, but the amount of time and human energy invested into creating and improving datasets can be quite large relative to finding another cool trick that can improve performance.
However, I wonder which will come first to make these systems "robust" for the average joe's real world uses for these perceptual systems; a large, well labeled dataset or more transformations and semi-supervised learning approaches?
Can you explain what the various reasons for not using a probability baseline (i.e., only if the model is pretty confident assign a label, otherwise say nothing) are? It's what we're being taught right now in ML classes, so I would like to fix my understanding.
In image captioning specifically we are in a dire need of data. To give you an idea, MS COCO is ~100K images. ImageNet Challenge is 1.2 million. The dataset is quite miniscule, which restricts the richness of models we can explore, and forces strong regularization concerns. The place I normally like to be is when my several hundred million parameter nets are underfitting - that's where neural nets really shine - and MS COCO is not that.
Also it's not only the size of the dataset, it's also the size/variety in the label space. ImageNet is quite comprehensive, with many varied labels. MS COCO is quite biased towards a narrow ~hundred classes.
I'd love to see a properly large dataset of images "from the wild", with no restrictions on content (unlike what is done in MS COCO), annotated with sentences. From my experience with adding data to models in these situations I'm quite certain this would work _significantly_ better.
Thanks a lot for your hard work! This is amazing work. I was wondering if you had any sense of which one of the models works best (or when one is better than another) -- Stanford, Toronto (Show, attend, tell), and Google (Show and tell) ?
So here's what's funny about the leaderboard. The NeuralTalk submission (near bottom of the list) is in fact a re-implementation of the Show and Tell model (my paper and Oriol's are basically identical models - we plug in the top of the CNN to an RNN on first time step. Except I use vanilla RNN and he used an LSTM (which turns out to work a bit better)). But it's an identical model - in that submission I had in fact used the LSTM implementation in NeuralTalk.
What's different between the two is the engineering portion: NeuralTalk was written in Python and ran on CPU without batches (so probably it did not converge as well), it did not do CNN finetuning (which helps a ton), and it did not do ensembles, and I used VGG while Oriol used GoogLeNet (though I don't expect much difference there). There are a few more tricks that Oriol talks about that give you a few small extra points, but that's basically it. I also don't want to take away from Oriol's top result (these small tricks and engineering are very valuable and hard to come up with), but I would be hesitant to draw conclusions about which models work best by looking at the model diagrams in a Figure, and comparing to numbers in the table.
That's why I've over time grown cynical about results in tables of papers that compare one work to another - there are too many variables to keep track of and it's confusing unless you know the full details. The truth is that results in tables are model + engineering + noise. The papers pretend that it's all model, but in fact the latter 2 have a huge impact. At least results that only compare two models in the same framework can be trusted a bit more to contain information, but even then you find that in practice people can be a lot more generous to their own models than their baselines. Hence the famous saying: "The second best model in the paper is in fact what you want to use".
To answer the original question though: The idea presented in Show Attend and Tell (spatial attention during caption generation) is clearly good and if compared properly I'm confident would turn out to work better. Also, the Berkeley paper seemed to have a nice architecture where there was a 2-layer LSTM but the image was only plugged in on the second layer (1st layer was an image-independent Language Model). In their controlled experiments this seemed to work well, I think it's an interesting idea, and I'd want to try to reproduce it. The models presented in my work and Oriol's are the simplest architecture that has the core nugget of the approach and that gets the job done, but I'd expect many bells and whistles on top of this to work better.
Agreed, isolating the contribution of the "model" itself can be borderline impossible, and the details of model engineering have a considerable impact on performance.
That's why I like Kaggle competitions: most approaches tried on a given challenge will likely be optimized to their very limits, so that it's the nature of the different approaches that ends up making the difference.
This algorithm needs badly temporal dimension, some kind of short term memory that lets it interpret using context. At the very least to filter out freaky readings of a train station when looking at the ground, best case scenario it would enable building deeper understating of its surroundings. Maybe not even memory, but Bayes filter to prime next estimation. Then throw movies at it.
Even as it is this could be adapted for the blind. I can imagine app that will simply build a model of what it sees and answer questions or warn about stairs/walls/roads/other dangers. There isnt all that much to make it as clever as a guide dog.
Is there a way to let the algorithm not make a guess if it doesn't find a strong match? I guess it would be better to not output anything than to output gibberish, right?
Thank you. You mentioned that you plan on adding a re-ranker. Is that a re-ranker that encourages diversity? Just like what is done in this paper: http://arxiv.org/pdf/1510.03055.pdf
23 comments
[ 4.9 ms ] story [ 51.6 ms ] threadhttps://vimeo.com/146492001
Something people don't fully appreciate about neural networks is that their performance is quite a strong function of their training data. In this case the training data is taken from the MS COCO dataset (http://mscoco.org/explore/). That's why, for example, when Kyle points the camera at himself the model says something along the lines of "man with a suit and tie" - there is a very strong correlation between that kind of an image in the data, and the presence of a suit and tie. With such a strong correlation the model doesn't have a chance to tease the two concepts apart. A similar problem would come up with an ImageNet model, where a similar image might be classified as "seatbelt", because there is no Person class there, and shots of people in that pose usually come from the seatbelt class. It happens to be the most similar concept in the data it has seen. Another example is if you pointed the model at trees it might hallucinate a giraffe, since the two are strongly correlated in the data. Or when Kyle points the camera at the ground I'm fully expecting it to say relatively random things, because I know that those kinds of images are very rare in the training data.
In other words, a lot of the "mistakes" are limitations of training data and its variety rather than something to do with the model itself, and it's easier to recognize this if you're familiar with the training data and its classes and distribution.
One simple low-hanging-fruit approach would be to include a large repository of additional data (e.g. all of ImageNet) and label it all as a "garbage" class. This way the model could at least learn to distinguish the kinds of images in its training data from the universe of images, and this could be used as one proxy of confidence.
Another simple proxy is to look at the probability of the generated sample, since usually the model tends to assign more diffuse probabilities in more uncertain cases. But this is also not a very clean approach for various reasons.
Another, and probably most appealing, approach would be something along the lines of Bayesian Neural Networks, ensembles, or approximations with dropout, where the disagreement between the predictions of all submodels can be used.
One more idea would be to have a (non-differentiable / REINFORCE?) penalty based on sentence likelihood using doc2vec or skip-thoughts to avoid the "blah and a blah on a blah" type errors that seem to be common in captioning.
One more would be to use TV / YouTube captions, but that data is extremely noisy - even more the COCO captions, unfortunately!
[1] http://arxiv.org/abs/1511.03719
It feels like getting something for free :) Of course there is a limit to how much signal you can extract from a noisy dataset, but the amount of time and human energy invested into creating and improving datasets can be quite large relative to finding another cool trick that can improve performance.
However, I wonder which will come first to make these systems "robust" for the average joe's real world uses for these perceptual systems; a large, well labeled dataset or more transformations and semi-supervised learning approaches?
[1] http://www.cs.stanford.edu/~acoates/papers/coatesleeng_aista... [2] http://cseweb.ucsd.edu/~elkan/posonly.pdf
If so, I'm guessing Google, maybe Facebook too, has plenty of data. What else is holding them back?
Also it's not only the size of the dataset, it's also the size/variety in the label space. ImageNet is quite comprehensive, with many varied labels. MS COCO is quite biased towards a narrow ~hundred classes.
I'd love to see a properly large dataset of images "from the wild", with no restrictions on content (unlike what is done in MS COCO), annotated with sentences. From my experience with adding data to models in these situations I'm quite certain this would work _significantly_ better.
Google's Show and Tell seems considerably superior to competing approaches.
What's different between the two is the engineering portion: NeuralTalk was written in Python and ran on CPU without batches (so probably it did not converge as well), it did not do CNN finetuning (which helps a ton), and it did not do ensembles, and I used VGG while Oriol used GoogLeNet (though I don't expect much difference there). There are a few more tricks that Oriol talks about that give you a few small extra points, but that's basically it. I also don't want to take away from Oriol's top result (these small tricks and engineering are very valuable and hard to come up with), but I would be hesitant to draw conclusions about which models work best by looking at the model diagrams in a Figure, and comparing to numbers in the table.
That's why I've over time grown cynical about results in tables of papers that compare one work to another - there are too many variables to keep track of and it's confusing unless you know the full details. The truth is that results in tables are model + engineering + noise. The papers pretend that it's all model, but in fact the latter 2 have a huge impact. At least results that only compare two models in the same framework can be trusted a bit more to contain information, but even then you find that in practice people can be a lot more generous to their own models than their baselines. Hence the famous saying: "The second best model in the paper is in fact what you want to use".
To answer the original question though: The idea presented in Show Attend and Tell (spatial attention during caption generation) is clearly good and if compared properly I'm confident would turn out to work better. Also, the Berkeley paper seemed to have a nice architecture where there was a 2-layer LSTM but the image was only plugged in on the second layer (1st layer was an image-independent Language Model). In their controlled experiments this seemed to work well, I think it's an interesting idea, and I'd want to try to reproduce it. The models presented in my work and Oriol's are the simplest architecture that has the core nugget of the approach and that gets the job done, but I'd expect many bells and whistles on top of this to work better.
That's why I like Kaggle competitions: most approaches tried on a given challenge will likely be optimized to their very limits, so that it's the nature of the different approaches that ends up making the difference.
This algorithm needs badly temporal dimension, some kind of short term memory that lets it interpret using context. At the very least to filter out freaky readings of a train station when looking at the ground, best case scenario it would enable building deeper understating of its surroundings. Maybe not even memory, but Bayes filter to prime next estimation. Then throw movies at it.
Even as it is this could be adapted for the blind. I can imagine app that will simply build a model of what it sees and answer questions or warn about stairs/walls/roads/other dangers. There isnt all that much to make it as clever as a guide dog.
Do you plan to add beamsearch?