2) It seems quite likely that they've just identified the problems described and fixed by [0, 1] (of which [0] is the much better paper).
In brief, CNNs guarantee translation invariance (ignoring boundary effects) through translation equivariance of their feature maps; this borks certain problems, most importantly a commonly used mechanism for instance segmentation as described in [0]. Consider trying to instance-segment two copy/pasted faces; translation equivariance and global pooling clearly make it impossible.
I strongly suspect that the problem described in the paper originates from a similar place, and thus could be solved in a week or so. I doubt it's a "key weakness of modern cv systems".
EDIT - a little further investigation shows that it was accepted as a workshop paper at ICLR, also provides the reviewers comments. https://openreview.net/forum?id=HymuJz-A-
So perhaps I'm being overly negative. Can't find any code release for it, would be interested if anyone can.
EDIT2 - Entertainingly, the workshop paper version of this and the CoordConv paper seem to use the same name for different toy tasks, "Not-so-CLEVR".
Thanks for the link to the reviewers' comments. I am not an academic, and I found some of the objections surprising, such as this one that might be seen as suggesting that the authors just had the wrong attitude:
"However, I think there's some chance that if the same tasks were in the hands of people who wanted CNNs or CNN+RN to work well, the results might have been different."
The reviewer goes on to write that he cannot say what people with the proper attitude would do differently.
The examples used in the article (in some cases the objects were identical; sometimes they were the same but with one object rotated in relation to the other) are solved very easily with OpenCV "template matching".
This is a toy/artificial example to emphasize and isolate a specific problem with CNNs. The reasoning behind it is that other tasks, for which CNNs are used in practice, would benefit from an ability to do well on this kind of matching tasks.
I view this not as a problem with CNN, but a problem with researchers picking the wrong tools. I've witnessed this behavior at large orgs and startups alike. Taking a model or approach, and feeding it input it's not designed for, is not good science.
It's much more effective and economical to build a simple hybrid which knows when to use CNNs vs other methods - and ensemble the results.
This is more like fundamental research, not applied problem-solving, engineering-like research.
When you know that you have a problem at hand, where template matching would be a better approach, you can use that. However, it could be that more complex tasks (where usually CNNs are used) for which it's not immediately obvious that this capability is required, would also benefit from this capability.
Ensembles are a practical solution, but it's kind of ugly. Remember, this is research into limitations and capabilities, not about solving particular tasks. Once the issue is identified it can be tackled and the resulting model may be nicer and more elegant than an ensemble.
(ml expert here)
This doesn't seem true. Neural networks are great at detection, so they should be similarly good at determining whether two objects are identical or not. The fact that their networks didn't do better than chance (as opposed to a "disappointing 10%" error rate") suggests that they trained their networks wrong.
There's only one problem in computer vision, and it's not unique to the field at all: balancing sensitivity vs specificity. Modern applications of ML don't solve that.
The authors didn't try very hard to solve the problem (although it's still slightly interesting).
Many networks can detect if two never-before-seen objects are the same or different. Simple example is Koch's Siamese Neural Networks[1]. More useful example is Facenet[2], which can tell if two faces are the same or different, even if never trained on these specific faces. Facenet learns an n-dimensional embedding and similarity is estimated by euclidean distance. These are cool papers by the way. The Omniglot dataset[3] used in Koch is a sort of an "inverse MNIST". Omniglot has 20 examples each of 1600 different characters, vs MNIST's 60,000 examples of 10 different characters.
This just goes to show how messed up the whole AI field is right now. It shouldn't be about "trying hard". Either they used the right network for their analysis, or they didn't.
It should be possible to do negative analysis of AI capabilities. But since there is almost no theory behind this stuff right now (just test sets and architectures targeting particular test sets), anyone can always claim that any negative analysis just didn't target the "right" architecture.
This does seem to reduce to one-shot learning, but face dissimilarity metrics won't work well to describe similarity between unstructured objects like the authors propose.
Faces have a limited set of common features that are readily separable, quantifiable, and comparable, which is why face recognition can be done rather well just using a SVM. When the set of common features falls to zero, as in this paper, a face-specific net architecture will not do one-shot recognition across independent faceted objects nearly as well.
I think you've missed the point slightly. This paper looks at two cases, one of which is the "two images, two embeddings, one operation" (eg siamese-style). They note that this works well. They then go on to note that the problem of "are these two components at aribtrary locations of an otherwise blank image the same or different" is not well solved. This cannot be addressed directly by Siamese nets (well, or at least kind-of-can't. Arguably relational networks are similar to taking pixel pairwise functions, and thus have some relation to Siamese nets.)
In any case, they directly exclude the case you mention.
My comment was on the article, which was flat out wrong, and thanks for pointing that out in the paper I had missed that.
Their paper was still unimaginative. Trying a handful of vanilla networks doesn’t prove anything.
The right framing would have been, “here’s an interesting case to work on” instead of pretending to have identified a basic limitation of neural networks. This is more like a research idea than research.
"Instead of showing the computer two objects in the same image, the researchers showed the computer the objects one at a time in separate images. The experiments showed that the algorithms had no problem learning same-or-different relationship as long as they didn’t have to view the two objects in the same image."
There is a lot of research on attention and saliency mechanisms. Itti, Koch, Niebur had a model. Adaptive resonance theory from Grossberg has a successive winner take all strategy as well. Global workspace theory from Baars similarly. It is a very logical cognitive architecture to come up with. If I'm correct also the new architecture from Hinton separates object recognition from the position information. That these are separate pathways in mammals is also already long known.
It would have been nice if they would have come up with a method in which these separate pathways and/or a sequential attention mechanism would arise from more basic principles and not hardcoded by researchers.
From what I understand, the task of their toy experiment is to identify multiple objects in an image. This can be easily solved by simple computer vision algorithms.
The proper solution can be implemented as following. Firstly either using sliding window or selective search like algorithm to generate potential object targets, secondly using trained object models on these targets.
In my opinion, it seems that the researchers didn't know how to use cnn.
22 comments
[ 3.1 ms ] story [ 68.5 ms ] thread1) It's not published in a good CV journal.
2) It seems quite likely that they've just identified the problems described and fixed by [0, 1] (of which [0] is the much better paper).
In brief, CNNs guarantee translation invariance (ignoring boundary effects) through translation equivariance of their feature maps; this borks certain problems, most importantly a commonly used mechanism for instance segmentation as described in [0]. Consider trying to instance-segment two copy/pasted faces; translation equivariance and global pooling clearly make it impossible.
I strongly suspect that the problem described in the paper originates from a similar place, and thus could be solved in a week or so. I doubt it's a "key weakness of modern cv systems".
[0] https://arxiv.org/abs/1807.10712 [1] https://arxiv.org/abs/1807.03247
EDIT - a little further investigation shows that it was accepted as a workshop paper at ICLR, also provides the reviewers comments. https://openreview.net/forum?id=HymuJz-A-
So perhaps I'm being overly negative. Can't find any code release for it, would be interested if anyone can.
EDIT2 - Entertainingly, the workshop paper version of this and the CoordConv paper seem to use the same name for different toy tasks, "Not-so-CLEVR".
"However, I think there's some chance that if the same tasks were in the hands of people who wanted CNNs or CNN+RN to work well, the results might have been different."
The reviewer goes on to write that he cannot say what people with the proper attitude would do differently.
I guess that's an undesirable symptom of CNNs (etc.) still being an empirical rather than theoretical science.
The examples used in the article (in some cases the objects were identical; sometimes they were the same but with one object rotated in relation to the other) are solved very easily with OpenCV "template matching".
It's much more effective and economical to build a simple hybrid which knows when to use CNNs vs other methods - and ensemble the results.
When you know that you have a problem at hand, where template matching would be a better approach, you can use that. However, it could be that more complex tasks (where usually CNNs are used) for which it's not immediately obvious that this capability is required, would also benefit from this capability.
Ensembles are a practical solution, but it's kind of ugly. Remember, this is research into limitations and capabilities, not about solving particular tasks. Once the issue is identified it can be tackled and the resulting model may be nicer and more elegant than an ensemble.
There's only one problem in computer vision, and it's not unique to the field at all: balancing sensitivity vs specificity. Modern applications of ML don't solve that.
Many networks can detect if two never-before-seen objects are the same or different. Simple example is Koch's Siamese Neural Networks[1]. More useful example is Facenet[2], which can tell if two faces are the same or different, even if never trained on these specific faces. Facenet learns an n-dimensional embedding and similarity is estimated by euclidean distance. These are cool papers by the way. The Omniglot dataset[3] used in Koch is a sort of an "inverse MNIST". Omniglot has 20 examples each of 1600 different characters, vs MNIST's 60,000 examples of 10 different characters.
[1] https://www.cs.cmu.edu/~rsalakhu/papers/oneshot1.pdf
[2] https://arxiv.org/abs/1503.03832
[3] https://github.com/brendenlake/omniglot
It should be possible to do negative analysis of AI capabilities. But since there is almost no theory behind this stuff right now (just test sets and architectures targeting particular test sets), anyone can always claim that any negative analysis just didn't target the "right" architecture.
Faces have a limited set of common features that are readily separable, quantifiable, and comparable, which is why face recognition can be done rather well just using a SVM. When the set of common features falls to zero, as in this paper, a face-specific net architecture will not do one-shot recognition across independent faceted objects nearly as well.
In any case, they directly exclude the case you mention.
Their paper was still unimaginative. Trying a handful of vanilla networks doesn’t prove anything.
The right framing would have been, “here’s an interesting case to work on” instead of pretending to have identified a basic limitation of neural networks. This is more like a research idea than research.
To be honest, I skipped it and went to the paper :) Apologies!
if the answer is no, then I don’t think that even feedback would help in the original problem..
“When computers look at an image, they can’t actually tell where one object in the image stops”.
“When children read French, they actually don’t know what the foreign words mean.”
Just means: Todo: teach them to.
There is a lot of research on attention and saliency mechanisms. Itti, Koch, Niebur had a model. Adaptive resonance theory from Grossberg has a successive winner take all strategy as well. Global workspace theory from Baars similarly. It is a very logical cognitive architecture to come up with. If I'm correct also the new architecture from Hinton separates object recognition from the position information. That these are separate pathways in mammals is also already long known.
It would have been nice if they would have come up with a method in which these separate pathways and/or a sequential attention mechanism would arise from more basic principles and not hardcoded by researchers.
The proper solution can be implemented as following. Firstly either using sliding window or selective search like algorithm to generate potential object targets, secondly using trained object models on these targets.
In my opinion, it seems that the researchers didn't know how to use cnn.