This shouldn't be news. Most computer vision NNs are CNNs where the convolutions are basically translating the image into small "textures". "AI sees textures" yeah because that's how we present data to it!
At that point it's less a task in the context of image processing and more one of graph learning / language processing, depending on how you want to formulate it.
Human vision certainly contains a component where we do 3d reconstruction and adjust our perception based on that. Many optical illusions depend on this step.
If we want more human like machine vision then having passes in image processing that deal with more abstract data sounds like a great idea
I'm a little confused by the assertion of "artificial intelligence's preference for texture over shape". The article makes it sound like an intrinsic issue with AI, when to my understanding it's just an outcome of popular image classification algorithms. Couldn't one create classifiers that depend more heavily on object shapes?
I'd guess animals have a lot of compute dedicated to finding out the shape based off of various visual cues:
Lighting changes indicating edges
Gradients indicating a smooth curve
Texture maps that discern between matte, smooth, translucent, often in depth enough to determine the texture material itself.
Reflectivity
Changes in reflectivity when reorienting oneself or the object
Binocular Vision
Parallax effect between frames
A large database matching objects to memories and experiences
Many other techniques for tracking and seeing movement.
In the above, Parallax and Binocular vision, and consulting one's memory are the only ways to determine the sizes of the objects being looked at.
It's why those optical illusions tiny/big rooms work so easily; neither we nor CV algorithms would be able to discern the sizes of objects without prior memory of such objects or by walking around it.
Yes, maybe. But it is at least unclear whether is inherently CNN or not (many people would vote for no). ImageNet training data is heavily texture dependent, therefore, the trained model leaning towards texture information more. Some recent papers did style transfer to randomize textures in order to train the model to focus more on the shape, the reported results seem good.
Exactly. I would also add (although without strong evidence to support my assertion) that it seems to be more a function of the training data than of the network architecture.
One of the most archetypal failure modes of current NN-based classifiers is, for example, happily reporting seeing a spotted feline given a picture of a couch with a leopard pattern.
Well, if I'm given the task to enumerate what I see in that image maybe I would also add the feline label. But I would add the picture_of attribute, if that's a thing.
I don’t think today’s classifiers are anywhere sophisticated enough to have a concept of something vs. a picture of something, a type of use–mention distinction, or ceci n’est un pipe if you will. The point was that people think they trained their classifier to recognize leopards, but in actuality it just learned to recognize the leopard coat pattern.
These networks are correlation-finding machines, and will simply latch onto the simplest correlation possible that produces the correct results. Only by explicitly controlling the correlations in the training data can you force the network to focus on the attributes that you wish.
This is why domain-randomization (such as the approach that NVIDIA is taking) helps support generalization -- it's a very direct and efficient form of regularization, removing correlations that do not generalize, forcing the network to work harder to find correlations that do generalize.
This also has the added benefit of being something that we can tie back to requirements and physical properties, an important ingredient in making these systems understood and safe.
My take on this: if we want vision ML to succeed at recognition in the same way as humans, perhaps we need to pre-process and present visual information in the same way as the human vision system? As far as I'm aware, we get a lot of info from our eyes about lines and orientation that assists in recognizing shapes.
I'm not well-informed about the current state of visual recognition DL, perhaps someone who is can tell us more about whether that approach makes sense.
When you train a deep convolutional neural network, the first couple of layers appear to take on this role, detecting simple features like edges and textures, which the higher layers build upon to see more complex objects.
TLDR: The use of texture is inherent to the ImageNet dataset and not to deep learning / ConvNet. Training on less-textured versions of ImageNet drives the ConvNet to focus more on shape.
Sure, but just because you can coerce an algorithm into doing something different does not mean the algorithm itself does not have fundamental tendencies.
There is a misunderstanding here. It's not the algorithm that emphasizes textures over shapes, it's the dataset. Texture is a more informative feature than shape in the datasets used.
"AI" can detect shapes just fine; simply use a different dataset and you'll get a different result with the same algorithm. I mean, just look at the classic MNIST: it's all shape, no texture, and neural nets work great.
One thing that’s interesting about textures is that humans have an incredible sensitivity to it by touch. You can detect the difference between a flat surface and a surface with angstrom-magnitude features.
"The participants probed the surfaces with the index finger of their preferred hand in a designated direction (perpendicular to the grooves) for as long as they wished and at loads and speeds that they established themselves"
"...it is observed that while the minimum pattern that could be distinguished from the unwrinkled reference surfaces had a wavelength of 760 nm, the amplitude of this pattern was only 13 nm.
This shows unambiguously that the human finger, with its coarse fingerprint structure in the sub-millimeter range, is capable of dynamically detecting surface structures many orders of magnitude smaller and indicates that nanotechnology may well have a role to play in haptics and tactile perception."
> Texture is a more informative feature than shape in the datasets used.
But the insight is that although both features are informative, classical training approaches result in networks that over-fit on texture information and ignore shape information.
Table 2 of the discussed paper finds that joint training with "Stylized ImageNet" (IN images after a style transfer) and classic IN followed by a fine-tuning pass on IN results in improved classification accuracy over IN training alone.
Not necessarily. Textures are high frequency low abstraction data forms, shapes are low frequency high abstraction data forms.
The former are easier to learn for multiple reasons:
The nature of higher frequency data means that you have more samples.
(There are far more furry textures in a picture of a cat than there are toes, paws, legs etc.)
Low abstraction data forms are also learnt in earlier layers of the network - more abstract forms are resolved through the understanding of how less abstract forms group together, making the low abstraction forms a prerequisite to more in depth learning.
There's some truth to this, but it's not the whole story. CNNs have limitations in terms of what they can represent and what they can learn. Counting is an example of this. If I create a class of monsters with 10 eyes, and another that is otherwise identical, but with 11 eyes, these will be difficult classes to tell apart.
When it comes to "shape", part of the problem is definitional. In the MNIST example, the network will fairly easily learn features that are nodes, like where two lines intersect, but it has trouble with paths, like "an unbroken line that smoothly intersects itself." CNNs have trouble distinguishing things like a spiral from a set of concentric circles because locally they look the same.
It seems to me that the number of eyes problem should be solvable. Just convolve with an eye detector and threshold appropriately. Then feed everything to a single neuron in the next layer and threshold at 10.5. No?
Solvable with human guidance, yes, but would any existing machine learning system come up with a successful strategy like this by itself? Most deep dream style images suggest the networks don't even realise that dogs have two eyes.
Texture is a more informative feature than shape in the datasets used.
TL;DR; Logical reasoning isn't based on a "prepondernace of informative." Human-like image-recognition seems, to me, to quire both sort-of statistical reasoning, which depends on the preponderance of information and logical reasoning, which doesn't (not directly).
That's seems sort-of true but "Informative" has a number of different implication. It seems pretty inevitable that a huge image dataset is going to carry the most data in the form of textures.
The thing, a human being can (often) extract considerable meaning from an image of a stick figure or a black and white outline of a big cat, neither of which involves that much data. Effectively, human image recognition is able to both use texture-qualities as a rough guestimate for the source of an image but also look at the overall image structure and see "differences that make a difference". I mean, I can look at a puzzle piece and determine it probably goes in particular area but that's only one facility, "knowing" what an image "is", involves more.
The key part is the 'bag of features' discussion. It's SUPER interesting work, demonstrating that classification works just as well if you cut up the image into square chunks and scramble them - the convolutional activations end up roughly similar, but all sense of shape is lost (as is any human recognition of the scrambled objects).
I think MNIST is too simple to have much bearing on the problem. MNIST shapes are two-dimensional; actual object classification requires 3D shapes (and all possible projections thereof), which is a much harder problem.
Since this is just a message board discussion, my wild conjecture would be that the model has no idea about the existence of the third dimension, or basic physical concepts like lighting and distance. And without that larger world model in which to situate things, texture is maybe the easiest thing to cling to.
The "bag of features" work seems plausible but I recall from a reddit discussion that "just as well" might be stretching it. I recall it worked OK for a fairly small set but not for a larger set. Still, some references would be nice.
I never saw a plane made of lego bricks but I recognized it instantaneously when I saw it for the first time. This hints about a different understanding process in deep learning and human brains. This is one of the points of the article.
Actually I don't think humans vision is based only on shapes or textures. There are many novel shapes of planes and we know that's a plane no matter what. We use understanding, with our knowledge of the world, not only the information coming from the eyes. I think it is much larger than what is encoded in a deep learning neural network.
Yes, you have only to look at the images generated by GANs to see that it's not inherent in the algorithm: images generated by GANs are not just a patchwork of textures, they have coherent shapes. They 'needed' shape in order to fool the discriminator, which in turn developed a sensitivity to shape in order to identify generated images (which probably do rely on texture early on) as fakes.
It seems they reach healthy adult sharpness by 6 months [1]. By the time they are sitting and starting to move they are certainly extremely good at finding tiny bits of fluff etc. to put in their mouths!
Does this have anything to do with AIs always learning from 2D photos when we learn from binocular vision so every image contains a lot more ‘shape’ information.
It seems like that's fairly important to me. Consider that even if you're monocular, you're still able to determine parallax by moving your head around to determine depth.
Not only this, but we when we see things even when we are still our eyes/head move a small amount. Shape information is preserved in these multiple perspectives relatively more than texture information.
49 comments
[ 2.9 ms ] story [ 106 ms ] threadIf we want more human like machine vision then having passes in image processing that deal with more abstract data sounds like a great idea
https://github.com/tensorflow/magenta/blob/master/magenta/mo...
Lighting changes indicating edges
Gradients indicating a smooth curve
Texture maps that discern between matte, smooth, translucent, often in depth enough to determine the texture material itself.
Reflectivity
Changes in reflectivity when reorienting oneself or the object
Binocular Vision
Parallax effect between frames
A large database matching objects to memories and experiences
Many other techniques for tracking and seeing movement.
In the above, Parallax and Binocular vision, and consulting one's memory are the only ways to determine the sizes of the objects being looked at.
It's why those optical illusions tiny/big rooms work so easily; neither we nor CV algorithms would be able to discern the sizes of objects without prior memory of such objects or by walking around it.
a good AI will eventually need to be able to identify this subtle difference, it is an elephant, made of cats, not a cat and not an elephant.
These networks are correlation-finding machines, and will simply latch onto the simplest correlation possible that produces the correct results. Only by explicitly controlling the correlations in the training data can you force the network to focus on the attributes that you wish.
This is why domain-randomization (such as the approach that NVIDIA is taking) helps support generalization -- it's a very direct and efficient form of regularization, removing correlations that do not generalize, forcing the network to work harder to find correlations that do generalize.
This also has the added benefit of being something that we can tie back to requirements and physical properties, an important ingredient in making these systems understood and safe.
I'm not well-informed about the current state of visual recognition DL, perhaps someone who is can tell us more about whether that approach makes sense.
For example https://www.researchgate.net/figure/Visualization-of-example..., where you can see (somewhat, if you zoom in) that layer 1 neurons are interested in very simple features, like strong horizontal edges, or particular gradients.
TLDR: The use of texture is inherent to the ImageNet dataset and not to deep learning / ConvNet. Training on less-textured versions of ImageNet drives the ConvNet to focus more on shape.
"AI" can detect shapes just fine; simply use a different dataset and you'll get a different result with the same algorithm. I mean, just look at the classic MNIST: it's all shape, no texture, and neural nets work great.
"...it is observed that while the minimum pattern that could be distinguished from the unwrinkled reference surfaces had a wavelength of 760 nm, the amplitude of this pattern was only 13 nm.
This shows unambiguously that the human finger, with its coarse fingerprint structure in the sub-millimeter range, is capable of dynamically detecting surface structures many orders of magnitude smaller and indicates that nanotechnology may well have a role to play in haptics and tactile perception."
[1] Feeling Small: Exploring the Tactile Perception Limits https://www.nature.com/articles/srep02617
But the insight is that although both features are informative, classical training approaches result in networks that over-fit on texture information and ignore shape information.
Table 2 of the discussed paper finds that joint training with "Stylized ImageNet" (IN images after a style transfer) and classic IN followed by a fine-tuning pass on IN results in improved classification accuracy over IN training alone.
The former are easier to learn for multiple reasons: The nature of higher frequency data means that you have more samples.
(There are far more furry textures in a picture of a cat than there are toes, paws, legs etc.)
Low abstraction data forms are also learnt in earlier layers of the network - more abstract forms are resolved through the understanding of how less abstract forms group together, making the low abstraction forms a prerequisite to more in depth learning.
When it comes to "shape", part of the problem is definitional. In the MNIST example, the network will fairly easily learn features that are nodes, like where two lines intersect, but it has trouble with paths, like "an unbroken line that smoothly intersects itself." CNNs have trouble distinguishing things like a spiral from a set of concentric circles because locally they look the same.
So does the human visual system; see e.g. https://en.wikipedia.org/wiki/Fraser_spiral_illusion .
The spiral vs. circles I agree with.
TL;DR; Logical reasoning isn't based on a "prepondernace of informative." Human-like image-recognition seems, to me, to quire both sort-of statistical reasoning, which depends on the preponderance of information and logical reasoning, which doesn't (not directly).
That's seems sort-of true but "Informative" has a number of different implication. It seems pretty inevitable that a huge image dataset is going to carry the most data in the form of textures.
The thing, a human being can (often) extract considerable meaning from an image of a stick figure or a black and white outline of a big cat, neither of which involves that much data. Effectively, human image recognition is able to both use texture-qualities as a rough guestimate for the source of an image but also look at the overall image structure and see "differences that make a difference". I mean, I can look at a puzzle piece and determine it probably goes in particular area but that's only one facility, "knowing" what an image "is", involves more.
I think MNIST is too simple to have much bearing on the problem. MNIST shapes are two-dimensional; actual object classification requires 3D shapes (and all possible projections thereof), which is a much harder problem.
Since this is just a message board discussion, my wild conjecture would be that the model has no idea about the existence of the third dimension, or basic physical concepts like lighting and distance. And without that larger world model in which to situate things, texture is maybe the easiest thing to cling to.
Actually I don't think humans vision is based only on shapes or textures. There are many novel shapes of planes and we know that's a plane no matter what. We use understanding, with our knowledge of the world, not only the information coming from the eyes. I think it is much larger than what is encoded in a deep learning neural network.
Understanding shapes/outlines is useful for manipulating objects and avoiding obstacles.
Maybe the object recognition develops a bit later and builds on top of how brains already understand the world.
[1] https://en.wikipedia.org/wiki/Infant_visual_development
Doesn't quite seem like a break through insight to me?