Somehow tangent but this made me think about this quote found on HN last year:
Context: Evolutionary algorithms and analog electronic circuits
> One thing stands out when you try playing with evolutionary systems.
Evolution is _really_ good at gaming the system. Unless you are very careful at specifying all of the constraints that you care about you can end up with a solution that is very clever but not quite what you had in mind. Here power consumption is the issue. If you tried to evolve a sturdy chair you might end up with something that is 1mm tall. or maybe a fuel efficient car that exploits continental drift.
I think it's the same here: The net is never gonna better than what it needs to be, and it is probably always gonna take the easy route.
You don't even need a neural net for that, take any global optimization method and give it a somewhat ill-defined scoring function, it will instantly run circles around you laughing.
There's an alife program called DarwinBots where small bots powered by mutating code compete against each other to survive and reproduce.
Given enough time, you'd expect the to develop clever behaviors, but instead they just fuzz-tested the sim and locked in on exploits of bugs or environment settings. They only got a bit more clever when connecting different sims running on different conditions.
Eyes already use different kinds and densities of sensors optimized for either detail and color or movement/edges. I wouldn't expect a single learning method, even after optimizing it to its limits, to be above what two or more layers of different methods could do, especially when trying to avoid exploits like the tank story.
Given enough time, you'd expect the to develop clever behaviors, but instead they just fuzz-tested the sim and locked in on exploits of bugs or environment settings.
Classic A-life! Also, not so different from the spirit of actual biology.
They only got a bit more clever when connecting different sims running on different conditions.
Diversity is very important for evolution on many levels. What many don't realize (especially, I note, evolution deniers) is that the ecosystem as a whole provides a very complex and continually varying epiphenomenal fitness function to any given organism.
If you don't have a sufficiently complex genotype phenotype mapping and the system is not evolvabke (See Gunter Wager's work) the you shouldn't expect more complex phenotype. Understanding a genetic representation is going to be an important step toward open ended evolutionary systems.
Darwinbots uses actual computer code to control the robots. This makes it really hard for evolution to work with. Most mutations just break the code, and very very few mutations create anything interesting. And the simulation is too slow to explore millions of different possibilities to make up for the difficulty. What makes it worse is they are usually asexual.
However I think that's ok. Most of the fun with darwinbots is programming your own bots. They used to be (still are?) competitions where people wrote their own bots and had them compete under different conditions.
> They only got a bit more clever when connecting different sims running on different conditions.
Part of the reason why a lot of these nets are trained with added noise, as well as drop-out (randomly disabling 50% of the hidden neurons, every training step).
Especially the drop-out tactic is particularly effective at preventing "exploits" of the neural net type, which otherwise appear in the form of large correlated weights (really big weights depending on other really big opposite weights to cancel out--it works, but it doesn't help learning).
Either way, adding noisy hurdles helps because exploits are usually edge cases, and noise makes them less dependable, as the region of fitness space very close to an exploitable spot, is usually not very high-ranking at all (which is why you don't want your classifiers ending up there).
From the outside, there appear to be significant advances; it's just they seem to come in clusters, rather than linearly. The news has turned into science fiction!
It would seem CNNs were a significant step up, but the author hints at inferring structure as the next tack to take.
From the outside, there appear to be significant advances; it's just they seem to come in clusters, rather than linearly. The news has turned into science fiction!
It would seem CNNs were a significant step up, but the author hints at inferring structure as the next tack to take.
I'm not sure it's absurdly overblown but individual advances/findings can be way overhyped or at least over-generalized. ML/AI has been incrementally delivering pretty impressive results within certain constraints. That's great but there's then a widespread tendency to extrapolate those results to the broader case--and then absurd/stupid-looking results happen.
We certainly see the same thing with autonomous vehicles. Given very accurate mapping and a particular set of environmental and type-of-road conditions, cars can do so well that it's tempting to say they're 95% of the way to fully-autonomous. But dump them in a Boston snowstorm and you see they're really not even close. (Which isn't to say that bounded use cases can't be very useful.)
Voice recognition has gotten a lot better. I'm almost impressed by my Amazon Echo. That said, for an arbitrary recording of say a conference presentation, you need to either use a human transcriber or expect to spend a LOT of time cleaning things up.
(A lot probably has to do with switching to more data-based approaches.)
I tried the unrotated sofa image on Wolfram's ImageIdentify and it correctly identified a settee [1]. So it presumably gathered that from the shape of the image rather than the pattern. It is peculiar though that it can't see the shape under a simple rotation. Or perhaps the margin of confidence levels between sofa and leopard were so narrow that a rotation was enough to tip it in favour of the leopard? I'd be interested to see the inner workings of this.
I had some success with pictures of sofas from a top view, but a lot came out as complete nonsense like "nail" or "light bulb". Seems on at least some patterns it is trained to view things in a particular orientation that you would normally see them in. I imagine that if it did exhaustive rotation searches on a lot of objects the results would often be completely incorrect.
That's right, CNNs tend to be more concerned with textures rather than with overall shapes. One problem is that CNNs discard a lot of pose information of the detected features during pooling. Another problem is that there is no top-down verification such as "hmm, leopards always have {heads, legs, tails ...}, I should scan the input for these... nope, it doesn't fit at all, I should exclude everything that has {heads, legs, tails ...} from my interpretation." In a human brain that likely happens in some distributed fashion without considering individual classes, but by just inhibiting everything that can't be verified upon looking twice (or more times).
> So I guess, there's still a lot of work to be done.
And I think this is the most interesting part.
One of the most depressing things about all of the "this image recognition algorithm performs better than humans on this task" is the idea that we've pretty much solved the problem, and it's just a matter of some more optimization and tweaking to handle a few edge cases.
This kind of problem, where the dominant solution simply gets it so wrong, and the problem cases are uncommon enough that any statistical solution is generally going to treat them as noise, reveals that in fact that there is likely plenty of room for entirely new, novel ways of approaching the problem to handle these kinds of cases better.
It's actually more exciting that there's so much more to be done, than to say "well, it's basically a solved problem, we just need to do some tweaking and optimization."
Definitely agree here, want to add a link to this paper which shows how far we still have to go (and questions whether the current models will ever replicate human vision): http://arxiv.org/abs/1412.1897
This reminds me of one of Richard Feynman's famous quotes: “We are trying to prove ourselves wrong as quickly as possible, because only in that way can we find progress.”
Indeed, discovering these "broken" edge cases is exactly what we need to converge upon a more correct solution.
I agree. I don't work with machine learning or neural nets, and thus don't have more than the most cursory laymans understanding of them. This article read really well and was quite informative of the problems this technology is facing.
This article would not come as a surprise to anyone who works with ConvNets. Sadly, that might not the case for those outside of the field, largely due to media's inadequate coverage of our advances (but this is common outside our field too). No one in the field really believes ConvNets see better than humans. They are very good single glance texture recognizers. It's as if you flashed an image and looked at it for a split second without giving yourself a chance to look around and take some time to gain any higher-level scene understanding. If you tried this with this image you might also think you had seen a leopard. Another point to make is not from modeling side but from data side. If in the training data the leopard texture is highly indicative of leopard, then the ConvNet will learn to strongly associate it as such. As the article mentions, a quick hack would be to make sure that your training data contains many leopard-textured items of different classes. You might then expect the ConvNet to seek other features to latch on to and become less reliant on the texture itself.
Also, we carried out an experiment on ImageNet and the outcome was that "One human labeler (me, incidentally) with a fixed amount of training and a slightly-above average determination reached ~5% top-5 error on a subset of ImageNet test set". The media sees this and it immediately gets spun to "AI now Super-Human. And we're all going to die." It makes a lot of us cringe every time.
Many people in Computer Vision now consider ImageNet "squeezed" out of juice - we're good at texture recognition and when an object is in plain view, and we're now searching for harder tasks and more dynamic range with respect to human performance, in areas such as harder 3D/Spatial tasks, Image Captioning, Visual Q&A, etc. The hope is that these harder datasets might in turn guide us in developing models with more nuanced understanding.
Hmm, in your opinion do you think this would be a good technique then for digitizing paper maps? And if so, could you point in the direction of a library or textbook you'd recommend?
What if you took these same Neural Networks as they exist now, and tweaked the input and the parameters slightly. For the input, use individual frames of an hour long video of a leopard (in order), and instead of having it just identify whether or not there is a leopard, have it identify what in each image is the leopard, and have it try to predict the next frame.
It seems that this is more like the way that we learn to identify things. Then once we establish an understanding of a base class (big cat) we can apply that same model to new cats that we have never seen before with just a picture.
Might you by chance be familiar with Rodney Brooks' work on subsumption architectures [1]? If not, I would summarize the underlying idea (my words not his) as "don't try to jump too many layers of abstraction in one go" [2].
So I wonder to what extent you would consider this a predictable outcome from the classifier in question not being part of a subsumptive architecture --- which at a guess would look like
- glance/texture responses fed into
- boundary-recognition layers fed into
- object persistence/tracking layers
- fed into abstract scene reasoning
It seems to me, as a non-vision researcher (I mainly worked in planning and control), that the most obvious counterargument to the image being a spotted cat is based on boundary/object/scene reasoning, and that it's "reasonable" for the texture/glance layer to say "looks a lot like a cat texture".
[2] I realize this may seem, superficially, anathema to deep network research, which advocates letting the network find its own intermediate levels of abstraction. But it's actually compatible in my view because Brooks advocates (again, paraphrasing quite a bit) that the separate layers should have different objective functions, and that in fact the need for different objective functions (in a prioritized order) is the cause of emergent layering in nature. "First, don't die. Second, find shelter. Third, find food etc." So one can imagine deep networks each finding their own locally useful abstractions for each objective function in the "Maslow" chain, while still having some macro architecture that tracks human-imposed design principles.
the limit is that training cannot force abstraction. you can only reach abstraction if you have enough neuron space and the data set is big enough to avoid over-fitting textures.
the problem is.. human vision doesn't work just by feeding a bitmap. we have structure to decode space relationships, shapes and maybe even shadow/light relations. no way we gonna see classificator working on color arrays matching our vision capabilities
Seems simple enough to feed a NN with that abstracted data.
However, the advantage to the texture approach is it's abstracted from a lot of other information. You don't want a classifier to say sofa, when it's a picture of a person on a sofa.
No, I actually would not think I saw a leopard. Humans are really good at recognizing things with faces and legs. Those humans that didn't have the ability to recognize a leopard in a split second were already eaten thousands of years ago.
After rotating the image 90 degrees, the predicted result changes substantially. The author should not be surprised. A convolutional neural network is translationally invariant, not rotationally invariant.
Standard convnets do not contain explicit rotational invariance (unless you include a layer such as this: arXiv:1506.02025v1). They can however learn rotational invariance if you feed them rotated images.
Let's say I didn't want to use the ImageNet or CaffeNet pre-trained models but wanted to train my own model (say, of thousands of images of sofas, leopards, jaguars, and cheetahs); are there any tutorials that walk through the process of building a CNN on your own data?
(I've seen the comments like https://news.ycombinator.com/item?id=9584325 and watched the lectures and youtube walkthroughs, but they're all theoretical and I'm looking for documented code to go along with that theory)
Obviously these classifiers do often focus on patterns, rather than shapes, and that's probably something that could be worked on, but I don't think an image classifier can possibly be expected to, at the level it is operating, identify the leopard-print sofa all on its own. Clearly there's a higher order process at work than image recognition here - after all, when a human is faced with a sofa-shaped object with a leopardskin pattern on it, there are two hypotheses that need to be evaluated: 1) this is a sofa patterned to look like a leopard; or 2) this is a leopard, shaped like a sofa. Rejecting the less plausible of those two scenarios is obviously a higher-order activity. If the image classifier is at least firing off the concepts 'leopard' and 'sofa' with some level of probability, it's doing its job pretty well.
I don't work with NN at all, but it kind of seems like the author set up their CNN with a large enough filter to see a whole spot at once, but not large enough to see a whole cat at once. Then he complains that it doesn't know what a cat looks like. Would it be possible to make a larger filter with a lower resolution such that overhead is the same as the smaller filter but it can get a higher-level view of the image?
Also, the author spends the first section of the article determining that it is in fact a jaguar-print sofa (which the model also confirms) but continues to throw around the word "leopard". They're not making it any easier for the future machine learning algorithms that try to identify an image by the text surrounding it. ;)
https://www.imageidentify.com/ correctly identified the images as "a small sofa". I think rotating the image is questionable, since there could be an algorithm for first orienting the image correctly based on light and shadow and then the image recognition could be run.
But those algorithms would be very limited in usefulness. Systems such as imageidentify.com should be at least trying an ensemble of algorithms, many of which I suppose should be invariant under translation and rotation.
Edit: There's a comment about invariance in this thread [1] and apparently CNNs are not invariant under rotation.
It could be that the dataset is so large there's very close matches to any common given picture, and the algorithm is actually awful at picking up details.
Giving it an image that we know has all the relevant details of a sofa, but it likely won't have close matches in the dataset, can give us an idea of how clever it is.
We did not train ImageIdentify to be invariant under arbitrarily large rotations. This is fairly easy to do: show the network couches rotated at all angles.
I felt like Alice down the rabbit hole after getting sucked into reading about TempleOS. Quite an unexpected side-effect of reading an article on neural networks!
I have seen some kaggle competitions do image transformations and put the data back into the training set to increase the robustness of the classifier. For instance, rotating images, slightly skewing them, etc.
I would propose that for this leopard problem, instead of just skewing the images, you also performed transformations on the COLOR and put the images back into the training set.
Maybe applying certain filters, such asdimming the saturation or contrast of images, so that the contrast between the leopoard spots were less visible (i.e. "A Leopoard in low lighting") - maybe this would force the neural net to learn more than just its print.
Knowing the right set of color filters to apply to all images could be tricky though.
I have seen some kaggle competitions do image transformations and put the data back into the training set to increase the robustness of the classifier. For instance, rotating images, slightly skewing them, etc.
I would propose that for this leopard problem, instead of just skewing the images, you also performed transformations on the COLOR and put the images back into the training set.
Maybe applying certain filters, such asdimming the saturation or contrast of images, so that the contrast between the leopoard spots were less visible (i.e. "A Leopoard in low lighting") - maybe this would force the neural net to learn more than just its print.
Knowing the right set of color filters to apply to all images could be tricky though.
The net was able to find tanks hiding in the trees with amazing accuracy. Too amazing. It turned out the photos of the hidden tanks were all photographed on a cloudy day. The images without tanks in a clear day.
They would be if you ran screaming from someone's living room because of the thing in the corner. I think it's because from a practical point of view its a different category of error?
I myself have researched leopard spots since I painted our toilet floor in them. It's a lead sheet, and the paint had worn off, which probably wasn't the healthiest thing. My housemates had filled the toilet with memorabilia from an African trip, so leopard-print paintjob it was.
Which entailed looking up leopardprint online. Very little of which actually looks like leopard rosettes, and now I have a problem with almost anything trying to pass itself off as leopardprint. Anyway, I can't say that my paintjob is a particularly good reproduction, but at least it's 'spiritually correct'... :)
The MNIST analogy reminds me of the "Teaching Me Softly" article that was posted here last year:
> When Vladimir Vapnik teaches his computers to recognize handwriting, he does something similar. While there’s no whispering involved, Vapnik does harness the power of “privileged information.” Passed from student to teacher, parent to child, or colleague to colleague, privileged information encodes knowledge derived from experience. That is what Vapnik was after when he asked Natalia Pavlovich, a professor of Russian poetry, to write poems describing the numbers 5 and 8, for consumption by his learning algorithms. The result sounded like nothing any programmer would write. One of her poems on the number 5 read,
> He is running. He is flying. He is looking ahead. He is swift. He is throwing a spear ahead. He is dangerous. It is slanted to the right. Good snaked-ness. The snake is attacking. It is going to jump and bite. It is free and absolutely open to anything. It shows itself, no kidding.
Brown_Cornerart
> All told, Pavlovich wrote 100 such poems, each on a different example of a handwritten 5 or 8, as shown in the figure to the right. Some had excellent penmanship, others were squiggles. One 5 was, “a regular nice creature. Strong, optimistic and good,” while another seemed “ready to rush forward and attack somebody.” Pavlovich then graded each of the 5s and 8s on 21 different attributes derived from her poems. For example, one handwritten example could have an ‘‘aggressiveness” rating of 2 out of 2, while another could show “stability” to a strength of 2 out of 3.
> So instructed, Vapnik’s computer was able to recognize handwritten numbers with far less training than is conventionally required. A learning process that might have required 100,000 samples might now require only 300. The speedup was also independent of the style of the poetry used. When Pavlovich wrote a second set of poems based on Ying-Yang opposites, it worked about equally well. Vapnik is not even certain the teacher has to be right—though consistency seems to count.
That article in turn reminded me strongly of "Metaphors We Live By" by Lakoff & Johnson, and the works they have written since, where they claim that humans make sense of the world using systems of rich, conceptual metaphors. As I understand, the work is well-known to machine learning researchers.
We're doing humans wrong. Maybe not all wrong, and of course, humans are extremely useful things, but think about it: sometimes it almost looks like we're already there. There always going to be an anomaly; lots of them, actually, considering all the things shaded in different patterns. Something have to change.
I agree that we aren't there, but we'll never be there, every system can be fooled, its just a question of 95%, 99% or 99.99%
This analogy doesn't hold, because the whole point of these classifiers is to classify things the same way humans would.
The fact that all classifiers -- including human beings -- fail in some cases is a separate issue. The goal is to create a computer classifier that succeeds and fails in the same cases humans do.
I don't follow. If you asked a human what the linked image looked like, they'd likely say a face, but if you then asked them what it actually was, they're all going to change their answer to a rock, even specifically a rock on Mars (if given a colour version of this image).
It's true that humans see patterns that aren't there, but does that detract from our ability to recognise objects?
Leopards (or jaguars) are complex 3-dimensional shapes with quite a lot of degrees of freedom (considering all the body parts that can move independently). These shapes can produce a lot of different 2d contours
My son keeps telling me that infants are fine with, say, a truck transforming into a clown (when it emerges from the other side of a visual barrier) but not with it transforming into TWO of something. Apparently, babies subjectively experience this (visual transformation) all the time -- mom moves a plate and what seemed like a big circle is now a flat line or whatever.
So humans apparently get tons and tons of experience with visually mapping 3d reality to mere 2d imagery. I have been thinking somewhat about this of late, in terms of physical attractiveness or "image" -- that pictures of a woman posted on a blog capture a 2d version of her but people interacting with her are interacting with a 3d living, moving creature who also has smell and a voice and her movements may be elegant or may be not elegant. Which is a thought process relevant to a project of mine, something people here surely will have no interest in. But where it is relevant to this article is that we are doing this wrong: Humans have thousands of hours of practice of looking at 3d reality and figuring out how it to interpret 2d images as representative of that 3d reality. Image recognition software is just dealing with 2d images. I don't see how it can hope to compete. Humans don't come preinstalled with the software to make that distinction. We acquire it with enormous repetition.
When do we make a robot and give it some baseline parameters and a learning algorithm (and set it loose in 3d reality and have to learn)? That is when we can get scared about human like AI that can compete on image recognition.
Indeed, arguments like "My mother didn't have to purchase 30,000 mugs to teach me what one looks like" miss the fact that we humans spent so much time (almost all of our waking time in fact) interpreting 3D reality, an endless stream of repetitive tasks.
112 comments
[ 2.5 ms ] story [ 164 ms ] threadhttp://image-net.org/search?q=sofa
There are no sofa's in this list, the closest thing I can find is a "studio couch, day bed": http://imagenet.stanford.edu/synset?wnid=n04344873
Context: Evolutionary algorithms and analog electronic circuits
> One thing stands out when you try playing with evolutionary systems. Evolution is _really_ good at gaming the system. Unless you are very careful at specifying all of the constraints that you care about you can end up with a solution that is very clever but not quite what you had in mind. Here power consumption is the issue. If you tried to evolve a sturdy chair you might end up with something that is 1mm tall. or maybe a fuel efficient car that exploits continental drift.
I think it's the same here: The net is never gonna better than what it needs to be, and it is probably always gonna take the easy route.
Given enough time, you'd expect the to develop clever behaviors, but instead they just fuzz-tested the sim and locked in on exploits of bugs or environment settings. They only got a bit more clever when connecting different sims running on different conditions.
Eyes already use different kinds and densities of sensors optimized for either detail and color or movement/edges. I wouldn't expect a single learning method, even after optimizing it to its limits, to be above what two or more layers of different methods could do, especially when trying to avoid exploits like the tank story.
Classic A-life! Also, not so different from the spirit of actual biology.
They only got a bit more clever when connecting different sims running on different conditions.
Diversity is very important for evolution on many levels. What many don't realize (especially, I note, evolution deniers) is that the ecosystem as a whole provides a very complex and continually varying epiphenomenal fitness function to any given organism.
However I think that's ok. Most of the fun with darwinbots is programming your own bots. They used to be (still are?) competitions where people wrote their own bots and had them compete under different conditions.
Part of the reason why a lot of these nets are trained with added noise, as well as drop-out (randomly disabling 50% of the hidden neurons, every training step).
Especially the drop-out tactic is particularly effective at preventing "exploits" of the neural net type, which otherwise appear in the form of large correlated weights (really big weights depending on other really big opposite weights to cancel out--it works, but it doesn't help learning).
Either way, adding noisy hurdles helps because exploits are usually edge cases, and noise makes them less dependable, as the region of fitness space very close to an exploitable spot, is usually not very high-ranking at all (which is why you don't want your classifiers ending up there).
It would seem CNNs were a significant step up, but the author hints at inferring structure as the next tack to take.
It would seem CNNs were a significant step up, but the author hints at inferring structure as the next tack to take.
We certainly see the same thing with autonomous vehicles. Given very accurate mapping and a particular set of environmental and type-of-road conditions, cars can do so well that it's tempting to say they're 95% of the way to fully-autonomous. But dump them in a Boston snowstorm and you see they're really not even close. (Which isn't to say that bounded use cases can't be very useful.)
But it might be different this time...
Right now we're at the stage of the "seven blind men and the elephant". Over time, our eyes will open and things will start making sense.
(A lot probably has to do with switching to more data-based approaches.)
If not, why else would this exist http://www.nvidia.com/object/tesla-supercomputing-solutions....?
I tried the unrotated sofa image on Wolfram's ImageIdentify and it correctly identified a settee [1]. So it presumably gathered that from the shape of the image rather than the pattern. It is peculiar though that it can't see the shape under a simple rotation. Or perhaps the margin of confidence levels between sofa and leopard were so narrow that a rotation was enough to tip it in favour of the leopard? I'd be interested to see the inner workings of this.
[1] http://i.imgur.com/6f6Co5O.png
I kept trying different ones and it kept identifying as "Bicycle Saddle"...
And I think this is the most interesting part.
One of the most depressing things about all of the "this image recognition algorithm performs better than humans on this task" is the idea that we've pretty much solved the problem, and it's just a matter of some more optimization and tweaking to handle a few edge cases.
This kind of problem, where the dominant solution simply gets it so wrong, and the problem cases are uncommon enough that any statistical solution is generally going to treat them as noise, reveals that in fact that there is likely plenty of room for entirely new, novel ways of approaching the problem to handle these kinds of cases better.
It's actually more exciting that there's so much more to be done, than to say "well, it's basically a solved problem, we just need to do some tweaking and optimization."
Indeed, discovering these "broken" edge cases is exactly what we need to converge upon a more correct solution.
Also, we carried out an experiment on ImageNet and the outcome was that "One human labeler (me, incidentally) with a fixed amount of training and a slightly-above average determination reached ~5% top-5 error on a subset of ImageNet test set". The media sees this and it immediately gets spun to "AI now Super-Human. And we're all going to die." It makes a lot of us cringe every time.
Many people in Computer Vision now consider ImageNet "squeezed" out of juice - we're good at texture recognition and when an object is in plain view, and we're now searching for harder tasks and more dynamic range with respect to human performance, in areas such as harder 3D/Spatial tasks, Image Captioning, Visual Q&A, etc. The hope is that these harder datasets might in turn guide us in developing models with more nuanced understanding.
It seems that this is more like the way that we learn to identify things. Then once we establish an understanding of a base class (big cat) we can apply that same model to new cats that we have never seen before with just a picture.
So I wonder to what extent you would consider this a predictable outcome from the classifier in question not being part of a subsumptive architecture --- which at a guess would look like
It seems to me, as a non-vision researcher (I mainly worked in planning and control), that the most obvious counterargument to the image being a spotted cat is based on boundary/object/scene reasoning, and that it's "reasonable" for the texture/glance layer to say "looks a lot like a cat texture".[1] https://en.wikipedia.org/?title=Subsumption_architecture
[2] I realize this may seem, superficially, anathema to deep network research, which advocates letting the network find its own intermediate levels of abstraction. But it's actually compatible in my view because Brooks advocates (again, paraphrasing quite a bit) that the separate layers should have different objective functions, and that in fact the need for different objective functions (in a prioritized order) is the cause of emergent layering in nature. "First, don't die. Second, find shelter. Third, find food etc." So one can imagine deep networks each finding their own locally useful abstractions for each objective function in the "Maslow" chain, while still having some macro architecture that tracks human-imposed design principles.
the problem is.. human vision doesn't work just by feeding a bitmap. we have structure to decode space relationships, shapes and maybe even shadow/light relations. no way we gonna see classificator working on color arrays matching our vision capabilities
However, the advantage to the texture approach is it's abstracted from a lot of other information. You don't want a classifier to say sofa, when it's a picture of a person on a sofa.
http://www.bespokesofalondon.co.uk/assets/Uploads/bespoke-so...
anyway it does work perfectly if that's what you need, but most proponent are trying to use deep nn to classify 'as good as humans do'
https://www.imageidentify.com/result/1ixb9603m9ix1
(I've seen the comments like https://news.ycombinator.com/item?id=9584325 and watched the lectures and youtube walkthroughs, but they're all theoretical and I'm looking for documented code to go along with that theory)
Also, the author spends the first section of the article determining that it is in fact a jaguar-print sofa (which the model also confirms) but continues to throw around the word "leopard". They're not making it any easier for the future machine learning algorithms that try to identify an image by the text surrounding it. ;)
https://www.imageidentify.com/result/1ixb9603m9ix1
Edit: There's a comment about invariance in this thread [1] and apparently CNNs are not invariant under rotation.
[1] https://news.ycombinator.com/item?id=9750133
Giving it an image that we know has all the relevant details of a sofa, but it likely won't have close matches in the dataset, can give us an idea of how clever it is.
Scary and fascinating.
I would propose that for this leopard problem, instead of just skewing the images, you also performed transformations on the COLOR and put the images back into the training set.
Maybe applying certain filters, such asdimming the saturation or contrast of images, so that the contrast between the leopoard spots were less visible (i.e. "A Leopoard in low lighting") - maybe this would force the neural net to learn more than just its print.
Knowing the right set of color filters to apply to all images could be tricky though.
I would propose that for this leopard problem, instead of just skewing the images, you also performed transformations on the COLOR and put the images back into the training set.
Maybe applying certain filters, such asdimming the saturation or contrast of images, so that the contrast between the leopoard spots were less visible (i.e. "A Leopoard in low lighting") - maybe this would force the neural net to learn more than just its print.
Knowing the right set of color filters to apply to all images could be tricky though.
This is a classic story of a neural net failure.
The net was able to find tanks hiding in the trees with amazing accuracy. Too amazing. It turned out the photos of the hidden tanks were all photographed on a cloudy day. The images without tanks in a clear day.
I myself have researched leopard spots since I painted our toilet floor in them. It's a lead sheet, and the paint had worn off, which probably wasn't the healthiest thing. My housemates had filled the toilet with memorabilia from an African trip, so leopard-print paintjob it was.
Which entailed looking up leopardprint online. Very little of which actually looks like leopard rosettes, and now I have a problem with almost anything trying to pass itself off as leopardprint. Anyway, I can't say that my paintjob is a particularly good reproduction, but at least it's 'spiritually correct'... :)
> When Vladimir Vapnik teaches his computers to recognize handwriting, he does something similar. While there’s no whispering involved, Vapnik does harness the power of “privileged information.” Passed from student to teacher, parent to child, or colleague to colleague, privileged information encodes knowledge derived from experience. That is what Vapnik was after when he asked Natalia Pavlovich, a professor of Russian poetry, to write poems describing the numbers 5 and 8, for consumption by his learning algorithms. The result sounded like nothing any programmer would write. One of her poems on the number 5 read,
> He is running. He is flying. He is looking ahead. He is swift. He is throwing a spear ahead. He is dangerous. It is slanted to the right. Good snaked-ness. The snake is attacking. It is going to jump and bite. It is free and absolutely open to anything. It shows itself, no kidding. Brown_Cornerart
> All told, Pavlovich wrote 100 such poems, each on a different example of a handwritten 5 or 8, as shown in the figure to the right. Some had excellent penmanship, others were squiggles. One 5 was, “a regular nice creature. Strong, optimistic and good,” while another seemed “ready to rush forward and attack somebody.” Pavlovich then graded each of the 5s and 8s on 21 different attributes derived from her poems. For example, one handwritten example could have an ‘‘aggressiveness” rating of 2 out of 2, while another could show “stability” to a strength of 2 out of 3.
> So instructed, Vapnik’s computer was able to recognize handwritten numbers with far less training than is conventionally required. A learning process that might have required 100,000 samples might now require only 300. The speedup was also independent of the style of the poetry used. When Pavlovich wrote a second set of poems based on Ying-Yang opposites, it worked about equally well. Vapnik is not even certain the teacher has to be right—though consistency seems to count.
http://nautil.us/issue/6/secret-codes/teaching-me-softly
That article in turn reminded me strongly of "Metaphors We Live By" by Lakoff & Johnson, and the works they have written since, where they claim that humans make sense of the world using systems of rich, conceptual metaphors. As I understand, the work is well-known to machine learning researchers.
https://upload.wikimedia.org/wikipedia/commons/7/77/Martian_...
We're doing humans wrong. Maybe not all wrong, and of course, humans are extremely useful things, but think about it: sometimes it almost looks like we're already there. There always going to be an anomaly; lots of them, actually, considering all the things shaded in different patterns. Something have to change.
I agree that we aren't there, but we'll never be there, every system can be fooled, its just a question of 95%, 99% or 99.99%
The fact that all classifiers -- including human beings -- fail in some cases is a separate issue. The goal is to create a computer classifier that succeeds and fails in the same cases humans do.
I don't follow. If you asked a human what the linked image looked like, they'd likely say a face, but if you then asked them what it actually was, they're all going to change their answer to a rock, even specifically a rock on Mars (if given a colour version of this image).
It's true that humans see patterns that aren't there, but does that detract from our ability to recognise objects?
My son keeps telling me that infants are fine with, say, a truck transforming into a clown (when it emerges from the other side of a visual barrier) but not with it transforming into TWO of something. Apparently, babies subjectively experience this (visual transformation) all the time -- mom moves a plate and what seemed like a big circle is now a flat line or whatever.
So humans apparently get tons and tons of experience with visually mapping 3d reality to mere 2d imagery. I have been thinking somewhat about this of late, in terms of physical attractiveness or "image" -- that pictures of a woman posted on a blog capture a 2d version of her but people interacting with her are interacting with a 3d living, moving creature who also has smell and a voice and her movements may be elegant or may be not elegant. Which is a thought process relevant to a project of mine, something people here surely will have no interest in. But where it is relevant to this article is that we are doing this wrong: Humans have thousands of hours of practice of looking at 3d reality and figuring out how it to interpret 2d images as representative of that 3d reality. Image recognition software is just dealing with 2d images. I don't see how it can hope to compete. Humans don't come preinstalled with the software to make that distinction. We acquire it with enormous repetition.
When do we make a robot and give it some baseline parameters and a learning algorithm (and set it loose in 3d reality and have to learn)? That is when we can get scared about human like AI that can compete on image recognition.
I suppose that's why Banach–Tarski is considered a paradox.