That seems to be concise summary of the articles point, but my first reaction is that it's a superficial distinction.
One of the researches is Max Tegmark, author of Our Mathematical Universe. In short, he suggests every theory becomes it's own universe; physical laws are representations of patterns in some kind of calculation or proof.
As he explains, although every possible theory is a universe, "most" complex theories can be reduced to simple ones. So it seems plausible that those simple patterns are more common and more likely to be found in a random (mathematical or physical) universe.
This means that neural networks are likely to work with those common patterns. One might say it because that's closer to the nature of physical laws. But more directly, that's because of the nature of mathematical laws.
Neural networks are trained with information from the Universe. So, in my opinion, it should not be shocking at all that neural networks store some projection/model/echo of the universe (e.g. physical laws, maths, etc.).
You should watch Max Tegmark's talk "Connections between Physics and Deep Learning" if this interests you [1].
Additionally, a paper that has everyone excited about deep connections between the mathematical analysis of physical systems and the hierarchical feature learning paradigm speaks of the connection in terms of the Renormalization Group [2].
Regardless of it's practical utility, the philosophical implications do tickle the intellect. On a dreamy note, I wonder if it would be possible to draw Category-theoretic parallels between some physical theories and statistical learning theory. There is so much to learn, and I am trying my best to teach myself (on the side) the mathematics that they don't teach in my CS grad school. So much to learn, such little time. :)
I think that theoretical physics is an example of confusing map with the territory. Just because you can work out some mathematical mumbo-jumbo does not necessarily mean that it has any connection to reality.
This applies especially well to string theory which is just layers of pointless mathematics, each layer trying to make up for obvious deficiencies of lower layers.
The answer is that neural networks are a somewhat crude approximation of Solomonoff induction. Solomonoff induction is an ideal, perfect machine learning method. It only assumes that the universe is a random computer program, and tries to infer which one.
It's not that no one understands, it's that journalists don't know that their's research that has addressed this question. The problem with intuitively understanding neural networks is that most of the weighted edges are spurious. Meaning you can remove most of them (WiJ -> 0) without negatively affecting the mapping function solution. Failing to do this we stare at a fully connected neural network which gives us no clues why it works. However , once you get rid of these spurious interactions (through perturbation analysis), then This will reveal the underlying circuit diagram for the network.
Often these threshold integration neurons are just approximating a simpler Boolean function so it really comes down to digital logic.
Techniques for pruning networks can be found here:
Pruning works for understanding sparse networks, but not for holographic networks, which can encode a much higher information density. Understanding a holographic network requires a large amount of effort, similar to determining the content of an optical holograph. You can try to develop a model, but if the information density is very high, it's no faster than observing the response to various inputs.
The problem becomes even more difficult if there is no clear distinctions between layers in the network, as in the brain.
A network should be no more complex than it needs to be. If you're not pruning spurious interactions then you're implicitly assuming that each interaction is information bearing. If you're not pruning spurious interactions from holographic networks then you're assuming that they're all functionally relevant. I promise you they're not.
You're also assuming that holographic networks are somehow necessary for most of the tasks we require. The human brain, while having billions of neurons is still quite sparsely connected.
Pruning the network leads to a smaller network. How does that help with understanding? Does compressing the network from 100M parameters to 10M parameters reveal "the underlying circuit diagram" that you can understand?
> Despite the huge success of deep neural networks, nobody is quite sure how they achieve their success.
Is that true?
After reading Jeff Hawkings - "On intelligence" book (2004), for me it's pretty much clear why they are so successful on that set of tasks. Despite of HTM model built by Jeff being kind of different from DNN, the idea and origin of brain and consciousness described pretty well.
All modern advances in neuroscience just prove how strong the theory in this book is.
Jeff Hawkins has some pretty heterodox theories of intelligence, many of which are not backed up by modern neuroscience. Furthermore, as neuroscience advances, the similarities between so-called artificial neural networks and actual neurons are rapidly receding. Much of the current success of convolutional neural networks is (loosely) inspired by a particular part of the anatomy of the eye, but non-visual processing and abstract reasoning seems to take place under much more complicated patterns that ANNs may not be able to replicate as easily.
for me it's pretty much clear why they are so successful on that set of tasks
What's less clear is why HTM has not been successful on any tasks since 2004. I remember a few years ago they entered Kaggle competition to detect anomalies in EEG. Their team finished in ~150th place (out of 500). Even though anomaly detection in time series data was supposed to be their strongest feature.
19 comments
[ 2.7 ms ] story [ 36.5 ms ] threadOne of the researches is Max Tegmark, author of Our Mathematical Universe. In short, he suggests every theory becomes it's own universe; physical laws are representations of patterns in some kind of calculation or proof.
As he explains, although every possible theory is a universe, "most" complex theories can be reduced to simple ones. So it seems plausible that those simple patterns are more common and more likely to be found in a random (mathematical or physical) universe.
This means that neural networks are likely to work with those common patterns. One might say it because that's closer to the nature of physical laws. But more directly, that's because of the nature of mathematical laws.
Additionally, a paper that has everyone excited about deep connections between the mathematical analysis of physical systems and the hierarchical feature learning paradigm speaks of the connection in terms of the Renormalization Group [2].
Regardless of it's practical utility, the philosophical implications do tickle the intellect. On a dreamy note, I wonder if it would be possible to draw Category-theoretic parallels between some physical theories and statistical learning theory. There is so much to learn, and I am trying my best to teach myself (on the side) the mathematics that they don't teach in my CS grad school. So much to learn, such little time. :)
[1] https://youtu.be/5MdSE-N0bxs
[2] http://arxiv.org/abs/1410.3831
This applies especially well to string theory which is just layers of pointless mathematics, each layer trying to make up for obvious deficiencies of lower layers.
The answer is that neural networks are a somewhat crude approximation of Solomonoff induction. Solomonoff induction is an ideal, perfect machine learning method. It only assumes that the universe is a random computer program, and tries to infer which one.
Often these threshold integration neurons are just approximating a simpler Boolean function so it really comes down to digital logic.
Techniques for pruning networks can be found here:
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/
The problem becomes even more difficult if there is no clear distinctions between layers in the network, as in the brain.
You're also assuming that holographic networks are somehow necessary for most of the tasks we require. The human brain, while having billions of neurons is still quite sparsely connected.
Is that true? After reading Jeff Hawkings - "On intelligence" book (2004), for me it's pretty much clear why they are so successful on that set of tasks. Despite of HTM model built by Jeff being kind of different from DNN, the idea and origin of brain and consciousness described pretty well. All modern advances in neuroscience just prove how strong the theory in this book is.
What's less clear is why HTM has not been successful on any tasks since 2004. I remember a few years ago they entered Kaggle competition to detect anomalies in EEG. Their team finished in ~150th place (out of 500). Even though anomaly detection in time series data was supposed to be their strongest feature.