I love the fact that I'm seeing how stuff works. Before theory, people built huge buildings, cars, airplanes, magical electrical circuits etc.
Neural networks are the same thing. Bunch of heuristics that will later be proven in theory. Yet still work incredibly well. Like cathedrals did without any mathematics.
Says a lot about how tinkering with heuristics is powerful.
Well, I'm pretty sure there was not even Euclid's geometry thousand years after he invented it in any architecture recipe book, yet they built brutally large structures.
We have 13th century books from architects that built cathedrals filled with recipes with 0 mathematics, all heuristics, rules.
Are you sure? For example, the Florence Cathedral (Il Duomo) required nontrivial, novel mathematics. Brunelleschi didn't succeed in that effort simply through iterative engineering.
I think in general it would be more appropriate to say that fundamental theory tends to predate its applications, and then the theory and applications evolve as one informs the other in a positive feedback cycle. Likewise a lot of improvements to rocket science are due to iterative engineering, but we couldn't get off the ground (literally) without calculus.
Brunelleschi was working at the height of the Renaissance, in the early 1400s. They weren't short of book learning in renaissance Florence, but that's why it's called the renaissance. I imagine medieval cathedral building, e.g. Notre Dame starting more than 200 years previously, might have looked very different. They couldn't build a dome like Brunelleschi back then.
Similarly, there's a ton of math in the applied side of neural network research too. It's not math vs no math. It's a question of foundational "General Theory of X" math.
I’m reminded of this paper [0] on the relations between depth and width and expressivity and optimization. I think that ResNext’s concept of cardinality likely plays a meaningful role.
I think a important advance is being able to build neural networks in a modular fashion by constructing a toolkit of components and techniques one can reasonably expect, when combined, to produce certain result. As the field advances, this toolbox will become richer and better defined and delineated, forming basic building blocks. Just like functions, loops and virtual methods came out of the chaos of assembly.
Deep Learning is currently an empirical science guided by intuition of practitioners. A main principle in experimental sciences is that a theory without predictive power is not considered a full-fledged theory. As such, unless they are interesting predictions coming from their theory (rather than only barely justifying existing empirically observed phenomena), this is just speculative theory that I would not use the phrase "Foundations Built" for.
As an example of this general litmus test for a theory see e.g. Eddington's confirmation of GR: https://en.wikipedia.org/wiki/Tests_of_general_relativity#De... . If there are hitherto unknown phenomena in DL predicted by this theory then I'd stand corrected and concede that there may be something to these theories.
Yeah, it's not foundations like elementary or axiomatic.
An actual theory of ANN and then of NNN will be of more momentous import than of any previous phenomena, will usher in a fundamentally transformative age where we understand ourselves, and will surely require a fundamental breakthrough in pure mathematics... probably a great many.
The article seems pretty upfront about the fact that we are nowhere near a solid theory. One of many quotes to that effect: “ […] beginning to build the rudiments of a theory of neural networks.”
The main point seems to be why such a theory is highly desirable. To that end, it does a far better job than the somehat tired Feynman aphorism about good theories “giving us more than we give them”. Namely that the current state of the art of “intuition” and vast trial-and-error runs being somewhat embarrassing for a field so closely related to math and statistics.
The current state of neural networks is somewhat analogous to how cathedrals were built in the medieval era.
Master architects had a good understanding of then-ancient geometry and applied that knowledge plus knowledge of arches from Roman times to create the great cathedrals found throughout Europe.
The architects didn't have a deep understanding of material science (stone) and other forces that are taken into account in the modern era, but they had enough knowledge from experience and experimentation combined with relatively simple geometry to create strong structures that have stood the test of centuries.
The analogy between that and modern practical AI/neural network implementations is that experts in the field have good basic knowledge of how to construct neural networks but there isn't yet an advanced theoretical knowledge of the science that allows for precise predictions and modeling. i.e. Experts are still at the level of experimentation and noting the results and getting useful results without a super-deep understanding of how neural networks produce useful results.
Hypotheses have been/are being proposed and tested right now. Expect that as a deeper level of experimental results are achieved we'll soon have a sound theoretical model that will lead to profound advances in the field.
I've beaten this drum before but another area to look at is network connectivity. When visualizing a neural network we typically see a fully connected network from one layer to the next, but if you set the value of many weights to zero you'll find that they have absolutely no effect on the function of the network. So basically we're looking at a lot of spurious interactions and this just clouds our thinking. In fact the network topology is what's driving network function here but we're not doing a good job exposing that intuitively. Computational research on Gene Regulatory Networks--which are also modeled with the same NN math--have shown how network topology is really the key driver of function.
For anyone who is interested, I published a paper during my PhD that has about 195 citations. It shows that under an evolutionary process that Artificial Gene Regulatory Networks select out spurious network interactions, leaving you with what's functionally necessary (Also see the supplementary). I came from AI before starting my PhD and believe that this could be important for building a foundation for NN as well.
20 comments
[ 3.6 ms ] story [ 60.1 ms ] threadNeural networks are the same thing. Bunch of heuristics that will later be proven in theory. Yet still work incredibly well. Like cathedrals did without any mathematics.
Says a lot about how tinkering with heuristics is powerful.
We have 13th century books from architects that built cathedrals filled with recipes with 0 mathematics, all heuristics, rules.
I think in general it would be more appropriate to say that fundamental theory tends to predate its applications, and then the theory and applications evolve as one informs the other in a positive feedback cycle. Likewise a lot of improvements to rocket science are due to iterative engineering, but we couldn't get off the ground (literally) without calculus.
https://arxiv.org/abs/1705.05502 https://arxiv.org/abs/1810.00393
[0] https://openreview.net/forum?id=BJjquybCW
https://arxiv.org/abs/1606.05340
As an example of this general litmus test for a theory see e.g. Eddington's confirmation of GR: https://en.wikipedia.org/wiki/Tests_of_general_relativity#De... . If there are hitherto unknown phenomena in DL predicted by this theory then I'd stand corrected and concede that there may be something to these theories.
An actual theory of ANN and then of NNN will be of more momentous import than of any previous phenomena, will usher in a fundamentally transformative age where we understand ourselves, and will surely require a fundamental breakthrough in pure mathematics... probably a great many.
The main point seems to be why such a theory is highly desirable. To that end, it does a far better job than the somehat tired Feynman aphorism about good theories “giving us more than we give them”. Namely that the current state of the art of “intuition” and vast trial-and-error runs being somewhat embarrassing for a field so closely related to math and statistics.
Master architects had a good understanding of then-ancient geometry and applied that knowledge plus knowledge of arches from Roman times to create the great cathedrals found throughout Europe.
The architects didn't have a deep understanding of material science (stone) and other forces that are taken into account in the modern era, but they had enough knowledge from experience and experimentation combined with relatively simple geometry to create strong structures that have stood the test of centuries.
The analogy between that and modern practical AI/neural network implementations is that experts in the field have good basic knowledge of how to construct neural networks but there isn't yet an advanced theoretical knowledge of the science that allows for precise predictions and modeling. i.e. Experts are still at the level of experimentation and noting the results and getting useful results without a super-deep understanding of how neural networks produce useful results.
Hypotheses have been/are being proposed and tested right now. Expect that as a deeper level of experimental results are achieved we'll soon have a sound theoretical model that will lead to profound advances in the field.
[0] http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.441...
For anyone who is interested, I published a paper during my PhD that has about 195 citations. It shows that under an evolutionary process that Artificial Gene Regulatory Networks select out spurious network interactions, leaving you with what's functionally necessary (Also see the supplementary). I came from AI before starting my PhD and believe that this could be important for building a foundation for NN as well.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2538912/