If you are looking to have a mathematical understanding of the field then I really enjoyed "Understanding Machine Learning: From Theory to Algorithms" by Shalev-Shwartz and Ben-David.
After a cursory glance, my feeling is that "The Mathematics of Neuronal Networks" would be a better title?
In recent years, the term "Artificial Intelligence" is often used instead of "Neuronal Networks".
I wouldn't be surprised if this will change again. If there is evidence that it will not and Neuronal Networks are for some reason the optimal medium for intelligence, I would love to read about it.
Came here to say the same thing. This is not a paper on mathematical artificial intelligence, at last as I understand it, not in the way Goertzel or Hutter write about it. I was actually thinking this would be something like Hutter's universal artificial intelligence.
This is a very good point. To anyone interested in what Artificial Intelligence looked like before neural networks, the "Artificial Intelligence - A Modern Approach" by Stuart Russell and Peter Norvig is a good book on the subject.
Now, NNs are the ones getting results at computer vision and natural language, and more. I think most people would say that other ML approaches are computational statistics. The goalpost for AI keeps moving.
If you are truly interested in the math of AI I think PAC Bayes learning is more appropriate and your book is Understanding Machine Learning [1] (not an easy read). A more gentle intro would be Learning From Data [2]. If someone recommends a book/paper it would be awesome, I'm always on the look.
NN > ML. The proof is that nematode or fly, I dont remember, with its simple NN fully mapped, and still remaining a mystery how it works. ML, which is just a matrix multiplication at its core, is a laughably simplistic model of NN.
I think one can't compare deep learning to machine learning since they have different purposes. While there have been great strides in the interpretability of NNs, the analytic models of classical machine learning are favored from a computational and interpretability perspective. Conversely, if the relationships in our data are constantly evolving (e.g. the change in the stochastic process followed by a time-series from one interval to another), then NNs are more appropriate.
Although it tends towards Deep Learning as opposed to AI, I highly recommend Bishop's Pattern Recognition and Machine Learning. It not only provides a solid Bayesian perspective, but also comments on the subtleties of applying theory. For the latter, its discussion of overparameterization in (I think?) the first chapter comes to mind.
> Neuronal Networks are for some reason the optimal medium for intelligence
No, that's not the same 'intelligence' ("General Intelligence") as the "I" side in Artificial Intelligence.
The term 'intelligence' applied to Artificial Neural Networks makes sense, as such: to reach a procedural solution it takes an engineer; the engineer is said to have reached the solution because "intelligent"; ANNs are (semi-)automated builders of function approximators; ANNs are said "intelligent" because they "reach solutions" (like the engineer would have done - "approximator":"engineer"="Artificial":"Natural" Intelligence). And of course they are not "intelligent", while they are in some sense. It's just an expression, it's rhetoric (it requires considerate interpretation).
That some sort of "intelligence" is achievable in other ways, or that ANNs spawn as an idea from anatomical considerations of naturally intelligent entities, or that ANNs could help in modelling general intelligence (etc.), is tangential.
ANNs dont have the key component of intelligence - ability to learn. If our brains worked like ANNs, they'd be wired once at the brains factory, they'd produce the same output given the same input, and learning anything new would need a complete rewiring at the factory, possibly erasing other learnings (often unpredictably).
Even if ANNs could update themselves on the fly, in a reasonably incremental manner, they'd just imitate insect-level reactions. Intelligence would need those ANNs to have a virtual reality, run thought experiments there and learn from that.
Well this does not mean that some mechanisms similar to those we are studying cannot be part of the original or that they cannot be part of a General Intelligence system. Even if that were simply "adjusting a structure to function and information retention". The very "compositional" nature of the perceived that makes Convolutional ANNs work was noted after biomimicry.
You're asking for online learning. This is certainly possible with modern techniques - define an unsupervised loss and update the weights with gradients as new data is encountered. Federated learning also works along these lines.
'Raw' online learning is unpopular because you have no garauntee that the network won't do something funky in the field.
That said, I think there are production systems in the world which learn on a day by day basis. Eg, take all of the logs from the last day and use them to update the production model for tomorrow. Then there's enough data that you don't risk a bad step. Think of it as learning while dreaming...
I think in Greek 'neura' and 'neuron' are equivalent. The perception that '-on' indicate the single or ideal entity ("ion, electron, muon...") comes from Whewell and Faraday ('-on' ex "ienai": the "goer" particles) and fully postdates the use of 'neuron'.
Because AI basically "stole" the term neural networks from neuroscience, neuroscientists now often speak of "neuronal networks" while referring to biological ones, to avoid confusion. So I would recommend to not use "neuronal networks" in this context.
Is "Nothing about this paper is new" an appropriate response when the abstract explicitly mentions that it's a survey paper? Maybe I'm just sensitive, but to me that sounds dismissive rather than purely informative.
Well, the poster may have just rephrased the formula («survey article»), or may have noted it for the benefit of the skimmer, or just used it for the conclusion out of their judgement "for an overview it does its job". «sensitive ... sounds»: the author is apparently a native German speaker and not only different minds and cultures weigh expressions differently: there could even be a linguistic bias. We are mandated and obliged to «Assume good faith».
Let L(uy,y) = fy denote a parametric partial differential equation with y being a parameter from a high-dimensional parameter space Y ⊆ Rp and uy the associated solution in a Hilbert space H. After a high-fidelity discretization...
These AI folks are clearly very clever, but they don't actually believe that's got anything to do with how human thinking works, right?
AI leaders occasionally seem to pontificate on AGI, yet it's difficult for a layperson like me to see how their expertise (as seen in publications like this) translates.
Whenever I see "Hilbert space" remarks in a paper that doesn't concern abstract math, I roll my eyes and imagine a chef who starts his youtube video with "Today we are going to cook a salad. For better precision, we are going to use knifes of triangular shape, three dimensional plates of circular shape, and the 100% dihydrogen of oxyde solution." All this only to obfuscate the fact that the entire recipe is just chopping a few carrots.
A form of "plural majestatis" which paradoxically implies that the individual author has only relative importance
(«paradoxically» as the majestas itself is founded on representing something higher, not of some high status of the individual which happens to be endowed - it may sound like boasting but on the contrary it would be a downplay of the individual).
In other contexts, that "we" can have even more foundations: "I could never have done this without the work of others", "Not just me but all those who think alike" etc.
Using the "royal we" is standard in mathematics. If you ever see someone do this in ordinary writing you can bump up your odds that they're a mathematician. Another giveaway is using "modulo" to mean "except for".
Does anybody know of textbooks, or articles, material etc. that focus on the /consequences/ of the mathematics of ANNs? For example: why in this problem using more layers is more or less efficient than using bigger layers, or branching evolutions of the outcomes as opposed to intensive computation of a single flow etc.
A reasoned summary of the tricks, in a way ("this works because of that").
Along the same lines, here is a course on the "Mathematical Foundations of Machine Learning" [1] (from GT, but other institutions also having offerings) and is a section in the paper in this post (so deeper dive possible):
Pity that it uses Artificial Intelligence instead of the more correct and precise term Machine Learning, especially since Mathemticians are supposed to be more precise.
It is mathematically correct: what concerns ML concerns AI.
The interest of the author is explicitly ("precisely") on AI, where, statedly, «the current “workhorse” of artificial intelligence [are] namely deep neural networks».
Poorly thought out morning spitball coming. One of the reasons approximation theorems are so unsatisfying is that they are always of the form "for function class X there exists an architecture A of complexity O(N) such that blah". And then this is compared favorably with some other function class whose dimension is O(N). But there's something tricky about this: you leave the architecture unspecified. You are comparing a single space of functions with an enormous number of spaces, one for each architecture with the specified complexity, and then saying "well if I pick the right architecture I win". Doesn't seem like a fair comparison.
more spitball -- the term "AI" changes the context from machine-side to application-side. "AI" implies to the listener that the machine is doing something that only thinking humans could do.. emphasis on the humans, while sets of math that seek minimized error, emphasize the behavior of the functions.
Shouldn't "the mathematics of artificial intelligence" be subsumed by computational statistics? It may be that only very general and weak statements are possible about "deep" NN's, but then that's merely a fact about the territory we're dealing with. Why should this be regarded as a "new" field?
Oh yeah well aware that it was the first restricted Boltzmann Machine. It minimizes energy, but that’s just harmony with a sign change. When Smolensky Hinton and Rummelhart collaborated, they decided to call it “goodness of fit.” So neural networks still optimize for harmony by another name. I think that’s so cool. The original Smolensky paper is worth reading. Good ideas.
60 comments
[ 3.4 ms ] story [ 125 ms ] threadhttps://mml-book.github.io/
After learning the math, which machine learning books come next?
In recent years, the term "Artificial Intelligence" is often used instead of "Neuronal Networks".
I wouldn't be surprised if this will change again. If there is evidence that it will not and Neuronal Networks are for some reason the optimal medium for intelligence, I would love to read about it.
As an aside, I will forever remember the rough geography of Romania. I still plan to do an A* route visit of the country some day :D
Now, NNs are the ones getting results at computer vision and natural language, and more. I think most people would say that other ML approaches are computational statistics. The goalpost for AI keeps moving.
If you are truly interested in the math of AI I think PAC Bayes learning is more appropriate and your book is Understanding Machine Learning [1] (not an easy read). A more gentle intro would be Learning From Data [2]. If someone recommends a book/paper it would be awesome, I'm always on the look.
[1] https://www.cs.huji.ac.il/w~shais/UnderstandingMachineLearni... [2] https://work.caltech.edu/telecourse
Machine learning contains ANNs as a sub-discipline. Other non-ANN topics in ML include ensembled trees, Gaussian processes, and sampling theory.
[1] - https://www.microsoft.com/en-us/research/people/cmbishop/prm...
No, that's not the same 'intelligence' ("General Intelligence") as the "I" side in Artificial Intelligence.
The term 'intelligence' applied to Artificial Neural Networks makes sense, as such: to reach a procedural solution it takes an engineer; the engineer is said to have reached the solution because "intelligent"; ANNs are (semi-)automated builders of function approximators; ANNs are said "intelligent" because they "reach solutions" (like the engineer would have done - "approximator":"engineer"="Artificial":"Natural" Intelligence). And of course they are not "intelligent", while they are in some sense. It's just an expression, it's rhetoric (it requires considerate interpretation).
That some sort of "intelligence" is achievable in other ways, or that ANNs spawn as an idea from anatomical considerations of naturally intelligent entities, or that ANNs could help in modelling general intelligence (etc.), is tangential.
Even if ANNs could update themselves on the fly, in a reasonably incremental manner, they'd just imitate insect-level reactions. Intelligence would need those ANNs to have a virtual reality, run thought experiments there and learn from that.
This, of course, as an aside.
'Raw' online learning is unpopular because you have no garauntee that the network won't do something funky in the field.
That said, I think there are production systems in the world which learn on a day by day basis. Eg, take all of the logs from the last day and use them to update the production model for tomorrow. Then there's enough data that you don't risk a bad step. Think of it as learning while dreaming...
Edit: also see the comment from member martopix, nearby ( https://news.ycombinator.com/item?id=30988150 )
Just as the paper itself clearly states. So your comment comes across as rather rude.
[1] https://www.amazon.com/Mathematical-Methods-Artificial-Intel...
These AI folks are clearly very clever, but they don't actually believe that's got anything to do with how human thinking works, right?
(«paradoxically» as the majestas itself is founded on representing something higher, not of some high status of the individual which happens to be endowed - it may sound like boasting but on the contrary it would be a downplay of the individual).
In other contexts, that "we" can have even more foundations: "I could never have done this without the work of others", "Not just me but all those who think alike" etc.
A reasoned summary of the tricks, in a way ("this works because of that").
http://yann.lecun.com/exdb/publis/pdf/bengio-lecun-07.pdf
https://ai.facebook.com/blog/advancing-ai-theory-with-a-firs...
HN discussion:
https://news.ycombinator.com/item?id=27559017
My own review comment:
https://news.ycombinator.com/item?id=27564506
Edit: Corrected the third link. Thanks mdp2021 for notifying me of the copy-paste error.
I would just correct the links:
-- article: https://arxiv.org/abs/2106.10165
-- book: https://arxiv.org/pdf/2106.10165
-- your summary: https://news.ycombinator.com/item?id=27564506 (the link to the post is that "under" the date. You posted two identical links here)
[1.] https://jrom.ece.gatech.edu/mfml-f20-notes/
The interest of the author is explicitly ("precisely") on AI, where, statedly, «the current “workhorse” of artificial intelligence [are] namely deep neural networks».
Poorly thought out morning spitball coming. One of the reasons approximation theorems are so unsatisfying is that they are always of the form "for function class X there exists an architecture A of complexity O(N) such that blah". And then this is compared favorably with some other function class whose dimension is O(N). But there's something tricky about this: you leave the architecture unspecified. You are comparing a single space of functions with an enormous number of spaces, one for each architecture with the specified complexity, and then saying "well if I pick the right architecture I win". Doesn't seem like a fair comparison.