The reasoning for author's premise that sigma summation and integrals are leaky abstractions is unclear.
The two are their own independent abstractions; while perhaps complicated to fully understand, they will work exactly as designed and, provably so, from a logical standpoint. If they were leaky, they would display some kind of shortcoming that didn't allow them to convey parts of the core idea they were created to represent.
On the other hand, an NN implementation of any sufficiently complex concept is almost guaranteed to be leaky as it is too complicated to be provably correct, and will have too great a test surface to verify exhaustively. There are likely to exist edge cases where the NN fails outright that will be never discovered until used in a specific scenario. That seems leaky.
Can someone set me straight on what the author was trying to convey?
The neural network counterpoint feels odd, but with respect to summations and integrals as leaky abstractions I think the author's point is that deploying them effectively as tools typically depends on a mountain of hierarchical knowledge on top of which elementary calculus is built. I don't know that this is necessarily a requirement -- one could conceive of a world where integrals are taught purely in terms of methods for moving in and out of that concept domain (analogous to the train/predict interface common in ML), and where people are simply taught algebraic rules for manipulating integrals entirely in the realm of integrals without relying on their pedagogical grounding in limits (sort of like how we have high-level descriptions of layers, drop-out, and other concepts that are at the same level of abstraction as neural networks themselves). I think the author claims that this latter approach is rare in mathematics, and it seems they make the stronger claim that mathematical concepts aren't generally amenable to that kind of strategy with integrals and summations as specific examples of where it's impossible. FWIW, I disagree with that assertion, but nevertheless it seems to be what the author is trying to say.
one could conceive of a world where integrals are taught purely in terms of ... algebraic rules for manipulating integrals entirely in the realm of integrals without relying on their pedagogical grounding in limits
This is absolutely how I was first taught integral calculus.
∫x^n = x^{n+1}/(n+1) + C
∫e^x = e^x + C
∫(f+g) = ∫f + ∫g
etc. Only later did I really learn the details of limits and the rigorous epsilon-delta underpinnings in real analysis.
I used to think this was a bad thing, but now I’m not so sure. The word “calculus” does mean a system for calculation.
Leakiness is not referring to "correctness" or any sort of practical shortcoming.
It actually means that the abstraction is not actually abstracting the concept it is representing, and thus requires you to understand lower level concepts in order to grasp it.
An example would be that the C programming language is a leaky abstraction because sometimes you need to understand and write Assembly code.
That doesn't make C bad or incorrect, it just means that at some point, you'll need to break through the abstraction and learn what it was trying to simplify.
How does this definition make the Neural Networks not leaky? They sometimes work, they sometimes don't work, if you tweak a few parameters (the number of layers, the number of neurons in each layer, ...) perhaps they work or perhaps no.
Also, how does this definition make the Fourier transform not leaky? From the article:
> A leaky abstraction is the rule rather than the norm in mathematics. The only popular counterexample that comes to mind are integral transform (think Laplace and Fourier Transforms).
I don't particularly agree with the author, as I think they are mixing up the mathematical theory and the computational part of it.
Yes you can pull Tensorflow and start creating layers without much knowledge in Algebra, just like you can use cryptography software without a PhD in Number theory, or apply Photoshop filters without understanding convolution.
But this is only possible because _other_ people do have that knowledge and distilled it down to a form you can use from the get go.
These mathematical "abstractions" are not the same as "abstractions" in natural sciences: they are only a succinct notation that provides leverage to perform more reasoning with less writing, not a simplified and separate model of a system.
For example, positional notation decreases the size of natural number representations exponentially compared to straightforward application of Peano's axioms or the like, but the numbers themselves remain exactly the same.
Arguably the neural net is the leakiest of all abstractions. Inputting enough data, preventing overfitting. Not having a human readable implementation because of the computer figured it out with brute force. Those are all complex details leaking out of the abstraction. Machine learning has its uses in processing large amounts of data, but it is the opposite of a designed algorithm with self-contained abstractions.
When I prove, say, Cauchy's theorem about groups, it doesn't matter which concrete group I'm operating on to prove it. If I write a sigma to sum some vectors, it doesn't matter if the vectors are coordinates in 3D space, functions defined on an interval, or just numbers. Mathematical abstractions are pretty much the only ones that don't leak.
What, that you have to understand summation to understand what the sigma refers to? Well, duh. That doesn't make the sigma a leaky abstraction, because the sigma is not an abstraction at all! Not even a trivial one. The sigma is notation. And it's hardly news that you have to understand the concepts behind a notation to understand notation itself.
The author seems to misunderstand what an abstraction actually is.
All signs refer directly to abstractions in the mind that can be compiled into a variety of useful forms. Yours does not take precedent in this case, as it is plain to see that sigma is under specified due to the huge range of concepts it’s practical usage covers. Your attempt to prescribe that which should be prescribed is nothing but authoritarianism.
Indeed. The author is looking for abstractions where they are not. Abstractions are in Algebra in general and in Category Theory in particular. The essence of Algebra is abstraction. All Algebras start from some intended model but they abstract it into axioms, i.e. "interface-level properties", which completely hide the "implementation".
>If I write a sigma to sum some vectors, it doesn't matter if the vectors are coordinates in 3D space, functions defined on an interval, or just numbers.
you're wrong. every "weird" technical hypothesis in a theorem is a leak. even in your super simple example there's a caveat: the sum has to be finite (that mean no you can't perform any sum of any set of vectors).
edit: i got downvoted so here's a better example that i just ran across:
manifolds with boundary can have empty boundaries. where does this come up? exact 1-forms on S^1 integrate to zero. why? because by Stokes. how? S^1 has a boundary but it's empty.
> I find it rather weird that mathematics is usually taught to people on a historical basis.
It very much is not. If you tried to learn calculus the way Newton learned it (with fluxions, or worse, awkwardly stated in the style of Euclid’s elements) or the way Leibniz learned it (ill-defined infinitesimals), you’re gonna be in for a bad time. We synthesise, reformulate, and reteach all the time. There’s a lot more that has changed since Euclid’s time than merely the notation.
I skimmed the rest of the article, but the philosophical mathematical preface is questionable.
I'd like you to elaborate a bit more on the ill-defined inifinitesimals, because that sounds a bit like the way I was taught calculus in 2006. But maybe it's just that I studied engineering and we got the "light" part and not too rigurous. Or maybe we were taught in a horrid way and didn't know better.
Infinitesimals have a long and complicated history (e.g. 1 = 0.999...) and their lack of a formal definition until recent years has lead to erroneous arguments such as thinking that (dx/dy) (dy/dz) (dz/dx) equals 1:
They are well-defined, but the usual framework that explains them ("non-standard analysis") is extremely abstract.
If you are dealing with polynomials/algebraic geometry only then there is a very simple, totally rigorous approach using nilpotents. And there is a newer approach called "alpha calculus" that I know nothing about but here is an example: https://arxiv.org/pdf/0807.3477.pdf
They are well-defined now, but when Leibniz and Cauchy were working with them, they were not, and used carelessly could lead to errors. Their "solution" was to just pretend those errors didn't happen ("don't do that, then") instead of trying to come up with a consistent system of rules for infinitesimals.
>Tell a well educated ancient Greek: “ 95233345745213 times 4353614555235239 is always equal to 414609280180109235394973160907, this must be fundamentally true within our system of mathematics”… and he will look at you like you’re a complete nut.
What is this supposed to mean? I honestly don't understand. They most certainly realized that although tedious to compute it is just as well-defined as 2*3=6. What is he implying here?
Archimedes dealt with much bigger numbers in The Sand Reckoner (he worked through his numbering system up to 10^(8*10^16)). He certainly wouldn't flinch at those.
Putting aside the weird preface about observation and mathematics, which misses the point that science builds a theory, which is a formal system that recapitulates the relations among observations, the neural network stuff shows very deep lack of understanding. For example:
> an autoencoder (AE) is basically one of the easiest ones me to explain, implement and understand
Then a list describing it. But that list does not constitute an explanation unless you have a pretty good intuitive understanding of linear algebra and function composition.
> So essentially, neural networks become a sort of mathematical abstraction that isn’t very leaky.
I think what the author really means by "leaky" is "not opaque." Mathematical structures that you can analyze in a well defined way (not opaque/"leaky") like integrals vs mathematical structures that the tools to dig into in a deep way don't exist yet (opaque/"not leaky").
Except that such understanding of neural nets is a major current research program.
So to all the youngsters who might be looking at this, this article is the ravings of a crank who doesn't know what he's talking about.
I took this more to mean the author liked NNs because they were not transparent. To model something with an NN you don't need to know all of the mathematical constructs that go into the NN.
A lot of fields in math expose a kind of API that abstracts away details quite well.
You can use SVMs without fully understanding the specific QP optimization algorithm underneath, you can understand the kernel trick without all RKHS math etc.
Or in classical vision, you can use keypoint extraction without caring about the details.
Or in linear algebra you can understand what SVD is and does without knowing the detailed steps of computing it.
34 comments
[ 1367 ms ] story [ 2233 ms ] threadThe reasoning for author's premise that sigma summation and integrals are leaky abstractions is unclear.
The two are their own independent abstractions; while perhaps complicated to fully understand, they will work exactly as designed and, provably so, from a logical standpoint. If they were leaky, they would display some kind of shortcoming that didn't allow them to convey parts of the core idea they were created to represent.
On the other hand, an NN implementation of any sufficiently complex concept is almost guaranteed to be leaky as it is too complicated to be provably correct, and will have too great a test surface to verify exhaustively. There are likely to exist edge cases where the NN fails outright that will be never discovered until used in a specific scenario. That seems leaky.
Can someone set me straight on what the author was trying to convey?
Yep, someone definitely needs to take a course in Abstract Algebra.
This is absolutely how I was first taught integral calculus.
etc. Only later did I really learn the details of limits and the rigorous epsilon-delta underpinnings in real analysis.I used to think this was a bad thing, but now I’m not so sure. The word “calculus” does mean a system for calculation.
It actually means that the abstraction is not actually abstracting the concept it is representing, and thus requires you to understand lower level concepts in order to grasp it.
An example would be that the C programming language is a leaky abstraction because sometimes you need to understand and write Assembly code.
That doesn't make C bad or incorrect, it just means that at some point, you'll need to break through the abstraction and learn what it was trying to simplify.
Also, how does this definition make the Fourier transform not leaky? From the article:
> A leaky abstraction is the rule rather than the norm in mathematics. The only popular counterexample that comes to mind are integral transform (think Laplace and Fourier Transforms).
Yes you can pull Tensorflow and start creating layers without much knowledge in Algebra, just like you can use cryptography software without a PhD in Number theory, or apply Photoshop filters without understanding convolution.
But this is only possible because _other_ people do have that knowledge and distilled it down to a form you can use from the get go.
For example, positional notation decreases the size of natural number representations exponentially compared to straightforward application of Peano's axioms or the like, but the numbers themselves remain exactly the same.
When I prove, say, Cauchy's theorem about groups, it doesn't matter which concrete group I'm operating on to prove it. If I write a sigma to sum some vectors, it doesn't matter if the vectors are coordinates in 3D space, functions defined on an interval, or just numbers. Mathematical abstractions are pretty much the only ones that don't leak.
What, that you have to understand summation to understand what the sigma refers to? Well, duh. That doesn't make the sigma a leaky abstraction, because the sigma is not an abstraction at all! Not even a trivial one. The sigma is notation. And it's hardly news that you have to understand the concepts behind a notation to understand notation itself.
The author seems to misunderstand what an abstraction actually is.
All signs refer directly to abstractions in the mind that can be compiled into a variety of useful forms. Yours does not take precedent in this case, as it is plain to see that sigma is under specified due to the huge range of concepts it’s practical usage covers. Your attempt to prescribe that which should be prescribed is nothing but authoritarianism.
This, I think, is the main problem with the article.
I also found the opening line curious:
> > I find it rather weird that mathematics is usually taught to people on a historical basis.
I think this is much more common in philosophy than in maths.
you're wrong. every "weird" technical hypothesis in a theorem is a leak. even in your super simple example there's a caveat: the sum has to be finite (that mean no you can't perform any sum of any set of vectors).
edit: i got downvoted so here's a better example that i just ran across: manifolds with boundary can have empty boundaries. where does this come up? exact 1-forms on S^1 integrate to zero. why? because by Stokes. how? S^1 has a boundary but it's empty.
It very much is not. If you tried to learn calculus the way Newton learned it (with fluxions, or worse, awkwardly stated in the style of Euclid’s elements) or the way Leibniz learned it (ill-defined infinitesimals), you’re gonna be in for a bad time. We synthesise, reformulate, and reteach all the time. There’s a lot more that has changed since Euclid’s time than merely the notation.
I skimmed the rest of the article, but the philosophical mathematical preface is questionable.
https://en.wikipedia.org/wiki/Triple_product_rule
If you are dealing with polynomials/algebraic geometry only then there is a very simple, totally rigorous approach using nilpotents. And there is a newer approach called "alpha calculus" that I know nothing about but here is an example: https://arxiv.org/pdf/0807.3477.pdf
What is this supposed to mean? I honestly don't understand. They most certainly realized that although tedious to compute it is just as well-defined as 2*3=6. What is he implying here?
> an autoencoder (AE) is basically one of the easiest ones me to explain, implement and understand
Then a list describing it. But that list does not constitute an explanation unless you have a pretty good intuitive understanding of linear algebra and function composition.
> So essentially, neural networks become a sort of mathematical abstraction that isn’t very leaky.
I think what the author really means by "leaky" is "not opaque." Mathematical structures that you can analyze in a well defined way (not opaque/"leaky") like integrals vs mathematical structures that the tools to dig into in a deep way don't exist yet (opaque/"not leaky").
Except that such understanding of neural nets is a major current research program.
So to all the youngsters who might be looking at this, this article is the ravings of a crank who doesn't know what he's talking about.
You can use SVMs without fully understanding the specific QP optimization algorithm underneath, you can understand the kernel trick without all RKHS math etc.
Or in classical vision, you can use keypoint extraction without caring about the details.
Or in linear algebra you can understand what SVD is and does without knowing the detailed steps of computing it.
I don't see neural nets as very special in this.
Try explaining the intrinsic meaning of written language in written language. You can't close the loop without making the result trivial.