Normally reverse mode automatic differentiation is used and is called backpropagation since everything old has to have a new and shiny name in machine learning a/k/a continuous optimization ;)
I concur. When I first started taking online coursework in machine learning, I was in almost every unit asking myself, didn't I cover this already in my non-linear optimization class at Uni.?
The main difference is that a lot of practitioners using ML have very little advanced mathematical background. Let's be honest, you can't understand much of optimization without at least multivariable calculus and linear algebra.
I think this is a very bad thing, but basically what happens when things become "commoditized".
I feel it's the same with web developers who have no formal background in Computer Science. All these frameworks reinventing "paradigms" that have been studied since the 60's.
I'm not sure what your point is. People use tools without understanding the underlying principles all the time, frequently in ways that are more (tangibly) practical and interesting than a person who does would use them.
I'd say that depends entirely on context. It also has diminishing value each "level" one goes. I mean, understanding quantum degeneracy pressure doesn't make physicists more effective at being solid.
I have had the same thoughts too. What also had me a bit puzzled in the ML community is the lack of discussion on local optimality, and constraint/Lagrangian loss functions. But there is probably a subtlety that I'm missing.
It's the first open source implementation compatible with the modern ecosystem. Software development isn't about novel math, it's about novel applications and deployments
I've always been curious why none of the silicon companies have attempted to provide hardware (or at least ISA extensions) supported AD, backprop, etc.
14 comments
[ 3.1 ms ] story [ 36.2 ms ] threadAs someone just starting to study machine learning it's inspiring and exciting seeing this kind of thing.
I think this is a very bad thing, but basically what happens when things become "commoditized".
I feel it's the same with web developers who have no formal background in Computer Science. All these frameworks reinventing "paradigms" that have been studied since the 60's.