I've believed since college that math is the worst taught of all academic subjects. I never had a math professor that engaged the class, and practical applications were never mentioned.
Even worse the language of math was never adequately explained. You were expected to learn the meaning of some new squiggle or greek letter by... its context? Symbols and terms you'd never seen before were just thrown at you without explanation. Definitions when given sounded circular or were given in terms of other things that had never been defined clearly.
I had one professor who just turned his back to the class and wrote things on the board. Students were meant to just magically understand.
I'll never forget in year one of college being absolutely stuck in calculus. I called up my dad and he tried to help me a bit, then he stopped and said: "you know what a derivative is, right?" I said no, not really. They tried to explain in class but it didn't click. He said "a derivative of a function is the rate of change of that function." I thought for a second and then said "thanks, now I understand calculus." I was un-stuck instantly. My professors never explained it that clearly.
Later on I took classes in things like population genetics and evolutionary dynamics. The professors of those classes explained the relevant math better than my math professors did to the point that if I'd taken those classes first I would have done better in the math classes that were their prerequisites.
Maybe you've just a bad math department? None of my math lecturers ever just "wrote things on the board".
> He said "a derivative of a function is the rate of change of that function."
I'm confused by this comment. That's the intuition that you see in just about every introductory calculus textbook: that the derivative is the instantaneous rate of change.
Maybe things were different back when OP took calc but I’d bet any standard calculus textbook used for instruction in the past 20 years mentions this many times.
Mine didn't lead with it. It may have been buried in there somewhere but it wasn't highlighted as critically important. Instead you just got the mechanics of differentiating and integrating functions thrown at you without explanation.
This was back in the late 1990s and early 2000s. I hope things have improved.
We had a very rigurous program in highschool back in Eastern Europe, it was so good that taking clac in college in the US was redundant. However, I remember having a similar click moment when I visually or intuitively understood derivatives and integrals. Before that it juggling math mechanics, which in itself is also not bad and helps build a certain muscle that can be used later on. But I am familiar with the fumbling a bit through some math classes.
My conclusion is different though. I think there are different types of thinkers and different teaching styles and when the subject becomes very complex the disparity between the two is exacerbated. And luck plays the role in matching up with the right teacher/professor for you.
It's unfortunate you were in a geographic location and a time that exposed you to poor math teachers.
I was taught calculus in high school, well before university, in Australia in the late 1970s.
Visual analogies such derivatives being tangential rates of change and 2nd derivatives being local curvatures were taught and drawn on the board in the first week of two years of Calc I and Calc II classes that came before leaving high school to attend university.
A good modern university program probably would have made you take "remedial" math courses first, where they don't assume that you already know general concepts like what a derivative is. Some classes (like proofs) introduce notation and get you comfortable with using definitions, but most people never take anything like that. It's normal not to really grasp a course until you take another one that relies on it.
> I've believed since college that math is the worst taught of all academic subjects. I never had a math professor that engaged the class, and practical applications were never mentioned.
Nearly all math textbooks and lectures mention practical applications. The problem rather seems to be that you have/had a different understanding of "practical" than your math professor.
I've tutored too many kids/young adults who had this exact story. Too often I got them after the midterm, when their whole 4 year plan was suddenly in jeopardy. I'm happy to say that there are a few more engineers out there that I helped get un-stuck.
I always had the benefit of being the tutor, though. I have to imagine that teaching a calculus one course is near impossible with 30 students on 30 different levels of experience and aptitude.
That said, I think that first year of calculus is often taught valuing mechanics over concepts.
I like to imagine what 12 years of math would look like if people weren't chained to a timeline. Here's khanacademy, go!
That doesn’t imply the teachers were bad and your father a brilliant teacher.
Chances are your teachers said almost, if not exactly, the same, but it didn’t stick yet.
In my first analysis class, the professor said we shouldn’t worry if we couldn’t complete any of the exercises, adding something along the lines of “If, during the course, you look back at the exercises from four weeks ago, you will find you can easily do them, and won’t even understand why anybody could have trouble doing them”. That turned out to be o so true.
These aren't the right lessons to learn from this issue.
Comparing floats for equality is not a good idea, not just because of the 0.1+0.2!=0.3 behaviour (which, yes, is at least consistent), but because floats - or numerical methods in general - are simply not meant to be used for exact results. If that's what you want - e.g. big integer or fixed point arithmetic - there are other types you should be using. So the takeaway is not that JS has a "glitch" or is messing up - but a misunderstanding of the guarantees of numerical methods.
Also, the suggested solution to compare actual vs. expected value by having their absolute distance be less than some epsilon is half correct, but what you really should be looking at is the relative error, i.e. the absolute error divided by the magnitude of the number.
> but what you really should be looking at is the relative error
This actually depends on the specific problem you're talking about, and there's also the issue of forward vs. backward error. Unless you're doing something that requires additional care, unit-testing against abs(expected - actual) < 0.001 is relatively safe (and if you are, then God help you).
100% agree on the rest of the comment, requiring numerical methods to converge to a specific IEEE 754 double precision floating point is just dumb.
I'm not very experienced with this in practice and can mostly only speak to what I was taught in numerical methods, but it was my understanding that a) relative errors don't arbitrarily depend on your units and b) they seem to be better behaved in theory, or at least, it's how algorithms are typically analysed - e.g. if you multiply two numbers that both have some associated error, the relative error is the sum of the individual relative errors (plus quadratic terms which you would ignore).
> Comparing floats for equality is not a good idea
This is very true, but the mention of data science is really significant here.
If you're trying to productionize a neural network or something similar, you ideally want to be able to pin down every source of noise to make sure your results are reproducible, so you can evaluate whether a change in your results reflects a change in the data, a change in the code, or just luck from your initial weights - with a big enough network, for all you know it decided that the 10th decimal place of some feature was predictive. I wouldn't be surprised if the author brings up tanh as an example specifically because it's a common activation function.
If you're pinning all your Javascript dependencies, but the native code it dynamically links to gets ripped out from under you, it could completely mislead you about what produces good results. Similarly, if you want to be able to ship a model to run client-side, it would be nice if you could just test it on Node with your Javascript dependencies pinned and be reasonably confident it'll run the same on the browser.
Of course, if you can't do that it's not the end of the world, since you can compensate by doing enough runs, and tests on the target platform, to convince yourself that it's consistent, but it's a lot nicer if things are just as deterministic as possible.
If you want true reproducibility you can save everything in an image/container. Lots of data science companies offer this for end-to-end reproducibility.
> for all you know it decided that the 10th decimal place of some feature was predictive
isn't this kind of a misfeature of the model? If it really depends that highly on a function output that's not guaranteed to be precise even in the best of cases and certainly not when you have to assume noisy input data... that doesn't seem to be particularly robust.
>> because floats - or numerical methods in general - are simply not meant to be used for exact results
But one should expect a single function does not vary in the last 5 digits as one example did. That's over 16 bits. Why should anyone use double precision if that's the kind of slop an implementation can have?
> But one should expect a single function does not vary in the last 5 digits as one example did.
Why not? The whole selling point of floats is that they're a fast approximation to real arithmetic in a bunch of useful cases. I'd personally be much more miffed by an excessively slow implementation of a transcendental than one which was off even by a large number of bits. If there's a fast solution with better precision then that's great, but low precision by itself doesn't strike me as particularly problematic.
> Why should anyone use double precision if that's the kind of slop an implementation can have?
Because by using float64 instead of float32 you can cheaply get a ton more precision even with sloppy algorithms. If there's only 16ish bits of slop you could probably get a full-precision float32 operation just by using an intermediate cast to float64 (proof needed for the algorithm in question of course).
Counting the number of different digits has no useful meaning. The example you cite has two results that differ by 1 part in 10 quadrillion. If you have a good reason care about the precise implementation of hyperbolic functions to that level of precision then you don't rely on whatever you happen to stumble on in some version of nodejs.
If you're talking about the tanh example, that differs in the last five bits of the mantissa. You don't want to count the number of different bits because differences in the more significant bits are worse. Count the ulps instead.
Yes, the implementation of Node’s Math package functions is changing, and yes, it's not otherwise standardized.
And one should not expect that the CPUs will ever standardize the results of the functions there. We are de facto left only with IEEE 754, and there are even regarding that continuous attempts to implement less.
So what's left is the wiggle room, but not only in JavaScript as the specification or specific interpreters, but in other languages too.
"stdlib" sounds good, but do they guarantee to never change any implementation of some math function?
I don't agree what the author saw was really "a glitch in the language itself." It's directly allowed by the language to happen.
Don't compare floats vs floats with an exact comparison.
I've recently seen a version of this error where someone's code was calculating how many bits a field needed to be to hold the maximum value they cared about.
bits = floor(log(max_value) / log(2)) + 1
This is playing with fire: it trusts that dividing one approximated irrational value divided by another approximated irrational value will yield a ratio equal or greater than the desired value when max_value is a power of two. Depending exactly how those approximations are rounded, log(8)/log(2) might return 2.999999999999 (bug!), or it might be 3.000000000 (their assumption), or epsilon more than 3.0 (not a bug).
This is one reason why even code that looks "obviously correct" probably deserves a test. If your assumption is true on one platform but not another, the test's failure on it will alert you of the fact.
That's especially a shame, because log2 of a floating point value is particularly easy to calculate, and most math libraries will internally calculate logs of other based (like ln) by finding the equivalent log2 first and then converting it.
And if you want just the floor of the base-2 log, you almost don't even need to calculate it at all; it's right there in the exponent bits.
most of this is solved by using big numbers, knowing about precision expansion for different fundamental operations, real numbers, and when to use rounding. it is an insightful article but honestly i wouldnt expect to use the float type for precise math. computers are fast as hell. a bigger question to me is where to actually use floats since they never seems to be what i want.
You use them when their downsides aren't excessive (e.g. the error is tolerable) and/or when their upsides are attractive (e.g. you need the speed of raw hardware floats, and you might be doing math on numbers of vastly different orders of magnitude).
Specific examples include differential equation solvers (vastly different orders of magnitude to deal with), plotting libraries (only have to be correct to within a pixel), and a ton of other graphics-related operations for that matter (only have to be correct to within a pixel, and most classes of small errors won't be important anyway).
Some amount of error can even be desirable (see a lot of the work the last half-decade in reduced precision neural networks).
Floats aren't bad -- they're a tool in your toolbox, and they won't hammer every nail.
For almost all of the problems I work on, I only care that the relative error is bounded. Floating point is perfect for me and, I would guess, most scientific calculations.
When Knuth wrote TeX he based all of its mathematical calculations on integer arithmetic. Its one of the reasons that software remains accurate and consistent regardless of platform. Using the corresponding stdlib approach in javascript is not so crazy, if your goal is accurate, consistent results across platforms including future not-yet-released platforms.
you can use integer arithmetic if you're dealing with integer values - or rational ones, since quotients are just (equivalence classes of) pairs of integers.
It doesn't really work when you're dealing with transcendental functions such as the trigonometric ones. There is simply no way to get "accurate" results for such calculations in general, unless of course, you deal with everything symbolically.
> What I was seeing between Node versions really should have been a bug in the testing library, or something in my code, or maybe in simple-statistics itself. But in this case, digging deeper revealed that what I was seeing was exactly what you don’t expect: a glitch in the language itself.
This is such a disheartening conclusion. It really was a bug in your code, describing it as a "glitch in the language" is disingenuous. Your code was assuming higher level of precision (or accuracy? I get those always mixed) from the system was providing, that is not a glitch in the language, but bug in your code making that false assumption.
This also leads to the question where did the original "13098426.039156161" reference value come from? Has he actually verified by hand that it is correct to the last decimal? WolframAlpha gives lot more decimals for the value, how did he decide that exact number of digits?
Of course hindsight 20/20, but numerical code just is generally much more difficult and complex than many people give credit to, at least before they get bitten by some issue like the author.
Sorry if I sound rude, but if one needs 15-digit precision for trigonometric functions, is javascript really a suitable choice for such a task (whatever it is)?
> Intel also bears some blame for overstating the accuracy of their trigonometric operations by many magnitudes. That kind of mistake is especially tragic because, unlike software, you can’t patch chips.
It is a great rabbit hole, however I have to note that the raised issue seems to be with float32, while JavaScript uses float64 ?
38 comments
[ 2.5 ms ] story [ 86.4 ms ] threadEven worse the language of math was never adequately explained. You were expected to learn the meaning of some new squiggle or greek letter by... its context? Symbols and terms you'd never seen before were just thrown at you without explanation. Definitions when given sounded circular or were given in terms of other things that had never been defined clearly.
I had one professor who just turned his back to the class and wrote things on the board. Students were meant to just magically understand.
I'll never forget in year one of college being absolutely stuck in calculus. I called up my dad and he tried to help me a bit, then he stopped and said: "you know what a derivative is, right?" I said no, not really. They tried to explain in class but it didn't click. He said "a derivative of a function is the rate of change of that function." I thought for a second and then said "thanks, now I understand calculus." I was un-stuck instantly. My professors never explained it that clearly.
Later on I took classes in things like population genetics and evolutionary dynamics. The professors of those classes explained the relevant math better than my math professors did to the point that if I'd taken those classes first I would have done better in the math classes that were their prerequisites.
> He said "a derivative of a function is the rate of change of that function."
I'm confused by this comment. That's the intuition that you see in just about every introductory calculus textbook: that the derivative is the instantaneous rate of change.
This was back in the late 1990s and early 2000s. I hope things have improved.
My conclusion is different though. I think there are different types of thinkers and different teaching styles and when the subject becomes very complex the disparity between the two is exacerbated. And luck plays the role in matching up with the right teacher/professor for you.
I was taught calculus in high school, well before university, in Australia in the late 1970s.
Visual analogies such derivatives being tangential rates of change and 2nd derivatives being local curvatures were taught and drawn on the board in the first week of two years of Calc I and Calc II classes that came before leaving high school to attend university.
Nearly all math textbooks and lectures mention practical applications. The problem rather seems to be that you have/had a different understanding of "practical" than your math professor.
I always had the benefit of being the tutor, though. I have to imagine that teaching a calculus one course is near impossible with 30 students on 30 different levels of experience and aptitude.
That said, I think that first year of calculus is often taught valuing mechanics over concepts.
I like to imagine what 12 years of math would look like if people weren't chained to a timeline. Here's khanacademy, go!
https://betterexplained.com/articles/colorized-math-equation...
Chances are your teachers said almost, if not exactly, the same, but it didn’t stick yet.
In my first analysis class, the professor said we shouldn’t worry if we couldn’t complete any of the exercises, adding something along the lines of “If, during the course, you look back at the exercises from four weeks ago, you will find you can easily do them, and won’t even understand why anybody could have trouble doing them”. That turned out to be o so true.
Comparing floats for equality is not a good idea, not just because of the 0.1+0.2!=0.3 behaviour (which, yes, is at least consistent), but because floats - or numerical methods in general - are simply not meant to be used for exact results. If that's what you want - e.g. big integer or fixed point arithmetic - there are other types you should be using. So the takeaway is not that JS has a "glitch" or is messing up - but a misunderstanding of the guarantees of numerical methods.
Also, the suggested solution to compare actual vs. expected value by having their absolute distance be less than some epsilon is half correct, but what you really should be looking at is the relative error, i.e. the absolute error divided by the magnitude of the number.
This actually depends on the specific problem you're talking about, and there's also the issue of forward vs. backward error. Unless you're doing something that requires additional care, unit-testing against abs(expected - actual) < 0.001 is relatively safe (and if you are, then God help you).
100% agree on the rest of the comment, requiring numerical methods to converge to a specific IEEE 754 double precision floating point is just dumb.
This is very true, but the mention of data science is really significant here.
If you're trying to productionize a neural network or something similar, you ideally want to be able to pin down every source of noise to make sure your results are reproducible, so you can evaluate whether a change in your results reflects a change in the data, a change in the code, or just luck from your initial weights - with a big enough network, for all you know it decided that the 10th decimal place of some feature was predictive. I wouldn't be surprised if the author brings up tanh as an example specifically because it's a common activation function.
If you're pinning all your Javascript dependencies, but the native code it dynamically links to gets ripped out from under you, it could completely mislead you about what produces good results. Similarly, if you want to be able to ship a model to run client-side, it would be nice if you could just test it on Node with your Javascript dependencies pinned and be reasonably confident it'll run the same on the browser.
Of course, if you can't do that it's not the end of the world, since you can compensate by doing enough runs, and tests on the target platform, to convince yourself that it's consistent, but it's a lot nicer if things are just as deterministic as possible.
> for all you know it decided that the 10th decimal place of some feature was predictive
isn't this kind of a misfeature of the model? If it really depends that highly on a function output that's not guaranteed to be precise even in the best of cases and certainly not when you have to assume noisy input data... that doesn't seem to be particularly robust.
But one should expect a single function does not vary in the last 5 digits as one example did. That's over 16 bits. Why should anyone use double precision if that's the kind of slop an implementation can have?
Why not? The whole selling point of floats is that they're a fast approximation to real arithmetic in a bunch of useful cases. I'd personally be much more miffed by an excessively slow implementation of a transcendental than one which was off even by a large number of bits. If there's a fast solution with better precision then that's great, but low precision by itself doesn't strike me as particularly problematic.
> Why should anyone use double precision if that's the kind of slop an implementation can have?
Because by using float64 instead of float32 you can cheaply get a ton more precision even with sloppy algorithms. If there's only 16ish bits of slop you could probably get a full-precision float32 operation just by using an intermediate cast to float64 (proof needed for the algorithm in question of course).
And one should not expect that the CPUs will ever standardize the results of the functions there. We are de facto left only with IEEE 754, and there are even regarding that continuous attempts to implement less.
So what's left is the wiggle room, but not only in JavaScript as the specification or specific interpreters, but in other languages too.
"stdlib" sounds good, but do they guarantee to never change any implementation of some math function?
I don't agree what the author saw was really "a glitch in the language itself." It's directly allowed by the language to happen.
I've recently seen a version of this error where someone's code was calculating how many bits a field needed to be to hold the maximum value they cared about.
This is playing with fire: it trusts that dividing one approximated irrational value divided by another approximated irrational value will yield a ratio equal or greater than the desired value when max_value is a power of two. Depending exactly how those approximations are rounded, log(8)/log(2) might return 2.999999999999 (bug!), or it might be 3.000000000 (their assumption), or epsilon more than 3.0 (not a bug).And if you want just the floor of the base-2 log, you almost don't even need to calculate it at all; it's right there in the exponent bits.
Specific examples include differential equation solvers (vastly different orders of magnitude to deal with), plotting libraries (only have to be correct to within a pixel), and a ton of other graphics-related operations for that matter (only have to be correct to within a pixel, and most classes of small errors won't be important anyway).
Some amount of error can even be desirable (see a lot of the work the last half-decade in reduced precision neural networks).
Floats aren't bad -- they're a tool in your toolbox, and they won't hammer every nail.
It doesn't really work when you're dealing with transcendental functions such as the trigonometric ones. There is simply no way to get "accurate" results for such calculations in general, unless of course, you deal with everything symbolically.
This is such a disheartening conclusion. It really was a bug in your code, describing it as a "glitch in the language" is disingenuous. Your code was assuming higher level of precision (or accuracy? I get those always mixed) from the system was providing, that is not a glitch in the language, but bug in your code making that false assumption.
This also leads to the question where did the original "13098426.039156161" reference value come from? Has he actually verified by hand that it is correct to the last decimal? WolframAlpha gives lot more decimals for the value, how did he decide that exact number of digits?
Of course hindsight 20/20, but numerical code just is generally much more difficult and complex than many people give credit to, at least before they get bitten by some issue like the author.
However, the point was not so much about precision, it was about consistency.
It is a great rabbit hole, however I have to note that the raised issue seems to be with float32, while JavaScript uses float64 ?