If your code changes behavior based on the timing of the various sections, you have a bug, a race condition. The solution isn't to lock down your code and build with only one configuration. The solution is to fix the bug.
In the same way, if the behavior of your code is dependent on the order of operations in floating point, it is very likely that you are doing something wrong. The solution is not careful ordering of floating point operations, but using algorithms appropriate to floating-point (and avoiding floating point when it isn't applicable.)
Code that has been ported to many platforms tends to be more stable, because the heterogeneous environments have exposed bugs and flawed assumptions in the code's construction.
Simple use case: a multiplayer game containing physics. I'd like the physics to be reasonably accurate, but more than anything it needs to be consistent, because I want to get the same results on both players' machines, and then the only thing I have to send over the slow internet is a command stream.
No physics engine is stable in that sense. Neither is the real world. You need full determinism to do what lmm is saying. Source: Worked on this exact thing for commercial multiplayer games.
Many interesting physical phenomena are deterministic but chaotic, in the sense that arbitrarily tiny changes in initial conditions can rapidly lead to arbitrarily large changes in behavior.
Would you say that anyone who tries to make a game that has the same property is "doing something wrong"?
You're probably imagining a short-lived game. Think of a game that goes on for months at a time. There are ways around this, but none of them as elegant as a floating point routine which produces the same results on any platform across any ordering of calculations.
> But if infinitesimal disturbances to the results makes a practical difference to the game
They don't! The system can tolerate answers which are slightly wrong... as long as all machines calculate the SAME slightly-wrong answer.
The same logic applies to synchronized PRNGs. It doesn't (directly) matter what the "random" dice-roll result is, as long as all clients get the same answer.
Ah, thank you for this response. It "clicked" for me: Game developers are using the low bits (below significance) as pseudorandom numbers! They are overloading a new function on top of the existing use of floating-point numbers.
It's a feature of physics, not physics engines. Many process in the real world are exponential in nature, so even a very small difference gets magnified very quickly.
I read the article so you don't have to! The author dreams of consistent behavior so he can get:
- Reproducible results across platforms
- Reproducible results with different build settings (such as debug builds or optimization levels)
- Useful test cases for regression testing
- Test cases that help him find misbehavior in the optimizer
- An improved chance of finding more subtle bugs, absent 'noise' from small changes in floating point operations
In particular, he gives examples of how changes to nearby code can change the output of floating point operations, due to hairiness in how registers are allocated. You can see how this would make testing quite painful.
These all seem like worthy goals that deserve a better response than "you're doing it wrong", along with some non-sequitur about race conditions.
Regarding race conditions, I was referring to this:
> It's not just floating point that's inconsistent across modes – it's code snippets with behavior undefined by the language, buggy dependence on timing, optimizer bugs, conditional compilation, etc.
There's quite a few algorithms that are generally vulnerable to precision issues because they can affect comparison order, such as k-means clustering. "Don't do that then" is not a useful answer.
I understand that avoids some platform-capability differences, but does it also protect from issues with execution-order or ghostly-coupling between threads?
The original post said he'd be happy to trade performance for consistency.
He could also use fdlibm ( http://netlib.org/fdlibm/index.html ), at least for anything other than primitive operations. But, again, it trades performance for consistency.
I have often found myself closer to the opposite end of the spectrum. Many machine learning algorithms and applications (certainly not all) can tolerate a fair degree of inaccuracy. This is often justified by the fact that the training data is typically noisy to begin with and is a random sample from the space it lives in. If your algorithm can be stable under those conditions it ought to be stable under floating point jitter.
This usually means you need regularization, or smoothing, or low pass filter, or averaging in some guise or the other (in some abstract way they are the same thing). There are some frequently used operations in ML that are very unstable under inaccuracies, numeric differentiation is one, just dont do it without smoothening. Inverting matrices is another, but you better have a real solid reason to ever invert a matrix, yes there are reasons when you do really want the inverse.
In any case the final point is that I have had to baby talk the compilers into, "yes, please please dont worry about it, just do these commutative (for real numbers) operations in any order that you want". This exposes more parallelism and other compiler optimizations. Often I would gladly take the hit over some precision in higher order decimal places to gain speed. But this should not be done blindly, and a smattering of numerical analysis helps gauging when it would be safe to do so. I doubt whether numerical analysis figures prominently in the trajectory of a graduating machine learner, although I think it ought to.
Indeed. Do you have some specific examples in mind, I would love knowing more. The usual suspects, for example fold_left, fold_right would typically adhere to fixed order for floating point operations. A reason Java is a FLOPs killer is that its standard is too strictly specified, it does not leave much room for the runtime to take advantage of these features. Even order of argument evaluation for functions are specified to be left to right.
If you're a game developer, you should look at this Floating Point Determinism article, which shows some additional techniques for implementing deterministic physics with floating point.
Getting consistently worse results? :) He didn't even mention when matlab 5 started using his 64b only recommendation, the results were getting in-precise and matlab was getting unusable. http://www.cs.berkeley.edu/~wkahan/MxMulEps.pdf
I cannot take a floating point rant seriously without knowing Kahan's arguments. And a crazy requirement to compare it to slow and false debugging code does not really help.
29 comments
[ 3.5 ms ] story [ 68.5 ms ] threadIf your code changes behavior based on the timing of the various sections, you have a bug, a race condition. The solution isn't to lock down your code and build with only one configuration. The solution is to fix the bug.
In the same way, if the behavior of your code is dependent on the order of operations in floating point, it is very likely that you are doing something wrong. The solution is not careful ordering of floating point operations, but using algorithms appropriate to floating-point (and avoiding floating point when it isn't applicable.)
Code that has been ported to many platforms tends to be more stable, because the heterogeneous environments have exposed bugs and flawed assumptions in the code's construction.
Would you say that anyone who tries to make a game that has the same property is "doing something wrong"?
They don't! The system can tolerate answers which are slightly wrong... as long as all machines calculate the SAME slightly-wrong answer.
The same logic applies to synchronized PRNGs. It doesn't (directly) matter what the "random" dice-roll result is, as long as all clients get the same answer.
- Reproducible results across platforms
- Reproducible results with different build settings (such as debug builds or optimization levels)
- Useful test cases for regression testing
- Test cases that help him find misbehavior in the optimizer
- An improved chance of finding more subtle bugs, absent 'noise' from small changes in floating point operations
In particular, he gives examples of how changes to nearby code can change the output of floating point operations, due to hairiness in how registers are allocated. You can see how this would make testing quite painful.
These all seem like worthy goals that deserve a better response than "you're doing it wrong", along with some non-sequitur about race conditions.
> It's not just floating point that's inconsistent across modes – it's code snippets with behavior undefined by the language, buggy dependence on timing, optimizer bugs, conditional compilation, etc.
He could also use fdlibm ( http://netlib.org/fdlibm/index.html ), at least for anything other than primitive operations. But, again, it trades performance for consistency.
Sure, 0.1 can be represented exactly. But that's about the only issue with floating-point that that solves.
This usually means you need regularization, or smoothing, or low pass filter, or averaging in some guise or the other (in some abstract way they are the same thing). There are some frequently used operations in ML that are very unstable under inaccuracies, numeric differentiation is one, just dont do it without smoothening. Inverting matrices is another, but you better have a real solid reason to ever invert a matrix, yes there are reasons when you do really want the inverse.
In any case the final point is that I have had to baby talk the compilers into, "yes, please please dont worry about it, just do these commutative (for real numbers) operations in any order that you want". This exposes more parallelism and other compiler optimizations. Often I would gladly take the hit over some precision in higher order decimal places to gain speed. But this should not be done blindly, and a smattering of numerical analysis helps gauging when it would be safe to do so. I doubt whether numerical analysis figures prominently in the trajectory of a graduating machine learner, although I think it ought to.
* (a,b,c) can take advantage of those tricks, whereas the programmer can still be explicit and go * (a, * (b, c)), for example.
http://gafferongames.com/networking-for-game-programmers/flo...
I cannot take a floating point rant seriously without knowing Kahan's arguments. And a crazy requirement to compare it to slow and false debugging code does not really help.