Not had chance to look at this in detail but this seems more likely to be an artefact of the rounding in FMADD vs separate instructions rather than a 'flaw' in the instruction.
There seems to have been a similar issue in Chromium a few years ago.
"Having said that, fmadd is doing exactly what it should do; it isn't
broken. The model that gave you a result of 0x4 _is_ broken, but the
hardware isn't."
I think this is the key. My understanding is the FMADD on arm64 does what the spec says it should do, so there's nothing broken.
Next up though is whether fused multiply add is part of the IEEE floating point standards, and whether the arm implementation meets these. Chances are it does (I see references to it from 2008). If the x64 FMADD produces difference results, it's likely that either the standard is wooly in some respects and allows wiggle room (so they both meet the standard), or, the Intel/AMD implementation pre-dates the standard and isn't compliant.
I'm sure someone who knows these standards and processors better than me will be along in a minute to comment.
I wouldn't be surprised if the difference in rounding is an innate property of how the math behind an FMA instruction works and that guaranteeing the same rounding (which is essentially rounding twice, once after mul and once after add) heavily impacts the performance benefits. Thus implementing FMA with an inconsistent rounding to separate mul and add instruction is in the interest of performance.
The standard mandates that the result of fma is rounded as-if an infinite precision result is computed internally followed by a single correct rounding. There isn't any wiggle room.
For finite inputs, the results are unambiguously specified by the IEEE spec. For singularities involving zero/infinite inputs, the IEEE spec allows two sets of behavior, which basically correspond to x86 and everyone else.
> The operation fusedMultiplyAdd(x, y, z) computes (x × y) + z as if with unbounded range and precision, rounding only once to the destination format. [...] fusedMultiplyAdd differs from a multiplication operation followed by an addition operation.
[elided exception discussion]
I expect that the ARM processor precisely implements this standard, as does x86-64. Any "bugs" here are the user misunderstanding how the precision is supposed to work.
Don't CPU compilers in general, not just ARM ones, avoid using FMA for normal arithmetic unless you use -ffast-math? I don't think FMAs are inaccurate per se, but they will produce differently-rounded results than multiplications and additions, and moreover you have to do a bunch of reassociation to make good use of them. C and C++ expect precise math by default.
gcc will avoid FMA if you specify `-std=c17` (or another C standard, instead of the default "GNU" variant). Note that it has to avoid FMA in that case because fusing (a * b) + c would not comply with the C17/IEEE-754 spec (see -fexcess-precision in the manual).
It's good to know there's a performance-accuracy tradeoff.
Is the error the same in non-Apple targets? Does replacing FMADD with FNMADD (negative mult) or FNSUB change the magnitude of error in this case? Also, would the relative error be the same if the numbers were random integers or random floats of a given range (-1 to 1; -0.01 to 0.01; -100 to 100; 0 to 1) rather than integers in sequence?
-----
In most robotics and AI applications, calculations are unlikely to suffer greatly from inaccuracy but benefit from speed. What are the applications where someone would require floating point math AND both extreme speed and extreme accuracy?
> What are the applications where someone would require floating point math AND both extreme speed and extreme accuracy?
Scientific computation.
I'm not even sure, it is a trade-off between (computational) speed and accuracy.
In my experience, error-analysis was a mandatory course for physics, but not in robotics.
The error analysis was manual and time-consuming.
In robotics, no one cared why it occasionally produced NaNs, etc.. people just added a small number here and there.
That's faster to program.
To put it more positively, I think, the view here is more having an application, which is robust in the face of errors / edge cases. What counts is the output (action).
While in science the correct model is part of the output. Edge cases are an important part of it.
>In most robotics and AI applications, calculations are unlikely to suffer greatly from inaccuracy but benefit from speed.
FMA instructions are more accurate and faster compared to a sequence of separate multiply and add/sub instructions, not less. The article literally talks about it in the beginning. The issue is reproducibility, which with floats can be quite bad even without FMA.
Also both robotics and neural networks (I dislike the tendency to call it "AI") can be quite susceptible to errors. Gradient-based optimization algorithms can suffer noticeably from calculation errors. This is why you usually want to train ANNs using 64-bit floats.
It's important in finance. Not so much because the last decimals are really important, but when they perform reconciliation they want to ensure that numbers remain consistent over multiple runs, and if one machine returns different values it comes up as errors when the reports are compared, which can trigger all sorts of processes that can be very costly.
The amount of effort and extra hardware required to maintain this reproducibility across different hardware (cpu generations, GPU hardware/drivers, etc) feels insane to me. For example, people seem extremely interested in performance, but insist on using the much slower modes in MKL to maintain the bottom few bits of their golden data.
Do you find this genuinely important? If not, have you had any success encouraging e.g. validating against golden data in a non-bitwise comparison?
I personally don't find it important. And I have seen that the demand of having numbers match exactly comes from a misunderstanding of how floating point calculations work.
Sometimes this is understood by the banks and the solution is to implement a tolerance in the comparisons. In other cases other solutions may have to be used, such as forcing certain reports to be computed on certain software/hardware.
That's not what I'm talking about. Floating point computations happen when doing risk calculations, for example computing risk numbers in a Black-Sholes calculation.
A difference in the 15'th decimal has no impact on the risk number.
I encountered this in a scientific computing context where a Monte Carlo simulation was designed to give reproducible results across runs. I've forgotten which architectures (Power, Alpha?) hit this but basically some fraction of the test suite failed because the same starting conditions gave slightly different results due to FMADD using a greater precision internal representation before the final round, which meant that over time the particles being simulated would be in different positions (very close, of course).
As far as I can tell, this article doesn’t give any example where fused multiply-add gives a worse result than multiply, then add.
Without such a smoking gun, I would guess the explanation for this is that “subtract c, then divide by b” is a better inverse for “multiply by b, then add c” than for the fused multiply-add.
This seems to be a flawed approach. Comparing the result to sub/div is not computing the error in the fused-mac. It is calculating the difference between one rounding operation and two.
To do this correctly requires:
* performing the calculation with a higher degree of accuracy (and precision).
* comparing the two approaches against the reference results.
To do this you should look into either writing a simulation of ieee with a larger mantissa or using an arbitrary precision library.
I think you can model the process as a random walk, and I'd expect 2 roundings/iteration to stray away from the true value about sqrt(2) more. But it seems that the fused iteration is actually worse than the manual one. Could be this is because of the reverse operation and inter-iteration dependencies.
I wrote a quick-and-dirty test using a simple dot product as the basis for testing precision. (Dot product-like operations are a relatively common use case for FMA).
Which result is more precise depends on the input data (I randomized it), but it seems that most commonly, there was no actual difference using FMA and not using it, and when there was a difference, it was even odds which one was the more precise result.
The consensus seems to be that there likely isn't a problem with ARM64 FMADD compared to the IEEE standard. In light of this and the recommendation in the post - which seems to be potentially misleading.
"If you use different tools and want to ensure best results from floating point arithmetic on ARM64, you may wish to check that code generation doesn’t use fused instructions, particularly on large loops which could accumulate significant errors. It’s worth bearing in mind that authoritative texts on floating-point arithmetic are also extremely cautious about the use of such fused instructions."
I do hope that the original poster updates the post to reflect some definitive and validated conclusions. (And also not rely on Betteridge's Law for the headline!)
A difference between the fused instruction and the two separate ones is expected. The fused instruction is designed to avoid a rounding step between the multiplication and add.
The article makes no mention if the fused instruction is producing incorrect results or just different results. That is the critical question.
The result of (IEEE 754) operations -- including PLUS, MINUS and FMA -- is specified bit-for-bit, at least for `isnormal()` numbers. It is not a matter of vaguely "more precise" or "less precise".
The test performed brings no information towards answering the question, because the loss of precision precision in one particular complex algorithm can perfectly well happen despite one individual operation having better worst-case precision.
To check the unlikely event of a flaw in M1's FMA, one would need to compare the output of invidual FMAs with MPFR's output on the same operands.
> ARM64 v Intel: I haven’t attempted to look at Intel processor fused instructions
Instead, this test could have been a clue. FP64 results follow the spec and hence must be bit-for-bit equal across platforms (as long as one avoids legacy x87 on Intel).
Okay I got nerdsniped by this, reimplemented and reproduced it. As others mentioned, the article does not show a flaw in the M1's FMA. The results are the same on x86_64. It is just due to how this particular formula interacts with floating-point rounding, and it all depends on the order of magnitude of the numbers.
The basic operation here is
x = (((a * b) + c) - c) / b
which would yield (x == a) in exact arithmetic.
Let's denote by fp(y) the floating-point number closest to y. If you don't use FMA, then the middle of the formula is essentially fp(fp(M + c) - c) for some M. When you pick the natural choice of initial a, b and c having the same order of magnitude, this gives exactly M in 99.99% of cases. Then you're just measuring the difference between a and (a * b) / b. Instead, FMA gives you x = fp(fp(fp(a * b + c) - c) / b), which is admittedly more complex to study.
But as you pick larger and larger c values, you can observe that FMA becomes more accurate compared to the other version. When c reaches about 0.5e+6 times larger than a and b, FMA starts winning. Note that a is incremented 1M times as the proposed algorithm iterates, so this corresponds to (a * b) and c having the same orders of magnitude on average, as the loop progresses.
You can probably go much more in-depth in studying this, and explain it better. But essentially this is a non-issue, just floating-point being floating-point.
If you want to play with the code, you can find it here:
Also, the article concludes that FMA hurts accuracy, and this is at best misleading (FMA is better in the worst case). But the example given does highlight something important:
Depending on the case, the accuracy gains can be tiny (so much so that in this example the FMA version does indeed return larger errors in most runs), and the performance gains can be small too.
On the other hand, the price to pay for letting the compiler freely decide whether/when to fuse * and + is huge: reproducibility. Since x87 was deprecated, having floating-point results be bit-for-bit consistent across platforms, compilers and optimization flags has been amazing for debugging. Losing that is a step back and a recipe for headaches.
I was surprised that gcc enables -fexcess-precision=fast by default (in GNU mode), hence enabling transparent FMA and potential headaches. Specifying the standard (e.g. -stdc=c17) fixes that.
Iirc x87 was even worse on some processors depending on the processor state, the processor could cache the CPU state on context switch and revert the x87 register to a f64, leading to irreproducibility at the runtime level (randomly fails if you get an unlucky thread thrash)
> When c reaches about 0.5e+6 times larger than a and b, FMA starts winning.
It should be noted that 1e+6 sticks out as right on the rounding limit of single precision floating point numbers, which is about 1e+7. Thus it suggests that the residue amplification from b ceases to be a problem once b simply falls out of the picture as it gets close to a roundoff error.
Therefore, this test isn't evaluating FMA as much as it is showcasing how floating point division amplifies numerical residues.
38 comments
[ 2.8 ms ] story [ 97.2 ms ] threadThere seems to have been a similar issue in Chromium a few years ago.
https://bugs.chromium.org/p/chromium/issues/detail?id=401117
Where the conclusion was:
"Having said that, fmadd is doing exactly what it should do; it isn't broken. The model that gave you a result of 0x4 _is_ broken, but the hardware isn't."
Next up though is whether fused multiply add is part of the IEEE floating point standards, and whether the arm implementation meets these. Chances are it does (I see references to it from 2008). If the x64 FMADD produces difference results, it's likely that either the standard is wooly in some respects and allows wiggle room (so they both meet the standard), or, the Intel/AMD implementation pre-dates the standard and isn't compliant.
I'm sure someone who knows these standards and processors better than me will be along in a minute to comment.
I'm just guessing here tho.
> The operation fusedMultiplyAdd(x, y, z) computes (x × y) + z as if with unbounded range and precision, rounding only once to the destination format. [...] fusedMultiplyAdd differs from a multiplication operation followed by an addition operation.
[elided exception discussion]
I expect that the ARM processor precisely implements this standard, as does x86-64. Any "bugs" here are the user misunderstanding how the precision is supposed to work.
GLSL and the like are different.
Doesn't seem o be the case: MSVC avoids FMA, while GCC prefers it: https://godbolt.org/z/e9EcfjdeY
gcc and icc will use fma under `-O3`, but Clang won't.
Is the error the same in non-Apple targets? Does replacing FMADD with FNMADD (negative mult) or FNSUB change the magnitude of error in this case? Also, would the relative error be the same if the numbers were random integers or random floats of a given range (-1 to 1; -0.01 to 0.01; -100 to 100; 0 to 1) rather than integers in sequence?
-----
In most robotics and AI applications, calculations are unlikely to suffer greatly from inaccuracy but benefit from speed. What are the applications where someone would require floating point math AND both extreme speed and extreme accuracy?
Scientific computation.
I'm not even sure, it is a trade-off between (computational) speed and accuracy.
In my experience, error-analysis was a mandatory course for physics, but not in robotics. The error analysis was manual and time-consuming.
In robotics, no one cared why it occasionally produced NaNs, etc.. people just added a small number here and there.
That's faster to program.
To put it more positively, I think, the view here is more having an application, which is robust in the face of errors / edge cases. What counts is the output (action).
While in science the correct model is part of the output. Edge cases are an important part of it.
FMA instructions are more accurate and faster compared to a sequence of separate multiply and add/sub instructions, not less. The article literally talks about it in the beginning. The issue is reproducibility, which with floats can be quite bad even without FMA.
Also both robotics and neural networks (I dislike the tendency to call it "AI") can be quite susceptible to errors. Gradient-based optimization algorithms can suffer noticeably from calculation errors. This is why you usually want to train ANNs using 64-bit floats.
Do you find this genuinely important? If not, have you had any success encouraging e.g. validating against golden data in a non-bitwise comparison?
Sometimes this is understood by the banks and the solution is to implement a tolerance in the comparisons. In other cases other solutions may have to be used, such as forcing certain reports to be computed on certain software/hardware.
You remind me of the alleged UK Post Office scandal:
https://www.bbc.com/news/business-56718036
A difference in the 15'th decimal has no impact on the risk number.
The truth is, I have faith in Murphy's law.
Without such a smoking gun, I would guess the explanation for this is that “subtract c, then divide by b” is a better inverse for “multiply by b, then add c” than for the fused multiply-add.
To do this correctly requires:
* performing the calculation with a higher degree of accuracy (and precision).
* comparing the two approaches against the reference results.
To do this you should look into either writing a simulation of ieee with a larger mantissa or using an arbitrary precision library.
Which result is more precise depends on the input data (I randomized it), but it seems that most commonly, there was no actual difference using FMA and not using it, and when there was a difference, it was even odds which one was the more precise result.
"If you use different tools and want to ensure best results from floating point arithmetic on ARM64, you may wish to check that code generation doesn’t use fused instructions, particularly on large loops which could accumulate significant errors. It’s worth bearing in mind that authoritative texts on floating-point arithmetic are also extremely cautious about the use of such fused instructions."
I do hope that the original poster updates the post to reflect some definitive and validated conclusions. (And also not rely on Betteridge's Law for the headline!)
The article makes no mention if the fused instruction is producing incorrect results or just different results. That is the critical question.
The result of (IEEE 754) operations -- including PLUS, MINUS and FMA -- is specified bit-for-bit, at least for `isnormal()` numbers. It is not a matter of vaguely "more precise" or "less precise".
The test performed brings no information towards answering the question, because the loss of precision precision in one particular complex algorithm can perfectly well happen despite one individual operation having better worst-case precision.
To check the unlikely event of a flaw in M1's FMA, one would need to compare the output of invidual FMAs with MPFR's output on the same operands.
> ARM64 v Intel: I haven’t attempted to look at Intel processor fused instructions
Instead, this test could have been a clue. FP64 results follow the spec and hence must be bit-for-bit equal across platforms (as long as one avoids legacy x87 on Intel).
The basic operation here is
which would yield (x == a) in exact arithmetic.Let's denote by fp(y) the floating-point number closest to y. If you don't use FMA, then the middle of the formula is essentially fp(fp(M + c) - c) for some M. When you pick the natural choice of initial a, b and c having the same order of magnitude, this gives exactly M in 99.99% of cases. Then you're just measuring the difference between a and (a * b) / b. Instead, FMA gives you x = fp(fp(fp(a * b + c) - c) / b), which is admittedly more complex to study.
But as you pick larger and larger c values, you can observe that FMA becomes more accurate compared to the other version. When c reaches about 0.5e+6 times larger than a and b, FMA starts winning. Note that a is incremented 1M times as the proposed algorithm iterates, so this corresponds to (a * b) and c having the same orders of magnitude on average, as the loop progresses.
You can probably go much more in-depth in studying this, and explain it better. But essentially this is a non-issue, just floating-point being floating-point.
If you want to play with the code, you can find it here:
https://godbolt.org/z/WWqdcrzdh
Depending on the case, the accuracy gains can be tiny (so much so that in this example the FMA version does indeed return larger errors in most runs), and the performance gains can be small too.
On the other hand, the price to pay for letting the compiler freely decide whether/when to fuse * and + is huge: reproducibility. Since x87 was deprecated, having floating-point results be bit-for-bit consistent across platforms, compilers and optimization flags has been amazing for debugging. Losing that is a step back and a recipe for headaches.
I was surprised that gcc enables -fexcess-precision=fast by default (in GNU mode), hence enabling transparent FMA and potential headaches. Specifying the standard (e.g. -stdc=c17) fixes that.
It should be noted that 1e+6 sticks out as right on the rounding limit of single precision floating point numbers, which is about 1e+7. Thus it suggests that the residue amplification from b ceases to be a problem once b simply falls out of the picture as it gets close to a roundoff error.
Therefore, this test isn't evaluating FMA as much as it is showcasing how floating point division amplifies numerical residues.
The number line for floats is non-linear. It has holes in it.