It's odd. Someone told me about that on irc at the Time. People were feeling strongly about the topic. Quackery, mythomany,.. nobody wanted to hear about anything but IEEE standards.
Lots and lots of people have proposed replacements for IEEE floats that fix (or claim to fix) various problems with floats. 90% of them are utter nonsense, so many people default to assuming that any new one they encounter is also utter nonsense, and they usually turn out to be right.
> Quackery, mythomany,.. nobody wanted to hear about anything but IEEE standards.
Mostly because there is no good evidence that what is being proposed is better.
This isn't the olden days when it was difficult to demonstrate on a large enough CPU and dataset. Today we have cloud computing. If you create something better, you can demonstrate it by putting it into a numerics application and blow everybody away.
The CFD people are always looking for better solutions. The numerical simulation people are always constrained.
Until you do that, people have a right to blow you off.
That only really compares posits to Float16 in a domain where the paper admits roundoff error mostly isn't really an issue even at single precision.
I'd be much more interested to see how these handle stiff systems with multiple time constants. That's a domain where everything has problems and improvement is going to actually move some things from completely infeasible to actually simulatable.
That's a much stronger use case. And you don't have to worry about implementation efficiency since the improvement ratio is "infinity"--you went from can't do it at all to actually being able to do it.
Here is the official site[1] of the project. There is a request[2] to add it in the Scryer Prolog[3] (ISO Prolog implementation in Rust). And implementations in Rust[4] itself, and Julia[5] language.
Besides claimed efficiency improvements, posits are claimed to have advantages for implementing numerical algorithms correctly compared to IEEE floats. A big part of Gustafson's book The End of Error is devoted to that, claiming to show that a range of numerical algorithms are implemented better with posits vs IEEE floats (better meaning some mix of clearer, easier to produce stability, better error bounds, etc.). It's something of a competitor to interval arithmetic in that respect [1].
A 'softposit' implementation can at least let people experiment with writing posit algorithms to investigate the claimed algorithm-design benefits, even if it's not going to beat hardware floats in speed.
Yes, those implementation are slow compared to classical floating points. The goal, as I understand it, is to let people experiment with Posits in order to prove that they bring something to the table.
As far as I read https://posithub.org/docs/Posits4.pdf there are FPGA implementations.
Also in systems where floating point arithmetic isn't supported in hardware (i.e. Arduino Uno) you rely on soft fp.
Benchmark performance analysis of accuracy and speed on AVR platforms would indeed be very interesting and an actual, possible, immediate implementation scenario...
He will keep failing to replace IEEE floating point as long as he insists on making NEGATIVE infinity the same as POSITIVE infinity.
Also, IEEE 754 floating point standard guarantees the results of addition, subtraction, multiplication, division, and square root to be the exact correctly rounded value, ie a deterministic result, contrary to what he says.
Not so sure. I mean, yes, what you say is right, but there are problems nevertheless, see eg Wikipedia:
> Reproducibility
> The IEEE 754-1985 allowed many variations in implementations (such as the encoding of some values and the detection of certain exceptions). IEEE 754-2008 has strengthened up many of these, but a few variations still remain (especially for binary formats). The reproducibility clause recommends that language standards should provide a means to write reproducible programs (i.e., programs that will produce the same result in all implementations of a language), and describes what needs to be done to achieve reproducible results.
Reproducibility is an orthogonal issue to Posits vs IEEE754.
Most developers prefer speed over reproducibility, and are encouraged to use denormal-to-zeros, fast-math optimizations, fused MAC, approximated square root function and whatever else is available to achieve results.
The IEEE754 standard provides a guarantee for deterministic results, and many multi-precision and interval arithmetic libraries depend on this guarantee to be true to function properly.
IEEE754 defines unique -infinity and +infinity values, and any "new and improved" standard that breaks this axiom is just incompatible with all existing floating-point libraries written in the last +30 years.
"These claims pander to Ignorance and Wishful Thinking." Kahan (main author of IEEE754) on Posits claims.
You're free to voice your own opinion, but I take some issue with people asserting theirs as if they speak for "most developers". Especially if it comes from a new account with a name like "Gustafnot". That doesn't exactly scream "unbiased" to me.
Having written scientific numerical software for decades, and having been in situations where I want high speed or I want reproducibility, and having worked with hundreds of developers, I agree wholeheartedly with Gustafnot - the vast majority of developers prefer performance over bitwise reproducibility. Lose a few bits here and there, and most don’t care, because they treat floats as fuzzy to begin with (and almost never care about reproducibility since it’s very hard to obtain due to compilers, libraries, etc.) . But slow down code, and they sure notice quickly.
If you really want to claim the opposite, do you have evidence? Or experience that it’s true?
He argues that underflow/overflow are usually caused by bugs in the code. Having so many special numbers implies reducing the numeric representation (there are a lot of NaN in IEEE-754) and a hardware overhead to deal with all the special cases. That small hardware overhead can add up when working with thousands of FPU.
The vast majority of application dosen't need fine control of the FPU. There is always will be hardware for the few application need the ieee-754 features.
Obviously for any 32 bit value there’s only 2^32 possible values. So logically, a better number format is all about the distribution of values and reducing redundancy.
It sounds like the basic idea here is that rather than a fixed amount of significant figures, you get more precision in the middle. Meanwhile, you can also get larger exponents. Is that right?
The ideas are (1) gradual instead of hard overflow [IEEE floats do gradual underflow via denormals], (2) crowding representable numbers more precisely around 1 while letting very large or very small numbers get less and less precise, vs. floating point which (except for denormals) is scale free within its range, (3) making a format which can be extended by just adding bits without new definitions, so that you can get more or less precise numbers as needed by just adding zeros or truncating, (4) a it more uniform logic for basic arithmetic with fewer edge cases.
It's a bit sad that many of the creator’s claims are exaggerated and his examples cherry-picked though.
> floating point which (except for denormals) is scale free within its range
Presumably that's what you mean by scale free, but let's spell it out:
It's denser around 0, then has uniform density between, say, 1/8 and 1/4, half that density between 1/4 and 1/2, half that between 1/2 and 1, half that between 1 and 2, etc. etc.
If x = 1.0e22, you can add a million and it's still the same.
I recall in some of these earlier conversations, people pointing out that there are game engines where they invert some of their calculations to prevent artifacts caused by loss of significant figures with certain numbers.
that's the general idea. There is an additional tradeoff - error analysis in non-extreme (aka subnormal) cases is easier for IEEE than posits.
The quire stuff has been thrown in there, too, because while we're boiling the ocean, we might as well encourage people to "do the right thing". think of the quire as a really large accumulator cache for things like matrix vector multiplication or tensor outer products, which I hear, AI uses a lot.
How does the implementation of an FPU for this compare? I thought the existing IEEE 754 floating point standard was focused on reduced complexity of addition and multiplication hardware. This seems more complicated.
having implemented it, with unoptimized verilog (ok, i wrote a verilog generator to generate it), it requires about 30% fewer LUTs on a FPGA relative to berkeley hardfloat implementation.
Is the variable length regime handling that much easier to deal with (space-wise) than the NaN and subnormal handling needed in IEEE floats? I'd think that the regime scheme would effectively be equivalent to creating a multitude of different-width subnormal routes. Is it really the NaN handling that kills IEEE float performance?
It's basically a barrel shifter; for addition you're going to need it anyways. Multiplication is a bit nastier, but most of multiplier gates are the adder gates anyways. I made a useful insight that negative numbers are basically the same as positives, with a "minus two" invisible bit.
Here is a sample 8-bit multiplier. All code was generated using a verilog DSL I wrote in Julia for the specific purpose. All verilog is tested by transpiling to c using verilator and mounting the shared object into a Julia runtime with a Julia implementation.
Unrelated as far as I can tell. Posits are an alternative floating point format, not an array compression algorithm. The “compression” benefits from being able to use posit floats instead of ieee754 doubles, because of the better precision.
Not really... there is a standard 32-bit and 64-bit representation of posits. They use the same memory as a float or a double, just with better accuracy and other desirable properties. You only reduce storage space if you switch from “double” to “float”, which you can only do if you didn’t need most of the precision of a double in the first place. They’re not THAT much better.
It occurs to me that maybe you are referring to unums? The "unum" format discussed in "End of Error" is variable sized. However they since dropped that design and switched to fixed-sized number formats (16-, 32-, or 64-bit) with adjustable division between exponent and fraction bits, the "posit."
So "posit" numbers get you the ability to tradeoff precision and range, but they're not anymore highly compressed than regular old IEEE floats. Unless, as Gustafson argues, you didn't need a double in the first place and the added features of posits let you switch to a float.
Zfp is lossy compression just FYI. Fpzip (an earlier invention by Lindstrom et al) has a lossless mode. They really push zfp more though as it has nice features like random access decoding and (except on my data) higher compression ratios.
Not really. I suspect they are referring to the original x87 FP functions with their odd 80 bit FP stack and operations, where the exact result depends on when intermediate results are written to RAM and when they are kept in the FP stack, which might depend on intransparent things like compiler optimisations.
They could refer to parallel systems, most likely GPUs, where it is common to have slightly different results even on the same operation, on the same system.
This is due to the fact that several pipelines will do calculations in parallel and depending on which pipeline ends first (which can depend on many factors like current temperature) the sums can happen in a different order, leading to different rounding results.
You could have this kind of behavior if you use atomic functions - but you need to explicitly opt-in actually using those and it is not limited to GPU.
We have this in our GPU computations, this is very annoying. The errors are usually too small to matter, but they make comparing binary results / reproducible builds impossible. Here is an example I just made:
$ python3
>>> s = [10**-x for x in range(16)]
>>> s
[1, 0.1, 0.01, 0.001, 0.0001, 1e-05, 1e-06, 1e-07, 1e-08, 1e-09, 1e-10, 1e-11, 1e-12, 1e-13, 1e-14, 1e-15]
>>> sum(s)
1.1111111111111112
>>> sum(reversed(s))
1.111111111111111
(it uses CPU for simplicity, but GPU has exactly the same math)
Summation of an array is surprisingly non-trivial. Below a discussion about it with many interesting links (including Kahan summation and pairwise summation) if you want to go down some unexpected rabbit holes.
Somewhere I've seen a little routine (by Stefan Karpinski of Julia fame) that starts with a fixed pre-determined array of numbers, and you give it a desired target number, and it permutes the numbers of the array such that when you sum them up (naively, from left to right) the result is your desired target number [2].
If you don't care about cancellation error and are only concerned with perfect bitwise / reproducible results, then sort your numbers.
Sorting the numbers is necessary to minimize cancellation error. But absolute minimum cancellation error can only be done sequentially: the parallel version may return different results.
l = list(s)
l.sort()
sum(l)
l = list(reversed(s))
l.sort()
sum(l)
Both return 1.111111111111111 now. Sorting is the "synchronization" step you need to get all your parallel tasks back into a defined order.
> They could refer to parallel systems, most likely GPUs, where it is common to have slightly different results even on the same operation.
Parallel systems have great difficulty in defining an order of operations. Floating-point numbers are non-associative, so order matters for bitwise compatibility.
With that being said, you CAN define an order, and it CAN be done in parallel. The parallel-prefix sum is a fully defined order of operations for example, and if you sort all numbers (from smallest to largest), you'll get a more accurate result.
The + operations always happen in the same order, even on a GPU. So you should get the same results every time with floating-point math. But it is non-intuitive for many people to work with non-associative floating-point math.
-----------
Note that writing a sequential algorithm with fully defined order is still grossly difficult! IMO, if anyone wants a bit-accurate simulation, use 64-bit integers instead.
If you absolutely must use floats, then define a proper order of all operations, and include sorting the numbers (!!) to force a fully defined order as much as possible. You may need to sort your pool of numbers many, many times per calculation. But that's what you have to do to get a defined ordering.
it doesn't happen much in practice, but the official IEEE spec is silent about it, so you can toss it into whatever bin contains 'undefined' behaviour of C.
>> where the exact result depends on when intermediate results are written to RAM and when they are kept in the FP stack, which might depend on intransparent things like compiler optimisations.
One thing I've never really been sure about is how context-switching affects this. I've always assumed this could also be a source of non-deterministic floating-point results: when the OS decides to switch threads, it has to save the CPU & FPU state, and restore it when switching back. Since context switches are (practically speaking) non-deterministic on a pre-emptive OS, this would imply truncating 80-bit to 64-bit floating point values could also happen on non-deterministic points in time.
Does anyone know if this is actually true? Or do the full 80-bits get saved/restored on context switches?
1. Yes -- On old x86 systems, the x87 registers were 80-bits and the bottom bits were undefined. You could theoretically have a 1-bit difference on some results depending on what the bottom 26 bits (that were undefined, because you had 64-bit floats most of the time, even though the machine did things 80-bits at a time).
2. No -- Modern x86 systems use SSE registers, which are 64-bits. There are a whole slew of configuration options, but if Windows / Linux does their job, you should have the same rounding-errors across your program (unless you manually change the rounding options, but that's your own fault if you go that route).
3. Kinda yes -- Floating point operations are NON-associative. (A+B)+C does NOT equal A+(B+C). The best example is Python:
253 is the first value which starts to "round off" 1.0 in double-precision. So (1+2.053) == 2.053, because the value was rounded off.
Due to #3, even if you did everything correctly (ie: set the rounding flags so they were consistent), if your arithmetic happened in slightly different orders (very common in network-games, where different players may have their updates in slightly different orders), you end up with a 1-bit error between clients... which completely borks your simulation.
Because #3 is nonintuitive, many people think that you get different results when using floats. But its simply due to non-associativity.
> On old x86 systems, the x87 registers were 80-bits and the bottom bits were undefined. You could theoretically have a 1-bit difference on some results
I spent a long time once debugging an issue arising from this. Code was basically:
double x = y;
[...]
if (x>y) fail;
Nothing in between the assignment and the check changed the value of x or y, yet the check triggered the fail condition. Turned out to be that one of the value stayed in the 80 bit x87 register and one was written to RAM as a 64 bit value, then loaded back into an x87 register for the check, resulting in the inequality. Running in a debugger wrote both values out of the registers before the check, making the problem unreproducible.
Yes, that's a hard bug to catch, but if the mantra of "every floating point comparison must use a context-appropriate epsilon" is followed, then it wouldn't have happened in the first place.
Absolutely, and the fix involved adding a tolerance check. Not my code, I was the new guy at the time, and my job was trying to fix the bugs nobody else wanted to deal with...
I hate that mantra, as it produced more bugs and inaccuracies in the code I have to work with than necessary. Actually you should be able to rely on the equality of a value with its copy no matter what. We are past the FPU days, so I think we can let go of these mindless epsilons sprinkled over the code.
(Even during the FPU days, every ; should have rounded each value to the size of the variables, unless you use a special compiler switch to make FP math faster)
Are your sure of this? It's my understanding that SSE registers require the use of a special API and standard floating point operations do not use them.
But the last time I worked with them was writing a SIMD vector library 7 years ago.
-mfpmath=unit
Generate floating-point arithmetic for selected unit unit. The choices for unit are:
‘387’
Use the standard 387 floating-point coprocessor present on the majority of chips and emulated otherwise. Code compiled with this option runs almost everywhere. The temporary results are computed in 80-bit precision instead of the precision specified by the type, resulting in slightly different results compared to most of other chips. See -ffloat-store for more detailed description.
This is the default choice for non-Darwin x86-32 targets.
> It's my understanding that SSE registers require the use of a special API and standard floating point operations do not use them.
No, SSE registers are used by most compilers that do floating point these days. They support all the usual IEEE float math operations and a host of bit twiddling operations as well as vector operations. The vector operations do still require using compiler intrinsics in C++, although some autovectorization does occur in gcc, icc, and llvm.
Wrt #3, I've had huge errors (I think on the order of 10e-3) in non linear curve fitting spectroscopy algorithms because of this. One of the physics research fellows in the group looked at me like I was an idiot for not knowing the order of multiplication mattered (I still have no clue why he would think it's a standard thing to know this).
Lots of researchers cargo cult floating point programming. I had a Fortran program a few months ago where the author did "if var > 0.99 and var < 1.01 then" etc., where "var" was an integer. I tried to search back where and how that was ever done, but no scenario made any sense ("var" was a categorical variable and always had been). So I went back to some of the original authors and there too they looked at me like I was an idiot and said "you should never test for equality, always check for distance within a certain epsilon". So I asked about the difference between integers and floating point and then they "didn't have time for technical details in code written 10 years ago." Shrug, they pay me to fix that particular sort of weirdness I guess.
In all likelyhood, the reason that's there isn't because a programmer was an idiot. Almost certainly, that variable was originally a float, then some time later another programmer came in and refactored it into an int, but missed fixing this condition, since it technically "works" for ints as well.
This is a good argument for stricter typechecking than anything else. In Rust, a condition like that wouldn't compile because of the mixed types. That's probably a good call in my opinion: it alerts the programmer that there's something funky going on, and it forces the programmer to be explicit about how the comparison should actually happen.
I do find myself doing the "cargo cult" thing from time to time. I was working on a codebase where Lua was embedded in C++, and there were all these flags that had to work in both systems. Because Lua only supports 64-bit floats as a numeric type, the natural type for these were doubles, and "by convention" they only ever had the values 0.0 or 1.0. It was a bit silly of me, but everytime i saw the line `if (fSomeFlag == 1.0f) ...` in the code, I would wince. It just feels wrong to do that to floats. I found myself regularly writing `if (fSomeFlag > 0.5f) ...` instead.
Of course, I was wrong about this. These weren't values determined as a result of arithmetic, they were always set to whole numbers. There was never going to be a case where the actual value of the flag would be 0.9999998f or whatever. But still: once you've been burned by the "floating point equality comparison", you live in fear of it forever after.
It was the same in the situation I was talking about; all numbers that would always be integers because of their nature (and in the data source they came from, they were integers, there was no way this could ever have been floating point, or that the values ever had any non-0 digits after the comma). And well, they probably weren't 'idiots' in the absolute sense, in fact I'm 100% sure that they are much smarter (and/or just plain better humans in other ways) than I am in many respects.
My point was that many researchers can only cobble together somewhat barely working software by taking snippets from their undergraduate textbooks (or googling/stack overflowing, for those under 40), and have no idea about many/most of the underlying principles. If only they would recognize that, and leave the software development to professionals, instead of treating it as an inconsequential implementation detail that is beneath them. Then again, if they did, they wouldn't have to pay me what they do to fix it up afterwards. So meh?
> . No -- Modern x86 systems use SSE registers, which are 64-bits. There are a whole slew of configuration options, but if Windows / Linux does their job, you should have the same rounding-errors across your program
Knock, knock. Who is there? Floating-point contraction.
Python will happily do this too, if you remove the ".0" in the example, the values become integers, instead of floats, and the computation is precise.
The point is to demonstrate a property of IEEE floats, not the language. I'd expect CL has some means of using IEEE floats too, and with the appropriate syntax, the same issues could be demonstrated.
This may be good for numerical simulation, but I doubt this will ever replace common ieee754 float in general purpose programs. Two problems I can see right away:
- Sometimes, one needs precision for numbers not close to 1. A UNIX timestamp is a good example -- for the current date, it provides ~1uS resolution in 64-bit ieee754. I could not calculate what the resolution would be for posits, but I suspect much worse.
- The lack of positive and negative infinites will break naive min/max calculations which start accumulator with +/-inf.
Now, you might say that those things should not be done, and that software that uses those patters is defective. But even then, there is a lot of software like this, and so posits will never become a default "float" type.
UNIX timestamps are normally stored as 32-bit or 64-bit signed integers, not as floating-point. If you want better than 1-second precision, then the type "struct timespec" (specified by POSIX) gives you nanosecond precision. Fixed-point types can also be used in languages that support them.
UNIX time is seconds since epoch (hence year 2038, that's the limit for a signed 32b time_t).
gettimeofday() and clock_gettime() provide higher resolution timestamps (respectively µs and ns), using typedefs instead of just numbers.
Some APIs return floating-point UNIX time in order to provide sub-second accuracy (the decimal part is the fractional second). Python's time.time() does that for instance.
Yeah, timespec (or time_interval) is a proper way to go, but it is quite a pain to work with -- you need a library even if you just want to subtract the numbers.
On the other hand, floating-point time is pretty common in scripting languages -- for example Python has time.time(); ruby has Time.now.to_f. It is not perfect, but great for smaller scripts: fool proof (except for the precision loss), roundtrips via any serialization format, and easy to understand. And no timezone problems at all!
I read "drop-in" as dropped in to new code. Certainly this could never be dropped in transparently at an OS or processor level. As long as the change is opt-in, neither of the things you mention are problems. I think a new world where programmers don't have to be aware of negative zeros, etc, would be appealing to a lot of people.
I think this person is saying that it should break existing code not the pattern.
The solution seems simple to me though. Whatever syntax programmers are using to initialize floats to +/-inf, make that ensure to posit min and max as well.
The proposed format has changed several times since 2015, so not all those threads are discussing the same thing as this one.
Posits are Type III Unums IIRC, the original proposal was with Type I Unums.
Previous proposals were variable length formats, which is pretty much a nonstarter for a variety of reasons. This proposal (posits) is a fixed length format with variable length fields.
The bottom line is they are a tradeoff; they certainly aren't purely better than IEEE floats. Plus they seem to have a non-trivial hardware cost (one of the linked papers shows the adder is twice as large as the equivalent IEEE adder and has double the latency!)
Hm, perhaps the answer is a subtle definition of "rounding error".
A rounding error is when you have a number, lets say 0.75, and due to rounding it is recorded as 1.00. The "rounding error" is 0.25.
An alternative to rounding to 1.00 would be to have a mechanism which says "the value is between 0.50 and 1.50". This way, there is no actual rounding, as it doesn't commit to a rounded value, so there is technically no "rounding error".
A neat advantage of recording an interval rather than rounding is that the "error" is preserved in the data through arithmetic, so if there is some following code that runs x * 100, a rounding mechanism would say "the value is 100" whereas an interval mechanism would say "the value is between 50 and 150". Then, if the user only looks at the output, it will be clear that there is a wide error range and something needs to be fixed, rather than the output indicating a precise answer when really it suffers from significant rounding errors.
Yes, this is the question I come to. The article betrays its own claim...
> Gustafson said that for training, 32-bit floating point is overkill and in some cases doesn’t even perform as well as the smaller 16-bit posits
If posits have a bit number, then they are not of variable accuracy. This is simply sloppy explaining.
At some point, rounding has to happen (call it an interval if you want, but you're not being helpful). As a programmer, I WANT rounding to happen. I depend on it. You don't break out the floats unless you are prepared for some information to be lost. If you try to never lose any information, then you wind up with a monstrosity like Maple and Mathematica in which all steps need to be curated by hand to occasionally reduce the giant glob of fractions and implicit solves into something concrete, but imperfect.
One form of rounding error that comes up in floating point conversations has to do with data aggregation. If the average value of a series is 3.5 but after rounding the average drops to 3.4, then your rounding has lost some fidelity, and this upsets some people. This counts as a rounding error to them.
That claim can only be made with respect to using a Kulisch accumulator (what he calls the quire) for accumulation.
This can represent the exact sum of any number of floating point values (or products of floating point values) with the intermediate operations preserved at infinite precision, up to a single rounding at the end.
e.g., for computing an inner product of vectors \sum_i x_i y_i with x_i and y_i in floating point, a fused multiply-add operation (as on today's computers) would perform something like:
where what is performed within the rounding function r() is done to infinite precision and a single rounding at the end. This is also only true if you are accumulating into the same value, and not splitting this operation across multiple values, otherwise there would be additional rounding steps performed.
Using a Kulisch accumulator the result would be:
r(\sum_i x_i y_i)
similarly also only preserving precision if everything is done in a Kulisch accumulator.
You can add a Kulisch accumulator to IEEE floating point or any other floating point as well. In fact the second? ever computer with floating point (the Zuse Z3) performed FP addition using a similar accumulator, but sums-of-products probably didn't preserve the multiplied value though.
Posits and other floating variants are seriously cool, and Gustafson work is amazing.
Sadly, the guy has a very annoying writing style that makes him sound like a crackpot. The advantages of posits would shine much more if they were not mixed with ridiculous language (posits are floating point numbers, thus they cannot replace them) and outlandish claims (IEEE floating point is deterministic, to the apparent contradiction of many sentences written by Gustafson). It does not help either that Gustafson work only seems to attire the attention of semi-illiterate journalists who do not really understand what they are talking about.
>IEEE floating point is deterministic, to the apparent contradiction of many sentences written by Gustafson
What about this then, which somebody comments below:
"On old x86 systems, the x87 registers were 80-bits and the bottom bits were undefined. You could theoretically have a 1-bit difference on some results depending on what the bottom 26 bits (that were undefined, because you had 64-bit floats most of the time, even though the machine did things 80-bits at a time)."
And below:
"I spent a long time once debugging an issue arising from this. Code was basically:
double x = y;
[...]
if (x>y) fail;
Nothing in between the assignment and the check changed the value of x or y, yet the check triggered the fail condition. Turned out to be that one of the value stayed in the 80 bit x87 register and one was written to RAM as a 64 bit value, then loaded back into an x87 register for the check, resulting in the inequality. Running in a debugger wrote both values out of the registers before the check, making the problem unreproducible."
This behavior might not be IEEE defined, but even as IEEE undefined (and implementation dependent), it still qualifies those IEEE compliant implementations as non-deterministic.
Or how about:
"parallel systems, most likely GPUs, where it is common to have slightly different results even on the same operation, on the same system. This is due to the fact that several pipelines will do calculations in parallel and depending on which pipeline ends first (which can depend on many factors like current temperature) the sums can happen in a different order, leading to different rounding results."
Those are not problems a new number format can fix though. The issues with x87 precision are due to compilers freely converting between different representations without this being indicated in the program.
Posits also can't achieve associativity in all conditions, you need to use "quire" accumulators with higher precision. However, if the decision of using quires were left to the compiler, you'd end up with the exact same problem of unclear precision as for x87. To avoid that, those accumulators will need to be annotated in the code, and straightforward programs that do not use them will continue to exhibit non-associativity.
The observation that all of the +-*/ operations in a basic block can be reduced to a single round with the quire would get you a reasonable way with the compiler alone.
Overall I'm not sold on the quire thing. It seems like a great big accumulator is a solution that is fairly independent from the floating point format.
> >IEEE floating point is deterministic, to the apparent contradiction of many sentences written by Gustafson
> "parallel systems, most likely GPUs, where it is common to have slightly different results even on the same operation, on the same system. This is due to the fact that several pipelines will do calculations in parallel and depending on which pipeline ends first (which can depend on many factors like current temperature) the sums can happen in a different order, leading to different rounding results."
> due to fp associativity?
As far as I understand, that's not quite the case (on Fermi+ Nvidia, as previous generations had no real support for proper IEEE single compute).
There are pipelines in parallel, but almost all non-determinism in the order of execution (resulting in fp associativity) is exposed to the programmer, except for atomic instructions. Those can even depend on clock skew across PCIe or NVLink, if they operate on remote memory.
NVLink supports remote execution in a recent version (I suggest the microbenchmarking-derived microarchitecture analysis papers for Volta and Turing, even if they are less hands-on the Nervana's maxas (with the SGEMM walk-through, which I really recommend)), and thus any hope of reliably tricking synchronization behavior via undefined opcodes/scheduler instructions trickery is deliberately out of the window.
On IBM POWER9 CPUs they can apparently delegate execution of atomic operations to the host CPU, which might (I don't know enough about POWER9 cache coherency /NUMA protocols) delegate them over the NUMA fabric to a different rack-mount unit bonded into the same NUMA mesh.
TL;DR:
Recent Nvidia GPUs do not suffer implicit non-determinism in IEEE floats unless you use hardware atomics.
Non-determinism is exposed and introduced in explicit concurrent data structures [0].
[0]: If you count a datastructure with a simple spinlock guarding it as a concurrent datastructure.
PS: I'll add references / links later on the laptop.
”This behavior might not be IEEE defined, but even as IEEE undefined (and implementation dependent), it still qualifies those IEEE compliant implementations as non-deterministic.”
Although IEEE is non-deterministic for other reasons (the transcendental example below is a good illustration) this argument is incorrect.
Performing operations on a non-IEEE 80-bit representation and then using a non-IEEE comparison is not a valid argument about non-determinism in IEEE.
If you have an 80 bit object, convert it to a 64 bit object, then convert it back to an 80 bit object, and you expect to drop zero bits, of course you'll run into problems. Posits can't fix these problems.
Desktop graphics cards intentionally ignore IEEE 754 rules if it allows them to squeeze out a little bit of performance. (or die space, or power consumption depending on the application) If the bad guy is drawn half a pixel out of position or if his shirt is slightly the wrong shade of red, nobody cares. But if your graphics card draws at 72 frames per second and your competitor's cards draws at 73, every publication is going to say their card is better than yours. Posits won't fix this either.
These problems you point out aren't technical problems, they're institutional ones. IMHO it's generally a bad idea to try to fix institutional problems with technical solutions, but I went try to stop you from attempting it.
> IEEE floating point is deterministic, to the apparent contradiction of many sentences written by Gustafson
How can that be? The IEEE standard does not place many requirements on the precision of the transcendental functions, and AFAICT a transcendental doesn't even need to return the exact same answer twice. That is, it is correct for a compiler to, if it can, evaluate a transcendental at compile-time with infinite precision, and if it cannot, call a run-time function that computes it. Math libraries can be dynamically linked, and all can produce different results here, so it is not that a compiler can go out of its way to make things deterministic either.
I am implementing a high performance posit arithmetic unit for uni thesis. The idea of using Golomb-Rice prefix tree to encode the exponent is genius. Having said that.
I totally agree his papers make a lot of bogus claim.
For instance, the claim that "posits can be a direct replacement of floating point" is not true. There are a few tricks, such the Fast Invertion Square Root that abuses the floating point format to do fast arithmetic operation. This would break replacing with posit.
Also he claims that early FPGA posit implementation uses less area and is has high performance than a Floating Point implementation. This cannot be true^. The posit unit IS A floating point unit with a encoder/decoder unit. The posit floating point unit will actually have a larger word size when decoded than ieee floating point because it has more precision than ieee format. Also the features such different rounding
supported by ieee but not by posit uses a negligible extra area.
I'm eager to read your thesis, then! Please, make sure to include a (short) historical account from a neutral point of view. Most notably, Kahan's rejection of unums/posits mixes mostly reasonable criticism with a few points that are not relevant; and then Gustafson holds on to these irrelevant points and replies with his usual word salad. It is quite a sad state of affairs for what could be an enlightening scientific discussion.
The Golomb-Rice prefix tree for the exponents is certainly cool. It has the air of Gosper's "continued logarithms", which are like the same idea but turned to the max: use always the same mantissa (which is implicit) and represent only the exponent to a high precision.
These architectures are evaluated on recent FPGA hardware and compared to their IEEE-754 counterpart. The standard 32 bits posit adder is found to be twice as large as the corresponding floating-point adder. Posit multiplication requires about 7 times more LUTs and a few more DSPs for a latency which is 2x worst than the IEEE-754 32 bit multiplier.
There are a few FPGA implementations.
The posit hub website provides a table with all of them. Each has different trade-offs. The implementation I working on is focused on hardware reuse and high frequency operation.
I will get in touch once I finish the section on Posit. I would appreciate some feedback! I'm writing in my language, but hopefully the google translator will work well.
Gustation is being too ambitious trying to reinvent the whole floating point framework. I think he should focus in what really matters: the compact encoding. A lot of application could save memory bandwidth using some kind of posit inspired format. A pack/unpack unit that converted the posit encoding to floating point inside the core would could be game changing.
He also proposed a good idea on code ergonomic: Make the accumulator a type (quire in the posit). Making the programmer aware of the accumulator can lead to better to code.
By the way, reading the Kahan work in floating point I could understand why Gustafson want's to avoid the IEEE. The Kahan implementation favored Intel, not necessarily the community.
Anyway. My point of views are from the hardware side of the things. The format seems to have some arithmetic properties that might be useful for mathematicians.
> posits are floating point numbers, thus they cannot replace them
Everyone who reads that who is cognitively typical will understand "floating-point" (which, by the way, needs a dash, if we are going to nitpick) as a nickname for "IEEE 754 floating-point" or "traditional floating-point", rather than an insinuation that posits are, in contrast, fixed-point or something other than floating-point.
Do these techniques make it easier or harder to implement hardware floating point units?
Storage and transmission speed have progressed exponentially while execution units have become the bottleneck. A floating point format that is 20% denser or more accurate but that requires a 2x number of gate delays to implement is major step backwards, except maybe in highly specialized applications.
From [1]: " The standard 32 bits posit adder is found to be twice as large as the corresponding floating-point adder. Posit multiplication requires about 7 times more LUTs and a few more DSPs for a latency which is 2x worst than the IEEE-754 32 bit multiplier." It's for an FPGA implementation.
This being said, I object to your premise: transmission and memory storage are getting comparatively more costly vs computation. From "Computer Architecture, A Quantitative Approach", 6th edition in figure 1.13 there are some values for the TSMC 45nm process, so hardly the leading edge and it only gets worse with finer nodes:
- 32 bits integer multiplication: 3.2 pJ
- 32 bits float multiplication: 3.7 pJ
- 32 bits read in a small 8 kB SRAM: 5 pJ
- external DRAM 32 bits read: 640 pJ
We have a lot of transistor nowadays, and as long as the computation can be pipelined halving the memory accesses could be a win (TBC).
Also on from the same Inria team as [1], and also covering hardware cost but more than this: "Posits: the good, the bad and the ugly":
https://hal.inria.fr/hal-01959581v3/document
I'm not sure the inria team used the correct optimization for posit addition. You do yourself a disservice by treating a posit like a float and bifurcate the cross (negative/positive) from the negative/negative and positive/positive branches; since posits are twos complements the posit adder should be smaller, not bigger.
Memory bandwidth relative to instruction to instruction latency has gone way down. Memory is slower now relative to the amount of math one can do.
We are also swimming in extra silicon, so even if what you are saying is true (2x the number of gates, all adding to the delay), posits would still be a win.
You are arguing from an arbitrary what-if position, it doesn't look good.
Choosing to have -infinity and various nans can be done independently from choosing to add regime bits, one could as well design this system to include them by adding more special values.
I didn't find a resource specifying how many exponent bits to actually use? (The 'es' value)
I don't see anything different in this system than regular floats that would ensure consistent results across machines/compilers/...? It's floats plus unary encoded regime bits to have var length exponent, so everything that can make floats inconsistent can still happen here: non commutativity, different exponent sizes or precisions of intermediate values, different rounding modes, ...
Thanks, came to say the same thing - wondered what William "Father of IEEE 754" Kahan had to say about this.
I haven't had time to digest it all, but he is certainly critical of Gustafson's proposals. Not sure though the articles linked above cover the latest and greatest Unum III. At any rate, I'd pay close attention to Kahan's critique.
This sounds really good. I find floating points completely unusable for any situation where accuracy is important. It's fine when being vaguely in the right ballpark is good enough, but I don't want to have to deal with 2 + 4.1 = 6.1000000001, or x/1000000 + y/1000000 != (x+y)/1000000.
Well, those will not be solved by posits, it is still an approximate floating point format. What it does is redistribution the precision to what Gustafson considers a better default and dropping edge cases in order to get more bits for precision.
edit: One of several examples found in the "Posits: the good, the bad and the ugly" paper linked in the thread : 10.0 * 2.0 = 16.0 in posit8
Really? My impression from the article was that this was exactly one of the things posits were supposed to fix, at least for numbers with small exponents.
I'm not sure how 10.0 * 2.0 = 16.0. I'm not sure what posit8 means, but it can only be correct if it switches halfway from base 8 representation to base 10, which is a bit weird, but at least the calculation is correct. (Otherwise it would be so incorrect to be unusable for anything.)
I am serious (and recommend reading the paper I quoted to have a good understanding of the trade-of offered by posits).
Overall, posits are a new trade-of that will give you better precision (nothing exact, it is still an approximation) when you manage to keep all your number in a small range. Once you get out of that range precision drops significantly (whereas the precision of classical floating points drops gradually).
Posit8 are equivalent to 8 bits floating points (minifloats) making them an easy target for pathological cases but the example still illustrate the fact that, contrary to floating-point arithmetic, multiplication by a multiple of two is not exact with posits (one of several good properties we take for granted and would lose when switching to posits).
The problem you describe here arises from using binary fractions, that is, a power of 2 as a denominator. You cannot represent the decimal fraction 0.1 as a binary fraction. You would have the same problem representing 1/3 with decimal fractions. It just does not work.
You can solve it by switching to decimal floating points, they are defined by IEEE as well:
This has always been interesting to me. It's been awhile since I read about it last, though, and I remember reading some criticisms or concerns that posits (or unums? what's the difference?) would end up being slower for some reason. I don't remember the arguments though, or where I saw them; I think the idea was that there were some edge cases that were common enough in practice that overall it would slow things down. It would be nice to see a balanced discussion of the ideas (pros and cons).
> Better yet, he claims the new format is a “drop-in replacement” for standard floats, with no changes needed to an application’s source code.
Claims like that are best taken with a grain of salt:
> It also does away with rounding errors, overflow and underflow exceptions, subnormal (denormalized) numbers, and the plethora of not-a-number (NaN) values. Additionally, posits avoids the weirdness of 0 and -0 as two distinct values.
Ok, so posits will probably work fine as a drop-in replacement when my application makes simple use of floats. But, assuming my application is doing non-trivial math, it's probably aware of the above edge cases. Thus, dropping in posits might have lots of weird side effects where I had to work around weird side effects of floats.
I think we can assume that the people who are sophisticated enough to “use floats correctly” are also sophisticated enough to know the article is not aimed at them.
Your boss, on the other hand, may try to armchair architect you, which is a real (and underreported) problem,
> But, assuming my application is doing non-trivial math, it's probably aware of the above edge cases. Thus, dropping in posits might have lots of weird side effects where I had to work around weird side effects of floats.
Yeah, but posits appear to have fewer weird edge-cases. Plus the people trying out posits are constantly trying to find methods to work around the limitations too.
During this years conference on posit maths Florent de Dinechin had a really nice talk bringing up all current issues with posits and ways that floating point maths had found workarounds over the year, as a kind of challenge to make posits catch up[0][1]. The community took it really well, as far as I can see, and Gustafson in particular seemed delighted because he genuinely wants everyone to start using better numerical methods.
Very few developers truly understand floating point representation. Most think of it as base-10, and put in horrific kludges and workarounds when they discover it doesn't work as they (wrongly) expected. I shudder to think how many e-commerce sites use `float` for financial transactions!
So as far as I'm concerned, whatever performance cost these alternate methods may have, it would be well worth it to avoid the pitfalls of IEEE floats. Intel chips have had BCD support in machine code; I'm surprised nobody has made a decent fixed point lib that is widely used already.
> Most think of it as base-10 [...] I'm surprised nobody has made a decent fixed point lib that is widely used already.
Note that fixed radix point does not solve the common issues with representing rational base 10 fractions. A base10 fixed radix solution would, but so would IEEE754's decimal64 spec, which would eliminate representation error when working exclusively in the context of base10 e.g finance, but these are not found in common hardware and do not help reduce propagation of error due to compounding with limited precision in any base.
Considering that back in the 1980's we figured out that you should use some kind of 'integer' or 'bcd' based type for financial calculations; it is astonishing that by almost 2020 people are making these same mistakes over and over again.
A 64-bit integer is big enough to express the US National Debt in Argentine Pesos.
> it is astonishing that by almost 2020 people are making these same mistakes over and over again.
It's actually logical: the number of developers doubles roughly every 5 years. It means that half of the developers have less than 5 years of experience. If they don't teach you this in school (university), you will have to learn from someone who knows, but chances are the other developers are as clueless as you.
It's not enough to represent one US cent in original Zimbabwean dollars though as of 2009. Although those 'first' dollars hadn't been legal for quite some time, the currency having gone through three rounds of massive devaluation and collapse, totalling something like 1x10^30 by then.
Replacing all the IEEE 754 hardware with posits won't fix this, though.
If you don't care about performance, then the actual solution has no dependency on hardware:
1. Replace the default format for numbers with a decimal point in suitably high level languages with a infinite precision format.
2. Teach people using other languages about floating point and how they may want to use integers instead.
The end. No multi-generation hardware transition required.
IMO, IEEE 754 is an exceptionally good format. It has real problems, but they aren't widely known to people unfamiliar with floats (e.g. 1.0 + 2.0 != 3.0 isn't one of them).
> 1. Replace the default format for numbers with a decimal point in suitably high level languages with a infinite precision format.
Unlimited precision in any radix point based format does not solve representation error. If you don't understand why:
- How many decimal places does it take to represent 1/3 (infinite, AKA out of memory)
- Now how many places does it take to represent 1/3 in base3? (1)
If you are truly only working with rational numbers and only using the four basic arithmetic operations, then only a variable precision fractional representation (i.e a numerator and demonstrator, which is indifferent to underlying base) will be able to store any number without error (if it fits in memory). Of course if you are using transcendental functions or want to use irrational numbers e.g PI then by definition there is no numerical solution to avoid error in any finite system.
Great points. I agree - the performance hit is simply the cost of being accurate and having predictable behaviour.
I'm not suggesting we replace all our current HW with chips that implement posits (let's fix branch prediction first!!). More that FP should be opt-in for most HLLs.
The use of binary in the numerator is not the problem with using floats for financial math, it is the use of powers of 2 in the denominator.
The numerator is just an integer and integers are just integers and the base doesn't matter. But if the exponent is base 2, then you can have 1/2, 1/4, 1/8 on the base but not 1/5 or 1/10.
In the paper "Do Developers Understand IEEE Floating Point" the authors surveys phd student and faculty members from computer science and found out that your observation is true: most people don't know how fp works.
They have the survey online at [1] in case you want to see how much you know about fp behavior.
That was a waste of time: 6 pages of questions, and instead of "grading" it and letting me know where I stand on FP, it says "Thanks for the gift of your time".
Interestingly, they got one of their own answers wrong.
The question is:
if a and b are numbers, it is always the case that (a + b) == (b + a)
and their notes on it:
Is a simple statement involving the commutativity over addition true for floating point? Generally, floating point arithmetic follows the same commutativity laws as real number arithmetic.
They make it clear in their notes that "are numbers" includes infinities but not NaNs. Now consider the case where a = inf and b = -inf. Then inf + (-inf) is NaN,
and (-inf) + inf is NaN, and NaN != NaN.
>>> a = float('inf')
>>> b = float('-inf')
>>> a + b == b + a
False
Rational numbers are a good way to do many financial calculations (extra points for doing something useful with negative denominators!), since many financial calculations are specified with particular denominators (per cent, per mille, basis points; halves, sixths, twelfths, twenty-sixths, fifty-seconds of a basis point, etc.).
However, as soon as you start doing anything interesting, you have limited precision as a matter of course.
If there's one thing I've always really appreciated about Groovy, it's that it used BigDecimal as the default for fractions, because 9 times out of 10, you need accuracy more than you need high performance and large exponents (and if you do need high performance, you wouldn't be using Groovy anyway).
Sadly most languages don't support something like that out of the box.
Sorry, but you actually sounds like one those people who dosen't really know how FP work.
> I shudder to think how many e-commerce sites use `float` for financial transactions!
The float IEEE-754 represent up to 9 decimal digits (or 23 binary digits) with precision. The double, represent 17 decimal digits. The error upper bound is (0.00000000000000001)/2 per operation. Likely irrelevant for most e-commerce.
Also, the database stores in currency values using fixed point.
> Intel chips have had BCD support in machine code
BCD is floating point encoding not fixed point. AFAIK, only Intel supports it and very precariously.
> I'm surprised nobody has made a decent fixed point lib that is widely used already.
Nonsense. If you do any scientific computation you have likely have Boost, GMP, MPFR installed in your system. They support arbitrary precision arithmetic with integer (aka fixed point), rational and floating point.
> Sorry, but you actually sounds like one those people who dosen't really know how FP work.
LOL, sure ok. Worked on banking systems for 2 years and been doing scientific computing for many more. Pretty comfortable with fixed and floating point.
> [error bounds] Likely irrelevant for most e-commerce.
Those bounds are theoretical, and there are plenty of occasions I have come across in the past when rounding errors were observed. It was forbidden in the bank to use floating point! We went to enormous lengths to ensure numerical accuracy and stability across systems.
I think this article has a pretty good explanation:
> Nonsense. If you do any scientific computation you have likely have Boost, GMP, MPFR installed in your system. They support arbitrary precision arithmetic with integer (aka fixed point), rational and floating point.
Yes, absolutely right; I have used several of those 3rd party libs myself, as well as hand-rolling fixed point code (esp for embedded systems). I didn't write what I intended. I meant to say that very few languages have first-class fixed point in their standard library. So long as the simple `float` is available as a POD, people will (mis-)use it.
I think in a general purpose HLL, a fixed decimal type should be the default, and you should have to opt in to IEEE-754 floating point.
First, I would like to apologize for my attitude on the original post, my choice of words was very inappropriate for this forum. Second. I would like to thank you for bringing another point of view.
I'm talking about the average joe e-commerce.
The precision requirement of Financial institution and large e-commerce are more strict than most website.
For the average e-commerce the extra cost of using a decimal arithmetic framework might not be reasonable. Of course, software engineers should use currency types when they are available.
The example used by the website is misleading. They print a value with all the significant digits. In reality, e-commerces are only concerned with two digits after the decimal point. I run the same program in C with one billion (-1) iterations and the printed value with two decimal digits was exact. It was the the as it would be if I used decimal arithmetic.
> I think in a general purpose HLL, a fixed decimal type should be the default, and you should have to opt in to IEEE-754 floating point.
Most modern languages have some support to rational, fixed point and currency arithmetic. Nowadays, even GCC's C supports Fixed Point arithmetic [1].
The hardware implementation of floating point arithmetic is concerned about efficiency hardware and support for a lot of uses cases. The ranges and accuracy needed in different application varies widely. The BCD encoding is very inefficient hardware-wise compared to binary encoding.
In summary, I do agree people with working currencies should use the proper currency type, but I also think using IEEE 754 is usually fine, specially in the front-end. Also, I don't think the trade offs of changing from binary arithmetic to decimal arithmetic are not worth it for most people and hardware system.
> It also does away with rounding errors, overflow and underflow exceptions, subnormal (denormalized) numbers, and the plethora of not-a-number (NaN) values.
All of those are important and useful features. Presenting their
absence as some kind of advantage shows that Gustafson has no clue.
200 comments
[ 2.7 ms ] story [ 259 ms ] thread[0] https://dl.acm.org/citation.cfm?id=3148220
Mostly because there is no good evidence that what is being proposed is better.
This isn't the olden days when it was difficult to demonstrate on a large enough CPU and dataset. Today we have cloud computing. If you create something better, you can demonstrate it by putting it into a numerics application and blow everybody away.
The CFD people are always looking for better solutions. The numerical simulation people are always constrained.
Until you do that, people have a right to blow you off.
I'd be much more interested to see how these handle stiff systems with multiple time constants. That's a domain where everything has problems and improvement is going to actually move some things from completely infeasible to actually simulatable.
That's a much stronger use case. And you don't have to worry about implementation efficiency since the improvement ratio is "infinity"--you went from can't do it at all to actually being able to do it.
Err, it's math, there doesn't need to be "evidence". They can do the calculations for themselves and see...
If you want me to believe something, the jobs is yours to cough up the convincing evidence.
[1] https://posithub.org/index
[2] https://github.com/mthom/scryer-prolog/issues/6
[3] https://github.com/mthom/scryer-prolog
[4] https://gitlab.com/burrbull/softposit-rs
[5] https://juliacomputing.com/blog/2016/03/29/unums.html
A 'softposit' implementation can at least let people experiment with writing posit algorithms to investigate the claimed algorithm-design benefits, even if it's not going to beat hardware floats in speed.
[1] The interval arithmetic people don't seem very happy with his comparisons though: http://frederic.goualard.net/publications/MR3329180.pdf
Also, IEEE 754 floating point standard guarantees the results of addition, subtraction, multiplication, division, and square root to be the exact correctly rounded value, ie a deterministic result, contrary to what he says.
> Reproducibility
> The IEEE 754-1985 allowed many variations in implementations (such as the encoding of some values and the detection of certain exceptions). IEEE 754-2008 has strengthened up many of these, but a few variations still remain (especially for binary formats). The reproducibility clause recommends that language standards should provide a means to write reproducible programs (i.e., programs that will produce the same result in all implementations of a language), and describes what needs to be done to achieve reproducible results.
https://en.wikipedia.org/wiki/IEEE_754#Reproducibility
Most developers prefer speed over reproducibility, and are encouraged to use denormal-to-zeros, fast-math optimizations, fused MAC, approximated square root function and whatever else is available to achieve results.
The IEEE754 standard provides a guarantee for deterministic results, and many multi-precision and interval arithmetic libraries depend on this guarantee to be true to function properly.
IEEE754 defines unique -infinity and +infinity values, and any "new and improved" standard that breaks this axiom is just incompatible with all existing floating-point libraries written in the last +30 years.
"These claims pander to Ignorance and Wishful Thinking." Kahan (main author of IEEE754) on Posits claims.
You're free to voice your own opinion, but I take some issue with people asserting theirs as if they speak for "most developers". Especially if it comes from a new account with a name like "Gustafnot". That doesn't exactly scream "unbiased" to me.
If you really want to claim the opposite, do you have evidence? Or experience that it’s true?
Yet electron is not only a thing for hobbyists, but something even large companies bet their livelihood on.
The vast majority of application dosen't need fine control of the FPU. There is always will be hardware for the few application need the ieee-754 features.
It sounds like the basic idea here is that rather than a fixed amount of significant figures, you get more precision in the middle. Meanwhile, you can also get larger exponents. Is that right?
It's a bit sad that many of the creator’s claims are exaggerated and his examples cherry-picked though.
Presumably that's what you mean by scale free, but let's spell it out:
It's denser around 0, then has uniform density between, say, 1/8 and 1/4, half that density between 1/4 and 1/2, half that between 1/2 and 1, half that between 1 and 2, etc. etc.
If x = 1.0e22, you can add a million and it's still the same.
The quire stuff has been thrown in there, too, because while we're boiling the ocean, we might as well encourage people to "do the right thing". think of the quire as a really large accumulator cache for things like matrix vector multiplication or tensor outer products, which I hear, AI uses a lot.
Here is a sample 8-bit multiplier. All code was generated using a verilog DSL I wrote in Julia for the specific purpose. All verilog is tested by transpiling to c using verilator and mounting the shared object into a Julia runtime with a Julia implementation.
https://github.com/interplanetary-robot/mullinengine/blob/ma...
[1] https://github.com/LLNL/zfp
[2] https://helper.ipam.ucla.edu/publications/bdcws2/bdcws2_1504...
So "posit" numbers get you the ability to tradeoff precision and range, but they're not anymore highly compressed than regular old IEEE floats. Unless, as Gustafson argues, you didn't need a double in the first place and the added features of posits let you switch to a float.
wait...what? Is this real?
This is due to the fact that several pipelines will do calculations in parallel and depending on which pipeline ends first (which can depend on many factors like current temperature) the sums can happen in a different order, leading to different rounding results.
Somewhere I've seen a little routine (by Stefan Karpinski of Julia fame) that starts with a fixed pre-determined array of numbers, and you give it a desired target number, and it permutes the numbers of the array such that when you sum them up (naively, from left to right) the result is your desired target number [2].
https://discourse.julialang.org/t/accurate-summation-algorit...
EDIT: [2] https://discourse.julialang.org/t/array-ordering-and-naive-s...
Sorting the numbers is necessary to minimize cancellation error. But absolute minimum cancellation error can only be done sequentially: the parallel version may return different results.
Both return 1.111111111111111 now. Sorting is the "synchronization" step you need to get all your parallel tasks back into a defined order.Parallel systems have great difficulty in defining an order of operations. Floating-point numbers are non-associative, so order matters for bitwise compatibility.
With that being said, you CAN define an order, and it CAN be done in parallel. The parallel-prefix sum is a fully defined order of operations for example, and if you sort all numbers (from smallest to largest), you'll get a more accurate result.
https://en.wikipedia.org/wiki/Prefix_sum#Algorithm_1:_Shorte...
The + operations always happen in the same order, even on a GPU. So you should get the same results every time with floating-point math. But it is non-intuitive for many people to work with non-associative floating-point math.
-----------
Note that writing a sequential algorithm with fully defined order is still grossly difficult! IMO, if anyone wants a bit-accurate simulation, use 64-bit integers instead.
If you absolutely must use floats, then define a proper order of all operations, and include sorting the numbers (!!) to force a fully defined order as much as possible. You may need to sort your pool of numbers many, many times per calculation. But that's what you have to do to get a defined ordering.
One thing I've never really been sure about is how context-switching affects this. I've always assumed this could also be a source of non-deterministic floating-point results: when the OS decides to switch threads, it has to save the CPU & FPU state, and restore it when switching back. Since context switches are (practically speaking) non-deterministic on a pre-emptive OS, this would imply truncating 80-bit to 64-bit floating point values could also happen on non-deterministic points in time.
Does anyone know if this is actually true? Or do the full 80-bits get saved/restored on context switches?
The FPU state is saved using the F(N)SAVE x86 instruction, which stores 108 bytes to memory, which contains all visible FPU state.
1. Yes -- On old x86 systems, the x87 registers were 80-bits and the bottom bits were undefined. You could theoretically have a 1-bit difference on some results depending on what the bottom 26 bits (that were undefined, because you had 64-bit floats most of the time, even though the machine did things 80-bits at a time).
2. No -- Modern x86 systems use SSE registers, which are 64-bits. There are a whole slew of configuration options, but if Windows / Linux does their job, you should have the same rounding-errors across your program (unless you manually change the rounding options, but that's your own fault if you go that route).
3. Kinda yes -- Floating point operations are NON-associative. (A+B)+C does NOT equal A+(B+C). The best example is Python:
>>> (1+1)+2.053 9007199254740994.0 >>> 1+(1+2.053) 9007199254740992.0
253 is the first value which starts to "round off" 1.0 in double-precision. So (1+2.053) == 2.053, because the value was rounded off.
Due to #3, even if you did everything correctly (ie: set the rounding flags so they were consistent), if your arithmetic happened in slightly different orders (very common in network-games, where different players may have their updates in slightly different orders), you end up with a 1-bit error between clients... which completely borks your simulation.
Because #3 is nonintuitive, many people think that you get different results when using floats. But its simply due to non-associativity.
The newlines also helped, by the way, because I was wondering what putting the two numbers next to each other was meant to indicate.
I spent a long time once debugging an issue arising from this. Code was basically:
Nothing in between the assignment and the check changed the value of x or y, yet the check triggered the fail condition. Turned out to be that one of the value stayed in the 80 bit x87 register and one was written to RAM as a 64 bit value, then loaded back into an x87 register for the check, resulting in the inequality. Running in a debugger wrote both values out of the registers before the check, making the problem unreproducible.What was the check for?
(Even during the FPU days, every ; should have rounded each value to the size of the variables, unless you use a special compiler switch to make FP math faster)
Modern applications and compilers largely do not use x87, but they can, and x87 results depending on hidden bits persists in modern x86_64 CPUs.
Are your sure of this? It's my understanding that SSE registers require the use of a special API and standard floating point operations do not use them.
But the last time I worked with them was writing a SIMD vector library 7 years ago.
No, SSE registers are used by most compilers that do floating point these days. They support all the usual IEEE float math operations and a host of bit twiddling operations as well as vector operations. The vector operations do still require using compiler intrinsics in C++, although some autovectorization does occur in gcc, icc, and llvm.
For 32-bit systems, you still theoretically have to check, but I think every halfway modern compiler also uses SSE2 as the default target.
This is a good argument for stricter typechecking than anything else. In Rust, a condition like that wouldn't compile because of the mixed types. That's probably a good call in my opinion: it alerts the programmer that there's something funky going on, and it forces the programmer to be explicit about how the comparison should actually happen.
I do find myself doing the "cargo cult" thing from time to time. I was working on a codebase where Lua was embedded in C++, and there were all these flags that had to work in both systems. Because Lua only supports 64-bit floats as a numeric type, the natural type for these were doubles, and "by convention" they only ever had the values 0.0 or 1.0. It was a bit silly of me, but everytime i saw the line `if (fSomeFlag == 1.0f) ...` in the code, I would wince. It just feels wrong to do that to floats. I found myself regularly writing `if (fSomeFlag > 0.5f) ...` instead.
Of course, I was wrong about this. These weren't values determined as a result of arithmetic, they were always set to whole numbers. There was never going to be a case where the actual value of the flag would be 0.9999998f or whatever. But still: once you've been burned by the "floating point equality comparison", you live in fear of it forever after.
My point was that many researchers can only cobble together somewhat barely working software by taking snippets from their undergraduate textbooks (or googling/stack overflowing, for those under 40), and have no idea about many/most of the underlying principles. If only they would recognize that, and leave the software development to professionals, instead of treating it as an inconsequential implementation detail that is beneath them. Then again, if they did, they wouldn't have to pay me what they do to fix it up afterwards. So meh?
Knock, knock. Who is there? Floating-point contraction.
The point is to demonstrate a property of IEEE floats, not the language. I'd expect CL has some means of using IEEE floats too, and with the appropriate syntax, the same issues could be demonstrated.
- Sometimes, one needs precision for numbers not close to 1. A UNIX timestamp is a good example -- for the current date, it provides ~1uS resolution in 64-bit ieee754. I could not calculate what the resolution would be for posits, but I suspect much worse.
- The lack of positive and negative infinites will break naive min/max calculations which start accumulator with +/-inf.
Now, you might say that those things should not be done, and that software that uses those patters is defective. But even then, there is a lot of software like this, and so posits will never become a default "float" type.
gettimeofday() and clock_gettime() provide higher resolution timestamps (respectively µs and ns), using typedefs instead of just numbers.
Some APIs return floating-point UNIX time in order to provide sub-second accuracy (the decimal part is the fractional second). Python's time.time() does that for instance.
On the other hand, floating-point time is pretty common in scripting languages -- for example Python has time.time(); ruby has Time.now.to_f. It is not perfect, but great for smaller scripts: fool proof (except for the precision loss), roundtrips via any serialization format, and easy to understand. And no timezone problems at all!
Wouldn't the equivalent pattern just be to start at +/- typemax(Posit) or whatever? As you would for integers, which lack infinity.
The solution seems simple to me though. Whatever syntax programmers are using to initialize floats to +/-inf, make that ensure to posit min and max as well.
2016: https://news.ycombinator.com/item?id=11573172
2017: https://news.ycombinator.com/item?id=15617633
https://news.ycombinator.com/item?id=14669913
Many other articles (but not many comments):
https://hn.algolia.com/?sort=byDate&dateRange=all&type=story...
https://hn.algolia.com/?sort=byDate&dateRange=all&type=story...
Posits are Type III Unums IIRC, the original proposal was with Type I Unums.
Previous proposals were variable length formats, which is pretty much a nonstarter for a variety of reasons. This proposal (posits) is a fixed length format with variable length fields.
https://marc-b-reynolds.github.io/math/2019/02/06/Posit1.htm... https://hal.inria.fr/hal-01959581v2 https://hal.inria.fr/hal-02131982
The bottom line is they are a tradeoff; they certainly aren't purely better than IEEE floats. Plus they seem to have a non-trivial hardware cost (one of the linked papers shows the adder is twice as large as the equivalent IEEE adder and has double the latency!)
What? Surely it does not have infinite precision with a finite number of bits, and also doesn't seem to be a rational number representation.
A rounding error is when you have a number, lets say 0.75, and due to rounding it is recorded as 1.00. The "rounding error" is 0.25.
An alternative to rounding to 1.00 would be to have a mechanism which says "the value is between 0.50 and 1.50". This way, there is no actual rounding, as it doesn't commit to a rounded value, so there is technically no "rounding error".
A neat advantage of recording an interval rather than rounding is that the "error" is preserved in the data through arithmetic, so if there is some following code that runs x * 100, a rounding mechanism would say "the value is 100" whereas an interval mechanism would say "the value is between 50 and 150". Then, if the user only looks at the output, it will be clear that there is a wide error range and something needs to be fixed, rather than the output indicating a precise answer when really it suffers from significant rounding errors.
> Gustafson said that for training, 32-bit floating point is overkill and in some cases doesn’t even perform as well as the smaller 16-bit posits
If posits have a bit number, then they are not of variable accuracy. This is simply sloppy explaining.
At some point, rounding has to happen (call it an interval if you want, but you're not being helpful). As a programmer, I WANT rounding to happen. I depend on it. You don't break out the floats unless you are prepared for some information to be lost. If you try to never lose any information, then you wind up with a monstrosity like Maple and Mathematica in which all steps need to be curated by hand to occasionally reduce the giant glob of fractions and implicit solves into something concrete, but imperfect.
I read the article and took a glance at https://www.johndcook.com/blog/2018/04/11/anatomy-of-a-posit..., but I'm still completely unable to answer this question of how accuracy is shed.
Sloppy journalism.
This can represent the exact sum of any number of floating point values (or products of floating point values) with the intermediate operations preserved at infinite precision, up to a single rounding at the end.
e.g., for computing an inner product of vectors \sum_i x_i y_i with x_i and y_i in floating point, a fused multiply-add operation (as on today's computers) would perform something like:
r(x_n y_n + r(... + r(x_2 y_2 + r(x_1 y_1 + 0))...))
where what is performed within the rounding function r() is done to infinite precision and a single rounding at the end. This is also only true if you are accumulating into the same value, and not splitting this operation across multiple values, otherwise there would be additional rounding steps performed.
Using a Kulisch accumulator the result would be:
r(\sum_i x_i y_i)
similarly also only preserving precision if everything is done in a Kulisch accumulator.
You can add a Kulisch accumulator to IEEE floating point or any other floating point as well. In fact the second? ever computer with floating point (the Zuse Z3) performed FP addition using a similar accumulator, but sums-of-products probably didn't preserve the multiplied value though.
Sadly, the guy has a very annoying writing style that makes him sound like a crackpot. The advantages of posits would shine much more if they were not mixed with ridiculous language (posits are floating point numbers, thus they cannot replace them) and outlandish claims (IEEE floating point is deterministic, to the apparent contradiction of many sentences written by Gustafson). It does not help either that Gustafson work only seems to attire the attention of semi-illiterate journalists who do not really understand what they are talking about.
What about this then, which somebody comments below:
"On old x86 systems, the x87 registers were 80-bits and the bottom bits were undefined. You could theoretically have a 1-bit difference on some results depending on what the bottom 26 bits (that were undefined, because you had 64-bit floats most of the time, even though the machine did things 80-bits at a time)."
And below:
"I spent a long time once debugging an issue arising from this. Code was basically:
Nothing in between the assignment and the check changed the value of x or y, yet the check triggered the fail condition. Turned out to be that one of the value stayed in the 80 bit x87 register and one was written to RAM as a 64 bit value, then loaded back into an x87 register for the check, resulting in the inequality. Running in a debugger wrote both values out of the registers before the check, making the problem unreproducible."This behavior might not be IEEE defined, but even as IEEE undefined (and implementation dependent), it still qualifies those IEEE compliant implementations as non-deterministic.
Or how about:
"parallel systems, most likely GPUs, where it is common to have slightly different results even on the same operation, on the same system. This is due to the fact that several pipelines will do calculations in parallel and depending on which pipeline ends first (which can depend on many factors like current temperature) the sums can happen in a different order, leading to different rounding results."
due to fp associativity?
This is exactly the kind of fringe shitfuckery that must be avoided at all costs if you want to promote posits seriously.
Posits also can't achieve associativity in all conditions, you need to use "quire" accumulators with higher precision. However, if the decision of using quires were left to the compiler, you'd end up with the exact same problem of unclear precision as for x87. To avoid that, those accumulators will need to be annotated in the code, and straightforward programs that do not use them will continue to exhibit non-associativity.
Overall I'm not sold on the quire thing. It seems like a great big accumulator is a solution that is fairly independent from the floating point format.
It could still be a good idea to mandate a standard version in the spec for consistent behavior.
> "parallel systems, most likely GPUs, where it is common to have slightly different results even on the same operation, on the same system. This is due to the fact that several pipelines will do calculations in parallel and depending on which pipeline ends first (which can depend on many factors like current temperature) the sums can happen in a different order, leading to different rounding results."
> due to fp associativity?
As far as I understand, that's not quite the case (on Fermi+ Nvidia, as previous generations had no real support for proper IEEE single compute).
There are pipelines in parallel, but almost all non-determinism in the order of execution (resulting in fp associativity) is exposed to the programmer, except for atomic instructions. Those can even depend on clock skew across PCIe or NVLink, if they operate on remote memory. NVLink supports remote execution in a recent version (I suggest the microbenchmarking-derived microarchitecture analysis papers for Volta and Turing, even if they are less hands-on the Nervana's maxas (with the SGEMM walk-through, which I really recommend)), and thus any hope of reliably tricking synchronization behavior via undefined opcodes/scheduler instructions trickery is deliberately out of the window.
On IBM POWER9 CPUs they can apparently delegate execution of atomic operations to the host CPU, which might (I don't know enough about POWER9 cache coherency /NUMA protocols) delegate them over the NUMA fabric to a different rack-mount unit bonded into the same NUMA mesh.
TL;DR: Recent Nvidia GPUs do not suffer implicit non-determinism in IEEE floats unless you use hardware atomics.
Non-determinism is exposed and introduced in explicit concurrent data structures [0].
[0]: If you count a datastructure with a simple spinlock guarding it as a concurrent datastructure.
PS: I'll add references / links later on the laptop.
Although IEEE is non-deterministic for other reasons (the transcendental example below is a good illustration) this argument is incorrect.
Performing operations on a non-IEEE 80-bit representation and then using a non-IEEE comparison is not a valid argument about non-determinism in IEEE.
Desktop graphics cards intentionally ignore IEEE 754 rules if it allows them to squeeze out a little bit of performance. (or die space, or power consumption depending on the application) If the bad guy is drawn half a pixel out of position or if his shirt is slightly the wrong shade of red, nobody cares. But if your graphics card draws at 72 frames per second and your competitor's cards draws at 73, every publication is going to say their card is better than yours. Posits won't fix this either.
These problems you point out aren't technical problems, they're institutional ones. IMHO it's generally a bad idea to try to fix institutional problems with technical solutions, but I went try to stop you from attempting it.
How can that be? The IEEE standard does not place many requirements on the precision of the transcendental functions, and AFAICT a transcendental doesn't even need to return the exact same answer twice. That is, it is correct for a compiler to, if it can, evaluate a transcendental at compile-time with infinite precision, and if it cannot, call a run-time function that computes it. Math libraries can be dynamically linked, and all can produce different results here, so it is not that a compiler can go out of its way to make things deterministic either.
For instance, the claim that "posits can be a direct replacement of floating point" is not true. There are a few tricks, such the Fast Invertion Square Root that abuses the floating point format to do fast arithmetic operation. This would break replacing with posit.
Also he claims that early FPGA posit implementation uses less area and is has high performance than a Floating Point implementation. This cannot be true^. The posit unit IS A floating point unit with a encoder/decoder unit. The posit floating point unit will actually have a larger word size when decoded than ieee floating point because it has more precision than ieee format. Also the features such different rounding supported by ieee but not by posit uses a negligible extra area.
^ Unless, ofc, he is comparing apples to oranges.
The Golomb-Rice prefix tree for the exponents is certainly cool. It has the air of Gosper's "continued logarithms", which are like the same idea but turned to the max: use always the same mantissa (which is implicit) and represent only the exponent to a high precision.
The summary:
These architectures are evaluated on recent FPGA hardware and compared to their IEEE-754 counterpart. The standard 32 bits posit adder is found to be twice as large as the corresponding floating-point adder. Posit multiplication requires about 7 times more LUTs and a few more DSPs for a latency which is 2x worst than the IEEE-754 32 bit multiplier.
Gustation is being too ambitious trying to reinvent the whole floating point framework. I think he should focus in what really matters: the compact encoding. A lot of application could save memory bandwidth using some kind of posit inspired format. A pack/unpack unit that converted the posit encoding to floating point inside the core would could be game changing.
He also proposed a good idea on code ergonomic: Make the accumulator a type (quire in the posit). Making the programmer aware of the accumulator can lead to better to code.
By the way, reading the Kahan work in floating point I could understand why Gustafson want's to avoid the IEEE. The Kahan implementation favored Intel, not necessarily the community.
Anyway. My point of views are from the hardware side of the things. The format seems to have some arithmetic properties that might be useful for mathematicians.
Everyone who reads that who is cognitively typical will understand "floating-point" (which, by the way, needs a dash, if we are going to nitpick) as a nickname for "IEEE 754 floating-point" or "traditional floating-point", rather than an insinuation that posits are, in contrast, fixed-point or something other than floating-point.
Storage and transmission speed have progressed exponentially while execution units have become the bottleneck. A floating point format that is 20% denser or more accurate but that requires a 2x number of gate delays to implement is major step backwards, except maybe in highly specialized applications.
This being said, I object to your premise: transmission and memory storage are getting comparatively more costly vs computation. From "Computer Architecture, A Quantitative Approach", 6th edition in figure 1.13 there are some values for the TSMC 45nm process, so hardly the leading edge and it only gets worse with finer nodes: - 32 bits integer multiplication: 3.2 pJ - 32 bits float multiplication: 3.7 pJ - 32 bits read in a small 8 kB SRAM: 5 pJ - external DRAM 32 bits read: 640 pJ
We have a lot of transistor nowadays, and as long as the computation can be pipelined halving the memory accesses could be a win (TBC).
Also on from the same Inria team as [1], and also covering hardware cost but more than this: "Posits: the good, the bad and the ugly": https://hal.inria.fr/hal-01959581v3/document
[1] "Hardware cost evaluation of the posit number system", https://hal.inria.fr/hal-02131982/document
Memory bandwidth relative to instruction to instruction latency has gone way down. Memory is slower now relative to the amount of math one can do.
We are also swimming in extra silicon, so even if what you are saying is true (2x the number of gates, all adding to the delay), posits would still be a win.
You are arguing from an arbitrary what-if position, it doesn't look good.
I didn't find a resource specifying how many exponent bits to actually use? (The 'es' value)
I don't see anything different in this system than regular floats that would ensure consistent results across machines/compilers/...? It's floats plus unary encoded regime bits to have var length exponent, so everything that can make floats inconsistent can still happen here: non commutativity, different exponent sizes or precisions of intermediate values, different rounding modes, ...
Long story short: es=log2(nbits) -3
https://people.eecs.berkeley.edu/~wkahan/EndErErs.pdf
https://people.eecs.berkeley.edu/~wkahan/SORNers.pdf
I haven't had time to digest it all, but he is certainly critical of Gustafson's proposals. Not sure though the articles linked above cover the latest and greatest Unum III. At any rate, I'd pay close attention to Kahan's critique.
edit: One of several examples found in the "Posits: the good, the bad and the ugly" paper linked in the thread : 10.0 * 2.0 = 16.0 in posit8
I'm not sure how 10.0 * 2.0 = 16.0. I'm not sure what posit8 means, but it can only be correct if it switches halfway from base 8 representation to base 10, which is a bit weird, but at least the calculation is correct. (Otherwise it would be so incorrect to be unusable for anything.)
Overall, posits are a new trade-of that will give you better precision (nothing exact, it is still an approximation) when you manage to keep all your number in a small range. Once you get out of that range precision drops significantly (whereas the precision of classical floating points drops gradually).
Posit8 are equivalent to 8 bits floating points (minifloats) making them an easy target for pathological cases but the example still illustrate the fact that, contrary to floating-point arithmetic, multiplication by a multiple of two is not exact with posits (one of several good properties we take for granted and would lose when switching to posits).
You can solve it by switching to decimal floating points, they are defined by IEEE as well:
https://en.wikipedia.org/wiki/Decimal_floating_point
Claims like that are best taken with a grain of salt:
> It also does away with rounding errors, overflow and underflow exceptions, subnormal (denormalized) numbers, and the plethora of not-a-number (NaN) values. Additionally, posits avoids the weirdness of 0 and -0 as two distinct values.
Ok, so posits will probably work fine as a drop-in replacement when my application makes simple use of floats. But, assuming my application is doing non-trivial math, it's probably aware of the above edge cases. Thus, dropping in posits might have lots of weird side effects where I had to work around weird side effects of floats.
Your boss, on the other hand, may try to armchair architect you, which is a real (and underreported) problem,
Yeah, but posits appear to have fewer weird edge-cases. Plus the people trying out posits are constantly trying to find methods to work around the limitations too.
During this years conference on posit maths Florent de Dinechin had a really nice talk bringing up all current issues with posits and ways that floating point maths had found workarounds over the year, as a kind of challenge to make posits catch up[0][1]. The community took it really well, as far as I can see, and Gustafson in particular seemed delighted because he genuinely wants everyone to start using better numerical methods.
[0] https://www.youtube.com/channel/UCOstJ2IVC4Y8mbgN0IsowKw
[1] https://www.youtube.com/watch?v=tcX2nRCdZvs
So as far as I'm concerned, whatever performance cost these alternate methods may have, it would be well worth it to avoid the pitfalls of IEEE floats. Intel chips have had BCD support in machine code; I'm surprised nobody has made a decent fixed point lib that is widely used already.
Note that fixed radix point does not solve the common issues with representing rational base 10 fractions. A base10 fixed radix solution would, but so would IEEE754's decimal64 spec, which would eliminate representation error when working exclusively in the context of base10 e.g finance, but these are not found in common hardware and do not help reduce propagation of error due to compounding with limited precision in any base.
A 64-bit integer is big enough to express the US National Debt in Argentine Pesos.
It's actually logical: the number of developers doubles roughly every 5 years. It means that half of the developers have less than 5 years of experience. If they don't teach you this in school (university), you will have to learn from someone who knows, but chances are the other developers are as clueless as you.
If you don't care about performance, then the actual solution has no dependency on hardware:
1. Replace the default format for numbers with a decimal point in suitably high level languages with a infinite precision format.
2. Teach people using other languages about floating point and how they may want to use integers instead.
The end. No multi-generation hardware transition required.
IMO, IEEE 754 is an exceptionally good format. It has real problems, but they aren't widely known to people unfamiliar with floats (e.g. 1.0 + 2.0 != 3.0 isn't one of them).
Thanks!
The issue with floating point arrises when comparing very close numbers.
Unlimited precision in any radix point based format does not solve representation error. If you don't understand why:
If you are truly only working with rational numbers and only using the four basic arithmetic operations, then only a variable precision fractional representation (i.e a numerator and demonstrator, which is indifferent to underlying base) will be able to store any number without error (if it fits in memory). Of course if you are using transcendental functions or want to use irrational numbers e.g PI then by definition there is no numerical solution to avoid error in any finite system.I'm not suggesting we replace all our current HW with chips that implement posits (let's fix branch prediction first!!). More that FP should be opt-in for most HLLs.
The numerator is just an integer and integers are just integers and the base doesn't matter. But if the exponent is base 2, then you can have 1/2, 1/4, 1/8 on the base but not 1/5 or 1/10.
They have the survey online at [1] in case you want to see how much you know about fp behavior.
[1] http://presciencelab.org/float
FYI the actual paper is https://ieeexplore.ieee.org/document/8425212 and you can get the PDF here: http://pdinda.org/Papers/ipdps18.pdf
The question is:
and their notes on it: They make it clear in their notes that "are numbers" includes infinities but not NaNs. Now consider the case where a = inf and b = -inf. Then inf + (-inf) is NaN, and (-inf) + inf is NaN, and NaN != NaN.> They make it clear in their notes that "are numbers" includes infinities but not NaNs.
This is definitely not very clear on the form, thought.
Another comment pointed out to the paper with the answers.
A great resource to learn about the FP corner cases is the great Random ASCII blog:
https://randomascii.wordpress.com/category/floating-point/
However, as soon as you start doing anything interesting, you have limited precision as a matter of course.
Sadly most languages don't support something like that out of the box.
Where would one go to better understand how floating points are represented?
> I shudder to think how many e-commerce sites use `float` for financial transactions!
The float IEEE-754 represent up to 9 decimal digits (or 23 binary digits) with precision. The double, represent 17 decimal digits. The error upper bound is (0.00000000000000001)/2 per operation. Likely irrelevant for most e-commerce.
Also, the database stores in currency values using fixed point.
> Intel chips have had BCD support in machine code
BCD is floating point encoding not fixed point. AFAIK, only Intel supports it and very precariously.
> I'm surprised nobody has made a decent fixed point lib that is widely used already.
Nonsense. If you do any scientific computation you have likely have Boost, GMP, MPFR installed in your system. They support arbitrary precision arithmetic with integer (aka fixed point), rational and floating point.
LOL, sure ok. Worked on banking systems for 2 years and been doing scientific computing for many more. Pretty comfortable with fixed and floating point.
> [error bounds] Likely irrelevant for most e-commerce.
Those bounds are theoretical, and there are plenty of occasions I have come across in the past when rounding errors were observed. It was forbidden in the bank to use floating point! We went to enormous lengths to ensure numerical accuracy and stability across systems.
I think this article has a pretty good explanation:
https://dzone.com/articles/never-use-float-and-double-for-mo...
> Nonsense. If you do any scientific computation you have likely have Boost, GMP, MPFR installed in your system. They support arbitrary precision arithmetic with integer (aka fixed point), rational and floating point.
Yes, absolutely right; I have used several of those 3rd party libs myself, as well as hand-rolling fixed point code (esp for embedded systems). I didn't write what I intended. I meant to say that very few languages have first-class fixed point in their standard library. So long as the simple `float` is available as a POD, people will (mis-)use it.
I think in a general purpose HLL, a fixed decimal type should be the default, and you should have to opt in to IEEE-754 floating point.
I'm talking about the average joe e-commerce. The precision requirement of Financial institution and large e-commerce are more strict than most website. For the average e-commerce the extra cost of using a decimal arithmetic framework might not be reasonable. Of course, software engineers should use currency types when they are available.
> https://dzone.com/articles/never-use-float-and-double-for-mo...
The example used by the website is misleading. They print a value with all the significant digits. In reality, e-commerces are only concerned with two digits after the decimal point. I run the same program in C with one billion (-1) iterations and the printed value with two decimal digits was exact. It was the the as it would be if I used decimal arithmetic.
> I think in a general purpose HLL, a fixed decimal type should be the default, and you should have to opt in to IEEE-754 floating point.
Most modern languages have some support to rational, fixed point and currency arithmetic. Nowadays, even GCC's C supports Fixed Point arithmetic [1].
The hardware implementation of floating point arithmetic is concerned about efficiency hardware and support for a lot of uses cases. The ranges and accuracy needed in different application varies widely. The BCD encoding is very inefficient hardware-wise compared to binary encoding.
In summary, I do agree people with working currencies should use the proper currency type, but I also think using IEEE 754 is usually fine, specially in the front-end. Also, I don't think the trade offs of changing from binary arithmetic to decimal arithmetic are not worth it for most people and hardware system.
[1] https://gcc.gnu.org/onlinedocs/gcc-4.9.0/gcc/Fixed-Point.htm...
All of those are important and useful features. Presenting their absence as some kind of advantage shows that Gustafson has no clue.
One can then solve some ODEs to check how these implementation are performing.
Ofcourse the speed will be far slower but one can get number of instructions.