> In ancient times, floating point numbers were stored in 32 bits.
I thought in ancient times, floating point numbers used to be 80 bit. They lived in a funky mini stack on the coprocessor (x87). Then one day, somebody came along and standardized those 32 and 64 bit floats we still have today.
FP4 1:2:0:1 (other examples: binary32 1:8:0:23, 8087 ep 1:15:1:63)
S:E:l:M
S = sign bit present (or magnitude-only absolute value)
E = exponent bits (typically biased by 2^(E-1) - 1)
l = explicit leading integer present (almost always 0 because the leading digit is always 1 for normals, 0 for denormals, and not very useful for special values)
M = mantissa (fraction) bits
The limitations of FP4 are that it lacks infinities, [sq]NaNs, and denormals that make it very limited to special purposes only. There's no denying that it might be extremely efficient for very particular problems.
If a more even distribution were needed, a simpler fixed point format like 1:2:1 (sign:integer:fraction bits) is possible.
There is a relevant Wikipedia page about minifloats [0]
> The smallest possible float size that follows all IEEE principles, including normalized numbers, subnormal numbers, signed zero, signed infinity, and multiple NaN values, is a 4-bit float with 1-bit sign, 2-bit exponent, and 1-bit mantissa.
There's an "Update:" note for a next post on NF4 format. As far as I can tell this is neither NVFP4 nor MXFP4 which are commonly used with LLM model files. The thing with these formats is that common information is separated in batches so not a singular format but a format for groups of values. I'd like to know more about these (but not enough to go research them myself).
When you have so few bits, does it really make sense to invent a meaning for the bit positions? Just use an index into a "palette" of pre-determined numbers.
As a bonus, any operation can be replaced with a lookup into a nxn table.
0.0 + x = x
NaN + x = NaN
+1.0 + -1.0 = 0.0
+1.0 + +1.0 = NaN
-1.0 + -1.0 = NaN
-0.0 = 0.0
-(+1.0) = -1.0
-(-1.0) = +1.0
-NaN = NaN
x - y = x + (-y)
NaN * x = NaN
+1.0 * x = x
-1.0 * x = -x
0.0 * 0.0 = 0.0
/0.0 = NaN
/+1.0 = +1.0
/-1.0 = -1.0
/NaN = NaN
x / y = x * (/y)
More interestingly, how to implement in logic gates. Addition with a 2's complement full adder and NaN detector. Negation with a 2's complement negation circuit. Reciprocal with a 0.0 detector.
Multiplication with a unique logic circuit (use a Karnaugh map):
I'll use custom notation =? ≤≥? <? ≤? for comparison to distinguish from = < ≤.
x =? x = True
Otherwise, a =? b = False
NaN ≤≥? NaN = False
Otherwise, a ≤≥? b = a =? b
-1.0 <? 0.0 = True
-1.0 <? +1.0 = True
0.0 <? +1.0 = True
Otherwise, a <? b = False
a >? b = b <? a
a ≤? b = (a <? b | a ≤≥? b)
a ≥? b = (a >? b | a ≤≥? b)
In logic gates: For =?, bitwise equality. For ≤≥?, bitwise equality and a NaN detector. For <?, use:
ab <? cd = a&b&~c | ~a&~b&~c&d
I separate =? from ≤≥?. =? compares value, while ≤≥? compares order. NaN has no ordering, so it compares false. IEEE float only uses ≤≥? and names it ==.
I too want fewer bits of mantissa in my floating point!
But what I wish is that there had been fp64 encoding with a field for number of significant digits.
strtod() would encode this, fresh out of an instrument reading (serial). It would be passed along. It would be useful EVEN if it weren't updated by arithmetic with other such numbers.
Every day I get a query like "why does the datum have so many decimal digits? You can't possibly be saying that the instrument is that precise!"
Well, it's because of sprintf(buf, "%.16g", x) as the default to CYA.
Also sad is the complaint about "0.56000 ... 01" because someone did sprintf("%.16f").
I can't fix this in one class -- data travels between too many languages and communication buffers.
In short, I wish I had an fp64 double where the last 4 bits were ALWAYS left alone by the CPU.
> It would be passed along. It would be useful EVEN if it weren't updated by arithmetic with other such numbers.
It would be useful if you could then pass it to an "about equal" operator, too.
I don't need to know that the alternator is putting out 13.928528V, and sure as hell I know you're not measuring that accurately. It's precise but wrong.
I want an "about equals" thing so I can say "if Valt == 14 alt_ok=true" kind of thing but tag it to be "about 14" not "exactly 14".
> In ancient times, floating point numbers were stored in 32 bits.
This was true only for cheap computers, typically after the mid sixties.
Most of the earliest computers with vacuum tubes used longer floating-point number formats, e.g. 48-bit, 60-bit or even weird sizes like 57-bit.
The 32-bit size has never been acceptable in scientific computing with complex computations where rounding errors accumulate. The early computers with floating-point hardware were oriented to scientific/technical computing, so bigger number sizes were preferred. The computers oriented to business applications usually preferred fixed-point numbers.
The IBM System/360 family has definitively imposed the 32-bit single-precision and 64-bit double-precision sizes, where 32-bit is adequate for input data and output data and it can be sufficient for intermediate values when the input data passes through few computations, while otherwise double-precision must be used.
21 comments
[ 3.9 ms ] story [ 42.0 ms ] threadSomeome didn't try it on GPU...
I thought in ancient times, floating point numbers used to be 80 bit. They lived in a funky mini stack on the coprocessor (x87). Then one day, somebody came along and standardized those 32 and 64 bit floats we still have today.
In ancient times, floats were all 60 bits and there was no single precision.
See page 3-15 of this https://caltss.computerhistory.org/archive/6400-cdc.pdf
S:E:l:M
S = sign bit present (or magnitude-only absolute value)
E = exponent bits (typically biased by 2^(E-1) - 1)
l = explicit leading integer present (almost always 0 because the leading digit is always 1 for normals, 0 for denormals, and not very useful for special values)
M = mantissa (fraction) bits
The limitations of FP4 are that it lacks infinities, [sq]NaNs, and denormals that make it very limited to special purposes only. There's no denying that it might be extremely efficient for very particular problems.
If a more even distribution were needed, a simpler fixed point format like 1:2:1 (sign:integer:fraction bits) is possible.
> The smallest possible float size that follows all IEEE principles, including normalized numbers, subnormal numbers, signed zero, signed infinity, and multiple NaN values, is a 4-bit float with 1-bit sign, 2-bit exponent, and 1-bit mantissa.
[0] https://en.wikipedia.org/wiki/Minifloat
It seems that life is imitating art.
https://github.com/sdd/ieee754-rrp
[1] https://tom7.org/nand/
Or does that matter - its the kernel that handles the FP format?
Shouldn't that be m mantissa bits (not y) -- i.e. typo here -- or am I misunderstanding something?
As a bonus, any operation can be replaced with a lookup into a nxn table.
Multiplication with a unique logic circuit (use a Karnaugh map):
But what I wish is that there had been fp64 encoding with a field for number of significant digits.
strtod() would encode this, fresh out of an instrument reading (serial). It would be passed along. It would be useful EVEN if it weren't updated by arithmetic with other such numbers.
Every day I get a query like "why does the datum have so many decimal digits? You can't possibly be saying that the instrument is that precise!"
Well, it's because of sprintf(buf, "%.16g", x) as the default to CYA.
Also sad is the complaint about "0.56000 ... 01" because someone did sprintf("%.16f").
I can't fix this in one class -- data travels between too many languages and communication buffers.
In short, I wish I had an fp64 double where the last 4 bits were ALWAYS left alone by the CPU.
It would be useful if you could then pass it to an "about equal" operator, too.
I don't need to know that the alternator is putting out 13.928528V, and sure as hell I know you're not measuring that accurately. It's precise but wrong.
I want an "about equals" thing so I can say "if Valt == 14 alt_ok=true" kind of thing but tag it to be "about 14" not "exactly 14".
I think Cray doubles were 128 bits, and their singles were 64… which makes it seem like smaller floats are just a continuation of the eternal trend.
This was true only for cheap computers, typically after the mid sixties.
Most of the earliest computers with vacuum tubes used longer floating-point number formats, e.g. 48-bit, 60-bit or even weird sizes like 57-bit.
The 32-bit size has never been acceptable in scientific computing with complex computations where rounding errors accumulate. The early computers with floating-point hardware were oriented to scientific/technical computing, so bigger number sizes were preferred. The computers oriented to business applications usually preferred fixed-point numbers.
The IBM System/360 family has definitively imposed the 32-bit single-precision and 64-bit double-precision sizes, where 32-bit is adequate for input data and output data and it can be sufficient for intermediate values when the input data passes through few computations, while otherwise double-precision must be used.