Fully agree with the article (as my comment history would attest). I wonder why people worry so much more about "determinism" over conformance to a spec. (a compiler can be nondeterministic and correct which is not particularly good but definitely not worse than being incorrect)
Maybe most devs don't care so much about the language specification and just expect the code to do vaguely what it looks it should do intuitively? This is not very clear to me, at least for libraries I guess a lot of people don't read API docs and just call the API hoping it does what the name says (preconditions be damned) in which case nondeterminisc observable behavior would be more problematic than nonconformance to a spec (API doc) they don't read.
1st: Even with temp 0 and seed set, you can still get variations, I've heard the explanation relates to batching with other tenants and race conditions in the gpu parallel processing.
Also slower int quantized models may reduce or eliminate this fluctuation.
2nd: we need to talk about the difference between determinism and chaos. Because I don't give a fuck if "Write me the firmware for a TC scanner" always produces the same output, I care whether it will be different if it's not capitalized.
3 comments
[ 4.4 ms ] story [ 22.7 ms ] threadMaybe most devs don't care so much about the language specification and just expect the code to do vaguely what it looks it should do intuitively? This is not very clear to me, at least for libraries I guess a lot of people don't read API docs and just call the API hoping it does what the name says (preconditions be damned) in which case nondeterminisc observable behavior would be more problematic than nonconformance to a spec (API doc) they don't read.
1st: Even with temp 0 and seed set, you can still get variations, I've heard the explanation relates to batching with other tenants and race conditions in the gpu parallel processing.
Also slower int quantized models may reduce or eliminate this fluctuation.
2nd: we need to talk about the difference between determinism and chaos. Because I don't give a fuck if "Write me the firmware for a TC scanner" always produces the same output, I care whether it will be different if it's not capitalized.