Would your aarch64 test suite have caught it? Testing is fine and all, but it can only find bugs the test authors predicted. There will always be some issues that test suites miss.
It's obvious to add regression tests for this and related issues now that they've been found. And great! That's what regression tests are for. But testing is always going to be fundamentally incomplete in that it can only cover non-obvious things after they've already been encountered.
I would've expected a compiler to have a huge integration/regression test suite which just consists of other pieces of software's test suites. Grab the N most popular Haskell programs with test suites, compile them, run their test suites, verify that they still pass on the new version.
> Crater is a tool for compiling and running tests for every crate on crates.io (and a few on GitHub). It is mainly used for checking the extent of breakage when implementing potentially breaking changes and ensuring lack of breakage by running beta vs stable compiler versions.
I'm not sure what you mean by this. `cabal-install` is quite deterministic; you can even freeze the index date for full build plan determinism.
The largest challenge in this area is the fact that core libraries' interface sometimes change. Our [head.hackage](https://gitlab.haskell.org/ghc/head.hackage/) infrastructure is one way of addressing this. I agree that it would be great to leverage `head.hackage` to test GHC against a larger body of packages.
It definitely _can_ be deterministic, but my experience of pulling down packages without a frozen index is that it rarely compiles. This, inevitably is because the plan depends on the current state of hackage. A combination of “it might pick a different minor version”, “different versions can cause cascading effects”, “many libraries have breaking changes” and “Haskell is extremely unforgiving of potential breaks” makes it very difficult to use without either a) being really good or b) downloading a freeze index. (I download freeze indexes.)
The whole core libraries thing is difficult, in that I can see it’s a real problem for everyone constantly updating their packages (and consuming them), but also that most of the changes are actually good ideas.
The issue is that basic math doesn't work. I don't dispute that no test suite can catch everything, but yes I do expect that any reasonable test suite that compiles and runs code would have caught this.
I'm still confused, but I hope this is correct. IIUC the problem here is that they do something like
(a // b ) * b
where // is the integer division. They are not breaking all math operations, just math operations that have a very specific pattern.
Also, I'm not sure if it's important that 128 <= b <= 255, because the main problem seams to be they are extending a number and the leading 1 in the binary representation is a problem (???).
So you must test a very specific math pattern, with very specific numbers, and then hope that other parts of the optimizer don't optimize the problem away before the bug in the compiler makes a mistake.
I don't know too much about the automatic test suit in Haskell, but I know more about the automatic test suit in Racket. In Racket there are a lot of test written manually, and some randomized test, and also tests for many of the packages. But I really doubt there is a test somewhere for this specific case.
[Is there a cross language regression test collection? Something that collects weird cases, so other languages maintainers can take a look, translate the code and add it to the tests.]
That is correct. The problem is only triggered by very particular patterns of integer operations which happen to get lowered to C-- where an operand is bound to a local register; this register then gets inappropriately mutated by the code generated by the backend.
As you say, GHC, like most language implementations, has an extensive [testsuite](https://gitlab.haskell.org/ghc/ghc/-/tree/master/testsuite) (10k tests which result in over 30k testcases). Sadly, none of these tests covered this particular case.
Moreover, we also have a systematic [property testing framework](https://gitlab.haskell.org/ghc/test-primops) testing the backends and primitive operations provided by the compiler. This testsuite also didn't test this form of program since it focuses on testing the expression fragment of the Cmm language (and therefore won't generate local register bindings). This is a blind spot that I am going to look into addressing.
It's pretty advanced in the cycle for something this nasty, but 9.4.2 is still "beta"-y. Usually you wait for a x.x.4 release to bump the MV. This sort of thing could be more explicit though.
Ouch, this is a nasty bug. Looks like it’s a secondary consequence of some arm specific issue in the 9.4 support for sized word primitive unlifted types. Good thing folks caught it relatively early in the life cycle of ghc 9.4! (Ghc 8.10.7 is the most recent bug fix release in the 8.10 series, and I try to make sure I support it in my own hackery)
This looks to have a pretty straightforward fix hopefully.
In the space of miscompilation bugs this seems pretty easy to catch. A pretty nasty one happened in ghc 7.6/7.8 , where the register allocator didn’t realize float and double registers were the same registers, so code that mixed the type had total garbage!
Guessing the split here is between internal and external use-cases. Maybe 7% of e.g. Amazon EC2 is ARM, because that's the percentage of orgs AWS and other cloud vendors predict are "ready" for arm. But internally? How much of Amazon S3 is ARM? How much of Cloudflare? Google Search? Netflix? I'd expect pretty high numbers in these vertically-integrated DC niches — they can literally do whatever they want to solve these problems, and one of the largest KPIs for any of these services would be performance + scalability (i.e. requests served per OpEx dollar.)
Why do you think it’s gaining market share? Because more of ARM is deployed than non-ARM.
I couldn’t find the direct source any more, but I remember that in the year 2020 AWS deployed more ARM than non-ARM, and today it is at around 80% ARM and 20% non-ARM for a newly deployed system. You only get to single digit market share because of the enormous amount of non-ARM that still exists and is being used. ARM is growing faster than non-ARM. That’s what I’ve said.
It isn't time to ask that question though. At least, I don't think so and I've been using GHC Haskell for a decade now and for many different commercial use-cases to boot.
Complexity is positively correlated with bugs. Many studies have shown that.
GHC is complex, perhaps too complex. I don't track GHC closely but it seems GHC does seem to suffer from more than a fair share of embarrassing compilation bugs. This feeling is derived anecdotally so I may be wrong. Happy to be corrected.
Someone more conversant with GHC will have to chime in and agree/disagree. Don't let your love of Haskell get in the way of providing an objective opinion :-)
This is reductive thinking under the guise of "objectivity" imo.
What does "complex" even mean here? How do you measure it or quantify it? You can't - and then your "objective" house of cards comes tumbling down. You can't correlate anything with hand waves.
GHC is doing fine. I say this as someone more tuned into it than you. Moralizing and catastrophizing about this bug is a waste of everyone's time. The bug is fixed, the process inefficiencies that contributed to it happening are being corrected. It's software development 101.
do you feel the same way about other libraries or software? Do you have reasoning that you can articulate that GHC is unnecessarily complex in ways that other comparable software is not?
GHC is indeed a large piece of software and, like all software, has its share of bugs. Some of these bugs are indeed attributable to GHC's complexity or, rather, the complexity of its domain: compiling high-level languages to efficient machine code is not easy. To determine whether GHC is too complex we need a definition of what constitutes "too complex". I think a reasonable definition would be "too complex to be managed via abstraction". In general I think GHC does a reasonably good job here. There is a reason why Haskell programs passes through four different intermediate languages as it works its way through GHC. Each of these languages, and associated compiler passes, can focus on a specific set of needs constrained by a specific set of invariants.
Where this approach to encapsulating complexity can fall short is where we must maintain hard-to-check relationships between the state of the machine at runtime and the compiler. This is the source of many of the bugs which are most readily attributable to GHC's complexity (e.g. [#13615](https://gitlab.haskell.org/ghc/ghc/-/issues/13615) and [#14346](https://gitlab.haskell.org/ghc/ghc/-/issues/14346) come to mind here).
However, the sign-extension issue noted in this thread is, in my opinion, not one of these issues. In many ways it is a very boring bug, arising from a simple misunderstanding of the semantics of a function in the code generator (namely, that `getSomeReg` isn't guaranteed to return a fresh local register). There are a number of ways that this could have been caught: better code review, more thorough tests, better use of type safety in the backend. It's certainly an serious (and, frankly, embarrassing) bug, but I don't think it is a sign that the compiler has accrued more complexity than can be managed.
Put simply: No. The native code generator for aarch64 is relatively new, and is bound to have bugs in it that haven't been caught, as others have mentioned, because no one thought to test for this specific thing. There's already a patch for this bug, and a new regression test to catch it. These things happen in all compilers, and all software, relatively often.
I don't understand how a bug in GHC has blown up more than any bug in any recent GCC or clang - probably because the bug reporter decided that hyperbole was going to be their tool of choice instead of behaving like an adult.
> I don't understand how a bug in GHC has blown up more than any bug in any recent GCC or clang - probably because the bug reporter decided that hyperbole was going to be their tool of choice instead of behaving like an adult.
It didn't blow up more or less than bugs in other language implementations. When there are serious bugs in GCC, LLVM the JVM, etc. it always goes to HN's front page.
Also, aarch64 Haskell is almost exclusively not in production (although there's probably some iOS/macOS apps out there - very cool if so!). This bug shows up on devs' M1 MBPs and the CI used to build their Nix cache.
It's a nasty bug, but in reality it isn't some P0 monstrosity despite how "eXtReMeLy aLaRmInG" it seems.
37 comments
[ 3.2 ms ] story [ 93.0 ms ] thread"CCall testsuite doesn't test `signed` arguments and results. This would have helped catch [a previous ghc regression on aarch64]".
https://gitlab.haskell.org/ghc/test-primops/-/issues/3
It's obvious to add regression tests for this and related issues now that they've been found. And great! That's what regression tests are for. But testing is always going to be fundamentally incomplete in that it can only cover non-obvious things after they've already been encountered.
https://rustc-dev-guide.rust-lang.org/tests/crater.html
https://github.com/rust-lang/crater
https://crater.rust-lang.org/
Even then such a bug as GHC's can conceivably not be detected.
The largest challenge in this area is the fact that core libraries' interface sometimes change. Our [head.hackage](https://gitlab.haskell.org/ghc/head.hackage/) infrastructure is one way of addressing this. I agree that it would be great to leverage `head.hackage` to test GHC against a larger body of packages.
The whole core libraries thing is difficult, in that I can see it’s a real problem for everyone constantly updating their packages (and consuming them), but also that most of the changes are actually good ideas.
I'm still confused, but I hope this is correct. IIUC the problem here is that they do something like
where // is the integer division. They are not breaking all math operations, just math operations that have a very specific pattern.Also, I'm not sure if it's important that 128 <= b <= 255, because the main problem seams to be they are extending a number and the leading 1 in the binary representation is a problem (???).
So you must test a very specific math pattern, with very specific numbers, and then hope that other parts of the optimizer don't optimize the problem away before the bug in the compiler makes a mistake.
I don't know too much about the automatic test suit in Haskell, but I know more about the automatic test suit in Racket. In Racket there are a lot of test written manually, and some randomized test, and also tests for many of the packages. But I really doubt there is a test somewhere for this specific case.
[Is there a cross language regression test collection? Something that collects weird cases, so other languages maintainers can take a look, translate the code and add it to the tests.]
As you say, GHC, like most language implementations, has an extensive [testsuite](https://gitlab.haskell.org/ghc/ghc/-/tree/master/testsuite) (10k tests which result in over 30k testcases). Sadly, none of these tests covered this particular case.
Moreover, we also have a systematic [property testing framework](https://gitlab.haskell.org/ghc/test-primops) testing the backends and primitive operations provided by the compiler. This testsuite also didn't test this form of program since it focuses on testing the expression fragment of the Cmm language (and therefore won't generate local register bindings). This is a blind spot that I am going to look into addressing.
This looks to have a pretty straightforward fix hopefully.
In the space of miscompilation bugs this seems pretty easy to catch. A pretty nasty one happened in ghc 7.6/7.8 , where the register allocator didn’t realize float and double registers were the same registers, so code that mixed the type had total garbage!
https://www.youtube.com/watch?v=fdyh3YQ-ZWI
I couldn’t find the direct source any more, but I remember that in the year 2020 AWS deployed more ARM than non-ARM, and today it is at around 80% ARM and 20% non-ARM for a newly deployed system. You only get to single digit market share because of the enormous amount of non-ARM that still exists and is being used. ARM is growing faster than non-ARM. That’s what I’ve said.
It is time to ask if GHC itself has become too complicated.
It isn't time to ask that question though. At least, I don't think so and I've been using GHC Haskell for a decade now and for many different commercial use-cases to boot.
GHC is complex, perhaps too complex. I don't track GHC closely but it seems GHC does seem to suffer from more than a fair share of embarrassing compilation bugs. This feeling is derived anecdotally so I may be wrong. Happy to be corrected.
Someone more conversant with GHC will have to chime in and agree/disagree. Don't let your love of Haskell get in the way of providing an objective opinion :-)
What does "complex" even mean here? How do you measure it or quantify it? You can't - and then your "objective" house of cards comes tumbling down. You can't correlate anything with hand waves.
GHC is doing fine. I say this as someone more tuned into it than you. Moralizing and catastrophizing about this bug is a waste of everyone's time. The bug is fixed, the process inefficiencies that contributed to it happening are being corrected. It's software development 101.
Where this approach to encapsulating complexity can fall short is where we must maintain hard-to-check relationships between the state of the machine at runtime and the compiler. This is the source of many of the bugs which are most readily attributable to GHC's complexity (e.g. [#13615](https://gitlab.haskell.org/ghc/ghc/-/issues/13615) and [#14346](https://gitlab.haskell.org/ghc/ghc/-/issues/14346) come to mind here).
However, the sign-extension issue noted in this thread is, in my opinion, not one of these issues. In many ways it is a very boring bug, arising from a simple misunderstanding of the semantics of a function in the code generator (namely, that `getSomeReg` isn't guaranteed to return a fresh local register). There are a number of ways that this could have been caught: better code review, more thorough tests, better use of type safety in the backend. It's certainly an serious (and, frankly, embarrassing) bug, but I don't think it is a sign that the compiler has accrued more complexity than can be managed.
I don't understand how a bug in GHC has blown up more than any bug in any recent GCC or clang - probably because the bug reporter decided that hyperbole was going to be their tool of choice instead of behaving like an adult.
It didn't blow up more or less than bugs in other language implementations. When there are serious bugs in GCC, LLVM the JVM, etc. it always goes to HN's front page.
It's a nasty bug, but in reality it isn't some P0 monstrosity despite how "eXtReMeLy aLaRmInG" it seems.