The study in question is the Imperial College study projecting the spread of COVID-19. It used (apparently an earlier version of) this code to generate the data for its analysis.
This study was, of course, the basis for lockdowns around the world.
You sound like an anti-scientific snowflake, FWIW. I don't see how your position actually advances the scientific discourse; you didn't open up any PRs to improve the codebase, and without any other options, it sounds like you just want the scientists to go away and stop trying to end the pandemic.
> This study was, of course, the basis for lockdowns around the world.
[citation needed]
Afaik this was a lone study that might have been relevant only to the UK debate (and very late at that). Italy had already been in full lockdown for weeks before it was published, quite a few other European countries had gone in shutdown or closed borders, and obviously Asian countries had already taken countermeasures for months.
Being so cavalier with the truth is, of course, why intelligent people don't take seriously a lot of anti-science attitudes.
The timelime doesn't match tho. The article is dated 19/03 and mentions a study released the previous week (so 12/03 or late). The italian lockdown started on the 22/02 [0] for a few towns, and a couple of weeks later was extended to the entire peninsula.
According to Le Monde (~ French NY Times), it's this research that prompted the French government to urgently implement this "lock down" solution. It was criticized by some French scientists in the government's own expert group.
But I don't really blame Ferguson. If President Macron listens to him, or Alex Jones, or any other crank or random guy with an opinion and a model, instead of his own experts, he is the one to blame. I think the culture of the "safety principle" and excessive caution is also to blame.
I've honestly lost track of all the simulation papers over the last few weeks, is there anything in particular in this one that's not reflected by any one of those? I don't think I've seen a single one that didn't come to suggesting a scale that calls for some lockdown requirements.
At least in Europe it wasn't a major part of decisions. When it was published, it mostly corroborated what people here were already seeing, that urgent care facilities got - or were going to get - overwhelmed in a few weeks if contact tracing failed to contain close to all cases.
There is also a "control" group of sorts in us Swedes, as the Swedish health authority choose to act differently, initially. Their recommendation as of now is essentially equivalent to quarantine, except for non-adults. It's important to understand that it is really to instigate anything like a curfew in Sweden, at least unless we get attacked military by another country. The laws as they stand don't really allow that. So the government is also somewhat limited in what they can do.
The situation here is that urgent care has been overwhelmed for about a month, and a lot more people have died or been gravely ill than during even really bad flu seasons. We're also probably far from done yet, as it's much easier to get people to stay indoors when it's cold and rainy outside, and as its been getting warmer more people have been disrespecting the recommendations.
I think this sets a bad example. In many fields the current standard is that code isn't published at all. If bad coding practices will cause you to get bullied into retracting your paper, that's just another argument for why people won't publish their code. Of course if somebody does find a big that invalidates the results of the paper given the stated assumptions, that's another matter, but that doesn't seem to be what's alleged here.
Why exactly? We could also probably require a website to test the application, a mobile app on all platforms and then a native OS apps on all platforms. I apologise for being facetious but writing code is a serious, specialized task. It should be left to the people who specialise in it, scientists aren't the specialists for it.
I think it's important to understand that we only require information to generate algorithms written in the paper. As such, assumptions and limitations should be listed enough to generate the material in the paper. One way to do so is code and data, it is certainly not the only way.
I empathise with what you say, but it's also true that maybe things could be better.
We don't expect universities to build their own laboratories and halls of residence from first principles - they will call on property developers, architects, and contractors. It would be good if there were more IT services, built by specialists, that academics could tap when required - so they would spend more time advancing their fields of expertise and less time writing mediocre code. Things like PythonAnywhere / Notebooks, Lambda etc are a step forward, but there is still ample space for advancement.
Getting software engineers interested in your science isn't easy. Though in this particular case there ought to have been enough funding. I'm not knowledgeable about its specific developments.
Not generally, no. Even if they had access to the code at the time of review, which is hardly ever the case in my experience, there's simply no time. I also don't see a point in doing so, most papers only describe minute changes to some state of the art thing that's well established so you trust in the academic integrity and experience of authors to get that part right and usually focus on parts that matter over things like the ones this issue brings up.
Even for real reproductions, that require work just like they do in any other scientific domain, the points in that issue seem pretty irrelevant. There's things like FAIR standards on reproducibility but as far as the status quo goes that repo doesn't look too bad. I could not care less if tests for some project are badly written, at least it's in written in a non-obscure language and shows a somewhat sane structure. What's next? Calling for redactions because somebody didn't follow the same tabs vs spaces paradigm?
There's nothing in that issue w.r.t. whatever scientific finding this was used for and this general phenomenon is really fascinating. Instead of a constructive discussion with the technical or scientific folk that put their work out there, or engagement with the politicians that drew conclusions based on those scientific findings, you get github issues and Twitter threads mixing a bunch of unrelated concerns.
I’ve never heard of a reviewer writing 15K lines of code. That would be a serious undertaking for someone who is reviewing as an unpaid community service in their free time outside of doing their own research/teaching/etc.
Maybe a very dedicated reviewer might check a short simulation code that can be written in a few hours max, but even that is exceptionally rare in my experience. Reviewers usually look at the results and validation simulations in the manuscript, and if they look reasonable trust the code doesn’t have major bugs affecting accuracy.
Exactly, the person I replied to was saying there's no need to open source the code, it's enough for the paper to describe it. My point is that unless it's re-implemented and verified (which obviously it isn't) as part of reviewing the paper, that's not the same at all. It's like reviewing 'Assuming X, Y' and saying 'well I can't verify X, but yep, X->Y checks out. Y.'.
> My point is that unless it's re-implemented and verified (which obviously it isn't) as part of reviewing the paper, that's not the same at all.
Even if it is re-implemented and verified it won't be the same. There are a million possible conditions for "re-implementation". Where would you draw the line? How many "re-implementations" and "types of conditions" do you need to be sure?
You probably need to have expectation mismanagement from a scientific paper. A scientific article will only say "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions". That is the crux of ANY empirical science - Computer Science or otherwise. That's all that they are paid (and incentivised) for. A computational scientific experiment is not a software product. If you want more, you need to pour in more funding and give them more resources explicitly for those purposes. Either that, or if you want to be 100% verified - you can choose to read theoretical papers where mathematical proofs are "verification".
> "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions".
Is what I'm calling the assumption (the whole sentence). The implementation is essentially the methodology, and IMO therefore needs to be included.
Note I'm arguing against 'academic, paper-supporting, code does or should not need to be open sourced', I'm not saying it needs to be 'software product' quality, written in the same way, to the same expectations, packaged, or anything like that. Just available for someone to say 'wait a second, you didn't try it under conditions, X,Y,Z, because Y gets negated here', or whatever.
> Of course if somebody does find a [bug] that invalidates the results of the paper given the stated assumptions, that's another matter
And obviously the 'risk' of that happening isn't a reason not to publish it - you don't not publish the paper for risk that peer review reveals an error!
Of all the things wrong with this study, bad code is probably the least (bar actual bugs which mess with results). In fact, it's par on course for code in non-cs academia. There's such a thing as the "physicist code" stereotype.
It's generally not used to push policy affecting billions of people, though.
But as I said in another comment, let's blame whoever listened to that garbage fire of a model, not the model itself. We don't listen to Minecraft modders for structural engineering advice either, and we did, they would not be the ones to blame for collapsing buildings.
The policy is mostly a reaction to what happened in Wuhan, Iran, and Italy. Especially Italy. The mass graves filling up are a pretty good motivator, as it turns out.[1]
Well I would rather have the code already available for transparency and then we can discuss about the quality later than it not to be open-source at all.
I wouldn't expect research scientists to write an elegant and well structured program that ticks all the coding practices boxes at the first time as they are not "software engineers" unless that's their area of research or expertise. They want to present their results and the "code quality" comes secondary to them which can be done later. Surely the Linux source code wasn't cleanly structured in its first open-source release.
However, a quick skim at the source, one may suggest that the authors were writing C++ in a style of a C programmer. Maybe one can run a clang-analyzer on the source to find all sorts of issues, I guess.
This was hardly the only element that UK policymakers had to take into account. By the time this study was published, numbers were already accelerating and most of Europe was already in lockdown.
Hard to know really. I think it's likely the big question at the time was how bad will this get. And they did run and use this simulation. Frankly people who are willing to use this for anything should have no say in anything more important than their own papers.
There is the great (Carl Sagan?) quote "extraordinary claims require extraordinary evidence". If other parts of the puzzle are of the same quality, I want to see heads rolling. The PCR test for example.
You're trying to apply software development techniques to a discipline where testing is usually done in a much different way. Yes, we should all try to be perfect and write the most magnificent code with the most bulletproof testing methods, but reality is different.
How about you write tests that clearly prove that the results from this simulation are absolutely wrong? The code is right there! And you're a "software engineer"! Then start talking about retracting papers.
> a discipline where testing is usually done in a much different way
This reminds me of the horror mainstream code monkeys express when they see the source code for automotive firmware. Thing is automotive people functional test the shit out of everything. And they don't let jr developers refactor proven good code because they don't like the way it looks.
I wrote a paper with co-authors asking for open source code in science (https://www.nature.com/articles/nature10836) but this GitHub issue is stupid. Calling for retraction because you don't like the code smell is dumb. Work on the code and see what its flaws are before asking for retraction.
Yes, I wish the original, apparently C code, had been released but let's fix the bugs in the code.
This is the correct approach. We, the "software engineers", can jump in and help. If it eventually comes out that the results were wrong then the papers should be retracted. At the moment there is no proof of that.
That's the beauty of open-sourcing the code: people can help or verify. Needlessly shitting on other people's work without any proof is just disheartening to me.
I'm reminded of the idiom "Don't throw the baby out with the bathwater". And I agree that software engineers should more actively contribute their expertise here.
With that said, if a research paper's main contribution is a model whose results were evaluated using a code with major bugs then retraction of the paper is only professional.
> if a research paper's main contribution is a model whose results were evaluated using buggy code
There's no proof of that at all in this case. The person complaining didn't find any scientifically valid problem, only that he personally for some not cleanly stated reason doesn't like the tests provided, which is clearly stupid.
Since when should any tests be such that some random guy on github must like them? If the code was used by the experts, and they maybe even used it for years, who says that these experts are in any way obliged to publish all their logs of their use of that code (which doesn't have to even exist in publishable form)?
Even if all that they possibly iteratively did for potentially years could be condensed to some tests, who says that it wouldn't take too long (as in, months, years) for that? It's practically just a matter of good will that the code is published at all, in any form at all.
Not only that. It's an awful precedent to shoot down any research based on the implementation of it. That's more than just a misunderstanding; it's a fundamental misunderstanding of how research and scientific processes work.
The issue claims the tests are broken because they're checking for checksums instead of actual data -- but what they really do is to check for the sanity of the implementation to be actually implementing the mathematical models that are underlying to it.
I had the displeasure of working with this sort of "scientific" code. My experience is that anything this messy is just wrong. It's not about appearance. It is so complex, messy, untestable and twisted that it is in the end incapable of giving the correct answer.
The overall conclusion may be correct, who knows, but based on my experience, I do not believe, at all, any numerical predictions of this code.
If that is indeed true you should write a paper about it, obviously if simply checking the source code for poor formatting is enough to confidently state it produces incorrect results that would be hugely important.
Did you read the code? It’s not formatting or minor anal things like that. There are 10s of variables defined, functions, all without comment, cryptic names updating other cryptic names, stateful global vars updated promiscuously by everyone...
There's some other criticism (which I won't link because the text is partisan, and it's elsewhere on HN already) with some links that point to (potential or perceived) issues:
Both of these issues don't have any connection at all with any claim that the code could produce scientifically wrong results.
The 116 is "if I use it this special way I would expect something else and not what I see" which can be simply answered with "well, don't do THAT."
The simulations by design are not expected to produce exactly the same results in different runs. That's why they are simulations. The reporter tried some "partial save" and expected something else.
The 30, if I understand correctly, is observed on Cray to behave in some minor detail not exactly the same as on the PC (which again doesn't have to even mean that the output is scientifically wrong). Seriously? If one is a Cray user, he can fix it, so what?
The evidence shows that this model did not have good predictive power. For example, the predictions made about Sweden did not come to pass. The predictions made about the US did not come to pass - specifically, 5400 hospitalizations were predicted by by March 19 for NYC, but only 750 actually occurred - off by an order of magnitude before lockdown measures could have any impact.
Did this model provide clarity and insight for the decision making process? Or, did it instead induce a false sense of panic?
Here's some more info as to what this repo does and does not contain, and why what was released is not enough.
These variable names are all what you typically see in mathematics-heavy numerical computation code. Sometimes it’s better to let the index symbols be just symbols rather than assigning long names; otherwise it’s going to be even more difficult to read and understand.
Academic code has always been a mess (from the standpoint of professional software developers). Anyone who has ever worked with code from academia knows this. If you think this code is messy, check out any random Matlab simulation strung together by any random series of grad students who aren't professional developers in any random lab.
People in industry don't always do better, either. Look at the notebook code shared on Github by Kevin Systrom for the rt.live site. It's just as messy. But the point is that he shared it and that helps everyone see what is going on and lets anyone improve on the work.
Publicly sharing model code on Github is a recent development and is amazing for everyone. It gets the work out faster, allows for feedback on methodology and coding style and generally lets everyone get smarter over time. We shouldn't discourage this.
Posting retraction demands in Github issues based on coding style is not helpful and shows a lack of communication skills. What is helpful is providing useful, specific bug reports, submitting improvements, or sharing your own, improved model. If you want to demand a retraction, do it via alternate means.
Remember, this crisis appeared suddenly on the world stage and whoever had a model sitting around got thrust into the forefront of public discussion. But scientists don't make policy. Policy makers make policy. A github issue is not the place to blame someone for a policy you disagree with. That only discourages people from sharing their work which would be a loss for everyone.
Your reading is very charitable. The political debate is a complete gutter, and everyone is ready to jump at absolutely anything that can advance his argument in the slightest.
As you can see in this very thread, jMyles is hardly unbiased on the matter. He's just playing politics, code standards are an excuse.
I worked on an academic open source Java project following my BSc and after years of MSc + industry experience as backend dev, I cringe really hard at what I produced back then.
So hard in fact, that I am about to invest time to fix that mess (because I like the project).
This is such a dangerous precedent. This type of call out will only discourage the release of source code making legitimate peer review more difficult.
It is if you know that result is valid, for example if you have validated it some other way and this is a regression test, or you calculated it by hand.
I specialise in legacy code. I've seen my fair share of abysmal code that works. This is pretty awful, but far from the worst I've ever seen in my career -- even in systems where public safety was critical. I've not looked at this code in any real detail, but I suspect John Carmack (who has doubtless seen a fair amount of complex C code in his life) has it right.
On the plus side, if anyone wants to practice refactoring gnarly C code: here's your opportunity.
The twits point that Carmack personally worked on that very code (to prepare it for the Github release)!
He explicitly writes there:
"it turned out that it fared a lot better going through the gauntlet of code analysis tools I hit it with than a lot of more modern code. There is something to be said for straightforward C code. Bugs were found and fixed, but generally in paths that weren't enabled or hit."
"the performance scaling using OpenMP was already pretty good, and this was not the place for one of my dramatic system refactorings. Mostly, I was just a code janitor for a few weeks, but I was happy to be able to help a little."
and
"I can’t vouch for the actual algorithms, but the software engineering seems fine."
As shown, he even points that the believes "a lot of more modern code" would have been worse than that "straightforward C", which matches my experiences.
This is the only time I've ever actually been embarrassed to be part of this profession. Maybe other cases i can just say it's limited to this other subgroup that doesn't include me...
This github issue sounds a lot like it is politically motivated. He is clearly trying to make the leap from "I don't like the tests in this project" to "public policy is wrong." No actual evidence, just cranky poo-pooing on the project.
Just because unit tests aren't provded in the codebase, it doesn't mean testing hasn't been done. Quite often it's difficult to model certain things in tests, e.g. Linux kernel drivers, embedded devices etc.
However, it seems that the author of the GitHub issue has personal/political problems with the Imperial researchers because of the lockdown: https://pastebin.com/LadbM3E1.
@OP: So let's get this straight, you created that GitHub issue within one hour of seeing the repository, rallied whatever Discord server that is to join your cause and put it on HN for reach?
As an academic and software engineer, please point me to where I can file an issue for you to retract your issue.
The issue is crazy. It basically come down to there is no unit test? Unit test traditionally is not done for numerical code in Fortran or C. We sent people to the moon on code without unit test. We build atomic bombs on code without unit test. It doesn’t mean the code is not tested. Those numerical code are just tested in a different way.
Also remember, all models are wrong, some are useful. Is this particular model useful? Probably.
I suppose it's only fair that if software engineers think they can play epidemiologists, that epidemiologists could think they can play software engineers.
It's worth noting that this issue was posted by a moderator of the /r/LockdownSkepticism/ subreddit to push a particular political agenda, and now this repo is being brigaded by users from said subreddit.
This is politically motivated and just plain dumb. Someone with the intention of finding holes posted it with the assumption that the code must be wrong because it’s build in a way that makes it hard for that individual to find potential errors. That doesn’t mean that there are critical errors and this “issue” should not have been submitted unless there are, but the author is impatient and has a foregone conclusion so the issue was submitted and pushed in social media instead of doing the actual work of looking for critical errors.
And likely there are no critical errors, only minor ones as is typical of heavy simulation code bases.
You don’t do testing via unit and integration tests, you do testing through simulating known systems and building confidence in code correctness over time. And by reviewing the mathematics behind the models, and reviewing that the code matches the math.
That’s harder for outsiders who want to discredit you to dig into that and point fingers at it. But not at all impossible, however in the science community we don’t start pointing fingers until we have actual criticism to back our critique.
83 comments
[ 3.1 ms ] story [ 169 ms ] threadThis study was, of course, the basis for lockdowns around the world.
So find a bug in the implementation which can invalidate one of their claims. Lack of tests, is not a bug.
Its model for China predictions seems to be quite fitting.
[citation needed]
Afaik this was a lone study that might have been relevant only to the UK debate (and very late at that). Italy had already been in full lockdown for weeks before it was published, quite a few other European countries had gone in shutdown or closed borders, and obviously Asian countries had already taken countermeasures for months.
Being so cavalier with the truth is, of course, why intelligent people don't take seriously a lot of anti-science attitudes.
As I said, "around the world" they were already taking countermeasures.
[0] https://metro.co.uk/2020/02/25/towns-italy-lockdown-coronavi...
But I don't really blame Ferguson. If President Macron listens to him, or Alex Jones, or any other crank or random guy with an opinion and a model, instead of his own experts, he is the one to blame. I think the culture of the "safety principle" and excessive caution is also to blame.
https://www.lemonde.fr/planete/article/2020/03/15/coronaviru...
There is also a "control" group of sorts in us Swedes, as the Swedish health authority choose to act differently, initially. Their recommendation as of now is essentially equivalent to quarantine, except for non-adults. It's important to understand that it is really to instigate anything like a curfew in Sweden, at least unless we get attacked military by another country. The laws as they stand don't really allow that. So the government is also somewhat limited in what they can do.
The situation here is that urgent care has been overwhelmed for about a month, and a lot more people have died or been gravely ill than during even really bad flu seasons. We're also probably far from done yet, as it's much easier to get people to stay indoors when it's cold and rainy outside, and as its been getting warmer more people have been disrespecting the recommendations.
I think it's important to understand that we only require information to generate algorithms written in the paper. As such, assumptions and limitations should be listed enough to generate the material in the paper. One way to do so is code and data, it is certainly not the only way.
We don't expect universities to build their own laboratories and halls of residence from first principles - they will call on property developers, architects, and contractors. It would be good if there were more IT services, built by specialists, that academics could tap when required - so they would spend more time advancing their fields of expertise and less time writing mediocre code. Things like PythonAnywhere / Notebooks, Lambda etc are a step forward, but there is still ample space for advancement.
Even for real reproductions, that require work just like they do in any other scientific domain, the points in that issue seem pretty irrelevant. There's things like FAIR standards on reproducibility but as far as the status quo goes that repo doesn't look too bad. I could not care less if tests for some project are badly written, at least it's in written in a non-obscure language and shows a somewhat sane structure. What's next? Calling for redactions because somebody didn't follow the same tabs vs spaces paradigm?
There's nothing in that issue w.r.t. whatever scientific finding this was used for and this general phenomenon is really fascinating. Instead of a constructive discussion with the technical or scientific folk that put their work out there, or engagement with the politicians that drew conclusions based on those scientific findings, you get github issues and Twitter threads mixing a bunch of unrelated concerns.
Maybe a very dedicated reviewer might check a short simulation code that can be written in a few hours max, but even that is exceptionally rare in my experience. Reviewers usually look at the results and validation simulations in the manuscript, and if they look reasonable trust the code doesn’t have major bugs affecting accuracy.
Even if it is re-implemented and verified it won't be the same. There are a million possible conditions for "re-implementation". Where would you draw the line? How many "re-implementations" and "types of conditions" do you need to be sure?
You probably need to have expectation mismanagement from a scientific paper. A scientific article will only say "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions". That is the crux of ANY empirical science - Computer Science or otherwise. That's all that they are paid (and incentivised) for. A computational scientific experiment is not a software product. If you want more, you need to pour in more funding and give them more resources explicitly for those purposes. Either that, or if you want to be 100% verified - you can choose to read theoretical papers where mathematical proofs are "verification".
> "We tried this idea under conditions X,Y,Z and it works with A,B,C metrics and I, J, K assumptions".
Is what I'm calling the assumption (the whole sentence). The implementation is essentially the methodology, and IMO therefore needs to be included.
Note I'm arguing against 'academic, paper-supporting, code does or should not need to be open sourced', I'm not saying it needs to be 'software product' quality, written in the same way, to the same expectations, packaged, or anything like that. Just available for someone to say 'wait a second, you didn't try it under conditions, X,Y,Z, because Y gets negated here', or whatever.
And obviously the 'risk' of that happening isn't a reason not to publish it - you don't not publish the paper for risk that peer review reveals an error!
But as I said in another comment, let's blame whoever listened to that garbage fire of a model, not the model itself. We don't listen to Minecraft modders for structural engineering advice either, and we did, they would not be the ones to blame for collapsing buildings.
[1] https://www.theguardian.com/world/2020/mar/12/coronavirus-ir...
I wouldn't expect research scientists to write an elegant and well structured program that ticks all the coding practices boxes at the first time as they are not "software engineers" unless that's their area of research or expertise. They want to present their results and the "code quality" comes secondary to them which can be done later. Surely the Linux source code wasn't cleanly structured in its first open-source release.
However, a quick skim at the source, one may suggest that the authors were writing C++ in a style of a C programmer. Maybe one can run a clang-analyzer on the source to find all sorts of issues, I guess.
There is the great (Carl Sagan?) quote "extraordinary claims require extraordinary evidence". If other parts of the puzzle are of the same quality, I want to see heads rolling. The PCR test for example.
How about you write tests that clearly prove that the results from this simulation are absolutely wrong? The code is right there! And you're a "software engineer"! Then start talking about retracting papers.
This reminds me of the horror mainstream code monkeys express when they see the source code for automotive firmware. Thing is automotive people functional test the shit out of everything. And they don't let jr developers refactor proven good code because they don't like the way it looks.
Yes, I wish the original, apparently C code, had been released but let's fix the bugs in the code.
That's the beauty of open-sourcing the code: people can help or verify. Needlessly shitting on other people's work without any proof is just disheartening to me.
With that said, if a research paper's main contribution is a model whose results were evaluated using a code with major bugs then retraction of the paper is only professional.
There's no proof of that at all in this case. The person complaining didn't find any scientifically valid problem, only that he personally for some not cleanly stated reason doesn't like the tests provided, which is clearly stupid.
Since when should any tests be such that some random guy on github must like them? If the code was used by the experts, and they maybe even used it for years, who says that these experts are in any way obliged to publish all their logs of their use of that code (which doesn't have to even exist in publishable form)?
Even if all that they possibly iteratively did for potentially years could be condensed to some tests, who says that it wouldn't take too long (as in, months, years) for that? It's practically just a matter of good will that the code is published at all, in any form at all.
The issue claims the tests are broken because they're checking for checksums instead of actual data -- but what they really do is to check for the sanity of the implementation to be actually implementing the mathematical models that are underlying to it.
The overall conclusion may be correct, who knows, but based on my experience, I do not believe, at all, any numerical predictions of this code.
That does lend itself to retracting the paper.
This I take issue with.
Yes, it is a political proclamation without any practical substance, and should be recognized as such.
- https://github.com/mrc-ide/covid-sim/issues/116
- https://github.com/mrc-ide/covid-sim/issues/30
The 116 is "if I use it this special way I would expect something else and not what I see" which can be simply answered with "well, don't do THAT."
The simulations by design are not expected to produce exactly the same results in different runs. That's why they are simulations. The reporter tried some "partial save" and expected something else.
The 30, if I understand correctly, is observed on Cray to behave in some minor detail not exactly the same as on the PC (which again doesn't have to even mean that the output is scientifically wrong). Seriously? If one is a Cray user, he can fix it, so what?
Did this model provide clarity and insight for the decision making process? Or, did it instead induce a false sense of panic?
Here's some more info as to what this repo does and does not contain, and why what was released is not enough.
https://www.aier.org/article/imperial-college-model-applied-...
https://github.com/mrc-ide/covid-sim/issues/168
Looks linked to the awful style, though. When you have 700 LOC functions starting with
it does tend to happen that you reuse a variable that you shouldn't have reusedJust smells like FORTRAN to me.
People in industry don't always do better, either. Look at the notebook code shared on Github by Kevin Systrom for the rt.live site. It's just as messy. But the point is that he shared it and that helps everyone see what is going on and lets anyone improve on the work.
Publicly sharing model code on Github is a recent development and is amazing for everyone. It gets the work out faster, allows for feedback on methodology and coding style and generally lets everyone get smarter over time. We shouldn't discourage this.
Posting retraction demands in Github issues based on coding style is not helpful and shows a lack of communication skills. What is helpful is providing useful, specific bug reports, submitting improvements, or sharing your own, improved model. If you want to demand a retraction, do it via alternate means.
Remember, this crisis appeared suddenly on the world stage and whoever had a model sitting around got thrust into the forefront of public discussion. But scientists don't make policy. Policy makers make policy. A github issue is not the place to blame someone for a policy you disagree with. That only discourages people from sharing their work which would be a loss for everyone.
Your reading is very charitable. The political debate is a complete gutter, and everyone is ready to jump at absolutely anything that can advance his argument in the slightest.
As you can see in this very thread, jMyles is hardly unbiased on the matter. He's just playing politics, code standards are an excuse.
So hard in fact, that I am about to invest time to fix that mess (because I like the project).
lol
I specialise in legacy code. I've seen my fair share of abysmal code that works. This is pretty awful, but far from the worst I've ever seen in my career -- even in systems where public safety was critical. I've not looked at this code in any real detail, but I suspect John Carmack (who has doubtless seen a fair amount of complex C code in his life) has it right.
On the plus side, if anyone wants to practice refactoring gnarly C code: here's your opportunity.
He explicitly writes there:
"it turned out that it fared a lot better going through the gauntlet of code analysis tools I hit it with than a lot of more modern code. There is something to be said for straightforward C code. Bugs were found and fixed, but generally in paths that weren't enabled or hit."
"the performance scaling using OpenMP was already pretty good, and this was not the place for one of my dramatic system refactorings. Mostly, I was just a code janitor for a few weeks, but I was happy to be able to help a little."
and
"I can’t vouch for the actual algorithms, but the software engineering seems fine."
As shown, he even points that the believes "a lot of more modern code" would have been worse than that "straightforward C", which matches my experiences.
However, it seems that the author of the GitHub issue has personal/political problems with the Imperial researchers because of the lockdown: https://pastebin.com/LadbM3E1.
@OP: So let's get this straight, you created that GitHub issue within one hour of seeing the repository, rallied whatever Discord server that is to join your cause and put it on HN for reach?
As an academic and software engineer, please point me to where I can file an issue for you to retract your issue.
Also remember, all models are wrong, some are useful. Is this particular model useful? Probably.
Doesn't this put the code far above most academic code by having some tests?
You don’t do testing via unit and integration tests, you do testing through simulating known systems and building confidence in code correctness over time. And by reviewing the mathematics behind the models, and reviewing that the code matches the math.
That’s harder for outsiders who want to discredit you to dig into that and point fingers at it. But not at all impossible, however in the science community we don’t start pointing fingers until we have actual criticism to back our critique.