The solution is obvious - provide a github repository with all the code for the research. Provide clear ways to reproduce all the results of the research from raw data by running build.
Maybe even a container image / VM to ensure there's no subtle enviroment differences that alter the results. Even better with reproducible builds specifically so you can confirm it was created from the published source.
IMO that's an overkill. It's sufficient that the result is reproduced once. If slightly different environment would produce enormously different results, that's interesting by itself.
The goal is that you can continue research based on previous results, not that you can reproduce exact results in an obtuse VM.
The core of the article is first recommendations on writing scripts to disambiguate cleanup tasks in data and organization/records management of files. It's then recommendations to use open source languages for scripting / analysis and for using git to maintain it all.
The trouble you run into is in the phrase "Provide clear ways to reproduce all the results."
There is "clear ways to," then there is "an exact method to." Academic life is cutthroat and research tenure and continued employment rely on showing that other peers are using their research. It's common practice to write an article in such a way that it looks "clear" but requires consultation with the researcher to understand precisely.
Ambiguity is part and parcel. The tough need here is core journals in each field being more rigorous on the sussing out of not-quite-good-enough methods statements to force researchers to be more open.
But if somebody would read the publication in, say, 20 years, then the whole platform on which the computation has to run might have become obsolete. Suddenly, you have to look for emulators. Or even for emulators running emulators.
You won't succeed at making a "pull request" in 2035 against a piece of code written in 2015.
The person who wrote the code has another job now. The programming language it's written in is unsupported and unsafe to run. The site it's hosted on was sold to a holding company and then taken down. The entire concept of a "pull request" has been replaced by some new idea that's as foreign to us as a pull request would be to developers in 1995.
Right. I look down the road when the drive isn't to migrate to Python 3, but from it, to PyGo 1.4. And damn-it, what is this PoGo thing? We had PIP! PIP worked! Why are this tri-ennials continually reinventing stuff we had completing worked out by the early 20's?!
In my experience, scientists are like extremely junior developers with PhDs.
They name functions "wstok" instead of "whitespace_tokenizer" because it uses fewer keystrokes.
They can't be bothered to commit code. And when you do manage to get them to commit code, it is isn't the same code. It is close to the same code, but different. They think "close enough" is close enough, even though they recognize that without the exact code, it isn't the same thing. "Just a few tweaks will get you from that code to my code!"
This is a very difficult problem for a startup to solve.
EDIT 1 I read about other commenters writing about containers or just provide a github repository. The problem isn't the availability of the tools, the problem is convincing them to use the tools.
EDIT 2 Scientists don't give a damn about reading code, or what happens to the code the moment it leaves their hands. To quote one, "Why would I use a coding style? Why did you read the code? Code is not meant to be read!" This was a serious conversation.
EDIT 3 The most difficult thing is remembering they're not developers who care about developer tools. They're scientists who want to get results to their very exciting experiments.
I don't think that a startup can solve their coding problems. Only scientists can solve their own coding problems by having the diligence and attention to detail that one would expect of a high school dropout with 2 weeks of a coding bootcamp under their belt.
"The code is bad therefore it's not science" is a questionable and even arrogant attitude.
In more cases than not, the software is doing something that was previously done by hand by some grad student or support staff(1). If those people made a mistake, then tough luck. Hopefully you'd catch it with a sanity check on the output. Were those people who relied on manual labor for centuries before the invention of modern computers not doing science? Absurd.
IMO, what's happening is that scientists are doing the same sort of validation that they always did -- placing no trust in the process (the code), and instead looking at checkpoints in the computation process and sanity checking the final output.
None of this is to say that there isn't room for improvement -- there is. But it's possible to do great science while writing shitty code. Hell, it's possible to do great Computer Science while writing shitty code!
(1) The vast majority of scientists programming are not, in fact, doing anything revolutionary with computation. It's just a more efficient way of doing what they used to do by hand. And those that are using code in an essential way typically (not always) have higher quality code.
The startup I work at employees scientists, therefore the scientists are part of the startup, and it is our collective responsibility to create an atmosphere of good science.
Creating that atmosphere, in the face of years of habits and history, is a very difficult problem for a startup to solve.
This is accurate. I am a good example of this phenomenon. Scientists don't know how to code in a way that is intelligible to someone else, and in academic shops, there's probably only going to be one person who writes/understands a program in the first place. This one person probably views coding purely as an icky means to an end, and his adviser thinks it's a waste of time if it even reaches his knowledge whatsoever. As soon as this person moves on, the software he was working on will be rendered nonfunctional.
Part of it is complete ignorance of standards and best practices; another part is a complete lack of documentation (too much time spent on non-science activity to justify making it). It's a problem because scientists can always make use of information technology, but they don't quite have the focus to prioritize it. Honestly, passing it off to non-science devs hasn't worked well in my experience, there's a knowledge gap that's really hard to get over.
I think the core of the problem is that coding and documentation are not considered part of the "science". Contrast it with writing papers, which is understood as communication with scientific peers, the community, and therefore is "science" -- and, of course, is one of the measures of scientific progress. Writing code is not. Anecdotally, writing papers about code is (on occasion), so maybe there is a way to convince the scientific community that the code is important too?
Some Scientists are used to describing something by stripping away all the irrelevant matrix and concisely stating the idea at issue. Maybe they write code that way - as little boilerplate as possible, just a few concise lines that capture the algorithm.
Or maybe they write hash, I don't know. But their approach isn't 'let me craft some good code', its 'let me describe my thesis' which will end up in a different place than any trained software professional.
I think a more fundamental problem is that coding and documentation aren't actually taught in a formalized way to many scientists, so you pick up the habits that worked for you at the time, and just carry those forward.
A little harsh, but yes in general scientists don't write great code or use the right tooling. The reason is simple : the purpose of science is not to write code but to write papers.
That's not _at all_ to say this state of affairs is a good thing, but the practical reality is that scientists are judeged by academic papers, not code.
So, time taken learning how to code is time taken away from learning to write papers. Time taken designing code, testing it, documenting it, is time taken away from doing the research that goes into the paper. And so on.
Furthermore, most scientific code is actually scripting built on underlying frameworks. For example, plugging together a bunch of analytic tools to process some data. Or transforming data from one format to another before running it through a machine learning framework.
There _are_ scientists who write well documented, well designed framework code that is properly stored in a VCS and has release cycles, code review, etc etc. However, it is usually done in 'free time' - such as it is - and with limited resources.
There's also very little incentive to write code that's ultra-transportable. For the most part, I'm using it, for this one specific thing, and that thing alone.
I once refactored some code of mine to make it way more accessible and automated. It was, to be blunt, a complete waste of time.
I think that's a bit unfair to label all scientists like that. There are many of us that care about writing good code and working in version control. You just need to browse github to see many such people. There's not much incentive to actually do this in an academic environment, however, other than improving their own ability to improve and maintain their own codes. A well documented code available for everyone is usually well down the list of most scientist's priorities, for understandable reasons when you must publish papers.
Although I have quite complex programs I have written, which do benefit from software engineering methodologies, most of what I write are single-use scripts to automate tasks or try out ideas. These definitely aren't for public consumption and shouldn't be - they're more extensions of the thinking process than products of the research.
All generalities are unfair to some extent, it's just how generalities work, so I tend to read them as a quantified statement based on the number of people in the group that the person making the statement is likely to have interacted with. In this case, it seems as if the poster works with scientists on a regular basis, and that is his experience with those he has worked with. That being said, I'll pile on some of my own anecdata in just saying that in school I was split between physics and computer science and my initial approaches towards software development were certainly influenced by some of the thought patterns that the poster was expressing. I've come around on a lot of things, but software developers definitely have a different mindset about software because we develop it.
Yes, I'm also aware that I made a general statement about generalities, though I think this one is a mostly correct generality.
I have had a similar experience working with financial engineers and traders turned programmers (I am a programmer). They don't seem to value software the same way as programmers. They seem to view their ideas as the most important aspect of their work, and writing software is only a way of expressing their ideas. The problem with this is that if you don't feel that their are non purely functional requirements worthy of your time, pretty soon your code base will be impossible to work on.
Ive seen some bad stuff, including thousand line methods and production software whose source code is nowhere to be found. But the biggest forehead slap that I think I've encountered is when working on an automated trading system. I had just started working on the system when the original trader who had developed the algo quit. I saw him after work at a bar a couple weeks later and made some comment about how difficult it was to work on that codebase. He said that he made some of the trading logic purposefully difficult to understand so that it would be difficult for someone to read the code, get a job elsewhere, and replicate his work for a different company. Thats an insane way to write software. It assumes that all of the value is in the idea itself (and this algo was hardly rocket science by the way) rather than in a functional piece of software that is easy to modify and debug.
The other problem with it is that they don't appreciate that probably more than 90% of all code (including the code that touches their ideas) is for unsexy reporting purposes only. The domain expertise accounts for probably 1% to 5% of any business's code, and generally it's the easiest code to design, test, and change later on because it follows very tightly with well-worn practices in scientific computing. You might invent a great new trading strategy, but implementing it is still just the same old vector math and efficient algorithms stuff everyone's been doing for decades.
But with the logging, reporting, parameter management, data provenance, data resource management, etc., etc., the actual creative design of the code matters much more, and a failure to make it extensible is way more expensive to fix down the road than tweaking some O(n^2) scientific algorithm.
As an aside, this is one reason that I think algorithmic brain teasers are a really stupid thing for interviews or predicting positive business impact. With brainteasers, you are testing for rote memorization of a set of optimization techniques (data structure implementations) that maybe touches 1%-5% of your easiest-to-optimize-from-textbooks-or-wikipedia code, whereas the other 95% can almost always be assumed to rely on library implementations of these kinds of structures, and the high-level design of components, particularly optimizing it so that the addition or refactoring of features later is not expensive, is the dominant concern for adding actual business value.
It's just another status game like everything else. Programming labor is asserted to be a commodity item to the company, so that managers can justify lower wage, worse working conditions, etc., for programmers than for business domain experts who are supposedly less easily replaced. But the reality is that a reasonably smart programmer can probably pick up the domain expertise to a high level, even to the same level as a Ph.D. researcher, in a short time, whereas for some reason because of some kind of status-based mental block or lack of curiosity, the domain experts seem incapable of picking up legitimate programming skills.
When most of the code is for reporting, designing the reporting system is a huge priority. But when domain expertise carries political status, the importance of system design is often neglected in favor of campaigning for the supposed critical importance of the favored domain area.
Im not sure its a 90-10% split between the importance of infrastructure to business logic code in our case, but I don't think you necessarily need to make the distinction.
Something that I've run into a few times is when an FE will come up with a proof of concept that needs to be built out into something more robust, and it takes longer to build it out than to come up with the concept in the first place. This is often because the original POC is more or less a direct translation from pure math, while the real application needs to be worked into a multithreaded program for performance reasons. That often comes with significant changes to the way the business logic gets executed.
Software isn't merely a concrete implementation of pure mathematics. It is a logical system that bears some resemblance to math, but has a number of details (hierarchal memory model, network latency, etc...) that make it act quite differently. I think you can iterate on an idea faster and with fewer bugs if you treat it as a software problem from the get go rather than working on the idea and the software separately.
For example, I once had a frustrating argument with an FE about why a program he was working on wasn't working correctly. He thought I didn't understand the normal distribution and how the tails never quite reach zero. What he didn't understand is that the computer doesn't care what a normal distribution is supposed to be, if you try to deal with numbers on the order of 1e-100 in a float in c++, it might as well be zero.
>In my experience, scientists are like extremely junior developers with PhDs.
So from your experience, you are assuming every scientist disdains coding and does it poorly, and every programmer is a master at writing clean and working code.
Perhaps a scientist can teach you about the follies of anecdotal evidence.
This is exactly the problem that the Center for Open Science [https://cos.io/] is hoping to solve with the Open Science Framework [https://www.osf.io].
These are the same people behind the large Reproducibility Project that was on the top of HN not too long ago.
I work in electric utility research. Pretty much every paper gives some simulation results. Getting the source? The only option is emailing and hoping the researchers respond, want to share, and have code that runs on something other than laptop of the student that built it. I haven't ever succeeded in getting working source code.
There are many simple improvements that could be done. Just a github repo for each paper would be a monumental step forward. Installation instructions that have been tested and proven to work would be valuable. And then good coding practices, smart reuse of existing packages and integration with other projects would be beyond awesome (and pretty hard).
GNU Guix is starting to be used for reproducible science in the bioinformatics industry. Much better than bundling opaque binary VM/container images or just having some scripts that bitrot.
It's funny how the reaction of most people in this thread is to bash on 'the other sciences' for bad coding practice, while completely ignoring how inadequate their own practices are in creating reproducible programs.
The heart of the problem is described in the article: It's point and click interfaces (and yes, this includes regurgitating out commands into the terminal), which are expected to be followed by the dot, where they could be automated by the machine if the program were ever to be completed to be reproducible.
A big problem is a non-computer scientist is probably working in a lab on some ancient machine running ancient software - he has no control over the machine, and the system-admin is so far behind because his main job is to reproduce the (unreproducible) software written by so called 'computer scientists'. He struggles so much that he has to share his work with thousands of others as a 'package maintainer', and is grateful that so many other package maintainers exists, because without eachother, they would all have absolutely zero chance of reproducing anything.
It's time to stop bashing the coding practices of other people folks, and look in the mirror. We are the friction that causes code in scientific research to be unreproducible - it's not the code itself. If we're to educate non-computer scientists in how to create reproducible research, surely the absolute minimum is that we do so ourselves.
And so far, Nix and Guix are the only two projects (afaik) which are seriously attempting to tackle this. If you call yourself a 'computer scientist', and you regularly write research (which all code is), then start living up to the name and make it reproducible. This means you should be using Nix or Guix, and packaging your software for it. Without such tool to reproduce the software, you're suffering from the same reproducibility problems this article is highlighting about the other sciences.
When I do simulations in my research, I put the code on GitHub. As the article points out, the solutions are there, some people just aren't interested in using them. Practically speaking, replication doesn't happen very often, because there's no profit in it. What we need is an incentive to replicate existing studies. Maybe replicating an existing study should be a requirement for a masters degree, or for certain PhD-level research methods courses.
It's bizarre to me how byzantine the scientific community's grasp of programming is. Variables are named things like "namxaqqs" and "stoggs1_r3". 4,000 line monster functions are rampant. If scientific code were a garden, stuff like this would be the weeds:
Ax_1 += Bx_1 + Cx_1 # Add Bx_1 to Cx_1
Ax_2 += Bx_2 + Cx_2 # Same thing, but for Ax_2
Ax_3 += Bx_3 + Cx_3 # "
Ay_1 += By_1 + Cy_1 # "
...
Code with comparable functionality is copied and pasted instead of being factored into a single function. Everything is tightly-coupled: changing one line of code is like pulling the keystone out of a bridge designed by an oyster chef. If there are any functions, then calling one mutates at least 7 global variables and induces 13 side effects that are more unpredictable than eigenstate selection. File formats are non-standard and consist largely of one giant concatenation of every variable in the program (all converted to strings of course).
Not-invented-here-syndrome is a badge of honor (LAPACK? Bah! I'll write my own Gaussian elimination routine for this matrix with a million entries). Libraries are embraced with the exuberance of a picky eater encountering durian (as a rule of thumb, anything that is open source and has been vetted by thousands of users is probably untrustworthy).
It is considered a waste of time to learn basic CS algorithms — efficiency is merely an implementation detail, so problems that could have been solved with a clever algorithm and an iPhone are instead brute-forced using millions of hours of supercomputer time. Complexity classes are the abstract nonsense of computer science — it's much easier to throw more hours at the problem (so what if it's NP-hard? My algorithm probably converges to the global minimum. Why wouldn't it?)
When garbage-collected languages are used, programs spend 99% of their time allocating and deallocating small quantities of memory in tightly nested inner loops (16 of them, no less). "Inlining" means putting comments inside of the code instead of above it (if there even are any comments). "Cache locality" has something to do with GPS systems. "Hashing", "recursion", and "quicksort" are the names of recently-announced smartphones. And doesn't "SIMD" stand for the Society for Inherited Metabolic Disorders?
/rant
It's a mess. Granted, there are researchers who write very high quality code, but they are few and far between. I think the main problem is that a lot of students/professors who get involved in computational research never had a good CS background. Perhaps they had one "Computing for Engineers" class that taught Matlab or Python, but that isn't nearly sufficient for research-quality code. While I had the advantage of taking up programming as a hobby during childhood, most graduate students have never programmed before in their life. They're expected to learn something like C++ in a week (that's not a hypothetical example).
Many researchers are afraid to publish their code alongside their paper because they know the code is low-quality. And that awareness causes a lot of insecurities. If someone finds a bug in their 2-3 year research project, then their entire conclusion might be invalidated. But I don't think that's the biggest worry (most scientists ultimately want to know the truth about their subject of study). I think the biggest fear is of losing prestige, losing a chance for tenure, or having funding revoked.
To fix this problem, there is a crucial and urgent need for the academic community to reduce the penalty associated with making honest mistakes.
Mistakes are simply part of the research process. Humans are fallible — everyone is going to mess up at some point. And instead of propagating this academic "chilling effect", it would be much better for...
A project just launched yesterday is Depsy [1], from Impact Story. Depsy tries to give credit to the coders who write scientific software that's used in research. One element at play is that it's not worth it from a reputational standpoint to go through the hard work of getting your code ready to release publicly. It's the same problem we all have with open sourcing hacked together side projects (except even relatively shoddy side projects do help build a programmer's reputation). Depsy wouldn't totally solve that problem, but if there was a good way to boost your reputation by contributing public code then you might see changes in the industry. But like all things in academia, you have to start with reputation and work your way to a solution from there.
If ever there was a case for programming as part of computer literacy, surely this must be it? Give children an introduction to some basic version control (also useful for writing in general!) and basic programming and maintainence (eg: give a programming assignment in 2nd grade, have them improve it in 4th grade).
That should leave some scars that can be built on in the phd. program.
(And either make sure they have a full number tower (probably best for "most"), or make sure they know that computers, despite their name, suffer from fundamental dyscalculia -- the ability to manipulate decimal numbers).
I remember some time ago somebody proposed that scientists should supply entire virtual machines with all the relevant setup and data for a better reproducibility.
It was before the dawn of Docker and alike, so now this idea is probably a bit less insane.
https://github.com/Factual/drake from the data science toolbox is useful. "Drake is similar to GNU Make, but designed especially for data workflow management"
Very interesting, and we were talking about the exact same process at an Open Data event this weekend - documenting the data processing work people do so others can reproduce and build on it.
One thing that is not mentioned at all in this as far as I can see is what about personal data? When the experiment handles personal data - a medical trial is the most likely candidate - the details shouldn't be published sometimes, only anonymised aggregate tables. Are there any guidelines for handling that?
I don't disagree that badly-written scientific code exists (both in academia and in the industry), but I want to point out a couple of things for people who have only a software engineering background.
1. Keep in mind that most scientific code for numerical analysis, being based in mathematics, follows the convention for using algebraic symbols, e.g. even in physics, we write
F=m*a
instead of the wordy version, which we overcame a couple of centuries ago. So using shortened variable names [1] comes from that background, and using longer names as in Java-world seems like a regression.
2. Writing code like [2]
Ax_1 += Bx_1 + Cx_1 # Add Bx_1 to Cx_1
Ax_2 += Bx_2 + Cx_2 # Same thing, but for Ax_2
Ax_3 += Bx_3 + Cx_3 # "
Ay_1 += By_1 + Cy_1 # "
is known as loop unrolling [3,4] and is used for optimization.
Anecdote. I suggested that a bright young PhD science student switch from point-click software to scripting software for his analysis. He replied that he would never do that; he was "too old to learn something like that."
If this issue is interesting to you from either the researcher or software engineering side please consider working with, taking classes from, or just supporting
Greg Wilson's Software Carpentry http://software-carpentry.org
or the spin off Data Carpentry.
Full disclosure:
I am not affiliated in any way shape or form.
Just a fan of the intent.
This is an interesting topic but the article is very pie in the sky.
Talking of members of the public using scientific research (In high school even), code reuse to save money etc. is a bit silly.
For starters I'd imagine code, like an arts student theses is copyright the researcher?
As mentioned 10% is getting code working. 90% is getting it working for others. Researchers just don't have time or skills. And it loses most of it's 'possible' value without that 90%
But as a start a simplified git framework would be nice. Git is way to hard for researchers and other easier version controls won't be suggested cause everyone loves git. Not allowing branching for instance in the gui of some sort of easyGit program.
>- The code is a mess - Most code is written by grad students, it's often sloppy scripts that are duck taped together to get the numbers and graphs needed for the paper. I know some students who essentially have to rebuild things from scratch if they need to expand on what they did for their thesis. Releasing this type of code is embarrassing to the dept, advisor, and student.
>- Don't want to support it - Even with disclaimers, there'll inevitably be someone who downloads the code, and emails asking for help/support. Depending on the quality of the code some may even claim that your results are false because they couldn't get it running on their application. I'm all for peer review and agree that code should be available, and further i know that the scenario is probably a rare occurrence, but still some would rather not publish it to avoid it entirely.
>- Simply don't know where or how - granted i'm in the engineering school (not cs) so milage may vary, but you'd be surprised how many students i've talked to that just don't know much about github or repositories (even svn).
>- There's no incentive - students are already overworked, what benefits do they gain from publishing their code? It helps academic research, yes. But most just want to finish their phd and move on to the next thing, anything else just takes time away from them accomplishing that goal.
>- Funding conflicts - part of the code i use was developed by another student who was funded by a particular grant where a deliverable was required at the end. Thus, anything i release cannot include that code, which inevitably breaks the rest. I do not know the particulars of the grant or why it is not allowed to be released, this is just the response i receive when i ask to put my code online.
54 comments
[ 3.1 ms ] story [ 109 ms ] threadBonus points: Hirschware as software license.
The goal is that you can continue research based on previous results, not that you can reproduce exact results in an obtuse VM.
The core of the article is first recommendations on writing scripts to disambiguate cleanup tasks in data and organization/records management of files. It's then recommendations to use open source languages for scripting / analysis and for using git to maintain it all.
The trouble you run into is in the phrase "Provide clear ways to reproduce all the results."
There is "clear ways to," then there is "an exact method to." Academic life is cutthroat and research tenure and continued employment rely on showing that other peers are using their research. It's common practice to write an article in such a way that it looks "clear" but requires consultation with the researcher to understand precisely.
Ambiguity is part and parcel. The tough need here is core journals in each field being more rigorous on the sussing out of not-quite-good-enough methods statements to force researchers to be more open.
Using open formats and tools would likely help.
And this is even an example of a combination which once was considered mainstream.
The person who wrote the code has another job now. The programming language it's written in is unsupported and unsafe to run. The site it's hosted on was sold to a holding company and then taken down. The entire concept of a "pull request" has been replaced by some new idea that's as foreign to us as a pull request would be to developers in 1995.
They name functions "wstok" instead of "whitespace_tokenizer" because it uses fewer keystrokes.
They can't be bothered to commit code. And when you do manage to get them to commit code, it is isn't the same code. It is close to the same code, but different. They think "close enough" is close enough, even though they recognize that without the exact code, it isn't the same thing. "Just a few tweaks will get you from that code to my code!"
This is a very difficult problem for a startup to solve.
EDIT 1 I read about other commenters writing about containers or just provide a github repository. The problem isn't the availability of the tools, the problem is convincing them to use the tools.
EDIT 2 Scientists don't give a damn about reading code, or what happens to the code the moment it leaves their hands. To quote one, "Why would I use a coding style? Why did you read the code? Code is not meant to be read!" This was a serious conversation.
EDIT 3 The most difficult thing is remembering they're not developers who care about developer tools. They're scientists who want to get results to their very exciting experiments.
EDIT 4 (as commented in reply to woah: we ("a startup") want to solve this problem because we employ scientists. See : https://news.ycombinator.com/item?id=10539433
EDIT 5 We've made great progress on working with scientists, and we're hoping to open source some of our tooling and write about what we've learned.
In more cases than not, the software is doing something that was previously done by hand by some grad student or support staff(1). If those people made a mistake, then tough luck. Hopefully you'd catch it with a sanity check on the output. Were those people who relied on manual labor for centuries before the invention of modern computers not doing science? Absurd.
IMO, what's happening is that scientists are doing the same sort of validation that they always did -- placing no trust in the process (the code), and instead looking at checkpoints in the computation process and sanity checking the final output.
None of this is to say that there isn't room for improvement -- there is. But it's possible to do great science while writing shitty code. Hell, it's possible to do great Computer Science while writing shitty code!
(1) The vast majority of scientists programming are not, in fact, doing anything revolutionary with computation. It's just a more efficient way of doing what they used to do by hand. And those that are using code in an essential way typically (not always) have higher quality code.
The startup I work at employees scientists, therefore the scientists are part of the startup, and it is our collective responsibility to create an atmosphere of good science.
Creating that atmosphere, in the face of years of habits and history, is a very difficult problem for a startup to solve.
Part of it is complete ignorance of standards and best practices; another part is a complete lack of documentation (too much time spent on non-science activity to justify making it). It's a problem because scientists can always make use of information technology, but they don't quite have the focus to prioritize it. Honestly, passing it off to non-science devs hasn't worked well in my experience, there's a knowledge gap that's really hard to get over.
Or maybe they write hash, I don't know. But their approach isn't 'let me craft some good code', its 'let me describe my thesis' which will end up in a different place than any trained software professional.
That's not _at all_ to say this state of affairs is a good thing, but the practical reality is that scientists are judeged by academic papers, not code.
So, time taken learning how to code is time taken away from learning to write papers. Time taken designing code, testing it, documenting it, is time taken away from doing the research that goes into the paper. And so on.
Furthermore, most scientific code is actually scripting built on underlying frameworks. For example, plugging together a bunch of analytic tools to process some data. Or transforming data from one format to another before running it through a machine learning framework.
There _are_ scientists who write well documented, well designed framework code that is properly stored in a VCS and has release cycles, code review, etc etc. However, it is usually done in 'free time' - such as it is - and with limited resources.
I once refactored some code of mine to make it way more accessible and automated. It was, to be blunt, a complete waste of time.
Although I have quite complex programs I have written, which do benefit from software engineering methodologies, most of what I write are single-use scripts to automate tasks or try out ideas. These definitely aren't for public consumption and shouldn't be - they're more extensions of the thinking process than products of the research.
Yes, I'm also aware that I made a general statement about generalities, though I think this one is a mostly correct generality.
Ive seen some bad stuff, including thousand line methods and production software whose source code is nowhere to be found. But the biggest forehead slap that I think I've encountered is when working on an automated trading system. I had just started working on the system when the original trader who had developed the algo quit. I saw him after work at a bar a couple weeks later and made some comment about how difficult it was to work on that codebase. He said that he made some of the trading logic purposefully difficult to understand so that it would be difficult for someone to read the code, get a job elsewhere, and replicate his work for a different company. Thats an insane way to write software. It assumes that all of the value is in the idea itself (and this algo was hardly rocket science by the way) rather than in a functional piece of software that is easy to modify and debug.
But with the logging, reporting, parameter management, data provenance, data resource management, etc., etc., the actual creative design of the code matters much more, and a failure to make it extensible is way more expensive to fix down the road than tweaking some O(n^2) scientific algorithm.
As an aside, this is one reason that I think algorithmic brain teasers are a really stupid thing for interviews or predicting positive business impact. With brainteasers, you are testing for rote memorization of a set of optimization techniques (data structure implementations) that maybe touches 1%-5% of your easiest-to-optimize-from-textbooks-or-wikipedia code, whereas the other 95% can almost always be assumed to rely on library implementations of these kinds of structures, and the high-level design of components, particularly optimizing it so that the addition or refactoring of features later is not expensive, is the dominant concern for adding actual business value.
It's just another status game like everything else. Programming labor is asserted to be a commodity item to the company, so that managers can justify lower wage, worse working conditions, etc., for programmers than for business domain experts who are supposedly less easily replaced. But the reality is that a reasonably smart programmer can probably pick up the domain expertise to a high level, even to the same level as a Ph.D. researcher, in a short time, whereas for some reason because of some kind of status-based mental block or lack of curiosity, the domain experts seem incapable of picking up legitimate programming skills.
When most of the code is for reporting, designing the reporting system is a huge priority. But when domain expertise carries political status, the importance of system design is often neglected in favor of campaigning for the supposed critical importance of the favored domain area.
Something that I've run into a few times is when an FE will come up with a proof of concept that needs to be built out into something more robust, and it takes longer to build it out than to come up with the concept in the first place. This is often because the original POC is more or less a direct translation from pure math, while the real application needs to be worked into a multithreaded program for performance reasons. That often comes with significant changes to the way the business logic gets executed.
Software isn't merely a concrete implementation of pure mathematics. It is a logical system that bears some resemblance to math, but has a number of details (hierarchal memory model, network latency, etc...) that make it act quite differently. I think you can iterate on an idea faster and with fewer bugs if you treat it as a software problem from the get go rather than working on the idea and the software separately.
For example, I once had a frustrating argument with an FE about why a program he was working on wasn't working correctly. He thought I didn't understand the normal distribution and how the tails never quite reach zero. What he didn't understand is that the computer doesn't care what a normal distribution is supposed to be, if you try to deal with numbers on the order of 1e-100 in a float in c++, it might as well be zero.
So from your experience, you are assuming every scientist disdains coding and does it poorly, and every programmer is a master at writing clean and working code.
Perhaps a scientist can teach you about the follies of anecdotal evidence.
These are the same people behind the large Reproducibility Project that was on the top of HN not too long ago.
http://www.artifact-eval.org/
There is a separate committee charged with evaluating software (and other artifacts, like data sets), that come along with a paper.
This is a nice start to a process that could be adopted in non-CS fields for the evaluation of statistical results or software that analyzes data.
I work in electric utility research. Pretty much every paper gives some simulation results. Getting the source? The only option is emailing and hoping the researchers respond, want to share, and have code that runs on something other than laptop of the student that built it. I haven't ever succeeded in getting working source code.
There are many simple improvements that could be done. Just a github repo for each paper would be a monumental step forward. Installation instructions that have been tested and proven to work would be valuable. And then good coding practices, smart reuse of existing packages and integration with other projects would be beyond awesome (and pretty hard).
The heart of the problem is described in the article: It's point and click interfaces (and yes, this includes regurgitating out commands into the terminal), which are expected to be followed by the dot, where they could be automated by the machine if the program were ever to be completed to be reproducible.
A big problem is a non-computer scientist is probably working in a lab on some ancient machine running ancient software - he has no control over the machine, and the system-admin is so far behind because his main job is to reproduce the (unreproducible) software written by so called 'computer scientists'. He struggles so much that he has to share his work with thousands of others as a 'package maintainer', and is grateful that so many other package maintainers exists, because without eachother, they would all have absolutely zero chance of reproducing anything.
It's time to stop bashing the coding practices of other people folks, and look in the mirror. We are the friction that causes code in scientific research to be unreproducible - it's not the code itself. If we're to educate non-computer scientists in how to create reproducible research, surely the absolute minimum is that we do so ourselves.
And so far, Nix and Guix are the only two projects (afaik) which are seriously attempting to tackle this. If you call yourself a 'computer scientist', and you regularly write research (which all code is), then start living up to the name and make it reproducible. This means you should be using Nix or Guix, and packaging your software for it. Without such tool to reproduce the software, you're suffering from the same reproducibility problems this article is highlighting about the other sciences.
Not-invented-here-syndrome is a badge of honor (LAPACK? Bah! I'll write my own Gaussian elimination routine for this matrix with a million entries). Libraries are embraced with the exuberance of a picky eater encountering durian (as a rule of thumb, anything that is open source and has been vetted by thousands of users is probably untrustworthy).
It is considered a waste of time to learn basic CS algorithms — efficiency is merely an implementation detail, so problems that could have been solved with a clever algorithm and an iPhone are instead brute-forced using millions of hours of supercomputer time. Complexity classes are the abstract nonsense of computer science — it's much easier to throw more hours at the problem (so what if it's NP-hard? My algorithm probably converges to the global minimum. Why wouldn't it?)
When garbage-collected languages are used, programs spend 99% of their time allocating and deallocating small quantities of memory in tightly nested inner loops (16 of them, no less). "Inlining" means putting comments inside of the code instead of above it (if there even are any comments). "Cache locality" has something to do with GPS systems. "Hashing", "recursion", and "quicksort" are the names of recently-announced smartphones. And doesn't "SIMD" stand for the Society for Inherited Metabolic Disorders?
/rant
It's a mess. Granted, there are researchers who write very high quality code, but they are few and far between. I think the main problem is that a lot of students/professors who get involved in computational research never had a good CS background. Perhaps they had one "Computing for Engineers" class that taught Matlab or Python, but that isn't nearly sufficient for research-quality code. While I had the advantage of taking up programming as a hobby during childhood, most graduate students have never programmed before in their life. They're expected to learn something like C++ in a week (that's not a hypothetical example).
Many researchers are afraid to publish their code alongside their paper because they know the code is low-quality. And that awareness causes a lot of insecurities. If someone finds a bug in their 2-3 year research project, then their entire conclusion might be invalidated. But I don't think that's the biggest worry (most scientists ultimately want to know the truth about their subject of study). I think the biggest fear is of losing prestige, losing a chance for tenure, or having funding revoked.
To fix this problem, there is a crucial and urgent need for the academic community to reduce the penalty associated with making honest mistakes.
Mistakes are simply part of the research process. Humans are fallible — everyone is going to mess up at some point. And instead of propagating this academic "chilling effect", it would be much better for...
[1] http://depsy.org/ and the blog post announcing it: http://blog.impactstory.org/introducing-depsy/
That should leave some scars that can be built on in the phd. program.
(And either make sure they have a full number tower (probably best for "most"), or make sure they know that computers, despite their name, suffer from fundamental dyscalculia -- the ability to manipulate decimal numbers).
It was before the dawn of Docker and alike, so now this idea is probably a bit less insane.
One thing that is not mentioned at all in this as far as I can see is what about personal data? When the experiment handles personal data - a medical trial is the most likely candidate - the details shouldn't be published sometimes, only anonymised aggregate tables. Are there any guidelines for handling that?
[0]http://www.nature.com/news/interactive-notebooks-sharing-the... [1]https://news.ycombinator.com/item?id=8563028
1. Keep in mind that most scientific code for numerical analysis, being based in mathematics, follows the convention for using algebraic symbols, e.g. even in physics, we write
F=m*a
instead of the wordy version, which we overcame a couple of centuries ago. So using shortened variable names [1] comes from that background, and using longer names as in Java-world seems like a regression.
2. Writing code like [2]
is known as loop unrolling [3,4] and is used for optimization.[1] https://news.ycombinator.com/item?id=10539078
[2] https://news.ycombinator.com/item?id=10540446
[3] https://en.wikipedia.org/wiki/Loop_unrolling
[4] http://stackoverflow.com/questions/2349211/when-if-ever-is-l...
I've come across near duplicates of this code and its always "that's how we did it after punch cards."
Full disclosure: I am not affiliated in any way shape or form. Just a fan of the intent.
Talking of members of the public using scientific research (In high school even), code reuse to save money etc. is a bit silly.
For starters I'd imagine code, like an arts student theses is copyright the researcher?
As mentioned 10% is getting code working. 90% is getting it working for others. Researchers just don't have time or skills. And it loses most of it's 'possible' value without that 90%
But as a start a simplified git framework would be nice. Git is way to hard for researchers and other easier version controls won't be suggested cause everyone loves git. Not allowing branching for instance in the gui of some sort of easyGit program.
>- The code is a mess - Most code is written by grad students, it's often sloppy scripts that are duck taped together to get the numbers and graphs needed for the paper. I know some students who essentially have to rebuild things from scratch if they need to expand on what they did for their thesis. Releasing this type of code is embarrassing to the dept, advisor, and student.
>- Don't want to support it - Even with disclaimers, there'll inevitably be someone who downloads the code, and emails asking for help/support. Depending on the quality of the code some may even claim that your results are false because they couldn't get it running on their application. I'm all for peer review and agree that code should be available, and further i know that the scenario is probably a rare occurrence, but still some would rather not publish it to avoid it entirely.
>- Simply don't know where or how - granted i'm in the engineering school (not cs) so milage may vary, but you'd be surprised how many students i've talked to that just don't know much about github or repositories (even svn).
>- There's no incentive - students are already overworked, what benefits do they gain from publishing their code? It helps academic research, yes. But most just want to finish their phd and move on to the next thing, anything else just takes time away from them accomplishing that goal.
>- Funding conflicts - part of the code i use was developed by another student who was funded by a particular grant where a deliverable was required at the end. Thus, anything i release cannot include that code, which inevitably breaks the rest. I do not know the particulars of the grant or why it is not allowed to be released, this is just the response i receive when i ask to put my code online.