The official Python documentation is excellent, and in many ways goes beyond providing just a list of existing modules and what they do. Sometimes if I'm bored I'll actually just pull up documentation for something I'm not 100% familiar with and have a look around, and I almost always find something new and useful. A couple of interesting ones are [0][1], and [2] is a nice starting point for discovering more. Not everyone's cup of tea, but I also found it enjoyable dive into asyncio with the docs.
I'm curious how you would do data oriented programming in a language with no type system and no control over memory layout. And I guess the answer is "you can't, but JITs might exist someday that do it for you"
But you can't wave your hands around and say compiler optimizations will fix performance problems - they can, but they're not magic, and the arrow in the proverbial knee for optimization passes are language semantics that make them impossible to realize (forcing the authors to either abandon the passes, or rely on things like dynamic deoptimization which is not free).
By using only coding patterns that are known to JIT well and lower level primitive types and containers if provided by the language. Maximizing the use of packages written in native code also helps.
The resulting code is even more annoying to write than using a lower level language typed language in the first place, but ecosystem access sometimes makes up for it.
Hopefully tools like mypyc get better, letting well-typed python code with reasonable usage patterns be compiled to reasonably efficient native code.
Last time I used it I was pleased with the performance benefits but it couldn't even compile all files in a module to a single shared library, despite this being mentioned as possible (and recommended) in the docs. Maybe I was doing something wrong, but they don't answer their github issues often, alas.
Any little thing helps though, it's one thing for throwaway scripts to be inefficient, but applications? At a large scale it is a monstrous waste of time and literal energy.
Those are basically contradiction of terms. Orienting the program structure around the data necessarily requires control over memory layout and how it is interpretted.
Unfortunately, the terms "data-oriented design" and "data-oriented programming" refer to two different styles of programming. Data-oriented design is the approach to programming made popular by Mike Acton's CppCon keynote—as you say, it focuses on the layout of objects in memory to make the processing of data take advantage of the underlying hardware.
Data-oriented programming is a style of programming that, as far as I know, originates in the Clojure community. It emphasizes using general data structures (vectors, dictionaries) to store all data and make code more re-usable. It has nothing to do with good cache utilization, pre-fetching, or avoiding branch mispredictions.
It's a shame that two styles of programming which are almost diametrical opposites share such similar names.
From the look of the article, it's discussing data-oriented design, but in Python, and I agree that it's kind of a weird match.
There is no book on data oriented design as far as I know. It would be great if someone could take Mike Actons talk and similar talks and condense the ideas into a book filled with real world examples.
I'm not sure what either you or the parent consider a proper type system, but the difference between Python and (say) Java's type system is night and day.
If you using the Typing module you might be able to get a linter to check things for you, but that's a far cry from what other typed languages have.
It actually isn't saying that. What do you think Python is made of, under the hood? It's C modules.
The argument is not that NumPy is written in C, but that it amortizes the cost of Python overhead over multiple data, rather than incur it on each datum.
> In practice, scientific computing users rely on the NumPy family of libraries e.g. NumPy, SciPy, TensorFlow, PyTorch, CuPy, JAX, etc..
this is a somewhat confusing statement. most of these libraries actually don't rely on numpy. e.g. tensorflow ultimately wraps c++/eigen tensors [0] and numpy enters somewhere higher up in their python integration
> Yet, [the scientists] struggle to move away from Python, because of network effects, and because Python’s beginner-friendliness is appealing to scientists for whom programming is not a first language.
I don't believe it's the whole story.
In my case, during my 13 years in academia, I saw my field going away from C++ and towards python. Not because of network effects (it was the opposite: it was more difficult to not use what everybody was using), or because scientists were not able to program (the entry language of the whole field was C++, and python arrived only because scientists with a deep knowledge of C++ started to themselves switch the core library to be usable with python).
I think something that computer scientists forgot when they consider the subject is that the way computer scientists do software is just not working when you do science.
In science, you use coding as an exploratory tool. You are lucky if 10% of your code ends up being used in your final publication. Because the 90% was only there to understand and to progress towards the proper direction. For this reason, things like declaring variables, which is very important when one makes a professional software, are too costly to be useful when you need to write down a piece of code that you will ever run once to check a small hypothesis, especially when you have another language not requiring it.
Another aspect is that you will present your scientific results to your colleagues, not your code (they are not interested in that), and they will come up with questions or good ideas, all very good for science, but rarely compatible with the way your algorithm was built in the first place, and you will need to shoe-horn it into your code (to test it) without taking 3 weeks. In this case, python flexibility and hackability is very useful.
It's also visible in the popularity of things like Jupyter notebooks (I have to acknowledge it even if I personally don't like working with such tools), which reuse a working approach similar to what was done in mathematica and matlab, that were created with the scientific workflow in mind.
I'm sure python simplicity has played a role. But I have the feeling that some people are totally oblivious on the fact that there may be other reasons.
That is also how "computer scientists" and software engineers work. Our time is just valued a lot higher so we've come up with techniques to make our work more efficient and faster, like structuring our code well using types and function signatures.
The added bonus is you get science that's you know, repeatable. Because the difference between industrial code and prototype code is that it gets run so often there can't be glaring mistakes; it must be repeatable by default. We have different techniques for dealing with these problems, but writing good code is orthogonal to that (I don't think scientists need to be running static analysis, doing deep reviews, and having extensive integration/unit/mock testing throughout their code).
I used to be a software engineer, and now I'm a data scientist. Not exactly the same thing as a real scientist, but I suspect that we have common cause in this area.
One of the hard lessons in the transition was realizing that things that allowed me to work more efficiently when I was a software engineer instead reduce my efficiency in my new career.
You might get a decent analogy of the difference by comparing photographs of the first transistor with pictures of every subsequent transistor. The first transistor's clearly going to be terrible in any production application. But the same characteristics that make it so terrible for practical use were also, to varying degrees, essential to or characteristic of the exploratory process that led to its creation.
It's similar for my R&D code. In order to do my R&D work more efficiently and effectively, I need to do things that would be unholy in production code. This is why there's a separate and essential productionizing step where my output is heavily revamped and possibly even completely rewritten in a different programming language.
re: repeatability, I've discovered that it, too, means something slightly different in a science context than it does in an engineering context.
All good points. I think the network effects are more important than they were in the past, because the network has gotten bigger. Today, new graduates make sure to put Python on their resumes, even if they've barely touched it. A colleague who left for another job thanked me for encouraging him to learn Python. Contrast that with 13 years ago, roughly when I switched to Python. My colleagues thought it was some weird toy, and encouraged me to learn C# instead. The past 13 years have also seen most programmers get over their aversion to open-source tools.
It's true, as mentioned in another comment, that repeatability is important. But at least in experimental science, the repeatability of an experiment is a bigger hurdle than making the same code run twice. I use Python code to control my experiments, and the ability to read my way through a complex workflow is quite valuable.
One thing I like is being able to bodge together a huge blob of data and metadata into a dict that's easy to store and unpack. That encourages me to keep more experimental data and metadata, and use it later.
Python and its libraries have sprawled to the point where it's anything but simple.
It's nice to be able to collect your data and then do analysis on it in one language. Data work is mostly about getting your data in a place where you can work with it. Rarely are you handed a dataset that is shiny and ready for analysis or regression ...
Python also has the benefit of tooling, where you can use tools you're familiar with and still work on most codebases.
I just finished working on an exploratory data science paper that was in the making for the last 5 years. In the beginning, I set the goal to version the whole process in git. It was incredibly difficult to not accumulate an immense mountain of tech debt during this whole time. We had 4 internal review rounds (5 researchers from two universities), then 2 immediate rejections ("not fitting the scope of journal"), which I considered Major Revisions (because you need to significantly adjust the paper to each new journal) and at the final journal now 1 major revision, maybe another one (but I am not sure yet). All in all, there were about 7 Major Revisions, each taking about 3 to 6 months of 8 hour shifts.
The paper now consists of 10 (chained) Jupyter Notebooks and about 16000 lines of code with 1200 git revisions (the notebooks are versioned in Markdown using Jupytext). I cannot account for the fundamental changes that were necessary due to all the external input - this is really very difficult to plan for in the beginning, when you set out for such a project. I hear my IT colleague arguing "you need tests for all of this!" and then I face another day where I need to write 25 new functions in 6 hours, while preparing lectures, workshops, assess master theses etc. - impossible to me, without the flexibility of python. It is at least pretty different to how things work in production at companies, where you have at least some clear goals/rules.
> The paper now consists of 10 Jupyter Notebooks and about 16000 lines of code with 1200 git revisions
I‘ve been there too! And about 1 year after the paper was finally published, fresh starters in my lab couldn’t run the code anymore because of an obscure Pandas error caused by a version change. Then Python 2.7 was shelved and created more problems.
In conclusion the considerable effort I put in to create Jupyter notebooks recreating every single figure from the paper was not worth it.
I created Docker Container Images for every versioned "release" of the paper that we maintain in our Gitlab Registry (a CI builds those images automatically from `.gitlab-ci.yml`, `docker-compose.yml` using `docker:dind`), so you can pull a specific Docker Image for every version of the paper that will definitely have the correct dependencies (because the Jupyter Notebooks were tested with each specific version), including Jupyter itself.
That is awesome and should be the minimum standard, but even just getting the entire infrastructure set up is something almost all scientists are not willing to do.
In my experience, one should be happy if code is version controlled in a proper way. Too often, it isn't. In ML this might be a little different but in my field at least (electrical engineering) this is not the case at all.
Yes, I know and I was pretty stubborn with my initial goals to go through with this.. (probably will also cost my employment, but I learned a lot and it was thus still worth it).
> Another aspect is that you will present your scientific results to your colleagues, not your code (they are not interested in that),
Ah, yes you do not present your code and that's a flaw of science. Because nobody can really reproduce your results without, but it seems modern science gave up on the whole reproducibility thing.
> they are not interested in that
Ah no… they are very much interested in that, but you don't give them out because you want to keep adding little details in the form of new papers, rather than letting other groups compete on the same research. So keeping the code proprietary makes it difficult for other groups to do better than you.
It's all very unscientific and uncooperative really. Let's not pretend it's "science"…
> but it seems modern science gave up on the whole reproducibility thing.
It looks like you don't understand the definition and the point of reproducibility.
If you just click on a button to recompute the results from the same code, this is not "reproducibility". The point of reproducibility is to have other scientists able to confirm your results in a fully independent way. They should be able to do it from scratch, without your code, and reach the same conclusion.
Using the code is therefore against reproducibility: people who use the code are not reproducing the experiment.
It is not restricted to the code. At CERN, the Atlas experiment and the CMS experiment are designed to be able to observe the same phenomenons, in order to bring independent measurements. Because of the notion of independence and reproducibility, while working in the same building, there are strong instructions to not share any non-published details with the members of the other experiment. The reason is that if there is an error in the reasoning, or if we observe a statistical fluctuations, or if there is a bug in the code, when the other experiment comes up with their results, their results will be unbiased.
Additionally, it is good practice to have several team doing their analysis in parallel on the same subject, each using their own code. This is called "cross-check analyses" and is a very important scientific procedure that is able to discover code bugs way more efficiently than unit test or code review.
> but you don't give them out
Why do you say we don't give them out? The code is available to reviewers. It's on the collaboration git repository and anyone in the collaboration can check it out.
They are not interested in that the same way they are not interested in the schematic of the Intel processor that was used in the laptop of the person who run the algorithm.
They are, however, interested in understanding deeply the reasoning and the logic that led to the conclusion, to the point that, if they want, they can rewrite their own code (in any language they want) and reach the same result. They don't see the code (not because they can't, but because they are smart enough to understand what reproducibility means), but they know what the code is doing.
> It's all very unscientific and uncooperative really. Let's not pretend it's "science"…
You clearly have no idea what you are talking about. Therefore, your conclusions are worthless.
> It looks like you don't understand the definition and the point of reproducibility. If you just click on a button to recompute the results from the same code, this is not "reproducibility". The point of reproducibility is to have other scientists able to confirm your results in a fully independent way. They should be able to do it from scratch, without your code, and reach the same conclusion.
I highly disagree with that. In machine learning for instance, it's very important to be able to check authors' claims because otherwise you can't know if they are lying or not, or if they cheated or not ("what do you mean we can't use our test data to improve our training process?")
Plus, you can't improve on what others did if they don't let you access their own code and dataset. "We got better results than previous works it seems, but we can't know for sure because maybe that's because our method is better, maybe that's because our code is better, we will never know." My ML approach is more efficient than yours because you wrote it in python on your laptop and I wrote it in C and spent a lot of time to optimize it, plus I have a faster computer. Yeah sure. That's not science.
What you are talking about is important, but it is not what corresponds to the notion of "reproducibility in science".
Ideally, what we need is being able to 1) reproduce from scratch, 2) rerun the experiment with the code used (which has a lot of pragmatical advantages too).
In fact, I would argue that even the examples you are giving are not "reproducibility". Reproducibility means that if you redo it _from scratch_, you reach the same conclusion. If the conclusion is "this algorithm describe in this paper is 20+-10% better than the algorithm described in this other paper, in those conditions", then, ideally, what you need to do is to read the first paper and follow the explanation to recreate the algorithm, read the second paper and follow the explanation to recreate the algorithm, read the conditions and reproduce them yourself, and then, you check if you get a compatible conclusion.
As for improving on top of others, again, the scientific paper explains the algorithm, just apply the recipe in your personal implementation, and you will have the same result.
Even better: if you don't, you may have found a bug that exists in their implementation, and you have made a scientific progress that you would not have obtained if you would have just cut-and-pasted their code.
But as I've said, the ideal is to have both. Still, the sentence that we reacted on as "not scientific because not compatible with reproducibility" is still valid. It is true that, in practice, the code is not very useful in science (it may be a bit useful, but not as crucial as some may think), and it is true that this does not lead to a reproducibility problem, because the goal of the concept of reproducibility is to guarantee the existence of _independent_ implementations to check if we reach the same conclusion.
Code is little more than a very formal way to describe an algorithm, the difference between a detailed algorithm and a piece of code is very small in many cases. And in the cases where the given algorithm is general and no code is given, reproducibility is at stakes.
But ideally you want both: the description of the procedure to rewrite an equivalent program, and the actual code that gave the results detailed in the article. This is happening more and more frequently, and this is a good thing.
> Code is little more than a very formal way to describe an algorithm, the difference between a detailed algorithm and a piece of code is very small in many cases. And in the cases where the given algorithm is general and no code is given, reproducibility is at stakes.
If you follow discussions about how to comment code properly, you will see that an important thing to understand to comment code properly is the fact that your comment should not describe the code, but explain the code. In science, the explanation is a very very very important part of what the community wants. The fact that this distinction exists for comments is the proof that "just the code" is not enough.
Then, yes, of course, it does not hurt to give the code. I have NEVER said otherwise. But just re-read the discussion: my point is that, in practice and in reality, the EXPLANATION is way more important than the description in scientific research, and, because of that, the scientific community is way more interested in discussing the explanation rather than the description (not that it is 100% useless, the same way that discussing plenty of other things is not 100% useless).
> ... reproducibility is at stakes.
True. But in this case, trying to solve that by giving the code is the worst way of solving reproducibility. If the EXPLANATION is not well done, than the science is not there. If you can't reproduce because you don't have a good explanation, than, the solution is to fix the explanation, not to give an unexplained code. Describing the code will NOT fix that, but will let bad science being used ("well, I have the code. I don't know exactly why they have done like that, but I can run the code, so let's use it")
> But ideally you want both: the description of the procedure to rewrite an equivalent program, and the actual code that gave the results detailed in the article. This is happening more and more frequently, and this is a good thing.
No one so far in this discussion disagrees with that. If I was not trying to strongly give the benefit of the doubt, I would be tempted to conclude from this discussion that this disproportionate reaction to just "they are not interested in the code" is from few persons who are totally oblivious of how science works, but are somehow triggered by the idea that the code is not the most important thing in the scientists eyes.
It just comforts me in the idea that some computer scientists have no idea of what are the tensions when doing science (some computer scientists, not all) and just refuse even to consider that maybe it is simply slightly different.
> It looks like you don't understand the definition and the point of reproducibility. If you just click on a button to recompute the results from the same code, this is not "reproducibility".
Cool. Except they don't give you the source even when the paper is about the source. So if you want to make sure their benchmarks make sense you can't.
How do i know? It has happened to me.
> You clearly have no idea what you are talking about. Therefore, your conclusions are worthless.
I'm sorry by being right I've offended your sensibility to the point that you need to insult me to feel whole again.
Something happened to you. But it is not a "reproducibility problem".
I guarantee you, at CERN, several teams, and several experiments, reproduce the results without sharing the code. The Higgs boson was observed with 5 sigma of statistical significance, by Atlas AND CMS, and they did NOT share any code.
How do I know? It has happened to me.
The problem of reproducibility is not about sharing the code. If the paper does not provide a complete explanation and does not have the code, you will not be able to reproduce it. But if the paper provides a complete explanation and does not have the code, you will be able to reproduce it. Worst, if the paper does not provide a complete explanation and provide the code, you will NOT be able to reproduce the experiment: you will be able to re-run the code, including the part of the code that contains unsound hypotheses, incorrect equations, and ill-advised choices, without having the elements to understand why those things should not be like that. If you have the elements to understand "wait, this code should not do that", then you have the explanation that allows you to rewrite some code from scratch without needing the code in a first place.
> I'm sorry by being right I've offended your sensibility to the point that you need to insult me to feel whole again.
Sorry if I offended you. The fact that you have no idea of what you are talking about is the result of an observation: you truly don't understand the concept of "reproducibility in science". It is not an opinion, it is a demonstrated fact: I have given real life example where reproducibility is achieved despite not having used the code (for example, the reproducibility of the observation of the Higgs boson despite the code not being shared), while you argued that the lack of code is a problem for reproducibility.
The part on the conclusions being worthless is a simple logic consequence: if you base your premises on incorrect understanding, then your conclusion should be discarded.
Sorry if you became so emotional that you were unable to read a simple observation and logic statement without imagining it was a personal attack.
Wow, that's probably the worst response you could have made.
First, the CERN example was a counter-example. If you say "A implies B" and that someone shows an example where A does not imply B, it does not matter if this is just one example, it still demonstrates that your affirmation is bullshit.
Second, you cannot just say "let's just pretend that the rest of the world just works like I want. See, I'm therefore right". If "cern" is just a part of "science", then, it's YOUR JOB to give me the demonstration that the rest of science is different. Until then, your affirmation is the one that is pseudo-science.
Until now, all we have is, on one side, someone who has experience in science and who has provided a mechanism to explain why your affirmation is bullshit + a specific counter-example. And on the other side, someone who has just made some big affirmation with nothing below it. Sorry, but I'm pretty sure that everyone can see that you have absolutely nothing to defend your affirmation that was just motivated by a childish "I'm a computer scientist, so my methods are the best, and I'm offended that someone can say that these methods are great in many cases but not in all, so I will invent reasons to pretend they are somehow wrong"
46 comments
[ 1.9 ms ] story [ 118 ms ] threadEdit: maybe this one on Numpy may be more relevant: https://youtu.be/u2yvNw49AX4
[0] https://docs.python.org/3/howto/descriptor.html [1] https://docs.python.org/3/library/collections.html [2] https://docs.python.org/3/
> >>> multiply_by_two = homogenous_array * 2
will be calculated most of the times using a BLAS library - whichever you are using (https://numpy.org/devdocs/user/building.html)
But you can't wave your hands around and say compiler optimizations will fix performance problems - they can, but they're not magic, and the arrow in the proverbial knee for optimization passes are language semantics that make them impossible to realize (forcing the authors to either abandon the passes, or rely on things like dynamic deoptimization which is not free).
The resulting code is even more annoying to write than using a lower level language typed language in the first place, but ecosystem access sometimes makes up for it.
Hopefully tools like mypyc get better, letting well-typed python code with reasonable usage patterns be compiled to reasonably efficient native code.
Last time I used it I was pleased with the performance benefits but it couldn't even compile all files in a module to a single shared library, despite this being mentioned as possible (and recommended) in the docs. Maybe I was doing something wrong, but they don't answer their github issues often, alas.
Any little thing helps though, it's one thing for throwaway scripts to be inefficient, but applications? At a large scale it is a monstrous waste of time and literal energy.
Data-oriented programming is a style of programming that, as far as I know, originates in the Clojure community. It emphasizes using general data structures (vectors, dictionaries) to store all data and make code more re-usable. It has nothing to do with good cache utilization, pre-fetching, or avoiding branch mispredictions.
It's a shame that two styles of programming which are almost diametrical opposites share such similar names.
From the look of the article, it's discussing data-oriented design, but in Python, and I agree that it's kind of a weird match.
There is this one. It has been posted here a few times.
EDIT: I'd meant to link to the updated one:
https://www.dataorienteddesign.com/dodbook/
I'm not sure what language you're referring to? Neither of those is true of Python.
If you using the Typing module you might be able to get a linter to check things for you, but that's a far cry from what other typed languages have.
Python's type discipline is stronger, but dynamic.
Java's type discipline is static, but weaker.
Both are most certainly typed. An untyped language would be one like Forth or most assembly languages.
Python + C modules = Speed
Nothing new here, move along.
The argument is not that NumPy is written in C, but that it amortizes the cost of Python overhead over multiple data, rather than incur it on each datum.
this is a somewhat confusing statement. most of these libraries actually don't rely on numpy. e.g. tensorflow ultimately wraps c++/eigen tensors [0] and numpy enters somewhere higher up in their python integration
[0] https://github.com/tensorflow/tensorflow/blob/master/tensorf...
> Yet, [the scientists] struggle to move away from Python, because of network effects, and because Python’s beginner-friendliness is appealing to scientists for whom programming is not a first language.
I don't believe it's the whole story.
In my case, during my 13 years in academia, I saw my field going away from C++ and towards python. Not because of network effects (it was the opposite: it was more difficult to not use what everybody was using), or because scientists were not able to program (the entry language of the whole field was C++, and python arrived only because scientists with a deep knowledge of C++ started to themselves switch the core library to be usable with python).
I think something that computer scientists forgot when they consider the subject is that the way computer scientists do software is just not working when you do science.
In science, you use coding as an exploratory tool. You are lucky if 10% of your code ends up being used in your final publication. Because the 90% was only there to understand and to progress towards the proper direction. For this reason, things like declaring variables, which is very important when one makes a professional software, are too costly to be useful when you need to write down a piece of code that you will ever run once to check a small hypothesis, especially when you have another language not requiring it. Another aspect is that you will present your scientific results to your colleagues, not your code (they are not interested in that), and they will come up with questions or good ideas, all very good for science, but rarely compatible with the way your algorithm was built in the first place, and you will need to shoe-horn it into your code (to test it) without taking 3 weeks. In this case, python flexibility and hackability is very useful.
It's also visible in the popularity of things like Jupyter notebooks (I have to acknowledge it even if I personally don't like working with such tools), which reuse a working approach similar to what was done in mathematica and matlab, that were created with the scientific workflow in mind.
I'm sure python simplicity has played a role. But I have the feeling that some people are totally oblivious on the fact that there may be other reasons.
That is out of touch with scientific research. We may change overnight completely the inputs, the core logic and the outputs.
Having to babysit function signatures, manage memory and types throughout these activities is just draining.
The added bonus is you get science that's you know, repeatable. Because the difference between industrial code and prototype code is that it gets run so often there can't be glaring mistakes; it must be repeatable by default. We have different techniques for dealing with these problems, but writing good code is orthogonal to that (I don't think scientists need to be running static analysis, doing deep reviews, and having extensive integration/unit/mock testing throughout their code).
One of the hard lessons in the transition was realizing that things that allowed me to work more efficiently when I was a software engineer instead reduce my efficiency in my new career.
You might get a decent analogy of the difference by comparing photographs of the first transistor with pictures of every subsequent transistor. The first transistor's clearly going to be terrible in any production application. But the same characteristics that make it so terrible for practical use were also, to varying degrees, essential to or characteristic of the exploratory process that led to its creation.
It's similar for my R&D code. In order to do my R&D work more efficiently and effectively, I need to do things that would be unholy in production code. This is why there's a separate and essential productionizing step where my output is heavily revamped and possibly even completely rewritten in a different programming language.
re: repeatability, I've discovered that it, too, means something slightly different in a science context than it does in an engineering context.
What the hell is this?
It's true, as mentioned in another comment, that repeatability is important. But at least in experimental science, the repeatability of an experiment is a bigger hurdle than making the same code run twice. I use Python code to control my experiments, and the ability to read my way through a complex workflow is quite valuable.
One thing I like is being able to bodge together a huge blob of data and metadata into a dict that's easy to store and unpack. That encourages me to keep more experimental data and metadata, and use it later.
Python and its libraries have sprawled to the point where it's anything but simple.
Python also has the benefit of tooling, where you can use tools you're familiar with and still work on most codebases.
I just finished working on an exploratory data science paper that was in the making for the last 5 years. In the beginning, I set the goal to version the whole process in git. It was incredibly difficult to not accumulate an immense mountain of tech debt during this whole time. We had 4 internal review rounds (5 researchers from two universities), then 2 immediate rejections ("not fitting the scope of journal"), which I considered Major Revisions (because you need to significantly adjust the paper to each new journal) and at the final journal now 1 major revision, maybe another one (but I am not sure yet). All in all, there were about 7 Major Revisions, each taking about 3 to 6 months of 8 hour shifts.
The paper now consists of 10 (chained) Jupyter Notebooks and about 16000 lines of code with 1200 git revisions (the notebooks are versioned in Markdown using Jupytext). I cannot account for the fundamental changes that were necessary due to all the external input - this is really very difficult to plan for in the beginning, when you set out for such a project. I hear my IT colleague arguing "you need tests for all of this!" and then I face another day where I need to write 25 new functions in 6 hours, while preparing lectures, workshops, assess master theses etc. - impossible to me, without the flexibility of python. It is at least pretty different to how things work in production at companies, where you have at least some clear goals/rules.
I‘ve been there too! And about 1 year after the paper was finally published, fresh starters in my lab couldn’t run the code anymore because of an obscure Pandas error caused by a version change. Then Python 2.7 was shelved and created more problems.
In conclusion the considerable effort I put in to create Jupyter notebooks recreating every single figure from the paper was not worth it.
In my experience, one should be happy if code is version controlled in a proper way. Too often, it isn't. In ML this might be a little different but in my field at least (electrical engineering) this is not the case at all.
Ah, yes you do not present your code and that's a flaw of science. Because nobody can really reproduce your results without, but it seems modern science gave up on the whole reproducibility thing.
> they are not interested in that
Ah no… they are very much interested in that, but you don't give them out because you want to keep adding little details in the form of new papers, rather than letting other groups compete on the same research. So keeping the code proprietary makes it difficult for other groups to do better than you.
It's all very unscientific and uncooperative really. Let's not pretend it's "science"…
It looks like you don't understand the definition and the point of reproducibility. If you just click on a button to recompute the results from the same code, this is not "reproducibility". The point of reproducibility is to have other scientists able to confirm your results in a fully independent way. They should be able to do it from scratch, without your code, and reach the same conclusion.
Using the code is therefore against reproducibility: people who use the code are not reproducing the experiment.
It is not restricted to the code. At CERN, the Atlas experiment and the CMS experiment are designed to be able to observe the same phenomenons, in order to bring independent measurements. Because of the notion of independence and reproducibility, while working in the same building, there are strong instructions to not share any non-published details with the members of the other experiment. The reason is that if there is an error in the reasoning, or if we observe a statistical fluctuations, or if there is a bug in the code, when the other experiment comes up with their results, their results will be unbiased.
Additionally, it is good practice to have several team doing their analysis in parallel on the same subject, each using their own code. This is called "cross-check analyses" and is a very important scientific procedure that is able to discover code bugs way more efficiently than unit test or code review.
> but you don't give them out
Why do you say we don't give them out? The code is available to reviewers. It's on the collaboration git repository and anyone in the collaboration can check it out.
They are not interested in that the same way they are not interested in the schematic of the Intel processor that was used in the laptop of the person who run the algorithm.
They are, however, interested in understanding deeply the reasoning and the logic that led to the conclusion, to the point that, if they want, they can rewrite their own code (in any language they want) and reach the same result. They don't see the code (not because they can't, but because they are smart enough to understand what reproducibility means), but they know what the code is doing.
> It's all very unscientific and uncooperative really. Let's not pretend it's "science"…
You clearly have no idea what you are talking about. Therefore, your conclusions are worthless.
I highly disagree with that. In machine learning for instance, it's very important to be able to check authors' claims because otherwise you can't know if they are lying or not, or if they cheated or not ("what do you mean we can't use our test data to improve our training process?")
Plus, you can't improve on what others did if they don't let you access their own code and dataset. "We got better results than previous works it seems, but we can't know for sure because maybe that's because our method is better, maybe that's because our code is better, we will never know." My ML approach is more efficient than yours because you wrote it in python on your laptop and I wrote it in C and spent a lot of time to optimize it, plus I have a faster computer. Yeah sure. That's not science.
Ideally, what we need is being able to 1) reproduce from scratch, 2) rerun the experiment with the code used (which has a lot of pragmatical advantages too).
In fact, I would argue that even the examples you are giving are not "reproducibility". Reproducibility means that if you redo it _from scratch_, you reach the same conclusion. If the conclusion is "this algorithm describe in this paper is 20+-10% better than the algorithm described in this other paper, in those conditions", then, ideally, what you need to do is to read the first paper and follow the explanation to recreate the algorithm, read the second paper and follow the explanation to recreate the algorithm, read the conditions and reproduce them yourself, and then, you check if you get a compatible conclusion.
As for improving on top of others, again, the scientific paper explains the algorithm, just apply the recipe in your personal implementation, and you will have the same result. Even better: if you don't, you may have found a bug that exists in their implementation, and you have made a scientific progress that you would not have obtained if you would have just cut-and-pasted their code.
But as I've said, the ideal is to have both. Still, the sentence that we reacted on as "not scientific because not compatible with reproducibility" is still valid. It is true that, in practice, the code is not very useful in science (it may be a bit useful, but not as crucial as some may think), and it is true that this does not lead to a reproducibility problem, because the goal of the concept of reproducibility is to guarantee the existence of _independent_ implementations to check if we reach the same conclusion.
But ideally you want both: the description of the procedure to rewrite an equivalent program, and the actual code that gave the results detailed in the article. This is happening more and more frequently, and this is a good thing.
If you follow discussions about how to comment code properly, you will see that an important thing to understand to comment code properly is the fact that your comment should not describe the code, but explain the code. In science, the explanation is a very very very important part of what the community wants. The fact that this distinction exists for comments is the proof that "just the code" is not enough.
Then, yes, of course, it does not hurt to give the code. I have NEVER said otherwise. But just re-read the discussion: my point is that, in practice and in reality, the EXPLANATION is way more important than the description in scientific research, and, because of that, the scientific community is way more interested in discussing the explanation rather than the description (not that it is 100% useless, the same way that discussing plenty of other things is not 100% useless).
> ... reproducibility is at stakes.
True. But in this case, trying to solve that by giving the code is the worst way of solving reproducibility. If the EXPLANATION is not well done, than the science is not there. If you can't reproduce because you don't have a good explanation, than, the solution is to fix the explanation, not to give an unexplained code. Describing the code will NOT fix that, but will let bad science being used ("well, I have the code. I don't know exactly why they have done like that, but I can run the code, so let's use it")
> But ideally you want both: the description of the procedure to rewrite an equivalent program, and the actual code that gave the results detailed in the article. This is happening more and more frequently, and this is a good thing.
No one so far in this discussion disagrees with that. If I was not trying to strongly give the benefit of the doubt, I would be tempted to conclude from this discussion that this disproportionate reaction to just "they are not interested in the code" is from few persons who are totally oblivious of how science works, but are somehow triggered by the idea that the code is not the most important thing in the scientists eyes.
It just comforts me in the idea that some computer scientists have no idea of what are the tensions when doing science (some computer scientists, not all) and just refuse even to consider that maybe it is simply slightly different.
Cool. Except they don't give you the source even when the paper is about the source. So if you want to make sure their benchmarks make sense you can't.
How do i know? It has happened to me.
> You clearly have no idea what you are talking about. Therefore, your conclusions are worthless.
I'm sorry by being right I've offended your sensibility to the point that you need to insult me to feel whole again.
Something happened to you. But it is not a "reproducibility problem".
I guarantee you, at CERN, several teams, and several experiments, reproduce the results without sharing the code. The Higgs boson was observed with 5 sigma of statistical significance, by Atlas AND CMS, and they did NOT share any code.
How do I know? It has happened to me.
The problem of reproducibility is not about sharing the code. If the paper does not provide a complete explanation and does not have the code, you will not be able to reproduce it. But if the paper provides a complete explanation and does not have the code, you will be able to reproduce it. Worst, if the paper does not provide a complete explanation and provide the code, you will NOT be able to reproduce the experiment: you will be able to re-run the code, including the part of the code that contains unsound hypotheses, incorrect equations, and ill-advised choices, without having the elements to understand why those things should not be like that. If you have the elements to understand "wait, this code should not do that", then you have the explanation that allows you to rewrite some code from scratch without needing the code in a first place.
> I'm sorry by being right I've offended your sensibility to the point that you need to insult me to feel whole again.
Sorry if I offended you. The fact that you have no idea of what you are talking about is the result of an observation: you truly don't understand the concept of "reproducibility in science". It is not an opinion, it is a demonstrated fact: I have given real life example where reproducibility is achieved despite not having used the code (for example, the reproducibility of the observation of the Higgs boson despite the code not being shared), while you argued that the lack of code is a problem for reproducibility.
The part on the conclusions being worthless is a simple logic consequence: if you base your premises on incorrect understanding, then your conclusion should be discarded.
Sorry if you became so emotional that you were unable to read a simple observation and logic statement without imagining it was a personal attack.
Maybe studying very small particles made you forget about the rest of the world?
First, the CERN example was a counter-example. If you say "A implies B" and that someone shows an example where A does not imply B, it does not matter if this is just one example, it still demonstrates that your affirmation is bullshit.
Second, you cannot just say "let's just pretend that the rest of the world just works like I want. See, I'm therefore right". If "cern" is just a part of "science", then, it's YOUR JOB to give me the demonstration that the rest of science is different. Until then, your affirmation is the one that is pseudo-science.
Until now, all we have is, on one side, someone who has experience in science and who has provided a mechanism to explain why your affirmation is bullshit + a specific counter-example. And on the other side, someone who has just made some big affirmation with nothing below it. Sorry, but I'm pretty sure that everyone can see that you have absolutely nothing to defend your affirmation that was just motivated by a childish "I'm a computer scientist, so my methods are the best, and I'm offended that someone can say that these methods are great in many cases but not in all, so I will invent reasons to pretend they are somehow wrong"