HN Explorer

Ask HN: Should I publish my research code?

421 points by jarenmf ↗ HN
I'm looking for advice om whether I should publish my research code? The paper itself is enough to reproduce all the results. However, the implementation can easily take two months of work to get it right.

In my field many scientists tend to not publish the code nor the data. They would mostly write a note that code and data are available upon request.

I can see the pros of publishing the code as it's obviously better for open science and it makes the manuscript more solid and easier for anyone trying to replicate the work.

But on the other hand it's substantially more work to clean and organize the code for publishing, it will increase the surface for nitpicking and criticism (e.g. coding style, etc). Besides, many scientists look at code as a competitive advantage so in this case publishing the code will be removing the competitive advantage.

362 comments

[ 2.7 ms ] story [ 300 ms ] thread
PragmaticPulp
Is your goal to help advance the science and our general knowledge? Publish the code. You don’t even need to clean it up. Just publish. Don’t worry about coding style nitpicks. Having the code and data available actually protects you from claims of fabrication or unseen errors in hidden parts of your research.

On the other hand, if your goal is only to advance your own career and you want to inhibit others from operating in this space any more than necessary to publish (diminish your “competitive advantage”) then I guess you wouldn’t want to publish.

Icko
Not sure if posting the paper only is even the best move. I personally never work with papers with no code published. Just not worth the effort to reproduce them, when I can use the second SOTA for nearly no performance penalty and much less effort.

All the groundbreaking papers in deep learning in the last decade had code published. So if you're aiming for thousands of citations, you need code.

_aavaa_
Can confirm.

I personally look at any paper without code with great suspicion. The reviewers certainly did not try to reproduce your results, and I have no guarantee that a paper without code has enough information for me to reproduce.

I always go for the papers with code provided.

erwincoumans
Agreed, reproducibility helps a lot, and it is very easy to get details wrong when reimplementing. Having the source code is a bit plus.
petschge
As a reviewer I have reproduced results with my own independent plasma simulation code. And I have had a reviewer write in a report about my paper "result X seemed strange, but I ran it with my code and it does it too. I don't know why, but the results is valid". In my opinion that is even better than just rerunning the same code.
YetAnotherNick
> All the groundbreaking papers in deep learning in the last decade had code published. So if you're aiming for thousands of citations, you need code.

I am in this field and I would say less than 10% of the top papers have code published by the author, and those are most of the time another 0.1% improvement in imagenet. All the libraries that you generally use are likely to be recreated by others in this field. Lot of most interesting work's code never come out like alphazero/muzero, GPT-3 etc.

pottertheotter
This is very domain specific. OP said it is not the norm to do publish code in his field. I have a PhD and in my field it is the same. So much so that I can't think of any paper in my field that has code published. Therefore, a paper with no code would not be at a disadvantage.

Personally, it is a pet peeve I have about my field. But there is no incentive for a new researcher to publish code as it decreases barriers to entry. As much as it's nice to say that researching in academia is about progressing science, as a researcher, you are your own startup trying to make it (i.e., get tenure).

NamTaf
Disclaimer: Not an academic, and my whilst undergrad thesis included code it was so broken that when others saw it I had nothing to lose except my pride.

Personally, I would. Open source is a form of peer review, and if you're wanting to stand by your paper as peer-reviewable then I believe the code should be included in that. Generally speaking, I feel more researchers need to open up their code to peer review because generally speaking, research code tends to not have the same robustness against mistakes (through coding convention as well as tests) as professional software development. I shudder to think how many papers have flawed results that no one realises and are just accepted, because no one can spare the effort rebuilding the code from scratch and without any prior reference in order to verify said results.

I don't think you need to clean it up. You're not competing for a coding elegance competition, but rather allowing someone to find bugs if they exist and point it out, just as they would peer reviewing your paper.

More cynically, spaghetti code probably helps as a defense against people ripping off your code, so if you're worried about your competitive advantage then not cleaning it up is a form of security through obscurity :)

Cynox
Just do a super-minimal cleaning and upload to Zenodo or similar, then stick the DOI to the code and input/output files in your paper somewhere. 99% certain your reviewers will not brother to look at your code. 10 years from now someone new looking into the same topic gets a leg up. Don't feel obligated to update, clarify, or even think about the code ever again. If you want to build a community or something, then by all means go for github, but providing code along with your paper should be something automatic and quick, not adding an unwanted burden.
nicec2
Two times I have published my research code - both times I have found many other papers/projects plagiarized my work without giving me any credit. This happens way more than you would think, especially if you are working under less known advisor, and at less known university.

As the other comment said, if you care about "advancing the science", and won't mind stuff like the above happening, then go for it. In my experience, it is not worth it.

vitorsr
> Two times I have published my research code - both times I have found many other papers/projects plagiarized my work without giving me any credit. This happens way more than you would think, especially if you are working under less known advisor, and at less known university.

This has been very much my experience.

jonnycomputer
I wonder how often it is the case that code isn't considered an academic product per se, and so free to use. May have to make it very explicit.
webmaven
> I wonder how often it is the case that code isn't considered an academic product per se, and so free to use.

As an outsider with occasional glimpses into academia, I've observed a bimodal response:

On the one hand, code seems to be seen as more available for reuse in terms of copy-and-paste, particularly in the exploratory phase.

On the other hand, code reuse is somewhat less likely to trigger an acknowledgement and citation.

This unacknowledged copy-and-paste borrowing of code seems in turn to be one influential factor inhibiting the derived code from being shared in turn.

> May have to make it very explicit.

That certainly can't hurt. I've certainly seen that reusable tools aren't cited as often as they are used, which in turn inhibits efforts from being made by academics in the production and refinement of such tools (the citation being the primary currency and reward mechanism for publication in academia).

Explicitly asking for citations (and getting them) helps with that, though the more general issue of "releasing open source software isn't seen as 'publication' for academic status purposes such as being considered for tenure" remains a problem, though it varies considerably by field, institution, and department.

But if given some thought and extra effort, a researcher might actually get more papers (possibly with different sets of collaborators) out of roughly the same body of work, especially if they look more broadly for appropriate conferences and journals.

BTW, again as an outsider with occasional glimpses into academia, it seems to me that there are many vacant niches for cross-disciplinary journals and conferences focused on reusable assets such as datasets and tools necessary for research.

iamcreasy
This in my experience as well. How did you find it out though?
skulumani
I always published all of my code/papers/source for my publications. I never made anything "revolutionary" but I still felt it was important to produce reproducible research, even if relatively insignificant.

This was kind a change for my advisor who was definitely less interested in that aspect of research. I think this is an issue in academia and needs to change.

Also, ultimately if someone wants to copy and publish your work as their own it will be relatively easy to show that and the community as a whole will recognize it.

Also, for me it felt good when another student/researcher was aided by my work.

https://shankarkulumani.com/publications.html

You don't need to clean it up or make the code presentable. Everyone knows it's research grade code. Most important part is that you have the code in a state that you can reuse in the future for another publication.

I've been saved multiple times by being able to easily go back to decade old work and reproduce plots.

tagh
Publishing the code does have some selfish benefits too: better chance of people building on your research (and citing it).
dagw
It is at least as likely that they'll take your code, integrate it into their own research, and never mention you at any point in the process. So you have to be OK with that.
neilv
Can you ask scientists who are very experienced in your field and successful in the career track that you want to be?

Separate from that, is there fairly new chatter in your field about reproducible science, publishing code and data, etc.? If so, what's the current thinking there about how valuable this is to collective science, and how that should affect the sometimes unfortunate conflicts of interest between career and science?

mhh__
Publish the code.

If someone has comments about style ask them to improve it for you.

Worry about maintaining things after someone asks for maintenance, the vast majority of code is never read again.

anon_123g987
One argument against publishing code is that maybe there's an error (or more) in your code, which validates your possibly mistaken theory, and forcing an independent reimplementation by others would uncover this problem.
techwind
That's a good point I'd not considered. I suppose it's ultimately a risk scientists/researchers take to not lead humanity down a broken path however I'm of the firm belief the truth will "out" and transparency almost always trumps opaqueness
dqpb
The fact that there may be an error is an argument in favor of publishing the code.
anon_123g987
Publishing the code can increase the number of people tinkering with it, and possibly debugging it, it's true. But people just going with it, start using it without looking into the details, and blindly trusting the author (i.e. being lazy) sounds pretty realistic, too.
chriskanan
Having a polished public implementation can lead to a massive increase in the number of citations a paper recieves, if it is really a useful system. Some of my papers I think would have received far fewer citations if I had not released the code. Of course, if it is a really niche area with only a handful of researchers, this may not be true.
lumost
The economic incentive of science is for your work to be replicated and cited. Not publishing the code and data means your work is harder to reuse for subsequent studies and will hurt citations.

If it's uncommon to release code then I'd doubt anyone in the peer review will review it.

nomilk
> it will increase the surface for nitpicking and criticism

Anyone who programs publicly (via streaming, blogging, open source) opens themselves up for criticism, and 90% of the time the criticism is extremely helpful (and the more brutally honest, the better).

I recall an Economist magazine author made their code public, and the top comments on here were about how awful the formatting was. The criticism wasn't unwarranted, and although harsh, would have helped the author improve. What wasn't stated in the comments is that by publishing their code, the author already placed themselves ahead of 95% of people in their position who wouldn't have had the courage to do so. In the long run, the author will get a lot better and much more confident (since they are at least more aware of any weaknesses).

I'd weigh up the benefits of constructive (and possibly a little unconstructive) criticism and the resulting steepening of your trajectory against whatever downsides you expect from giving away some of your competitive advantage.

CJefferson
Do you really mean 90% of the criticism is extremely helpful? Or did you mean 90% was useless.

I've published 100,000s of lines of code from my research over 20 years, and I think I've had exactly one useful comment from someone who wasn't a close collaborator I would have been sharing code with anyway.

I still believe research code should be shared, but don't do it because you will get useful feedback.

bagacrap
Interesting. Are the unhelpful comments coming from academics or random peanut gallery folks?
UncleMeat
Peanut gallery, in my experience. The number of people who I've never met before who decide to complain about hardcoded file paths or run a linter and tell me my paper must therefore be garbage is frustratingly high.

This seems to depend on a paper getting a modest amount of media traction. That seems to set off the group of people who want to complain about code online.

CJefferson
peanut gallery are likely to give stupid feedback. Academics are likely to ask for help using my code -- which is nice but doesn't (usually) contribute anything useful to me, and takes up time I could be spending on other things.
Helmut10001
I had the same experiences, but only publishing for 5 years so far. I still try to puplish everything openly, but I do not expect any responses anymore. In none of my papers, the reviewers appeared to have even seen the Jupyter Notebooks I attached as HTML. The papers are cited, some more, some less, but there is no reaction towards the source code. I still don't regret publishing it.
bagacrap
This. Feedback (less loaded term than "criticism") is something you should want. You can obviously ignore tabs vs spaces types of comments but if your code takes 2 months to get right then it probably still has bugs in it after 2 months and it would be a win if others started finding them for you. Also, if the style is really that bad then it could be obscuring bugs that would otherwise be easy to spot (missing braces, etc), and you might find bugs while fixing it up.

ps always use an auto formatter/linter. I can't believe we ever used to live without them. So much time used to be wasted re-wrapping lines manually and we'd still get it wrong.

goodpoint
> 90% of the time the criticism is extremely helpful

Citation needed. I have rarely seen valuable feedback from random visitors from the internet.

analog31
Depends on the climate of the field you're in, and where you're at in your career. There are fields where entire research groups routinely harvest preliminary ideas from graduate student publications, and then finish them and rush to publication before the student realizes what's happened.

I'd say, grad student owes nobody anything until they finish, because they're bearing the greatest risk of losing priority, and the openness of science is being used against them. Nothing lost by waiting until they have their degree in the bag before sharing. Then clean it up and use it as part of your portfolio. Or append it to your thesis. Advancing science after you've secured your career is a fair compromise.

I love open source and open science, but also look back on my own graduate studies, and I chose a topic that was protected by virtue of a large capital investment plus domain knowledge that was not represented by code. Also, my thesis predates widespread use of the Internet. ;-)

nicec2
I can attest to this. I myself am victim of this. My undergraduate thesis was plagiarized by two other papers. Code was 80% the same, they just added some trivial things. No citing of my work at all.

Look at my other comment for more explanation - if you are working under less known advisor, or at less known university, there is a high chance that this will happen if your work is good.

spaniard89277
Write to the journal they published in and call them out.
analog31
This kind of thing is best done after the thesis is in the bag. A student is racing against the clock. Grad study has many kinds of hard failures. At the most extreme, your advisor could up and die. The focus has to be on finishing. That's how you get out.
nicec2
Matter of fact, I did, even with help of my advisor. The journal did not take any action (it is Q1 open access journal), since they come up with all kinds of mental gymnastics why it is not copied (which boiled down it is NOT 100% the same).

That was for the first occurrence. For the 2nd one, we just did not bother because it hurts my advisor's reputation as well. It is not in the interest of journal to admit the mistake once they made it -- they will fight you about it and try to keep their reputation/image up.

cowpig
> There are fields where entire research groups routinely harvest preliminary ideas from graduate student publications, and then finish them and rush to publication before the student realizes what's happened.

Can you provide a source, or example of this? What does the Amazon of academia look like?

analog31
Biology, and synthetic chemistry. Unfortunately all anecdotal. I live near a major research university, have lots of friends who are involved at all levels, and relatives who are even closer to it. It tends to be in areas that require minimal capital investment to pivot into a new study. Also, the student pursuing the original idea is hampered by their own emerging skills. "My student's thesis just got scooped" is something that every professor has experienced or knows about.

My field, physics, much harder. Building my experiment required a bunch of expensive equipment (maybe half a million in today's dollars), gear that I built myself, the technique of operating it, and so forth.

My career, much harder. I work in business. You learn about my ideas when a patent comes out. ;-)

chrisseaton
> it will increase the surface for nitpicking and criticism

You're supposed to welcome criticism and 'nitpicking' as a scientist.

lanstin
Not from untrained randos on the internet. Signal noise ratio and prior of “not a nutjob” have to be high enough to offset the cost of lost focus.
dahart
That’s a bit of a dismissive straw man. The quote was explicitly referring to nitpicking of things unrelated to the research. You intentionally snipped the very next three words “e.g. code style”. Contrary to your implication, it is not a scientist’s job to welcome any and all nitpicking and criticism, which is why there are professional moderated platforms for relevant science critique, as opposed to criticism.
MaxBarraclough
This back-and-forth has happened before on HN. [0]

As I rambled at the time, [1] it seems to me (non-scientist) that publication norms are well behind the times. Researchers shouldn't be in a position to decide how much they graciously deign to disclose for independent review. If the scientific publication process permits researchers to withhold details they fear won't withstand independent review, that means it's failing to do its job.

[0] https://news.ycombinator.com/item?id=24261706

[1] https://news.ycombinator.com/item?id=24264376

jaclaz
As always I may be wrong, but the (admittedly very few) times I find an article/paper based or revolving around code that is interesting/useful for some purposes I read the "code is available on request" (or similar) as the (in-) famous Fermat's Last theorem note: Hanc marginis exiguitas non caperet.

Nowadays margins are large enough and cost nothing or next to nothing, and you don't probably have any other use of your code, so what would be the advantage for you in not publishing it?

What kind of competitive advantage does it give to you? (what many scientists think might be not as relevant as what you think about this "competitive advantage" secifically in your specific case/field)

About "cleaning it", why?

I mean, if as-is it works (but it is "ugly") it still works, what if in the process of "cleaning it" you manage to introduce a bug of some kind?

Unless you plan to also re-test it after the cleaning, I guess it would be better to not clean it at all.

michaelt
> What kind of competitive advantage does it give to you?

For every paper introducing the revolutionary Algorithm X, there are a bunch of follow-up papers like "Algorithm X applied to self-driving cars", "Algorithm X applied to smartphones", "Algorithm X with some tweaks that provide marginal improvements", "Algorithm X but using consumer-grade hardware" and so on.

If every other lab has to spend several months to replicate your first paper, you and your colleagues can spam out the follow-up papers before anyone else can catch up. This makes your publication count go up.

Other means for achieving similar effects include delaying the publication of your code, or releasing undocumented spaghetti-code with missing dependencies and entirely comprised of one-character variable names.

Of course, this stuff comes at a cost: Making it harder for people to use your work makes them less likely to use your work. So it might be better for your citation count to release the code - and in any case, who goes into research hoping their ideas will be ignored?.

jonnycomputer
NDA requires you to share data several months after reporting it. But in many cases, data collection has not even completed by then. Theoretically someone could scoop you by analyzing your data before all data collection is completed (e.g. N=100, instead of N=120). I'd think that would be career suicide if it were found out, but the risk of it happening doesn't exactly provide much of a incentive to make it any easier on them.
del82
> it's substantially more work to clean and organize the code for publishing, it will increase the surface for nitpicking and criticism (e.g. coding style, etc).

Matt Might has a solution for this that I love: Don't clean & organize! Release it under the CRAPL[0], making explicit what everyone understands, viz.:

"Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it into running; and it's not pretty code. Academic code is about 'proof of concept.'"

[0] https://matt.might.net/articles/crapl/

carterschonwald
A thousand times this. A working demonstrator of a useful idea that is honest about its limits is so valuable. Mover over most commercial code is garbage! :)
ur-whale
Please do absolutely publish your code.

If only to help people who simply can't read academic papers because it's not a language their brain is wired to parse, but who can read code and will understand your ideas via that medium.

[EDIT]: to go further, I have - more than once - run research code under a debugger (or printf-instrumented it, whichever) to finally be able to get an actual intuitive grasp of the idea presented in the paper.

Once you stare at the actual variables while the code runs, my experience is it speaks to you in a way no equation ever will.

archontes
I'd like to piggyback and say that increasing the surface for nitpicking and criticism is exactly why OP should release his code. It improves the world's ability to map data to the phenomenon being observed. It becomes auditable.

Certainly don't clean it up unless you're going to repeat the experiment with the cleaned up code.

klyrs
Agreed on both points! As somebody who bridges research and production code, I can typically clean code faster than I can read & understand it. It really helps to have a known-good (or known-"good") example so that I can verify that my cleanup isn't making undesired changes.

And, yeah. I've found some significant mistakes in research code -- authors have always been grateful that I saved them from the public embarrassment.

galgalesh
I do this with my code and can highly recommend it.

Supplying bad code is a lot more valuable than supplying no code.

Also in my experience, reviewers won't actually review your code, even though they like it a lot when you supply it.

armchairhacker
Anecdotally most of the research papers I see and have worked on publish their code but don't really clean it up. Even papers by big companies like Microsoft Research. Still significantly better than not publishing the code at all.
LandR
> "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it into running; and it's not pretty code. Academic code is about 'proof of concept.'"

What do you know, it turns out the professional software developers I work with are actually scientists and academics!!

paulluuk
They don't call it "Computer Science" for nothing ;)
actually_a_dog
Maybe a little OT, but, I'd rather it be called "computing science." Computers are just the tool. I believe it was Dijkstra who famously objected to it being called "computer science," because they don't call astronomy "telescope science," or something to that effect.
TremendousJudge
I never thought about this, but it's right. However, to get more nitpicky, most of the uses of "Comput[er|ing] Science" should be replaced with "Computer Engineering" anyway. If you are building something for a company, you are doing engineering, not science research
mcguire
Be careful there. If you start calling what you're doing 'engineering', people will want you to know statics, dynamics, and thermodynamics.
andrewzah
The average developer isn't often doing "engineering". Until we have actual standards and a certification process, "engineer[ing]" doesn't mean anything.

The average software developer doesn't even know much math.

Right now, "software engineer" basically means "has a computer, -perhaps- knows a little bit about what goes on under the hood".

TremendousJudge
I'm not talking about the "average developer", I'm talking about college graduates having a "Computer Science" degreee but in practice being "Computer engineers"
andrewzah
College degrees aren't standardized and most of the time don't really mean anything. Ask some TAs for computer science courses about how confident they are in undergrads' ability to code.

There isn't a standard license that show that someone is proficient in security, or accessibility, or even in how computer hardware or networking work at a basic level.

So all we're doing is diluting the term "engineer", so as to not mean anything.

The only thing the term "software engineer" practically means is: they have a computer. It's meaningless, just a vanity title meant to sound better than "developer".

actually_a_dog
> The average software developer doesn't even know much math.

Well, I know stupid amounts of math compared to the average developer I've encountered, since I studied math in grad school. Other than basic graph traversal, I only remember one or two times I've gotten to actually use much of it.

tjs8rj
Engineering is something like “deliberately configuring a physical or technological process to achieve some intended effect”. That applies whether you’re building a bridge or writing fizzbuzz
bhelkey
All my job titles have been of the form Software Engineer plus a modifier.

I believe they are referring to what the degree currently known as Computer Science should be called.

actually_a_dog
Yes. Specifically, it's referring to what the academic discipline ought to be called.

Incidentally, I don't think I know any SWEs who actually majored in software engineering. I know one who didn't even bother to graduate high school and therefore has no academic credential whatsoever, a couple of music majors, a few math majors, and a lot of "computer science" majors, but I can't think of anyone who actually got a "software engineering" bachelor's degree. Hell, I even know one guy with a JD. I think I know 1 or 2 who have master's degrees in "software engineering," but that's it.

actually_a_dog
I agree. I can't find it right now, but there was an article on HN within the past few days talking about how software engineering is often more like writing and editing a book than engineering. That makes perfect sense to me. Code should be meant primarily to communicate with people, and only incidentally to give instructions to computers. It seems to be lost to the sands of time who said this first, but it is certainly true that code is read by humans many more times than it is written. Therefore, ceteris paribus, we should optimize for readability.

Readable code is often simple code, as well. This also has practical benefits. Kernighan's law[0] states:

> Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it.

Dijkstra says it this way:

> The competent programmer is fully aware of the strictly limited size of his own skull; therefore he approaches the programming task in full humility, and among other things he avoids clever tricks like the plague.[1]

Knuth has a well-documented preference for "re-editable" code over reusable code. Being "re-editable" implies that the code must be readable and understandable (i.e. communicate its intention well to any future editors), if not simple.

I know that I have sometimes had difficulty debugging overly clever code. I am not inclined to disagree with the giants on this one.

---

[0]: https://www.defprogramming.com/q/188353f216dd/

[1]: https://www.cs.utexas.edu/users/EWD/transcriptions/EWD03xx/E...

erk__
Peter Naur agreed with that which is why it is called "Datalogi" in Denmark and Sweden, his English term Datalogy never really caught on though. He wrote it in a letter to the editor in 1966 and it has pretty much been used since then here as Naur founded the first Datalogy university faculty a few years later.

This also lead to a strange naming now we have data science as well where it is called "Data videnskab" which is just a literal translation of the English term.

[0]: https://dl.acm.org/doi/10.1145/365719.366510 (sadly behind a wall)

asddubs
informatics may be the closest english analogue. incidentally also what computer science is called in german (informatik)
Sebb767
> incidentally also what computer science is called in german (informatik)

Computerwissenschaften (literally computer science) exists, too, but it's the less common word.

stuartd
> his English term Datalogy never really caught on though

Perhaps because no one had any idea about how it would be pronounced?

hezag
In Portuguese it's called "Ciência da Computação" (computing science or science of computation).
peoplefromibiza
notoriously Philip Wadler says that computer science has two problems: computer and science.

It's not about computers and "You don't put science on your name if you're a real science"

He prefers the name informatics.

source: https://youtube.com/watch?v=IOiZatlZtGU

dmlorenzetti
You don't put science on your name if you're a real science

Having flashbacks to when a close friend was getting an MS in Political Science, and spent the first semester in a class devoted to whether or not political science is a science.

wisemang
Kinda like countries that feel the need to put “democratic” in their name.
viraptor
Also applies to newspapers which include some version of "truth" in the title. (Russian propaganda "Pravda", Polish tabloid "Fakt", etc.)
rsj_hn
An economist friend of mine told me that once, so this isn't just a CS quip.
Frost1x
"Information science" is basically long form of "informatics" so that breaks it I'd say. Also, "information" tends to imply a focus on state and places computational aspects (operations performed on information) as second hand.

I've yet to find a classification I really like but this is an interesting take. I still tend to like CIS (Computing and Information Sciences). The problem with CS is it focuses on computation and puts state as second class. The problem with IS is it focuses on state and puts computing as second class. To me, both are equally important.

peoplefromibiza
TBF informatics is "the study of computational systems"

I have studied informatics, but don't call myself an informatic, because I am not doing any research.

I call myself a programmer, because I am not doing engineering either.

kragen
Informática is a common term in Spanish, as you probably know.
anamax
Materials Science is about as sciency as you can get.
goodpoint
This is not what licenses are for!! They are not statements about the quality of your work or anything similar.

Use standard and well understood licenses e.g. GPL for code and CC for documentation. The world does not need more license fragmentation.

ur-whale
> This is not what licenses are for!!

You must be fun at parties :)

jll29
For instance, the Java license explicitly forbids the use in/for real-time critical systems, and such limitations are good to stress in a license so that they may reach legal force, also to protect the author(s).

Incidentally, I've seen people violate the Java "no realtime" clause.

jerven
Used to, OpenJDK is licensened under GPLv2 with the classpath excemption that allows this for years. If not running an OpenJDK build it depends on your vendor license.
goodpoint
And it makes the license non-opensource.

Plus, the usual "no warranty" is strong enough to protect the authors anyways.

skrebbel
This has explicit usage limitations that matter in science land, which is very much the kind of thing that belongs in a license.

Eg:

   You are permitted to use the Program to validate scientific claims
   submitted for peer review, under the condition that You keep
   modifications to the Program confidential until those claims have
   been published.
Moreover, sure, lots of the license is text that isn't common in legal documents, but there's no rule that says legal text can't be quirky, funny or superfluous. It's just most practical to keep it dry.

In this particular case, however, there's very little risk of actual law suits happening. There is some, but the real goal of the license is not to protect anyone's ass in court (except for the obvious "no warranty" part at the end), but to clearly communicate intent. Don't forget that this is something GPL and MIT also do besides their obvious "will likely stand up in court" qualities. In fact I think that communicating intent is the key goal of GPL and MIT, and also the key goal of CRAPL.

From this perspective, IMO the only problem in this license is

    By reading this sentence, You have agreed to the terms and
    conditions of this License.
This line makes me sad because it makes a mockery of what's otherwise a pretty decent piece of communication. Obviously nobody can agree to anything just by reading a sentence in it. It should say that by using the source code in any way, you must agree to the license.
goodpoint
> clearly communicate intent

Again, this is not how a license work. You can express your intents, ideas and desires in a README file and in many other ways.

The license is nothing more than a contract that provides rights to the recipient under certain conditions. Standing up in court is its real power and only purpose.

That's why we should prefer licenses that stood up in court and have been written by lawyers rather than developers or scientists.

skrebbel
I strongly disagree. Contracts very much primarily communicate intent, ideally in such a way that they also stand up in court. People regularly argue over details in contracts, people regularly look up things in contracts, also when there is no court to be seen and no intention anywhere to go to court. The vast vast vast majority of contracts never make it to court.

Plenty of contracts aren't even written down. When you buy a loaf of bread at the bakery, you make an oral contract about that transaction.

The idea that contracts, or licenses, need to be written in dull legalese and be pretty much impenetrable to be useful or "valid" or whatever, is absolutely bonkers. Lawyers like you to think that but it's not true. It's an urban legend.

If you need to make sure that you can defend your rights in court, then sure, you're probably going to need some legalese (but even then there's little harm in also including some intent - it's just not very common). Clearly that's not the goal here. No scientist is gonna sue another scientist who asked for support and got angry about not getting any even though the code was CRAPL licensed.

goodpoint
> Plenty of contracts aren't even written down.

That's a well known fact. And it's besides the point.

> Lawyers like you to think that but it's not true.

Is that a conspiracy theory? Writing long, detailed contracts on a persistent medium is safer: it lowers the risk of he-said-she-said scenarios and ambiguities.

That is meant to save you tons of legal expenses.

> No scientist is gonna sue another scientist

Then there is no need for such license in the first place. Just a readme file.

aqfamnzc
By existing, you have agreed to the terms and conditions.
npteljes
I agree. There's a lot of confusion surrounding even the most established ones, so there's no need to further muddy the situation with newer licenses. In my opinion a "fun" license, with its untested legal ambiguity, restricts usage more than a well established license with a similar level of freedoms.
zitterbewegung
I think this could be done much better by putting a very restrictive license like GPLv3 / AGPL and then in the README putting in that I don't support this project at all and ignore everything associated with wherever you are hosting it.

Using this license would actually make me suspect that your results aren't even valid and I don't trust many experiments that don't release source code.

HenrikB
In case OP, and others don't know, it is the copyright holders that can decide on the license. Copyright holders are the persons who contributed to the code. In this case, it sounds like OP is the sole author and therefore the sole copyright holder.

You cannot change the past, but as a copyright holder, you can always set a new license for future releases.

Thus, OP, if you're uncertain, I definitely was when I started out, go with a restrive license as recommended here (GPL). That, together with publishing the code online (e.g. GitHub, Gitlab, ...) as well as a with your article, will give you some protection against plagiarism. Anyone who use include parts of your code for their research code, will have to share theirs code the same. If you later on feel like you want to relax the license, you can always change it to, say, MIT.

bee_rider
I like it so far, other than

4) You recognize that any request for support for the Program will be discarded with extreme prejudice.

I think that should be a "may" rather than a "will." If I find out someone is using my obscure academic code, and they ask for help, I'd be pretty pumped to help them (on easy requests at least).

inanutshellus
The point of the license is to set your expectations as low as possible. Then, when you actually /do/ get support, you'll be ecstatic rather than non-plussed.
leksak
When phrased like this,

> 4) You recognize that any request for support for the Program will be discarded with extreme prejudice.

There is no way I'd even make a request for support.

bee_rider
Exactly. If nothing else, a request for support has a chance of being an indication that there's somebody else in the field that cares about some aspect of the problem. I might not act on it, but it is good to have some other human-generated signal that says "look over there."
kragen
Discarding a request for support with extreme prejudice might entail using LinkedIn to look up the boss of the person who asked you for support, then phoning them up to complain about the request for support, or it might entail filing for a restraining order against the person requestings support. The point of this clause is to intimidate people out of making the request in the first place.
bee_rider
That's a pretty over the top reading.
kragen
You should probably familiarize yourself with the meaning of the phrases it's alluding to, "dismissed with extreme prejudice" and "terminated with extreme prejudice".
jandrese
Yes, if you feel you have to make it "release ready" then you'll never publish it. I'm pretty sure a good majority of the code is never released because the original author is ashamed of it, but they shouldn't be. Everybody is in the same boat.

The only thing I would add is a description of the build environment and an example of how to use it.

rectang
The CRAPL is a "crayon license" — a license drafted by an amateur that is unclear and so will prevent downstream use by anyone for whom legal rigor of licensing is important.

https://matt.might.net/articles/crapl/

> I'm not a lawyer, so I doubt I've written the CRAPL in such a way that it would hold up well in court.

Please do release your code, but please use a standard open source license. As for which one, look to what your peers use.

vngzs
Please don't use this license. Copy the language from the preamble and put it in your README if you'd like, but the permissions granted are so severely restricted as to make this license essentially useless for anything besides "validation of scientific claims." It's not an open-source license - if someone wished to include code from CRAPL in their MIT-licensed program, the license does not grant them the permission to do so. Nor does it grant permission for "general use" - the software might not be legally "suitable for any purpose," but it should be the user's choice to decide if they want to use the software for something that isn't validation of scientific claims.

I am not a lawyer, just a hardcore open-source advocate in a former life.

gzer0
In the research industry, it is well established for anyone wanting to publish or utilize / include another's research in their own, to contact the source author and receive explicit permission to do so.

More often than not, they are more than willing to help.

blagie
"Well-established" norms are a barrier to newcomers and perpetuate power structures (including racial, ethnic, national, and socio-economic).

That's not theoretical; I know many people who were or would be embarrassed to ask.

Explicitly posting things is helpful.

planb
I‘m a proponent of MIT and BSD style licenses normally, but this calls for something like AGPL: Allow other researchers and engineers to improve upon your code and build amazing things with it. If someone wants to use your work to earn money, let them understand and reimplement the algorithms and concepts, that’s fine too.
kragen
That's probably not viable under US copyright law, especially with the Bright Tunes Music v. Harrisongs Music precedent; if someone is going to reimplement the algorithms and concepts without a copyright license, they're better off not reading your code so they don't have to prove in court, to a computer-illiterate jury, that the aspects their code had in common with your code were really purely functional and not creative.
danuker
> the permissions granted are so severely restricted as to make this license essentially useless

Indeed, also there're things like "By reading this sentence, You have agreed to the terms and conditions of this License.". That can't hold up in court! How can I know in advance what the rest of the conditions say before agreeing to them?

Then again, I am not a lawyer either.

vimax
While I whole heatedly agree with you, I would seriously question anyone trying to reuse research code in production without completely reimplementing it from scratch.
LamaOfRuin
From the post: "The CRAPL says nothing about copyright ownership or permission to commercialize. You'll have to attach another license if you want to classically open source your software."

It is explicitly the point of the license that the code is not for those purposes, because it's shitty code that should not be reused in any real code base.

kragen
That's not a good excuse for putting your readers at legal risk of copyright infringement. A real, non-shitty code base could easily be a "derivative work" of the shitty code.
caconym_
Yeah, this seems a bit verbose and overbearing to me. The open-source licenses I've used myself include something like this, which seems quite sufficient:

> THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

From a non-academic's point of view I might include a brief disclaimer in the README too, explaining the context for the code and why it's in the state it's in, but there's no obligation to hold the user's hand. To be frank, anybody nitpicking or criticizing code released under these circumstances with the expectation that the author do something about it can go fuck themselves.

Competitive advantage, on the other hand, is a perfectly valid reason to hold code back. There may also be some cost in academia to opening an attack surface for any sort of criticism, even irrelevant criticism made in obvious bad faith. Based on what I've heard about academia, this wouldn't surprise me.

PeterisP
It looks quite explicitly designed as a short-term temporary license for the period when the main paper is unpublished and you'd be expected to keep the code non-public (due to e.g. reviewing anonymity requirements), so the basic open source freedoms are explicitly not included.

I would expect that anyone wanting to actually publish their code should publish the code with a "proper" license after the reviewing process is done and the relevant paper is published.

bmitc
> Academic code is about "proof of concept."

Why does he think that but presumably not the same about the paper itself and the “equations”, plots, etc. contained within?

It’s really not that hard to write pretty good code for prototypes. In fact, I can only assume that he and other professors never allowed or encouraged “proof of concept” code to be submitted as course homework or projects.

cycomanic
I think you don't understand the realities of work in scientific/academic organisations. Unless you work in computer science you likely never received any formal education on programming except for some numerical methods in c/matlab/fortran course during your studies (which often also focus more on the numerical methods and not the programming). So pretty much every person, just learned by doing.

Moreover you are not being paid for writing reasonable programs you're paid for doing science. Nobody would submit "prototype" papers, because they are the currency of academic work. There is lots of time spend on polishing a paper before submission, but doing that for code is generally not appreciated because nobody will see this on your CV.

bmitc
I understand it fine. Like I said, it’s really not that hard to write pretty good code for prototypes. I'm not saying the code needs to be architected for scale or performance or whatever else needless expectation. I don't have a formal education in programming or computer science and write clean code just fine, as do some other non-computer science people I've worked with in advanced R&D contexts. And then some (many?) don't. It's not really about educational background, it's more about just being organized. Even when someone is "just" doing science, a lot of times, the code explicitly defines what's going on and has major impacts on the results. (Not to mention that plenty of people with computer science backgrounds write horrible code.)

If code is a large part of your scientific work, then it's just as important as someone who does optics keeping their optics table organized and designed well. If one is embarrassed by that, then too bad. Embarrassment is how we learn.

Lastly, you're describing problems with the academic world as if they are excuses. They're reasons but most people know the academic world is not perfect, especially with what is prioritized and essentially gamified.

cycomanic
I'm not making excuses I'm just talking about the realities. For the significant majority of researchers I know about version control is still program_v1.py, program_v1.py, program_final.py, program_final2.py (and this is the good version, at least they are using python), so talking about clean code is still quite a bit away. I'm teaching PhD students and it's even hard to convince them, because they just look at how to get the next publishable result. For academics it becomes even more unrealistic, most don't even have time to go to a lab, they are just writing grants typically.

I'm actually a strong supporter of open science, release my code OSS (and actually clean it up) and data (when possible). But just saying it's easy and there is no cost, is just setting the wrong expectations. Generally unless you enjoy it, spending significant time on your code is not smart for your career. Hopefully things are changing, but it is a slow process.

Funny that you talk about optics I know some of the most productive people in my field and their optical tables are an absolute mess (as is their code btw). They don't work at universities though.

bmitc
I still think it’s really not that hard, and actually, it’s really not even about code. It’s really just about organization, because as you point out, not everyone is great at it. But for example, messy optics tables, labs, or whatever do in fact cause problems, like efficiency and knowing what “state” of supporting tools yielded what result and several other derivative problems. I think my push would be just applying even a modicum of organization on supposedly ancillary things will go a long way rather than accepting them as reality.

I understand the realities and have even been a part of PowerPoint development where slides are emailed back and forth. Sometimes one just has to go with things like that. But I have also seen the reality of stubbornness. I have tried introducing source code control to scientists or even stuff like wikis, all supported and already up and running by IT and used by other groups. Scientists and engineers, especially those with PhDs, can a bit rejective and set in their ways. I have been told flat out by different people that they wouldn’t use tools like a wiki or that Python and C was all they ever needed. I have even noticed physicists saying “codes” for software instead of “code”. It’s fairly rampant, and I have seen it in research papers, published books, and in industry in person. I have never seen that use of “codes” anywhere else. That alone is evidence of a certain amount of culture and institutionalization of doing things incorrectly but viewed as acceptable within the field.

I have written code in some research contexts. I get the constraints. One just needs to take it seriously. But organization, in general, often takes a back seat in basically any field. The only way to change things like this are like anything, which is to push against culture.

kube-system
MIT and BSD are established and well accepted licenses, literally named after the academic institutions where they originated. Licenses are legal documents, part of what makes them "explicit what everyone understands" is their legally recognized establishment.

If you want to set expectations, this can simply be done in a README. Putting this in a license makes no sense. Copyright licenses grant exceptions to copyright law. If you're adding something else to it, you're muddying the water, not making it better.

woah
I'm not a scientist so maybe I don't get it, but it seems like code could be analogized to a chemist's laboratory (roughly). If a chemist published pictures of their lab setup in their paper, and it turned out that they were using dirty glassware, paper cups instead of beakers, duct taping joints, etc etc, wouldn't that cast some doubt on their results? They would be getting "nitpicked" but would it be unfair? Maybe their result would be reproducible regardless of their sloppy practices, but until then I would be more skeptical than I would be of a clean and orderly lab.
rubyist5eva
You don't need an esoteric license, just use a standard license like MIT>

relevant section from MIT license:

"THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE."

gwd
> Don't clean & organize!

FWIW I basically did this: My thesis numbers were run on a branch based on the unstable version of an upstream project that was going through a major refactoring. I took a tarball of the VCS tree at that point in time and posted it online. Over the years 3-4 people have asked for the tarball; nobody has ever come back to me with any more questions. I can only assume they gave up trying to make it work in despair.

I think I tried to build it a couple of years ago, and it wouldn't even build because gcc has gotten a lot more strict than it used to be (and the upstream project has -Werror, so any warnings break the compilation).

I think it's definitely worth doing, but I think you need to be realistic about how much impact that kind of "publishing" is really going to have.

beckman466
> Matt Might has a solution for this that I love: Don't clean & organize! Release it under the CRAPL[0]

> "Generally, academic software is stapled together on a tight deadline; an expert user has to coerce it into running; and it's not pretty code. Academic code is about 'proof of concept.'"

This is brilliant!

edit: it seems that, aside from that great snippet from the text, the license itself isn't so great. another comment [1] has a great analysis of the actual license and suggests using a superior solution (copying the preamble part yet still using MIT/(A)GPL).

[1] https://news.ycombinator.com/item?id=29937180

CJefferson
I've posted a huge amount of academic code (I've linked to a small number at the end). I think you should, but it won't help advance your career immediately. However, I still think it's better for science.

What is useful is if you can produce code people can build on and do their own cool stuff with -- then they will cite you. However, getting something to a state where it is tested for all reasonable inputs, has some basic docs, etc. is a hard untaking.

https://github.com/minion/minion (C++ constraint solver)

https://github.com/stacs-cp/demystify (Python puzzle solver)

https://github.com/peal/vole (Rust group theory solver)

[deleted]
(comment deleted)
erwincoumans
Thanks, agreed. Small note: it is not clear what Minion is doing, from just visiting the github repo. Perhaps add "C++ constraint solver" in the github description, but it is still unclear: it could be a rigid body constraint solver for games? Maybe add a link to a paper?
CJefferson
Yes, I should practice making things more accessible :)

In practice Minion is generally used as a backend to Conjure ( https://conjure.readthedocs.io/en/latest/ ), which provides a much nicer input language.

erwincoumans
Thanks, I was not familiar with Conjure and general Constraint Programming. I haven't seen it in real-time appications for games or robotics (usually highly optimized domain specific constraint solvers are used there, for rigid body, fluid sim, cloth, deformables etc)
spaniard89277
Yes, please, the state of affairs currently is that it's impossible to get code, data, and pretty much anything besides the actual paper.

To me at least sends a signal of people hiding stuff. That's not good. It made me distrust some papers in the past. I tried to reach out with no success.

cowpig
What is the purpose of doing research?

If the purpose is to push human knowledge forward, then it seems backwards not to publish everything.

Personally, I've found it difficult in my various careers to date when I've been put in positions where the actions that serve my immediate interests are in any way in conflict with my underlying principles or overarching goals. It's demotivating and deflating.

If I were in your position, I would publish everything and let myself feel pride in what I did. Even if we're all just insignificant specks in the grand scheme of things, pursuing a greater purpose can help make it feel like something matters.

dqpb
> it's substantially more work to clean and organize the code for publishing

Regardless of whether or not you release the code, you should do this.

It’s so common for people to think that cleaning/refactoring/documenting code is a waste of time, but it’s exactly the opposite.

The point at which the code is working, but not yet polished is exactly the prime “teachable moment” for improving your skills as a programmer and for refining your knowledge of the domain the program solves for. (This is true no matter how skilled or knowledgeable you already are).

Your brain is perfectly primed to do this now, so don’t let that go to waste.

phkahler
>> Besides, many scientists look at code as a competitive advantage so in this case publishing the code will be removing the competitive advantage.

That "competitive advantage" is just holding everyone back, slowing progress. This is particularly annoying to hear coming from "research" which I thought was supposed to be advancing the state of the art for the benefit of society. That's ostensibly the reason for publishing papers right, to disseminate knowledge? Or is it really just to increase ones ego and get paid?

Not saying you should publish code, just that deliberately keeping secrets in your field seems to go against what I thought you were doing.

erwincoumans
Agreed, perhaps there is some competition in citations for follow up work, releasing source code makes it easier to get 'scooped' on your future plans section? (not saying that optimizing for citations is a good thing)