79 comments

[ 3.6 ms ] story [ 178 ms ] thread
If is wrong disavow is one possible way, another would be to repeat the experiment and publish a correction criticizing your own article.

On the other hand, if your name is on an article, you must do your homework and know if what you are signing is realistic or not.

However, what you say in the second paragraph is exactly what the reviewers should ask the authors and question.
> if your name is on an article, you must do your homework and know if what you are signing is realistic or not.

Definitely not the case when you're an undergraduate trying to get your foot in the door with your first paper.

Isn’t it the case that when you’re an undergraduate you do the grunt work and collect and process all the base data, and the risk is actually on the side of the supervisor?
The base data in this case was (supposedly) collected by the Canadian Centre for DNA Barcoding, rather than anyone in the lab. However that Centre denies having the samples in their database.
As far as I know, writing a research paper at all as an undergraduate is unusual.

In the US, afaik, the purpose of undergraduate work is to learn the material of the field. A Phd program then teaches the student how to do research in a field and the student goes on to do actual research only then.

An undergraduate would do a research paper as independent study or a special department class I think. With the provision that advisor is doing a lot of hand holding.

So handing fake data to someone's whose effectively a complete newby and letting them process it and publish it as their first achievement, is really despicable as the article seems to describe.

If an undergraduate accepts to cheat as exchange to have a career in science maybe simply shouldn't have a career in science.
There's pretty much no way to do that in interdisciplinary publications. In those cases the main reason you have the other people on board is because you don't understand their field well enough.

And there are plenty of cases where you won't notice anything is off unless you actually examine the raw data in detail, which isn't really realistic as it's an enormous amount of effort and duplicated work. There are certainly cases where co-authors should notice that something is off about the data. But there are plenty of situations where there is no reason at all to suspect a problem with the data, and where there is no way to know that someone is simply providing false data.

> In those cases the main reason you have the other people on board is because you don't understand their field well enough.

“Don’t put your name on something you don’t understand” is a good rule to live by. All the authors share responsibility. We cannot accept credit when all works well and reject blame when it does not.

> But there are plenty of situations where there is no reason at all to suspect a problem with the data, and where there is no way to know that someone is simply providing false data.

Publishing an article with better results is definitely the best thing to do then. Short of that, publishing a letter to the editor sharing concerns is a good way of putting your doubts on record. Just removing your name looks too much like washing your hands of the whole thing.

A call replicate a paper that's demonstrably fraudulent seems even more of a cover-up of the fraud as just doing nothing.

The original paper was published while the first author was an undergraduate, btw, so they probably didn't have a complete grasp of the issue.

This is a widespread problem. A close friend went through a similar experience collaborating at another supposedly reputable Canadian university.

He reported the academic dishonesty (copies of figures from another paper presented as new experimental results), and the university went to great lengths to keep the whole thing quiet.

5 years later, the lead researcher who faked the results still works at the university, the paper has not been retracted, and my friend left what became a pretty hostile lab (and AFAIK is still under NDA)

At that point, what is left aside from trying to publicly remove your own name from a paper you know is fraudulent?

modern academia has become a meta game where its more about being able to navigate the system(get grant funding, go with status quo to get tenure, collude with others for citations, etc.) rather than actually being good at the field. We spend drastically more government money on research funding than in the past yet get less in return

see the Alzheimer's amyloid plaque cartel- https://www.statnews.com/2019/06/25/alzheimers-cabal-thwarte...

https://www.upi.com/Health_News/2019/12/31/Amyloid-plaques-m...

“It is difficult to get a man to understand something, when his salary depends on his not understanding it.”

> modern academia has become a meta game where its more about being able to navigate the system rather than actually being good at the field.

Sounds like the entire history of academia (and most human endeavors, honestly)

> We spend drastically more government money on research funding than in the past

You didn't define "in the past" but pure science spending has generally declined as a percentage of GDP and has barely changed in inflation adjusted dollars in the last twenty years.

> yet get less in return*

As measured how?

Don't get fooled by that stat. In actual dollars not inflation adjusted dollars. If the US is making 10% more because exports of coal the fact that they spend 3% on coal investment means science gets more money at a lower percentage.
> Don't get fooled by that stay.

Which stat? The poster didn't provide any.

> In actual dollars not inflation adjusted dollars.

Scientists are paid salaries and need to eat and stuff, so it doesn't really make sense to me to ignore inflation there.

I agree, as someone who's seen how the sausage gets made, it's discouraging that such a noble endeavor remains so vulnerable to financial conflict of interest from the individual to the institutional level.

We'll still ultimately learn more about what's true in our universe, but it will take even longer and be more expensive than necessary.

How are the activities you listed a meta game instead of just a regular game with no Greek prefix?
If the game is scientific research, the metagame is as described.
Thanks. I thought the game was ego satisfaction.
Agreed. I do sometimes wonder if academia shouldn't be blown up and replaced wholesale with something different.

The new thing would have to have different incentives, or the same problems would just re-occur.

Why blow it up? Just create something that parallels it, and organize it in a way you think will work better. Then we can decide which one to blow up once we see how it measures up.

Some examples of where this is being done: open-access journals, study pre-registration, requirement to publish source code. No one's saying the old way is banned now, it exists in parallel with all those innovations. Some will stick, others will fizzle out.

> Just create something that parallels it

... then see your new system get attacked from all angles by the existing establishment. Smears on TV, 'fact-checkers' in social networks, bans and cancellations, etc.

The stakeholders of the existing system you are trying to replace won't go quietly into the night.

They're not exactly going to go quietly into the night with an attempt to "blow it up" either.
One good reason why the amyloid hypothesis persists in attracting support is that it is currently the only way to account for how the mutations in APP and presenilin cooperate to cause early onset familial Alzheimer's disease.

In particular, out of the 695 amino acids that constitute the APP protein, all the mutations that cause disease are located within the 42 amino acids that comprise the amyloid peptide, or just next to it.

NDA and academia? I guess that's considered normal but it would be good if there were checks and balances on the NDAs.
The only NDAs I'm familiar with are with vendors... maybe different in other fields.
... the MIT standard form NDA includes a clause which, in effect, says "You will not talk about the existence of this NDA." I'm not sure you'd know.
(comment deleted)
I’ve never had to sign an NDA and my whole career is in academia, and a lot of it with funding and partners from industry. The only colleagues I know (some of them working on nuclear materials, so quite sensitive) who are in this kind of situation is because of security clearance, and universities have nothing to do with it.

It’s definitely not normal, and should actually raise some red flags. I would like to know which institution does this sort of things.

MIT uses NDAs and non-disparage agreements to cover up research fraud. It's not uncommon.
I am not really convinced that it is “not uncommon”. Maybe MIT does but again, I have never had a colleague sign any. And I personally would not work for an institution that does.
How would you know? You might be working for an institution that does this right now.

I was intimidated into signed an NDA which says I can't talk about the NDA. The threats were serious, and I don't talk about that NDA.

Unless you're at senior levels of your university, you wouldn't know about sketchy stuff like this going on.

The way to solve this would be to extend restrictions on federal grant funding and financial aid to only encompass institutions which don't use tools like this.

I've signed a couple, but they've all been due to working with private industry.
You know about the fraudsters publishing about ivermectin?
I know you're just trying to make a point here, but not all of the Ivermectin research was fraudulent. There was some (seemingly) legit work from India/Bangladesh where Ivermectin did significantly increase the chance of survival among COVID patients.

(To be fair, these same regions where the legit studies are from also have a high prevalence of parasitic worm infections, but it's still a poignant reminder that removing complicating factors can help heal the primary disease.)

Thank you for bringing this up because so much of the Ivermectin discourse forgets about it and I continue to be very glad that doctors in those areas have an extra tool for saving lives.

The anglosphere yelling at itself in counterproductive ways would, I presume, have happened anyway albeit over a different treatment, so this seems to me like it was at least close to a pure win and "fewer dead people" is a pretty solid win, all things considered.

It's been said before but once it was decided having papers published became a target everything started to go to shit.
The root issue is that many people want to become scientists but other people are only willing to fund far fewer scientists.

When the supply of scientists far exceeds the demand, there must be a way to decide who gets jobs and funding. Some people favor objective measurements, because they are less prone to corruption and nepotism than subjective ones. But when you have objective measurements, people will try to game them, and some will use dishonest means.

I think tech company hiring is in a loosely similar situation. No one thinks whiteboard coding is an awesome way to test candidates. Everyone agrees that it doesn't reflect what programmers do day-to-day, and everyone knows there are lots of strong programmers out there who don't do well in whiteboard settings. But I'm not aware of any viable alternatives. The good old fashioned approach of "judge people based on what school they went to, what companies they worked for, and who they know" is probably worse for everyone. The ideal approach might be "hire anyone who asks and then fire them after a year if they're not doing well", but that's very expensive for everyone involved.
There are tons of alternatives, anywhere from take home tests to just gauging their general experience and answers to some technical questions.
Personally I like the idea of hiring a prospective employee as a contractor to do a small but real task. That will give you a more reliable read than any interview. I don’t understand why this is not a more common practice.
That does seem like a great approach when it works, but I expect it wouldn't work for the majority of people/positions. A few thoughts off the top of my head:

- You need to be available for contracting work, but many people prefer to go from one full-time job to another. Or they might not want to leave their current job, unless they get a much better offer.

- You need to be in the right location. Relocating without a job offer is risky.

- Especially strong candidates are likely to get many offers when they go looking, which is a good position to be in for salary negotiations. But work-as-interview arrangements make it hard to interview with more than one company at a time.

- Anything resembling visa support is probably off the table.

> gauging their general experience and answers to some technical questions

This plus a very easy coding question (think fizzbuzz) works really well at a small enough company that you can ensure that all interviewers:

1. Are far enough above the hiring bar that they can comfortably tell when other people are (people near the bar have a harder time distinguishing whether someone is a bit above or below the bar)

2. Are invested enough in the company to want to do a good job interviewing

3. Are thoughtful enough to gauge the competencies the company needs for the role. Those needs are not universal, I know developers who thrive in super technical no-nonsense roles who flounder in product-y work.

If a company is very large or growing rapidly it is nearly impossible to guarantee these things. My experience is that companies that don't adopt something like the industry norms lose control of their hiring bar. In some cases this is fine (sometimes you need quantity, not quality) but if the business relied on that high technical bar there's generally a cascading effect of initiatives failing, good engineers leaving, and more initiatives failing because of it. I've seen this tank a few startups in the 50-100 engineer rage.

One of the standouts of my business school education was the recruiting class where our professor spent most of his time drilling in how bad most of the ways that we assess people are, but that you have to learn to manage to cognitive dissonance of it because that's what most places are going to do anyway.
actually, new professor should publish enough number of research article or they will be fired by organizations
In my experience in academia, albeit not in the sciences, dishonesty far from the main problem. There is a big difference between fundamental research of wide and lasting interest, and research that ticks enough boxes to be published. Because the second is a lot easier to do than the first, and is what an academic's performance is judged on, everyone ends up spewing out mediocre papers that, intellectually, take us nowhere. Research becomes degraded as a means to the end of a career.
You see the same dynamic in Hollywood with movie actresses. The biggest difference is that the official grant application racket is replaced with an open-air underground system of sexual favors to studio executives.
When a measure becomes a target, it ceases to be a good measure
https://en.wikipedia.org/wiki/Goodhart%27s_law

Absolute fav :)

Best guesses I've got for how to combat the situation: - Have a diversity of metrics that are somewhat independent, somewhat anti-correlated (if I'm using the right words) - Have non-measurable goals that you refer back to

Anyone else got ideas for hedging against the failure mode of Goodhart's Law? :)

Yes, changing metrics often in unpredictable ways.

I have seen this in a company where they would change employee objectives every year, and also promote and demote people into unexpected new roles. This seemed to result in everyone understanding that next year things can be very different, and does not give people enough time to corrupt a metric. So everyone needs to build good working relations with superiors and subordinates, and not only try to meet objectives but also do well for the company in general, because next year roles can be reversed.

This system is tiring for everyone though...

you have two options -- empirical metrics or holistic or subjective evaluation. the weakness of the former is that it is subject to gaming, the latter is subject to bias and nepotism.
Sure. Which alternative metric would you prefer we use for allocating jobs and funding, and why do you expect that a generation from now, it won't be gamed into absurdity?
I understand that Melinda Gates is looking for different charities to invest in.

I wonder if she'd fund researchers to attempt to replicate crucial publications / studies.

I expect that she has enough money to put a serious dent in this problem, at least if she focused on some specific area.

It doesn't work like that. When you do actual science at the bench you find out fairly quickly whose work is solid and whose is like weak reeds. The trouble is the misaligned incentives, everyone has good reason to keep silent for fear of attracting the attention of the funding agency. If something is afoot (the rumours are usually true), soon you are looking at 7-figure damages.
I don't understand why that invalidates my idea though.

In this case, Gates would be the funding agency. And its published findings would hopefully steer much academic teaching and research away from bad assumptions.

Just to elaborate on the "it doesn't work that way" part.

Science is hard. Replicating an experiment is costly in time and money.

The papers in scientific field are organized like a tree. A good paper by a good and credible researcher will be a base node that other papers branch off of. A poor or not credible paper will be leaf that won't be cited by others. Testing all the poor papers would be tremendous waste of time. And it wouldn't catch fraudsters as such because not being replicable isn't proof of fraud.

Talking about replicating this stuff misses the point.

Science can't survive if the answer to faking claims is to replicate every study. The reason is that replicating a study done in complete seriousness and honesty is hard and is not guaranteed to yield the same result. So "can't be replicated" shouldn't be a black mark - "was basically made up" should be a black mark.

And also, if faking stuff becomes standard with replication the main guard, an easy way to get points is announce the fake replication of a given study.

> replicating a study done in complete seriousness and honesty is hard and is not guaranteed to yield the same result

Yes, a study failing to replicate is not in itself evidence of fraud. Nor is it definitive evidence that the effect shown in the original doesn't exist, since a replication can be messed up too.

But science needs replication because we're ultimately trying to learn about the world, and an effect we can't reliably produce is usually an effect we're not sure exists. Replication sometimes catching dishonesty is a bonus: failure to replicate calls the effect into question, which calls the method used to find that effect into question, which sometimes calls into question whether that method was actually used. (Or in the case of bad statistics, whether it was chosen to dishonestly show a certain result.) It's the appropriate propagation of doubt.

> Yes, a study failing to replicate is not in itself evidence of fraud. Nor is it definitive evidence that the effect shown in the original doesn't exist, since a replication can be messed up too.

It’s not a trial to determine the guilt of a supposed fraudster. This is what institutions should do because fraud is a breach of contract (in all sane institutions).

However, the main problem is made up results in the scientific record. This is what we attempt to solve by retracting articles. And from the point of view of the scientific record, there is no meaningful difference between a made up result and something that cannot be replicated.

This is what we attempt to solve by retracting articles. And from the point of view of the scientific record, there is no meaningful difference between a made up result and something that cannot be replicated.

I don't think "the scientific record" is more than a metaphor that's referred to in talking of retraction. Still, if one were to imagine a hypothetical scientific record, you would have to imagine something that's inherently malleable. Even if most basic results are never going to be disproved, so fraction of even the most basic result are going to be disproved.

The thing, it's unusual but not unheard-of for a researcher to come up with a result, have it not be replicated, have the result languish and then have a later researcher take advantage of the result, either show it could be replicated or otherwise explaining what was actually going on.

But if the explanation for a strange result is "they faked it", then the impact of the situation is different.

Moreover, non-reproducibility is a reason for retraction but I don't think it's the main reason and well know papers that haven't reproduced aren't necessarily retracted Afaik.

Which is to say, I think there's still an important difference between non-reproducible for unknown reasons and know-to-be-fabricated data.

Article on retraction practices and "correcting the scientific record" https://pubs.acs.org/doi/10.1021/acs.chemmater.9b00897

I fail to see the point of a study that can't be replicated. Assuming the replication is performed correctly, failure to replicate pretty much means either deliberate fraud (unlikely) or an honest "mistake" (or whatever word you prefer to use to describe a situation where you thought you measured and described a real-world effect in a manner that's in any way useful, but in fact you did not).
At this point we are all just living in The Wire all the time.

So many of our institutions have been corrupted from the inside by the more ruthless nature of careerism that’s taken over the modern world. Bond rating agencies endorse junk debt, doctors say opiates aren’t addictive any more, epidemiologists say the politics of your protest determine if it’s safe during a pandemic, elite educational institutions tell you going into crippling debt will be good for you.

Literally every institution, including ones supposed to serve the public, is rotting from the inside out from people bullshitting to keep their job or get promoted.

https://youtu.be/_ogxZxu6cjM

Without necessarily endorsing the premise of the comment, I wonder what can be done to prevent this. Has humanity ever been successful at preventing institutions from regressing to exclusively serving their own needs? It's not clear if as a species we can actually pull that off, at least not in perpetuity.
Implicit in your comment is that once upon a time things were different. That is not at all clear to me. Rather, science, government, public health, etc have always been largely a mess, and whatever you conjure up in your idealized view of "the good old days" in those human endeavours is just good old fashioned selection bias.
In academia, things were different. Science didn't really get like this until tenure became hyper-competitive. In the early days of the British tradition, science was a leisurely task by nobles. After WWII, academia was expanding rapidly, and there was a shortage of professors. Academic jobs were easy to come by.

The strong incentives to cheat is very new in academia. I've seen academia for about a quarter-century, and at least MIT went from a pretty honest place to a shark tank full of crooks.

That's not a general "the world was once better," but a very specific "academic fraud has grown out-of-control in the past 25 years."

The other big change happened when Clinton doubled the NIH budget in the 90s. This made a lot more room for shark tanks full of crooks (sadly).
> It's not uncommon for research journals to defer to institutional misconduct investigations, in line with recommendations from the Committee on Publication Ethics (COPE).

I wonder whether any dark humor was involved when making that initialism.

(comment deleted)
I wonder if the problems with the data would have been discovered sooner if data sharing been enforced. It's clear that the reason the article was retracted by the editor-in-chief was because of the lack of data:

It appears that sequences were neither uploaded to Genbank nor to the BOLD system at the time of the study. It does appear that sequences related to this article have been uploaded to Genbank in September 2020, but as this happened six years after publication and no voucher information is presented in the article, post-publication review was unable to confirm whether these sequences were indeed derived from the research described in this article.

https://link.springer.com/article/10.1007%2Fs10531-021-02316...

If I could I'd disavow every paper that has my name on it except for one. But, if I want my children to have a good education I can't.