> But the human reviewers didn't do much better: they correctly identified only 68% of the generated abstracts and 86% of the genuine abstracts. They incorrectly identified 32% of the generated abstracts as being real and 14% of the genuine abstracts as being generated.
Isn't the headline actually "Abstracts written by ChatGPT fool scientists 1/3 of the time"? Having never written one myself, wouldn't the abstract be the place where ChatGPT shines, being able to write unsubstantiated information confidently? I imagine getting into the meat of the paper would quickly reveal issues.
I think it has an important difference for ChatGPT - it likes to generate numbers that make absolutely no sense. A human that lies will try to generate sufficiently correct data to convince. ChatGPT often won't even make an attempt to produce values that fit.
I guess that's good for us meat beings. Better for an AI to incompetently lie than competently lie.
I wonder if having AI models available would make it significantly easier to identify material created by that model. Seems it would be easier, but it would still be a big problem. Nets can be trained to identify ai vs not ai on a given model. And that's without needing access to the model weights, just training examples. But when there are N potential models...
>they correctly identified only 68% of the generated abstracts and 86% of the genuine abstracts.
I think you and I are basically in alignment... what this tells me is that 14% of real abstracts are so bad that other human beings call their BS. Meanwhile, this AI stuff is kinda working 32% of the time in generating legitimately interesting ideas.
So at that point - yeah, that sounds about right. The 32% is still so low that it shows AI is not anywhere near maturity, whereas 14% of human-generated is crap.
And, yeah - a short blurb like an abstract seems to be exactly the kind of text that ChatGPT is conditioned to do well generating. As others below note - once a human starts reading the rest, the alarm bells trigger.
(Buuuut let us also enjoy the fact that ML can generate a decent-looking abstract to a scientific paper 32% of the time. This represents massive progress; I would venture that most humans cannot write such an abstract convincingly.)
I would argue the opposite. Give anyone who can write reasonnably the abstract of a few dozen papers in a given field, and I am confident they would be able to produce a convincing bullshit abstract. Most abstracts in a given field sound extremely similar, and are mostly keyword dropping with an ounce of self-promotion. I actually think that the only reason ChatGPT did not produce a higher rate of convincing abstracts is that the reaserchers assessing them knew they had a high probability to look at an AI result and were thus extra careful. Most of the abstracts correctly labelled as AI would probably be considered legit (which does not mean good) if sent to a conference without warning.
It really would depend on the quality of the work/paper. I bet mis-identification by humans of hand written or generated abstracts goes down in higher tier conferences vs. lower tier ones, since the writing standards in lower tier conferences can be pretty bad (well, in CS at least).
All true I think. Interesting thing is how many researchers read just the abstract of papers. I’d say it’s quite common to read just the abstract and some future captions, and only dig into details on a few papers.
That's the main purpose of the abstract. There are several levels of reading a paper: checking the title, reading the abstract, reading intro/conclusions and skimming over the table, or reading the whole thing. You start at the first level and continue or stop depending on the interest of the paper for your work (and available time).
This works very well to more or less stay on top of things with today's frantic publication pace in many disciplines.
Writing text that feels plausibly real is ChatGPT’s specialty.
Fake scientific papers that are written with the language, vocabulary, and styling of an academic paper have been a problem for a long time. The supplement and alternative medicine industries have been producing fake studies at high volumes for years now. ChatGPT will only make it accessible to a wider audience.
Isn't that the reason we have trusted scientific peer review journals? I mean, why trust a paper that hasn't been vetted by a trusted source? The same is true in news media - I don't give any stock to news content that isn't published by a well-trusted source (and I do pay for subscriptions, e.g. AAAS, Financial Times, etc., for that very reason). I guess I don't understand the concern - the world has always been filled with junk information and we have tried and true systems in place already to deal with it.
Considering how much intentionally fake garbage got published, this doesn't surprise me at all... and this is not just random scientists, but scientists who should (atleast theoretically) know enough to be able to notice it's gibberish.
I'm reminded of the somewhat recent news of a line of Alzheimer's research being based on a fabricated paper that was only caught many years later [0].
Previously, we've relied on a number of heuristics to determine if something is real or not, such as if an image has any signs of a poor photoshop job, or if a written work has proper grammar. These heuristics work somewhat, but a motivated adversary can still get through.
As the quality of fakes gets better, we'll need to develop better tools to dealing with them. For science, this could, hopefully, result in better work replicating previous works.
I'm quite likely being overly optimistic, but there's a chance for positive outcomes here.
The requirement to detect something fake is quite easy, and we knew it for a long time: publish all data code and everything to make the expriments reproducible.
Even if everything is fake, the code has value for further research.
It would be nice to have that as a minimun standard at this point, as I would prefer to see much less publications that can be trusted more than the current situation.
If a developer codes up an AI to scour the web, write an article, and submit it to a scientific journal without letting the developer see the article, is the developer doing science?
If someone trains a translation model between languages they don't know, is that someone a translator?
I guess the users of said model would be "translators" as they would be doing the translation (without necessarily knowing the languages either).
unsure if anyone is "doing science". Doing science is applying the scientific method.
Making conjectures, deriving predictions from the hypotheses as logical consequences, and then carrying out experiments or empirical observations based on those predictions.
Not sure AI is up to that, and it's debatable if it'll ever be able to make and test conjectures. There is a difference between symbol manipulation (like outputting text) and actual conjecture.
Not really (I mean it helps, but it doesn't make it easy). It can still be very difficult to reproduce legitimate results, even with help from the original researcher. And faked results can last a long time even without reproduction because it's very easy to assume you're doing something wrong.
Well, given that all paper abstracts have to follow the same structure with the same keywords and be conservative to get a chance to get published, it makes sense that ChatGPT shines there.
IMHO, it says more about the manic habits of journal editors than anything else.
That's a feature, not a bug. It means that when you have 100 papers to check for applicability to something that you are researching you can do so in a minimum of time.
Not really IME; you have to go through layers of bullshit aiming at making the paper seems more important that it is, not hurting the feelings of Pr. Curmudgedon that could be a reviewer, fitting the grant that funded the paper, hiding the weak points of the study, adhering to the current Scientific Serious Professional Way Of Writing™, not disturbing the flawed, but socially accepted consensus in this particular field, ... and so on that are actually burying what should actually have been in the abstract.
What I see as wrong here is an AI witch-hunt. AI is a tool. And it would be the same as calling the baning the use of a car cause horses exist. Obviously the disruption is happening, which is always a good thing as it should lead to progress.
On the other hand, all kinds of technology have been regulated to minimize adverse effects. The trouble with software is that it is evolving faster than regulators can keep track of, and it is very hard to police even if regulated.
There is only so much peer review can actually accomplish. Mostly a reviewer can tell if the work was performed with a certain amount of rigor and the results are supported by the techniques used to test the claimed results. It doesn't guaranty there were no mistakes made. Having others reproduce the results is the only true way to verify an experiment. Unfortunately you don't get tenure for reproducing other people work.
At least one nice side-effect of this could be that only reproducible research with code provided will matter in the future (this should already be the case but for some reason isn't yet). What's the point of trusting a paper without code if ChatGPT can produce 10 such papers with fake results in less than a second
ChatGPT can produce code too. Therefore I think this may call for something more extreme — at risk of demonstrating my own naïveté about modern science, perhaps only allowing publication after replication, rather than after peer-review?
Ideally yes, for a paper to be accepted it should be reproduced, if ChatGPT is ever able to produce code that runs and produce SOTA results then I guess we won't need researchers anymore
There is however a problem when the contents of the papers costs thousands/millions of $ to be reproduced (think GPT3, DALLE, and most of the papers coming Google, OpenAI, Meta, Microsoft). More than replication, it would require fully open science where all the experiments and results of a paper are publicly available, but I doubt tech companies will agree with that.
Ultimately it could also end up with researchers only trusting papers coming from known labs/people/companies
Indeed and other sciences seems even harder to reproduce/verify (e.g. how can mathematicians efficiently verify results if chatgpt can produce thousands of wrong proofs)
> there are already ways to automate testing in their domain.
Do you mean proof assistant like Lean ? From my limited knowledge of fundamental math research, I thought most math publications these days only provide a paper with statements and proofs, but not with a standardized format
Only a tiny fraction of existing maths can be done with proof assistants currently, and as a result very very few papers use them. In most current research automated testing would be impossible or orders of magnitude more work; in many areas mathematicians are working with things centuries ahead of where proof assistants are up to, and working at a much higher level of abstraction. Also, many maths papers have important content that is not proofs (and many applied maths papers contain no proofs at all).
I'm thinking of the LHC or the JWST: billions of dollars for an essentially unique instrument, though each produces far more than one paper.
Code from ChatGPT could very well end up processing data from each of them — I wouldn't be surprised if it already has, albeit in the form of a researcher playing around with the AI to see if it was any use.
Reproduction of experiments generally comes after publication, not before acceptance. Reviewers of a paper would review the analysis of the data, and whether the conclusions are reasonable given the data, but no one would expect a reviewer to replicate a chemical experiment, or the biopsy of some mice, or re-do a sociological survey or repeat observation of some astronomy phenomenon, or any other experimental setup.
Reviewers work from an assumption that the data is valid, and reproduction (or failed reproduction) of a paper happens as part of the scientific discourse after the paper is accepted and published.
If an software system can generate abstracts, good. Nobody got into research for love of abstract-writing.
It is a tool. Ultimately researchers are responsible for their use of a tool, so they should check the abstract and make sure it is good, but there’s no reason it should be seen as a bad thing.
> if scientists can’t determine whether research is true, there could be “dire consequences”
Yeah well we can't tell that now either. Maybe we can finally start publishing raw data alongside these "trust us we found something" papers that people evaluate based on the reputation of the journal and the authors.
As someone else pointed out, that system has already derailed decades of Alzheimer's research. It's stupid and broken and it should have changed a long time ago.
I know it was just titles, but I was having a good day on "arxiv vs snarxiv" if I did better than random chance. And that was just a Markov text generator, no fancier AI needed.
Not really. No one writes a paper with the aim of convincing reviewers that they are human, and the reviewers aren't trying to determine whether the authors are human.
As a researcher, I would expect any researcher to be able to generate fake abstracts. However, I suspect that generating a whole paper that had any interest would be nigh on impossible for AI to do. An interesting paper would have to have novel claims that were plausible and supported by a web of interacting data.
That's not how we should use it - give a few ideas, a bunch of notes, results and ask the model to write the text. It will write better text than most, but we are responsible for prompting it with valid data.
Nice trick for ChatGPT, but this will not destroy science.
Nobody takes a serious decision reading only the abstract. Look at the tables, look at the graphs, look at the strange details. Look at the list of authors, institutions, ...
Has it been reproduced? Has the last few works of the same team been reproduced? And if it's possible, reproduce it locally. People claim that nobody reproduce other teams works, but that's misleading. People reproduce other teams works unofficially, or with some tweaks. An exact reproductions is difficult to publish, but if it has a few random tweaks ^W^W improvements, it's more easy to get it published.
The only time I think people read only the abstract is to accept talks for conference. I've seen a few bad conference talks, and the problem is that sometimes the abstracts get posted on like in bulk without further check. So the conclusion is don't trust online abstracts, always read the full paper.
EDIT: Look at the journal where it's published. [How could I have forgotten that!]
I'm quite confident that there are cliques within "science" which are admitted without as much as a glance at the body of the papers. Some people simply cannot be bothered to get past the paywalls, others accept on grounds outside the content of the paper, like local reputation or tenure. Others are asked to review without the needed expertise, qualification, or time to properly understand the content. Even the most honorable reviewers make mistakes and overlook critical details. Then there are the set of papers which are (rightfully so) largely about style, consistency, and honestly, fashion.
How can we yield results from an industry being lead by automated derivatives of the past?
Is an AI-generated result any less valid than one created by a human with equally poor methods?
Will this issue bring new focus on the larger problems of the bloated academic research community?
Finally, how does this impact the primary functions of our academic institutions... teaching.
> How can we yield results from an industry being lead by automated derivatives of the past?
Even a human researcher need experiments to validate ideas. AI can generate plausible ideas, so why not run the experiment and let it learn from the outcomes? The source of learning comes from experimentation, that's how models escape the derivative trap. AlphaGo invented move 37, proving AI can be creative and smart.
Interesting questions. I don't think most of them have a definitive answer, so this is my opinion.
> I'm quite confident that there are cliques within "science" which are admitted without as much as a glance at the body of the papers.
It's possible. I don't know every single area, but I guess it's not common in most serious branches od science.
> Some people simply cannot be bothered to get past the paywalls, others accept on grounds outside the content of the paper, like local reputation or tenure.
My friends says they can ask Alexandra for a copy. A few years ago it was also common to ask a friend in another site that has a copy.
> Others are asked to review without the needed expertise, qualification, or time to properly understand the content. Even the most honorable reviewers make mistakes and overlook critical details. Then there are the set of papers which are (rightfully so) largely about style, consistency, and honestly, fashion.
That's why you RTFP instead of thrusting the journal or the reviewer or just the abstract. I've seen "dubious" papers published in good journal.
> How can we yield results from an industry being lead by automated derivatives of the past?
It's running now in natural intelligence. Every time there is a interesting paper, other groups run to publish variants or combinations with other result to reach the anual quota, or get enough point for the graduate student. Once the GAI can read all papers and combine them, there will be very few low hanging fruits to pick.
> Is an AI-generated result any less valid than one created by a human with equally poor methods?
For now, AI is to stupid to get the details right. That's why it's important to read the full paper instead of only the abstract. One AI is intelligent enough, the result may be as bad as a cheating human. Let's hope the AI has a good PRNG to create credible noise, because that's a part humans do badly.
> Will this issue bring new focus on the larger problems of the bloated academic research community?
Nah.
> Finally, how does this impact the primary functions of our academic institutions... teaching.
I don't understand the question. You fear that professor in some areas will just send fake papers written by ChatGPT4.0 and the journal and the community will not notice? There are a lot of predatory journals, open-peer-review journals and other bad journals that are publishing a lot of crap. Usually by professor in bad universities or just as a fancy achievement for the c.v.. A good AI will increase the amount of crap, but it will be just ignored.
No, actually I'm curious if this could open up the schedule for professors who want to spend more time teaching to design and develop better curriculums for their students. But I'm probably being overly optimistic about the number of profs who actually want to teach.
There are some universities that don't require research, so if you want to only teach you can go to one of them. It's a solved problem. Anyway, the best universities require research, opting out of research so you get less money, less prestige, and not the top students.
Also, accumulating published papers is important to get a new position in the future, so only teaching is a risk for your future career.
For teaching some of the topics to advanced students, it's important to have people that do research and is updated with the cutting edge results. Also to babysit the graduate students so they can publish their results. For teaching the students in the first years, research is probably overrated.
There are many interesting result that fly under the radar. If you don't want to count big things like LIGO, my favorite to talk about is magnetoresitance, IIRC the giant one https://en.wikipedia.org/wiki/Giant_magnetoresistance [It's not my specialization, so I may have a few details wrong.]
You can explain it to a technical friend.
Start explaining about the the two possible spin of electrons, and how it cause the existence of two currents inside a conductor. This is not important in a normal conductor like cooper, but it's important inside a magnetic conductor like iron. So yo get a different resistance for each one of the two currents, the one that has a spin that is in the same direction than the magnetic field and the other with the oposite spin.
You can make a sandwich of iron-cooper-iron. If there is no external magnetic field, the two iron parts have oposite magnetization and the total resistance is higher. If there is an external magnetic field, the two iron parts have the same magnetization and the total resistance is lower. Anyway, the difference is not too much.
[Ideally, your fiend should be boring now, uninterested about the abstract currents no one cares about.]
It get's more interesting if you have many layers of iron and cooper, because the difference is higher and they call is "giant". It is used in the heads to read hard disks, like in the laptop of your friend. [Your friend will never see it coming!]
It's interesting because it mix weird abstract quantum properties and engineering to make it more efficient and you get a device that everyone has. For some weird reason, no one talks about it. And the sad part is that SSD are killing the punch line of the story :( .
> Nobody takes a serious decision reading only the abstract.
I am not sure if this is sarcasm or not.
Literally the whole world besides the researches read mainly the abstract and make even life changing decisions based on it. (just look at any twitter discussion linking to a paper)
I don't consider twitter discussion "serious decisions". When a thread here reach 100 comments, the quality of the discussion usually drops. I don't want to imagine how bad it's in twitter.
The problem is people overdosing from snake oil they read in twitter. I think there were a few cases with ivermectine, hydroxychloroquine and chlorine dioxide [1]. People should not self medicate, or at least understand proportions and that the dose makes the poison. They should see a medical doctor [2], that at least understand proportions and that the dose makes the poison and that they should follow the advice from the FDA made by people that read a few of research papers and extended reports, instead of only one press release about a preprint written by a moron that somehow got a position in a university or hospital.
I guess politician don't read the full paper (except Merkel?), but they hire experts to read the articles and give advice. If you are a politician and the "expert" is reading only the abstract without checking the full paper and the journal and other related stuff, you should fire the moron.
[1] The chemistry of chlorine dioxide is too simple and it's obvious that it can't possible work, so it makes me angry. The other stuff don't work, there were no good reasons to expect them to work, but at least it's not obvious with elementary chemistry that it can't work.
[2] I have horror stories about medical doctos too. Always ask for a second opinion for important stuff.
> And if I can use ChatGPT to write an abstract for me from my paper, let's go!
Is ChatGPT located in a central repository or cloud? Is centralized? If that, probably a bad idea.
A private company having access to your abstract before you publish it could easily lead to problems like plagiarism (even worse, automatized plagiarism) or give an unfair advantage to one in two teams running to publish the same result. Science has a lot of this cases.
It's weird to me that scientists make so much hay of the Sokal affair given how unscientific it is.
It's a single data point. Did anyone ever claim the editorial process of Social Text caught 100% of bunk? If not, how do we determine what percent it catches based on one slipped-through paper?
I'd expect scientists to demand both more reproducibility and more data to draw conclusions from one anecdote.
I think part of the problem comes to the sheer amount of jargon in even the simplest research paper. During my time in graduate school (CS) I would often do work that used papers in mathematics (differential geometry) for some of the stuff I was researching. Even having been fairly well versed in the jargon of both fields I was often left dumbfounded reading a paper.
This would seem to me a situation that is easily exploited by an AI that generate plausible text. If you pack enough jargon into your paper you will probably make it past several layers of review until someone actually sits down and checks the math/consistency which will be, of course, off in a way that is easily detected.
It's a problem academia has in general. Especially in STEM fields they have gotten so specialized that you practically need a second PhD in paper reading to even begin to understand the cutting edge. Maybe forcing text to be written so that early undergrads can understand it (without simplifying it to the point of losing meaning) would prevent this as an AI would likely be unable to do such feat without real context and understanding of the problem. Almost like adversarial Feynman method.
TBH, a paper's abstract is supposed to summarize the purpose and findings in the paper, so auto-generation of what is otherwise "repeating what the rest of the paper says" should be considered a win; it's automating boring work.
If ChatGPT can't do that (i.e. if it's attaching abstracts disjoint from the paper body), it's not the right tool for the job. A tool for that job would be valuable.
The real issue for me is that the bot might generate incorrect text, imposing a yet-higher burden on readers who already find it difficult to keep up with the literature. It is hard enough, working sentence by sentence through a paper (or even an abstract) wondering whether the authors made a mistake in the work, had difficulty explaining that work clearly, or wasted my time by "pumping up" their work to get it published.
The day is already too short, with an expansion of journals. But, there's a sort of silver lining: many folks restrict their reading to authors that they know, and whose work (and writing) they trust. Institutions come into play also, for I assume any professor caught using a bot to write text will be denied tenure or, if they have tenure, denied further research funding. Rules regarding plagiarism require only the addition of a phrase or two to cover bot-generated text, and plagiarism is the big sin in academia.
Speaking of sins, another natural consequence of bot-generate text is that students will be assessed more on examinations, and less on assignments. And those exams will be either hand-written or done in controlled environments, with invigilators watching like hawks, as they do at conventional examinations. We may return to the "old days", when grades reflected an assessment of how well students can perform, working alone, without resources and under pressure. Many will view this as a step backward, but those departments that have started to see bot-generated assignments have very little choice, because the university that gives an A+ to every student will lose its reputation and funding very quickly.
Isn't it how abstracts shall be ? - excluding phenomenal characteristics like: different formulas to get it, human or author involvement, creativity; in pure form being scientific form of some work, like an equation catching the essence without flaws or distractions - and that's what computers are for. to proceed, then humans may don't have to ??
But I'm lost at what those scientists are trying to find.. (?)
118 comments
[ 3.1 ms ] story [ 168 ms ] threadIsn't the headline actually "Abstracts written by ChatGPT fool scientists 1/3 of the time"? Having never written one myself, wouldn't the abstract be the place where ChatGPT shines, being able to write unsubstantiated information confidently? I imagine getting into the meat of the paper would quickly reveal issues.
This is a tautology: the thing that can be validated can be validated.
I wonder if having AI models available would make it significantly easier to identify material created by that model. Seems it would be easier, but it would still be a big problem. Nets can be trained to identify ai vs not ai on a given model. And that's without needing access to the model weights, just training examples. But when there are N potential models...
Edit: Second paragraph added later.
I think you and I are basically in alignment... what this tells me is that 14% of real abstracts are so bad that other human beings call their BS. Meanwhile, this AI stuff is kinda working 32% of the time in generating legitimately interesting ideas.
So at that point - yeah, that sounds about right. The 32% is still so low that it shows AI is not anywhere near maturity, whereas 14% of human-generated is crap.
And, yeah - a short blurb like an abstract seems to be exactly the kind of text that ChatGPT is conditioned to do well generating. As others below note - once a human starts reading the rest, the alarm bells trigger.
I don't know that interesting has anything to do with whether people found them plausible.
The bar is not "this is legitimately interesting," it's "I've seen real abstracts worse than this."
This works very well to more or less stay on top of things with today's frantic publication pace in many disciplines.
Fake scientific papers that are written with the language, vocabulary, and styling of an academic paper have been a problem for a long time. The supplement and alternative medicine industries have been producing fake studies at high volumes for years now. ChatGPT will only make it accessible to a wider audience.
https://en.wikipedia.org/wiki/Sokal_affair
https://en.wikipedia.org/wiki/List_of_scholarly_publishing_s...
Previously, we've relied on a number of heuristics to determine if something is real or not, such as if an image has any signs of a poor photoshop job, or if a written work has proper grammar. These heuristics work somewhat, but a motivated adversary can still get through.
As the quality of fakes gets better, we'll need to develop better tools to dealing with them. For science, this could, hopefully, result in better work replicating previous works.
I'm quite likely being overly optimistic, but there's a chance for positive outcomes here.
[0]: https://www.science.org/content/article/potential-fabricatio...
Even if everything is fake, the code has value for further research.
It would be nice to have that as a minimun standard at this point, as I would prefer to see much less publications that can be trusted more than the current situation.
I'd call that human-computer science partnership. If it checks out, it's not fake. Nonhuman scientists are still scientists.
If someone trains a translation model between languages they don't know, is that someone a translator?
I guess the users of said model would be "translators" as they would be doing the translation (without necessarily knowing the languages either).
Making conjectures, deriving predictions from the hypotheses as logical consequences, and then carrying out experiments or empirical observations based on those predictions.
Not sure AI is up to that, and it's debatable if it'll ever be able to make and test conjectures. There is a difference between symbol manipulation (like outputting text) and actual conjecture.
https://en.wikipedia.org/wiki/Robot_Scientist
Adam is capable of:
- hypothesizing to explain observations
That's not to say scientists shouldn't publish their data; they should.
In the limit, any claims one makes are rejected unless you also provide a way to reproduce the claim.
That's a lot of work, but perhaps the world doesn't need more monkeys and typewriters creating faux Shakespeare.
(https://pubmed.ncbi.nlm.nih.gov/36549229/)
IMHO, it says more about the manic habits of journal editors than anything else.
There is however a problem when the contents of the papers costs thousands/millions of $ to be reproduced (think GPT3, DALLE, and most of the papers coming Google, OpenAI, Meta, Microsoft). More than replication, it would require fully open science where all the experiments and results of a paper are publicly available, but I doubt tech companies will agree with that.
Ultimately it could also end up with researchers only trusting papers coming from known labs/people/companies
Kinda needed to be, given the rise of computer-generated proofs starting with the 4-colour theorem in 1976.
Do you mean proof assistant like Lean ? From my limited knowledge of fundamental math research, I thought most math publications these days only provide a paper with statements and proofs, but not with a standardized format
Code from ChatGPT could very well end up processing data from each of them — I wouldn't be surprised if it already has, albeit in the form of a researcher playing around with the AI to see if it was any use.
Reviewers work from an assumption that the data is valid, and reproduction (or failed reproduction) of a paper happens as part of the scientific discourse after the paper is accepted and published.
It is a tool. Ultimately researchers are responsible for their use of a tool, so they should check the abstract and make sure it is good, but there’s no reason it should be seen as a bad thing.
Yeah well we can't tell that now either. Maybe we can finally start publishing raw data alongside these "trust us we found something" papers that people evaluate based on the reputation of the journal and the authors.
As someone else pointed out, that system has already derailed decades of Alzheimer's research. It's stupid and broken and it should have changed a long time ago.
https://www.science.org/content/article/potential-fabricatio...
And if AI can manage that, well: https://xkcd.com/810/
Nobody takes a serious decision reading only the abstract. Look at the tables, look at the graphs, look at the strange details. Look at the list of authors, institutions, ...
Has it been reproduced? Has the last few works of the same team been reproduced? And if it's possible, reproduce it locally. People claim that nobody reproduce other teams works, but that's misleading. People reproduce other teams works unofficially, or with some tweaks. An exact reproductions is difficult to publish, but if it has a few random tweaks ^W^W improvements, it's more easy to get it published.
The only time I think people read only the abstract is to accept talks for conference. I've seen a few bad conference talks, and the problem is that sometimes the abstracts get posted on like in bulk without further check. So the conclusion is don't trust online abstracts, always read the full paper.
EDIT: Look at the journal where it's published. [How could I have forgotten that!]
How can we yield results from an industry being lead by automated derivatives of the past?
Is an AI-generated result any less valid than one created by a human with equally poor methods?
Will this issue bring new focus on the larger problems of the bloated academic research community?
Finally, how does this impact the primary functions of our academic institutions... teaching.
Even a human researcher need experiments to validate ideas. AI can generate plausible ideas, so why not run the experiment and let it learn from the outcomes? The source of learning comes from experimentation, that's how models escape the derivative trap. AlphaGo invented move 37, proving AI can be creative and smart.
> I'm quite confident that there are cliques within "science" which are admitted without as much as a glance at the body of the papers.
It's possible. I don't know every single area, but I guess it's not common in most serious branches od science.
> Some people simply cannot be bothered to get past the paywalls, others accept on grounds outside the content of the paper, like local reputation or tenure.
My friends says they can ask Alexandra for a copy. A few years ago it was also common to ask a friend in another site that has a copy.
> Others are asked to review without the needed expertise, qualification, or time to properly understand the content. Even the most honorable reviewers make mistakes and overlook critical details. Then there are the set of papers which are (rightfully so) largely about style, consistency, and honestly, fashion.
That's why you RTFP instead of thrusting the journal or the reviewer or just the abstract. I've seen "dubious" papers published in good journal.
> How can we yield results from an industry being lead by automated derivatives of the past?
It's running now in natural intelligence. Every time there is a interesting paper, other groups run to publish variants or combinations with other result to reach the anual quota, or get enough point for the graduate student. Once the GAI can read all papers and combine them, there will be very few low hanging fruits to pick.
> Is an AI-generated result any less valid than one created by a human with equally poor methods?
For now, AI is to stupid to get the details right. That's why it's important to read the full paper instead of only the abstract. One AI is intelligent enough, the result may be as bad as a cheating human. Let's hope the AI has a good PRNG to create credible noise, because that's a part humans do badly.
> Will this issue bring new focus on the larger problems of the bloated academic research community?
Nah.
> Finally, how does this impact the primary functions of our academic institutions... teaching.
I don't understand the question. You fear that professor in some areas will just send fake papers written by ChatGPT4.0 and the journal and the community will not notice? There are a lot of predatory journals, open-peer-review journals and other bad journals that are publishing a lot of crap. Usually by professor in bad universities or just as a fancy achievement for the c.v.. A good AI will increase the amount of crap, but it will be just ignored.
No, actually I'm curious if this could open up the schedule for professors who want to spend more time teaching to design and develop better curriculums for their students. But I'm probably being overly optimistic about the number of profs who actually want to teach.
Also, accumulating published papers is important to get a new position in the future, so only teaching is a risk for your future career.
For teaching some of the topics to advanced students, it's important to have people that do research and is updated with the cutting edge results. Also to babysit the graduate students so they can publish their results. For teaching the students in the first years, research is probably overrated.
It's just been a while since I was inspired by anyones research I guess.
You can explain it to a technical friend.
Start explaining about the the two possible spin of electrons, and how it cause the existence of two currents inside a conductor. This is not important in a normal conductor like cooper, but it's important inside a magnetic conductor like iron. So yo get a different resistance for each one of the two currents, the one that has a spin that is in the same direction than the magnetic field and the other with the oposite spin.
You can make a sandwich of iron-cooper-iron. If there is no external magnetic field, the two iron parts have oposite magnetization and the total resistance is higher. If there is an external magnetic field, the two iron parts have the same magnetization and the total resistance is lower. Anyway, the difference is not too much.
[Ideally, your fiend should be boring now, uninterested about the abstract currents no one cares about.]
It get's more interesting if you have many layers of iron and cooper, because the difference is higher and they call is "giant". It is used in the heads to read hard disks, like in the laptop of your friend. [Your friend will never see it coming!]
It's interesting because it mix weird abstract quantum properties and engineering to make it more efficient and you get a device that everyone has. For some weird reason, no one talks about it. And the sad part is that SSD are killing the punch line of the story :( .
I am not sure if this is sarcasm or not.
Literally the whole world besides the researches read mainly the abstract and make even life changing decisions based on it. (just look at any twitter discussion linking to a paper)
I don't consider twitter discussion "serious decisions". When a thread here reach 100 comments, the quality of the discussion usually drops. I don't want to imagine how bad it's in twitter.
The problem is people overdosing from snake oil they read in twitter. I think there were a few cases with ivermectine, hydroxychloroquine and chlorine dioxide [1]. People should not self medicate, or at least understand proportions and that the dose makes the poison. They should see a medical doctor [2], that at least understand proportions and that the dose makes the poison and that they should follow the advice from the FDA made by people that read a few of research papers and extended reports, instead of only one press release about a preprint written by a moron that somehow got a position in a university or hospital.
I guess politician don't read the full paper (except Merkel?), but they hire experts to read the articles and give advice. If you are a politician and the "expert" is reading only the abstract without checking the full paper and the journal and other related stuff, you should fire the moron.
[1] The chemistry of chlorine dioxide is too simple and it's obvious that it can't possible work, so it makes me angry. The other stuff don't work, there were no good reasons to expect them to work, but at least it's not obvious with elementary chemistry that it can't work.
[2] I have horror stories about medical doctos too. Always ask for a second opinion for important stuff.
Is ChatGPT located in a central repository or cloud? Is centralized? If that, probably a bad idea.
A private company having access to your abstract before you publish it could easily lead to problems like plagiarism (even worse, automatized plagiarism) or give an unfair advantage to one in two teams running to publish the same result. Science has a lot of this cases.
https://en.wikipedia.org/wiki/Sokal_affair
It's a single data point. Did anyone ever claim the editorial process of Social Text caught 100% of bunk? If not, how do we determine what percent it catches based on one slipped-through paper?
I'd expect scientists to demand both more reproducibility and more data to draw conclusions from one anecdote.
This would seem to me a situation that is easily exploited by an AI that generate plausible text. If you pack enough jargon into your paper you will probably make it past several layers of review until someone actually sits down and checks the math/consistency which will be, of course, off in a way that is easily detected.
It's a problem academia has in general. Especially in STEM fields they have gotten so specialized that you practically need a second PhD in paper reading to even begin to understand the cutting edge. Maybe forcing text to be written so that early undergrads can understand it (without simplifying it to the point of losing meaning) would prevent this as an AI would likely be unable to do such feat without real context and understanding of the problem. Almost like adversarial Feynman method.
If ChatGPT can't do that (i.e. if it's attaching abstracts disjoint from the paper body), it's not the right tool for the job. A tool for that job would be valuable.
The day is already too short, with an expansion of journals. But, there's a sort of silver lining: many folks restrict their reading to authors that they know, and whose work (and writing) they trust. Institutions come into play also, for I assume any professor caught using a bot to write text will be denied tenure or, if they have tenure, denied further research funding. Rules regarding plagiarism require only the addition of a phrase or two to cover bot-generated text, and plagiarism is the big sin in academia.
Speaking of sins, another natural consequence of bot-generate text is that students will be assessed more on examinations, and less on assignments. And those exams will be either hand-written or done in controlled environments, with invigilators watching like hawks, as they do at conventional examinations. We may return to the "old days", when grades reflected an assessment of how well students can perform, working alone, without resources and under pressure. Many will view this as a step backward, but those departments that have started to see bot-generated assignments have very little choice, because the university that gives an A+ to every student will lose its reputation and funding very quickly.
But I'm lost at what those scientists are trying to find.. (?)