This is going to be like Photoshop usage. Very quickly it will be expected that papers use AI to write parts, just as fixing white balance in Photoshop doesn’t require alerting the viewer that the photo has been enhanced. More extensive use may need to be acknowledged. But I’m not sure…design tools are rarely credited except in cases like a Fx heavy movie
Absolubtely. Im involved in writing right now and i've caught people using its output. It's hard to make an outright ban, heck ive used it myself for things like tricky LaTeX formatting (make this stuff into a nice table with color boxes corresponding to the rgb values).
But as with all papers the issue is just a lack of dilligence, be it people straight up using hallucinated sources or the paper bizzarely fawning over every aspect of the project, even things that are objectively rubbish and author knows that when questioned (such as a paragraph praising the high quality of servo motors used when they are blamed for test failure in a later section). Someone taking the time to use it well will use it well, someone lazy will still be lazy.
I clearly remember getting in trouble in class each year for using Wikipedia.
My podunk teachers couldn't fathom the difference between directly citing Wikipedia (just cite the source it references instead) and merely reading it.
Some teachers notice if you only cite sources on the Wikipedia page, too. More likely if the whole class is writing on the same topic.
The work-around is to write it from Wikipedia, but find sources by searching on Google Books. Being able to read a couple pages around each part you’re citing is enough, so their page allowance for restricted works is usually plenty.
Hit your citation count and real-book-source requirement (if they still do that second one) without leaving your chair. Barely more work than writing from Wikipedia and not covering your tracks.
My wikipedia account is over two decades old, and I often bring this up when talking bee -keeping stuff... them both being "bottom up organization that results in a beautiful exchange of information."
This experience "went full circle" for me, January 2023... when ChatGPT cited SOMETHING I HAD WRITTEN into wikipedia (re: transistor density).
This is a more serious problem than that, because AI does in fact make stuff up, and will bullshit logical connections that make superficial sense but don't match the system under discussion. I think a better analogy is airbrushing photos to remove people; it changes the facts of the photo in a way people care about. Likewise, it's a problem if people sloppily use GenAI to fix up text or graphs or whatever, and it creates things that match the paper really well but don't match reality. Those are _exactly_ the kinds of problems that will slip through peer review very easily.
Fixing white balance, or a thousand other old-school photo changes, is very different; the underlying structure of the photo is there and nobody cares about white balance or most of those old corrections, because our eyes and brains are constantly mucking with them anyway. People care about airbrushing or GenAI-"airbrushing" parts of the picture, because the information content we care about is changed.
> I think a better analogy is airbrushing photos to remove people; it changes the facts of the photo in a way people care about.
I was thinking about this a lot recently with the announcement of the Pixel 8. One of the big announced features was "Magic Editor", where you could edit all your photos to automatically put smiles on people who were frowning, remove people altogether like that inconvenient homeless person or garbage truck: https://blog.google/products/pixel/google-pixel-8-pro/
This may be a bit "get off my lawn", but honestly I was disgusted. Why do people feel their everyday family photos need to match some Insta-bullshit ideal? Some of my favorite old family pics are where something "wrong" or unexpected happens.
I don't fault Google for this, and I think it's natural to want to touch up photos when you're taking an important picture. But I feel like we're careening towards a future where we're starting to forget what reality actually is. When TikTok came out with their extremely realistic "makeup filters", I remember seeing a video where a photographer commented that she would take pictures of women who were shocked at how "ugly" they were because they weren't used to seeing "un-Facetuned" photos of themselves.
Sorry for the semi-rant, just feels like a situation where any individual, small technological improvement makes sense, but in totality we end up at a place that is overall worse than where we started.
I think that once you start editing a picture in a way that isn’t equivalent to just having different camera settings you aren’t capturing a moment anymore, at best you’re creating a new artwork which is fine but I don’t think it’s good to pretend to yourself or others that it’s actually what happened
Agreed, but that is clearly not how Google is positioning "Magic Editor":
"Magic Editor in Google Photos is a new experimental editing experience that uses generative AI to help you bring your photos in line with the essence of the moment you were trying to capture."
After all, why take a picture of your family when your kid is making a silly face when you can instead "fix" him so that you now look like the Stepford-family, JC Penney ad that God intended.
I don't think authors need to credit AI in their written works, but I do think this is the double-edged sword of claiming partly AI-generated content as your own (for reference, the WGA recently won the right to claim AI-inspired and partially AI-created work as a human writer's own work without having to credit the AI). I think it's fair for a scientific human author to take full credit for papers that are partially written by AI, but any logical errors and hallucinations inserted by AI should also be attributed to the human author. It's the human author's responsibility to ensure the final written work is accurate.
Edit: for further context, I have published a few scientific papers in recent years, and using AI is not all that different from a senior author relying on junior authors. Early on in my career, I created the first draft of most papers, and my more senior co-authors would verify my work and create direct edits or suggestions for me to revise the draft. The final paper was created after a few rounds of this cycle.
I don't mean to be super nitpicky, but I've got to strongly disagree with your comparison to photoshop / photography.
There's been an ongoing debate in the Photography community going back as far as photographic editing has existed, regarding when and how acknowledgement of editing has been performed, whether it be literal, physical cutting and pasting of printed media, or a digital equivalent.
Some of the more famous personalities in Photography seem to lean to something like this:
"Minor adjustments like removing dust from the lens, adjusting colors to match the 'true' colors as it was taken (or to make it 'more visually pleasing'), cropping and level correction are fine and expected. The use of things like Sky Replacement [1] or content-aware fills [2] should be attributed or referenced somewhere if they are the focal point of the photograph."
There are others who do much more "artistic" styles of photography [3] where they argue something like:
"No person in their right mind would believe that there isn't some amount of editing going into nearly photograph, so why mention it at all?"
or
"Because editing has been so heavily used, by so many famous photographers for so long, without mentioning editing, it's fine if people don't mention the use of digital (or physical) editing."
or
"Popular services like Instagram have built-in filters that users can use to make themselves look different - should the people who post photos that have those filters on also be pressured into mentioning that their photo was edited? If not, what's the difference?"
There's really not any solid consensus on what is "correct" when it comes to editing attribution. Most tend to agree that doing the bulk of the work "in-camera" (cropping by moving in closer, for example) will make your life easier later on, but, there are some outliers that strongly advocate that any form of editing in post, is "not photography".
Things get even weirder when you get into print. The same photograph printed on different mediums of paper, or with different printers, or different ink, or different DPI settings can come out looking noticeably different even to laymen (imagine a smeared blotchy photo versus a crisp, clear one - there are times where you might want the photo to look somewhat blotchy for aesthetic reasons). In a way, choosing one form of medium over another is a kind of editing, because the end presentation is measurably different from the original. A quick example might be noticing the difference between modern movies that are shot on digital vs traditional film. There is a kind of "feel" to the end product, because of the medium that was used to record and distribute it.
TL;DR The photography community is very much not in agreement to what amount of editing is, or is not permissible, or when it should, or should not be attributed or brought up, or even how that information should be conveyed. Some are fine without any mention whatsoever because of historical reasons. Others want it mentioned under the use of certain tools or criteria. Still other don't consider a photographer to be a "real" photographer at all, and instead should be called a "stills editor" (or something like that) if they use any editing at all, outside of "in-camera work".
Good points. As you point out, it depends on the community. Photojournalism and nature photography have much higher standards for what is acceptable than advertising or personal photographs and artistic photographer is different again. I remember when Galen Rowell burned a hole in a slide with a loupe and had to write an apology for fixing it digitally.
And you could make that analogy for writing too. For creative writing, knock yourself out (though there is already a raging debate on this topic), but for scientific writing you need to be more careful. If I was going to extend the photo journalism example to papers, then there is a difference between using AI to change the data (change the sky) than as a grammar checker (fix the white balance).
Ah, I think I see the error that I made. When I read your comment about white balance, I took it to mean "things like white balance" as in "almost everything", not just minor corrections. My mistake! :)
Cropping can make a huge difference in the meaning or messaging imparted by a photograph. It's akin to picking a phrase out of a longer speech or passage and interpreting it by itself. It's potentially a far more impactful edit than color balancing, adjusting contrast, or those sorts of things. On the other hand it could just be removing an irrelevant distraction that was captured in the original framing.
Totally agree and I thought about that example. But cropping is an inevitable part of photography (or painting) and was done in camera or in the darkroom before digital tools with identical power and results (albeit, cropping digitally is a lot, lot, lot faster). To push this analogy to its breaking point, cropping digitally is like using a keyboard to write instead of a pen.
The hilarious part isn't that a human didn't write the paper. Instead, consider that in some obscure environmentalism journal, a paper might only ever be read by a few dozen people around the world anyway. Even a good paper on some groundbreaking study might have very narrow interest after all.
But we're fast approaching an era where only the LLMs will read the thing, and only to spit out a summary of it. No human wrote it, no humans will read it.
The biggest journals will of course avoid this fate the longest, maybe even putting it off indefinitely. And some of the most useless journals have been doing worse for a long time. At some point though, doesn't academia have to wake up and realize how pointless all of this is?
I was thinking something a bit similar about news in the future. I’m already thinking I should try to put together an automated daily briefing for myself using a list of sources and ask LLMs to write a summary briefing. But then also consider in the future the news sources will also likely be generating some of the articles with LLMs. This LLMs reading LLM output is a curious development. Same for emails also btw. I already have an LLM redraft most of my emails as it makes them more readable and concise.
It's darkening the public usage, the private channels for research and communication will become disproportionately more valuable.
It's odd not to see Google or Meta try and make a copyright strike system for written text. Copy and paste LLMs are driving people away from google's ad bucks, filling FB with junk. I guess 'the web' is no longer worth defending, de facto.
This has been true for a very long time. A lot of writing is perfunctory. It's written just because it is required, but nobody really cares what it says. It's just a cover sheet on a TPS report.
It's easy to forget how many people don't actually have English as a first language.
With so many journals demanding papers submitted in a language not native to the author, use of assistive technology to convert the paper so the underlying information can be made known should be expected.
If it's just a better Google Translate, then I don't see the problem. If it's used to actually write the first version because people can't write or sort out their ideas... I dunno.
Is there any evidence that it's primarily used by non-native English speakers for this purpose, though? (edit: just saw the list of authors, probably this is the case, at least in this context).
> If it's used to actually write the first version because people can't write or sort out their ideas... I dunno.
For what it's worth, I'm also 10,000% on board with that use case. I have definitely stared at a blank piece of paper with writer's block because I couldn't write the introduction for a paper before. If ChatGPT gets somebody past that block, more power to it.
Every paper is required to start with some novel variant of "from the dawn of time humanity has wondered if..." when all you want to say is "We discovered some stuff, here it is." I am super in favor of automating the step that nobody wants to write, nobody wants to read, and no journal will accept without its presence.
Reporting novel scientific findings was never supposed to be an exercise in fluent English authorship mastery.
I haven't written scientific papers, but for other reports or writeups I always start with the meat of what I want to say. I write the introduction later, once I know what it is that I'm introducing.
> I am super in favor of automating the step that nobody wants to write, nobody wants to read, and no journal will accept without its presence.
You make a good point. Maybe the bits that can be automated are the zero-value bits, and therefore the best conclusion would be to get rid of them?
In other words, if it can be automated, it's not real content.
I'd be wary of people writing summaries or conclusions using ChatGPT though, because ChatGPT can write very convincing crap that completely misses the point. Someone who is careless enough to forget removing the "Regenerate response" part of the output may be as careless reviewing whether the conclusion truly says what they meant it to say.
As an aside, I've asked ChatGPT 3.5 to provide interpretations of works of fiction of which I know the most common ones (e.g. "this is a parable about [...]", "this is a warning against authoritarianism", etc), and ChatGPT sometimes comes up with stuff that completely misses the point; but if you aren't familiar with the work (haven't read it except for a summary) you could mistake it for a valid interpretation!
Could this just be a translation? A non native speaker pasting their paper into ChatGPT and asking it to translate it to English, and then copying the results and hoping for the best?
Well a lot of the more egrerious errors ive seen when trying to get it to write academic writing comes from it not having a full context of what the problem is/whats been done/ what you are trying to do. Ive been turning over the idea in my head that for one to fully give it all the required context one might as well just use that context as the piece of writing in the first place. There is no advantage into stretching the relevant information to 6 paragraphs instead of 2. Not to say it should never be used but trying to use it while being thorough and actually reading its output has usually led me to extending context more and more and eventually just writing the section myself.
I don't think you should just blindly accept the results. I just write what I want to say stream of consciousness, drop it into chatgpt and then review the result to see where it improved and where it didn't.
A prompt such as "Why are all the dolphins dying?" would be unacceptable.
A prompt such as rewrite "explain that the dolphins are dying because 30 degree rise in ocean temperaturs made their environments unlivable" that reformats the language into a academic style I feel is totally acceptable and efficient if you read what it spits out and agree and edit from there.
I am very sympathetic to people who are using it to write better in language X when their native language is Y.
I think most other people should not use it for writing, because the hard part is thinking about (A) what you _really_ want to say (B) what your audience cares about (C) mistaken assumptions by you or your audience, which need correction in the text. And I think people using ChatGPT to write will give it a vague idea of (A) and usually nothing about (B) and (C).
If you look at the papers from the list, some of the authors respond to the comment which asks for clarifications on ChatGPT usage (for those who didn't read TFA, this is detected mostly because they cut & pasted ChatGPT's response without removing extraneous parts like "Regenerate response" from the UI! So not only do they rely on ChatGPT, they also don't know how to use it well).
What's interesting is that even when they reply, authors don't seem to respond to the ethics/standards issue of actually using ChatGPT. Well, some do ("I will not do it again" is basically what one of the authors says, which reads like a childish response to me), but many claim it was a "mistake" -- the mistake being... forgetting to remove the boilerplate and "regenerate response" garbage! Uh-oh.
I think an acceptable use of AI would be in translation, basically a better Google Translate. But is this what's happening here?
I have been experimenting with generating powershell scripts using chatgpt, and aside from a few syntax errors or localization, the scripts are a great starting point.
Instead of writing the whole script. I can have ChatGPT write it, then I just have to clean it up, address any errors and localize.
Yeah, exactly this. Paragraphs of condensed scientific text after six rounds of collaborator revisions are typically a mess - ChatGPT is great for suggesting a more readable alternative, and then further crafting into something actually-good.
I think a lot of the hand-wringing neglects that there's a range of ways to use LLMs. Text-refactor is obviously perfectly ethical, unless you're gunning for the Butlerian Jihad, no worse than the grammar checker in Word.
Do note TFA is about using ChatGPT to write parts of research papers without disclosure. Nothing to do with using it to write code.
Hence my closing line, "[in the context of writing research papers] a valid use of AI would be as a better Google Translate [for non-native English speakers]."
The papers' authors may be trying to translate their broken English into proper English, and might be asking ChatGPT to regenerate a "translation" that changes the meaning.
"Regenerate response" [1] is what appears at the bottom of every ChatGPT response. It doesn't even mean they clicked on it; it just means they carelessly cut & pasted the response without cleaning it up, which carried over bits of ChatGPT's UI.
Whether or not they used ChatGPT just as a translation service (an acceptable use, in my opinion), in many cases they didn't disclose this usage even when the rules of submission mandated it.
--
[1] I just checked and now it just says "Regenerate". It must have been "Regenerate response" in the past.
There's a huge fraudulent scene when it comes to scientific papers, IMO largely driven by the "schoolification" of universities. Many authors seem to consider writing and publishing a research paper as just another form of an exam and thus cheating as the logical approach.
It's not so much the undeclared use of ChatGPT that bothers me but the egregious sloppiness of it. It was already obvious that writing scientific papers has evolved into a cargo cult, but this shows that it's much worse than previously thought.
Authors, referees, and journals go through the motions of what lead to genuine scientific advancement in the past, but no actual value is being created at any step in the process for that vast majority of papers. Of course most papers will never be ground breaking but at least in the past there was some value being created in the process.
It seems totally unclear whether ChatGPT is being used to invent fictional methodology or literature review etc., which is extremely unethical, or whether it's simply being used to check and fix grammar and tone for non-native speakers, which is perfectly fine.
But regardless, this is extremely troubling in terms of quality. The fact that "regenerate response" gets left in tells us that even if it's just for grammar correction:
1) The authors aren't even minimally reviewing if the ChatGPT version didn't accidentally change the meaning of the sentences
2) Peer reviewers clearly aren't even reading the paper
3) The journal isn't even reading the paper to introduce a minimal level of quality control
ChatGPT seems to be the smallest problem here. And thankfully, it is revealing much more serious problems about quality checks in academia at all.
The question is, will anything be done about it? Not about ChatGPT, but about the horrifically irresponsible processes that would let these slip through?
EDIT: Oh, wow. Lack of coffee this morning. I read that as "fictional mythology", NOT "fictional methodology."
Ignore me. :)
-----
I want to pull apart one specific thing you mentioned, essentially:
> It seems totally unclear whether ChatGPT is being used to invent fictional methodology [...], which is extremely unethical
In the context of generating something fictional, why is using ChatGPT unethical?
One reason I'm asking: I have always been a big fan of NaNoWriMo[1] (National Novel Writing Month, every November). This year, I'm interested in exploring how to use ChatGPT -- not necessarily to generate a ton of prose, but to essentially help flesh out some plot ideas that I've had.
Part of the back and forth I've had with it is coming up with some interesting scenarios for my story that I hadn't really thought about and I'm keen on exploring a bit more.
I am a ding dong. I read that as "fictional mythology" and my mind immediately went down a different rabbit hole. Yes, I see now why that would be unethical.
But ChatGPT isn’t deciding to use fictional methodology in its paper, it’s being asked to generate a fictional methodology and complying.
The human is the problem in that scenario, not the LLM. The unethical choice happens at the point of deciding to lie about methodology, before LLM is involved.
It may seem like a meaningless distinction, but it’s the difference between LLMs causing trouble and people causing trouble, which is important for the larger debate.
Not meaningless, but a little obvious. I don't think anyone was accusing ChatGPT of being duplicitous.
I think one problem with LLMs and the hype around them is that people are placing far too much confidence in them, and therefore misuse them.
That it's even conceivable that people might blame a tool for unethical behaviour is an indicator of this.
The idea that an LLM would be able to correctly generate your methodology for you is another.
I find ChatGPT simultaneously incredible and terrible. It has no concept of what it knows and what it doesn't, so there's nothing I would trust it to do without a human checking the output.
This reminds me of when I asked it to generate a fake research paper describing how my dog is the best in the world as a joke and it refused because apparently creating fake research papers is unethical, even when it’s an obvious joke.
I asked again in a new chat and it made me a really nice one with funny things like a formula showing the amount of force he uses to open doors and fake data proving statistically that he is the best
They mean generating fictional methodology and then passing it off as real in a scientific paper is unethical. This is also unethical if you don’t use ChatGPT, of course
If I recall correctly, a number of years ago, plenty prior to ChatGPT, there was an incident with a scientific journal (don't recall the name though). Perhaps others here may know of this incident too...
So apparently someone submitted a paper that was entirely nonsense. This was intentional but not explicit as it was just a stunt. The paper had all the right fancy/technical words, the right structure, likely even images, and apparently a convincing conclusion. But it was all about nothing, nothing substantial at all, nada. Had a peer reviewer actually read the paper, it would have not been published and summarily tossed. But it did go print and oh boy, the journal's board/powers-that-be were pissed. I didn't follow the story much past that point but I think something was supposedly proposed to prevent this.
I guess that would be it! Thanks for pointing out that Wikipedia entry.
I was in college at the time majoring in Physics, and yep, that was one of the headline news among a journal or two that arrived there. Good memories, indeed...
I can confirm using chatGPT for paraphrasing abstract, conclusion etc but I always edit it to make sure that it conveys the same meaning and has no errors.
That is true in principle. But as scientific papers, this is supposed to be extremely precise, highly iterated, and thoroughly reviewed content.
If there are very obvious mistakes like a leftover pasted in button text, it calls into question whether the authors (or reviewers) were responsibly verifying the content contributed by a tool that is known to sometimes deliver wildly but subtly incorrect results, and thus the quality of the paper content as a whole.
This just let's you see which writers are incompetent which is a good thing. People who copy and paste chat GPT without checking would have shit papers regardless.
1. Bad grammar in a paper is incredibly irritating, which is particularly unfair for people who are non-native English speakers. If ChatGPT can help people to convert awkward English into nicer work without changing the ideas or confabulating, that seems like a win. I don't see why people need to "declare" their usage of ChatGPT for these purposes. Is that a standard now?
2. The major issue here is that the authors copied a response and accidentally also brought some button text with it. This is given to imply that people aren't reading ChatGPT's output. But I think this is just as likely to be benign. People do all sorts of silly nonsense in papers (including me), like leaving author comments in final drafts. It's bad and sloppy, but it isn't usually a crime.
3. Many papers on public servers are junky. Many journals and conferences are equally junky. Search for any bad string and you'll find it everywhere, and non-scientists will be scandalized.
4. I like the people at Retraction Watch and think they're doing the Lord's work. However sometimes I worry that immersing yourself in wickedness makes you see wickedness everywhere, and also creates an (unconscious) incentive to sensationalize bad stuff. I hope they're aware of this.
If I type a series of bullet points into ChatGPT and ask it to re-phrase them in better English prose, am I plagiarizing? If it's not plagiarism but I screw up at copy/pasting, does it become plagiarism? I don't know! It is not something I personally would ever do because I like my own writing. But I'm also not going to throw that accusation at ESL authors who are using a writing tool (like Autocorrect-on-steroids) to convey original ideas to an audience, at least not until we've had a much longer community discussion on using these tools.
ETA I would urge you not to throw the death penalty at people whose crime may be more akin to shoplifting. This is bad both for the shoplifters and for people who want to fight serious crime. Plagiarism is one of the most serious accusations in science: casually tossing it around like this risks our ability to enforce it against serious offenders.
It's a difference in quality large enough to constitute a difference in kind. Autocorrect nominally takes text in your own words as typed and converts them into your words as intended. Chatgpt used as you describe does not produce text in the author's own words.
As children in grade school we are taught a series of rules about plagiarism that are designed to make us extremely sensitive about what we copy from. We're taught this because if you let a group of children do whatever they want, they'll copy stuff straight off Wikipedia or each other and thus they won't learn how to write properly. So we teach them that writing every word yourself is the most important goal to aspire to.
As an adult in a scientific field, what actually matters to me is having original scientific ideas conveyed to me in an efficient manner. I'm not really interested in applying arbitrary elementary-school rules on authors, as long as they aren't stealing ideas and text from other people (thus misallocating credit in ways that are harmful to progress.) I think as a community we should proceed from these principles, then work the details out from there.
> As an adult in a scientific field, what actually matters to me is having original scientific ideas conveyed to me in an efficient manner. I'm not really interested in applying arbitrary elementary-school rules on authors, as long as they aren't stealing ideas and text from other people.
The elementary school rules, at least the ones I weas taught decades ago, are pretty much “don’t steal ideas and text from other people”. Like, literally, other than for formatting, there were exactly three rules avout plagiarism I was taught on this in elementary school, and only one of them becomes slightly fuzzier outside of that context:
(1) if you use exactly wording from someone else, put it in quotes and cite the source,
(2) if you use an idea from someone else, cite the source but don't ise quotes, and
(3) if an idea comes from three or more independent sources, you don’t need to cite it, it can be treated as “common knowledge”.
Outside of elementary school, yeah, whether something is genuinely common knowledge or should be cited to multiple sources is fuzzier and context dependent, but the other two rules apply pretty directly.
If the source is Microsoft Word's grammar correction engine, do we require people to quote and cite that? The question here is whether "quote and cite" should apply to ChatGPT (used exclusively for re-wording existing text.) And I think my principles in the post above provide a better framework for thinking about that question than the grade-school rules (and rules for citing texts, which is again a different situation.)
Then the rules you were taught were far more lax than those I learned. According to the rules I was taught, copying lines from your own prior work without attribution would also be plagiarism.
And it's fine if that's what matters to you, but I think there's legitimate room for contention on the meaning of authorship or plagiarism when it comes to something like this. If a work is written in one language and translated by another person into English, I'd expect the translator to be credited. Is the sole purpose of crediting the translator acknowledging their effort or does it also serve to acknowledge that the other author can't necessarily vouch for translation or as an indication of provence? If it's not translated by a human, should the translation software be similarly credited? If the writing process consisted of one person writing a bulleted list outline and another person converting that to full prose, I'd certainly expect that person to be credited as an author. What about the case where that step was done by software?
My point is that it's not what matters to me. Our rules shouldn't be vacuous, they should be founded in real goals that advance science. In science we want people to communicate ideas clearly, we want the ideas to be accurate, and we want to allocate credit properly to authors. Adding a rule that says "you must say you used ChatGPT or you get punished" is unlikely to advance any real goal -- it will just lead to most authors adding that footnote, and a few people getting arbitrarily punished for forgetting it.
> If I type a series of bullet points into ChatGPT and ask it to re-phrase them in better English prose, am I plagiarizing?
I think the OP's point is that you could be, and should check, as such models sometimes reproduce parts their training set. You should check what you send, or, in other words, take responsibility for the tools you use -- the tools that are bread and butter are somewhat excused, but LLM are not at the moment.
The authors could be required to declare only to the reviewers that ChatGPT was used for the purpose of cleaning up the language. Reviewers could then be alert to check for artifacts like this, verify that no references were hallucinated, etc.
Perhaps a footnote would be added to mention that ChatGPT was used in this way.
I don't know. The use of an LLM to help produce something isn't necessarily a bad thing -- but in my mind, it does reduce my trust in what the thing is saying. It means that if I actually care about what the paper (or whatever) is talking about, I need to spend a lot more time confirming what it's saying. If I don't care much about what the paper is talking about, I'm more likely just to discount the paper and move on.
The challenge here is if their English skills are so poor that they need a ML model to help make it readable, their English skills are likely so poor that they will not spot grammatical changes that adjust the meaning of the work. That's a problem when the next person finds that paper on Google Scholar and cites it - you get a telephone game in the literature. It's hard to remove bad "facts" from the literature after a few people cite it, since you also need to go after a whole chain of papers, hoping their journals are willing to issue a correction. God help you if it's in the intro section and the authors complain because of academia's "corrections are bad" mindset. Anything that increases the BS rate of the literature is a bad thing.
Also, if you go to contact the author or view a talk, and suddenly their English is broken and completely different than what they wrote, you start questioning whether they wrote it or paid someone to without disclosing it.
IMO it's worth disclosing for many of the same reasons you disclose funding, ethics approval, author contributions, and conflicts of interest at the end of an article: it's metadata that protects your credibility as an author by showing that you're not doing anything underhanded.
Peer review is a mixed bag. It's the best system we've come up with, but still can be hit or miss.
Depends on how much the journal is willing to push the authors (predatory journals don't give a heck, prestigious journals usually care more because they've got a rep to uphold). Reviewers can be anyone from an expert in the field, a competitor who gives you a hard time (usually they're not actively malicious though), to some random prof doing it because the editor couldn't find anyone else, or even a grad student in the lab of a prof (it's a common training exercise).
I've had articles with anywhere from 1-3 reviewers, there is no standard beyond "the editor is a basic crackpottery filter, there is an external reviewer, and a copy editor makes it fit for viewing"
If the system is not catching "I am a language model", I have zero confidence in its ability to detect crackpottery, much less more insidious things like P-hacking.
Using a LLM to expedite writing is not fake science.
You can use LLMs to create fake science, but also use it expedite writing up real science. I have experimented with feeding in a paper that I've read, I know is relevant, and direct the LLM to extract this point out and connect it to this other point. I then go through and edit. Its faster to rewrite existing text (for me) than to generate it the first time.
We should NOT be demonizing tools that increase our scientists productivity when they use it responsibly, rigorously, and ethicly.
Does anybody still believe in the peer reviewed journal system? After all these years and the countless stories of passing off fake information?
We know people make up their data. We know peer reviewers sign off on it without actually looking it over. And we know the journals publish it anyway, and make a lot of money off of it.
Why does anybody even care if we've added another layer of fabrication to the mix of something that's already so full of lies?
This feels very dismissive of the efforts put in by genuine participants into the system. Exceptions and examples of malfeasance are not sufficient to indict all aspects.
I believe something that is not falsifiable is not science. So what would failure look like to you if it isn’t “> 50% off science is fake”? What do you call failure of this system?
A system that generates plausible, seemingly authoritative information, but often makes hard to detect errors ranging from minor to outright lies is dangeorous. This goes double when the information is either difficult or impossible to verify.
This shouldn't be surprising, since the most effective and dangerous liars tell the truth most of the time.
> but often makes hard to detect errors ranging from minor to outright lies is dangeorous.
Oh, I agree. However, the peer-review system does mitigate this problem to a useful degree. Without such a system, it would be impossible to have any degree of trust in any paper at all.
> What we're really finding out is that journals are often little better than LLMs as information sources.
It depends on the journal. There are absolutely crap ones out there that need to be ignored. But there are also good ones out there that have earned their reputation. Even there, BS can get through of course -- but to say that such journals are mostly unreliable is seriously overstating the issue.
What percentage of papers in your average, reputable journal have been replicated?
And how can one easily determine, while looking at a particular paper, whether it has been replicated? And whether those doing the replication have any undisclosed ties to the original?
At an epistemological level, the idea of a knowledge source like a journal where the information is only deemed reliable if personally verified seems problematic. Why even have it if all of its uncountable claims are indistinguishable from very clever lies, and attempts to quantify the extent of those lies indicate that they are pervasive?
With this system, I’m many domains, a -majority- of published research is not reproducible. Possibly worse than a coin flip. Do you really think this is “reasonably well”.
I predict that by 2030, probably 99.9% of all content and published discoveries will be AI-assisted, and outcompete manual stuff for grants and money. Manual content will be as niche as knitted sweaters on etsy.
In cases where it's being used to clean up grammar, perhaps the new standard should be providing a diff between the original paper and the cleaned up version.
I don’t really think the issue is that fact that ChatGPT (or perhaps another llm) was used.
The issue is that it was not checked. Perhaps they used it just to translate as a final step and did not have a fluent person to check it over (with the requisite level of understanding) LLMs are new, and soon to be superseded by multimodal AI, so the understanding of them is currently quite immature, in both usage and consumption of.
111 comments
[ 0.58 ms ] story [ 168 ms ] threadBut as with all papers the issue is just a lack of dilligence, be it people straight up using hallucinated sources or the paper bizzarely fawning over every aspect of the project, even things that are objectively rubbish and author knows that when questioned (such as a paragraph praising the high quality of servo motors used when they are blamed for test failure in a later section). Someone taking the time to use it well will use it well, someone lazy will still be lazy.
Exactly. It's a self-solving problem as long as draconian luddites don't outright ban LLMs.
My podunk teachers couldn't fathom the difference between directly citing Wikipedia (just cite the source it references instead) and merely reading it.
Pain.
You also shouldn't cite sources you haven't read yourself.
The work-around is to write it from Wikipedia, but find sources by searching on Google Books. Being able to read a couple pages around each part you’re citing is enough, so their page allowance for restricted works is usually plenty.
Hit your citation count and real-book-source requirement (if they still do that second one) without leaving your chair. Barely more work than writing from Wikipedia and not covering your tracks.
This experience "went full circle" for me, January 2023... when ChatGPT cited SOMETHING I HAD WRITTEN into wikipedia (re: transistor density).
Fixing white balance, or a thousand other old-school photo changes, is very different; the underlying structure of the photo is there and nobody cares about white balance or most of those old corrections, because our eyes and brains are constantly mucking with them anyway. People care about airbrushing or GenAI-"airbrushing" parts of the picture, because the information content we care about is changed.
I was thinking about this a lot recently with the announcement of the Pixel 8. One of the big announced features was "Magic Editor", where you could edit all your photos to automatically put smiles on people who were frowning, remove people altogether like that inconvenient homeless person or garbage truck: https://blog.google/products/pixel/google-pixel-8-pro/
This may be a bit "get off my lawn", but honestly I was disgusted. Why do people feel their everyday family photos need to match some Insta-bullshit ideal? Some of my favorite old family pics are where something "wrong" or unexpected happens.
I don't fault Google for this, and I think it's natural to want to touch up photos when you're taking an important picture. But I feel like we're careening towards a future where we're starting to forget what reality actually is. When TikTok came out with their extremely realistic "makeup filters", I remember seeing a video where a photographer commented that she would take pictures of women who were shocked at how "ugly" they were because they weren't used to seeing "un-Facetuned" photos of themselves.
Sorry for the semi-rant, just feels like a situation where any individual, small technological improvement makes sense, but in totality we end up at a place that is overall worse than where we started.
"Magic Editor in Google Photos is a new experimental editing experience that uses generative AI to help you bring your photos in line with the essence of the moment you were trying to capture."
Then here is the example video they show of a sample photo edit: https://storage.googleapis.com/gweb-uniblog-publish-prod/ori...
After all, why take a picture of your family when your kid is making a silly face when you can instead "fix" him so that you now look like the Stepford-family, JC Penney ad that God intended.
Edit: for further context, I have published a few scientific papers in recent years, and using AI is not all that different from a senior author relying on junior authors. Early on in my career, I created the first draft of most papers, and my more senior co-authors would verify my work and create direct edits or suggestions for me to revise the draft. The final paper was created after a few rounds of this cycle.
There's been an ongoing debate in the Photography community going back as far as photographic editing has existed, regarding when and how acknowledgement of editing has been performed, whether it be literal, physical cutting and pasting of printed media, or a digital equivalent.
Some of the more famous personalities in Photography seem to lean to something like this:
"Minor adjustments like removing dust from the lens, adjusting colors to match the 'true' colors as it was taken (or to make it 'more visually pleasing'), cropping and level correction are fine and expected. The use of things like Sky Replacement [1] or content-aware fills [2] should be attributed or referenced somewhere if they are the focal point of the photograph."
There are others who do much more "artistic" styles of photography [3] where they argue something like:
"No person in their right mind would believe that there isn't some amount of editing going into nearly photograph, so why mention it at all?"
or
"Because editing has been so heavily used, by so many famous photographers for so long, without mentioning editing, it's fine if people don't mention the use of digital (or physical) editing."
or
"Popular services like Instagram have built-in filters that users can use to make themselves look different - should the people who post photos that have those filters on also be pressured into mentioning that their photo was edited? If not, what's the difference?"
There's really not any solid consensus on what is "correct" when it comes to editing attribution. Most tend to agree that doing the bulk of the work "in-camera" (cropping by moving in closer, for example) will make your life easier later on, but, there are some outliers that strongly advocate that any form of editing in post, is "not photography".
Things get even weirder when you get into print. The same photograph printed on different mediums of paper, or with different printers, or different ink, or different DPI settings can come out looking noticeably different even to laymen (imagine a smeared blotchy photo versus a crisp, clear one - there are times where you might want the photo to look somewhat blotchy for aesthetic reasons). In a way, choosing one form of medium over another is a kind of editing, because the end presentation is measurably different from the original. A quick example might be noticing the difference between modern movies that are shot on digital vs traditional film. There is a kind of "feel" to the end product, because of the medium that was used to record and distribute it.
TL;DR The photography community is very much not in agreement to what amount of editing is, or is not permissible, or when it should, or should not be attributed or brought up, or even how that information should be conveyed. Some are fine without any mention whatsoever because of historical reasons. Others want it mentioned under the use of certain tools or criteria. Still other don't consider a photographer to be a "real" photographer at all, and instead should be called a "stills editor" (or something like that) if they use any editing at all, outside of "in-camera work".
[1] https://helpx.adobe.com/photoshop/using/replace-sky.html
[2] https://helpx.adobe.com/photoshop/using/content-aware-fill.h...
[3] johngossman ↗ Good points. As you point out, it depends on the community. Photojournalism and nature photography have much higher standards for what is acceptable than advertising or personal photographs and artistic photographer is different again. I remember when Galen Rowell burned a hole in a slide with a loupe and had to write an apology for fixing it digitally.
And you could make that analogy for writing too. For creative writing, knock yourself out (though there is already a raging debate on this topic), but for scientific writing you need to be more careful. If I was going to extend the photo journalism example to papers, then there is a difference between using AI to change the data (change the sky) than as a grammar checker (fix the white balance). registeredcorn ↗ Ah, I think I see the error that I made. When I read your comment about white balance, I took it to mean "things like white balance" as in "almost everything", not just minor corrections. My mistake! :) SoftTalker ↗ Cropping can make a huge difference in the meaning or messaging imparted by a photograph. It's akin to picking a phrase out of a longer speech or passage and interpreting it by itself. It's potentially a far more impactful edit than color balancing, adjusting contrast, or those sorts of things. On the other hand it could just be removing an irrelevant distraction that was captured in the original framing. johngossman ↗ Totally agree and I thought about that example. But cropping is an inevitable part of photography (or painting) and was done in camera or in the darkroom before digital tools with identical power and results (albeit, cropping digitally is a lot, lot, lot faster). To push this analogy to its breaking point, cropping digitally is like using a keyboard to write instead of a pen.
But we're fast approaching an era where only the LLMs will read the thing, and only to spit out a summary of it. No human wrote it, no humans will read it.
The biggest journals will of course avoid this fate the longest, maybe even putting it off indefinitely. And some of the most useless journals have been doing worse for a long time. At some point though, doesn't academia have to wake up and realize how pointless all of this is?
It's odd not to see Google or Meta try and make a copyright strike system for written text. Copy and paste LLMs are driving people away from google's ad bucks, filling FB with junk. I guess 'the web' is no longer worth defending, de facto.
The is probably going to be true for other types of documents as well, such as business emails. See this article: https://marketoonist.com/2023/03/ai-written-ai-read.html
With so many journals demanding papers submitted in a language not native to the author, use of assistive technology to convert the paper so the underlying information can be made known should be expected.
Is there any evidence that it's primarily used by non-native English speakers for this purpose, though? (edit: just saw the list of authors, probably this is the case, at least in this context).
For what it's worth, I'm also 10,000% on board with that use case. I have definitely stared at a blank piece of paper with writer's block because I couldn't write the introduction for a paper before. If ChatGPT gets somebody past that block, more power to it.
Every paper is required to start with some novel variant of "from the dawn of time humanity has wondered if..." when all you want to say is "We discovered some stuff, here it is." I am super in favor of automating the step that nobody wants to write, nobody wants to read, and no journal will accept without its presence.
Reporting novel scientific findings was never supposed to be an exercise in fluent English authorship mastery.
You make a good point. Maybe the bits that can be automated are the zero-value bits, and therefore the best conclusion would be to get rid of them?
In other words, if it can be automated, it's not real content.
I'd be wary of people writing summaries or conclusions using ChatGPT though, because ChatGPT can write very convincing crap that completely misses the point. Someone who is careless enough to forget removing the "Regenerate response" part of the output may be as careless reviewing whether the conclusion truly says what they meant it to say.
As an aside, I've asked ChatGPT 3.5 to provide interpretations of works of fiction of which I know the most common ones (e.g. "this is a parable about [...]", "this is a warning against authoritarianism", etc), and ChatGPT sometimes comes up with stuff that completely misses the point; but if you aren't familiar with the work (haven't read it except for a summary) you could mistake it for a valid interpretation!
fix prose, absolubtely. To write?
Well a lot of the more egrerious errors ive seen when trying to get it to write academic writing comes from it not having a full context of what the problem is/whats been done/ what you are trying to do. Ive been turning over the idea in my head that for one to fully give it all the required context one might as well just use that context as the piece of writing in the first place. There is no advantage into stretching the relevant information to 6 paragraphs instead of 2. Not to say it should never be used but trying to use it while being thorough and actually reading its output has usually led me to extending context more and more and eventually just writing the section myself.
A prompt such as "Why are all the dolphins dying?" would be unacceptable.
A prompt such as rewrite "explain that the dolphins are dying because 30 degree rise in ocean temperaturs made their environments unlivable" that reformats the language into a academic style I feel is totally acceptable and efficient if you read what it spits out and agree and edit from there.
I think most other people should not use it for writing, because the hard part is thinking about (A) what you _really_ want to say (B) what your audience cares about (C) mistaken assumptions by you or your audience, which need correction in the text. And I think people using ChatGPT to write will give it a vague idea of (A) and usually nothing about (B) and (C).
What's interesting is that even when they reply, authors don't seem to respond to the ethics/standards issue of actually using ChatGPT. Well, some do ("I will not do it again" is basically what one of the authors says, which reads like a childish response to me), but many claim it was a "mistake" -- the mistake being... forgetting to remove the boilerplate and "regenerate response" garbage! Uh-oh.
I think an acceptable use of AI would be in translation, basically a better Google Translate. But is this what's happening here?
I have been experimenting with generating powershell scripts using chatgpt, and aside from a few syntax errors or localization, the scripts are a great starting point.
Instead of writing the whole script. I can have ChatGPT write it, then I just have to clean it up, address any errors and localize.
I think a lot of the hand-wringing neglects that there's a range of ways to use LLMs. Text-refactor is obviously perfectly ethical, unless you're gunning for the Butlerian Jihad, no worse than the grammar checker in Word.
You and the comment you replied to seem to be making an argument about something neither my own comment nor TFA said.
Nobody is claiming LLMs or ChatGPT aren't useful.
The argument is about using it to write parts of research papers without disclosing this usage. Nobody mentioned using ChatGPT to write code.
Hence my closing line, "[in the context of writing research papers] a valid use of AI would be as a better Google Translate [for non-native English speakers]."
Whether or not they used ChatGPT just as a translation service (an acceptable use, in my opinion), in many cases they didn't disclose this usage even when the rules of submission mandated it.
--
[1] I just checked and now it just says "Regenerate". It must have been "Regenerate response" in the past.
Authors, referees, and journals go through the motions of what lead to genuine scientific advancement in the past, but no actual value is being created at any step in the process for that vast majority of papers. Of course most papers will never be ground breaking but at least in the past there was some value being created in the process.
But regardless, this is extremely troubling in terms of quality. The fact that "regenerate response" gets left in tells us that even if it's just for grammar correction:
1) The authors aren't even minimally reviewing if the ChatGPT version didn't accidentally change the meaning of the sentences
2) Peer reviewers clearly aren't even reading the paper
3) The journal isn't even reading the paper to introduce a minimal level of quality control
ChatGPT seems to be the smallest problem here. And thankfully, it is revealing much more serious problems about quality checks in academia at all.
The question is, will anything be done about it? Not about ChatGPT, but about the horrifically irresponsible processes that would let these slip through?
Ignore me. :)
-----
I want to pull apart one specific thing you mentioned, essentially:
> It seems totally unclear whether ChatGPT is being used to invent fictional methodology [...], which is extremely unethical
In the context of generating something fictional, why is using ChatGPT unethical?
One reason I'm asking: I have always been a big fan of NaNoWriMo[1] (National Novel Writing Month, every November). This year, I'm interested in exploring how to use ChatGPT -- not necessarily to generate a ton of prose, but to essentially help flesh out some plot ideas that I've had.
Part of the back and forth I've had with it is coming up with some interesting scenarios for my story that I hadn't really thought about and I'm keen on exploring a bit more.
[1] https://en.wikipedia.org/wiki/National_Novel_Writing_Month
The human is the problem in that scenario, not the LLM. The unethical choice happens at the point of deciding to lie about methodology, before LLM is involved.
It may seem like a meaningless distinction, but it’s the difference between LLMs causing trouble and people causing trouble, which is important for the larger debate.
I think one problem with LLMs and the hype around them is that people are placing far too much confidence in them, and therefore misuse them.
That it's even conceivable that people might blame a tool for unethical behaviour is an indicator of this.
The idea that an LLM would be able to correctly generate your methodology for you is another.
I find ChatGPT simultaneously incredible and terrible. It has no concept of what it knows and what it doesn't, so there's nothing I would trust it to do without a human checking the output.
I asked again in a new chat and it made me a really nice one with funny things like a formula showing the amount of force he uses to open doors and fake data proving statistically that he is the best
So apparently someone submitted a paper that was entirely nonsense. This was intentional but not explicit as it was just a stunt. The paper had all the right fancy/technical words, the right structure, likely even images, and apparently a convincing conclusion. But it was all about nothing, nothing substantial at all, nada. Had a peer reviewer actually read the paper, it would have not been published and summarily tossed. But it did go print and oh boy, the journal's board/powers-that-be were pissed. I didn't follow the story much past that point but I think something was supposedly proposed to prevent this.
Such alarmist tones would sound really silly when applied to other dumb productivity tools.
In the end, the quality of the paper is what should be judged.
If there are very obvious mistakes like a leftover pasted in button text, it calls into question whether the authors (or reviewers) were responsibly verifying the content contributed by a tool that is known to sometimes deliver wildly but subtly incorrect results, and thus the quality of the paper content as a whole.
1. Bad grammar in a paper is incredibly irritating, which is particularly unfair for people who are non-native English speakers. If ChatGPT can help people to convert awkward English into nicer work without changing the ideas or confabulating, that seems like a win. I don't see why people need to "declare" their usage of ChatGPT for these purposes. Is that a standard now?
2. The major issue here is that the authors copied a response and accidentally also brought some button text with it. This is given to imply that people aren't reading ChatGPT's output. But I think this is just as likely to be benign. People do all sorts of silly nonsense in papers (including me), like leaving author comments in final drafts. It's bad and sloppy, but it isn't usually a crime.
3. Many papers on public servers are junky. Many journals and conferences are equally junky. Search for any bad string and you'll find it everywhere, and non-scientists will be scandalized.
4. I like the people at Retraction Watch and think they're doing the Lord's work. However sometimes I worry that immersing yourself in wickedness makes you see wickedness everywhere, and also creates an (unconscious) incentive to sensationalize bad stuff. I hope they're aware of this.
ETA I would urge you not to throw the death penalty at people whose crime may be more akin to shoplifting. This is bad both for the shoplifters and for people who want to fight serious crime. Plagiarism is one of the most serious accusations in science: casually tossing it around like this risks our ability to enforce it against serious offenders.
It's a difference in quality large enough to constitute a difference in kind. Autocorrect nominally takes text in your own words as typed and converts them into your words as intended. Chatgpt used as you describe does not produce text in the author's own words.
As an adult in a scientific field, what actually matters to me is having original scientific ideas conveyed to me in an efficient manner. I'm not really interested in applying arbitrary elementary-school rules on authors, as long as they aren't stealing ideas and text from other people (thus misallocating credit in ways that are harmful to progress.) I think as a community we should proceed from these principles, then work the details out from there.
The elementary school rules, at least the ones I weas taught decades ago, are pretty much “don’t steal ideas and text from other people”. Like, literally, other than for formatting, there were exactly three rules avout plagiarism I was taught on this in elementary school, and only one of them becomes slightly fuzzier outside of that context:
(1) if you use exactly wording from someone else, put it in quotes and cite the source,
(2) if you use an idea from someone else, cite the source but don't ise quotes, and
(3) if an idea comes from three or more independent sources, you don’t need to cite it, it can be treated as “common knowledge”.
Outside of elementary school, yeah, whether something is genuinely common knowledge or should be cited to multiple sources is fuzzier and context dependent, but the other two rules apply pretty directly.
Then the rules you were taught were far more lax than those I learned. According to the rules I was taught, copying lines from your own prior work without attribution would also be plagiarism.
That something concerns matters of little personal interest to you doesn't make it vacuous.
I think the OP's point is that you could be, and should check, as such models sometimes reproduce parts their training set. You should check what you send, or, in other words, take responsibility for the tools you use -- the tools that are bread and butter are somewhat excused, but LLM are not at the moment.
Perhaps a footnote would be added to mention that ChatGPT was used in this way.
Also, if you go to contact the author or view a talk, and suddenly their English is broken and completely different than what they wrote, you start questioning whether they wrote it or paid someone to without disclosing it.
IMO it's worth disclosing for many of the same reasons you disclose funding, ethics approval, author contributions, and conflicts of interest at the end of an article: it's metadata that protects your credibility as an author by showing that you're not doing anything underhanded.
The fact that this slipped by the paper's author, their editor, *and the journal review* is not benign.
I agree with the author here: is no one reading these papers before putting their names behind them? Is the entire journal system entirely fraudulent?
Depends on how much the journal is willing to push the authors (predatory journals don't give a heck, prestigious journals usually care more because they've got a rep to uphold). Reviewers can be anyone from an expert in the field, a competitor who gives you a hard time (usually they're not actively malicious though), to some random prof doing it because the editor couldn't find anyone else, or even a grad student in the lab of a prof (it's a common training exercise).
I've had articles with anywhere from 1-3 reviewers, there is no standard beyond "the editor is a basic crackpottery filter, there is an external reviewer, and a copy editor makes it fit for viewing"
You can use LLMs to create fake science, but also use it expedite writing up real science. I have experimented with feeding in a paper that I've read, I know is relevant, and direct the LLM to extract this point out and connect it to this other point. I then go through and edit. Its faster to rewrite existing text (for me) than to generate it the first time.
We should NOT be demonizing tools that increase our scientists productivity when they use it responsibly, rigorously, and ethicly.
We know people make up their data. We know peer reviewers sign off on it without actually looking it over. And we know the journals publish it anyway, and make a lot of money off of it.
Why does anybody even care if we've added another layer of fabrication to the mix of something that's already so full of lies?
I believe something that is not falsifiable is not science. So what would failure look like to you if it isn’t “> 50% off science is fake”? What do you call failure of this system?
Sure, why not? Yes, it's imperfect and BS can get through. But on the whole, it works reasonably well.
This shouldn't be surprising, since the most effective and dangerous liars tell the truth most of the time.
Oh, I agree. However, the peer-review system does mitigate this problem to a useful degree. Without such a system, it would be impossible to have any degree of trust in any paper at all.
> What we're really finding out is that journals are often little better than LLMs as information sources.
It depends on the journal. There are absolutely crap ones out there that need to be ignored. But there are also good ones out there that have earned their reputation. Even there, BS can get through of course -- but to say that such journals are mostly unreliable is seriously overstating the issue.
The idea of peer-reviewed journals is great. The reality is a circle jerk.
And how can one easily determine, while looking at a particular paper, whether it has been replicated? And whether those doing the replication have any undisclosed ties to the original?
At an epistemological level, the idea of a knowledge source like a journal where the information is only deemed reliable if personally verified seems problematic. Why even have it if all of its uncountable claims are indistinguishable from very clever lies, and attempts to quantify the extent of those lies indicate that they are pervasive?
https://en.m.wikipedia.org/wiki/Replication_crisis
And comments
I predict that by 2030, probably 99.9% of all content and published discoveries will be AI-assisted, and outcompete manual stuff for grants and money. Manual content will be as niche as knitted sweaters on etsy.