Not sure about you, but I sometimes simply stop reading an article if the tone or language is 1 SD away from "Standard English". It can be hard enough understanding the content of a paper without having to interpret and parse the language they're using.
Exactly, it sort of increases the effort and decreases likelihood of reading, so support for authors there is a good thing.
At the same time, the LLM tools will do more than just enhance the language in a narrow way but rather also "streamline" it, which reduces what can be expressed. That probably will lead to a drop in the maximum complexity that can be expressed. We already see that in other areas where language is already down to really simple stuff with all complexity being removed (whether it is there or not.
>That probably will lead to a drop in the maximum complexity that can be expressed.
I'm not following your argument here. Are you saying that rather than getting help from an LLM, future researchers will be forced to use the LLM's output as-is?
No. However, there will be expectations forming around the language in papers based on how LLMs use language: People using LLMs to enhance readability will have an impact as the language would gravitate towards something like "maximum likelihood" text (incl text length). It will be very easy to read but also limits what can be expressed within such a "corset".
If editors make such a decision, how would that be any different than a publication deciding to set a readability bar based on something like Flesch–Kincaid?
You can request an LLM to write in a particular style. You can request a paragraph to be written in an Allan Poe style (complex prose) or in the style of US Navy Technical Manuals (simple to the point language).
What we associate with chatGPT style prose is just the model's 'default voice' but can arguably be curbed.
I'd never actually thought to get chatgpt to rewrite papers for me, there's some very badly written ones (fully aware that English may not be easy for many people to write in) papers that I have been setting aside for ages.
While true, scientific papers are written with an aim to be reasonably easy to parse. Turning scientific papers into literary works is likely to fail (novelty bits aside), same as it fails in business settings etc. We already have pretty streamlined language because people demand it in a lot of places - LLMs will accelerate that because it reduces the effort to get there.
My point is that the output style of LLMs like chatGPT is not static. The 'default voice' uses fairly embellished language but there is nothing stopping a user from requesting a plain language output, such as what you would find in a US Navy manual (whose target audience is a huge cohort) .
I know, but my point is that people will converge on a certain compact and simple style for scientific papers because that will be favored for fast and easy reading (as seen in business, for example).
Such a style isn't necessarily easy to write but the LLMs will help people to get there.
less bullshit would be nice than more but we're so good at polluting, we'll just start shitting in the noosphere as well until it also becomes uninhabitable for humans just like our own planet.
There's a wrong way and a right way to use ChatGPT/AI tools for scientific publishing. The right way is as a source of ideas and for feedback. The wrong way asking ChatGPT to write anything for you, which is sent off to be published.
I'm quite sure that people who use the wrong way will end up regretting it.
And yet, if your article gets rejected when you write it yourself, and gets published when you have AI rewrite it for you to use all the right buzzwords based on previously published articles -- that's a good (financial) reason to use AI for writing.
Research inherently requires attention and attention is inherently scarce, so unfortunately people do and will compete for attention however they can, metrics or not.
> let KPIs get into research/education and healthcare
Research, sure, that’s a creative profession. But education and healthcare have obviously measurable outcomes, some of which are predictably desired ex ante.
This is why I've started to argue against "peer review." Because no one can decide a paper's correctness/legitimacy by sitting down and reading the work. The method just doesn't scale and certainly doesn't in settings that are zero sum like conferences (see the absolute shitshow that is ML peer review). It's become a entire waste of time and money.
The alternative I suggest is just submitting works to Open Review so there can be open discussions on the works. It removes the hostility from the setting and self biases towards reviewers/commenters being parties that are experts in that sub-niche as they're going to tend to be the only ones reading those papers, and especially reading closely.
Metrics are fucking hard and we don't improve systems by removing nuance from metrics. Yet as complexity has increased in our world, this has seemed to be the direction we've gone. It's bad for science and the current methods have far too many false negatives (improper rejects) and false positives (improper accepts) to validate it as a meaningful signal. We've tried so many ways to fix it while maintaining the institution that it's time to just try letting go of it. I honestly cannot find a great argument for keeping these journals around (yes, there are arguments, but not many ,,strong,, arguments, and specifically ones that lead to better science).
And yet, when AI rewrites your paper for you, and you blindly send it to be reviewed, likely also by an AI, it may well have fabricated or misrepresented the actual research and congratulations you've published some bullshit.
And no, humans are not just as bad as AI at this. I don't know why that meme persists - if humans were just as bad, much less worse, at hallucination and confabulation as AI at scale we would never have gotten past the stone age as a species. It would be useful to keep humans in the loop but the inevitable exponential explosion of AI generated research will simply be too much for peer review to handle (see the same thing happening with fiction writing, and several publications having to shut down because they were drowning in AI garbage.)
But it's not as if it matters. As with everything else that incorporates AI, quality isn't relevant, only quantity and profitability.
Or at some point change the rewarding system to favor quality. Voila! But why is everybody assuming that the AI-enhanced work will not be checked before submission? I'm no scientist but my workflow starts with my draft and ends with me correcting/adapting whatever AI changes did, whenever AI gets involved.
Exactly. We have to have mechanisms to support replication, which is the only means of validating work. Rather, adding evidence to the work's claim. We're too caught up in this naive notion of novelty, which today more strongly correlates with how well read one is in this massive ocean of papers.
> Or at some point change the rewarding system to favor quality.
I am a scientist, and I don't think anyone knows how to do this. There also is very little skin in the game for this. Bureaucrats want dumb metrics that they can point to to determine success regardless if that metric is meaningful or not. There is a strong pressure to publish quickly, which directly conflicts with a pressure for quality. Many prominent Nobel Laureates, Fields Metal Winners, Turing Prize Winners, and so on have discussed how they themselves could not have thrived in the current environment and its insanity. But who is going to change this? Honestly the ones that are hurt the most are the grad students at the non-top 10-20 universities. Everyone else has found "success" in the system (as in learning how to play that game) and they are highly incentivized to maintain that system and their status. Research is a completely different game today than it was even 20 years ago and we have not adjusted our system accordingly, and worse, many want to pretend that it hasn't changed. An interesting simple example is that research teams have exploded in size (especially at large universities) which makes for a lot of Nobel drama (max 3 per team). But there are also many other simple and more nuanced points that exist, but few want nuance. Either way, I'm absolutely certain that our current metrics do not strongly align with producing good science.
I understand then that there's a known and increasing tension in this domain. Instead of seeing it crash and burn, do you see any glimmer of hope? I mean we still need science and we still need research, so we need a solution to be able to go on with it, even if it's outside the current framework...
Oh, of course. Most academics don't do it for the money. It's not like they're paid much. Though that can incentivize cheating in another way because they want money. But there will always be people that don't give a fuck and just want to do good research regardless of the metrics being used. I am insistent that to be a good scientist you must be somewhere on the side of "anti-authority." Because your job relies on challenging concepts, and especially well known and widely agreed upon concepts. Generally look for people who are passionate, will rant, but importantly rant with nuance. Those are the people passionate about their research and not the metrics.
The problem is we're just throwing a lot of money down the drain, wasting a lot of time, and generating distrust of the system. Any "crash and burn" is never going to lead to an extinction in any sense. It's just about general society level if we want to do good science or just do noisy science (all science is noisy). But you can never do good science if you remove nuance. This is why I hate that ML (and CS in general) uses conference systems as the main platform for publishing. It is ridiculous to think you can have a good system when it is highly competitive, zero sum, highly time consuming, there's no discussion between authors and reviewers (you may get a one page rebuttal, but your reviewers comments are often disjoint and vague), and you're being judged by those you're competing with. It's just a silly notion to believe this is useful.
The solution is actually not hard. I refer to the larger phenomena as Goodhart's Hell. The solution is to stop using metrics as targets. Metrics are guides. If you don't have a deep understanding of what your metric actually measures, how well your metric aligns to the thing you're intending to measure (never 100%), and if you don't understand the biases to your data, you're fucking doomed to this bureaucratic hell. Noise is inherent to complex systems and aggregation is the bane of evaluation of complex feature spaces. Just remember that its models all the way down and that all models are wrong (despite some models being better than others).
That would be the large number of salient examples.
I expect humans to form a (possibly normal) distribution in this regard rather than being all of equal quality, and people on the high end would permit development even if the mean was worse than even the best current AI.
If we were however all clumped together, then the very same meme you're complaining about here would itself be an example of humans doing exactly what you're saying AI do.
Except, since we're talking about scientific research, and not all humans perform scientific research, the only relevant group to consider would be humans who do scientific research.
I don't think it's common for researchers to randomly fabricate sources, citations and results. It happens, but it isn't mere coincidence when it doesn't happen. The "replicability crisis" would appear to be a counterexample, but it isn't. Actual attempts at research aren't the same category of error as randomly generated research.
When AI gets something right on the other hand it's entirely coincidental, because AI doesn't have any concept of "truth" or "falsehood," or the real world context in which it operates. AI just relates text tokens to one another, with some unavoidable randomness, trying to match something that looks like the desired result. To equate that process with the result of actual human beings doing science and publishing research seems like you're just reaching to justify the validity of replacing researchers with AI.
> Except, since we're talking about scientific research, and not all humans perform scientific research, the only relevant group to consider would be humans who do scientific research.
Sure, but I expect that to change the distribution, not much else.
> I don't think it's common for researchers to randomly fabricate sources, citations and results. It happens, but it isn't mere coincidence when it doesn't happen.
Hopefully not, but even in science, misunderstanding the cited work is a thing, as is disbelieving it. For the former, "perceptrons can't do XOR" became "AI is impossible", while the Black-Scholes equation was misapplied to ultimately lead to the global financial crisis. For the latter, the salient examples are older (e.g. the importance of washing hands between autopsies and midwifery), or things where I can't find the historical scientific consensus because of the modern cultural noise (e.g. thinking men and women have different numbers of ribs because of the book of Genesis and not checking).
In this regard, I think the replication crisis is valid, even though it (presumably) involved real research and real results, because widespread citations didn't fully account for the limits of that research.
To put it another way, while I agree with this:
> To equate that process with the result of actual human beings doing science and publishing research seems like you're just reaching to justify the validity of replacing researchers with AI.
With scientists I'm referring more to the game of telephone that necessarily occurs between any given researcher and anyone using their work.
For the general population it's only that game, hence all the people who think scientists in the 70s were worried about an ice age, or the newspaper stories that collectively divide the world into things that cause or cure cancer.
If you are referring specially to people who want an LLM to actually perform novel research, rather than merely write up lab notes in LaTeχ, then you would be correct that this is a terrible idea. Even for pure mathematics, you need a different type of AI, better suited to that task — they do exist though, and have been used successfully, but IIRC are not anything like as general with regard to maths and experiments as LLMs are with language.
> And no, humans are not just as bad as AI at this. I don't know why that meme persists - if humans were just as bad, much less worse, at hallucination and confabulation as AI at scale we would never have gotten past the stone age as a species.
Knowing this would require a counterfactual machine of some sort, thus the fact must be hallucinated into existence, no?
I'm quite happy to read something in a paper that ChatGPT or equivalent has written, just as i'm happy to read something that has been written with the assistance of other tools such as Grammarly, LanguageTool, Spell Check etc. I see it as the (human) authors responsibility to ensure that what has been written is factually correct.
The one time I ask an LLM AI to write something for me is after I think I have a final draft, I stay in the chat window that has all the history in it from where I worked through the paper. Then I ask the AI to write the paper for me. I do not take phrasing from it. I look for new ideas the AI inserted that I missed. There is usually at least one good new idea for me to steal.
I can imagine in the future authors selecting a checkbox with "I authored this manuscript without the help of any IA." before submitting a paper for review.
AND if caught, they would have the 'power' to lift the paper from their databases or mark it somehow (brand it with a (IA) logo perhaps?)
"I authored this manuscript without the help of any IA."
Where do you draw the line? Tools like Grammarly and similar are AI. Using google translate to translate a phrase from your native language to English to is AI. Hell the spellchecker that just autocorrected a typo I made in the previous sentence is probably AI. Should they all be banned?
People are focusing on the negative stuff here (lazy researchers "cheating"). But there's also a positive side to this. Generative AI is the ultimate research tool. It already knows the vast majority of research that you'll never get around to reading. You could spend your entire life reading every last second of it and you wouldn't come close to catching up. A lot of that stuff may be irrelevant (to your context) but it still knows about it.
And you can ask it questions about anything it ingested. Or ask it to criticize your text, find analogies to other work, or generally perform the role of a really diligent peer reviewer and editor before you even submit the work. That's all highly useful and it should lead to a higher quality of work for those researchers that use their tools well. I use gpt for Google docs and it's definitely helping me improve my text. I don't let it write whole sections/paragraphs. But I do use it to criticize, critique, and suggest improvements. I imagine a lot of students and researchers have been doing the same for the last year or so.
The same goes for reviewers. They can ask to extract key points, analyze the argumentation, find related work that the author might have missed, figure out where the authors are taking a few liberties with the facts/literature, etc. much more easily.
I've reviewed a fair amount of mostly badly written papers back in the day. This is not fun and rather laborious work. A lot of academic life is basically about reading each other's work and providing (hopefully) constructive criticism for articles that ultimately don't make the cut. I got rather good at that when I was still doing that. Any workshop, conference, or journal ends up rejecting way more articles than they accept. Especially the better publications. Some poor souls have to read all the rejected stuff. The price you pay for getting accepted is helping out with the peer reviews.
The process can be biased, political, unfair, and sometimes harsh. But it's better than not having a process. Generative AI can help with challenging unfair peer reviews, help reviewers extract key points, and zoom in on novel ideas/theories. Ultimately what you look for in an article is: Does it contribute something novel? Is the work contextualized properly relative to prior work? Is the work sound in its reasoning? Etc. Answering such questions positively basically means it's a good article. A generative AI can save a lot of time with this. Weeding out the bad articles is not that hard but a lot of work.
Just to plug that we have built MirrorThink.ai: an AI assistant specifically designed to help professional researchers in their day-to-day work.
(I am aware that self-promotion is allowed but somewhat frowned-upon in HN. I believe this is relevant to the context and I am particularly qualified to comment on it. But I'll strive to be as transparent as possible.)
We have intentionally steered away from assisting with writing work for the reasons outlined in the article. Our focus has been to save time in keeping up with their field: making literature reviews much faster, summarizing recent discoveries in each researchers' niche area of interest, and gathering how others have tackled similar highly-specialized research problems. Grounding LLMs in scientific fact and always providing verifiable sources.
We have spent many years before this building a business around our "Map of Science" at Scitodate, so we have strong connections with the community and a lot of experience in this domain. Both in terms of how science is done and how to reliably extract valuable knowledge from millions of documents with NLP. We now have one of the largest datasets of papers, patents, funding grants... Updated on a daily basis. With plenty of enrichment, connections and aggregations: we have done a lot of in-house research on disambiguation and entity-linking. We particularly have focused on mapping out the expertise, interests and achievements of all professional scientists, as well as the institutions they have worked at.
How do you get the chatbot to tell the truth more? I’m a scientist and this is my biggest showstopper with AI tools. I just can’t know if I can trust what it’s telling me.
Or, they realize that absence cannot be scientifically proven, and that scientists use language loosely, confusing those who take their loose language literally.
The study didn't "find" (discover) what they claim, rather, they didn't find validation that it "can" (the implementation of which varies per observer, sub-perceptually).
If you had to code something like this at work for a different domain, I bet you'd have no problem realizing that a nullable boolean is required to accurately model the problem space.
I recognize your clarification of “discovery” and conclusion from that research, but I do think there is a strong argument that in terms of the stochastic usage of a nonlinear system the “undefined” state of your nullable boolean is itself a falsey state.
You can argue whatever you like, but if the unknown IS actually known, why can't scientists tell us their secrets? How many people would have to be in on the scheme?
And this isn't just a one off, this is a systemic, institutional shortcoming, I encounter several instances of it every day just in my regular social media feeds.
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
In other words, AIs are useful when you can confidently evaluate their output for correctness.
Note that this also applies in novel scenarios, such as new scientific discovery. It's often much more difficult to come up with an answer than it is to verify it.
As a random example, modern AIs are very good at image recognition tasks, which would make them ideally suited for finding rare phenomena in all-sky surveys. The AI doesn't need to write the research paper! It can be valuable if it can just find candidates for detailed evaluation by human researchers.
Similarly, I've found AIs are quite good at spotting errors and inconsistencies. Don't make it write the paper... just ask it to proofread it for you.
Never rely on the AI's own "memory". GPT-4 can remember a lot of deep technical things correctly from its training, but it can also write non-factual statements that look perfectly reasonable.
We only use LLMs to retrieve and extract information from reliable sources (mostly papers and patents). If you provide extensive background information in the prompt, it will stick to that and avoid making up facts, and it has the added benefit of including verifiable references for each statement in the answer.
Again, we treat the AI mostly as an "automatic reader", it's very good at that, it's not hard to keep it factually grounded and that alone can save hours of research. But when asking it to do deep reasoning or original writing it can be a bit more tricky to keep it controlled.
Treat it like you're dealing with some vendor's salesperson. You have to bring your own source of truth, like the interfaces that let you "talk to your PDFs".
Prep it with data you know to be true, then start your discussion. If you show up unprepared and let it tell you what's real, it will bullshit you in its favor.
There's a free version if you register with an institutional email (university or company email domain), we want to support scientists (both academia and private). For full access it's pay-what-you-want right now (with a minimum of $5 or $10 depending what roll of the dice you get in the current A/B test).
Did the emails have different domains? We use the domain as a simple filter for the free version, but there might be a bug around to that. It would be helpful if you could help reproduce it.
They did! I used my personal first, which is just Firstname.Lastname@gmail.com which was the one that broke. The other one was Firstname@company.com which seemed to work fine.
This is exactly how I use ChatGPT right now. Like an extreme rubber ducky for research when I don't have anyone else I can talk to about something. Also, thank you for the self promotion, I'll check out MirrorThink!
- What's the product actually do? There's just one page with some vague descriptions and then a signup. I'd love to actually be pitched the product before signing up.
- What is the cost?
- How does literature search happen? What are the sources? How often is it updated? How sub-niche can it actually get? You specifically mention the researcher's niche interests but your examples are "Why can't we sleep when excited", "Companies selling 3D printers", "3D printers in Eurpore", and "3D printers for medical purposes." These are not niche topics, this is pretty broad and doesn't look like scientific research.
I'm sorry, but without more information this pitch and website just look like an information harvester. This may not be what you've actually built, but this is what it comes off as from someone who does scientific research.
These are valid remarks, we'll see about including them in the landing page.
Before going to the answers, just to note: if you are interested, we are looking to work closely with more researchers, to make sure it saves time for you in particular on a daily basis.
What does it do?
In very simple terms: It is ChatGPT (GPT-4) with awareness of our database of papers, patents, grants... It comes with many "tools" to use various information retrieval styles and sources, some more focused on question answering and some on wider reviews. There are also tools to gather "metadata" of science: researcher profiles, institutes, scientific companies...
What is the cost?
There's a free version if you register with an institutional email (university or company email). We generally want to make it open to professional researchers (academic or private). The full version is pay-what-you-want with a $5/$10 minimum.
How does literature search happen?
It depends on the "tool" you are using, but it generally involves retrieving a set of candidate relevant documents (papers, patents, websites...) and either read through full documents or select relevant snippets from them so it can do a wider review.
What are the sources?
Primarily papers (~100M right now), including ArXiv, PubMed, IEEE, the Elsevier suite and many more individual journals. As well as grants from most national and international funding organizations. And US patents, with ongoing expansion to other countries. Updated daily whenever possible.
We also have the option to complement scientific literature with web research, it can be good for example to review public documentation from scientific instrumentation and consumable providers on experimental protocols.
How sub-niche can it actually get?
It's hard to convey, it would be easiest if you just try it, it really can get as niche as you want it to, it can adapt to the level of expertise you operate at. If you are a researcher with a university email you can use most of the relevant functionality for free.
You are right, the examples are not very representative. The intent was to make them understandable to a wide array of researchers with different backgrounds, but still, yeah they could be better.
> if you are interested, we are looking to work closely with more researchers, to make sure it saves time for you in particular on a daily basis.
Sure, I can sign up. But can you clarify data retention and usage policies? As researchers we're always worried about being scooped. Despite complaining about the system and incentives in other comments, we're bound to the game we're forced to play to achieve scientific progress (if it weren't for this stupid game I'd research completely in the open). I can't find information on how data is used. This is especially important if you're discussing institutions like work, as many of them can't even use standard GPT4 accounts do to privacy concerns.
I'll provide some comments about what I personally would like but clearly personal preferences. I'll say that there's a lot that can be done in this space that doesn't even need AI.
For literature review:
Let's say I am a ML researcher (true) who focuses on 3D semantic image segmentation (not true) or some other appropriate niche (we should be able to get much more narrow btw). How do we deal with this literature search? As a simple example, let's say I wanted to get introduced to a topic that my colleague is working on. Finding a proper survey paper is often a surprisingly cumbersome task. Making this searchable (especially with a ranking not just by citations by some coverage metric -- too many 10 page "surveys") would be exceptionally useful. In addition, I'd love to have graphs that I can see paper or author connections through different sorting methods. A timeline is often highly useful to understand the progress of a research topic and especially helpful to making new surveys and onboarding new grad students (or anyone into a new topic).
Citation and Reading List:
I love Zotero, but it is so fucking limited it gets to me. Here's my issues:
There's no good way for me to create a prioritized reading list (which should be topic based). Sorting papers that come across my desk is a cumbersome task and they just don't get sorted. But I am able to assign prioritization values to these works (especially attached with a certain project or domain). But Zotero does not make this a realistically viable task. Realistically I have multiple browser windows and store arxiv links in tabs or use Obsidion (more cumbersome). This is actually something ML could help with, at least for the automatic classification (we can discuss other metrics too if you want).
Discovery is a challenge, especially in a new niche, but the bigger problem is actually wrangling the papers I've already discovered.
What also sucks is that I can't markup these works. Preferably on my iPad (OSX would be fine since easy to share). Adding notes is also a bit cumbersome. I often want to add notes to papers to categorize them as citation references to certain topics or place them into importance for certain niches, and this should be different from a reading list. And then comes the problem of sharing. Not great.
I said there's a lot that can be done without ML, and that is because I don't even have good tools to sort and catalogue the papers that come via natural discovery. This comes through multiple devices too, and idk if I'm going to get that on my computer or phone (twitter is still a great place for paper discovery, and comes with author comments/summaries. No tool catalogs this).
Another great pain is citations. Authors aren't helpful here, but grabbing the proper journal/conference citation is surprisingly a non-trivial task. They aren't always indexed by Semantic Scholar or Google, who both have quite large delays. But there's definitely no automatic correction in Zotero for when I add a arxiv paper (via plugin) that replaces the arxiv bib with the proper bib, even if that is indexed by GS/SS. Every time I write a paper this task has to be done and my zotero list gets updated.
Alright let me go over all the points. But you clearly care about finding a tool to help you with your work and are quite critical of any half-assed solutions. I would love to work together to literally make a custom tool just for you that can help with all of this, and keep going at it until it legitimately saves you a few hours a day. We have the experience, the resources and the will to do this. At no cost to you of course, I'd even be happy to discuss a comission if this ends up helping other researchers. Contact me if you'd like this: jon (at) scitodate (dot) com.
> How many people on the team have a PhD?
We are already working like this with a few other academics, as well as a number of pilots with R&D teams from industry. And yes my co-founder is a post-doc in theoretical quantum physics, and we have a number of other PhDs with experience in academia in the team, mostly chemistry and life-sciences. Myself I started early in the startup game, I have a masters in CS and have some experience working in academia (a couple of publications, supervised 4 masters thesis, interned with a couple of research groups...), but not a PhD just yet.
> as many of them can't even use standard GPT4 accounts do to privacy concerns
Azure guarantees full isolation and privacy, right now it's the only way to use GPT4 with such guarantees, and no other LLM right now can compete with GPT4 in the domain we operate in. From our side, we are working with a partner on full ISO certification, but obviously security and privacy have been a priority from the beginning.
> (regarding all your preferences on research tools)
There's too much here to cover in a comment, please contact me so we can discuss this further. Just to say that our use of LLMs has started recently (early this year), but we have 6 years of previous experience on doing things like creating global co-authorship and citation networks through state-of-the-art disambiguation and aggregating from many different sources, and working on large-scale information-retrieval infrastructure (complex boolean search with ranking). We have worked a lot with LLMs in the past year for MirrorThink to extract knowledge from the actual bodies reliably at large scale, but this is backed with a lot of non-AI substance built over years.
> I don't want these fancy ML tools to summarize and write papers for me
As stated in a previous comment, helping with writing has never been our focus. I agree with your characterization, sure LLMs can save some time with writing, but it's still hard to have them write coherent long-form text with real substance. And it's just kind of unethical, particulary in the sense that it just inflates the amount of text scientist need to keep-up with, turning small advancements into low-density decent looking publications just to get "points", this goes completely against what we are trying to do.
We do want to save a lot of time on research, and MirrorThink shows that LLMs can indeed add a lot in this area and that it is possible to keep them very factually grounded. But again, I'm not very attached to "fancy" AI, I was quite skeptical of this tech for a long time until it became hard to ignore this year. We'll do whatever works for you, however mundane. Frankly, our business needs are covered in the B2B side and I have a budget to simply contribute back to scientists, so that's the only goal.
> I'm 30 year old ABD, not an old tenured professor ... some aspects also have to be usable by my advisor or boss
That is ideal, we want people "in the treches" so to say, actually doing hands-on research every day. We'll figure it out in terms of accessibility to your supervisor, it's a great non-obvious point that is only apparent to those on the inside.
> we can discuss my anti-{journal,conference} dream and how to create a realistic peer review system
Would love that too, we've had lots of interal discussions in this area over the ...
I think using any AI tool that reports your findings for you represents failure and laziness in science.
It will become a circle: somebody uses AI to generate boring text (probably not even themselves will read it), then somebody else puts the boring text for AI to summarise. Nobody learns anything. Repeat.
This reminds me of M-x write-thesis RET, a usenet joke from the 90s IIRC. Apparently, since google is no longer indexing the past, I couldn't find a link. However, I still remember the post vividly, and it makes me sad that 25 years later, we have to have this discussion for real now. Back then, everyone in their right mind knew why automated scientific publication writing would be hilariously bad. These days, we have articles like this. Weird.
It’s a shame the article doesn’t talk more about the possible benefits of AI in academia. The ability for researchers to spot new patterns, discover new areas of research. The way information is generated, disseminated and then read by other researchers in this space is fairly prosaic if you think about it. Causally is a company I think is trying something in this space
That's because it's not how it's used right now. Just like elsewhere, it's primarly used to generate bullsh*t and flood journals with low quality generated content. Which now need to deal with this deluge of sewage drowning good articles.
> It’s a shame the article doesn’t talk more about the possible benefits of AI in academia. The ability for researchers to spot new patterns, discover new areas of research.
Honestly, which publicly available AIs are capable of doing that?
At least not the publicly available LLM AIs: I asked those quite many questions about scientific topics, but they could not give me any answer that helped me - it was a complete waste of time.
74 comments
[ 3.0 ms ] story [ 144 ms ] threadWhether that is net good or bad is difficult to say: broader perimeter of publishing vs narrower way to portray findings.
At the same time, the LLM tools will do more than just enhance the language in a narrow way but rather also "streamline" it, which reduces what can be expressed. That probably will lead to a drop in the maximum complexity that can be expressed. We already see that in other areas where language is already down to really simple stuff with all complexity being removed (whether it is there or not.
I'm not following your argument here. Are you saying that rather than getting help from an LLM, future researchers will be forced to use the LLM's output as-is?
What we associate with chatGPT style prose is just the model's 'default voice' but can arguably be curbed.
My point is that the output style of LLMs like chatGPT is not static. The 'default voice' uses fairly embellished language but there is nothing stopping a user from requesting a plain language output, such as what you would find in a US Navy manual (whose target audience is a huge cohort) .
Such a style isn't necessarily easy to write but the LLMs will help people to get there.
I'm quite sure that people who use the wrong way will end up regretting it.
Research, sure, that’s a creative profession. But education and healthcare have obviously measurable outcomes, some of which are predictably desired ex ante.
The alternative I suggest is just submitting works to Open Review so there can be open discussions on the works. It removes the hostility from the setting and self biases towards reviewers/commenters being parties that are experts in that sub-niche as they're going to tend to be the only ones reading those papers, and especially reading closely.
Metrics are fucking hard and we don't improve systems by removing nuance from metrics. Yet as complexity has increased in our world, this has seemed to be the direction we've gone. It's bad for science and the current methods have far too many false negatives (improper rejects) and false positives (improper accepts) to validate it as a meaningful signal. We've tried so many ways to fix it while maintaining the institution that it's time to just try letting go of it. I honestly cannot find a great argument for keeping these journals around (yes, there are arguments, but not many ,,strong,, arguments, and specifically ones that lead to better science).
And no, humans are not just as bad as AI at this. I don't know why that meme persists - if humans were just as bad, much less worse, at hallucination and confabulation as AI at scale we would never have gotten past the stone age as a species. It would be useful to keep humans in the loop but the inevitable exponential explosion of AI generated research will simply be too much for peer review to handle (see the same thing happening with fiction writing, and several publications having to shut down because they were drowning in AI garbage.)
But it's not as if it matters. As with everything else that incorporates AI, quality isn't relevant, only quantity and profitability.
The whole system favors publication quantity over quality, results over process.
The solution is to ask our AI overlord.
I think that will happen when we swap peer review for peer replication.
I am a scientist, and I don't think anyone knows how to do this. There also is very little skin in the game for this. Bureaucrats want dumb metrics that they can point to to determine success regardless if that metric is meaningful or not. There is a strong pressure to publish quickly, which directly conflicts with a pressure for quality. Many prominent Nobel Laureates, Fields Metal Winners, Turing Prize Winners, and so on have discussed how they themselves could not have thrived in the current environment and its insanity. But who is going to change this? Honestly the ones that are hurt the most are the grad students at the non-top 10-20 universities. Everyone else has found "success" in the system (as in learning how to play that game) and they are highly incentivized to maintain that system and their status. Research is a completely different game today than it was even 20 years ago and we have not adjusted our system accordingly, and worse, many want to pretend that it hasn't changed. An interesting simple example is that research teams have exploded in size (especially at large universities) which makes for a lot of Nobel drama (max 3 per team). But there are also many other simple and more nuanced points that exist, but few want nuance. Either way, I'm absolutely certain that our current metrics do not strongly align with producing good science.
Oh, of course. Most academics don't do it for the money. It's not like they're paid much. Though that can incentivize cheating in another way because they want money. But there will always be people that don't give a fuck and just want to do good research regardless of the metrics being used. I am insistent that to be a good scientist you must be somewhere on the side of "anti-authority." Because your job relies on challenging concepts, and especially well known and widely agreed upon concepts. Generally look for people who are passionate, will rant, but importantly rant with nuance. Those are the people passionate about their research and not the metrics.
The problem is we're just throwing a lot of money down the drain, wasting a lot of time, and generating distrust of the system. Any "crash and burn" is never going to lead to an extinction in any sense. It's just about general society level if we want to do good science or just do noisy science (all science is noisy). But you can never do good science if you remove nuance. This is why I hate that ML (and CS in general) uses conference systems as the main platform for publishing. It is ridiculous to think you can have a good system when it is highly competitive, zero sum, highly time consuming, there's no discussion between authors and reviewers (you may get a one page rebuttal, but your reviewers comments are often disjoint and vague), and you're being judged by those you're competing with. It's just a silly notion to believe this is useful.
The solution is actually not hard. I refer to the larger phenomena as Goodhart's Hell. The solution is to stop using metrics as targets. Metrics are guides. If you don't have a deep understanding of what your metric actually measures, how well your metric aligns to the thing you're intending to measure (never 100%), and if you don't understand the biases to your data, you're fucking doomed to this bureaucratic hell. Noise is inherent to complex systems and aggregation is the bane of evaluation of complex feature spaces. Just remember that its models all the way down and that all models are wrong (despite some models being better than others).
That would be the large number of salient examples.
I expect humans to form a (possibly normal) distribution in this regard rather than being all of equal quality, and people on the high end would permit development even if the mean was worse than even the best current AI.
If we were however all clumped together, then the very same meme you're complaining about here would itself be an example of humans doing exactly what you're saying AI do.
I don't think it's common for researchers to randomly fabricate sources, citations and results. It happens, but it isn't mere coincidence when it doesn't happen. The "replicability crisis" would appear to be a counterexample, but it isn't. Actual attempts at research aren't the same category of error as randomly generated research.
When AI gets something right on the other hand it's entirely coincidental, because AI doesn't have any concept of "truth" or "falsehood," or the real world context in which it operates. AI just relates text tokens to one another, with some unavoidable randomness, trying to match something that looks like the desired result. To equate that process with the result of actual human beings doing science and publishing research seems like you're just reaching to justify the validity of replacing researchers with AI.
Sure, but I expect that to change the distribution, not much else.
> I don't think it's common for researchers to randomly fabricate sources, citations and results. It happens, but it isn't mere coincidence when it doesn't happen.
Hopefully not, but even in science, misunderstanding the cited work is a thing, as is disbelieving it. For the former, "perceptrons can't do XOR" became "AI is impossible", while the Black-Scholes equation was misapplied to ultimately lead to the global financial crisis. For the latter, the salient examples are older (e.g. the importance of washing hands between autopsies and midwifery), or things where I can't find the historical scientific consensus because of the modern cultural noise (e.g. thinking men and women have different numbers of ribs because of the book of Genesis and not checking).
In this regard, I think the replication crisis is valid, even though it (presumably) involved real research and real results, because widespread citations didn't fully account for the limits of that research.
To put it another way, while I agree with this:
> To equate that process with the result of actual human beings doing science and publishing research seems like you're just reaching to justify the validity of replacing researchers with AI.
With scientists I'm referring more to the game of telephone that necessarily occurs between any given researcher and anyone using their work.
For the general population it's only that game, hence all the people who think scientists in the 70s were worried about an ice age, or the newspaper stories that collectively divide the world into things that cause or cure cancer.
If you are referring specially to people who want an LLM to actually perform novel research, rather than merely write up lab notes in LaTeχ, then you would be correct that this is a terrible idea. Even for pure mathematics, you need a different type of AI, better suited to that task — they do exist though, and have been used successfully, but IIRC are not anything like as general with regard to maths and experiments as LLMs are with language.
As a scientist, you should be aiming at communicating ideas in a pragmatic and verifiable way, not embellishing stuff via AI.
A real facepalm moment for scientists with wrong incentives.
Knowing this would require a counterfactual machine of some sort, thus the fact must be hallucinated into existence, no?
All other uses, including, specifically for generating ideas, are plagiarism and ground for expulsion.
Maybe ChatGPT will help accelerate the rate at which nonsense gets published and will end up speeding the collapse of these areas of "study".
The one time I ask an LLM AI to write something for me is after I think I have a final draft, I stay in the chat window that has all the history in it from where I worked through the paper. Then I ask the AI to write the paper for me. I do not take phrasing from it. I look for new ideas the AI inserted that I missed. There is usually at least one good new idea for me to steal.
AND if caught, they would have the 'power' to lift the paper from their databases or mark it somehow (brand it with a (IA) logo perhaps?)
Where do you draw the line? Tools like Grammarly and similar are AI. Using google translate to translate a phrase from your native language to English to is AI. Hell the spellchecker that just autocorrected a typo I made in the previous sentence is probably AI. Should they all be banned?
And you can ask it questions about anything it ingested. Or ask it to criticize your text, find analogies to other work, or generally perform the role of a really diligent peer reviewer and editor before you even submit the work. That's all highly useful and it should lead to a higher quality of work for those researchers that use their tools well. I use gpt for Google docs and it's definitely helping me improve my text. I don't let it write whole sections/paragraphs. But I do use it to criticize, critique, and suggest improvements. I imagine a lot of students and researchers have been doing the same for the last year or so.
The same goes for reviewers. They can ask to extract key points, analyze the argumentation, find related work that the author might have missed, figure out where the authors are taking a few liberties with the facts/literature, etc. much more easily.
I've reviewed a fair amount of mostly badly written papers back in the day. This is not fun and rather laborious work. A lot of academic life is basically about reading each other's work and providing (hopefully) constructive criticism for articles that ultimately don't make the cut. I got rather good at that when I was still doing that. Any workshop, conference, or journal ends up rejecting way more articles than they accept. Especially the better publications. Some poor souls have to read all the rejected stuff. The price you pay for getting accepted is helping out with the peer reviews.
The process can be biased, political, unfair, and sometimes harsh. But it's better than not having a process. Generative AI can help with challenging unfair peer reviews, help reviewers extract key points, and zoom in on novel ideas/theories. Ultimately what you look for in an article is: Does it contribute something novel? Is the work contextualized properly relative to prior work? Is the work sound in its reasoning? Etc. Answering such questions positively basically means it's a good article. A generative AI can save a lot of time with this. Weeding out the bad articles is not that hard but a lot of work.
Until people figure out which words to add in order to game the AI peer reviewer, or good work is missed because the lack of buzzwords.
(I am aware that self-promotion is allowed but somewhat frowned-upon in HN. I believe this is relevant to the context and I am particularly qualified to comment on it. But I'll strive to be as transparent as possible.)
We have intentionally steered away from assisting with writing work for the reasons outlined in the article. Our focus has been to save time in keeping up with their field: making literature reviews much faster, summarizing recent discoveries in each researchers' niche area of interest, and gathering how others have tackled similar highly-specialized research problems. Grounding LLMs in scientific fact and always providing verifiable sources.
We have spent many years before this building a business around our "Map of Science" at Scitodate, so we have strong connections with the community and a lot of experience in this domain. Both in terms of how science is done and how to reliably extract valuable knowledge from millions of documents with NLP. We now have one of the largest datasets of papers, patents, funding grants... Updated on a daily basis. With plenty of enrichment, connections and aggregations: we have done a lot of in-house research on disambiguation and entity-linking. We particularly have focused on mapping out the expertise, interests and achievements of all professional scientists, as well as the institutions they have worked at.
“LLMs can’t self-correct in reasoning tasks, DeepMind study finds“
https://news.ycombinator.com/item?id=37823543
Anyone who says otherwise is either ignorant of the underlying function of llms or trying to sell you something.
The study didn't "find" (discover) what they claim, rather, they didn't find validation that it "can" (the implementation of which varies per observer, sub-perceptually).
If you had to code something like this at work for a different domain, I bet you'd have no problem realizing that a nullable boolean is required to accurately model the problem space.
And this isn't just a one off, this is a systemic, institutional shortcoming, I encounter several instances of it every day just in my regular social media feeds.
This is just wrong lol.
GPT-4 logits calibration pre RLHF - https://imgur.com/a/3gYel9r
Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback - https://arxiv.org/abs/2305.14975
Teaching Models to Express Their Uncertainty in Words - https://arxiv.org/abs/2205.14334
Language Models (Mostly) Know What They Know - https://arxiv.org/abs/2207.05221
The needless condescension of your “lol” feels a bit premature.
How can you have self correction without superintelligence?
Use them to replace grad students / interns.
In other words, AIs are useful when you can confidently evaluate their output for correctness.
Note that this also applies in novel scenarios, such as new scientific discovery. It's often much more difficult to come up with an answer than it is to verify it.
As a random example, modern AIs are very good at image recognition tasks, which would make them ideally suited for finding rare phenomena in all-sky surveys. The AI doesn't need to write the research paper! It can be valuable if it can just find candidates for detailed evaluation by human researchers.
Similarly, I've found AIs are quite good at spotting errors and inconsistencies. Don't make it write the paper... just ask it to proofread it for you.
Etc...
We only use LLMs to retrieve and extract information from reliable sources (mostly papers and patents). If you provide extensive background information in the prompt, it will stick to that and avoid making up facts, and it has the added benefit of including verifiable references for each statement in the answer.
Again, we treat the AI mostly as an "automatic reader", it's very good at that, it's not hard to keep it factually grounded and that alone can save hours of research. But when asking it to do deep reasoning or original writing it can be a bit more tricky to keep it controlled.
Prep it with data you know to be true, then start your discussion. If you show up unprepared and let it tell you what's real, it will bullshit you in its favor.
As an aside, the scroll on your website stutters pretty badly for me (on Firefox).
There's a free version if you register with an institutional email (university or company email domain), we want to support scientists (both academia and private). For full access it's pay-what-you-want right now (with a minimum of $5 or $10 depending what roll of the dice you get in the current A/B test).
Edit: Hmmm, worked when I signed up with a different email.
- What is the cost?
- How does literature search happen? What are the sources? How often is it updated? How sub-niche can it actually get? You specifically mention the researcher's niche interests but your examples are "Why can't we sleep when excited", "Companies selling 3D printers", "3D printers in Eurpore", and "3D printers for medical purposes." These are not niche topics, this is pretty broad and doesn't look like scientific research.
I'm sorry, but without more information this pitch and website just look like an information harvester. This may not be what you've actually built, but this is what it comes off as from someone who does scientific research.
Before going to the answers, just to note: if you are interested, we are looking to work closely with more researchers, to make sure it saves time for you in particular on a daily basis.
What does it do?
In very simple terms: It is ChatGPT (GPT-4) with awareness of our database of papers, patents, grants... It comes with many "tools" to use various information retrieval styles and sources, some more focused on question answering and some on wider reviews. There are also tools to gather "metadata" of science: researcher profiles, institutes, scientific companies...
What is the cost?
There's a free version if you register with an institutional email (university or company email). We generally want to make it open to professional researchers (academic or private). The full version is pay-what-you-want with a $5/$10 minimum.
How does literature search happen?
It depends on the "tool" you are using, but it generally involves retrieving a set of candidate relevant documents (papers, patents, websites...) and either read through full documents or select relevant snippets from them so it can do a wider review.
What are the sources?
Primarily papers (~100M right now), including ArXiv, PubMed, IEEE, the Elsevier suite and many more individual journals. As well as grants from most national and international funding organizations. And US patents, with ongoing expansion to other countries. Updated daily whenever possible.
We also have the option to complement scientific literature with web research, it can be good for example to review public documentation from scientific instrumentation and consumable providers on experimental protocols.
How sub-niche can it actually get?
It's hard to convey, it would be easiest if you just try it, it really can get as niche as you want it to, it can adapt to the level of expertise you operate at. If you are a researcher with a university email you can use most of the relevant functionality for free.
You are right, the examples are not very representative. The intent was to make them understandable to a wide array of researchers with different backgrounds, but still, yeah they could be better.
Sure, I can sign up. But can you clarify data retention and usage policies? As researchers we're always worried about being scooped. Despite complaining about the system and incentives in other comments, we're bound to the game we're forced to play to achieve scientific progress (if it weren't for this stupid game I'd research completely in the open). I can't find information on how data is used. This is especially important if you're discussing institutions like work, as many of them can't even use standard GPT4 accounts do to privacy concerns.
I'll provide some comments about what I personally would like but clearly personal preferences. I'll say that there's a lot that can be done in this space that doesn't even need AI.
For literature review:
Let's say I am a ML researcher (true) who focuses on 3D semantic image segmentation (not true) or some other appropriate niche (we should be able to get much more narrow btw). How do we deal with this literature search? As a simple example, let's say I wanted to get introduced to a topic that my colleague is working on. Finding a proper survey paper is often a surprisingly cumbersome task. Making this searchable (especially with a ranking not just by citations by some coverage metric -- too many 10 page "surveys") would be exceptionally useful. In addition, I'd love to have graphs that I can see paper or author connections through different sorting methods. A timeline is often highly useful to understand the progress of a research topic and especially helpful to making new surveys and onboarding new grad students (or anyone into a new topic).
Citation and Reading List:
I love Zotero, but it is so fucking limited it gets to me. Here's my issues:
There's no good way for me to create a prioritized reading list (which should be topic based). Sorting papers that come across my desk is a cumbersome task and they just don't get sorted. But I am able to assign prioritization values to these works (especially attached with a certain project or domain). But Zotero does not make this a realistically viable task. Realistically I have multiple browser windows and store arxiv links in tabs or use Obsidion (more cumbersome). This is actually something ML could help with, at least for the automatic classification (we can discuss other metrics too if you want).
Discovery is a challenge, especially in a new niche, but the bigger problem is actually wrangling the papers I've already discovered.
What also sucks is that I can't markup these works. Preferably on my iPad (OSX would be fine since easy to share). Adding notes is also a bit cumbersome. I often want to add notes to papers to categorize them as citation references to certain topics or place them into importance for certain niches, and this should be different from a reading list. And then comes the problem of sharing. Not great.
I said there's a lot that can be done without ML, and that is because I don't even have good tools to sort and catalogue the papers that come via natural discovery. This comes through multiple devices too, and idk if I'm going to get that on my computer or phone (twitter is still a great place for paper discovery, and comes with author comments/summaries. No tool catalogs this).
Another great pain is citations. Authors aren't helpful here, but grabbing the proper journal/conference citation is surprisingly a non-trivial task. They aren't always indexed by Semantic Scholar or Google, who both have quite large delays. But there's definitely no automatic correction in Zotero for when I add a arxiv paper (via plugin) that replaces the arxiv bib with the proper bib, even if that is indexed by GS/SS. Every time I write a paper this task has to be done and my zotero list gets updated.
So kinda in ...
> How many people on the team have a PhD?
We are already working like this with a few other academics, as well as a number of pilots with R&D teams from industry. And yes my co-founder is a post-doc in theoretical quantum physics, and we have a number of other PhDs with experience in academia in the team, mostly chemistry and life-sciences. Myself I started early in the startup game, I have a masters in CS and have some experience working in academia (a couple of publications, supervised 4 masters thesis, interned with a couple of research groups...), but not a PhD just yet.
> as many of them can't even use standard GPT4 accounts do to privacy concerns
Azure guarantees full isolation and privacy, right now it's the only way to use GPT4 with such guarantees, and no other LLM right now can compete with GPT4 in the domain we operate in. From our side, we are working with a partner on full ISO certification, but obviously security and privacy have been a priority from the beginning.
> (regarding all your preferences on research tools)
There's too much here to cover in a comment, please contact me so we can discuss this further. Just to say that our use of LLMs has started recently (early this year), but we have 6 years of previous experience on doing things like creating global co-authorship and citation networks through state-of-the-art disambiguation and aggregating from many different sources, and working on large-scale information-retrieval infrastructure (complex boolean search with ranking). We have worked a lot with LLMs in the past year for MirrorThink to extract knowledge from the actual bodies reliably at large scale, but this is backed with a lot of non-AI substance built over years.
> I don't want these fancy ML tools to summarize and write papers for me
As stated in a previous comment, helping with writing has never been our focus. I agree with your characterization, sure LLMs can save some time with writing, but it's still hard to have them write coherent long-form text with real substance. And it's just kind of unethical, particulary in the sense that it just inflates the amount of text scientist need to keep-up with, turning small advancements into low-density decent looking publications just to get "points", this goes completely against what we are trying to do.
We do want to save a lot of time on research, and MirrorThink shows that LLMs can indeed add a lot in this area and that it is possible to keep them very factually grounded. But again, I'm not very attached to "fancy" AI, I was quite skeptical of this tech for a long time until it became hard to ignore this year. We'll do whatever works for you, however mundane. Frankly, our business needs are covered in the B2B side and I have a budget to simply contribute back to scientists, so that's the only goal.
> I'm 30 year old ABD, not an old tenured professor ... some aspects also have to be usable by my advisor or boss
That is ideal, we want people "in the treches" so to say, actually doing hands-on research every day. We'll figure it out in terms of accessibility to your supervisor, it's a great non-obvious point that is only apparent to those on the inside.
> we can discuss my anti-{journal,conference} dream and how to create a realistic peer review system
Would love that too, we've had lots of interal discussions in this area over the ...
It will become a circle: somebody uses AI to generate boring text (probably not even themselves will read it), then somebody else puts the boring text for AI to summarise. Nobody learns anything. Repeat.
Sokal-as-a-Service.
Honestly, which publicly available AIs are capable of doing that?
At least not the publicly available LLM AIs: I asked those quite many questions about scientific topics, but they could not give me any answer that helped me - it was a complete waste of time.
[Summarize the prior research in paper XXX.YYY focusing on the DNA synthesis section]
Bla BLA BLA
[Describe in two paragraphs how DNA synthesis works]
BLA BLA BLA