Launch HN: Undermind (YC S24) – AI agent for discovering scientific papers
We’re both physicists, and one of our biggest frustrations during grad school was finding research — There were a lot of times when we had to sit down to scope out new ideas for a project and quickly become a deep expert, or we had to find solutions to really complex technical problems, but the only way to do that was manually dig through papers on Google Scholar for hours. It was very tedious, to the point where we would often just skip the careful research and hope for the best. Sometimes you’d get burned a few months later because someone already solved the problem you thought was novel and important, or you’d waste your time inventing/building a solution for something when one already existed.
The problem was there’s just no easy way to figure out what others have done in research, and load it into your brain. It’s one of the biggest bottlenecks for doing truly good, important research.
We wanted to fix that. LLMs clearly help, but are mostly limited to general knowledge. Instead, we needed something that would pull in research papers, and give you exactly what you need to know, even for very complex ideas and topics. We realized the way to do this is to mimic the research strategies we already know work, because we do them ourselves, and so we built an agent-like LLM pipeline to carefully search in a way that mimics human research strategies.
Our search system works a bit differently from casual search engines. First, we have you chat back and forth with an LLM to make sure we actually understand your really complex research goals up front, like you’re talking to a colleague. Then the system carefully searches for you for ~3 minutes. At a high level, it does something similar to tree search, following citation rabbit holes and adapting based on what it discovers to look for more content over multiple iterations (the same way you would if you decided to spend a few hours). The 3 minute delay is annoying, but we’re optimizing for quality of results rather than latency right now. At the end there’s a report.
We’re trying to achieve two things with this careful, systematic agent-like discovery process:
1. We want to be very accurate, and only recommend very specific results if you ask for a specific topic. To do this, we carefully read and evaluate content from papers with the highest quality LLMs (we’re just reading abstracts and citations for now, because they’re more widely accessible - but also working on adding full texts).
2. We want to find everything relevant to your search, because in research it’s crucial to know if something exists or not. The key to being exhaustive is the adaptive algorithms we’ve developed (following citations, changing strategy based on what we find, etc). However, one cool feature of the automated pipeline is we can track the discovery process as the search proceeds. Early on, we find many good results, and later on they get more sparse, until all the good leads are exhausted and we stop finding anything helpful. We can statistically model that process, and figure out when we’ve found everything (it actually has an interesting exponential saturation behavior, which you can read a bit more about in our whitepaper (https://www.undermind.ai/static/Undermind_whitepaper.pdf), which we wrote for a previous prototype.)
You can try searching yourself here:
I did 2 searches. First I asked about a very specific niche thing. I gave me results but none I wanted. It looked like I missed a crucial piece of information. So I did the second search. I started with the final request it written for the previous search and added the information I though I missed. It gave me virtually the same results with a little sprinkle of what I was actually after. A few observations: 1. I'm not sure but it seems like it relies too much on citation count. Or maybe citations in papers make it think that the paper is absolutely a must read. I specifically said I'm not interested in what's in that paper and I still got those results. 2. I don't see much dissertations/theses in the result. I know for sure that there a good results for my request in a few dissertations. None of them are in the results. That said, while I didn't get exactly what I want I've found a few interesting papers even if they're tangential to the actual request. Regarding citations: we use them, but only for figuring out which papers to look at next in the iterative discovery process, not for choosing what to rank higher or lower at the end (unless you explicitly ask for citations). It's ranking based on topic match. If you're comfortable, posting the report URLs here can let us debug. The results from their website aren't sharable, but their lists of references do not seem relevant (ie. they miss the fact that shuttling needs to be in 3D, and the list of experiments for laser cooling to BEC is missing all of the relevant papers). I think, like other research tools, they're more focused on the summarization/extraction of information, rather than the discovery process (though they are similar to us in the way they say they do multi-stage retrieval and it takes some time). I love all the work that's being done in the field. Can't wait for science to be faster. And post.harvard.edu has been sunsetted for alums, so I don't have that email either. I gave it a version of my question, it asked me reasonable follow-ups, and we refined the search to: > I want to find randomized controlled trials published by December 2023, investigating interventions to reduce consumption of meat and animal products with control groups receiving no treatment, measuring direct consumption (self-reported outcomes are acceptable), with at least 25 subjects in treatment and control groups (or at least 10 clusters for cluster-assigned studies), and with outcomes measured at least one day after treatment begins. I just got the results back: https://www.undermind.ai/query_app/display_one_search/e5d964.... It certainly didn't find everything in my dataset, but: * the first result is in the dataset. * The second one is a study I excluded for something buried deep in the text. * The third is in our dataset. * The fourth is excluded for something the machine should have caught (32 subjects in total), but perhaps I needed to clarify 25 subjects in treatment and control each. * The fifth result is a protocol for the study in result 3, so a more sophisticated search would have identified that these were related. * The sixth study was entirely new to me, and though it didn't qualify because of the way the control group received some aspect of treatment, it's still something that my existing search processes missed, so right away I see real value. So, overall, I am impressed, and I can easily imagine my lab paying for this. It would have to advance substantially before it was my only search method for a meta-analysis -- it seems to have missed a lot of the gray literature, particularly those studies published on animal advocacy websites -- but that's a much higher bar than I need for it to be part of my research toolkit. (Also, if you want, you can share your report URL here, others will be able to take a look.) My wife does that for her day job (in the U.K. national healthcare system) and the systematic reviews have to be super well documented and even pre-registered on a system called PROSPERO. The published papers always have the full search strategy at the end. However, because I'm writing a methods-focused review -- we only look at RCTs meeting certain (pretty minimal) criteria relating to statistical power and measurement validity -- what I'm doing is closer to a combination of review of previous reviews (there have been dozens in my field) and a snowball search (searching bibliographies of papers that are relevant). I also consulted with experts in the field. however, finding bachelor's theses has been challenging, but many are actually relevant, so undermind was helpful there. What use case are you thinking of? In my opinion elicit has better looking UI and much more features and further along Feature-wise, we definitely have a lot of work to do :) What crucial pieces do you think we're missing? Their core feature now is a fast keyword search engine, but they also have a few advanced search features through their API (https://api.semanticscholar.org/api-docs/) like recommendations from positive/negative examples, but neither KW search nor these other systems are currently high enough quality to be very useful for us. FYI our core dataset for now is provided by Semantic Scholar, so hugely thankful for their data aggregation pipeline and open access/API. And yes, Semantic Scholar is a wonderful part of the academic commons. Fingers crossed they don't go down the jstor/oclc path. Claude 3.5 is reluctant to provide references—-although it will if coaxed by prompting. Undermind solves this particular problem. A great complement for my research question —- the evidence that brain volume is reduced as a function of age in healthy cognitively normal humans. In mice we see a steady slow increase that averages out to a gain of 5% between the human equivalents of 20 to 65 years of age. This increase is almost linear as a function of log of age. Here is the question that was refined with Undermind’s help: >I want to find studies on adult humans (ages 20-100+) that have used true longitudinal repeated measures designs to study variations in brain volume over several years, focusing on individuals who are relatively healthy and cognitively functional. I received 100 ranked and lightly annotated set of 100 citations in this format: >[1] Characterization of Brain Volume Changes in Aging Individuals With Normal Cognition Using Serial Magnetic Resonance Imaging
S. Fujita, ..., and O. Abe
JAMA Network Open
2023 - 21 citations - Show abstract - Cite - PDF
99.6% topic match
Provides longitudinal data on brain volume changes in aging individuals with normal cognition.
Analyzes annual MRI data from 653 adults over 10 years to observe brain volume trajectories.
Excludes populations with neurodegenerative diseases; employs true longitudinal design with robust MRI techniques. 1. It's on adults. 2. It's longitudinal over multiple years. 3. It studies variations in brain volume. 4. It focuses on healthy individuals. You can see the full results for that search text here: https://undermind.ai/query_app/display_one_search/e1a3805d35... this was the search <https://www.undermind.ai/query_app/display_one_search/cba773...> if you need a reference too it, ie bugs or performance monitoring... let’s say i am interested in coffee and i’d like to get new research papers on it. would this work? From a strategy perspective, we designed it for topic search because it makes more sense to find everything on a topic first, then filter for the most recent, if recent is what you want. That's because there is a lot of useful information in older articles (citation connections, what people discuss, and how), and gathering all that helps uncover the most relevant results. Conversely if you only ever filtered on articles in the last year, you might discover a few things, but you wouldn't have as much information to adapt to help the search work better. So, you can ask for articles on coffee (though ideally it should be something a bit more specific, or there will be thousands of results). Our system will carefully find all articles, then you can filter for 2024 articles or look at the timeline.133 comments
[ 4.4 ms ] story [ 234 ms ] thread