HN Explorer

Show HN: I put PubMed in a vector DB (pubmedisearch.com)

97 points by mpmisko ↗ HN
Hi HN,

As a researcher, I often found myself struggling with the limitations of keyword-based search when exploring PubMed papers. To address this, I created PubMed Search (https://www.pubmedisearch.com/), a tool that leverages a vector database to enable semantic search across medical research literature.

Some key features:

* Daily updates to ensure access to the latest articles

* Semantic search using latest & greatest embedding models

* Some additional useful info about the papers (tldr, journal, publication date, etc.)

Hope you find it useful!

27 comments

[ 5.1 ms ] story [ 79.3 ms ] thread
rkwz
Congrats on shipping!

I'm curious how the search results rankings work, doesn't look like it's based on date or number of citations, but seems to be deterministic (persists over multiple searches). I did a keyword search using one word.

mpmisko
Thanks!

It uses a vector search approach. Your query is embedded in a vector space using a language model and we find the closest vector to the query from the PubMed papers. This is a good summary of the techniques: https://learn.microsoft.com/en-us/azure/search/vector-search.... There are a couple more tricks but this is the gist.

The nice part is that this approach allows you to find relevant papers to your question. E.g, you can ask "Can secondhand smoke cause AMD?" and the very first few papers are answering your question (https://pubmedisearch.com/share/Can%20secondhand%20smoke%20c...). The more specific question, the better. :)

victor106
Cool idea!

What are some papers labeled "High Quality Article"? How do you determine that?

mpmisko
Just looking at stuff like citations and impact factors of journals.
kkielhofner
Nice!

Out of curiosity what model(s) are you using to generate the embeddings?

mpmisko
Glad you like it! I did this as a mini-project within our startup MediSearch (https://medisearch.io/) & the search pipeline is custom tuned for the problem.
lucas_crocker
This is very cool! 2 questions spring to mind:

1. How much did it cost to embed all those vectors and how many articles did you process? PMC is quite large.

2. Could elaborate a little more on your approach to ranking articles? Because I'm familiar with semantic search via embeddings put did you weight those with impact factors/citations? Like how does one even calculate that?

Anyhow, love the idea.

mpmisko
1. We cover all the articles on PMC. The exact cost is hard to estimate because we did a lot of iterations.

2. We do weight those ... it is a lot of trial and error and you have to have good & exhaustive benchmarks.

bdangubic
Hey mate, should search by PMID work? Like 35982160 is PMID for "Rare coding variation provides insight into the genetic architecture and phenotypic context of autism" - not seeing this publication at all in search results...
alex_duf
What storage did you go for, and what search approach?
mpmisko
We use pinecone and it is not ideal, looking at https://turbopuffer.com/ now. They look quite promising :)
yumraj
Did you compare pinecone against pgvector with Postgres? Self hosted of course
cchance
Isn't it funny how the best Choice somehow always comes back to Postgres in the end XD (for most)
yumraj
Yes, that’s where I’m these days. I don’t even think of venturing outside of Postgres these days, except for say things like Redis etc. where there are mature and established options for specific use cases.
mpmisko
Will definitely check pgvector, thanks for the pointer.
cchance
What kinda dimensions did you keep it relatively low to keep costs down?
dpifke
Very cool!

Related: the NIST TREC (Text REtrieval Conference) has had several competitions over the years related to improving the searchability of medical data: https://www.trec-cds.org/

If you have novel ideas in this area, you should consider participating. https://trec.nist.gov/

mpmisko
Thanks! Looks quite relevant
drycabinet
Maybe a stupid question, but how do you compare this against GPT-based search engines?
mpmisko
GPT-based search engines usually use some sort of a database to retrieve context for the LLM to summarize first. This is what people refer to as RAG these days: https://blogs.nvidia.com/blog/what-is-retrieval-augmented-ge....

Some of these GPT engines maintain their own vector DB to do semantic search, others are directly hooked into Bing / Google. So pubmedisearch.com would be one component of a GPT-based engine. We actually have a GPT-based engine here: https://medisearch.io/.

madhatter999
Very promising tool based on a couple of questions I asked it! How did the cleaning of documents look like?
mpmisko
Lots of annoying edge cases as you can imagine, nothing particularly glamorous.
mharig
All 10 thumbs up!

Edit: One suggestion: in the results list, please make the headings links to the articles, too.

mpmisko
Done! Let me know if you have other feedback.