I built an open library for psychedelic and consciousness research that pulls from over a dozen academic APIs every 20 minutes, dedupes them, organizes by relevance, and synthesizes evidence.
I just finished my master's in Psychedelics and Consciousness Studies, and built the library I wanted while studying. I started it over a year ago, but put it down until my practicum advisor suggested I dust it off and go hard on it during my last term.
As of this writing, there are over 35,700 papers by over 82,200 authors organized into 37 topics, growing daily. There's a 2D map of the whole corpus and evidence syntheses on the topic pages, plus the ability to generate your own syntheses with any prompt.
It's built with Rails 8.1, uses one Postgres (for full-text search, vectors, and job queue), and runs on a $24/mo DigitalOcean droplet that I just upgraded from the $12 tier and deploy to with Kamal. Total LLM spend so far is ~$28 in DeepSeek-V4-Flash, trending down after a big one-time backfill push.
Relevance was the hardest part to get right by far. Ensuring we keep papers on LSD the psychedelic vs Lumpy Skin Disease, or Ketamine for depression vs anesthesia in cats was no small feat. The design ended up being a cheap keyword prefilter, then an LLM rubric that accepts or rejects papers. More on that here: https://consciousnesslibrary.org/docs/article-pipeline.html
It's free, no ads, no signup to read anything, and a registered 501(c)(3) I fund myself for now. I'd really like to know what breaks or any other feedback, and would love to answer any questions about it!
So freakin' cool. Overjoyed by your efforts here. I notice when I search for certain different words like "brahmavihara" or just "vihara" sometimes I get different results due to slight differences in the titles of papers. But that might be a feature and not a bug [if there were many papers, for example].
Technical question: Is Gemini not as good at rifling through paper citations? I would think Google scholar would be a moat of sorts for Goog, but maybe every LLM has roughly equal access to it now?
Research question: Did you find any mind-blowing results with your awesome new tool?
Development question: Do you plan on extending this beyond just a search apparatus? I would personally love to have a "factoid list" that actually cites interesting factoids from each article and can link to the source document -- although that is probably a "don't bite off more than you can chew" juncture and what you have is already excellent. But a thought. In case you're interested in expanding on it.
5 comments
[ 0.22 ms ] story [ 7.0 ms ] threadI just finished my master's in Psychedelics and Consciousness Studies, and built the library I wanted while studying. I started it over a year ago, but put it down until my practicum advisor suggested I dust it off and go hard on it during my last term.
As of this writing, there are over 35,700 papers by over 82,200 authors organized into 37 topics, growing daily. There's a 2D map of the whole corpus and evidence syntheses on the topic pages, plus the ability to generate your own syntheses with any prompt.
It's built with Rails 8.1, uses one Postgres (for full-text search, vectors, and job queue), and runs on a $24/mo DigitalOcean droplet that I just upgraded from the $12 tier and deploy to with Kamal. Total LLM spend so far is ~$28 in DeepSeek-V4-Flash, trending down after a big one-time backfill push.
Relevance was the hardest part to get right by far. Ensuring we keep papers on LSD the psychedelic vs Lumpy Skin Disease, or Ketamine for depression vs anesthesia in cats was no small feat. The design ended up being a cheap keyword prefilter, then an LLM rubric that accepts or rejects papers. More on that here: https://consciousnesslibrary.org/docs/article-pipeline.html
It's free, no ads, no signup to read anything, and a registered 501(c)(3) I fund myself for now. I'd really like to know what breaks or any other feedback, and would love to answer any questions about it!
Technical question: Is Gemini not as good at rifling through paper citations? I would think Google scholar would be a moat of sorts for Goog, but maybe every LLM has roughly equal access to it now?
Research question: Did you find any mind-blowing results with your awesome new tool?
Development question: Do you plan on extending this beyond just a search apparatus? I would personally love to have a "factoid list" that actually cites interesting factoids from each article and can link to the source document -- although that is probably a "don't bite off more than you can chew" juncture and what you have is already excellent. But a thought. In case you're interested in expanding on it.