Just because the site says comprehensive does not mean it is comprehensive. Multiple names other databases find are not mentioned. Start from Joscha Bach...
A week ago I posted about an open database I’ve been building to cross reference Epstein case material. That post did way better than I expected (568k views, 4.6k upvotes) and it hugged my server to death twice.
Since then I basically did nothing but ingest, clean, and index more data. The database is now big enough that “just read the docs” is not advice, it’s a cry for help.
What it was last week
~6,000 documents
1,708 flights
2,700 emails
1,438 people
What it is now
1,522,060 documents (all DOJ releases we have access to so far), full text searchable
1,708 flights (1997 to 2019) with manifests where available
10,000+ emails indexed with threading
1,350 people (cleaned: removed duplicates + nuked a bunch of false connections)
638,000 docs run through redaction analysis
~1.8M individual redactions detected
~616k flagged by our tooling as “looks questionable, take a closer look”
~39,500 pages of text recovered from under black bars (you can see examples on the site)
107,000 named entities pulled out via NLP (people, orgs, places, dates)
1,530 audio/video transcripts
4,300+ photos/media (raid photos, exhibits, property shots, government releases)
That’s not a typo: 1.5 million documents.
If you search a phrase, it searches inside the actual pages (OCR where needed) and email bodies, not just titles.
So what changed, besides “everything is bigger”?
1) The redaction stuff is getting hard to ignore
I’m not saying “every redaction is evil.” Some of them obviously protect victims, minors, addresses, etc.
But the patterns are weird, and the volume is insane.
I also worked with u/Sea_Doughnut_8853, who independently processed 519k PDFs with their own pipeline. That let us sanity check a lot of what we’re seeing across the corpus.
We’re flagging ~616k redactions as “potentially improper” based on patterns (context, repetition, surrounding text). That does not mean “definitely corrupt.” It means “this is the pile worth human eyes.”
We also recovered a lot of hidden text. If you want to judge it yourself, the doc pages show the redaction density and any recovered text we can reliably extract.
2) Entity extraction is the only way to deal with this scale
107,000 entities means you can stop playing whack a mole with PDFs.
It’s still not “truth,” it’s just structure. But structure beats drowning.
3) This week’s real world developments are in there too
If you missed the news cycle, Congress has been pressuring DOJ about redactions, and Rep. Ro Khanna read six previously redacted names on the House floor:
Leslie Wexner
Salvatore Nuara
Zurab Mikeladze
Leonic Leonov
Nicola Caputo
Sultan Ahmed bin Sulayem
Important caveat: being named in a document is not proof of wrongdoing. People show up in emails, contact lists, forwarded threads, or because someone mentioned them.
Related:
Reporting says Wexner’s name appeared in an internal FBI document as “co conspirator,” but he has not been charged.
Maxwell invoked the Fifth in a House Oversight deposition and her lawyer floated testimony in exchange for clemency.
House Oversight depositions are scheduled: Wexner (Feb 18), Richard Kahn (Feb 25), Darren Indyke (Mar 5), plus Hillary Clinton (Feb 26) and Bill Clinton (Feb 27).
All of those items are indexed, with the underlying documents linked where available.
New tools since last week
Full text search: search inside 1.5M documents, 28k OCR entries, and 10k emails
AI research assistant: ask a question in plain ...
3 comments
[ 0.20 ms ] story [ 19.8 ms ] threadDOJ has more comprehensive search functionality.
https://www.justice.gov/epstein/search
https://reddit.com/r/Epstein/comments/1r3joqr/i_mapped_every...
-------
A week ago I posted about an open database I’ve been building to cross reference Epstein case material. That post did way better than I expected (568k views, 4.6k upvotes) and it hugged my server to death twice.
Since then I basically did nothing but ingest, clean, and index more data. The database is now big enough that “just read the docs” is not advice, it’s a cry for help. What it was last week
What it is now That’s not a typo: 1.5 million documents. If you search a phrase, it searches inside the actual pages (OCR where needed) and email bodies, not just titles.So what changed, besides “everything is bigger”? 1) The redaction stuff is getting hard to ignore
I’m not saying “every redaction is evil.” Some of them obviously protect victims, minors, addresses, etc. But the patterns are weird, and the volume is insane.
I also worked with u/Sea_Doughnut_8853, who independently processed 519k PDFs with their own pipeline. That let us sanity check a lot of what we’re seeing across the corpus.
We’re flagging ~616k redactions as “potentially improper” based on patterns (context, repetition, surrounding text). That does not mean “definitely corrupt.” It means “this is the pile worth human eyes.”
We also recovered a lot of hidden text. If you want to judge it yourself, the doc pages show the redaction density and any recovered text we can reliably extract. 2) Entity extraction is the only way to deal with this scale
107,000 entities means you can stop playing whack a mole with PDFs. It’s still not “truth,” it’s just structure. But structure beats drowning. 3) This week’s real world developments are in there too
If you missed the news cycle, Congress has been pressuring DOJ about redactions, and Rep. Ro Khanna read six previously redacted names on the House floor:
Important caveat: being named in a document is not proof of wrongdoing. People show up in emails, contact lists, forwarded threads, or because someone mentioned them.Related:
All of those items are indexed, with the underlying documents linked where available. New tools since last week