One of my favorite LLM use-cases is alternate history extrapolation. Since there’s no formal ‘right’ to a counterfactual, then it can’t be ‘wrong’, and it pretty much always clies me into other events of relevance that are hinge points on my history bead of interest.
Great use case of AI. Almost 4 years after the sensational release of GPT3.5, the best use case of AI is still being a powerful search engine that can gather information from all corners of the digital world.
This. They still make a lot of mistakes, but their working memory and processing of a massive corpus of data are obviously beyond what any human could ever do. But when they flag something interesting for expert review you can do some really good stuff with it.
Slightly tangential: One way I found of limiting impact from hallucinations was to get an LLM to produce, say, a Python program which processes the data in question. I was making some flashcards for astronomical data and tried to get an LLM to put it all together in Typst, but correcting the hallucinations got to be too much to handle, so I had it write a Typst template and also a Python program which would convert the ephemerides data fields to a Typst file sourcing the template. Worked flawlessly.
I've been using AI to help with my genealogical research, and it's been fantastic. I am loving pushing the family history back and catching mistakes that someone who many others share as a common ancestor have made.
I'm very interested in this topic. I have been using AI for genealogy, but not in ways that have been able to push family history back or catch mistakes (yet).
So how do you use the models more specifically? I have so far used them to write a - in some ways - better genealogy program, with some features I've sorely missed especially with respect to DNA genealogy. I've also tried to use them to help with the tedium of transcribing horrible handwriting, but the results have been so poor and with such a high level of hallucinations that I've not tried that for a while.
- ask it to generate Lean 4 proofs based off centimorgans
- extract subject, object, subjects from all source texts and create a graph database (maybe someone mentioned a red dog in their oral history, and someone else mentioned a sick dog in a eulogy, ask ai to explore weakly correlated connections, sometimes pays off)
> ask it to generate Lean 4 proofs based off centimorgans
That's not confidence inspiring. It'll give you proofs I'm sure, but with what assumptions? You need a full blown model for crossover, and even that would hardly help, because you don't know what assumptions went into the segment match. What size segments are filtered out, how many mismatches are ignored, how interpolation is done, how no-calls are treated etc. Let alone the cM value for services which don't give segments!
And then there's that the question for instance Bettinger's shared cM project asks (given this relationship, what shared cM level do you expect to see?) is actually very different from the question we most often ask (given this shared cM level, what relationship should I expect?). When you get into the distant cousin range, those questions can be very different indeed. And we're not even getting into endogamy.
They say the first principle is that you must not fool yourself, and you're the easiest person to fool. I'm worried that AI can be really powerful self-fooling tools!!
aha I appreciate the rant, and you obviously know what you are talking about.
I have used the Lean 4 approach and I wasn't proposing it solves the problem, but it has reduced the search space for me in certain circumstances. At best it's a nice way for agents to keep in their context windows potential relations, I made it create enormous amounts of hypothetical nodes and then visualized it so I could talk with my relatives about potential paths.
As a side note, I believe, if we had a better DNA site that had great API access (and exposed genomes), I think agents could probably solve genealogy, even as far as complex endogamics.
(I don't think centimorgans is that useful past 4 generations kind of thing)
Current models are amazing at one shot decoding hand written vital records. I thought there might be some friction, but if there is, I haven’t found any. I’ve unlocked lots of detail from records I already had just because I didn’t try to translate the handwriting due to the time required.
I’ve written my own genealogy app as well, it has reinvigorated my efforts. I have a multi-level data model (evidentiary below, curated above) that has helped me piece together people across a multitude of disparate sources. Claude has been my sidekick, and it’s amazing- especially for the more toilsome tasks like extracting people’s details from newspaper articles.
I have split up the data model into a handful of layers in order to separate concerns. The evidentiary layer are the raw bits - the media, sources, clippings, photos, etc. From there I derive "personas", which are the (incomplete and possibly inaccurate) representations of people from those media bits. Above that is the curation layer, where the curator takes those personas and compiles persons from them.
So if there are a bunch of articles that talk about someone's birthday but a couple give conflicting data, that is all present and recorded in the evidentiary layer, and it is the responsibility of the curator to decide which is the actual birthday for that person.
edit: the "below/above" terminology is a dependency relationship. The facts undergird the conclusions drawn from them. To flip that relationship would be to "retcon" facts to fit the conclusions a curator has already drawn, which would be a mistake.
I've been doing family history on my aboriginal Australian side, there were a bunch of Lutheran missions, and ever so kindly they digitized 800 pages for me, but it was all written in German. I transcribed and translated all of it.
I sent it back and they didn't acknowledge it, probably for various reasons, and I doubt they would like me linking it above.
Beyond being fascinating in general, I also found that from a different state in Australia there was an aboriginal who became a man of letters came to my ancestral state and was the first to write the language of the tribe, I think (not something easy to prove) it's the first ever written version of the tribal language circa ~1870 of Kuku Yalanji.
> I sent it back and they didn't acknowledge it, probably for various reasons, and I doubt they would like me linking it above.
Because if you study this stuff long enough, you'll find the LLMs hallucinate in this space far more than they do in a discourse or programming project.
It can be close enough to seem useful, but make critical errors that can cause rippling problems.
I've done a lot of the same sort of thing with Latin and chancery and while it can really help in understanding it on a personal level, it takes combing over what it has said and zooming in, comparing it with other information, and understanding the historical context. Otherwise it can completely mangle clauses, contexts, etc.
It is remarkable how well they can start to read things like chancery script, but again, many errors. Not reliable, especially in such large volumes. It would require extensive review. And they are probably getting flooded wth things like that now. And not all of them will be benevolent. History is laden with intentional corruptions and the flat, context-less reading that LLMs give can bury that even further.
Don't inundate researchers, churches, and records offices wth this stuff. But it is absolutely a space where it can help, with proper organization and engagement.
Or, more simply, affairs, unwed couples, incidences of rape that were a dark stain in the victim’s life, etc.
A friend of mine loves to think about heritage and genealogy and when I brought up genetic testing and “wouldn’t it be neat to find out if someone in your lineage wasn’t actually in it?” they were quick to point out that they’re not interested in that because it doesn’t matter and her parents are her parents (and her great great grandparents are her great great grandparents).
Discovering infidelity and/or abuse in your family tree 2-3 generations back often (but never ever always) does little but harm to one’s own genealogical journey.
My unlock in genealogy research was using https://www.familysearch.org/ with from Claude Code with Claude-in-Chrome (for browser control). You may need to occasionally confirm your humanity and limit Claude’s bot-ness, but it can still move quickly to gather the info.
familysearch.org + ancestry.com + genealogybank.com + archive.org + random institutional websites or government sources have been my top so far.
I throw it all together into a multi-agent team doing fact and source gathering, network tracking, location processing, and coverage analysis. Basically industrializing the research and creation of genealogical profiles of ancestors.
I'd caution others with believing caught "mistakes". (But am also interested in how you use it)
There is a farm cited as an ancestor origin by multiple deceased genealogist. Its a common POI that people with this shared ancestor try to find. Finding it could solidify the established line theory and finally debunk a small ancestor fraction's alternate theory.
Different LLM searches keep returning the same colony with invalid/made up references that don't even cite a partial name match. Manual searches throughout that area returns nothing as well. But since the LLM said so, a century+ of various research and work by professional genealogist gets severed from a crowd sourced public tree because "AI" returned a colony name that ended up helping the alt line.
I did eventually find what I believe is the origin farm, and it strengthens the history written by previous genealogist -- I tried to send the information to the tree maintainers and was outright ignored. LLMs fabricating locations apparently beats a listing in the National Heritage List for England of the exact place name, buildings from that time, and in one of the areas the larger english family is known to have been present.
Make sure you document why/how you determined that a previous research conclusion was a mistake -- generations after you may discover additional information and should be able to weigh the same evidence rather than simply accept (or reject) your novel conclusions.
FWIW, I have tried my best to make the 10s of thousands of books on https://SourceLibrary.org ergonomic for both agents and people. All feedback welcome.
To date, it is the largest collection of agent-accessible translations on the web. The mcp pulls texts and illustrations — and the API provides access to the embeddings. It’s free.
We are based at the Embassy of the Free Mind in Amsterdam, a UNESCO-recognized library of alchemy, magic and mysticism (among other related topics)
And, it’s worth saying, I’m very much inspired by Res Obscura’s line of curiosity-driven humanist inquiry…
Dive into some historical mysteries, there are so many!
Absolutely fascinated by it, both the contents, as well as getting the UI right. Getting the OCR right is also a major achievement. Is there a write-up somewhere on the technical aspects of the work? I have tried using llms for some 19th century books (in Greek) with limited success.
This is a brilliant idea and I've had the idea for gathering the worlds spiritual texts in a similar fashion.
One question I would have, why not put the extracted English txt contents onto github or some other repository? The API and MCP are definitely awesome features, but providing an LLM ready repository seems like it would be very useful as well. Your service then still serves the purpose for original scans/images/illustrations but the bulk of the content is then immediately searchable by an agent locally using the standard cli tools (grep/rg, etc).
49 comments
[ 0.27 ms ] story [ 10.6 ms ] threadSlightly tangential: One way I found of limiting impact from hallucinations was to get an LLM to produce, say, a Python program which processes the data in question. I was making some flashcards for astronomical data and tried to get an LLM to put it all together in Typst, but correcting the hallucinations got to be too much to handle, so I had it write a Typst template and also a Python program which would convert the ephemerides data fields to a Typst file sourcing the template. Worked flawlessly.
So how do you use the models more specifically? I have so far used them to write a - in some ways - better genealogy program, with some features I've sorely missed especially with respect to DNA genealogy. I've also tried to use them to help with the tedium of transcribing horrible handwriting, but the results have been so poor and with such a high level of hallucinations that I've not tried that for a while.
- ask it to generate Lean 4 proofs based off centimorgans
- extract subject, object, subjects from all source texts and create a graph database (maybe someone mentioned a red dog in their oral history, and someone else mentioned a sick dog in a eulogy, ask ai to explore weakly correlated connections, sometimes pays off)
That's not confidence inspiring. It'll give you proofs I'm sure, but with what assumptions? You need a full blown model for crossover, and even that would hardly help, because you don't know what assumptions went into the segment match. What size segments are filtered out, how many mismatches are ignored, how interpolation is done, how no-calls are treated etc. Let alone the cM value for services which don't give segments! And then there's that the question for instance Bettinger's shared cM project asks (given this relationship, what shared cM level do you expect to see?) is actually very different from the question we most often ask (given this shared cM level, what relationship should I expect?). When you get into the distant cousin range, those questions can be very different indeed. And we're not even getting into endogamy.
They say the first principle is that you must not fool yourself, and you're the easiest person to fool. I'm worried that AI can be really powerful self-fooling tools!!
I have used the Lean 4 approach and I wasn't proposing it solves the problem, but it has reduced the search space for me in certain circumstances. At best it's a nice way for agents to keep in their context windows potential relations, I made it create enormous amounts of hypothetical nodes and then visualized it so I could talk with my relatives about potential paths.
As a side note, I believe, if we had a better DNA site that had great API access (and exposed genomes), I think agents could probably solve genealogy, even as far as complex endogamics.
(I don't think centimorgans is that useful past 4 generations kind of thing)
Like if you said "curated below, evidentiary above", what should I understand has changed?
So if there are a bunch of articles that talk about someone's birthday but a couple give conflicting data, that is all present and recorded in the evidentiary layer, and it is the responsibility of the curator to decide which is the actual birthday for that person.
edit: the "below/above" terminology is a dependency relationship. The facts undergird the conclusions drawn from them. To flip that relationship would be to "retcon" facts to fit the conclusions a curator has already drawn, which would be a mistake.
I've been doing family history on my aboriginal Australian side, there were a bunch of Lutheran missions, and ever so kindly they digitized 800 pages for me, but it was all written in German. I transcribed and translated all of it.
https://drive.google.com/file/d/1en9gDgZRP7EmSHU72k6g2Iz3RRf...
I sent it back and they didn't acknowledge it, probably for various reasons, and I doubt they would like me linking it above.
Beyond being fascinating in general, I also found that from a different state in Australia there was an aboriginal who became a man of letters came to my ancestral state and was the first to write the language of the tribe, I think (not something easy to prove) it's the first ever written version of the tribal language circa ~1870 of Kuku Yalanji.
Because if you study this stuff long enough, you'll find the LLMs hallucinate in this space far more than they do in a discourse or programming project.
It can be close enough to seem useful, but make critical errors that can cause rippling problems.
I've done a lot of the same sort of thing with Latin and chancery and while it can really help in understanding it on a personal level, it takes combing over what it has said and zooming in, comparing it with other information, and understanding the historical context. Otherwise it can completely mangle clauses, contexts, etc.
It is remarkable how well they can start to read things like chancery script, but again, many errors. Not reliable, especially in such large volumes. It would require extensive review. And they are probably getting flooded wth things like that now. And not all of them will be benevolent. History is laden with intentional corruptions and the flat, context-less reading that LLMs give can bury that even further.
Don't inundate researchers, churches, and records offices wth this stuff. But it is absolutely a space where it can help, with proper organization and engagement.
A friend of mine loves to think about heritage and genealogy and when I brought up genetic testing and “wouldn’t it be neat to find out if someone in your lineage wasn’t actually in it?” they were quick to point out that they’re not interested in that because it doesn’t matter and her parents are her parents (and her great great grandparents are her great great grandparents).
Discovering infidelity and/or abuse in your family tree 2-3 generations back often (but never ever always) does little but harm to one’s own genealogical journey.
https://m.youtube.com/watch?v=c65QRaR16io
I throw it all together into a multi-agent team doing fact and source gathering, network tracking, location processing, and coverage analysis. Basically industrializing the research and creation of genealogical profiles of ancestors.
Wild times.
There is a farm cited as an ancestor origin by multiple deceased genealogist. Its a common POI that people with this shared ancestor try to find. Finding it could solidify the established line theory and finally debunk a small ancestor fraction's alternate theory.
Different LLM searches keep returning the same colony with invalid/made up references that don't even cite a partial name match. Manual searches throughout that area returns nothing as well. But since the LLM said so, a century+ of various research and work by professional genealogist gets severed from a crowd sourced public tree because "AI" returned a colony name that ended up helping the alt line.
I did eventually find what I believe is the origin farm, and it strengthens the history written by previous genealogist -- I tried to send the information to the tree maintainers and was outright ignored. LLMs fabricating locations apparently beats a listing in the National Heritage List for England of the exact place name, buildings from that time, and in one of the areas the larger english family is known to have been present.
I think this post is missing a lot of context and background. I haven't the foggiest what an "established line theory" is
Make sure you document why/how you determined that a previous research conclusion was a mistake -- generations after you may discover additional information and should be able to weigh the same evidence rather than simply accept (or reject) your novel conclusions.
To date, it is the largest collection of agent-accessible translations on the web. The mcp pulls texts and illustrations — and the API provides access to the embeddings. It’s free.
We are based at the Embassy of the Free Mind in Amsterdam, a UNESCO-recognized library of alchemy, magic and mysticism (among other related topics)
And, it’s worth saying, I’m very much inspired by Res Obscura’s line of curiosity-driven humanist inquiry…
Dive into some historical mysteries, there are so many!
One question I would have, why not put the extracted English txt contents onto github or some other repository? The API and MCP are definitely awesome features, but providing an LLM ready repository seems like it would be very useful as well. Your service then still serves the purpose for original scans/images/illustrations but the bulk of the content is then immediately searchable by an agent locally using the standard cli tools (grep/rg, etc).