HN Explorer

Ask HN: How to get the “average” of multiple texts?

3 points by UgR32zKhQTZFKPW ↗ HN
I am studying creation myth all over the world. I gathered a lot of them and I would like to know if there is an AI that could give me an "average" creation myth if I feed him all the texts.

Are you aware of somsething like this ?

3 comments

[ 2.5 ms ] story [ 21.2 ms ] thread
benj111
Is 'average' the correct question to ask (I take 'average' to be mean)?

It seems to me the commonalities would be what you want to look for. I'd be looking for the most common tropes, and then if the story's the goal, build it up from there.

alex-moon
It sounds like you want the semantic average, or, in other words, a centroid in semantic vector space. The approach I've used to do this in the past is Word2Vec, which excels at handling individual words. Word2Vec isn't going to be able to give you what you want per se but should give you somewhere to start in your search.

Incidentally this is the kind of thing LLMs are very good at. Have you tried just plugging them into ChatGPT/Bard?

ilaksh
If you really need to feed in a lot of text then look into Claude 2 because it has a 100000 token context window.

But Claude 2 or GPT-4 out of the box can probably do a reasonable job of that question without your having to feed the text in.

Try giving it a short description or name of each creation myth, asking it to provide a synopsis and then drill into main points for each at a high level. Then ask it to compare them.

Then finally just ask your question about the average. You could start with that but probably will get better results if you break it down first.