They are an information company, authoritative and trusted. They must be positioning themselves with data as a service, like the recent Reddit contract.
I mean, Ahrefs puts my traffic at half that, and my analytics report 20k sessions this month (not that it's high, but it's certainly not 15). No clue where Ahrefs gets those numbers from.
We need high quality data to train foundational models. Many of us thought textbooks, encyclopedias, etc were a great way to start. I have a list of names of every type of encyclopedia set on eBay for that exact reason. One was Britannica.
I hope this funds contracting top authors for the Encyclopaedia. If you check an article's authors, you'll see that many are written by experts in their field (professors, high-level practicioners, etc.) and some by leading people in their field. In the past they were written by people like Einstein and Freud. And often articles by leading experts can be very engaging, written with a personality and point of view (though they seem to cover all sides).
But it seems like many of those authors are long retired, maybe dead, and I wonder how old their contribution is (Britannica seems to update it). And many articles say they are written by the 'Editors of Encyclopaedia Britannica', though maybe that always was the case.
I have nothing against the editors, but an expert in that domain will have perspective, insight and current knowledge (i.e., not yet in textbooks) that is impossible to match, imho. I'd love to see them again hiring leading people to write.
Luminaries like that also have their hobby horses and grievances that may color their writing. The editors are people doing a lifelong work of curating all kinds of information, including through primary sources such as papers in peer reviewed scholarly journals, such as those written by the contributors you mention. So it stands to reason that there is a core contributed by a leader in their field at the time, which is subsequently maintained by the editors and contributors.
Yes, that would be great. A debate even, though that would be an odd format in an encyclopedia.
In Britannica the way it is now, I've noticed that related topics are sometimes written by experts with different viewpoints, and where they overlap you can see the contours of the terrain, so to speak.
> Luminaries like that also have their hobby horses and grievances that may color their writing
If you look at specialist academic encyclopaedias (e.g. the Stanford Encyclopaedia of Philosophy), very often the article on theory X is written by one of the major proponents of theory X – because only they feel sufficiently motivated to write it, and because the editors think it is charitable to grant the proponent of a theory the opportunity to produce their best defence of it. But, they'll make sure to include coverage of the criticisms of the theory, because that's what an academic is expected to do. And the editors will make sure the article overall, and the coverage of the criticisms in particular, is fair rather than overly slanted – e.g. by recruiting one of those critics as a reviewer.
Why would I trust that (even if it strongly supports my claim)?
And if true, I wonder what it would look like in the absence of Encyclopaedia Britannica, which accounts for 99% of the instances that I've seen the archaic spelling.
I agree that the combination of expert and editor is best, but you still want an expert contributing modern information, including what isn't yet in the textbooks. Imagine an expert in CPU manufacturing having written something 15-20 years ago; the editor would have trouble making sure it was now correct and complete.
One technique which reduces hallucinations, is to augment the LLM with a database of facts (such as the text of an encyclopaedia). You take the prompt, and you do a semantic search on the fact database (RAG, retrieval augmented generation) to find sentences/paragraphs/documents which relate to the prompt, and you add them to the prompt. Another related approach is function-calling, where instead of just doing a query based on the text of the prompt, you let the LLM analyse the prompt and come up with search terms, and then allow the LLM to do a search for the terms it has chosen. You can prompt and fine-tune the LLM to cite all its claims to the fact database.
Another approach that can help, is after you generate the response, you then submit the response to an LLM (even the same LLM), with a prompt to check it for errors (errors of reasoning, misrepresentation of citations, etc). Often, LLMs can do a decent job on catching their own errors, including hallucinations.
The more advanced an LLM is, the less likely it is to hallucinate, and the more likely it is to pick up on its own mistakes. Llama-7b hallucinates a lot more than GPT-4, and is far less likely to detect itself doing so.
You can't make an LLM foolproof, but you can't make humans foolproof either. Humans "hallucinate" too – e.g. students write essays with erroneous claims, cited to sources which never actually make those claims.
> One technique which reduces hallucinations, is to augment the LLM with a database of facts (such as the text of an encyclopaedia). You take the prompt, and you do a semantic search on the fact database (RAG, retrieval augmented generation) to find sentences/paragraphs/documents which relate to the prompt, and you add them to the prompt. Another related approach is function-calling, where instead of just doing a query based on the text of the prompt, you let the LLM analyse the prompt and come up with search terms, and then allow the LLM to do a search for the terms it has chosen. You can prompt and fine-tune the LLM to cite all its claims to the fact database.
Note that these are really very closely related, almost to the point of being slight variations on a single technique: both are “do a search on a database and include the results in a prompt to the LLM”; they differ in one uses something outside of the LLM to calculate the search query (typically, just an embedding of the user prompt used to search a vector DB), the other prompts the LLM so that it produces an appropriate search query.
> Note that these are really very closely related, almost to the point of being slight variations on a single technique: both are “do a search on a database and include the results in a prompt to the LLM”; they differ in one uses something outside of the LLM to calculate the search query (typically, just an embedding of the user prompt used to search a vector DB), the other prompts the LLM so that it produces an appropriate search query.
They can produce rather different results – the LLM can do a better job than just an embedding at picking out what's truly important in the prompt, which can increase the relevance of the returned documents. I've seen before where a vector search returns documents which contain some similar language to the prompt but which are actually talking about something completely unrelated. LLMs are less likely to do that sort of thing, because they have a much better understanding of what's important in the prompt and what's just fluff/waffle
Do you think the reason hallucinations occur in LLMs is because the training data includes the hallucination? You are very wrong. Hallucinations are not a training data problem, at least in the way you are implying.
Are hallucinations really only a problem with "non factual" training data? Looking at all the made up library methods that ChatGPT uses in the code it produces, it made me think this is a general LLM/GPT problem.
Just because it's printed in a book doesn't mean it's any more factual than any other source of information. Many studies and reviews have found Wikipedia to be more accurate overall than Encyclopedia Britannica. In fact, because it's in a book, it's harder to correct, and more insidious because of opinions like the one you just expressed.
In both cases, you should not treat the information as canonically or authoritatively accurate or factual. Biases, gaps, outright lies and fabrication exist in any large collection of human writing.
That comment is an amazing demonstration of the intellectual failure of Internet users (as you somewhat acknowledge) and the reason Wikipedia is so popular.
If you know it's heavily biased, maybe find something else?
The post I'm replying to was asking for a list of citations where Wikipedia was equally accurate to Encyclopedia Britannica. That wiki article has several of those citations (as well as Britannica's attempted refutations).
I understand that you can't read every link in a comment, but you certainly don't have to post a comment like this if you haven't at least looked into what I posted.
It's not a book; Britannica is a website now. They might sell you some books if you try, but that's not what they are editing for or producing.
Britannica's advantage is that it's written by experts and edited by professionals. If I want knowledge - for business, for my health, for legal, accounting, car repair, IT - I ask experts.
There are downsides - it's not nearly as large as Wikipedia, and it doesn't engage the crowd. But I know someone with expertise verified the facts, and the completeness, correctness, and consistency (the three C's of accuracy) of each article.
That will be interesting. I suspect that the hallucinations come from combining unrelated data in ways that stretch reality. However, using the general Internet as a source of Truth seems like it was never a good idea.
I took a brief look at their online article about the British empire. While it does briefly mention Jamaica requiring "conquest" in the origins section, it seems mostly oblivious to the consequences of the empire's "commercial ambitions" for the local populations. Not sure that would form a great "factual" source of truth to train an AI on.
An LLM that only used “factual encyclopedia data” as its training set would probably be very good at producing content that looked like encyclopedia articles, but bad at anything else.
Probably wouldn’t be any less prone to confabulation than any other LLM, and given how limited in coverage the sum of any factual encyclopedia is compared to typical LLM training sets, would have even a spottier base of factual information that most LLM’s have to work from.
> Hopefully there will be better alternatives (to LLMs) at some point that don’t hallucinate.
There are AIs that don’t “hallucinate” (a bad metaphor), e.g., most of the technologies developed in the previous rounds of AI – like expert systems.
I doubt that we will find more advanced, equally-or-more general AI techniques than LLMs that don’t “hallucinate”, what we’ll probably do is find better ways to build systems around LLMs (or more future AI architectures) that allow them to resolve “hallucination” better before presenting results.
It’s not as if natural intelligences don’t both literally hallucinate and, more relevant to what is described as “hallucination” in AIs, confabulate when pressed to answer without access to facts, and we know with them that grounding in physical reality via sensory data and having access to factual references doesn’t mitigate those effects (and we’ve also been able to demonstrate the latter with LLMs.)
You’re missing that you can reinforce information by repeating specific, data sets. They could use some repetitions of it to drill in the knowledge, put in a lot of content that’s less reliable in a different style, and re-run high-quality content afterwards to reinforce it.
The AI’s already accidentally do that for topics people repeat a lot on the Internet, esp political. ChatGPT used to fight with me over some of those. That it would not relent or stop redirecting me showed that they can be trained to follow authoritative sources, too.
Strange to learn that this is a private company, I'd always assumed that the Encyclopedia Britannica was produced by the UK government, in the same vein as the CIA World Factbook.
Given how long it's been since EB was at the peak of its commercial and cultural import, it's hard not to worry that this isn't just another example of Vampire Capitalism looking for a way to enrich itself while sucking a once great company dry.
It's a great resources almost never on SERP page 1. I hope they invest money in good SEO b/c there's some much garbage out there that EB deserves more love. And many things you look up don't change and the article written in 1980 is better than the LLM version written in 2024.
I wonder what the oldest company to go public is. This isn't easy to Google, because search engines want to give you the public company which is the oldest, even though that's not the same thing. I see this article about Birkenstocks, which company is apparently almost 250 years old. Surprising! But it does not mention Birkenstock being the oldest company at the time of IPO, so there must be an older one.
I'd look for a business in a trade that goes back to ancient times, is in demand today, and needs capital. Maybe something more institutional, like a academic institution? A hospital?
Also, I'd look in a country with both large securities markets and a long history (i.e., not the US). Maybe Japan, Italy, or France?
I believe the record is 613 years, by Stora Kopparbergs Bergslags AB, a Swedish mining company.
It was founded in the 13th century as Stora Kopparberg (first record is a private stock sale in 1288). It went public on the Stockholm Stock Exchange in 1901.
Today it's known as Stora Enso Oyi, following a merger with the Finnish company Enso in 1998.
I fail to see how becoming beholden to stakeholders will make an encyclopaedia a better publication. A real shame. Feels like nothing is sacred anymore.
72 comments
[ 5.0 ms ] story [ 123 ms ] threadI'm guessing it's the last two letters that are going to get them that valuation.
Encyclopaedia Britannica is positioning itself not only as a tech firm, but an AI firm.
What's going on?
Britannica has a contract with Reddit? I tried searching around online but didn't find anything.
But it seems like many of those authors are long retired, maybe dead, and I wonder how old their contribution is (Britannica seems to update it). And many articles say they are written by the 'Editors of Encyclopaedia Britannica', though maybe that always was the case.
I have nothing against the editors, but an expert in that domain will have perspective, insight and current knowledge (i.e., not yet in textbooks) that is impossible to match, imho. I'd love to see them again hiring leading people to write.
In Britannica the way it is now, I've noticed that related topics are sometimes written by experts with different viewpoints, and where they overlap you can see the contours of the terrain, so to speak.
If you look at specialist academic encyclopaedias (e.g. the Stanford Encyclopaedia of Philosophy), very often the article on theory X is written by one of the major proponents of theory X – because only they feel sufficiently motivated to write it, and because the editors think it is charitable to grant the proponent of a theory the opportunity to produce their best defence of it. But, they'll make sure to include coverage of the criticisms of the theory, because that's what an academic is expected to do. And the editors will make sure the article overall, and the coverage of the criticisms in particular, is fair rather than overly slanted – e.g. by recruiting one of those critics as a reviewer.
Edit. Ah!
Stanford Encyclopaedia of Philosophy. I read back a couple of comments and it clicked
And if true, I wonder what it would look like in the absence of Encyclopaedia Britannica, which accounts for 99% of the instances that I've seen the archaic spelling.
If they can also completely eliminate hallucinations, they could be onto something useful.
Sure it won't come from just having a good data set... they'll need to do other stuff as well. Hopefully not impossible. :)
Another approach that can help, is after you generate the response, you then submit the response to an LLM (even the same LLM), with a prompt to check it for errors (errors of reasoning, misrepresentation of citations, etc). Often, LLMs can do a decent job on catching their own errors, including hallucinations.
The more advanced an LLM is, the less likely it is to hallucinate, and the more likely it is to pick up on its own mistakes. Llama-7b hallucinates a lot more than GPT-4, and is far less likely to detect itself doing so.
You can't make an LLM foolproof, but you can't make humans foolproof either. Humans "hallucinate" too – e.g. students write essays with erroneous claims, cited to sources which never actually make those claims.
Heh, we need "post-LLM" AI's then that don't hallucinate at all. ;)
Note that these are really very closely related, almost to the point of being slight variations on a single technique: both are “do a search on a database and include the results in a prompt to the LLM”; they differ in one uses something outside of the LLM to calculate the search query (typically, just an embedding of the user prompt used to search a vector DB), the other prompts the LLM so that it produces an appropriate search query.
They can produce rather different results – the LLM can do a better job than just an embedding at picking out what's truly important in the prompt, which can increase the relevance of the returned documents. I've seen before where a vector search returns documents which contain some similar language to the prompt but which are actually talking about something completely unrelated. LLMs are less likely to do that sort of thing, because they have a much better understanding of what's important in the prompt and what's just fluff/waffle
No. What a bizarre take. Why would anyone think that?
Having a training data set that's not based on fiction sounds like it could be a useful thing.
If they can somehow get rid of the hallucinations too it could be good.
Sorry (not sorry) for assuming that the second sentence in your comment was related to the first.
Hopefully it's clearer now. ;)
In both cases, you should not treat the information as canonically or authoritatively accurate or factual. Biases, gaps, outright lies and fabrication exist in any large collection of human writing.
I'm not doubting this, but do you have citations?
https://en.wikipedia.org/wiki/Reliability_of_Wikipedia
If you know it's heavily biased, maybe find something else?
I understand that you can't read every link in a comment, but you certainly don't have to post a comment like this if you haven't at least looked into what I posted.
Britannica's advantage is that it's written by experts and edited by professionals. If I want knowledge - for business, for my health, for legal, accounting, car repair, IT - I ask experts.
There are downsides - it's not nearly as large as Wikipedia, and it doesn't engage the crowd. But I know someone with expertise verified the facts, and the completeness, correctness, and consistency (the three C's of accuracy) of each article.
At the end of the day it's just a single centralised source
Probably wouldn’t be any less prone to confabulation than any other LLM, and given how limited in coverage the sum of any factual encyclopedia is compared to typical LLM training sets, would have even a spottier base of factual information that most LLM’s have to work from.
> If they can completely eliminate hallucinations
Yes, but that’s orthogonal to the training set.
Of course. I didn't say otherwise.
> An LLM that ...
Note that I also wrote "AI" rather than "LLM". ;)
Hopefully there will be better alternatives (to LLMs) at some point that don't hallucinate.
My point is that having one of those (something that doesn't hallucinate) trained on factual data sounds like it'd be useful. :)
There are AIs that don’t “hallucinate” (a bad metaphor), e.g., most of the technologies developed in the previous rounds of AI – like expert systems.
I doubt that we will find more advanced, equally-or-more general AI techniques than LLMs that don’t “hallucinate”, what we’ll probably do is find better ways to build systems around LLMs (or more future AI architectures) that allow them to resolve “hallucination” better before presenting results.
It’s not as if natural intelligences don’t both literally hallucinate and, more relevant to what is described as “hallucination” in AIs, confabulate when pressed to answer without access to facts, and we know with them that grounding in physical reality via sensory data and having access to factual references doesn’t mitigate those effects (and we’ve also been able to demonstrate the latter with LLMs.)
The AI’s already accidentally do that for topics people repeat a lot on the Internet, esp political. ChatGPT used to fight with me over some of those. That it would not relent or stop redirecting me showed that they can be trained to follow authoritative sources, too.
I wonder what the oldest company to go public is. This isn't easy to Google, because search engines want to give you the public company which is the oldest, even though that's not the same thing. I see this article about Birkenstocks, which company is apparently almost 250 years old. Surprising! But it does not mention Birkenstock being the oldest company at the time of IPO, so there must be an older one.
https://www.npr.org/2023/09/13/1198324691/birkenstock-ipo-st...
Also, I'd look in a country with both large securities markets and a long history (i.e., not the US). Maybe Japan, Italy, or France?
It was founded in the 13th century as Stora Kopparberg (first record is a private stock sale in 1288). It went public on the Stockholm Stock Exchange in 1901.
Today it's known as Stora Enso Oyi, following a merger with the Finnish company Enso in 1998.