We would not generate novel knowledge if our only interface with the world was a short text based conversation.
In other words, I don't think that's a fundamental difference between humans and AI; we just haven't given AI any opportunity to build novel knowledge.
Part of that is because it can't really interact with the world yet, in a significant way. Part of it is because nobody has cracked on-line learning.
Actually when you give AI the ability to interact with a world, as in RL, it does build novel knowledge.
As true as "everyone is a calculator". There are superficial similarities, but that we achieve a similar result does not mean that we follow the same reasoning (or lack of).
In humans, language is based on the state of the world, and anticipates future states of the world. In an LLM, language is just language.
If an LLM says "my favorite flavor of ice cream is cookie dough", it's just a statistically plausible thing to say.
If I say "my favorite flavor if ice cream is cookie dough", it's because I'm trying to deceive someone, because for some reason I don't want them to know my real favorite flavor is cake batter.
You haven't identified any difference. In order to generate a really good statistically plausible thing to say you need a model of the state of the world and anticipated future states. This is very well known. I wish we didn't need to keep going over this basic stuff, but it somehow comes up again and again every time AI is discussed here.
Of a world, not necessarily the real world. One of the main drawbacks of philosophical rationalism is a mind's ability do deceive itself with constructs that sound plausible and are consistent with the world model. A healthy dose of empiricism is the best cure discovered so far.
In other words, less thinking and more experiments and observations. Thinking alone leads to ignorance.
I mean the logic behind it is sound. Just infantile compared to what we do. But yes I believe that all we do boils down to "if x is true and also y is true, therefore under these circumstances z must be false, I'll give it a go and find out". Plus a little qualified randomness, look at the large percentage of human discoveries being made accidentally (albeit by qualified professionals making best guesses).
Animals are not next token predictors. Most animals don't have language and don't generate sentences/ideas. It is unique feature of humans and big part of our success.
Animals do have sensors and can reason about real world quite intelligently. This is what LLM- models don't have. Surely, all the text in Internet has a lot of textual information about real world, but something "obvious to animal" might not be there.
I feel us humans are kind of both: We have sensors and experiences about real world, but also oral and written tradition.
Once we develope LLM- with cabability to sense and experience real word (perhaps with some evolutionary algorithm to make it better over time) it should start to be close to human beings.
The trivial interpretation is: every word written can be constructed by optimizing a prediction based on current state, what has been written so far, and a sufficiently complex model. This is true of anything computable: just make the method implicitly contain the program by assigning a high probability to any token that is consistent with running the computation one more step. It's also true of anything expressible: just brute force a solution that can be expressed in n words, then assign a high probability to the first word of these n words.
The profound but wrong interpretation is that intelligence is just statistical prediction according to some general-purpose algorithm, and that this algorithm is tractable. Consider something like solving a SAT problem. You're going to have a hard time using any tractable general-purpose algorithm to predict whether x_2 is true for the satisfying solution based on some long CNF statement plus "x_1 is false".
Now, what you _can_ do is augment your model so that if the previous tokens constitute a CNF-SAT instance plus a partial answer, then you cart these off to a SAT solver and output its next token. But the more you do this, the less force the "mere statistical prediction" part holds. The "next-token predictor" is just an interface to an assembly of different approaches; and often, these approaches (like the SAT solver) will output the whole solution all at once for free.
A device or system that, given a set of tokens, predicts the next token purely based on the previous tokens and its own initial state. It is incapable of coming up with a new token type or changing its own workings
On top of your definition being a bit circular, it is also like saying humans can't come up with new things, because we can't come up with new characters when talking online. We're limited by the character-set we're given but that doesn't mean we can't conceive of new characters and more importantly, can't represent new characters using existing character-sets using some encoding scheme to represent that new character. To that end, lets check with chatGPT if it can come up with a new and useful concept for a token, using existing characters to encode for it.
Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally:
P(tₖ|t₁,..,tₖ₋₁)
For example, the original OpenAI GPT tranformer model was trained to optimise the objective:
∑ log P(tₖ|tₖ₋ₙ,..,tₖ₋₁;Θ)
Where n the length of a sliding "context" window and Θ the parameters of a neural net model (here, a Transformer architecture) used to optimise the objectivbe.
> Typically in AI and CS what we mean by "next token predictor" is a system that maximises the probability of the k+1'th token given the k previous tokens. More formally:
> P(tₖ|t₁,..,tₖ₋₁)
Not a meaningful definition unless you say what probability distribution it's maximizing it for!
Because for any system that takes any actions (e.g. human beings), there's some probability distribution that that's true for...
No, the conditional distribution is a construction that transforms one probability distribution to another; you still haven’t said what the underlying distribution was.
Do you think it's possible that the person who literally wrote (multiple) papers with the answer AI gave, may phrase their question to the AI with similar, or even the exact phrasing they used in previous papers on the same subject ? If I load a bunch of lorem ipsum into AI scientist, labeled as scientific papers, then ask the AI "lorem ipsum" do you think it'll spit out some gibberish from those specific papers?
it is if you're presenting it as coming up with novel information rather than essentially acting as a search engine for information from its training data (which we already know ai does well, at least with some caveats)
Colour me unsurprised - even not knowing data leakage had occurred, the hypothesis was underwhelming, as I mentioned in a comment on an earlier discussion. I sometimes despair for the state of thinking in science these days given how quickly people fawn over entirely pedestrian thinking and work.
I mean, to be fair, "knowing all of the relevant literature" is a great first step for solving a problem! And a LLM can probably be better about not forgetting than humans are (though I'm guessing they do much worse on the hallucination front than humans do- humans tend to know when they are playing a hunch). But "This paper from a lower-tier journal two years ago suggests you should look at X" is a very valuable thing to have!
40 comments
[ 3.1 ms ] story [ 84.3 ms ] threadIn other words, I don't think that's a fundamental difference between humans and AI; we just haven't given AI any opportunity to build novel knowledge.
Part of that is because it can't really interact with the world yet, in a significant way. Part of it is because nobody has cracked on-line learning.
Actually when you give AI the ability to interact with a world, as in RL, it does build novel knowledge.
As true as "everyone is a calculator". There are superficial similarities, but that we achieve a similar result does not mean that we follow the same reasoning (or lack of).
Finally some logic. Token prediction alone - the behavior - isn't proof of a lack of intelligence.
Better reasoning gives better token predictions. And, sometimes, the ability to invent new tokens entirely.
If an LLM says "my favorite flavor of ice cream is cookie dough", it's just a statistically plausible thing to say.
If I say "my favorite flavor if ice cream is cookie dough", it's because I'm trying to deceive someone, because for some reason I don't want them to know my real favorite flavor is cake batter.
In other words, less thinking and more experiments and observations. Thinking alone leads to ignorance.
Animals do have sensors and can reason about real world quite intelligently. This is what LLM- models don't have. Surely, all the text in Internet has a lot of textual information about real world, but something "obvious to animal" might not be there.
I feel us humans are kind of both: We have sensors and experiences about real world, but also oral and written tradition.
Once we develope LLM- with cabability to sense and experience real word (perhaps with some evolutionary algorithm to make it better over time) it should start to be close to human beings.
The trivial interpretation is: every word written can be constructed by optimizing a prediction based on current state, what has been written so far, and a sufficiently complex model. This is true of anything computable: just make the method implicitly contain the program by assigning a high probability to any token that is consistent with running the computation one more step. It's also true of anything expressible: just brute force a solution that can be expressed in n words, then assign a high probability to the first word of these n words.
The profound but wrong interpretation is that intelligence is just statistical prediction according to some general-purpose algorithm, and that this algorithm is tractable. Consider something like solving a SAT problem. You're going to have a hard time using any tractable general-purpose algorithm to predict whether x_2 is true for the satisfying solution based on some long CNF statement plus "x_1 is false".
Now, what you _can_ do is augment your model so that if the previous tokens constitute a CNF-SAT instance plus a partial answer, then you cart these off to a SAT solver and output its next token. But the more you do this, the less force the "mere statistical prediction" part holds. The "next-token predictor" is just an interface to an assembly of different approaches; and often, these approaches (like the SAT solver) will output the whole solution all at once for free.
https://chatgpt.com/share/67bd04cf-1c50-8008-8606-ccac5df989...
And there you have it, novel constructed from available ones, with a genuine pragmatic and LLM specific use case.
https://arxiv.org/abs/2410.05864
Or are you wanting them to use keys that are not on their keyboard?
See page 3 of the OpenAI paper:
https://cdn.openai.com/research-covers/language-unsupervised...
Also see, for alternative formulations:
Hidden Markov Models:
https://en.wikipedia.org/wiki/Hidden_Markov_model
N-Gram word models:
https://en.wikipedia.org/wiki/Word_n-gram_language_model
> P(tₖ|t₁,..,tₖ₋₁)
Not a meaningful definition unless you say what probability distribution it's maximizing it for!
Because for any system that takes any actions (e.g. human beings), there's some probability distribution that that's true for...
Here is the normal distribution probability function.
https://en.wikipedia.org/wiki/Normal_distribution
You can see that there is an actual equation where all the numbers can be filled in except for X, which is the function input.
https://news.ycombinator.com/item?id=43105759
> "It's not just that the top hypothesis they provide was the right one," he said.
> "It's that they provide another four, and all of them made sense.
> "And for one of them, we never thought about it, and we're now working on that."
The Google's co-scientist still seems to make a useful assistant.