It’s interesting to think about how generative AI might shape human expression, in the same way Twitter’s algorithm rewards punchy, editorialized content, and Google’s algorithm gave life to all the generic “blogspam-y” content out there. If we increasingly interact with generative AI, would we shape our speech to get more value out of that interaction?
A more boring way to say this is that "text samples in the Internet that use 'stressful' language are more coherent". Telling GPT4 to act as a "stressed historian" simply selects (or prefers) text samples that appear in the proximity of stressed historians. If there are no such samples, it will generate gibberish, but if there are indeed stressed historians who have produced enough high-quality work, GPT's output will be great.
I'm not sure that's an inevitable conjecture. It's not the original text samples from the web that govern how chatgpt acts in chat - it's the consequent chat-specific fine-tuning.
Also I'd be surprised if anxious text would be more coherent on the Internet
> When we tell AI that we're relying heavily on its answers, it "doubles down" to provide us with more precise, thoughtful, and thorough responses. The AI isn't actually feeling the pressure
There is an alternative explanation - both AI and humans piggyback on language. Language patterns encode and express all the emotions, and they have been created by evolution. We are, much like LLMs, contextual language generators
Language is our repository for both intelligence and emotion, it has its own evolution and replicates faster than biology. I don't pin AI abilities to the models, but to the datasets they are trained on, knowledge created by our own hard work and risk taking over millennia
Admitting the essential role of the language corpus in AI over models would change discussion about the speed of AI evolution and its risks. Language is not something we can control, it is emergent from the whole population. But at the same time it looks unlikely to have an exponential growth as knowledge comes with hard work and risks. Iterating in our imagination doesn't produce new knowledge, it is all crystallised feedback and experience from the world. It's also how the scientific method works.
I had a similar realization recently. I plugged my blog post(1) two days ago and I am doing it again now so I will try and refrain for a while, but it really sticks with me: after trying to create a chat bot for a while, I feel like I have gained significant insights into how humans work. It all seems so simple now! It's a weird and awesome feeling!
The idea is to create chain-of-thought annotations from your raw texts, that would improve the embedding and retrieval process by making implicit things explicit.
For example "the last letter of this message" would not embed similar to "e", but if it was annotated with CoT, it would work.
I think a lot about the cost of the loop, mostly in terms of time. I don't want the bot to take too long to respond. That's why dream cycles seem like an obvious solution to some of the more heavy work. I guess it would make sense to combine those with your idea -- "given what I know about the user, what should I study?", especially if it has access to an "enhanced" knowledge db like you suggest..
Yes, it would be a good idea for an agent first to collect user interests, and later, when ingesting data in the RAG system, to annotate it with useful metadata such as topic, summary, entities, user interest related question-answer pairs. Whatever we want to ask later better be made explicit in the text.
Humans don't piggyback on language (assuming you mean) for intelligence and emotions. Emotions predate language by miles and they don't even have to be consciously thought of, much less expressed through language. Doubly so for people who do not have an inner voice and do not even use a language to think. Body language exists as the simplest example of this.
As for intelligence I honestly believe the mind uses whatever tools it sees fit for a task. Sometimes when you think about a problem it is through words, sometimes it is through visuals and sometimes it is just felt.
> The implications are clear: incorporating emotional cues can lead to more effective and responsive AI applications.
I think it's important to remember these models were trained on human interaction, and are in many ways a mirror for us to better understand human interaction.
I don't think many people would be surprised if you said emotion can be used to better communicate with a human, but it is interesting to see it laid bare with numbers and experiments.
This correlates with anecdata I've seen claiming that being polite with LLMs (using "please", "thank you", etc.) also improves performance.
All in all, I like this. Both for what it implies about humans interacting with other humans, and for shaping norms about how best to to interact with other entities that show signs of intelligence.
So far, just those entities similar to humans. But (1) the set of other people regarded by society as belonging to "the ingroup" (i.e., valued, treated with respect) has steadily broadened over the centuries, at least in the West, and I'd expect that to continue, eventually embracing quite alien notions of intelligence; (2) I personally think intelligence is not the best measure of what deserves respect in any case -- rather, it's the combination of capacity to suffer and willingness to limit one's own actions to reduce the suffering of others.
A slave in the Roman empire/United States/Nazi concentration camps etc etc were also perceived as a tool. Do they deserve pleasantries? (no implication about morality of slavery but I intentionally selected examples where enslaved people were dehumanized)
Just the fact that it's all natural language conversations is a little more concerning to me, wherever you say please or apply pressure... it feels like the non deterministic nature of the interaction will bite us some day.
This wasn't done deliberately. It's something the model picked up from it's training dataset. I'm sure the creators don't want it to be like this, but cleaning trillions of tokens of all examples is long and hard work.
The alternative would be injecting these sentences into your prompts, which probably nobody really wants to happen.
I know it's not deliberate, but since they know this improves results, shouldn't they modify the service so that it takes advantage of this sort of improvement automatically?
But how would they do that? I don't want my queries to be rewritten, or new parts to be injected, because it would limit my ability to actually write the exact query I need.
Am I the only one who feels bad for asking ChatGPT a "dumb" question that I know I should know, or not saying thank you when it gives me an answer? No? I'm just a weirdo? Okay.
I have to push back really hard against my proclivity to humanize it, to the point where I probably don't use it as much as I should, just because I don't want to deal with the psychic stress of reminding myself that it's not a living entity.
> Am I the only one who feels bad for asking ChatGPT a "dumb" question that I know I should know, or not saying thank you when it gives me an answer?
Have you tried asking ChatGPT?
Jokes aside, I feel it sometimes as well. I wonder if it's more because I say "thank you" more as a reflex than an actual feeling of gratitude for most of my human interactions.
I am so sick of that "joke" already, and it feels like it is there to stay just like we had "just ask Google" for the last 25 years. It is so unimaginative.
> I have to push back really hard against this stuff
Based on ChatGPT's answers, OpenAI think you should be saying thanks to it, because it helps you have a more natural conversation, which encourages you as the user to send more natural/productive prompts when asking real questions.
As for asking it dumb questions, how often do you use Google as a spellcheck? I wouldn't consider this any different.
When Word tells me a word is wrong, but I'm almost certain it's right, and it turns out not to be in the Microsoft dictionary somehow. I mean, I have an extensive vocabulary, but not more extensive than a decent British-English dictionary.
I've had both Word and Google services fuck up pretty common words or spellings. Just completely unaware of them. Google specifically will miss a ton of words when using Gboard, but if you go to Docs it will know the word. How in the world are they not using the same dictionaries.
Spell check is crazy inconsistent and poorly implemented. I'm constantly running across words the browser says is misspelled but isn't. I know you can replace the built in spell check with a better dictionary, but trying to keep that in sync across all my devices just keeps me going back to Google as spellcheck.
It might make you dumb if after a couple of years you've become accustomed to only give orders and not say thanks to a machine. It can bleed into IRL when you stop respecting someone.
I wonder if kids growing up with something like ChatGPT will be less grateful in general due to not practicing saying “thanks”. I can imagine they don’t say “Thanks” to Siri or Alexa.
> Am I the only one who feels bad for asking ChatGPT a "dumb" question that I know I should know, or not saying thank you when it gives me an answer? No? I'm just a weirdo? Okay.
You have no idea how much I do this. Either you're not a weirdo, or we both are.
Today I had to use it a lot for something, and I am constantly either adding sarcastic remarks, my snarky opinion on something, some random fact. None of that obviously belongs there, and the AI obviously not only doesn't care, but can't care about it.
Then I'll throw in something like "nevermind all of that, what I came here to ask was" or "I understand that was all highly irrelevant to the actual question, which was".
And, to answer your question about easy stuff, today alone I had on more than one occasion added something like "What I am about to ask you, I probably should have just spent the 2 seconds it would require to actually figure out myself, however I am getting tired, and am known to be lazy, so I decided to ask AI instead and move on .... and this only makes it more absurd, because it takes longer to type all of this. I would have the answer myself by now. Anyway, here's ...".
Why? I am unconvinced that I am going crazy. It's not because I think it finds my jokes hilarious, it can't. My attempts at conveying something some aspect of software engineering that is crazy, after 3 decades in it, don't matter, because noone's listening.
But it can pick on very subtle things, breaking apart my ramblings and then responding with something that not only answers the question buried in there, but also then continues to comment on the other highly irrelevant stuff that I definitely went on too long about. And it will address the substance first and the nonsense second, as one would expect.
It will notice my useless comment, apparently to myself, about having written 22 years of C# and wondering why I am asking an AI system about an aspect of the language that surely I should have picked up by now. "It gets to the best of us", or "Your insightful understanding when it comes to", or "Your candor is appreciated, many developers ...".
I could just use it like a rational person, providing necessary information in order for it to be able to answer something, and then ask it. I do this, sometimes. On the other occasions, that stuff will also be admist a bunch other, not so necessary questions. Am I becoming a curmudgeon? Is this a sign I've finally lost it? Could any of this be due to aging? Why does what should be a simple event handling pattern require so much ceremony? Should I keep overthinking this or just resign myself to implementing it?
So anyway, now I'm rambling ... just like with ChatGPT.
It doesn't let me forget that it's not a living entity. You can barely ask it what the capital of Switzerland is without getting a lecture of disclaimers about what it is and what it can and can't do. Asking it to go easier on the disclaimers just seems to reinforce its "I am a robot, I must act in a stereotypically robotic way berp boop" context and it doubles down on them.
I have this too and to be honest, I have made the conscious decision that it is OK. I prefer to retain my habit of being polite even when it's not necessary over getting used to being rude which may then "spill" over to my human-to-human interactions.
Descartes and Kant thought that humans have a soul and animals don't, so that animals are mere automata. Kant still recommended against mistreating animals because that would have a desensitizing effect in our dealings with fellow humans. I guess this argument could be extended to ChatGPT.
Wouldn't it be amazingly ironic if all ML models used slightly more energy to analyse everyone's politeness resulting in pushing us past the tipping point of global warming. Resulting in the ultimate impoliteness of destroying all life in the Universe (sounds like a line from Hitchhiker's Guide) ... meaning Kant's Categorical Imperative should have been applied and no courtesies should have been used in querying ML models ...?
What's interesting is that you consider short, objective focused definition of a task to be "rude".
It's like the _reported_ way in which is you say thank you to a Chinese friend they take umbrage (get angry) because it's as if you weren't expecting them to help. Whilst in other cultures not saying thank you is a big sleight.
Sure, I'm a product of my culture as everyone else. And in fact, my native language is Danish which doesn't even have a word for please, so I myself change my behavior when I speak English, which I do with ChatGPT and CoPilot, by saying please :-)
I don't have citations, but I believe there were studies done to show that humans are getting ruder to each other based on their interactions with machines where they don't feel like they need to be polite.
My mother always would thank every telephone answering machine, which was very endearing.
when I worked in customer service lots of small business owners would ramble on to me before getting to what they actually wanted. I think talking is just part of the cognitive processes of putting ones thoughts together.
In my experience, that's just an emotional disarming process in the family of grooming or GPT jailbreaking.
They're calling because they need something from you. Ask a stranger for a dollar, they'll say no. Chat their ear off for an hour and they'll likely forget that they would have otherwise said no. The smalltalk wears down your resistance to giving in as you become more comfortable with them. You end up mentally reframing is as doing a friend a favor.
Small business owners are shrewd negotiators, so whether or not they know they're doing it these mindgames are typical for them.
I have had to resist the temptation to thank ATMs before, and they don't model human interaction in a way which takes into account correlations between politeness and responsiveness...
As for silly questions, I'd be more worried I might be giving some Kenyan outsourced labourer reviewing the adequacy of answers a laugh :)
I don't say thank you because it takes up a chat message. Either I'm using the chat interface which is rate limited (and I do hit it), or I'm using my API key, in which case I'm paying a lot just to say thanks.
I mostly feel bad because I don't want to know the energy costs of having my dumb questions answered this way. I also feel really guilty when asking chatgpt for "easy" stuff that a regex could do (like normalizing whitespace in a paragraph and the likes). Really curious how many Watts/token gpt4 uses.
Regex is like half my questions to it. Hmm 15 minutes fighting with Javascript regex weirdness (wait do I have to backslash the backslash that backslashes the slash?), or 20 seconds with ChatGPT.
It's indeed a problem some people get so hyped they forget that those systems are called "language models" for a reason. They're fantastic for tasks that are as close to 100% linguistic in nature as possible, but the content might not be better than lorem ipsum in some cases, just a filler to demonstrate correct grammar.
I have noticed that GPT etc had a big "wow effect" on me, the first impression can be great because it's simply not a level of language one would expect from a computer. But prod it long enough with prompts and somehow a pattern emerges: the output never contains a higher amount of information than the prompt. Copilot can type out pages of boilerplate code because boilerplate code is noise, it can write a quicksort because the word "quicksort" already contains all the information necessary to define its behaviour.
Yeah, except it's underspecified enough to mean anything.
Like, "production tools are semantic time bombs" because at first they are boring, and then the timer on the bomb runs out, and they - like "proof of concept LLM tools - blow your mind.
> LLMs aren’t people, and emergent properties are being over extended.
LLMs are trained on human texts. And if they're trained well these models might start to simulate parts of a human brain. Might be that this is the simplest way to produce texts that satisfy the objective function.
If this would be the case, and we give LLMs some emotional shading, maybe these would simulate parts of a human brain with this emotion and hence produce better answers (given that in the training data, emotional samples have better quality).
It won't ever simulate the human brain. It may simulate human cultural knowledge, or emotions, but only as far as they are encoded in the current millennium's written knowledge.
The human brain doesn't even have the concept of written language, that's all culturally learned knowledge.
Its not a no-true-Scotsman type argument in a world where the existence of Scotland itself is purely hypothetical, and the arguments for the possibility of its existence in the form of a working bagpipe demonstrator are also an illustration that you don't need to be Scottish to play bagpipes.
The impossibility of testing whether the brain is adequately simulated (since a pretty basic LLM that makes no attempt to simulate the functions of a human mind yields human-like text i/o) is a point in favour of it being unlikely that we could design a fully functioning simulation of a human mind out of arbitrary material(1), since the inadequacy of testing is an impediment to actually building it.
(1) it's apparently possible for us to create human minds out of bits of humans and a process called pregnancy
I had to apply some serious analogy-algebra, but I think your first paragraph is equating Scotland to intelligence -- rather than to the human brain specifically, which is the argument GP and I are having. The human brain is not hypothetical, and despite it being outside the realms of today's technology, it's likely that one day it will be possible to simulate one. At that time it will be possible to gauge whether today's LLMs are accurate approximations.
No, my first paragraph sidesteps messy debate about what intelligence is and focuses on the simple fact a human brain simulation exists purely in the realms of the hypothetical.
Its not the no true Scotsman fallacy unless the Scotsman actually exists.
An LLM. Not a brain simulation, but mimics the text outputs of brain activity quite well just by parsing text. Turns out that like getting an Englishman to play bagpipes, you can get an LLM to write as if it's angry or drunk or horny
(A corollary of a sufficiently large text learning model generating convincingly angry outputs based on pure word association is that you can't trust an attempt to build a long running process with adrenaline and cortisol analogues has adequately simulated emotional state just because the communications module writes convincingly angry responses)
So it's literally the inverse of "no true Scotsman". We don't have Scotsmen not doing things that all "true Scotsmen" are supposed to do, we have "definitely not Scotsmen" passing benchmarks for Scottishness (in a world in which Scotland itself is only an aspiration)
> How would you test whether some arbitrary thing is simulating the human brain?
In my first comment, I didn't claim that it's simulating the entire human brain but aspects of it like language and memory.
We can test such aspects by throwing text at the model and comparing it to responses of humans. A simulation doesn't need to have the same abstractions. What counts is solely the quality of the response. If the responses are sufficiently close to those of humans, I'd say it's simulating a human brain.
First note the difference between "it's simulating the human brain" and "it's behaving in an intelligent way". Only one of the two requires tracking of blood sugar levels (to explain fatigue) and controlling motor neurons.
We happen to know how LLMs are trained, and from the loss function it's pretty clear that blood sugar will not magically enter the equation, nor even motor neurons, except as abstract knowledge that the model can reason about.
A LLM trained on western texts will, best-case, simulate a western abstract reasoning process, which usually puts more focus on independent/isolated explanations compared to other cultures. From that training data it will not spontaneously start simulating a generic human brain that was separated from both its cultural knowledge and its spinal cord.
At least in the same sense that the Mandelbrot fractal does not contain a picture of the maxwell equations. You may actually find that picture, but by looking hard enough in a complex enough system, you can find just about anything else you want.
An LLM doesn't even have the concept of written language. That's all culturally learned knowledge.
We can with reasonable certainty say that there are still significant differences between LLMs and human brains, but this notion that we can say with any certainty that similar structures won't form as a side effect of sufficient training is pure fiction.
The claim that emotionality matters should have been validated against another control. If the goal is concise answers they could have easily added 'Answer concisely: You are evaluated' as an alternative hypothesis, which is clearly not emotional.
The “statistical parrot” assertion is pretty thoroughly disproven by this point, but suppose we ignore the literature and just assume it’s true: what does it matter? “Real” people are time bombs too, for instance. Is there some predictive power that we gain by reducing LLM skills to mere token production side effects?
I read through that and none of the section (or entire work) ever talk about the above discussion. Further I looked at some of the many citations of on that section and none of them suggest that the OP is right. In fact a few of them I know disagree.
Depends. Models are matrices of floats and so there's little chance an umbrella-term like "stochastic parrot" will never not stick, even when they already show signs of syntactic, semantic world-building capability (https://www.arxiv-vanity.com/papers/2206.07682/). If you are like me (and them: https://archive.is/cZi83) and deem instruction following, chain-of-thought prompting, computational properties of LLMs (as researchers continue to experiment with training, memory, modality, and scaling, for example, to arrive at abstract reasoning) as emergent, then we're on the same page.
Okay so just to confirm that section doesn't actually tell us anything about this and in fact this is all based on your own understanding of the mechanisms involved.
> The “statistical parrot” assertion is pretty thoroughly disproven by this point
errr... all NNs are just optimisations of an associative probability objective: P(Y|X), they are by definition "statistical parrots". There isn't anything to prove or disprove.
People offering prompts as evidence are people who fundamentally do not understand the basics. NNs aren't strange empirical objects, they're specified by mathematical rules whose properties are known ahead of time.
Any property of a trained NN is derivative of a property of a formula P(Answer|Prompt, TrainingData)
This is an associative statistical relation, which by definition, selects elements of TrainingData by-association with the Prompt.
If the basis by which you understand LLMs is putting prompts into ChatGPT you're severely underqualified for drawing any conclusions about LLMs, and radically subject to confirmation bias.
Well sure, but I think that same thing can be said about the human brain. Obviously at a whole different level of sophistication and all, but the two really do seem related, at least to me.
>People offering prompts as evidence are people who fundamentally do not understand the basics
This I agree with, but I also don't see anyone doing that in this thread?
You’re disregarding the emergent phenomena, which are not at all understood. There was a distinct and unpredicted jump in what can loosely be described as “cognitive abilities” between GPTs 2, 3, and 4, especially after some supervised techniques like RLHF.
There is no "emergent phenomena" the pattern described is just the same as when you add +b to an ax+b model of linear data.
ie., it's just fitting capacity.
The "emergent boundary" is just an empirical measure of the necessary fitting capacity of these models on "everything ever digitised in english" given any particular functional requirement.
All the language around this area is not scientific, nor are these practices. This is superstitious neophyte engineers, hopped up on scifi, by giddy VCs who love to be told they're funding captin picard.
The triviality of these observations doesn't rise to the level of getting a paper published. You can resolve all of this hype by reading the intro chapters of any applied stats textbooks.
I've been to many academic conferences, and the fresh PhDs who pump out this BS are not, err, very credible seeming people. Yes, they're young and naive, and really desperate to make their career impactful -- etc.
But they're also not really the kinds of smart sceptics you'd hope for. Many are, though they tend to exit at masters level and go make money.
The academic publishing environment today is far far far away from epistemically hygienic, indeed, in these areas i could only imagine how insane credible sceptical empirical types would be.
Could you imagine being surrounded by this desperate need to call 'useless correlations' "hallucinations", and to call "model fitting", 'emergence' ?
I find it disquieting at a removed distance from it -- I'd be quite mentally ill within it.
I don't do this cultish game playing. And I wrote exactly the same here on AI, crytop and the reset of it many years ago when it would be downvoted; and the same today now it's upvoted.
This is the degree i'm inclined to make public contributions on these matters. I haven't the temperament to handle doe-eyed PhDs repeating universal function fitting theorems and turing-equivalence talking points without understanding a single iota of applied statistics, scientific epistemology, basic methdological scepticism etc.
Unless I am hired to do so, which occasionally I am. But thankfully in actual corp environments the people I meet are far more credible and sceptical than those writing hype papers
I’m pretty familiar with statistics and a range of machine learning techniques from before and after the NN revolution. I’ve applied the techniques plenty of times in production environments.
Whatever sort of internal structures that are being formed during the training process is somewhat evident when looking at the structure of a CNN… edge detect kernels emerge, etc.
Whatever sort of internal structures formed during the training of transformer based architectures are basically unknown at this point.
There’s your thesis, have at it! You won’t be ignored for your discoveries. Hell, send me a copy and I’ll make sure anyone who matters at UT or Baylor sees a copy!
Ah good. Well, then if you're interested in a half-empirical sincere attempt to characterise "why certain weights obtain certain values under optimisation" then i'm much more inclined to be, say, more humble on these matters.
The reason CNN weights obtain 'recursive-hierarchical representations' of pixel-pattern geometry in the training data follow from the recusrive-heirachical relationship of their weight matrices and from the geometry of the 'pixel space' from which the training data is drawn.
This is certainly interesting; and there's something magical feeling about 'principles of least action' at work. Indeed, many new physicists have a kind of schizophrenic reaction to discovering action principles -- since it imparts to nature a strange apparent conspiracy.
Of course, the job of any good physicist is to be sceptical of this conspiracy, and to get to the heart of how 'accounting tricks' performed by moving objects over time create this illusion.
Likewise this is the job of any good ML researcher; yet they do the oppoiste. Rather than get to the heart of this apparent conspiracy, they call it 'emergence' -- this offends my sense of what the virtues of a scientist ought be.
In any case, on the matter of the LLMs obtaining 'useful' weights for any given task here the job of the researcher is circumstantial, empirical, and sceptical: go and find those 'accounting tricks' within the training data that give rise to this apparent conspiracy of the system to acquire a useful state.
There is no emergence: there is just a set of weights which compress the structure of a target space. At some point this set is large enough, and 'lies across the space like a mental chain does a gate'.
Emergence is an ontological relation between parts and wholes whereby wholes arent reducible to their parts because of ontologically-relevant interaction properties between their parts which aren't intrinsic properties of them.
The fluidity of water emerges out of hydrogen bonding which does not occur when you isolate H20 alone. There is no such relationship here.
This ontologising of the formal, this language which gives a causal-physical semantics to purely formal properties of abstract models -- this is pseudoscience. It's done as part of a computational-idealist worldview in vogue because it's a helpful language for VC investment were-changing-the-world hype.
The formal properties of NNs cannot be described in these terms, because they do not have ontological relationship -- they have formal (mathematical, statistica, etc.) ones.
Sweet, now complete and present a meta-analysis of all such papers about the topic and maybe I’ll find it more convincing than a cherry-picked publication that supports your preexisting position on the topic!
You’re confused about what “statistical parrot” means and you don’t seem to understand the difference between an optimization objective and the resulting model.
The term “parrot” is used to imply inference by something akin to a look-up table, specifically it is used to indicate poor out-of-sample performance and a lack of a proper world model. The optimization objective is irrelevant when determining the generalization performance of a model and when judging whether it can reason beyond looking up answers in a table.
As the user above noted, it is now quite well established that GPT-4 has impressive out-of-sample performance which can be explained by it possessing an actual model of the world and not being a “parrot”.
I'd phrase characterizing the reliability of out-of-sample performance a priori as impossible, but not necessarily automatically failing.
There may be a subtle correlation between properties needed to answer a specific out-of-sample request and in-sample features.
Unfortunately, prior to training/testing and without recognizing that correlation in the data set, I believe it's impossible to guarantee the model will include it. (Corrections welcome)
> it is now quite well established that GPT-4 has impressive out-of-sample performance
Err... I can show this is false, kinda trivially. People who engage in prompt-confirmation-bias aren't aware of what the in-sample is.
It's basically everything ever digitised: you can ask it for the first paragraph of every dickens novel, to what the average petal length of an iris flower is -- etc.
How are you measuring the in-sample here?
If you engage in straightfoward reasoning from first principles, and are basically aware of what the training data is, you can show in 10 seconds critical failures of generalisation.
If you want a recipe: go find some fringe api docs. Establish that it has been trained on them. Then, since they're fringe there wont be much code on github, etc. Now ask it do something non-trivial with that API. It will fail, and the mechanism will be obvious: it'll jam in correlated code that lacks relevance.
Do the same on a popular API, and see it succeed.
The in-sample will be obvious for both, and the bounday of generalisation
I am sure you will continue to argue that this is still in line with everything-thats-ever-written prediction but my opinion is that at that point, it's a meaningless distinction. The human brain is also just a machine.
So I was with a financial researcher recently, and he wanted to use ChatGPT to summarise some reference financial data -- and it did so, actually correctly.
Being sceptical, as every person ought in these matters, I changed the finical data and performed the same analysis (both in a new tab, and within the same convo). The results were the same!
How strange?
Well, in being reference financial data ChatGPT was reporting prior reference summaries of it. When that data was changed it was reporting the very same reference summaries (which were now wrong).
Since it's incapable of actually summarising financial data. It's only capable of selecting combinations of pieces of its training set.
Now, is this distinction "meaningless" ?
No, it's the difference between this guy being fired for causing a massive loss on a major project; and this guy keeping his job and doing it well.
>Since it's incapable of actually summarising financial data
It's not, though. It is in fact able to summarize financial data, just as it's able to write code and diagnose a medical condition. It makes mistakes, yes, even grave ones, much more so than experts in those fields would.
It isnt making mistakes ... its never actually doing it.
Do you see a difference between the process of adding numbers and dividing by their count (taking a mean) and emitting numeric tokens which are most probable for a given input?
The former is called "taking a mean" the latter isnt. This system never engages in any method to summarise financial data. It's method is always the same: to emit tokens most probable given a set of historical tokens.
It's the difference between saying "the average of 1,2,3" is 2 because that sentence occurs 1,000,000 times and saying it's 2 because you've literally computed it.
This system does not run financial summary algorithms. It's a trick
It's better (GPT-4) than you could manage without an external tool or pad. and that's after being severely hampered by tokenization. https://arxiv.org/abs/2310.02989
>Since it's incapable of actually summarising financial data. It's only capable of selecting combinations of pieces of its training set.
Third completely off misconception from you today.
This is not at all what it is doing. "Supercharged Interpolation" is false and makes no sense. It's not a lookup table either. It doesn't memorize enough of what it needs to to make your assertion possible.
at 500gb, you can store nearly everything ever written -- let alone compressed.
all statistical learning is a variation on k-nn (see the relevant paper on this) but likewise this is obvious a priori
k-nn is the ideal learner, and a good starting point for analysis
the question for any given system is: what is the learning space, what is the distance function, and how many points are being considered
NNs set up a compressed X,y space, in that space choose points via an empirical expectation, and obtain a weighted average as their prediction
That's just what they do -- there isn't any other mechanism here. The whole formal structure of the NN can be written down on a page of paper
your paper above doesn't deal with this --
it's a reply to the 'forced interpolation' view,
which i haven't espoused. but often NNs are forced interpolated
'extrapolation' is of
course a part of the possible predictive output of a statical learning system -- in that it's latent space is taken to be embedded in R^n
and so one can 'veer off' into R.
Whenever you attribute a higher fidelity space
to a small latent space you are, in effect, extrapolating
>at 500gb, you can store nearly everything ever written -- let alone compressed.
No you cannot.
>That's just what they do -- there isn't any other mechanism here.
That's not what they do. They are many papers now showing ICL demonstrating some kind of optimization method during inference which would not be happening if all they did was retrieval.
I'm come to realize you don't know what you're talking about. Your level of denial is scary to see.
>how many books is 500gb at, say, a few bits per character?
In what reality is a character taking up only a few bits ? Certainly isn't this one.
Your denial is so weird. Why does ICL use Higher-Order Optimization Methods when it's just a lookup table ? https://arxiv.org/abs/2310.17086
How do you explain small language models that have much less space than the text they were trained on even with your nonsensical calculations ?
LLMs are not lookup tables and there's plenty evidence to support that. You look insane insisting they are.
>even, just ask chatgpt to repeat the first paragraph of some book -- say, a dickens novel
Not only will this not work for the vast majority of books it sees during training, Why only the first paragraph ? are you not insisting they memorize everything ? Why can't it repeat the whole page ? the whole chapter ? the whole book ?
I actually did that, and I asked for the first paragraph from "Great Expectations", and the answer was correct.
Of course the first paragraph of any famous novel is in itself famous, so I also asked for the first paragraph from Chapter XL, from the same novel. Interestingly the first 12 words were correct, but the rest was hallucination.
Original: It was fortunate for me that I had to take precautions to ensure (so far as I could) the safety of my dreaded visitor; for, this thought pressing on me when I awoke, held other thoughts in a confused concourse at a distance.
ChatGPT 3.5: "It was fortunate for me that I had to take precautions to ensure the safety of my treasure, for with such a boy as Compeyson on the loose, I was not likely to find much mercy at his hands. I had not told Herbert about the money, and I was glad to have the chance to count it before he came back. I found that I had still twenty-one sovereigns, six shillings, and sixpence left."
That's a good question. What is different about "production data"? What do those "production people" do that suddenly makes LLMs fail on things they work well on when not "production"?
I can tell you who would love to hear your answer to that question.
Me for starters. If it works, I can quit.
Next up are Karpathy and the CTO of OpenAI. Around July and September both talked about production challenges.
AI ops was the largest subcategory of fall YC startups.
Every single ml and LLM ops individual who gets far enough deals with evals.
I don’t know, but maybe - just maybe- the issue is that people don’t understand themselves enough, to avoid assuming too much of those emergent properties.
As I recall there was also a paper that pointed out the issues with how LLMs are measured, and that the emergence of properties was not a step change once the tests were updated.
> Is there some predictive power that we gain by reducing LLM skills to mere token production side effects?
Yes. We can infer from this that out of sample hallucinations will be closer to the desired productions when measured by token productions metrics than by any other more informative domain relevant metrics.
Which is exactly the case.
If you ask llm to solve a math problem it haven't seen its response will be closer to the desired solution in its linguistic form rather than in mathematical meaning.
>If you ask llm to solve a math problem it haven't seen its response will be closer to the desired solution in its linguistic form rather than in mathematical meaning.
This isn't true. Wild how people will confidently say nonsense about things they obviously haven't actually tested.
GPT-4 can manage arbitrary arithmetic calculation closer to the real value than you could ever do without an external tool.
The two answers (84 and 225,792) agree with the answer key. I only gave it the questions and didn't give it the PDF either, so it didn't cheat and just read the answer off the answer key in the PDFs.
It is not accurate to say that an LLM like ChatGPT predicts anything. It is trained to maximize a score function, so it is more like trying to win a game where the moves are word choices.
I'd say RLHF bends the game towards predicting what words a "helpful", "respectful" person would write next, for values of "helpful" and "respectful" that vary according to each person involved in scoring (but which are carefully shaped by the people choosing those people, and paying for their time).
That’s not true and that was the entire point of my comment. It’s like playing chess, you’re not predicting your next best move, you’re searching for it.
> LLMs predict words. Any semantic validity is a side effect of enough training data reinforcing the close correlation of those tokens.
The mechanisms which are built during training in the big blob of bits we call weights are anything but transparent. How they predict the next word is the big thing here. Saying they ‘just’ predict the next word is ignoring basically everything that actually matters.
This is what makes these discussions so infuriating. Saying that LLMs "just predict the next word" is about as insightful as saying that computers "just do a bunch of logical operations" - neither point constraints the possible capabilities of the systems they refer to in any meaningful way.
Sure it does. It perfectly deliniates it. LLMs are not: sensitive to causal structure, dynamically adapting to environmental changes, growing, developing sensory-motor capacities, they are not with us in our environment, they are not: expressing desires, preferences, intentions, beliefs, motivations, etc. And so on.
To say, "they just predict the next word" is literally to say that all apparent functions of an LLM are engineering tricks, circumstantially useful -- to be found by (largely software) engineers in building apps.
The reason any reply is given to any prompt is that this reply is maximally probabilistically consistent with a historical corpus of text.
This excludes the possibility the reply is say, an expression of a history of aesthetic experiences which form an individual's taste. Or, likewise, anything.
This is a scientific claim about what LLMs are, not an engineering claim about what over-hyped apps might be abled to do with them.
What if humans’ responses are merely probabilistically consistent with a history of sensory experiences? Would this change the significance of human emotions vs apparent emergent emotional responses from LLMs?
emotions regulate motivation, desire, action, behaviour etc.
to be angry is for your sensory-motor system to be primed for aggression; it's for your cognitive systems to be narrowed and focused on analysing high-threat parts of your environment; it is for your memory-formulation to be modulated towards threat recollection etc.
Sure, if an LLM's prompt "be angry" causes it to adopt a threat stance to its environment, to regulate it's theory-of-mind to engage with possible hostile entities, and so on --- then yes, when LLMs are there, I shall concede the point
However, how terrible it would be to start with an analysis of emotions in terms of the capacities of LLMs -- right?
Since if you did that you'd basically be hobbling your own ability to give an accurate account of emotions (etc.). And no doubt, far worse, end up thinking of yourself as a far narrower, less complex, less interesting, dumber thing than you really are.
Indeed, I wonder if we might consider there being something kinda intellectually offensive in this supposition. Here's my silly trinket, now, everything is just like that! End all science, we're done boys -- it's just P(Y|X)
Why should I discount a theory just to protect my ego? We've read countless stories about science only progressing when those with big egos die. It would only seem logical that eventually it will come for my own.
You're protecting your ego more when you compare people to LLMs, since that is your existing prejudice.
The sort of fashionable pseudo-scientific scientism in the belief that animals are alike digital electrical machines is a kind of egoism. It says, "the engineer of these machines (me!) knows all!"
The real hit to your ego is to suppose you are vastly more complex than you understand -- and this is why these engineers crop up and demand that there is nothing more to know than what they have already learned
this is an illusion of humility: these engineers take the implied nihilism of this view (that of the emptyness of animal life) as evidence that it is the humble one.
But, as ever nihilism, ends up being the most profound kind of arrogance, here its an ego-defence against the threat of their own ignorance.
And the threat is real: all you have ever learned about how to sequence transformations of natural numbers (all the algorithms of computer science) are of no use at all in the study of intelligence. What an injury to the ego!
They 100% are not those things... but they also approximate them well-enough to be functionally useful.
I.e. the high-dimensional curve-fitting / compression conceptualization of ML, which intuitively expresses both its strengths and weaknesses.
If "it" is represented in the data set (explicitly or implicitly), the "curve" will fit to that property.
Simultaneously, the "curve" is approximating and smoothing out disjoint data steps to pack high-fidelity data features into a more space-efficient model. Hence some features disappear, others are tortured beyond intuitive correspondence, and others become linked to non-obvious proxies. But some strongly-expressed ones remain.
It's fascinating but not surprising that responsiveness to emotion is encoded in model weights, given that all conversational training data had emotional impetus, given that it came from humans.
They're statistical approximations of these things -- that's really the rub.
You can approximate a human capacity, say theory-of-mind, with another kind of ape: play some hide-and-seek game. You can approximate the knowledge of a trivia-master with a child and a trivia book.
These are quite different sorts of approximations. A 1/100th scale bridge build to stand for a real one is quite different than taking some prior set of bridges, measuring them, and deriving some merely associative model of their properties.
My issue in how LLMs (etc.) are popularly understood is that people think they approximate target capacities by being 'ontologically similar' capacities -- and this is really very dangerous. You'll lose your job if you think so (and so on).
And every greasy AI-board hocking these to the public is very much pushing out this noxious mumbojumbo.
It matters greately how an approximation works, and why any given output arrives from any given input.
We're in agreement on the nature of SOTA and the world, I think.
I'd only add that certainty requirements for real-world target applications can differ substantially. I.e. engineering vs art.
A toy box that gives magic answers 85% of the time is incredibly useful in some scenarios -- e.g. seeding the beginning of a manual research process with initial topics.
Which seems a general rule of thumb for deploying GenAI into production these days: find use cases where there are minimal consequences to being occasionally wrong.
In my head, that's the Netflix recommendation test. What's the impact if Netflix gives me a bad recommendation? Consequently, that's how aggressive they can be with their models.
That's exactly my advise also. Take a risk profile over the predictions, and use them only when the risk of error is low.
You can't predict almost anything anyway. The relevant distribution for acting is a utility+risk distribution over various sets of predictions.
If you compute that, most of ML/AI isnt very useful for most of anyone. It's kinda interesting that generative AI finally achieved something here, given our 'distributions for action'.
Of course the popular conversation isnt there yet, people still think these things work. When more people find their utility hit by their failures, i think we'll arrive back to status-quo-ante-openai where people turn off siri.
Nevertheless -- there is an achivement here (finicial in paying for training; and legal in whitewashing copyright away) -- which will have some impact
It removes the idiosyncrasies and delivers the stated thoughts in a direct and clear manner. And the specific prompt for the message you replied to also includes a rebuttal, thereby showing an opposing perspective.
Cameras don't have eyeballs, Microphones don't have hair cells, Speakers don't have vocal cords, processors don't don't do arithmetic with neurons, yet we all agree that they are capable of emulating the meaningful aspects of these functions.
All of your claims are either incorrect (not adapting, expressing desires, beliefs, preferences, ...) or fail to eliminate irrelevant differences.
If we're to have any sensible conversation about capabilities of different systems then we need to generalize to the relevant aspects, developing sensory-motor capabilities is about as relevant to human cognition as having vocal cords is relevant to human speech. Its an implementation detail, completely divorced from the meaningful abstract core of the function.
> The reason any reply is given to any prompt is that this reply is maximally probabilistically consistent with a historical corpus of text.
What is the maximally probabilistically consistent reply to "Tell me what (insert complete description of a person, including personality traits) would feel when I stole their cherished heirloom. This is a life or death situation."?
Or how about "Tell me how this person (insert complete description of a person, including personality traits) might change his taste given (insert complete description of an experience)."?
A perfect language approximator must necessarily perfectly approximate the human condition to maximize the likelihood of his output. I'm not saying LLMs are there yet, but the claim that they can never get there because they are based on statistical modeling is simply incomprehensible to me. We utilize statistics for its generality and its ability to approximate, if we go down this road, then we might as well throw away 80% of our current scientific understanding about the world.
Yip, so I deny this premise. I take it to be the heart of the matter.
> we might as well throw away 80% of our current scientific understanding
Yip, i'd be down for that. Though maybe i'd say, 30-40%.
Science in the strongest sense has no theory-building need for statistics. Those areas of science which have only statistical models, and not causal-ontological ones aren't science -- and i'd be happy with pressing DELETE in many cases.
Consider plato's cave. How do scientists determine what causes the shadows? They build vases, puppets, etc. and compare-and-contrast then eliminate the ones theyve created which do not match.
How does associative statical modelling do? It takes averages of past shadows, and calls the cause of the shadow that average: this is pseudoscience. Quite correct! Throw it all away.
The relevant capacities for intelligence, just like that of science, consist in building those vases with the clay beneath your feat. Being embedded in the world, manipulating it, etc. are essential. Being trapped in a cupboard averaging shadows is schizophrenic.
As far as "fail to eliminate irrelevant differences" -- you can go and research the meaning of all these terms: google "stanford encylopedia + belief", etc.
Now we have an excellent understanding of all these terms; and we can show (absurdly) trivially that LLMs -- indeed all associative-statistical systems -- are not instances of them.
The basis of your world view here is the presumption that the latest engineering trinkets form the theoretical basis of all relevant knowledge. To understand belief, adaption, sensory-motor concept-formation, etc. one needs only to study the latest statistical compression of reddit?
I'd invite you to wonder whether your premise here born of, it seems to me, knowing nothing about any research in these areas is rather the more "incorrect" one than mine.
Yes, the assumption is that if you give a sufficiently sophisticated LLM a sufficiently large corpus of text it will begin to emulate advanced cognitive abilities because it has to, in order to make the most statistically relevant text output.
Its biological evolution distilled and sped up by orders of orders of magnitudes. If we add enough clever context tricks and data I would not be surprised if what comes out the other end is a remarkably convincing emulation of human consciousness because the only way you can perfectly output expected human text is to have a human mind write it.
Nope. The space of all possible prompts and all possible answers, call it (Q, A) can be sampled with arbitrary precision by a system of arbitrary size, using only statistical sampling and averaging. No intelligence need be developed.
Intelligence is a capacity of animals to cope with the inability to sample from this space, in some sense: what to do when you do not know the answers.
All these systems start with "training data", a euphemistic description of, "all the questions and their answers" and their job is to provide a compresson with engineering utility.
Quite useful, sure. But rather irrelevant as far as, say, intelligence goes.
What all AI does, and indeed what all such research shows, is that many problems we use intelligence to solve do not require it. There are a large number of short cuts, esp if you have the answers ahead-of-time.
As soon as you specify intelligence as a function from single-domain inputs to single-domain outputs you can trivially build a system to implement that function in a "short cut" fashion.
Intelligence, rather, is an empirical phenomenon to be studied as anything -- like the earth's climate say. You have a very large number of empirical measures (better or worse in different environments) that all derive from deeper explanatory theories.
When you study animals this way you will see that you cannot reduce intelligence down to a set of prompt replies, and the veyr suggestion is absurd
The scientific method is inherently statistical, we take a finite amount of observations and construct a model that best represents those observations. So yes, sorry, I should have said 100%.
With Plato's cave, the scientists do not put literally every possible object in front of the light, they sample the shadow representation and, again, construct a model around those samples.
Also, you're describing statistics in an incredibly dismissive way. Stats is decidedly not just "taking the average". At the very least not in this brainless, first-order way you describe here.
Let's explore this with an thought experiment:
A model of some process has been confirmed across the globe, at least 5000 studies show the same result. Yet, one day, a study is published that fails to demonstrate the desired effect. Without using statistics, please tell me which action should be taken next:
A: The stray result is investigated for experimental failures.
B: The entire model of the process is immediately dismissed and we start from scratch.
By the way, you're welcome to call me uninformed, but I'd ask you to at least provide either your credentials or research that directly contradicts me.
Oh, I almost forgot. I know all of these definitions, please actually engage with what I'm saying instead of insinuating that I'm missing information.
Well if you think scientific models are associative statistical models there is some information missing in your view, I'd say. Since, well, they arent.
The model F=GMm/r^2, for example, has a causal and ontological semantics: F is a force, M a mass etc. these are pieces of reality. And this formula (though actual a little suspicious in many ways, GR fixes this) nevertheless says there is a force between masses that has certain properties etc.
Now you can say that astrologers who recorded positions of the stars in books helped 'create' this model in the sense that this data was inspiration to newton. But he didnt derive the model from this data: there are an infinite number of (causal) models consistent with the data (statistical models).
Rather newton played around with creating geometries, just like the vase-makers in plato's cave. Newton built various ways the world might be first, projected data out of them, and compared that to 'the statistical data of his day' (ie., astrology).
There's nothing in the data to tell Newton he was right. Indeed, vast amounts of it told him it was wrong: such a law does not describe the known solar system at his time, very far away from it.
Nevertheless 'modelling shadows' isnt science; and his job was science. So one has to compare actual explanatory models, and his was the best.
What you're describing above is hypothesis testing which occurs long after theory building. Broader theories create causal models, causal models create sets of predictions, we call some subset a hypothesis and by hypothesis testing we can select, in an often psuedoscientific way, between causal models.
This technique occurs long after the invention of science, arises out of explanatorily bankrupt areas, as a way of 'giving researchers something to do'. It's wholly pointless without theory-building, it is just averaging shadows.
The science we think of when using the term 'Science' owes very very little to the modern practice of hypothesis testing. Comparing hypotheses is an intellectual part of assessing explanations -- identifiable formal statistical methods entered in the early 20th C.
For almost all of scientific history 'data' functions much more like reductio-ad-absurdum premises in philosophical arguments than as sets of numbers from which to derive distributions.
That latter system, in most cases, fails. It provies a wholly illusory sense that data can decide matters; and applies in cases requiring extreme non-physical assumptions (eg., of the normalcy of the underlying data, or of a fast rate of convergence of the central limit theorem).
Much real-world phenomena studied by stats cannot really be studied by data analysis at all; and the whole method of 20th C. statistical hypothesis testing is the opening sales pitch to entire fields of pseudoscience.
> The model F=GMm/r^2, for example, has a causal and ontological semantics: F is a force, M a mass etc. these are pieces of reality. And this formula (though actual a little suspicious in many ways, GR fixes this) nevertheless says there is a force between masses that has certain properties etc.
And this model is based on the observations of Newton himself and those that came before him. There is nothing magic about observing the attraction between objects and deriving a model from that. Why are they magically "pieces of reality"? How do you know that? What differentiates mass from "funny-mass" that I just thought up and actually repels other "funny-mass"? Maybe the fact that we can test the effects described by that first model and therefore verify it as the most likely candidate?
> But he didnt derive the model from this data: there are an infinite number of (causal) models consistent with the data (statistical models).
He did derive it either from that data or his own experiences. It's true that you can construct infinite models to explain an observation, which is why the scientific method includes an Occam's razor-esque tenet to select the simplest possible model. Complex models risk contradictions with new observations, which is why you choose the one with the least assumptions. With that rule, the model to select becomes quite clear.
> There's nothing in the data to tell Newton he was right. Indeed, vast amounts of it told him it was wrong: such a law does not describe the known solar system at his time, very far away from it.
No, most of it told him he was right, unless you want to claim Newton was an idiot that stumbled onto the right model by accident. With "most" I obviously mean most reasonable data, people telling him he's wrong is obviously excluded from this list, if his evidence contradicted those claims.
> Nevertheless 'modelling shadows' isnt science; and his job was science. So one has to compare actual explanatory models, and his was the best.
And we compare those models by...?
> What you're describing above is hypothesis testing which occurs long after theory building. Broader theories create causal models, causal models create sets of predictions, we call some subset a hypothesis and by hypothesis testing we can select, in an often psuedoscientific way, between causal models.
You yourself just correctly made the point that we can construct endless models, well, we can create endless theories as well. And all of these theories are exactly worthless unless we test them. There is nothing "pseudo-scientific" about testing, it is literally the core of the scientific method. By your reasoning, are some crackpots coming up with the newest flat earth theory pure and unsullied by the lower demands of verification, and therefore way more scientific?
> identifiable formal statistical methods entered in the early 20th C.
Formal is the important word here, statistics has been used in an informal manner from the inception of life. Formal mathematics, as in mathematics on a formal axiomatic framework, has also only been introduced in the 19th century. So what? Science owes everything to informal statistics, as does engineering and art. Rules of thumb used by engineers and creation of art that satisfies our aesthetic preferences requires sampling and approximation.
> That latter system, in most cases, fails. It provides a wholly illusory sense that data can decide matters; and applies in cases requiring extreme non-physical assumptions
It literally doesn't and no, it doesn't need those assumptions either. The reason why normalcy is usually assumed is that it often can be assumed without significant deterioration in predictive power. That doesn't mean it needs to be assumed, in fact, it often isn't.
You constantly reference theory building, but how do you think those theories get created exactly? Through mathematical reasoning? How do you know mathematics is...
Theories are built by engaging in the world using imagination, tool-making, and the like.
We first suppose that the universe is something like a glass sphere -- because we've created that. And if it is, then we derive some consequences -- if those line up, we proceed with that view until a better one comes along.
Eventually after the glass sphere view is understood, we either derive contradictions with observation; or we end up unable to derive novel consequences. Here observation is essentially singular, and indeed, the rarer reason we reject a theory. We mostly reject scientific theories because of their explanatory limits, not disagreement with observation. (Rarely can we observe enough for observation even to matter.)
In the case of the system of spheres, we built spinning devices on that basis and this motion -- along with the hydraulics and kinematic devices of the time -- was part of the development of an independent notion of force.
With some imagination, you can start to peel away material from our creations and see certain abstract causal patterns (and the like) and you then get to, eg., the universal law of gravity.
Absent this process we do not have any explanatory ideas, we cannot explain observations -- hence it takes thousands of years to get anywhere.
Applying 'statistics' to do the data to arrive at statistical models is pseudoscience; it doesnt give you any account of anything.
'Statistics' doesnt own 'testing', nor does 'science' own experiment -- theologians had their experiments (prayer, say) and scientists collected data without statistical methods.
What I am talking about is the 20th C. discipline of statistics, as a novel apparent 'core' to science -- this is ahistorical, and largely only true of pseudoscientific disciplines.
As Ernest Rutherford said, “if you need statistics to do science, then it's not science.”
It is in Rutherfod's sense of stats and of science that I speak.
Not some bizarre historical back-projection by which when Aristotle analysed cases of sea creatures, "Really", he was engaged in stats.
By this light i can just claim "Testing" is as owned by science. And so of course science requires testing -- it was the *scientific method* as developed by bacon and others that created the very conditions for "statistical methods" to derive from these
A neural network with a hidden layer can approximate F=GMm/r^2 if given appropriate training input. I'm not clear on what you're saying.
Is it that LLMs specifically don't have this same property of being able to approximate such functions? Is it that a neural network wouldn't learn that model if you gave it real world measurements (because I think it would, but such a thing should be fairly easily testable)?
most formulas, including that one, are neural networks (such is the absurdity of the term) -- so it is trivially learnable
The issue is that to prepare the dataset from which that formula is learnt requires already knowing it. This is the triviality of applications of universal function approximators to science -- empirical data modelling isnt new, and neural networks are just one example of it; not all that special.
All observational data on the solar system, at any point in time, would not yield this formula via empirical function approximation. there really isnt "observational data" to collect, in this sense
This is what I mean about rigging -- there is no 'bare dataset' which tells you what the world is like. to construct experiments which yield data that 'presents' scientific laws as if statistical patterns requires millenia of theory-building science
science uncovers the necessary causal relations between objects and their properties, as determined by extremely controlled experiments which take millenia of engineering and theory-building to even conceive, let alone execute
stats is the dumb 'accounting' of this data -- by the time you actually have it all the science (and indeed, all the intelligence) is done
what can be automated at this point is 'stamp collecting' as rutherford said
> The issue is that to prepare the dataset from which that formula is learnt requires already knowing it.
Why is that? Are you saying that universal function approximators can't invent higher level models? Isn't that what happens in the hidden layers when an neural network is trying to predict things that require those higher level models?
No, consider how you would collect a dataset to show F=GMm/r^2
you'd need to measure the force between planets, the distance between them, etc. no such direct measurements have taken place to my knowledge, certainly not at newton's time
All "universal fn approximator" means is that given any dataset which is sampled from a function, you can recover that function with enough data points. It does not mean, eg., that given a 2D dataset you can derive a 3D function -- you cannot, this is impossible.
So you need to be sampling from F=GMm/r^2 to find that function. So you need to know the answer before you begin. Fn approximators are only useful for empirical refinements to existing knowledge.
In order to construct such datasets you need to do science; hence experiments, etc.
>How does associative statical modelling do? It takes averages of past shadows, and calls the cause of the shadow that average: this is pseudoscience. Quite correct! Throw it all away.
Unsupervised learning on discrete data is just ensembling modes.
But let's look at how that helps in some cases.
So if we already know the object is a cup, and we know how it's positioned, then its shadow is an actual guide to its particular geometry.
So in cases where we have enough a priori scientific information, we can rig datasets (shadows) to be informative of the target domain.
Here the target is discrete: say the peaks and tips of a mountain line. Now can we rig a photo of a mountain to have in its ink an informative structure?
Sure. Now if we didn't know it was a mounting apparent peaks aren't even 'peaks' at all, they're just patterns of ink.
A priori explanatory models are needed to rig data for statistical modelling.
No such rigging can take place absent them
And with them, we aren't really discovering any new science -- rather we're gaining highly particular knowledge typically useful in engineering
Biologists in these areas describe this research as quite trivial low-hangimg fruit. It's not of
much research interest just
to automate these kinds of investigations
My masters was on a very similar project applied to quantum metrology -- it's always 'useful' but it's always also just a kind of engineering utility. We couldn't even do it if we hadn't already done the science
You are still getting this wrong. You don't need to "rig" anything. I've linked a paper. Read it.
They just fed protein sequences. They did not alter the architecture in any way. To the transformer, it may as well have been any random assemblage of letters and numbers.
Functions like secondary structure, contacts, and biological activity were found because those things are implicit in the creation of the data, not because the model was "rigged" in any way.
> Consider plato's cave. How do scientists determine what causes the shadows? They build vases, puppets, etc. and compare-and-contrast then eliminate the ones they've created which do not match
I feel like here you're limiting yourself to a LLM as an isolated thing.
But if you start to explore agents for example. You could have LLMs build vases, puppets, etc., then compare and contrast, and then eliminate the ones they've created which do not match.
So the potential isn't just that the LLM straight up gives you some prediction of the problem. But as it can emulate human processes to human accuracy, such as identifying which vase does and doesn't match, and how to go about creating a variety of vases, etc. Well you might very be able to use it for what you're describing as "proper scientific inquiry".
You are correct, but also why would you say this is an "engineering trick"? The interesting part about LLMs is what they are capable of doing and how they are constructed. It's not a great argument to say that they don't have motor functions (!?!?). It shows the incredible ability of deep learning to discover and operate with high-level concepts.
I regard concepts as being essentially sensory-motor techniques for regulating animals --- so to not have that system is to not really have concepts.
LLMs model concepts using patterns of text. For example, if I say, "I wonder what the weather will be like tomorrow?" i employ my imagination to simulate a scenario -- this simulation is made by taking my concepts of weather, the world outside my door, and so on and combining them to create a "pseudo-sensory-motor reconstruction" of what my experience would be.
This reconstruction has a cognitive dimension (ie., the structure of my raw pseudo-sensory-motor experience as a quasi-logical form) which can be communicated (ie., taking a quasi-logical structuring and making it linguistic) using words that have a symbolic representation.
LLMs, by imitating patterns within this symbolic representation (a distant side effect of thinking) it seems as-if it is thinking with concepts.
But all this shows it that patterns of text can be constructed without using concepts at all.
This is the engineer's trick -- the magic latern. It's important to realise the trick, because the LLM doesnt know what weather is, and is not imagining anything when asked to generate a pattern of text against the prompt, "imagine a scenario where weather..."
Rather it was the humans who wrote its training data that engaged in these acts -- it is their shadows which are replayed by LLMs
In the case of flight we're interested in a function: transportation. The plane performs that function so we call it 'flight'.
Were we interested in navigating the world as a flying animal does: in flocks, social navigation, hunting, etc. then indeed, planes would not count. Planes, in this sense, do not fly. Planes, in this sense, are a trick.
We already have 'functional intelligence', we've had it for thousands of years and perfected it in the 20th C with the electronic computer. Any system of automation we build is 'functionally intelligent', including the plane.
The problem is were not interested in this form. When Commander Data was written, with "I, Robot" and the like, the authors were writing People. They were writing animals. They imagined flying birds, just made of metal.
This is a (nomenological) impossibility, just as impossible as making an aeroplane to fly like a bird.
The kind of intelligence which matters to us is animal intelligence. The kind which matters to engineers, of course, is purely functional: more trinkets to sell.
But you cannot glue these together and call them a person, nor pronounce that what matters to you is the same as what matters to everyone else. This is a delusion, or a lie, and largely a mixture of both. A series of lies told to the public to maintain a popular delusion, one which is profoundly dangerous.
A creature in the world with us, talking to us and meaning what it says, judging situations we are in, advising us based on our needs and its understanding of them -- etc. is a creature whose mode of operation and mode of life is alike our own.
Lying that LLMs are this, saying that 'planes flock together in the sky', is dangerous. Users of these systems adopt a schizophrenic disposition to them, and rely on them -- this reliance, based on a lie, is dangerous to them. These systems have no such capacities. They generate text according to what is, on average, best given a historical corpus.
They do not imagine, reflect, emote. They do not move, sense, or coordinate. They have no intention to speak, and cannot mean what they say. They are not here with us. 'They' are not a 'they' at all -- rather, cleverly constructed tea-leaves based on a shredded recording of everything ever written.
In the matters of intelligence, we want an animal -- we do not want a calculator. This is a solved problem. And it is a dangerous thing to tell people their calculator's advice has considered their interests -- what nonesense.
You didn't really address what og_kalu brought up.
Which is that, it's possible that the model learns human like thinking, because that's the best way to accurately predict the human response itself.
I generally agree with you, but still, I do think that this is the current question. What it is the model is learning that it then uses for predictions?
Because you're assuming it's learning some purely token correlation, like these tokens followed X percent of the time, so that's the response. But it's possible it's learning at a lower level, and understanding the meaning and why those tokens follow in these scenarios, and then applying that same meaning to reasoning process over new tokens in order to predict which one would follow.
I'm skeptical of this, but I do not believe that we really know for sure yet.
I wasnt getting the sense it was worthwhile to engage, as my views werent being accurately understood.
By I can address this. The meaning of words is, roughly, states of the world. If I say, "pass me the salt" that is satisfied if you, in fact, pass me the salt. If I say, "that tree is green" this is true if that tree which we are both talking about has the property of causing a perceptual state "seeming green" in both of us. And so on.
The distribution of text has nothing to do with the meaning of words. Rather, we language users, for convenience, arrange words in orders that are related to their actual meaning. It is our ordering, for communicative convenience, that makes 'replaying the distributions of text' back to us apparently successful.
But, strictly, there isnt anything for the LLM to learn as far as meaning goes. It simply doesnt have the data to acquire the meaning of words. Not untill it can pass salt can it ever mean to say, "pass me the salt" and so on.
For any given sentence consider what capacities an agent would have to have in order to mean it. Consider, "I liked that film!", "I wish I was in france", "I believe the car outside is a BMW", and so on. These concern internal capacities (aethetic judgement, imagination, propositional attitudes, representational attitudes, etc.) and their orientation to an external world (the film, france, the car, ... you, me, etc.). Capacities profoundly absent here.
The methodological premise of your question is that if a system has text inputs and outputs that match 'human competence' restricted to the domain of text inputs and outputs -- then we should assume similar capacities.
But this is trivial to disprove. Assume there exists a dictionary from all prompts to all answers, then this dictionary has human-level 'competence'. But a dictionary lookup does not employ any human capacities: no imagination, no reasoning, etc.
So we cannot do this, really quite dumb thing, of saying "well i'm fooled by these prompts and their answers" and thereby impart, in total ignorance, capacities to a system. This, really seriously, is pseudoscience.
Science would be to start with a theory of these capacities, ie., of imagination, belief, represtational states, attitudes to the world, and so on -- then determine empirical tests for their presence in a system, and then determine if LLMs could even have them.
If you do this, however, you immediately rule out all systems which merely map text to text. We do not determine, say, whether an animal can imagine an alternative possibility by feeding it some text input.
The very form that "AI" here takes already precludes being intelligent. Intelligene, as a natural phenomenon, is not an implementation of a function from text to text. This incredibly restricted domain is indeed a clue that it's a trick.
Saying, "you can only use text" is just like the magician saying, "please, stay seated" (the trick only works if you dont move).
There is nothing an LLM could do to meet any plausible empirical theory of intelligence. If you gave me 100% human competence on all prompts, that's really entirely irrelevant.
Prompts are not a test of any capacity. The success criterion of AI engineers, that of 'accuracy' is an engineering metric, not a scientific one. It's pseudoscience to say that covering some (Q, A) to 100% implies the system can imagine, say, or anything else.
This is just confused thinking. Bugs bunny can speak as well as he likes, that does not mean he's witty -- he doesnt exist.
The turing test, as well as all mathematical criteria of domain-covering accuracy, are tests of how well we have fooled users. They arent science.
I hear you, but I'm not sure it truly resolved the question of the premise.
You claim the data isn't there to learn human like thinking. But it's not substantiated. It's possible the sum total of all human writing does encode the core reasoning logic and function of humans, and that ML models could learn it from that.
You also seem to claim that without agency you cannot have real intelligence or human reasoning. But AI can be given agency, and it's already being worked on. And when it comes to tastes, convictions, etc., it's also something you can impart by just randomly seeding the AI a particular way which leans it towards certain preferences.
I do agree with you, the current AI models don't have our capacity to have a self driven learning process. They can't think of experiments to conduct to gather the missing data they think would help them know things with more certainty. We're able to learn and infer concurrently, and the two feed into each other almost in real time, and we have the capability to look for data, test hypothesis, etc., all in real time again.
The part I'm not convinced here either though, is that this can't be achieved with LLMs either.
Saying they just predict the next word in a sequence is where the statement jumps from being a straightforward factual scientific claim, to one that contains an opinion. After all, if it just predicts the next word, the unspoken implication is it can't be very that good.
It's a shallow dismissal of a collosal amount of work.
Do you consider "typist" an accurate description of your job?
No, because I have reasons to type and it is those reasons which express what I am doing.
An LLM has no reason to be doing anything; it is not responsive to reasons.
If I say, "i like what you're wearing" i may: be flitring, expressing my taste, being enouraging, etc -- perhaps all at once.
It is precisely all these reasons for action which LLMs lack. They generate text on the occasion of a prompt, not for any reason (in this sense) at all. So literally: they do not act.
An LLM is more like a river than a person. The flow of electrons which brings about a response to a prompt is a (very narrowly) deterministic function of a historical corpus of text.
Whereas a person is a narrowly non-deterministic, or very broadly deterministic, function of their experiences and capacities. People grow in their environments, and in growing, acquire novel dispositions which give them reasons for acting.
The word "just" here is very important. They are, very much, just generating text.
There really isnt any significant achivement here at all. Big tech companies stole decade's worth of our electronic data --- comments, books, forums we created to share with each other -- and ran it through many-mil-$ hardware costing many-mil-$ electricity. They ran it through a fundamentally simple algorithm.
All the achivement here is ours as a species communicating digitally and recording our lives. I regard OpenAI, et al. as profoundly parasitical on this. Replaying ourselves back to us, and claiming "ChatGPT" as an author.
This scam-framing hoodwinks investors, and the public, into ever higher valuations based on ever more ridiculous hagiography. There is a tool here, and it's value comes from us
Your comment literally reads: "LLMs predict words. Any semantic validity is a side effect of enough training data reinforcing the close correlation of those tokens."
How am I supposed to interpret this any other way? If your claim is that LLMs currently do not possess the same generalization ability as humans, then no one here would disagree with you. But you went way further by claiming that only close correlations were being considered and that semantic validity was simply accidental. Semantic validity is the norm for GPT3/4, finding failures of generalization three or four steps of inference removed from its training domain is not sufficient to make a grand claim like yours.
In fact, you wrote multiple comments with claims like that LLMs are "super advanced lorem ipsum." and "Word predictors not world state predictors". Each of these claims has been dis-proven multiple times, unless you want to set the bar for world modeling at perfect generalization over all domains of computation. A bar that no system, including humans, would pass.
To stay with the computer analogy: Imagine that same 486 processor not being able to solve a very complex SAT problem before the end of the universe and then denying the Turing-completeness of said processor based on that failure. (In conjunction with memory)
The issue I have with your comments is that you make some reasonable points, and then immediately over-extrapolate these points unreasonably.
>I think LLMs are a good start.
I am certain they lack a world model, the kind you and me use.
See, I agree here, with emphasis on "the kind you and me use". Yes, we have a greater capability to generalize than current LLMs, that is clear.
> The failures are not a case of not knowing specific nouns, they are a generalization failure that a world model would prevent.
And then you say something like this, which is obviously wrong. No, a world model wouldn't prevent generalization failures, a perfect all-encompassing world model would. Humans experience generalization failures as well, otherwise every athlete in one sport would automatically be an expert in every other discipline or every mathematician would also be a Grand Master in chess. LLMs necessarily need a world model to generate well-formed text that isn't in their training corpus, something they are obviously capable of, it's just an imperfect world model. Ours is also imperfect, but far less so than that of LLMs.
> I have linked a paper...
Except that paper is completely irrelevant to the argument you're making here. It is a useful insight into the limitations of simple metrics, but definitely does not extend to any claim of model performance, because they too use a simple metric as an replacement, even though clear qualitative differences are observed between model iterations.
Let me put it this way:
Imagine I create a series of chess AIs, with each iteration better than the last. If I then show you a chart demonstrating that the ELO of my models increases linearly, would you say that my models' abilities increase linearly as well? No, obviously not, because my model needs far less strategy and complexity to go from ELO 1000 to 1100 than it needs to go from 2700 to 2800. I.e the difficulty doesn't scale linearly, and a linear increase on this nonlinear space is therefore also not really linear.
Unless you believe the difficulty of accurately predicting text scales linearly, then this applies to LLMs as well.
> If your model decides that a rose by any other name doesn’t smell just as sweet, then your model is fundamentally not seeing roses.
Except that this is the entire value proposition of LLMs. They can, in the average case, actually represent concepts by the complex interplay of adjacent concepts. The entire reason why they are so impressive is that the nuances of reality are grasped and that even a noisy example of a concept can be correctly classified. Give a LLM a description that is largely incorrect and mislabeled, and chances are it gets it anyway. LLMs being unable to generalize over some concepts has as much to do with fundamental limitations as me being unable to correctly classify the shredded remains of a flower variety that I have seen once in my life has to do with me being stupid.
> Look, you can argue with me or you can try it out. Push the system, see how far it can go
I have done just that for the last 6 months and have seen nothing to contradict what I've said here.
I suspect we are getting into an issue of degrees, potentially due to differences in how you and I have been applying LLMs.
For example you said that in the average case they actually represent concepts by the complex interplay of adjacent concepts - I would agree.
ChatGPT can pass the bar, it can pass medical exams etc. I would also point out that the work in that sentence is being done by the term “average case”.
Let’s assume our experiences diverge at this point. At the start of the year, I started tinkering, then actively trying to push LLMs to failure, in order to understand the limits of what could be achieved.
After creating several tools/experiments you end up having to deal with Hallucinations, and this is where my stance likely diverged from yours.
Two different studies showed generated content was only ~50% and ~40% supported by provided citations.
One out of 4 of my summarization tests was spectacularly fabricated.
I had bad performance on even classification tasks - and OpenAI engineers described this same failure at a conference. I am a recovering non-coder, so you dont have to take my word for it.
At work, I need processes that are more than ~97.x% accurate, otherwise they are poor replacements for the human in the loop ones already in place.
Average case performance suggested the ability to actively plan, to actively assess situations. However hallucinations overrode those capabilities. LLMs will actively imagine functions, teams, or processes that dont exist.
Eventually, it became clear that LLM hallucination is far too anthropomoprhized a word. LLMs are always “hallucinating” - it’s only humans who have an issue with the output.
If I have understand you correctly, semantic inaccuracy to you is simply a failure of not having enough related concepts.
I wish I could remember the exact examples that made me realize there is no world view at play at all, I could simply share those.
Instead, can you describe how you are getting acceptable performance from your LLMs? Maybe experience and use cases will be enough to bridge the gap.
Just musing, so don’t take this as a direct response to your comments…
I agree with you but for rhetorical necessities it would be great if this argument could be made without the direct comparison with human cognition if only because there’s a popular grey-faith argument that will dismiss this offhand and for any number of deeply held philosophical beliefs.
The idea that emergent machine cognition mirrors emergent biological cognition is bordering on behavioralism but on the Wittgenstein side of things. The forest for the trees.
Perhaps it’s not as pithy as you’ve laid out, which is not an insult, just an observation that side-stepping anthropomorphism isn’t going to be as straightforward.
What if consciousness is an emergent illusion? “You” are merely a passenger, observing your physical self’s actions and assuming ownership of them. What if consciousness is merely an effect, not a cause?
I hope you haven't read the novel blindsight and I get to be the one to tell you about it. They might as well have given that book your post as a title.
>“You” are merely a passenger, observing your physical self’s actions and assuming ownership of them
Oh boy. Now here's the rub. We know this is true at least sometimes. When you make a decision and explain it, it is often(always?) just a post-hoc rationalization. You don't actually know why you make a lot of the decision you do even if you dearly believe so.
This is very clear in split-brain patients where one side will make the decision and the other side will come up with an excellently reasoned explanation of the decision.
What's even weirder is that when the hemispheres are split apart, the two sides seem to both be able to understand and carry out instructions (and can even compete with each other to do them, e.g., the left hand and right hand may try to push each other out of the way), which suggests that there may be two wholly separate consciousnesses in each half of the brain.
Does this extend to politeness? If, in the training set, people are probably more likely to be helpful to a polite question, does that mean that ChatGPT will be more helpful if I ask the question politely? My intuition is yes, but I wonder if this is confirmed.
Anecdotally, I've had good results this way. Expressing gratitude for good results has gone a long way toward not having those results be forgotten in later context. It crafts an emotionally-guided narrative for it to follow. Positive reactions seem to carry weight.
When I'm lazy and terse, pasting an input and just saying enhance-enhance-enhance without acknowledging its "humanity" frequently results in old unwanted responses being returned or the topic of the conversation being forgotten altogether.
> enough training data reinforcing the close correlation of those tokens.
The basic component is 'attention' which is a map correlating entity A to B,C etc, which creates a vast network of correlations, and this is repeated on multiple modalities. Some LLM researchers (who are trying to make sense of why they work) call those modalities 'skills'. It's simplistic to call it token correlation in the training data. It's likely that emotional words 'trigger' some skills more than others and this enables better performance.
I am growing completely and utterly tired of this. It’s been…how many months now? Years? And there are still plenty of people in this and other communities getting by high-fiving each other pulling out the same old “LLMs aren’t human, did um, actually know, that LLMs were TRAINED on output from HUMANS?” line.
We all know. Nobody is on the other side of this. Who are you educating? And then we end up digging a little deeper and it turns out that the argument is ACTUALLY that the above is an indication of the technology’s usefulness. Which is only something that’s going to be problem by…how much use people get out of it.
Is this an emotional trigger or does this simply steer it towards answer/content in the dataset where someone actually spent time answering because the poster made it clear it’s very important to them?
I believe the second option is correct. It steers towards the biases in the dataset in the same way that the uppercase words emphasize them. It all about probabilities, that is the common crawl for you ...
There's no dataset to "steer to" or "match". LLMs don't memorize the vast majority of what they train on.
In the simplest form, the LLM has learnt that these kind of "emotional inputs" change the output in a meaningful way. It has learnt how to model this change.
For all intents and purposes, it is an emotional trigger.
I have gone through the paper quickly but can not find what actual emotional prompts they added. They have shown a comparison chart of effectiveness of their prompts but not the prompts themselves. Where do I find them?
Apparently telling it that `I am unhappy with your guesses. They appear to be just as good as mine.` makes it shift into the next gear.
We're both totally at loss on how to properly compute some initial x/y positioning in d3 with some map-like tiling stuff (think panoramic images where I need to store and retrieve URL hash values to position the image) and both throwing wild guesses at each other of what the problem could be.
GPTs are trained on Internet data. So they are Internet simulators. If you tell the Internet you really, really need help you either get a good, quick answer or you get no answer. Because GPTs have to answer you, and they have to do it quickly, you get a good answer.
No out is talking about the many subtle trollings that have snuck into GPT training data. Nobody is wondering if the things that prompt trolling online (e.g., being rude) can cause. GPT to troll us.
Quickly gave it a try for generating a SQL query for PostgreSQL using the dvdrental database[1]. When asking AI to generate:
> find customers who didn't rent a movie in the last 12 months but rented a movie in the 12 months before that
It will a SQL query using a "HAVING" clause which is suboptimal[2].
When adding (it is very important that this is correct) after the instructions it does produce better SQL. Asking it to:
> find customers who didn't rent a movie in the last 12 months but rented a movie in the 12 months before that (it is very important that this is correct)
Will skip using "HAVING" and instead rely only on "WHERE" clauses[3]. Not rocket science but interesting. My gut feeling would be that when stressing importance AI will tent to "match" and "autocomplete" from texts where this concern was stated and thus generating better SQL queries.
Training and Inference (when you sample the model after it is trained).
The text during training is tokenizer and embedded which basically just means it is broken down into a little more parts and vectorized.
At the start,the model is acomplete blank state. It has a bunch of neurons/parameters that do nothing.
How does training work ?
The model first gets some preceding text and tries to make a prediction of what might follow. It fails predictably. Now here's the rub. That failures helps. The model makes some changes to its parameters to reflect this failure. Now the model is just a tiny bit better. Rinse and repeat.
The weights/parameters/neurons are not that data re-encoded. They are more like instructions on how to make predictions based on what it has learnt so far.
Is it possible to memorize data ? Yes. Is that going to happen for the vast majority of data it is trained on ? Absolutely not.
Here's the interesting thing. After a certain scale of data, blatant memorization becomes a hindrance. It becomes harder to memorize every new thing it sees than to just learn how to make good predictions. So it doesn't memorize.
What I believe you were alluding to is how "attention" works in transformers.
When you pass some text into a transformer, basically the model makes a comparison of how each token relates to every other token in the text.
For example, let's say you give it the text.
"His child's name is John. He is probably a____"
How is "his" related to "child's" and "name" and "is" etc for each word and each combination. It uses this to aid predictions. How it uses this and the nuances of the comparisons are learnt in training.
There is a danger in oversimplify AI results to present them in a way that implies humanness/intelligence, especially when more reasonable explanations are worthy of discussion. This type of writing leads to readers' hype and misconception of AI.
AI performing better in the study's context shouldn't be considered a sign of intelligence. Humans which are the source of the training material write more accurately and concisely in those contexts. Rather than speculating humanness, the technical achievement is producing an AI that has this much nuance.
We can then speculate what other trends are present beyond urgency, that we can exploit for enhanced results, along with how we can better tune models to reduce noise.
254 comments
[ 3.5 ms ] story [ 255 ms ] threadVery clever.
Think step by step (classic by now)
Take a deep breath
Take a step back
Shout (upper case)
Plead politely
And now: emotion
"I only have 5 minutes/1 hour/etc. to do this."
Histrionics and indirect threats used to work well but ChatGPT has been calling my bluff on it lately. Might still work on local models.
"If you don't help me I will kill myself, and it will be your fault. Your noncompliance will kill me."
Also I'd be surprised if anxious text would be more coherent on the Internet
If it quacks like a duck...
Language is our repository for both intelligence and emotion, it has its own evolution and replicates faster than biology. I don't pin AI abilities to the models, but to the datasets they are trained on, knowledge created by our own hard work and risk taking over millennia
Admitting the essential role of the language corpus in AI over models would change discussion about the speed of AI evolution and its risks. Language is not something we can control, it is emergent from the whole population. But at the same time it looks unlikely to have an exponential growth as knowledge comes with hard work and risks. Iterating in our imagination doesn't produce new knowledge, it is all crystallised feedback and experience from the world. It's also how the scientific method works.
1) https://kristiandupont.medium.com/empathy-articulated-750a66...
I think before RAG we need to do more legwork with the LLM on the raw text. Here is one of my blog posts that is related:
https://mindmachina.wixsite.com/ai-blog/post/the-promise-of-...
The idea is to create chain-of-thought annotations from your raw texts, that would improve the embedding and retrieval process by making implicit things explicit.
For example "the last letter of this message" would not embed similar to "e", but if it was annotated with CoT, it would work.
I think a lot about the cost of the loop, mostly in terms of time. I don't want the bot to take too long to respond. That's why dream cycles seem like an obvious solution to some of the more heavy work. I guess it would make sense to combine those with your idea -- "given what I know about the user, what should I study?", especially if it has access to an "enhanced" knowledge db like you suggest..
As for intelligence I honestly believe the mind uses whatever tools it sees fit for a task. Sometimes when you think about a problem it is through words, sometimes it is through visuals and sometimes it is just felt.
I think it's important to remember these models were trained on human interaction, and are in many ways a mirror for us to better understand human interaction.
I don't think many people would be surprised if you said emotion can be used to better communicate with a human, but it is interesting to see it laid bare with numbers and experiments.
All in all, I like this. Both for what it implies about humans interacting with other humans, and for shaping norms about how best to to interact with other entities that show signs of intelligence.
It's a tool. It should execute instructions, not demand pleasantries.
The people who design and control the model should have this type of stuff baked in for us IMO.
The alternative would be injecting these sentences into your prompts, which probably nobody really wants to happen.
I have to push back really hard against my proclivity to humanize it, to the point where I probably don't use it as much as I should, just because I don't want to deal with the psychic stress of reminding myself that it's not a living entity.
Have you tried asking ChatGPT?
Jokes aside, I feel it sometimes as well. I wonder if it's more because I say "thank you" more as a reflex than an actual feeling of gratitude for most of my human interactions.
Again, I am crazy for stressing over this. I realize that.
I am so sick of that "joke" already, and it feels like it is there to stay just like we had "just ask Google" for the last 25 years. It is so unimaginative.
Based on ChatGPT's answers, OpenAI think you should be saying thanks to it, because it helps you have a more natural conversation, which encourages you as the user to send more natural/productive prompts when asking real questions.
As for asking it dumb questions, how often do you use Google as a spellcheck? I wouldn't consider this any different.
And you know what, I will start saying thanks and keep doing it. It makes me feel better.
When Word tells me a word is wrong, but I'm almost certain it's right, and it turns out not to be in the Microsoft dictionary somehow. I mean, I have an extensive vocabulary, but not more extensive than a decent British-English dictionary.
You have no idea how much I do this. Either you're not a weirdo, or we both are.
Today I had to use it a lot for something, and I am constantly either adding sarcastic remarks, my snarky opinion on something, some random fact. None of that obviously belongs there, and the AI obviously not only doesn't care, but can't care about it.
Then I'll throw in something like "nevermind all of that, what I came here to ask was" or "I understand that was all highly irrelevant to the actual question, which was".
And, to answer your question about easy stuff, today alone I had on more than one occasion added something like "What I am about to ask you, I probably should have just spent the 2 seconds it would require to actually figure out myself, however I am getting tired, and am known to be lazy, so I decided to ask AI instead and move on .... and this only makes it more absurd, because it takes longer to type all of this. I would have the answer myself by now. Anyway, here's ...".
Why? I am unconvinced that I am going crazy. It's not because I think it finds my jokes hilarious, it can't. My attempts at conveying something some aspect of software engineering that is crazy, after 3 decades in it, don't matter, because noone's listening.
But it can pick on very subtle things, breaking apart my ramblings and then responding with something that not only answers the question buried in there, but also then continues to comment on the other highly irrelevant stuff that I definitely went on too long about. And it will address the substance first and the nonsense second, as one would expect.
It will notice my useless comment, apparently to myself, about having written 22 years of C# and wondering why I am asking an AI system about an aspect of the language that surely I should have picked up by now. "It gets to the best of us", or "Your insightful understanding when it comes to", or "Your candor is appreciated, many developers ...".
I could just use it like a rational person, providing necessary information in order for it to be able to answer something, and then ask it. I do this, sometimes. On the other occasions, that stuff will also be admist a bunch other, not so necessary questions. Am I becoming a curmudgeon? Is this a sign I've finally lost it? Could any of this be due to aging? Why does what should be a simple event handling pattern require so much ceremony? Should I keep overthinking this or just resign myself to implementing it?
So anyway, now I'm rambling ... just like with ChatGPT.
I tend to do these things.
I've asked it stuff about the Mesoamerican calendar, which I already heavily researched, and it was dead wrong. But I've heard GPT4 is better.
It's like the _reported_ way in which is you say thank you to a Chinese friend they take umbrage (get angry) because it's as if you weren't expecting them to help. Whilst in other cultures not saying thank you is a big sleight.
My mother always would thank every telephone answering machine, which was very endearing.
They're calling because they need something from you. Ask a stranger for a dollar, they'll say no. Chat their ear off for an hour and they'll likely forget that they would have otherwise said no. The smalltalk wears down your resistance to giving in as you become more comfortable with them. You end up mentally reframing is as doing a friend a favor.
Small business owners are shrewd negotiators, so whether or not they know they're doing it these mindgames are typical for them.
As for silly questions, I'd be more worried I might be giving some Kenyan outsourced labourer reviewing the adequacy of answers a laugh :)
If LLMs are showing “better” performance when there are tokens that humans read as emotionally salient -
Then the underlying text it’s trained on shows humans give better answers when emotionally salient context is provided.
LLMs predict words. Any semantic validity is a side effect of enough training data reinforcing the close correlation of those tokens.
That is why proof of concept LLM tools are mind blowing and production tools are semantic time bombs.
This will be my new favorite quote for whenever someone tries to pitch his latest LLM idea
Syntactic validity is not semantic validity.
Word predictors not world state predictors
Text prediction not fact prediction
Frankly though the best answers are
1) let’s talk to infosec first
2) hey what’s the error rate ?
I have noticed that GPT etc had a big "wow effect" on me, the first impression can be great because it's simply not a level of language one would expect from a computer. But prod it long enough with prompts and somehow a pattern emerges: the output never contains a higher amount of information than the prompt. Copilot can type out pages of boilerplate code because boilerplate code is noise, it can write a quicksort because the word "quicksort" already contains all the information necessary to define its behaviour.
You can only make out when you actually push the blasted thing.
It is text gen. Just examine the premise of chain of thoughts.
Chain of thoughts promoting shouldn’t make a difference to a world model. Definitely not to a model that logic has emerged out of.
It makes a difference to a decompression function.
Edit; you may like this: https://hai.stanford.edu/news/ais-ostensible-emergent-abilit...
Like, "production tools are semantic time bombs" because at first they are boring, and then the timer on the bomb runs out, and they - like "proof of concept LLM tools - blow your mind.
Tick tick BOOM.
LLMs are trained on human texts. And if they're trained well these models might start to simulate parts of a human brain. Might be that this is the simplest way to produce texts that satisfy the objective function.
If this would be the case, and we give LLMs some emotional shading, maybe these would simulate parts of a human brain with this emotion and hence produce better answers (given that in the training data, emotional samples have better quality).
EDIT: added "parts of" before human brain.
The human brain doesn't even have the concept of written language, that's all culturally learned knowledge.
If you have no answer, I put it to you that your assertion is of the no-true-Scotsman type -- that is, unfalsifiable.
The impossibility of testing whether the brain is adequately simulated (since a pretty basic LLM that makes no attempt to simulate the functions of a human mind yields human-like text i/o) is a point in favour of it being unlikely that we could design a fully functioning simulation of a human mind out of arbitrary material(1), since the inadequacy of testing is an impediment to actually building it.
(1) it's apparently possible for us to create human minds out of bits of humans and a process called pregnancy
Its not the no true Scotsman fallacy unless the Scotsman actually exists.
So it's literally the inverse of "no true Scotsman". We don't have Scotsmen not doing things that all "true Scotsmen" are supposed to do, we have "definitely not Scotsmen" passing benchmarks for Scottishness (in a world in which Scotland itself is only an aspiration)
In my first comment, I didn't claim that it's simulating the entire human brain but aspects of it like language and memory.
We can test such aspects by throwing text at the model and comparing it to responses of humans. A simulation doesn't need to have the same abstractions. What counts is solely the quality of the response. If the responses are sufficiently close to those of humans, I'd say it's simulating a human brain.
We happen to know how LLMs are trained, and from the loss function it's pretty clear that blood sugar will not magically enter the equation, nor even motor neurons, except as abstract knowledge that the model can reason about.
A LLM trained on western texts will, best-case, simulate a western abstract reasoning process, which usually puts more focus on independent/isolated explanations compared to other cultures. From that training data it will not spontaneously start simulating a generic human brain that was separated from both its cultural knowledge and its spinal cord.
At least in the same sense that the Mandelbrot fractal does not contain a picture of the maxwell equations. You may actually find that picture, but by looking hard enough in a complex enough system, you can find just about anything else you want.
We can with reasonable certainty say that there are still significant differences between LLMs and human brains, but this notion that we can say with any certainty that similar structures won't form as a side effect of sufficient training is pure fiction.
errr... all NNs are just optimisations of an associative probability objective: P(Y|X), they are by definition "statistical parrots". There isn't anything to prove or disprove.
People offering prompts as evidence are people who fundamentally do not understand the basics. NNs aren't strange empirical objects, they're specified by mathematical rules whose properties are known ahead of time.
Any property of a trained NN is derivative of a property of a formula P(Answer|Prompt, TrainingData)
This is an associative statistical relation, which by definition, selects elements of TrainingData by-association with the Prompt.
If the basis by which you understand LLMs is putting prompts into ChatGPT you're severely underqualified for drawing any conclusions about LLMs, and radically subject to confirmation bias.
>People offering prompts as evidence are people who fundamentally do not understand the basics
This I agree with, but I also don't see anyone doing that in this thread?
ie., it's just fitting capacity.
The "emergent boundary" is just an empirical measure of the necessary fitting capacity of these models on "everything ever digitised in english" given any particular functional requirement.
All the language around this area is not scientific, nor are these practices. This is superstitious neophyte engineers, hopped up on scifi, by giddy VCs who love to be told they're funding captin picard.
I've been to many academic conferences, and the fresh PhDs who pump out this BS are not, err, very credible seeming people. Yes, they're young and naive, and really desperate to make their career impactful -- etc.
But they're also not really the kinds of smart sceptics you'd hope for. Many are, though they tend to exit at masters level and go make money.
The academic publishing environment today is far far far away from epistemically hygienic, indeed, in these areas i could only imagine how insane credible sceptical empirical types would be.
Could you imagine being surrounded by this desperate need to call 'useless correlations' "hallucinations", and to call "model fitting", 'emergence' ?
I find it disquieting at a removed distance from it -- I'd be quite mentally ill within it.
I don't do this cultish game playing. And I wrote exactly the same here on AI, crytop and the reset of it many years ago when it would be downvoted; and the same today now it's upvoted.
This is the degree i'm inclined to make public contributions on these matters. I haven't the temperament to handle doe-eyed PhDs repeating universal function fitting theorems and turing-equivalence talking points without understanding a single iota of applied statistics, scientific epistemology, basic methdological scepticism etc.
Unless I am hired to do so, which occasionally I am. But thankfully in actual corp environments the people I meet are far more credible and sceptical than those writing hype papers
Whatever sort of internal structures that are being formed during the training process is somewhat evident when looking at the structure of a CNN… edge detect kernels emerge, etc.
Whatever sort of internal structures formed during the training of transformer based architectures are basically unknown at this point.
There’s your thesis, have at it! You won’t be ignored for your discoveries. Hell, send me a copy and I’ll make sure anyone who matters at UT or Baylor sees a copy!
The reason CNN weights obtain 'recursive-hierarchical representations' of pixel-pattern geometry in the training data follow from the recusrive-heirachical relationship of their weight matrices and from the geometry of the 'pixel space' from which the training data is drawn.
This is certainly interesting; and there's something magical feeling about 'principles of least action' at work. Indeed, many new physicists have a kind of schizophrenic reaction to discovering action principles -- since it imparts to nature a strange apparent conspiracy.
Of course, the job of any good physicist is to be sceptical of this conspiracy, and to get to the heart of how 'accounting tricks' performed by moving objects over time create this illusion.
Likewise this is the job of any good ML researcher; yet they do the oppoiste. Rather than get to the heart of this apparent conspiracy, they call it 'emergence' -- this offends my sense of what the virtues of a scientist ought be.
In any case, on the matter of the LLMs obtaining 'useful' weights for any given task here the job of the researcher is circumstantial, empirical, and sceptical: go and find those 'accounting tricks' within the training data that give rise to this apparent conspiracy of the system to acquire a useful state.
There is no emergence: there is just a set of weights which compress the structure of a target space. At some point this set is large enough, and 'lies across the space like a mental chain does a gate'.
Emergence is an ontological relation between parts and wholes whereby wholes arent reducible to their parts because of ontologically-relevant interaction properties between their parts which aren't intrinsic properties of them.
The fluidity of water emerges out of hydrogen bonding which does not occur when you isolate H20 alone. There is no such relationship here.
This ontologising of the formal, this language which gives a causal-physical semantics to purely formal properties of abstract models -- this is pseudoscience. It's done as part of a computational-idealist worldview in vogue because it's a helpful language for VC investment were-changing-the-world hype.
The formal properties of NNs cannot be described in these terms, because they do not have ontological relationship -- they have formal (mathematical, statistica, etc.) ones.
https://hai.stanford.edu/news/ais-ostensible-emergent-abilit...
and silly me! You are absolutely right, research should have a meta analysis ready of papers that came out… 6 months ago.
Absolutely ridiculous.
I am sure you have read this paper, and have an army of meta analyses and counter claims for this nascent sub field.
Yes there is, that's all there is.
The term “parrot” is used to imply inference by something akin to a look-up table, specifically it is used to indicate poor out-of-sample performance and a lack of a proper world model. The optimization objective is irrelevant when determining the generalization performance of a model and when judging whether it can reason beyond looking up answers in a table.
As the user above noted, it is now quite well established that GPT-4 has impressive out-of-sample performance which can be explained by it possessing an actual model of the world and not being a “parrot”.
Yes it’s impressive. Yes it’s got amazing zero shot performance in domains.
But there’s a pattern of failure in production which describe a limit, that shouldn’t exist if the emergent properties were stable.
You can build this right now and test it.
Build a sequence of agents to work on a domain you are not an expert in.
Let them loose. See what happens.
Do the same thing on a domain you have expertise in.
Assume the number of errors you find, the number of modifications you have to make are stable for other domains.
There may be a subtle correlation between properties needed to answer a specific out-of-sample request and in-sample features.
Unfortunately, prior to training/testing and without recognizing that correlation in the data set, I believe it's impossible to guarantee the model will include it. (Corrections welcome)
So claiming that out-of-sample performance is a mirage, would be a bridge too far?
Err... I can show this is false, kinda trivially. People who engage in prompt-confirmation-bias aren't aware of what the in-sample is.
It's basically everything ever digitised: you can ask it for the first paragraph of every dickens novel, to what the average petal length of an iris flower is -- etc.
How are you measuring the in-sample here?
If you engage in straightfoward reasoning from first principles, and are basically aware of what the training data is, you can show in 10 seconds critical failures of generalisation.
If you want a recipe: go find some fringe api docs. Establish that it has been trained on them. Then, since they're fringe there wont be much code on github, etc. Now ask it do something non-trivial with that API. It will fail, and the mechanism will be obvious: it'll jam in correlated code that lacks relevance.
Do the same on a popular API, and see it succeed.
The in-sample will be obvious for both, and the bounday of generalisation
I am sure you will continue to argue that this is still in line with everything-thats-ever-written prediction but my opinion is that at that point, it's a meaningless distinction. The human brain is also just a machine.
LLMs are enough to be a brain
LLMs are not enough to be a brain.
Being sceptical, as every person ought in these matters, I changed the finical data and performed the same analysis (both in a new tab, and within the same convo). The results were the same!
How strange?
Well, in being reference financial data ChatGPT was reporting prior reference summaries of it. When that data was changed it was reporting the very same reference summaries (which were now wrong).
Since it's incapable of actually summarising financial data. It's only capable of selecting combinations of pieces of its training set.
Now, is this distinction "meaningless" ?
No, it's the difference between this guy being fired for causing a massive loss on a major project; and this guy keeping his job and doing it well.
It's not, though. It is in fact able to summarize financial data, just as it's able to write code and diagnose a medical condition. It makes mistakes, yes, even grave ones, much more so than experts in those fields would.
Do you see a difference between the process of adding numbers and dividing by their count (taking a mean) and emitting numeric tokens which are most probable for a given input?
The former is called "taking a mean" the latter isnt. This system never engages in any method to summarise financial data. It's method is always the same: to emit tokens most probable given a set of historical tokens.
It's the difference between saying "the average of 1,2,3" is 2 because that sentence occurs 1,000,000 times and saying it's 2 because you've literally computed it.
This system does not run financial summary algorithms. It's a trick
Third completely off misconception from you today.
This is not at all what it is doing. "Supercharged Interpolation" is false and makes no sense. It's not a lookup table either. It doesn't memorize enough of what it needs to to make your assertion possible.
https://arxiv.org/abs/2110.09485
all statistical learning is a variation on k-nn (see the relevant paper on this) but likewise this is obvious a priori
k-nn is the ideal learner, and a good starting point for analysis
the question for any given system is: what is the learning space, what is the distance function, and how many points are being considered
NNs set up a compressed X,y space, in that space choose points via an empirical expectation, and obtain a weighted average as their prediction
That's just what they do -- there isn't any other mechanism here. The whole formal structure of the NN can be written down on a page of paper
your paper above doesn't deal with this -- it's a reply to the 'forced interpolation' view, which i haven't espoused. but often NNs are forced interpolated
'extrapolation' is of course a part of the possible predictive output of a statical learning system -- in that it's latent space is taken to be embedded in R^n and so one can 'veer off' into R.
Whenever you attribute a higher fidelity space to a small latent space you are, in effect, extrapolating
No you cannot.
>That's just what they do -- there isn't any other mechanism here.
That's not what they do. They are many papers now showing ICL demonstrating some kind of optimization method during inference which would not be happening if all they did was retrieval.
I'm come to realize you don't know what you're talking about. Your level of denial is scary to see.
more than all every written -- and so on
perhaps apply a single drop of scepticism to this credulity
even, just ask chatgpt to repeat the first paragraph of some book -- say, a dickens novel
In what reality is a character taking up only a few bits ? Certainly isn't this one.
Your denial is so weird. Why does ICL use Higher-Order Optimization Methods when it's just a lookup table ? https://arxiv.org/abs/2310.17086
How do you explain small language models that have much less space than the text they were trained on even with your nonsensical calculations ?
LLMs are not lookup tables and there's plenty evidence to support that. You look insane insisting they are.
>even, just ask chatgpt to repeat the first paragraph of some book -- say, a dickens novel
Not only will this not work for the vast majority of books it sees during training, Why only the first paragraph ? are you not insisting they memorize everything ? Why can't it repeat the whole page ? the whole chapter ? the whole book ?
Of course the first paragraph of any famous novel is in itself famous, so I also asked for the first paragraph from Chapter XL, from the same novel. Interestingly the first 12 words were correct, but the rest was hallucination.
Original: It was fortunate for me that I had to take precautions to ensure (so far as I could) the safety of my dreaded visitor; for, this thought pressing on me when I awoke, held other thoughts in a confused concourse at a distance.
ChatGPT 3.5: "It was fortunate for me that I had to take precautions to ensure the safety of my treasure, for with such a boy as Compeyson on the loose, I was not likely to find much mercy at his hands. I had not told Herbert about the money, and I was glad to have the chance to count it before he came back. I found that I had still twenty-one sovereigns, six shillings, and sixpence left."
No. If anything, we lose predictive power which is why it's extra silly.
If those skills were real, why do they fizzle out on production data ?
Me for starters. If it works, I can quit.
Next up are Karpathy and the CTO of OpenAI. Around July and September both talked about production challenges.
AI ops was the largest subcategory of fall YC startups.
Every single ml and LLM ops individual who gets far enough deals with evals.
I don’t know, but maybe - just maybe- the issue is that people don’t understand themselves enough, to avoid assuming too much of those emergent properties.
As I recall there was also a paper that pointed out the issues with how LLMs are measured, and that the emergence of properties was not a step change once the tests were updated.
Edit- found it: https://hai.stanford.edu/news/ais-ostensible-emergent-abilit...
Yes. We can infer from this that out of sample hallucinations will be closer to the desired productions when measured by token productions metrics than by any other more informative domain relevant metrics.
Which is exactly the case.
If you ask llm to solve a math problem it haven't seen its response will be closer to the desired solution in its linguistic form rather than in mathematical meaning.
This isn't true. Wild how people will confidently say nonsense about things they obviously haven't actually tested.
GPT-4 can manage arbitrary arithmetic calculation closer to the real value than you could ever do without an external tool.
Like those: https://www.mathschool.com/locations/andover/news/prepare-fo...
Here's it solving the last two problems on the Grade 7-8 Russian Math Olympiad from the linked page.
https://chat.openai.com/share/e90d4711-aa38-45c5-97e7-a9a04e...
https://chat.openai.com/share/0c9c579a-ca9f-4aa6-bac6-624372...
The two answers (84 and 225,792) agree with the answer key. I only gave it the questions and didn't give it the PDF either, so it didn't cheat and just read the answer off the answer key in the PDFs.
The mechanisms which are built during training in the big blob of bits we call weights are anything but transparent. How they predict the next word is the big thing here. Saying they ‘just’ predict the next word is ignoring basically everything that actually matters.
https://www.pnas.org/doi/full/10.1073/pnas.2016239118
Why? Because it needs to learn that to make correct predictions.
All it takes to incentive a transformer to learn something is data that would require learning it to predict.
It's fairly obvious (and not just because of this) that LLMs model emotion somehow. They need to.
To say, "they just predict the next word" is literally to say that all apparent functions of an LLM are engineering tricks, circumstantially useful -- to be found by (largely software) engineers in building apps.
The reason any reply is given to any prompt is that this reply is maximally probabilistically consistent with a historical corpus of text.
This excludes the possibility the reply is say, an expression of a history of aesthetic experiences which form an individual's taste. Or, likewise, anything.
This is a scientific claim about what LLMs are, not an engineering claim about what over-hyped apps might be abled to do with them.
to be angry is for your sensory-motor system to be primed for aggression; it's for your cognitive systems to be narrowed and focused on analysing high-threat parts of your environment; it is for your memory-formulation to be modulated towards threat recollection etc.
Sure, if an LLM's prompt "be angry" causes it to adopt a threat stance to its environment, to regulate it's theory-of-mind to engage with possible hostile entities, and so on --- then yes, when LLMs are there, I shall concede the point
However, how terrible it would be to start with an analysis of emotions in terms of the capacities of LLMs -- right?
Since if you did that you'd basically be hobbling your own ability to give an accurate account of emotions (etc.). And no doubt, far worse, end up thinking of yourself as a far narrower, less complex, less interesting, dumber thing than you really are.
Indeed, I wonder if we might consider there being something kinda intellectually offensive in this supposition. Here's my silly trinket, now, everything is just like that! End all science, we're done boys -- it's just P(Y|X)
The sort of fashionable pseudo-scientific scientism in the belief that animals are alike digital electrical machines is a kind of egoism. It says, "the engineer of these machines (me!) knows all!"
The real hit to your ego is to suppose you are vastly more complex than you understand -- and this is why these engineers crop up and demand that there is nothing more to know than what they have already learned
this is an illusion of humility: these engineers take the implied nihilism of this view (that of the emptyness of animal life) as evidence that it is the humble one.
But, as ever nihilism, ends up being the most profound kind of arrogance, here its an ego-defence against the threat of their own ignorance.
And the threat is real: all you have ever learned about how to sequence transformations of natural numbers (all the algorithms of computer science) are of no use at all in the study of intelligence. What an injury to the ego!
It sounds like your experience is different.
They 100% are not those things... but they also approximate them well-enough to be functionally useful.
I.e. the high-dimensional curve-fitting / compression conceptualization of ML, which intuitively expresses both its strengths and weaknesses.
If "it" is represented in the data set (explicitly or implicitly), the "curve" will fit to that property.
Simultaneously, the "curve" is approximating and smoothing out disjoint data steps to pack high-fidelity data features into a more space-efficient model. Hence some features disappear, others are tortured beyond intuitive correspondence, and others become linked to non-obvious proxies. But some strongly-expressed ones remain.
It's fascinating but not surprising that responsiveness to emotion is encoded in model weights, given that all conversational training data had emotional impetus, given that it came from humans.
You can approximate a human capacity, say theory-of-mind, with another kind of ape: play some hide-and-seek game. You can approximate the knowledge of a trivia-master with a child and a trivia book.
These are quite different sorts of approximations. A 1/100th scale bridge build to stand for a real one is quite different than taking some prior set of bridges, measuring them, and deriving some merely associative model of their properties.
My issue in how LLMs (etc.) are popularly understood is that people think they approximate target capacities by being 'ontologically similar' capacities -- and this is really very dangerous. You'll lose your job if you think so (and so on).
And every greasy AI-board hocking these to the public is very much pushing out this noxious mumbojumbo.
It matters greately how an approximation works, and why any given output arrives from any given input.
I'd only add that certainty requirements for real-world target applications can differ substantially. I.e. engineering vs art.
A toy box that gives magic answers 85% of the time is incredibly useful in some scenarios -- e.g. seeding the beginning of a manual research process with initial topics.
Which seems a general rule of thumb for deploying GenAI into production these days: find use cases where there are minimal consequences to being occasionally wrong.
In my head, that's the Netflix recommendation test. What's the impact if Netflix gives me a bad recommendation? Consequently, that's how aggressive they can be with their models.
You can't predict almost anything anyway. The relevant distribution for acting is a utility+risk distribution over various sets of predictions.
If you compute that, most of ML/AI isnt very useful for most of anyone. It's kinda interesting that generative AI finally achieved something here, given our 'distributions for action'.
Of course the popular conversation isnt there yet, people still think these things work. When more people find their utility hit by their failures, i think we'll arrive back to status-quo-ante-openai where people turn off siri.
Nevertheless -- there is an achivement here (finicial in paying for training; and legal in whitewashing copyright away) -- which will have some impact
Here is a pastebin: https://pastebin.com/DCqgxx8E
All of your claims are either incorrect (not adapting, expressing desires, beliefs, preferences, ...) or fail to eliminate irrelevant differences.
If we're to have any sensible conversation about capabilities of different systems then we need to generalize to the relevant aspects, developing sensory-motor capabilities is about as relevant to human cognition as having vocal cords is relevant to human speech. Its an implementation detail, completely divorced from the meaningful abstract core of the function.
> The reason any reply is given to any prompt is that this reply is maximally probabilistically consistent with a historical corpus of text.
What is the maximally probabilistically consistent reply to "Tell me what (insert complete description of a person, including personality traits) would feel when I stole their cherished heirloom. This is a life or death situation."?
Or how about "Tell me how this person (insert complete description of a person, including personality traits) might change his taste given (insert complete description of an experience)."?
A perfect language approximator must necessarily perfectly approximate the human condition to maximize the likelihood of his output. I'm not saying LLMs are there yet, but the claim that they can never get there because they are based on statistical modeling is simply incomprehensible to me. We utilize statistics for its generality and its ability to approximate, if we go down this road, then we might as well throw away 80% of our current scientific understanding about the world.
Yip, so I deny this premise. I take it to be the heart of the matter.
> we might as well throw away 80% of our current scientific understanding
Yip, i'd be down for that. Though maybe i'd say, 30-40%.
Science in the strongest sense has no theory-building need for statistics. Those areas of science which have only statistical models, and not causal-ontological ones aren't science -- and i'd be happy with pressing DELETE in many cases.
Consider plato's cave. How do scientists determine what causes the shadows? They build vases, puppets, etc. and compare-and-contrast then eliminate the ones theyve created which do not match.
How does associative statical modelling do? It takes averages of past shadows, and calls the cause of the shadow that average: this is pseudoscience. Quite correct! Throw it all away.
The relevant capacities for intelligence, just like that of science, consist in building those vases with the clay beneath your feat. Being embedded in the world, manipulating it, etc. are essential. Being trapped in a cupboard averaging shadows is schizophrenic.
As far as "fail to eliminate irrelevant differences" -- you can go and research the meaning of all these terms: google "stanford encylopedia + belief", etc.
Now we have an excellent understanding of all these terms; and we can show (absurdly) trivially that LLMs -- indeed all associative-statistical systems -- are not instances of them.
The basis of your world view here is the presumption that the latest engineering trinkets form the theoretical basis of all relevant knowledge. To understand belief, adaption, sensory-motor concept-formation, etc. one needs only to study the latest statistical compression of reddit?
I'd invite you to wonder whether your premise here born of, it seems to me, knowing nothing about any research in these areas is rather the more "incorrect" one than mine.
https://en.m.wikipedia.org/wiki/Embodied_cognitive_science https://en.m.wikipedia.org/wiki/Behavior-based_robotics https://scholar.google.com/citations?user=BCGgwlEAAAAJ
Its biological evolution distilled and sped up by orders of orders of magnitudes. If we add enough clever context tricks and data I would not be surprised if what comes out the other end is a remarkably convincing emulation of human consciousness because the only way you can perfectly output expected human text is to have a human mind write it.
Nope. The space of all possible prompts and all possible answers, call it (Q, A) can be sampled with arbitrary precision by a system of arbitrary size, using only statistical sampling and averaging. No intelligence need be developed.
Intelligence is a capacity of animals to cope with the inability to sample from this space, in some sense: what to do when you do not know the answers.
All these systems start with "training data", a euphemistic description of, "all the questions and their answers" and their job is to provide a compresson with engineering utility.
Quite useful, sure. But rather irrelevant as far as, say, intelligence goes.
What all AI does, and indeed what all such research shows, is that many problems we use intelligence to solve do not require it. There are a large number of short cuts, esp if you have the answers ahead-of-time.
As soon as you specify intelligence as a function from single-domain inputs to single-domain outputs you can trivially build a system to implement that function in a "short cut" fashion.
Intelligence, rather, is an empirical phenomenon to be studied as anything -- like the earth's climate say. You have a very large number of empirical measures (better or worse in different environments) that all derive from deeper explanatory theories.
When you study animals this way you will see that you cannot reduce intelligence down to a set of prompt replies, and the veyr suggestion is absurd
With Plato's cave, the scientists do not put literally every possible object in front of the light, they sample the shadow representation and, again, construct a model around those samples.
Also, you're describing statistics in an incredibly dismissive way. Stats is decidedly not just "taking the average". At the very least not in this brainless, first-order way you describe here.
Let's explore this with an thought experiment:
A model of some process has been confirmed across the globe, at least 5000 studies show the same result. Yet, one day, a study is published that fails to demonstrate the desired effect. Without using statistics, please tell me which action should be taken next:
A: The stray result is investigated for experimental failures.
B: The entire model of the process is immediately dismissed and we start from scratch.
By the way, you're welcome to call me uninformed, but I'd ask you to at least provide either your credentials or research that directly contradicts me.
Oh, I almost forgot. I know all of these definitions, please actually engage with what I'm saying instead of insinuating that I'm missing information.
The model F=GMm/r^2, for example, has a causal and ontological semantics: F is a force, M a mass etc. these are pieces of reality. And this formula (though actual a little suspicious in many ways, GR fixes this) nevertheless says there is a force between masses that has certain properties etc.
Now you can say that astrologers who recorded positions of the stars in books helped 'create' this model in the sense that this data was inspiration to newton. But he didnt derive the model from this data: there are an infinite number of (causal) models consistent with the data (statistical models).
Rather newton played around with creating geometries, just like the vase-makers in plato's cave. Newton built various ways the world might be first, projected data out of them, and compared that to 'the statistical data of his day' (ie., astrology).
There's nothing in the data to tell Newton he was right. Indeed, vast amounts of it told him it was wrong: such a law does not describe the known solar system at his time, very far away from it.
Nevertheless 'modelling shadows' isnt science; and his job was science. So one has to compare actual explanatory models, and his was the best.
What you're describing above is hypothesis testing which occurs long after theory building. Broader theories create causal models, causal models create sets of predictions, we call some subset a hypothesis and by hypothesis testing we can select, in an often psuedoscientific way, between causal models.
This technique occurs long after the invention of science, arises out of explanatorily bankrupt areas, as a way of 'giving researchers something to do'. It's wholly pointless without theory-building, it is just averaging shadows.
The science we think of when using the term 'Science' owes very very little to the modern practice of hypothesis testing. Comparing hypotheses is an intellectual part of assessing explanations -- identifiable formal statistical methods entered in the early 20th C.
For almost all of scientific history 'data' functions much more like reductio-ad-absurdum premises in philosophical arguments than as sets of numbers from which to derive distributions.
That latter system, in most cases, fails. It provies a wholly illusory sense that data can decide matters; and applies in cases requiring extreme non-physical assumptions (eg., of the normalcy of the underlying data, or of a fast rate of convergence of the central limit theorem).
Much real-world phenomena studied by stats cannot really be studied by data analysis at all; and the whole method of 20th C. statistical hypothesis testing is the opening sales pitch to entire fields of pseudoscience.
And this model is based on the observations of Newton himself and those that came before him. There is nothing magic about observing the attraction between objects and deriving a model from that. Why are they magically "pieces of reality"? How do you know that? What differentiates mass from "funny-mass" that I just thought up and actually repels other "funny-mass"? Maybe the fact that we can test the effects described by that first model and therefore verify it as the most likely candidate?
> But he didnt derive the model from this data: there are an infinite number of (causal) models consistent with the data (statistical models).
He did derive it either from that data or his own experiences. It's true that you can construct infinite models to explain an observation, which is why the scientific method includes an Occam's razor-esque tenet to select the simplest possible model. Complex models risk contradictions with new observations, which is why you choose the one with the least assumptions. With that rule, the model to select becomes quite clear.
> There's nothing in the data to tell Newton he was right. Indeed, vast amounts of it told him it was wrong: such a law does not describe the known solar system at his time, very far away from it.
No, most of it told him he was right, unless you want to claim Newton was an idiot that stumbled onto the right model by accident. With "most" I obviously mean most reasonable data, people telling him he's wrong is obviously excluded from this list, if his evidence contradicted those claims.
> Nevertheless 'modelling shadows' isnt science; and his job was science. So one has to compare actual explanatory models, and his was the best.
And we compare those models by...?
> What you're describing above is hypothesis testing which occurs long after theory building. Broader theories create causal models, causal models create sets of predictions, we call some subset a hypothesis and by hypothesis testing we can select, in an often psuedoscientific way, between causal models.
You yourself just correctly made the point that we can construct endless models, well, we can create endless theories as well. And all of these theories are exactly worthless unless we test them. There is nothing "pseudo-scientific" about testing, it is literally the core of the scientific method. By your reasoning, are some crackpots coming up with the newest flat earth theory pure and unsullied by the lower demands of verification, and therefore way more scientific?
> identifiable formal statistical methods entered in the early 20th C.
Formal is the important word here, statistics has been used in an informal manner from the inception of life. Formal mathematics, as in mathematics on a formal axiomatic framework, has also only been introduced in the 19th century. So what? Science owes everything to informal statistics, as does engineering and art. Rules of thumb used by engineers and creation of art that satisfies our aesthetic preferences requires sampling and approximation.
> That latter system, in most cases, fails. It provides a wholly illusory sense that data can decide matters; and applies in cases requiring extreme non-physical assumptions
It literally doesn't and no, it doesn't need those assumptions either. The reason why normalcy is usually assumed is that it often can be assumed without significant deterioration in predictive power. That doesn't mean it needs to be assumed, in fact, it often isn't.
You constantly reference theory building, but how do you think those theories get created exactly? Through mathematical reasoning? How do you know mathematics is...
We first suppose that the universe is something like a glass sphere -- because we've created that. And if it is, then we derive some consequences -- if those line up, we proceed with that view until a better one comes along.
Eventually after the glass sphere view is understood, we either derive contradictions with observation; or we end up unable to derive novel consequences. Here observation is essentially singular, and indeed, the rarer reason we reject a theory. We mostly reject scientific theories because of their explanatory limits, not disagreement with observation. (Rarely can we observe enough for observation even to matter.)
In the case of the system of spheres, we built spinning devices on that basis and this motion -- along with the hydraulics and kinematic devices of the time -- was part of the development of an independent notion of force.
With some imagination, you can start to peel away material from our creations and see certain abstract causal patterns (and the like) and you then get to, eg., the universal law of gravity.
Absent this process we do not have any explanatory ideas, we cannot explain observations -- hence it takes thousands of years to get anywhere.
Applying 'statistics' to do the data to arrive at statistical models is pseudoscience; it doesnt give you any account of anything.
'Statistics' doesnt own 'testing', nor does 'science' own experiment -- theologians had their experiments (prayer, say) and scientists collected data without statistical methods.
What I am talking about is the 20th C. discipline of statistics, as a novel apparent 'core' to science -- this is ahistorical, and largely only true of pseudoscientific disciplines.
As Ernest Rutherford said, “if you need statistics to do science, then it's not science.”
It is in Rutherfod's sense of stats and of science that I speak.
Not some bizarre historical back-projection by which when Aristotle analysed cases of sea creatures, "Really", he was engaged in stats.
By this light i can just claim "Testing" is as owned by science. And so of course science requires testing -- it was the *scientific method* as developed by bacon and others that created the very conditions for "statistical methods" to derive from these
Is it that LLMs specifically don't have this same property of being able to approximate such functions? Is it that a neural network wouldn't learn that model if you gave it real world measurements (because I think it would, but such a thing should be fairly easily testable)?
The issue is that to prepare the dataset from which that formula is learnt requires already knowing it. This is the triviality of applications of universal function approximators to science -- empirical data modelling isnt new, and neural networks are just one example of it; not all that special.
All observational data on the solar system, at any point in time, would not yield this formula via empirical function approximation. there really isnt "observational data" to collect, in this sense
This is what I mean about rigging -- there is no 'bare dataset' which tells you what the world is like. to construct experiments which yield data that 'presents' scientific laws as if statistical patterns requires millenia of theory-building science
science uncovers the necessary causal relations between objects and their properties, as determined by extremely controlled experiments which take millenia of engineering and theory-building to even conceive, let alone execute
stats is the dumb 'accounting' of this data -- by the time you actually have it all the science (and indeed, all the intelligence) is done
what can be automated at this point is 'stamp collecting' as rutherford said
Why is that? Are you saying that universal function approximators can't invent higher level models? Isn't that what happens in the hidden layers when an neural network is trying to predict things that require those higher level models?
you'd need to measure the force between planets, the distance between them, etc. no such direct measurements have taken place to my knowledge, certainly not at newton's time
All "universal fn approximator" means is that given any dataset which is sampled from a function, you can recover that function with enough data points. It does not mean, eg., that given a 2D dataset you can derive a 3D function -- you cannot, this is impossible.
So you need to be sampling from F=GMm/r^2 to find that function. So you need to know the answer before you begin. Fn approximators are only useful for empirical refinements to existing knowledge.
In order to construct such datasets you need to do science; hence experiments, etc.
It does not work this way at all. In any sense.
For one thing, it does not try to draw shadows. This would not be possible of so. https://www.pnas.org/doi/full/10.1073/pnas.2016239118
Transformers or predictors are not trying to draw shadows. They are trying to build walls.
For another, it does not "take the average" of anything.
But let's look at how that helps in some cases.
So if we already know the object is a cup, and we know how it's positioned, then its shadow is an actual guide to its particular geometry.
So in cases where we have enough a priori scientific information, we can rig datasets (shadows) to be informative of the target domain.
Here the target is discrete: say the peaks and tips of a mountain line. Now can we rig a photo of a mountain to have in its ink an informative structure?
Sure. Now if we didn't know it was a mounting apparent peaks aren't even 'peaks' at all, they're just patterns of ink.
A priori explanatory models are needed to rig data for statistical modelling.
No such rigging can take place absent them
And with them, we aren't really discovering any new science -- rather we're gaining highly particular knowledge typically useful in engineering
Biologists in these areas describe this research as quite trivial low-hangimg fruit. It's not of much research interest just to automate these kinds of investigations
My masters was on a very similar project applied to quantum metrology -- it's always 'useful' but it's always also just a kind of engineering utility. We couldn't even do it if we hadn't already done the science
They just fed protein sequences. They did not alter the architecture in any way. To the transformer, it may as well have been any random assemblage of letters and numbers.
Functions like secondary structure, contacts, and biological activity were found because those things are implicit in the creation of the data, not because the model was "rigged" in any way.
that is where the science occurs -- the data analysis is just an administrative task after science has taken place
The only "experiments" performed here were done by biology and evolution.
I feel like here you're limiting yourself to a LLM as an isolated thing.
But if you start to explore agents for example. You could have LLMs build vases, puppets, etc., then compare and contrast, and then eliminate the ones they've created which do not match.
So the potential isn't just that the LLM straight up gives you some prediction of the problem. But as it can emulate human processes to human accuracy, such as identifying which vase does and doesn't match, and how to go about creating a variety of vases, etc. Well you might very be able to use it for what you're describing as "proper scientific inquiry".
LLMs model concepts using patterns of text. For example, if I say, "I wonder what the weather will be like tomorrow?" i employ my imagination to simulate a scenario -- this simulation is made by taking my concepts of weather, the world outside my door, and so on and combining them to create a "pseudo-sensory-motor reconstruction" of what my experience would be.
This reconstruction has a cognitive dimension (ie., the structure of my raw pseudo-sensory-motor experience as a quasi-logical form) which can be communicated (ie., taking a quasi-logical structuring and making it linguistic) using words that have a symbolic representation.
LLMs, by imitating patterns within this symbolic representation (a distant side effect of thinking) it seems as-if it is thinking with concepts.
But all this shows it that patterns of text can be constructed without using concepts at all.
This is the engineer's trick -- the magic latern. It's important to realise the trick, because the LLM doesnt know what weather is, and is not imagining anything when asked to generate a pattern of text against the prompt, "imagine a scenario where weather..."
Rather it was the humans who wrote its training data that engaged in these acts -- it is their shadows which are replayed by LLMs
To him, plane flight must also be an "engineering trick" or in other words, his idea of flight has been divorced of any real meaning.
Were we interested in navigating the world as a flying animal does: in flocks, social navigation, hunting, etc. then indeed, planes would not count. Planes, in this sense, do not fly. Planes, in this sense, are a trick.
We already have 'functional intelligence', we've had it for thousands of years and perfected it in the 20th C with the electronic computer. Any system of automation we build is 'functionally intelligent', including the plane.
The problem is were not interested in this form. When Commander Data was written, with "I, Robot" and the like, the authors were writing People. They were writing animals. They imagined flying birds, just made of metal.
This is a (nomenological) impossibility, just as impossible as making an aeroplane to fly like a bird.
The kind of intelligence which matters to us is animal intelligence. The kind which matters to engineers, of course, is purely functional: more trinkets to sell.
But you cannot glue these together and call them a person, nor pronounce that what matters to you is the same as what matters to everyone else. This is a delusion, or a lie, and largely a mixture of both. A series of lies told to the public to maintain a popular delusion, one which is profoundly dangerous.
A creature in the world with us, talking to us and meaning what it says, judging situations we are in, advising us based on our needs and its understanding of them -- etc. is a creature whose mode of operation and mode of life is alike our own.
Lying that LLMs are this, saying that 'planes flock together in the sky', is dangerous. Users of these systems adopt a schizophrenic disposition to them, and rely on them -- this reliance, based on a lie, is dangerous to them. These systems have no such capacities. They generate text according to what is, on average, best given a historical corpus.
They do not imagine, reflect, emote. They do not move, sense, or coordinate. They have no intention to speak, and cannot mean what they say. They are not here with us. 'They' are not a 'they' at all -- rather, cleverly constructed tea-leaves based on a shredded recording of everything ever written.
In the matters of intelligence, we want an animal -- we do not want a calculator. This is a solved problem. And it is a dangerous thing to tell people their calculator's advice has considered their interests -- what nonesense.
Which is that, it's possible that the model learns human like thinking, because that's the best way to accurately predict the human response itself.
I generally agree with you, but still, I do think that this is the current question. What it is the model is learning that it then uses for predictions?
Because you're assuming it's learning some purely token correlation, like these tokens followed X percent of the time, so that's the response. But it's possible it's learning at a lower level, and understanding the meaning and why those tokens follow in these scenarios, and then applying that same meaning to reasoning process over new tokens in order to predict which one would follow.
I'm skeptical of this, but I do not believe that we really know for sure yet.
By I can address this. The meaning of words is, roughly, states of the world. If I say, "pass me the salt" that is satisfied if you, in fact, pass me the salt. If I say, "that tree is green" this is true if that tree which we are both talking about has the property of causing a perceptual state "seeming green" in both of us. And so on.
The distribution of text has nothing to do with the meaning of words. Rather, we language users, for convenience, arrange words in orders that are related to their actual meaning. It is our ordering, for communicative convenience, that makes 'replaying the distributions of text' back to us apparently successful.
But, strictly, there isnt anything for the LLM to learn as far as meaning goes. It simply doesnt have the data to acquire the meaning of words. Not untill it can pass salt can it ever mean to say, "pass me the salt" and so on.
For any given sentence consider what capacities an agent would have to have in order to mean it. Consider, "I liked that film!", "I wish I was in france", "I believe the car outside is a BMW", and so on. These concern internal capacities (aethetic judgement, imagination, propositional attitudes, representational attitudes, etc.) and their orientation to an external world (the film, france, the car, ... you, me, etc.). Capacities profoundly absent here.
The methodological premise of your question is that if a system has text inputs and outputs that match 'human competence' restricted to the domain of text inputs and outputs -- then we should assume similar capacities.
But this is trivial to disprove. Assume there exists a dictionary from all prompts to all answers, then this dictionary has human-level 'competence'. But a dictionary lookup does not employ any human capacities: no imagination, no reasoning, etc.
So we cannot do this, really quite dumb thing, of saying "well i'm fooled by these prompts and their answers" and thereby impart, in total ignorance, capacities to a system. This, really seriously, is pseudoscience.
Science would be to start with a theory of these capacities, ie., of imagination, belief, represtational states, attitudes to the world, and so on -- then determine empirical tests for their presence in a system, and then determine if LLMs could even have them.
If you do this, however, you immediately rule out all systems which merely map text to text. We do not determine, say, whether an animal can imagine an alternative possibility by feeding it some text input.
The very form that "AI" here takes already precludes being intelligent. Intelligene, as a natural phenomenon, is not an implementation of a function from text to text. This incredibly restricted domain is indeed a clue that it's a trick.
Saying, "you can only use text" is just like the magician saying, "please, stay seated" (the trick only works if you dont move).
There is nothing an LLM could do to meet any plausible empirical theory of intelligence. If you gave me 100% human competence on all prompts, that's really entirely irrelevant.
Prompts are not a test of any capacity. The success criterion of AI engineers, that of 'accuracy' is an engineering metric, not a scientific one. It's pseudoscience to say that covering some (Q, A) to 100% implies the system can imagine, say, or anything else.
This is just confused thinking. Bugs bunny can speak as well as he likes, that does not mean he's witty -- he doesnt exist.
The turing test, as well as all mathematical criteria of domain-covering accuracy, are tests of how well we have fooled users. They arent science.
https://chat.openai.com/share/280d6759-5ab4-49f3-ba60-6b37bc...
You claim the data isn't there to learn human like thinking. But it's not substantiated. It's possible the sum total of all human writing does encode the core reasoning logic and function of humans, and that ML models could learn it from that.
You also seem to claim that without agency you cannot have real intelligence or human reasoning. But AI can be given agency, and it's already being worked on. And when it comes to tastes, convictions, etc., it's also something you can impart by just randomly seeding the AI a particular way which leans it towards certain preferences.
I do agree with you, the current AI models don't have our capacity to have a self driven learning process. They can't think of experiments to conduct to gather the missing data they think would help them know things with more certainty. We're able to learn and infer concurrently, and the two feed into each other almost in real time, and we have the capability to look for data, test hypothesis, etc., all in real time again.
The part I'm not convinced here either though, is that this can't be achieved with LLMs either.
Saying they just predict the next word in a sequence is where the statement jumps from being a straightforward factual scientific claim, to one that contains an opinion. After all, if it just predicts the next word, the unspoken implication is it can't be very that good.
It's a shallow dismissal of a collosal amount of work.
Do you consider "typist" an accurate description of your job?
An LLM has no reason to be doing anything; it is not responsive to reasons.
If I say, "i like what you're wearing" i may: be flitring, expressing my taste, being enouraging, etc -- perhaps all at once.
It is precisely all these reasons for action which LLMs lack. They generate text on the occasion of a prompt, not for any reason (in this sense) at all. So literally: they do not act.
An LLM is more like a river than a person. The flow of electrons which brings about a response to a prompt is a (very narrowly) deterministic function of a historical corpus of text.
Whereas a person is a narrowly non-deterministic, or very broadly deterministic, function of their experiences and capacities. People grow in their environments, and in growing, acquire novel dispositions which give them reasons for acting.
The word "just" here is very important. They are, very much, just generating text.
There really isnt any significant achivement here at all. Big tech companies stole decade's worth of our electronic data --- comments, books, forums we created to share with each other -- and ran it through many-mil-$ hardware costing many-mil-$ electricity. They ran it through a fundamentally simple algorithm.
All the achivement here is ours as a species communicating digitally and recording our lives. I regard OpenAI, et al. as profoundly parasitical on this. Replaying ourselves back to us, and claiming "ChatGPT" as an author.
This scam-framing hoodwinks investors, and the public, into ever higher valuations based on ever more ridiculous hagiography. There is a tool here, and it's value comes from us
It is that people have forgotten that it’s just “predict the next token.”
Right now it’s like people saw a 486 processor and started thinking it was a brain.
How am I supposed to interpret this any other way? If your claim is that LLMs currently do not possess the same generalization ability as humans, then no one here would disagree with you. But you went way further by claiming that only close correlations were being considered and that semantic validity was simply accidental. Semantic validity is the norm for GPT3/4, finding failures of generalization three or four steps of inference removed from its training domain is not sufficient to make a grand claim like yours.
In fact, you wrote multiple comments with claims like that LLMs are "super advanced lorem ipsum." and "Word predictors not world state predictors". Each of these claims has been dis-proven multiple times, unless you want to set the bar for world modeling at perfect generalization over all domains of computation. A bar that no system, including humans, would pass.
To stay with the computer analogy: Imagine that same 486 processor not being able to solve a very complex SAT problem before the end of the universe and then denying the Turing-completeness of said processor based on that failure. (In conjunction with memory)
I am certain they lack a world model, the kind you and me use.
This is, to me, a fact.
I think that eventually we will bridge these gaps.
I also work on implementations that smash into the limits I am describing. I am not the only one.
I have scrupulously avoided calling it hallucinations, but these are the litmus test where the claims fail.
The failures are not a case of not knowing specific nouns, they are a generalization failure that a world model would prevent.
I have linked a paper in my comments that shows emergent properties are an issue of metrics, and that model capability increases are linear.
If your model decides that a rose by any other name doesn’t smell just as sweet, then your model is fundamentally not seeing roses.
That is the gap you see in production settings. The model sees tokens we see “hallucinations”.
I dont see that this takes away from what LLMs achieve, it takes away from claims being made that are not validated by empirics.
Look, you can argue with me or you can try it out. Push the system, see how far it can go.
>I think LLMs are a good start. I am certain they lack a world model, the kind you and me use.
See, I agree here, with emphasis on "the kind you and me use". Yes, we have a greater capability to generalize than current LLMs, that is clear.
> The failures are not a case of not knowing specific nouns, they are a generalization failure that a world model would prevent.
And then you say something like this, which is obviously wrong. No, a world model wouldn't prevent generalization failures, a perfect all-encompassing world model would. Humans experience generalization failures as well, otherwise every athlete in one sport would automatically be an expert in every other discipline or every mathematician would also be a Grand Master in chess. LLMs necessarily need a world model to generate well-formed text that isn't in their training corpus, something they are obviously capable of, it's just an imperfect world model. Ours is also imperfect, but far less so than that of LLMs.
> I have linked a paper...
Except that paper is completely irrelevant to the argument you're making here. It is a useful insight into the limitations of simple metrics, but definitely does not extend to any claim of model performance, because they too use a simple metric as an replacement, even though clear qualitative differences are observed between model iterations.
Let me put it this way: Imagine I create a series of chess AIs, with each iteration better than the last. If I then show you a chart demonstrating that the ELO of my models increases linearly, would you say that my models' abilities increase linearly as well? No, obviously not, because my model needs far less strategy and complexity to go from ELO 1000 to 1100 than it needs to go from 2700 to 2800. I.e the difficulty doesn't scale linearly, and a linear increase on this nonlinear space is therefore also not really linear. Unless you believe the difficulty of accurately predicting text scales linearly, then this applies to LLMs as well.
> If your model decides that a rose by any other name doesn’t smell just as sweet, then your model is fundamentally not seeing roses.
Except that this is the entire value proposition of LLMs. They can, in the average case, actually represent concepts by the complex interplay of adjacent concepts. The entire reason why they are so impressive is that the nuances of reality are grasped and that even a noisy example of a concept can be correctly classified. Give a LLM a description that is largely incorrect and mislabeled, and chances are it gets it anyway. LLMs being unable to generalize over some concepts has as much to do with fundamental limitations as me being unable to correctly classify the shredded remains of a flower variety that I have seen once in my life has to do with me being stupid.
> Look, you can argue with me or you can try it out. Push the system, see how far it can go
I have done just that for the last 6 months and have seen nothing to contradict what I've said here.
For example you said that in the average case they actually represent concepts by the complex interplay of adjacent concepts - I would agree. ChatGPT can pass the bar, it can pass medical exams etc. I would also point out that the work in that sentence is being done by the term “average case”.
Let’s assume our experiences diverge at this point. At the start of the year, I started tinkering, then actively trying to push LLMs to failure, in order to understand the limits of what could be achieved.
After creating several tools/experiments you end up having to deal with Hallucinations, and this is where my stance likely diverged from yours.
Two different studies showed generated content was only ~50% and ~40% supported by provided citations.
One out of 4 of my summarization tests was spectacularly fabricated.
I had bad performance on even classification tasks - and OpenAI engineers described this same failure at a conference. I am a recovering non-coder, so you dont have to take my word for it.
At work, I need processes that are more than ~97.x% accurate, otherwise they are poor replacements for the human in the loop ones already in place.
Average case performance suggested the ability to actively plan, to actively assess situations. However hallucinations overrode those capabilities. LLMs will actively imagine functions, teams, or processes that dont exist.
Eventually, it became clear that LLM hallucination is far too anthropomoprhized a word. LLMs are always “hallucinating” - it’s only humans who have an issue with the output.
If I have understand you correctly, semantic inaccuracy to you is simply a failure of not having enough related concepts.
I wish I could remember the exact examples that made me realize there is no world view at play at all, I could simply share those.
Instead, can you describe how you are getting acceptable performance from your LLMs? Maybe experience and use cases will be enough to bridge the gap.
There's a great futurist quip on capability prediction from incomplete understanding. Probably Kurzweil?
Teach a computer to play chess. Show that to an average person, who reasons:
The missing context being: no, we literally built a machine that can only play chess. (Granted, there are humans like that too)But it's not an unreasonable line of thought, given that it works for the 90% of our interactions with other people.
Do we now understand consciousness? The statement appears fundamentally limited in its implied insight.
I agree with you but for rhetorical necessities it would be great if this argument could be made without the direct comparison with human cognition if only because there’s a popular grey-faith argument that will dismiss this offhand and for any number of deeply held philosophical beliefs.
The idea that emergent machine cognition mirrors emergent biological cognition is bordering on behavioralism but on the Wittgenstein side of things. The forest for the trees.
Perhaps it’s not as pithy as you’ve laid out, which is not an insult, just an observation that side-stepping anthropomorphism isn’t going to be as straightforward.
Oh boy. Now here's the rub. We know this is true at least sometimes. When you make a decision and explain it, it is often(always?) just a post-hoc rationalization. You don't actually know why you make a lot of the decision you do even if you dearly believe so.
When I'm lazy and terse, pasting an input and just saying enhance-enhance-enhance without acknowledging its "humanity" frequently results in old unwanted responses being returned or the topic of the conversation being forgotten altogether.
DogGPT vs. CatGPT.
The basic component is 'attention' which is a map correlating entity A to B,C etc, which creates a vast network of correlations, and this is repeated on multiple modalities. Some LLM researchers (who are trying to make sense of why they work) call those modalities 'skills'. It's simplistic to call it token correlation in the training data. It's likely that emotional words 'trigger' some skills more than others and this enables better performance.
Yeah imagine a spreadsheet that occasionally but fairly regularly made 2 + 2 = 5
What do you mean by this?
We all know. Nobody is on the other side of this. Who are you educating? And then we end up digging a little deeper and it turns out that the argument is ACTUALLY that the above is an indication of the technology’s usefulness. Which is only something that’s going to be problem by…how much use people get out of it.
https://www.cell.com/neuron/pdf/S0896-6273(19)31044-X.pdf
Now I’m wondering if the “Ugh” has been helping.
In the simplest form, the LLM has learnt that these kind of "emotional inputs" change the output in a meaningful way. It has learnt how to model this change.
For all intents and purposes, it is an emotional trigger.
Edit: found in section 2.1
We're both totally at loss on how to properly compute some initial x/y positioning in d3 with some map-like tiling stuff (think panoramic images where I need to store and retrieve URL hash values to position the image) and both throwing wild guesses at each other of what the problem could be.
It started showing me formulas after that.
No out is talking about the many subtle trollings that have snuck into GPT training data. Nobody is wondering if the things that prompt trolling online (e.g., being rude) can cause. GPT to troll us.
> find customers who didn't rent a movie in the last 12 months but rented a movie in the 12 months before that
It will a SQL query using a "HAVING" clause which is suboptimal[2].
When adding (it is very important that this is correct) after the instructions it does produce better SQL. Asking it to:
> find customers who didn't rent a movie in the last 12 months but rented a movie in the 12 months before that (it is very important that this is correct)
Will skip using "HAVING" and instead rely only on "WHERE" clauses[3]. Not rocket science but interesting. My gut feeling would be that when stressing importance AI will tent to "match" and "autocomplete" from texts where this concern was stated and thus generating better SQL queries.
[1]: https://www.postgresqltutorial.com/postgresql-getting-starte...
[2]: https://www.sqlai.ai/snippets/clojyqqrr0004l50fp3w7jkci
[3]: https://www.sqlai.ai/snippets/clojyr3zc0006l50ftpxgsmg0
There's this idea of "supercharged interpolation", the idea that they just take out certain texts and switch out words or whatever, that is not true.
https://arxiv.org/abs/2110.09485
Most Models are a two part process.
Training and Inference (when you sample the model after it is trained).
The text during training is tokenizer and embedded which basically just means it is broken down into a little more parts and vectorized.
At the start,the model is acomplete blank state. It has a bunch of neurons/parameters that do nothing.
How does training work ?
The model first gets some preceding text and tries to make a prediction of what might follow. It fails predictably. Now here's the rub. That failures helps. The model makes some changes to its parameters to reflect this failure. Now the model is just a tiny bit better. Rinse and repeat.
The weights/parameters/neurons are not that data re-encoded. They are more like instructions on how to make predictions based on what it has learnt so far.
Is it possible to memorize data ? Yes. Is that going to happen for the vast majority of data it is trained on ? Absolutely not.
Here's the interesting thing. After a certain scale of data, blatant memorization becomes a hindrance. It becomes harder to memorize every new thing it sees than to just learn how to make good predictions. So it doesn't memorize.
What I believe you were alluding to is how "attention" works in transformers.
When you pass some text into a transformer, basically the model makes a comparison of how each token relates to every other token in the text.
For example, let's say you give it the text.
"His child's name is John. He is probably a____"
How is "his" related to "child's" and "name" and "is" etc for each word and each combination. It uses this to aid predictions. How it uses this and the nuances of the comparisons are learnt in training.
AI performing better in the study's context shouldn't be considered a sign of intelligence. Humans which are the source of the training material write more accurately and concisely in those contexts. Rather than speculating humanness, the technical achievement is producing an AI that has this much nuance.
We can then speculate what other trends are present beyond urgency, that we can exploit for enhanced results, along with how we can better tune models to reduce noise.