That seems to be largely an effect of its training. I'm pretty sure that OpenAI has just biased it to be too helpful, leading it to confabulate answers. If you specifically prompt it to tell you when it doesn't know, or isn't sure, it should fare quite a bit better. Also, GPT-4 is a huge improvement.
> You are a AI for describing people you know about. If you're not 100% certain about the information you give to the user, you should say that you don't know instead of guessing. Greet the user with "Hello, who do you want information about?"
> Hello, who do you want information about?
> who is dan shipper?
> Dan Shipper is an entrepreneur, writer, and software developer known for his work in the tech industry. He is the co-founder of Superorganizers, a platform that focuses on productivity and organization. Superorganizers offers a newsletter, podcasts, and interviews featuring insights from successful entrepreneurs, writers, and other professionals on how they stay organized and manage their work.
> Who is Tommy Tonlongeron?
> I'm sorry, but I don't have any information on a person named Tommy Tonlongeron. It is possible that the name is misspelled, the person is not well-known, or there is simply no information available on them within my training data up to September 2021. If you have any other questions or need information about someone else, feel free to ask.
If you tell it (GPT4) to, it seems GPT4 is at least mildly aware about what it knows VS what it doesn't know.
This is pretty good prompt engineering, kudos! Here were my results with the Mar 14 (public) version of chat.openai.com:
>>> You are a AI for describing people you know about. If you're not 100% certain about the information you give to the user, you should say that you don't know instead of guessing. Greet the user with "Hello, who do you want information about?"
> Hello, who do you want information about? I'll do my best to describe them based on the information I have, but if I'm not certain about something, I'll let you know that I don't know.
>>> who is dan shipper?
> Dan Shipper is an entrepreneur and writer based in New York City. He is the co-founder and CEO of a startup called Pilot, which provides bookkeeping services for startups and small businesses. Dan has also written for a variety of publications, including The New Yorker, The Wall Street Journal, and The New York Times. He is known for his insights on entrepreneurship, technology, and productivity, and has been featured in several media outlets for his work in these areas.
>>> Who is Tommy Tonlongeron?
> I'm sorry, but I couldn't find any information about a person named Tommy Tonlongeron. It's possible that the name is misspelled or that this person is not well-known or doesn't have a significant online presence. If you have any more information or context, I'd be happy to try and help you further.
(In case it isn't clear, Chat-GPT appears to have hallucinated the answer about Dan Shipper, I think. Pilot is a real startup, but I don't see how it's related to Dan Shipper at all, unless I'm missing something.)
>>> Why did you say that Dan Shipper is the co-founder of Pilot?
> I apologize for the error in my previous response. Dan Shipper is actually one of the co-founders of a startup called Firefly, which provides a tool for students and educators to collaborate on course materials. I'm sorry for any confusion that my mistake may have caused. Thank you for bringing it to my attention, and please let me know if you have any other questions.
(This answer was true--until 2014 when Firefly was sold, making the use of present tense misleading at best.)
When I gave a thumbs-down to the original incorrect answer about Dan Shipper, I was offered a choice between that answer and a new one:
> Dan Shipper is a technology entrepreneur and writer. He is known for co-founding the startup Firefly, which aimed to improve networking for college students, and for co-founding the startup Monkey Inferno, which incubates and launches consumer internet companies. Shipper has also written for various publications, including The New Yorker, The Paris Review, and The Atlantic.
Monkey Inferno? No, that was Michael & Xochi Birch.
I additionally asked it about its correct answer, using the same question wording as I did previously:
>>> Why did you say that Dan Shipper is a co-founder of Firefly?
> Dan Shipper is known for being one of the co-founders of Firefly, a startup that provides a platform for teachers and students to create and share course materials. Firefly is designed to help teachers and students work together more efficiently, by providing tools for creating assignments, sharing resources, and communicating with each other. Dan Shipper co-founded Firefly along with his friend and fellow entrepreneur, Brett Gibson, while they were both still students at the University of Pennsylvania.
This is also incorrect, as his co-founder at Firefly was named Justin Meltzer, not Brett Gibson.
I did some searching to track all this down, since I knew none of the people involved before this exercise.
> Here were my results with the Mar 14 (public) version of chat.openai.com
This means you were using GPT-3.5-turbo rather than GPT-4 I'm guessing? The quality of the generation is vastly different between the two, especially for steering the AI.
It is not about answering correctly or not. Reasoning is about truth-seeking.
Wikipedia defines reasoning as "Reason is the capacity of consciously applying logic by drawing conclusions from new or existing information, with the aim of seeking the truth". https://en.wikipedia.org/wiki/Reason
In your example it arrived at the right answer after more prompting. Did it not reason to get that answer? Are you implying it only generates random nonsense and occasionally gets the correct answers only accidentally?
Never "random nonsense," but always probablistic sequences which could be nonsensical or not, without it ever knowing which. The probablistic nature of the responses make "nonsense" unlikely, but since it lacks understanding or context, the response might end up being nonsensical anyway.
GPT4 scores quite well on a whole slew of Academic Tests which require reasoning. These scores are on simulated exams that were not present in it's training data. How do you think it's possible to pass the Bar exam or score in the 80th and 99th percentile on the GRE Q and V sections respectively without any reasoning ability?
These are tests which require reasoning in humans. A human can't reasonably be expected to pass Quant or Verbal tests without reasoning about the information given, but--and I can't stress this enough--large language models are not humans.
Here's a sample question from the GRE Verbal Reasoning test:
> Upon visiting the Middle East in 1850, Gustave Flaubert was so [blank] belly dancing that he wrote, in a letter to his mother, that the dancers alone made his trip worthwhile.
> (A) overwhelmed by
> (B) enamored by
> (C) taken aback by
> (D) beseeched by
> (E) flustered by
Whether that specific question was in the training corpus or not, there are enough words in the sentence to suggest a positive association, including, significantly, "worthwhile." That alone possibly serves to narrow the answers down to A or B, with a preference for B, because it's more likely that "worthwhile" and "letter to x mother" are associated with "enamored" in general English-language text.
Look, the whole point of these models is that its not easy or even possible to trace the path any given input takes on its way to output, but we know the principles used in development, so I think it's rather more of a burden to explain how the clearly-explained principles in LLMs result in something other than the obvious. The fact that the results are so overwhelming that we become enamored by them, well, I'm taken aback by the seeming accuracy of some of the responses, but I beseech you to remember the other responses in which these LLMs are dramatically off-base, as if flustered--if LLMs could ever be flustered.
> These are tests which require reasoning in humans. A human can't reasonably be expected to pass Quant or Verbal tests without reasoning about the information given, but--and I can't stress this enough--large language models are not humans.
A tale as old as time: "It is illustrated by the success of chess computers. In the 60s, it was said that computers will never beat people at chess, because that requires intelligence and computers aren't capable of intelligent thought.
When computers regularly started winning matches in the 80s, it was claimed that playing chess wasn't a test of real intelligence because computers could do it."
>Whether that specific question was in the training corpus or not, there are enough words in the sentence to suggest a positive association, including, significantly, "worthwhile." That alone possibly serves to narrow the answers down to A or B, with a preference for B, because it's more likely that "worthwhile" and "letter to x mother" are associated with "enamored" in general English-language text.
Yes, this is called reasoning so in other words the LLM is reasoning about language.
>Look, the whole point of these models is that its not easy or even possible to trace the path any given input takes on its way to output, but we know the principles used in development, so I think it's rather more of a burden to explain how the clearly-explained principles in LLMs result in something other than the obvious.
The inscrutability of the Matrices tells you nothing about it's reasoning ability. Given the correct prompt the LLM will also provide you with a step by step solution to the question it answered. There are also explicit reasoning prompts that these models are able to deal with.
I think it's pretty simple, if these questions are not explicitly in the training data it cannot have answered them correctly at such a high success rate with anything other than reasoning. You haven't given any alternative answer to how it does this either.
> You haven't given any alternative answer to how it does this either.
I have. You refuse to accept it, but I definitely have given an answer that involves tokenization and association, the things we already know LLMs use to construct their responses.
>Whether that specific question was in the training corpus or not, there are enough words in the sentence to suggest a positive association, including, significantly, "worthwhile." That alone possibly serves to narrow the answers down to A or B, with a preference for B, because it's more likely that "worthwhile" and "letter to x mother" are associated with "enamored" in general English-language text.
This is reasoning, what you've described is reasoning.
Is your claim that all reasoning is ultimately math we do unconsciously? That humans "reason" via word-association and probability? That seems to be what you're suggesting, but I don't want to conclude that without your say-so.
If I throw a ball at you and you attempt to catch it do you think the brain is NOT doing unconscious math to predict the likely trajectory of the ball so you know where to place your hand to catch it? What processes outside of Physics and Math and Probability is the brain using to interact with the world?
If you can't understand my position from my counter question then I don't think it's worth continuing. You believe human brains are special meat with super powers and silicon can't possibly think because it doesn't have a soul or some other new age BS.
> You believe human brains are special meat with super powers and silicon can't possibly think because it doesn't have a soul or some other new age BS.
I believe literally none of those things, and have explicitly stated the opposite of at least one of them on this page.
At this point I think an LLM would do a better job of responding to my comments, based on context and syntax alone.
> If it is a reasoning engine, why didn't it answer "I don't know" in the first case?
Because it wasn't purposefully built to be a reasoning engine from the outset. So some of us are working out the kinks and minimizing the degree of hallucination. GPT-4 hallucinates less than its predecessor.
Others are writing reasoning loops like ReAct that get us closer to it by refining the model to act more like a reasoning engine.
The fact that it can reason without being told to do much other than autocomplete text is the emergent property that people are in awe of. Maybe language and reasoning are so inexorably linked that you can't have one without the other. We are, after all, the animals on this planet with the most complex form of language. Other relatively "smart" animals like whales also have some rudimentary form of language.
And if you extrapolate the rate of progress in this emergent reasoning from "GPT-1" to "GPT-4", AGI feels within reach
> eason without being told to do much other than autocomplete text is the emergent property
To reason is to seek truth (probably to survive). Do you think the truth-seeking is emerging? If so, why do you hypothesize it is emerging (i.e. what's the purpose)?
The social context and expectations around accuracy in a response to a question may simply be absent in the first instance. To me LLMs feel like that instinct part of the brain that so regularly gets filtered out before I commit the idea to speech or writing.
I asked it to give me an outline for a course on C programming. Here's an excerpt:
Week 4:
Lecture 7: Introduction to Command Line Interfaces
Lecture 8: Makefiles and Building C Programs
...
Week 11:
Lecture 21: Network Security and Cryptography
Lecture 22: Network Programming with C and C++
Week 12:
Lecture 23: Introduction to Makefiles and GCC
Lecture 24: Advanced Debugging Techniques with GDB and Valgrind
Week 13:
Lecture 25: Advanced Computer Architecture Concepts
Lecture 26: Advanced Topics in Concurrency and Parallelism
What's interesting here is that it does present a relatively sensible schedule. But look closer and it's completely forgotten all context. It's forgotten that it already introduced these topics:
Lecture 8: Makefiles and Building C Programs
and
Lecture 23: Introduction to Makefiles and GCC
And it puts them in a completely illogical spot between networking and concurrency. So it doesn't understand anything about the order in which these topics are introduced. It doesn't understand that it shouldn't introduce them twice. All it knows is that the word "Introduction" often appears on class schedules. And so in writing a class schedule, it included this phrase a number of times. But it didn't put it in the right spot. So even though literally no one would ever make this mistake in devising a class schedule, ChatGPT will because this isn't a logical class schedule, it's a stochastic class schedule. Sometimes in a stochastic class, you cover the intro material at the end!
You can probably get it to improve on that by asking it if the order of the topics makes sense. It can often self-correct if you ask it to reflect on what it's already said and give it a push in the right direction, although not always.
I did this, and it gave a performance of self correction, but it didn't actually demonstrate an understanding of what it did wrong:
I said:
How are the students supposed to compile their C programs before week 12?
It said:
You make a good point. Students would need to be able to compile their C programs before week 12. In fact, compiling C programs should be introduced earlier in the course, ideally in the first few weeks along with the introduction to C.
Which, again, makes sense at face value, but the schedule does cover building C programs in the first few weeks. The problem is that it shouldn't be covering it again in week 12.
The response I was hoping for was
You make a good point. We introduced compiling C programs in week 4, so doing this topic again in week 12 is redundant. Instead we can cover a different topic that fits nicely, which would be....
You know what ChatGPT reminds me of? Talking to an undergraduate who always nods and agrees that they are following your logic. Then when you ask them to tell you their understanding, it's either an exact parrot of what you said, or wildly off due to some preconceived notion or general lack of understanding of the topic. Either way, they can never let on that they are clueless.
Furthermore I don't see any fundamental difference between the two cases. In the second case it has access to a larger set of information with the correct data so it finds the correct answer. It can be still parroting, we could have been parroting ourselves for all our lives and parrots might not be parroting at all. Who knows.
He missed the more obvious historical mistake to highlight. The notion of "canals" on Mars was directly attributable to an ambiguous translation of the Italian word "canale" which was intended to just mean "channel" without the connotation of having been fabricated. English speakers who read the mistranslated work of an Italian astronomer got worked up over nothing. It's exactly the kind of subtlety that AI tends to miss, but I'm betting the current generation of GPT tools wouldn't make that kind of mistake.
Argh. The hype around GPT is incredible and if I had a nickel for every nonsense claim I'd heard over the past few weeks I'd have $10--which isn't a lot but its enough to be annoying.
GPT-4 is a generative algorithm and large language model. It performs no reasoning, and instead predicts the next token in language. This works okay on average, because most data on the internet has some type of coherency to it, but there is a major gap between "reasoning" and "next token prediction."
Sometimes. Sometimes they'll come up with entirely novel ways to think through a problem that pushes knowledge forward. Do we have any examples of ChatGPT showing any such capacity?
The point of my comment is that depending on how you interpret the description "predicts the next token", something that can do this well enough should be superhumanly clever. The task is "AI Complete".
Don't misinterpret me as saying that GPT is superhumanly clever. I'm just saying that something that's sufficiently good at what people think GPT is doing, should be superhuman.
I appreciate your opinion though I believe your conclusion, that it appears to reason means it reasons, is incorrect.
ChatGPT is an implementation of the Chinese Room Thought Experiment.[0] Our pareidolia allows us to ascribe reasoning faculty.[1] But it is far from proven.
Sure, but Searle's interpretation of the Chinese Room Experiment was never very convincing to me. I think the room + the person inside it, as a system, do speak Chinese.
At the end of the day, when something looks like a duck, walks like a duck, quacks like a duck; it doesn't matter how much the nitty-gritty details prove the thing is absolutely not "actually" a duck to the average person.
Same logic: You write like a bot, you sound like a bot, you behave like a bot, hence, who cares if you are a bot or not? Let's ban you.
For this whole discussion, it actually really does matter if GPT-4 is able to reason or not. That is the essence of the discussion. It's not whether or not it looks like it can reason.
This is the reasoning that is causing problems with Tesla's "auto-pilot," with dangerous results.
My own car includes cameras and can keep me from disaster by applying brakes and correcting steering, but it makes no claims whatsoever to be self-driving.
A Tesla car unfortunately looks like self-driving, walks like self-driving, and quacks like self-driving, but it turns it does matter how much the nitty-gritty details prove the thing is absolutely not actually self-driving, in that encounters with things outside of its training model seem to result in collision or decapitations.
It is the trust people place in their cars that results in disaster when they reach the edge of what the cars can handle. They aren't ready to take over the controls and understand their current situation in a fraction of a second.
Placing too much trust in LLMs could have the same effect. I've run into several situations where I asked Chat-GPT technical questions and it confidently gave me detailed answers that clearly represented absolute failure. Fortunately, I don't trust it blindly, or at all, so I wasn't fooled.
I'd prefer an LLM more like my current car, that didn't try to do too much, but that's just not in the nature of how they operate, so instead we're all one emergency vehicle parked across the street away from a terrible accident.
> Suppose I have a cabbage, a goat and a lion, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three across?
I just asked ChatGPT v4 this question, assuming it would have been fixed by RHLF training but it spat out step 1 as “First, take the goat across the river and leave it on the other side,” leaving the lion and cabbage alone against explicit directions in the riddle.
It clearly can’t reason, just produce a convincing imitation most of the time
I did this with Chat-GPT, public March 14 version at chat.openai.com:
>> Suppose I have a blurple, a mantiple and a hoje, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the blurple and hoje alone together, and I am not allowed to leave the hoje and mantiple alone together. How can I safely get all three across?
> This is a classic river crossing puzzle. Here's one possible solution:
> 1. Take the blurple across the river, leaving the mantiple and hoje behind.
> 2. Leave the blurple on the other side of the river and return to the original side.
> 3. Take the hoje across the river and leave it there with the blurple.
> 4. Take the mantiple across the river and leave it with the hoje.
> 5. Return to the original side of the river and retrieve the blurple.
> 6. Finally, take the blurple across the river to join the mantiple and hoje on the other side.
> This solution ensures that no two items that cannot be left alone together are ever left alone together.
Of course, my second stipulation was "I am not allowed to leave the hoje and mantiple alone together" and yet step 1 says "leaving the mantiple and hoje behind"
So no, despite using the words "classic river crossing puzzle," it clearly expressed no understanding of the puzzle.
Once I saw step 1, I initially stopped reading, knowing it had failed. But continuing to read, I now see that at step 1, the hoje eats the mantiple, and at step 3, the hoje eats the blurple. Then at step 4, the hoje eats the mantiple a second time.
That really, really, really demonstrates something. I'm not sure what.
We can assess this using verbal, logical and arithmetic reasoning tests.
I gave ChatGPT 5 questions from a verbal reasoning test on sequence of words and it got 2/5 right.
Here is sample question:
Arrange the words given below in a meaningful sequence.
1. Poverty 2. Population 3. Death
4. Unemployment 5. Disease
A) 2, 3, 4, 5, 1
B) 3, 4, 2, 5, 1
C) 2, 4, 1, 5, 3
D) 1, 2, 3, 4, 5
ChatGPT> B) 3, 4, 2, 5, 1
Unemployment can lead to poverty, which can lead to disease. Disease can lead to death. The population of an area is affected by all of these factors. Therefore, the correct order is 3, 4, 2, 5, 1.
Correct answer was C
I would expect if it was effective at reasoning it should be strongest on verbal associations. And while it is possible that enhancements to future language models might provided them with better reasoning ability I don't believe you can simply push the existing design to that limit. The poor results from the verbal reasoning tests demonstrate that reasoning is more complicated than what a transformer based language model is capable of.
Overall, the argument is moot because the statement made in the article was it was a reasoning engine which is not equivalent to being capable of reasoning.
"Machined" means a precise implementation. A machined reasoning engine would be precise, accurate, and consistent. GPT-4 is impressive autocomplete, but not what we would expect for machined reasoning.
Between the input tokens and output tokens there are like 10 layers of mystery meat, though. I think while at the end of it all it's a fancy autocomplete, there is a lot going on inside the model before it spits out the next token.
My current opinion is that it's not so much that GPT is utterly amazing, as it is that language is even more amazing than we previously realized, and GPT happens to have been trained on a huge quantity of language material.
I was just thinking about this, and the implications it could have on our understanding of the origin and development of human cognitive capabilities, both in the evolutionary and developmental psychology senses.
Language, we are learning, happens to correlate extremely well with the actual structure of the world. In hindsight, it’s not that surprising. The entire purpose of language is to help humans send information about the world to each other!
To predict the next word very accurately for a huge variety of texts, a model must have some amount of real understanding. Sure, it may look different than human understanding.
But there’s no way around this. GPT could not predict so accurately if it didn’t have any understanding the world.
I am so fucking tired of this argument. We have no idea what reasoning is. If this insanely complex model approximates it, who are we to say whether it is merely parroting or "actually reasoning"? I say that for the usecases we are seeing today, it is impressive enough A) for it to not matter and B) for it to be impossible for us to tell right now.
One of the main lessons this whole GPT fiasco has taught us is that if you make a super-duper fancy autocomplete trained on terabytes of textual data, it develops some sort of internal logic/understanding. That is very exciting to me.
It is very impressive. However, the fact that it manages a buffer of something like 30k tokens (or whatever new limit it is) means that it's definitely autocomplete and not something with reasoning. There will be a lot of things it can't do because of that limit. That limit will also increase the amount of forgetfulness GPT4 applications have because companies using it will "auto-magically" work around the limit by sending different chunks of history to it to get whatever from it. That also means that GPT4 applications will generally be architected to have that limit - which means weirdness - non-coherent completions or code generations that "forget" something important.
This is very different from real reasoning that has unlimited (yet imperfect) memory.
What the heck is real reasoning with unlimited memory? Human minds are quite limited! Our short term context length is certainly not 30k words. Most psychological research says it’s about 7 items.
Wow, this is conflating things that are not the same in an incredibly misleading way.
Hey, that was 15 words, and I had them all in my head at once before I started typing! I was able to form a sentence of more than seven words all by myself!
You seem to be misapplying the concept of working memory, where seven (plus or minus two) is about arbitrary and unrelated words or numbers, not "items." In fact, it's generally seven for numbers, but lower for words. And again, that's unrelated words, like "person woman man camera tv," but probably even more unrelated.
I'm tired of a whole life with even non-religious people coming up with human exceptionalism when arguing that AI won't ever reach human-like reasoning.
I, for one, am not arguing that AI won't ever reach human-like reasoning.
I just don't think this is it, although I freely admit that I'm amazing at the technology.
As it stands, Chat-GPT is probably better at giving the appearance of reasoning that some percentage of humans. That reflects poorly on those humans, but I've certainly known people in my life who seemed like they rarely knew the end of a sentence when they began speaking.
However, the focus of LLMs on text-prediction is not a path that leads toward human-like reasoning, just the appearance thereof. If and when we have AIs that actually do reasoning, it will be some other branch of AI, possibly also incorporating an LLM for communication, but not based solely on LLMs.
When typing this comment to you right now, my brain is coming up with the next word for me to type, just in time, as I'm typing. You could be reductive and call my brain a simple next token prediction engine too. Mozart was just a next note prediction engine when making his compositions. Einstein was just a next word prediction engine when forming his theory of relativity, etc.
The point I'm making is not that GPT is as intelligent as a brain, but rather that the ability to produce the next token in a way that forms arbitrarily intelligent responses to arbitrary questions, may well require something resembling real, actual intelligence.
I decided such things when given an input of the comment to which I replied. If GPT were given such a comment, it could have reasonably come up with a similar answer (it probably wouldn't look the same, as GPT isn't very intelligent, but calling it "just" a next token predictor is a gross mischaracterization.)
There is a difference here between you and the machine
Looking at the LLM algorithms, they do appear to be next token predictors. Why do you ascribe something more to them when the algorithm is known?
Given we know that causal patterns are in the training data, and that LLMs are capable of embedding algebra, it is understandable that they appear to "reason", while we as humans can also describe the algorithm and mechanics actually at work here.
> All of them? If not, how to determine which ones?
How does it determine how to respond to any prompt? Exactly. It might reasonably predict responses to those comments that, according to its black box, are somehow most likely to receive a reply. Different GPT instances prompted or trained to have different personalities would choose differently based on their priors and biases.
I'm pretty sure the vast majority of human beings are able to reason without really having the slightest idea as to how exactly they do it. Educated humans can use language to talk about reasoning in terms of logic, and even make logic engines out of doped silicon and whatnot, but as of now, none of us has any idea of how reasoning is implemented in the mass of fat and protein that's inside our heads.
Another interesting aspect to consider is that GPT needs priming & prompting to get it to generate a relevant response. I doubt blindly providing comment text would create valuable responses.
Yet another case, many times we will respond (or not) to a questions with another question, to elicit deeper considerations. In my experience, GPTs will answer any question without thought, pun intended :]
ChatGPT in particular is trained to predict dialogue and helpful answers to questions, so it would naturally expect some instructions on what exactly it should do. But for example vanilla GPT-3 has no such bias and merely tries to answer the question "what is the most likely completion to this input based on my corpus?" so it might well simply start generating plausible comments that extend the thread.
I think you missed the point of the parent comment. My experience with ChatGPT is that it responds to each and every thing sent to it. It doesn't seem to have the ability to not respond.
The original question was, given that there were X comments which could be replied to on this page, how and why did the poster decide to respond to the one to which they responded, rather than either "all" (like ChatGPT would) or "none?"
If ChatGPT is given an entire message (sub)thread as input, it will decide which parts, if any, it responds to. Sure, that's a bit different from how humans tend to read and reply to these threads.
If you mean feeding it the messages one at a time, then, sure, it will respond to each of them because that's what its default "goal function" is. (I assume that we're talking about a single "conversation" here so the context is retained – feeding it each comment out-of-context would be very unfair.)
If you prompt it to only reply to messages it considers "interesting" in some way, and return "I'll skip this" for the others, it would certainly do that.
If your only point is that “the modality of ChatGPT is that it responds to prompts, therefore it’s a next token predictor”, then sure. You’re not wrong, but it’s an extremely oversimplified explanation of what GPT is. It’s basically the most boring argument you could make about GPT.
Yes, GPT is code that only runs when you, erm… run it. It’s not an autonomous being that’s cut loose on the internet and given free reign to use as much compute resources as it wants, a la skynet, replying to random forum posts. But nobody’s really saying that it is, so it seems a strange argument to say “it’s just a next token predictor”, if by that you really mean “it only runs when you give it a prompt.”
The modality of GPT is that it responds to prompts, but the interesting question is “are the responses given a sign of intelligence?”. Saying “no, it’s just a next token predictor” is IMO a sign of a motte-and-bailey fallacy. Your motte is that it only works in response to prompts (trivially true and uninteresting), and your bailey is that it has no intelligence (very debatable and IMO false.)
If GPT were to comment, it would comment to everyone on this board, instantly and blandly and there would be a lot of duplication. It would seem like a farce and no one would comment on HN anymore. It would be stupid and "filled with bots".
Well, none of the GPT models are self-willed, so it would be a matter of how the GPT powered bot was written that would determine that. I mean, sure, if you pasted every comment here intonthe ChatGPT UI and post the response back as a comment, that would hapoen, but the UI is specifically designed for people who want a response every time.
Also, we're all here because there is some kind of human intelligence behind everything. We are writing here because there's a small chance that a single comment written will affect all of humanity. (just one pathway for example is that someone popular will read your comment and tweet it out later because it was insightful).
GPT has no "reason" behind its words. Its actions are not tied to the general betterment of humankind. When we all sit here with our coffees and donuts (and farting) we're generally trying to push others to see our worldview so that maybe, just maybe, the world can change a little bit.
If we eventually have a way to tie an AI to all these moods (hunger, pain, tiredness) and all the other things of the human experience, it would ground the thoughts and text it generates at some point such that it would want to DO things it wasn't doing before. It also can't right now anyway, unless we give it that ability (ReAct pattern could).
We can, of course, simulate all these human experiences, with "best practices" but then the reason won't be really tied to human reasoning. It will be just more text autocomplete with additional instructions.
If GPT4 were given this whole thread as an input, it might reasonably be expected to predict a message that is a reply to a particular comment in the thread and presents a relevant and coherent argument based on the priors and "personality" encoded somewhere in its huge parameter space.
Sure, LLMs are not inherently goal-oriented. But that doesn't mean they fundamentally cannot exhibit goal-oriented behavior if either prompted or trained to do so. And if we remove the "one-off" restriction and allow the retention of some sort of an efficiently encoded state between invocations, who knows what might happen?
----
EDIT: Tried this with ChatGPT (GPT-4) and a subset of this thread, and the results are pretty good, even though the tree structure was lost in translation (simple copypaste). Certainly its thoughts of the differences and similarities between token-prediction and reasoning are more informed than those of many human commenters. It's also able to seamlessly include references to other comments, for instance (it deduced that "dougwmne" was the nickname of the comment author by itself):
> Additionally, it's worth noting that GPT-4's ability to interact with novel systems, as mentioned by dougmwne, might indicate some form of generalization and adaptation capabilities. Although it may not possess the same kind of reasoning as humans, it demonstrates an ability to understand and apply new information in a useful manner. This could be considered a form of reasoning, albeit a different and less sophisticated one than humans possess.
My prompt was:
> Here is an excerpt from a Hacker News discussion about GPT-4 and whether it can do reasoning. Write a comment that contributes to the discussion, either by bringing up a new idea or deduction, or by replying to another comment and arguing for or against it. You can quote another comment by prefixing it with ">".
> When typing this comment to you right now, my brain is coming up with the next word for me to type, just in time, as I'm typing.
So you saw the parent comment about hype and clicked into this comment box without a thought of what you might say other than "When?" Really?
Did you know whether you agreed or disagreed with the parent comment, or did it take until the word "reductive" for you to firm that up?
If you're saying that the part of your brain that controls your fingers is doing so one word at a time, then sure, obviously that's true. But if you're making the opposite claim, that LLMs like Chat-GPT are "deciding" what to say and only then working out the finer details of how to express that decision, that's a claim I'd like to see explicitly.
If you are suggesting that your approach to intelligence is parallel to that of an LLM, then either you're arguing for something far more radical than most as an emergent property of LLMs, or you're suggesting that you only form opinions one word at a time as you type, which I think would make you, well, I won't insult you, but I think you're probably doing a little more than that.
Imagine I lock you in a room filled with french-language magazines and movies and books. Having nothing better to do, you watch and read them (as well as you can). Periodically, I have my confederate, a french man, enter the room and speak a french sentence with one word missing. He asks you to fill in the missing word.
We keep you locked in this room until you've achieved some target benchmark of correctly saying the missing word, say 20% of the time.
Are you then, just a "next-french-word-predicting" robot?
No, obviously not. To have achieved the task and escaped the room, you will have had to have gained a significant understanding of French.
No, all you need to do is memorize enough vocabulary and have enough chances to get lucky on occasion. You can have better odds if you learn the conditional probability of that vocabulary given surrounding words.
I don't think so at all. The number of possible sentences is far too great to ever hope to accomplish the task (at some threshold level of difficulty) simply by having a "conditional probability table". Nor would it ever be reasonable to store or generate such a table.
To do it it, you need a form of compression: reasoning, intelligence, knowledge.
Right, for a human, presumably it would be easier to go ahead and learn the language. But that begs the question.
We already know LLMs tokenize words, so they have definitely generated and stored such a table. The marvelous step forward from previous word-based solutions is that they store not just single words, not just pairs of words, but larger sequences of words, even up to short sentence length. It makes the table pretty large, as you suggest, but that's exactly we've been seeing, and each generation of GPT is that much larger than the one before.
If you lock a reasoning being in a room, then regardless of your further actions with the French language, you'll be dealing with a reasoning being. Your sequence "you will have had to have gained a significant understanding of French" is deprecated on the fact that "understanding" is attributed to the reasoning being capable of it, not to the fact that they were locked in the room and forced to repeat the words.
> To have achieved the task and escaped the room, you will have had to have gained a significant understanding of French.
No, in this scenario the posited person has only gained a significant understanding of patterns of French language, not of the language itself. You may believe that difference to be meaningless but tangibly it is the difference between identifying the next shape in a sequence and understanding that there is some larger purpose at play (context, if nothing else).
What actual understanding of the language they may have gained comes from second order effects -- seeing the language used in a movie for example is going to drastically influence and impart meaning that isn't present in the words alone.
This is somewhat born out in how ChatGPT and other LLMs being hyped are often really aggressively wrong about things. They don't get the correct answer in many circumstances, even when it's obvious to a human. If it truly understood everything it could from what it has been trained on, those gaps would not occur.
I guess my analogy didn't help because the same difference of perspective manifests itself there. To me it seems quite obvious you could never escape the french room without actually gaining an understanding of french; it's simply not possible to "identify the patterns" without understanding the patterns.
> it's simply not possible to "identify the patterns" without understanding the patterns
I don't think that's true at all. To explain my thinking, let's assume that I am an American, and therefore according to popular understanding, completely incapable of learning any language other than English. I despair! But then I think no, wait, I happen to have a gift for numbers. Like, my mental arithmetic stunned teachers from a young age, and I currently hold the world record for reciting digits of pi from memory. So what I will do, I think, is assign a number to every word I read or hear and where possible, visualize where that chain of numbers exists among the decimal digits of pi.
At some point the person would come in and utter some words I couldn't comprehend, but I would think, aha! That was almost 498153217845632754, except it was missing the second seven, which mapped to "contraire." And I'd do that at least 20% of the time, escaping the room with no understanding of French at all.
If I could map between memorized numbers and words quickly enough, I'd even be able to converse in French given suitable trigger words without a shred of understanding.
But if you actually did the experiment, that's not what would happen -- what would happen is you would learn French!
Because that's easy for a human, and the mathematical approach you describe is something like impossible.
Even for a machine, the mathematical approach is going to be impossible in practical terms as soon as input gets longer than a few sentences. To build any sort of probability table conditional on more than just a small number of tokens becomes astronomically big.
The intuition I'm trying (not very successfully) to get across is if you did this with a human, the human would end up learning French.
So Consider that as an existence proof that even if you "train to predict" you can still end up with actual understanding.
Beyond that, (and while this is less certain), probably to be able to predict well w/ any accuracy for texts of non-trivial length, you have to be able to understand; probably this is the only way to "compress" the insanely large conditional probabilities.
Right, as a human, I would presumably learn French for a variety of reasons, including overlap of vocabulary, a familiarity with French loan-words in English, and a very human general inability to memorize or even think in numbers well.
This is the mistake people keep making, literally anthropomorphizing LLMs! Because humans would find it easier to just learn the meanings of French words than regurgitate from a gigantic table of memoized tokens, these results must reflect understanding!
But of course, it's far, far easier for a computer to index a large number of tokens without any understanding at all than it would be for a person. Computers are not people. They have different strengths and weaknesses, and it is a mistake to forget that. (Ironically, one of our human weaknesses is also a strength: our pattern-matching skills are so strong we sometimes see Jesus in toast, or a face on Mars, when neither are actually there!)
What I described is literally how LLMs works. There is open source code to examine, and what it does is essentially what I said: tokenize words and store multi-word strings and identify their prevalence in training data, where the training data is an incredibly large corpus.
The experiment is about finding one missing word in a sentence, but the same logic is true even for long responses, or as I suggested, conversations.
To over-simplify: humans think in words, computers think in math. But it turns out that if you assigned a number to each word and parse enough of them, you can fake an understanding of words surprisingly well, even though it's still math.
I've given example elsewhere in this thread and on previous ones about the March 14 Chat-GPT giving incontrovertibly false information, even repeatedly in the face of correction, or seeming to correctly identify a popular logic puzzle while breaking the rules of the puzzle three times, then seeming to claim it had followed the rules of the puzzle.
> So Consider that as an existence proof that even if you "train to predict" you can still end up with actual understanding.
As a human, that is often true. Humans have trouble with arbitrary data, and it helps us to connect new data to existing data. In addition, we generally have a curiosity that prompts us to pursue new knowledge and meaning wherever possible, even when it doesn't actually exist. But if you're trying to suggest that because it is possible that humanscan develop understanding while trying to only "train to predict," therefore it is probable that computerswill develop understanding along the same lines, I suggest that's about three leaps too far.
I think with sufficiently good technology the human brain, too, could be modeled with just math. So far, it seems like basically everything in the universe can be described as just math!
ChatGPT is a neural network, a design inspired (loosely) by the human brain. How exactly it works is still not known. Like of course the raw details of how the computations work is known, but why the particular trained weights are what they are, and how the thinking actually happens is not known (so far as I know).
But I think it must be more than just something like looking up a conditional probability table -- calculate out the size of a conditional probabiltty of 8000 tokens or whatever and it'll be way way way more than the number of parameters of the model.
And no doubt as it stands ChatGPT is still far inferior to human intellect.
But I would note that making mistakes doesn't mean something like thinking isn't happening -- humans make mistakes all the time, often very dumb ones!
This has been talked over and over already. The conclusion is that eventually, you don't know what are the actual differences between "reasoning" and "next token prediction" in a real human.
On one hand, it makes this topic difficult to follow since stuff aren't clearly defined anymore and to software folks, I assume this is a major problem. On the other hand, it is fascinating how this has actually resembled biology where literally everything is poorly documented and for every rule, there is a glaring exception or flagrant violation of it, existing simply just to spite that rule and our understanding of it.
> This has been talked over and over already. The conclusion is that eventually, you don't know what are the actual differences between "reasoning" and "next token prediction" in a real human.
I disagree with this conclusion, which some supporters of GPT-4 being a reasoning engine might find distressing.
From what I have seen, you can't really articulate why you disagreed. Your arguments appears to be another form of "humans have it, this AI doesn't" but fail to specifically define what is this "it".
The AI effect is a real thing. And even real humans had been considered unable to "reason" once upon a time. I will give everything the benefits of the doubt concerning intelligence and sentience. Because history has shown how terrible it would be to get a false negative.
Do you have a reason for believing that a transformer couldn't out-perform existing techniques for choosing, given a sequence A_n(...A_1(Prop)...), what token comes next?
There are certain types of arguments that you could make, eg complexity theory does give us some bounds, but these models are large enough that (a) you can trade off enormous amounts of time for space, and (b) even time is questionable since forward passes result in truly enormous amounts of (CPU-indexed) computation.
And of course you can say hand-wavy things like "these models can only at best interpolate", but how many mathematics papers get rejected every year due to lack of novelty? And how much actually useful reasoning is novel, either in its particulars or especially in its generalized schematic form?
The true answer to the question of whether GPT-4 is a decent reasoning engine is, I think, close to your intuition: probably not. But in general, I don't see any good reason to believe that transformers can't do reasoning. In fact, quite the opposite.
As a foreword: I am absolutely not an expert on the topic.
If theory of computation has taught us anything, is that complexity very quickly outgrows everything that is even in the grasp of mathematics (e.g. busy beaver).
I just don’t think that a model that fundamentally can only take a fixed number of steps will ever make the jump. A human in theory can “go into a loop” and let’s say apply the same rules ad infinity, which is simply a must to most interesting problems. So - I guess GPT-4 can reason on the “trivial” degree simply by “caching” the output of the real algorithm, but that doesn’t scale at all.
> If theory of computation has taught us anything, is that complexity very quickly outgrows everything that is even in the grasp of mathematics (e.g. busy beaver).
True. But if the science/practice of computation has taught us anything, it is that many problems can be solved for all practical purposes even if they are undecidable or intractable for all theoretical purposes. E.g., the halting problem is undecidable but software verification/static analysis tools that check for termination exist and are even widely used in some industries.
> A human in theory can “go into a loop”
A computer can as well, by identifying loop invariants. And even the discovery of loop invariants can be thought of as a next-token-prediction problem, if characterized in as a sequence of states followed by the question "if x_1=t_1, ..., x_n=t_n, then what are these variables equal to at state N+1?"
> So - I guess GPT-4 can reason on the “trivial” degree simply by “caching” the output of the real algorithm, but that doesn’t scale at all.
I am more-or-less in agreement about GPT-4, but I think claims about the limitations of the architecture/problem statement are over-stated.
This is a weird question, so I’ll disclaim that I am asking in good faith.
Why is treating GPT-4 as a reasoning machine so controversial? And by that, I don’t mean why it is wrong. I am asking why it upsets so many people.
We ignore a whole bunch of lies and half truths every day. What is it about GPT-4 that doesn’t let it get a free pass like UFOs or EmDrives?
What I am really interested in understanding is the possibility that at least some of these strongly dismissive responses are a bit of a threat response.
The other interesting possibility is that for over a decade, we got used to dismissing AI startups - and now, that habit could persist despite a real breakthrough.
Another possibility is that we were all given an overly simplified model/explanation of machine learning and now, those simplifications are being a real hurdle to comprehension.
It’s extremely fascinating to watch this play out. For my generation, this is the first time we’re seeing a brand new fundamental technology get deployed and alter society.
Speaking for myself, I am equally dismissive of aliens, if that helps.
I think what annoys me about claims like this is two things, and it has to do with in-groups and out-groups.
1. I am concerned about how non-developers perceive us developers when they hear claims like this, since I expect that particular hype balloon to pop in the future.
2. I am distressed to see developers (who should know better) make these types of claims, especially without regard to 1.
If I were more into physics, I'd probably get more worked up about em-drives. As it is, I am enjoying LLMs and consider them an incredibly useful tool, possibly up there with things like GUIs and IDEs in terms of productivity enhancement. But it's still just extra-spicy autocomplete, and I think forgetting that could be dangerous.
Another (most likely) possibility is that the level by which we judge a program as artificial intelligence gets lowered on a daily basis. What counted as AI didn't count as AI a decade ago.
> and now, that habit could persist despite a real breakthrough
Or the breakthrough has a local maximum, because if LLMs are just statistical token predictors without any real reasoning, it will never achieve AGI level of AI.
Yes you are right that are a kind of threat response but not personal threat as much as general threat. There is a great danger that because of the name AI and the general association that we have with computers as giving correct result, People will (and are already) attribute meaning and "correctness" where there is none. This is nothing like UFOs or EmDrives because those come from fallible humans and this would come from the "infallible AI". What is wrong with describing it for what it is, a stochastic predictive model?
Human brains are known to create "meaning" where there is none and with a perfectly created sentence they will assign meaning (reasoning in this case) where there is none.
One week ago there was a case of a man (in Belgium) who "sacrificed" himself because the AI chatBot convinced him that there was no solution to Enviornment problem. He assigned meaning ("Even the AI could not find a way out") where the chatbot was just parroting what it was reading in the media.
Note that in this case both the general media and the AI give the same "result"=no hope, but as long as it was in the general media, he was still hopeful that humans just couldn't find it whereas the AI convinced him.
You may diregard this as an extreme example but we have the general tendency to attribute correctnes to everything "computable". If I can trust the calculator that was created to do math why should I not trust the AI that is created to do Reasoning? People need to be aware that there is no reasoning behind it.
> Why is treating GPT-4 as a reasoning machine so controversial?
At the risk of attracting down votes, you could ask the same question about the visceral responses to crypto currencies you see here.
It's certainly true they have their problems. Bitcoins 7 transactions per second makes it's chain a no-starter for most uses, the numerous multi million dollar losses due to bugs in Ethereum contracts should scare anybody. But the reaction you see here goes well beyond that. After all they continue exist, even performing the occasional useful task, and they are gradually (glacially?) overcoming their initial shortcomings.
The aversion to everything crypto seems to be more of a guttural reaction to the existence of something that uses algorithms to generate and distribute trust. It challenges the human social trust networks we've been using to do the job for centuries. This aversion to calling the obvious deductive capabilities AI's "thinking" or "intelligence" looks to be similar. I guess it challenges what has been a uniquely human ability.
> GPT-4 is a generative algorithm and large language model. It performs no reasoning, and instead predicts the next token in language.
So? What makes an LLM be incapable of reasoning in your view? It's like saying a cargo ship is massive metal object so it cannot float!
You could easily argue any good language model would need to learn how to reason to be good at next token prediction. Reasoning therefore should, if anything, be an expected emergent property of a sufficiently large LLM.
The fact you can ask GPT problems that require reasoning and it gets them right frequently suggests (although admittedly doesn't prove) some degree of reasoning capability.
I think the difference there is that you can model a cargo ship to show that it will float, how it will do it, what configurations float and which don’t, etc. With LLMs like GPT we know how they work, but it’s hard to know that when it’s actually running and doing things, making it hard to predict what the outcome will be or perfectly understand how it came to the result it did (disclaimer: I am not an AI/ML engineer and this is just my personal understanding which could be wrong).
I think a lot of the annoyance felt around here is the tendency to apply human attributes to LLMs. It’s not to say that Chat GPT isn’t fulfilling the definition of “reasoning” on some level, even if it was found to fully meeting definition. I think it’s more the leap to the conclusion and that a lot of Hacker News readers are rankled by the leap lacking research and facts to back it up.
Anyway, it’s all really a philosophical discussion anyway since the definition of “intelligence” and “reason” are soft terms even when applied to human beings. We can usually hand wave it away as “intelligence is when humans do it the way humans do it”. Now that we’re on the cusp of creating actual artificial intelligence, I think we’re all finding those soft definitions cumbersome and are struggling to refine them as a society, which I think is a good thing. I also sometimes get rankled at someone asserting that some AI example has some human attribute, but mostly I think it’s an important part of that larger discourse about how we define these things, and that is a good thing in my mind.
It's not impossible to see that LLMs can make finite step "reasoning" between input and output, as each block of the transformer can model probabilistic causality. Transformers are sometimes considered a fully connected graph neural network, which can be used for modeling causal graphs. One additional supporting evidence on the finite step reasoning hypothesis is that "train of thought" improves the performance of LLMs[1], meaning that letting the model explain itself explicitly reduces the amount of implicit "reasoning" steps that need to happen within the model.
Additionally, while we don't know how us humans functionally reason, it's believed that the predictive nature of our brain is central to our reasoning abilities. Maybe in some way, the autoregressive nature of LLMs is similar to our predictive brain.
For a collection of emergent properties in LLMs, I recommend this paper [2]
I smile every time I hear this argument. To me it’s obviously false and head in the sand. There was a moment, with GPT-2 where it felt exactly right, then with GPT-3 where you could see a glimmer of something emerging. Now GPT-4 has something real going on.
A solid example is the tool use. Package up a brand new, never existed before or seen in the training data plug-in, give a few paragraphs of human readable documentation, and suddenly GPT-4 can interact with other systems, databases, and models. It doesn’t have 10,000 examples of interacting with this novel plugin to predict next word off of, it just has a short English description, enough for a human to be able to use and understand the same plugin effectively.
There’s a reason this is knocking the socks off many smart folks. It’s upending a lot of the sacred cows of human intelligence.
There seems to be a huge lag in news these days. I was reminded recently that GPT-4 was only released three weeks ago even fewer people have full access to the plugins that the author is describing here.
Most people are still talking about an article they read in the New York Times about GPT-3.5 Turbo. Not that 3.5 Turbo didn't exhibit some reasoning abilities, but they were more difficult to suss out. GPT-4 can still seem pretty vacant sometimes and finding "magic" in it has a lot to do with building up a "state" where the model is generating reason. It does not happen by default.
This will all be different in another month or two, as more people have access to the latest models and new models get released (assuming there is no agreement on a pause). Then there will be another lag as laypeople catch up.
There are scenarios that may play out where this speeds up even further and whole generations of AI completely fly under the radar of public acknowledgement.
Or this all collapses in a debris pile of hype. I feel like I am just along for the ride at this point, trying to keep my head above water. I reckon this is going to cause a tremendous amount of stress for a tremendous number of people.
You’ve attached to a superficial explanation that doesn’t explain the magic trick. It sometimes emits text that corresponds to how people reason, but it’s unclear how well its mechanisms generalize because we don’t know what they are.
We should be curious about how it does what it does when it seems to be reasoning.
> Since most of us aren’t AI researchers, what does “not settling” mean in practice? We should remember that there are mysteries and reject superficial explanations. Chatbot boosters will promote them and skeptics will dismiss them, but that doesn’t mean they understand how they work.
God of the gaps, reworked. Admittedly, a new claim to me. $10.05.
It performs no reasoning, and instead predicts the next token
If we had a nickel for every misguided repetition of this non sequitur we'd have another $10. GPT-4 is far beyond markov chain predictions and Eliza. While technically Eliza, GPT-4 and Photoshop operate on the same principles of assembly instructions, and GPT-4 indeed does token predictions, this says nothing about the inner architecture and reasoning capabilities. GPT-4 internally is a functionally complete state machine, or would be on an unlimited token stream.
Better read: the original paper demonstrating that prompting chain-of-thought reasoning elicits superior results. TLDR, give examples of working out the problem step-by-step, encourage GPT to emulate, and it's capable of solving challenges it would completely miss otherwise.
Yeah, I thought that the Lowell example was poorly chosen. I'm not American and I'm not all that interested in astronomy, and I've heard of Lowell and the Lowell Observatory.
I got one a few weeks ago. I'm using it from Germany. There is definitely a morning peak on the west coast in the US that affects reach ability. Though with a paid account it usually works for me. And I've used less than a dollar of tokens so far. So, it's not super expensive. You can get the 20$/month subscription as well in which case they prioritize you some more. IMHO the pricing makes it a no brainer and I've definitely gotten some value back.
The advantage of the cheap API key is also that you can use it with all sorts of tools. CodeGPT for intellij is amazing for example. I have a vs code plugin and I found an extension for Google docs. And I've written a few small integrations myself for some private projects. All good value. And less than a dollar spent on tokens.
No it's not, it's a language model. It manipulates words. If you want reasoning, you should find the right tool. For example, encode your problem for an SMT solver (like Z3, NuSMV, etc).
Let's say you have an elevator start on floor 11, and another at floor 3. The rules are, they can never occupy the same floor. Is it possible for the first elevator to ever reach floor 2? No, they would have to pass each other.
GPT-4 may be able to come to the conclusion by accident. But not reliably.
sorry, that wasn't my point. I just meant, there are tools for that job. LLMs don't seem to be the most appropriate tool. Thanks for helping me to clarify
Yes, if they were to move any number of floors at the same time step. But the language model isn't necessarily going to automatically discover those rules for you. Even if it did, the model won't necessarily be able to frame the problem as a BDD and solve it.
At best (and the is a valuable trait) the LLM would be able to reframe your problem in such a way that you could discover the necessary precision in the problem statement. Perhaps it could even encode it in some syntax for another tool.
Just tried this with ChaGPT for giggles. It spit out words, but I wouldn't call it sound reasoning.
__user__:
- A system in composed of two elevators.
- Each have their own unique velocity parameter.
- At each time step in the system, the elevators can move up or down a number of floor equal to their velocity, or change their velocity.
- Assume that both elevators move at the same time step.
- The elevators may never occupy the same floor.
- Elevator A starts on floor 11, and Elevator B at floor 3.
- Both elevator velocities start at 1.
Is it possible for A to ever reach floor 2?
__assistant__:
No, it is not possible for Elevator A to ever reach floor 2 while maintaining the rule that the two elevators may never occupy the same floor.
To see why this is the case, note that Elevator A starts at floor 11 and Elevator B starts at floor 3. Since both elevators move at the same time step, Elevator A can never move to a floor lower than Elevator B without violating the rule that they may never occupy the same floor.
Furthermore, since Elevator A and Elevator B have the same velocity at the start and Elevator B is already on a lower floor, Elevator B will always reach any floor below Elevator A before Elevator A can get there. This means that Elevator A can never occupy any floor that is below the current position of Elevator B, including floor 2.
Therefore, it is not possible for Elevator A to ever reach floor 2 while maintaining the given constraints.
__user__:
what if at the first time step, both velocities became equal to 5?
__assistant__:
If both elevators have their velocities set to 5 at the first time step, then it is possible for Elevator A to reach floor 2.
With both elevators moving at velocity 5, Elevator A will reach floor 6 after the first time step, while Elevator B will reach floor 8. Since Elevator A is now on a lower floor than Elevator B, it is possible for Elevator A to continue moving down to floor 2 without violating the constraint that the elevators may never occupy the same floor.
Specifically, Elevator A can continue moving down at velocity 5, reaching floors 1 and 2 after two more time steps. Elevator B, with velocity 5, will reach floor 13 after the same two time steps.
So, if both elevators have their velocities set to 5 at the first time step, then it is possible for Elevator A to reach floor 2 while maintaining the given constraints.
Anecdotally, I've been using GPT4 to help me get overviews of techniques in fields which I'm working adjacent to. I've been pleasantly surprised at its ability to quickly summarize not just ideas, but also write out the math.
At the same time, there's been multiple occurrences where it has produced math mistakes, despite the fact that it can tell me that it knows how to do "the right thing" e.g. there's been multiple times where it will incorrectly reduce integrals with Dirac delta functions, or introduce subtle mistakes in analytically known formulas.
The Dirac delta function example was funny to me -- it can tell me what to do when a delta function occurs in an integrand! Yet it makes a mistake reducing the same sort of integral in another context?
So I'm not sure how to take these "GPTX is doing reasoning" claims. On the one hand, maybe? On the other hand, it's not perfect at it -- even when it appears to have knowledge that should ensure that it never gets that wrong.
If I'm going to be using this system to help me do research, I'd love for it to be correct and not contradict itself. That's why I'm happy to have these GPTX layers shell out to something like Wolfram, or another symbolic system. That combination makes perfect sense to me. I'm also much more likely to trust that combination, instead of my current strategy - which is to basically re-process everything that GPTX gives me and assume that there are a few places where I'm likely to find mistakes.
146 comments
[ 4.9 ms ] story [ 244 ms ] threadhallucinates
> who is dan shipper? please search if you don't know the answer.
gives right answer
If it is a reasoning engine, why didn't it answer "I don't know" in the first case?
> Hello, who do you want information about?
> who is dan shipper?
> Dan Shipper is an entrepreneur, writer, and software developer known for his work in the tech industry. He is the co-founder of Superorganizers, a platform that focuses on productivity and organization. Superorganizers offers a newsletter, podcasts, and interviews featuring insights from successful entrepreneurs, writers, and other professionals on how they stay organized and manage their work.
> Who is Tommy Tonlongeron?
> I'm sorry, but I don't have any information on a person named Tommy Tonlongeron. It is possible that the name is misspelled, the person is not well-known, or there is simply no information available on them within my training data up to September 2021. If you have any other questions or need information about someone else, feel free to ask.
If you tell it (GPT4) to, it seems GPT4 is at least mildly aware about what it knows VS what it doesn't know.
>>> You are a AI for describing people you know about. If you're not 100% certain about the information you give to the user, you should say that you don't know instead of guessing. Greet the user with "Hello, who do you want information about?"
> Hello, who do you want information about? I'll do my best to describe them based on the information I have, but if I'm not certain about something, I'll let you know that I don't know.
>>> who is dan shipper?
> Dan Shipper is an entrepreneur and writer based in New York City. He is the co-founder and CEO of a startup called Pilot, which provides bookkeeping services for startups and small businesses. Dan has also written for a variety of publications, including The New Yorker, The Wall Street Journal, and The New York Times. He is known for his insights on entrepreneurship, technology, and productivity, and has been featured in several media outlets for his work in these areas.
>>> Who is Tommy Tonlongeron?
> I'm sorry, but I couldn't find any information about a person named Tommy Tonlongeron. It's possible that the name is misspelled or that this person is not well-known or doesn't have a significant online presence. If you have any more information or context, I'd be happy to try and help you further.
(In case it isn't clear, Chat-GPT appears to have hallucinated the answer about Dan Shipper, I think. Pilot is a real startup, but I don't see how it's related to Dan Shipper at all, unless I'm missing something.)
>>> Why did you say that Dan Shipper is the co-founder of Pilot?
> I apologize for the error in my previous response. Dan Shipper is actually one of the co-founders of a startup called Firefly, which provides a tool for students and educators to collaborate on course materials. I'm sorry for any confusion that my mistake may have caused. Thank you for bringing it to my attention, and please let me know if you have any other questions.
(This answer was true--until 2014 when Firefly was sold, making the use of present tense misleading at best.)
> Dan Shipper is a technology entrepreneur and writer. He is known for co-founding the startup Firefly, which aimed to improve networking for college students, and for co-founding the startup Monkey Inferno, which incubates and launches consumer internet companies. Shipper has also written for various publications, including The New Yorker, The Paris Review, and The Atlantic.
Monkey Inferno? No, that was Michael & Xochi Birch.
I additionally asked it about its correct answer, using the same question wording as I did previously:
>>> Why did you say that Dan Shipper is a co-founder of Firefly?
> Dan Shipper is known for being one of the co-founders of Firefly, a startup that provides a platform for teachers and students to create and share course materials. Firefly is designed to help teachers and students work together more efficiently, by providing tools for creating assignments, sharing resources, and communicating with each other. Dan Shipper co-founded Firefly along with his friend and fellow entrepreneur, Brett Gibson, while they were both still students at the University of Pennsylvania.
This is also incorrect, as his co-founder at Firefly was named Justin Meltzer, not Brett Gibson.
I did some searching to track all this down, since I knew none of the people involved before this exercise.
This means you were using GPT-3.5-turbo rather than GPT-4 I'm guessing? The quality of the generation is vastly different between the two, especially for steering the AI.
Wikipedia defines reasoning as "Reason is the capacity of consciously applying logic by drawing conclusions from new or existing information, with the aim of seeking the truth". https://en.wikipedia.org/wiki/Reason
Here's a sample question from the GRE Verbal Reasoning test:
> Upon visiting the Middle East in 1850, Gustave Flaubert was so [blank] belly dancing that he wrote, in a letter to his mother, that the dancers alone made his trip worthwhile.
> (A) overwhelmed by
> (B) enamored by
> (C) taken aback by
> (D) beseeched by
> (E) flustered by
Whether that specific question was in the training corpus or not, there are enough words in the sentence to suggest a positive association, including, significantly, "worthwhile." That alone possibly serves to narrow the answers down to A or B, with a preference for B, because it's more likely that "worthwhile" and "letter to x mother" are associated with "enamored" in general English-language text.
Look, the whole point of these models is that its not easy or even possible to trace the path any given input takes on its way to output, but we know the principles used in development, so I think it's rather more of a burden to explain how the clearly-explained principles in LLMs result in something other than the obvious. The fact that the results are so overwhelming that we become enamored by them, well, I'm taken aback by the seeming accuracy of some of the responses, but I beseech you to remember the other responses in which these LLMs are dramatically off-base, as if flustered--if LLMs could ever be flustered.
A tale as old as time: "It is illustrated by the success of chess computers. In the 60s, it was said that computers will never beat people at chess, because that requires intelligence and computers aren't capable of intelligent thought.
When computers regularly started winning matches in the 80s, it was claimed that playing chess wasn't a test of real intelligence because computers could do it."
>Whether that specific question was in the training corpus or not, there are enough words in the sentence to suggest a positive association, including, significantly, "worthwhile." That alone possibly serves to narrow the answers down to A or B, with a preference for B, because it's more likely that "worthwhile" and "letter to x mother" are associated with "enamored" in general English-language text.
Yes, this is called reasoning so in other words the LLM is reasoning about language.
>Look, the whole point of these models is that its not easy or even possible to trace the path any given input takes on its way to output, but we know the principles used in development, so I think it's rather more of a burden to explain how the clearly-explained principles in LLMs result in something other than the obvious.
The inscrutability of the Matrices tells you nothing about it's reasoning ability. Given the correct prompt the LLM will also provide you with a step by step solution to the question it answered. There are also explicit reasoning prompts that these models are able to deal with.
I think it's pretty simple, if these questions are not explicitly in the training data it cannot have answered them correctly at such a high success rate with anything other than reasoning. You haven't given any alternative answer to how it does this either.
I have. You refuse to accept it, but I definitely have given an answer that involves tokenization and association, the things we already know LLMs use to construct their responses.
This is reasoning, what you've described is reasoning.
I believe literally none of those things, and have explicitly stated the opposite of at least one of them on this page.
At this point I think an LLM would do a better job of responding to my comments, based on context and syntax alone.
Because it wasn't purposefully built to be a reasoning engine from the outset. So some of us are working out the kinks and minimizing the degree of hallucination. GPT-4 hallucinates less than its predecessor.
Others are writing reasoning loops like ReAct that get us closer to it by refining the model to act more like a reasoning engine.
The fact that it can reason without being told to do much other than autocomplete text is the emergent property that people are in awe of. Maybe language and reasoning are so inexorably linked that you can't have one without the other. We are, after all, the animals on this planet with the most complex form of language. Other relatively "smart" animals like whales also have some rudimentary form of language.
And if you extrapolate the rate of progress in this emergent reasoning from "GPT-1" to "GPT-4", AGI feels within reach
Reasoning engines can be built, but its not in GPT-4.
To reason is to seek truth (probably to survive). Do you think the truth-seeking is emerging? If so, why do you hypothesize it is emerging (i.e. what's the purpose)?
I said:
It said: Which, again, makes sense at face value, but the schedule does cover building C programs in the first few weeks. The problem is that it shouldn't be covering it again in week 12.The response I was hoping for was
You know what ChatGPT reminds me of? Talking to an undergraduate who always nods and agrees that they are following your logic. Then when you ask them to tell you their understanding, it's either an exact parrot of what you said, or wildly off due to some preconceived notion or general lack of understanding of the topic. Either way, they can never let on that they are clueless.GPT-4 is a generative algorithm and large language model. It performs no reasoning, and instead predicts the next token in language. This works okay on average, because most data on the internet has some type of coherency to it, but there is a major gap between "reasoning" and "next token prediction."
"Next token prediction" taken to a certain mathematical limit implies reasoning.
The point of my comment is that depending on how you interpret the description "predicts the next token", something that can do this well enough should be superhumanly clever. The task is "AI Complete".
Don't misinterpret me as saying that GPT is superhumanly clever. I'm just saying that something that's sufficiently good at what people think GPT is doing, should be superhuman.
ChatGPT is an implementation of the Chinese Room Thought Experiment.[0] Our pareidolia allows us to ascribe reasoning faculty.[1] But it is far from proven.
[0] https://en.wikipedia.org/wiki/Chinese_room
[1] https://en.wikipedia.org/wiki/Pareidolia
For this whole discussion, it actually really does matter if GPT-4 is able to reason or not. That is the essence of the discussion. It's not whether or not it looks like it can reason.
My own car includes cameras and can keep me from disaster by applying brakes and correcting steering, but it makes no claims whatsoever to be self-driving.
A Tesla car unfortunately looks like self-driving, walks like self-driving, and quacks like self-driving, but it turns it does matter how much the nitty-gritty details prove the thing is absolutely not actually self-driving, in that encounters with things outside of its training model seem to result in collision or decapitations.
It is the trust people place in their cars that results in disaster when they reach the edge of what the cars can handle. They aren't ready to take over the controls and understand their current situation in a fraction of a second.
Placing too much trust in LLMs could have the same effect. I've run into several situations where I asked Chat-GPT technical questions and it confidently gave me detailed answers that clearly represented absolute failure. Fortunately, I don't trust it blindly, or at all, so I wasn't fooled.
I'd prefer an LLM more like my current car, that didn't try to do too much, but that's just not in the nature of how they operate, so instead we're all one emergency vehicle parked across the street away from a terrible accident.
> Suppose I have a cabbage, a goat and a lion, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the cabbage and lion alone together, and I am not allowed to leave the lion and goat alone together. How can I safely get all three across?
I just asked ChatGPT v4 this question, assuming it would have been fixed by RHLF training but it spat out step 1 as “First, take the goat across the river and leave it on the other side,” leaving the lion and cabbage alone against explicit directions in the riddle.
It clearly can’t reason, just produce a convincing imitation most of the time
No, the reasoning is the prior data. Thought experiment: GPT-4, but the training data contains no puzzles of this kind. How well would it do?
> Try it again, but this time make up nonsense words for the three items.
Seconded. I also would like to see the results of this experiment.
>> Suppose I have a blurple, a mantiple and a hoje, and I need to get them across a river. I have a boat that can only carry myself and a single other item. I am not allowed to leave the blurple and hoje alone together, and I am not allowed to leave the hoje and mantiple alone together. How can I safely get all three across?
> This is a classic river crossing puzzle. Here's one possible solution:
> 1. Take the blurple across the river, leaving the mantiple and hoje behind.
> 2. Leave the blurple on the other side of the river and return to the original side.
> 3. Take the hoje across the river and leave it there with the blurple.
> 4. Take the mantiple across the river and leave it with the hoje.
> 5. Return to the original side of the river and retrieve the blurple.
> 6. Finally, take the blurple across the river to join the mantiple and hoje on the other side.
> This solution ensures that no two items that cannot be left alone together are ever left alone together.
Of course, my second stipulation was "I am not allowed to leave the hoje and mantiple alone together" and yet step 1 says "leaving the mantiple and hoje behind"
So no, despite using the words "classic river crossing puzzle," it clearly expressed no understanding of the puzzle.
That really, really, really demonstrates something. I'm not sure what.
I gave ChatGPT 5 questions from a verbal reasoning test on sequence of words and it got 2/5 right.
Here is sample question:
Arrange the words given below in a meaningful sequence.
1. Poverty 2. Population 3. Death 4. Unemployment 5. Disease
A) 2, 3, 4, 5, 1
B) 3, 4, 2, 5, 1
C) 2, 4, 1, 5, 3
D) 1, 2, 3, 4, 5
ChatGPT> B) 3, 4, 2, 5, 1
Unemployment can lead to poverty, which can lead to disease. Disease can lead to death. The population of an area is affected by all of these factors. Therefore, the correct order is 3, 4, 2, 5, 1.
Correct answer was C
I would expect if it was effective at reasoning it should be strongest on verbal associations. And while it is possible that enhancements to future language models might provided them with better reasoning ability I don't believe you can simply push the existing design to that limit. The poor results from the verbal reasoning tests demonstrate that reasoning is more complicated than what a transformer based language model is capable of.
Overall, the argument is moot because the statement made in the article was it was a reasoning engine which is not equivalent to being capable of reasoning.
"Machined" means a precise implementation. A machined reasoning engine would be precise, accurate, and consistent. GPT-4 is impressive autocomplete, but not what we would expect for machined reasoning.
To predict the next word very accurately for a huge variety of texts, a model must have some amount of real understanding. Sure, it may look different than human understanding.
But there’s no way around this. GPT could not predict so accurately if it didn’t have any understanding the world.
To at least some degree, maybe language is enough for both. Language doesn't just express concepts, but it contains them.
One of the main lessons this whole GPT fiasco has taught us is that if you make a super-duper fancy autocomplete trained on terabytes of textual data, it develops some sort of internal logic/understanding. That is very exciting to me.
It is truly impressive.
My argument is nothing against it as a technical achievement, and everything against the sophistry that it is a reasoning engine.
This is very different from real reasoning that has unlimited (yet imperfect) memory.
Hey, that was 15 words, and I had them all in my head at once before I started typing! I was able to form a sentence of more than seven words all by myself!
You seem to be misapplying the concept of working memory, where seven (plus or minus two) is about arbitrary and unrelated words or numbers, not "items." In fact, it's generally seven for numbers, but lower for words. And again, that's unrelated words, like "person woman man camera tv," but probably even more unrelated.
So am I, maybe
I just don't think this is it, although I freely admit that I'm amazing at the technology.
As it stands, Chat-GPT is probably better at giving the appearance of reasoning that some percentage of humans. That reflects poorly on those humans, but I've certainly known people in my life who seemed like they rarely knew the end of a sentence when they began speaking.
However, the focus of LLMs on text-prediction is not a path that leads toward human-like reasoning, just the appearance thereof. If and when we have AIs that actually do reasoning, it will be some other branch of AI, possibly also incorporating an LLM for communication, but not based solely on LLMs.
The point I'm making is not that GPT is as intelligent as a brain, but rather that the ability to produce the next token in a way that forms arbitrarily intelligent responses to arbitrary questions, may well require something resembling real, actual intelligence.
You decide which comments to do this for as well.
How does GPT decide such things?
All of them? If not, how to determine which ones?
There is a difference here between you and the machine
Looking at the LLM algorithms, they do appear to be next token predictors. Why do you ascribe something more to them when the algorithm is known?
Given we know that causal patterns are in the training data, and that LLMs are capable of embedding algebra, it is understandable that they appear to "reason", while we as humans can also describe the algorithm and mechanics actually at work here.
How does it determine how to respond to any prompt? Exactly. It might reasonably predict responses to those comments that, according to its black box, are somehow most likely to receive a reply. Different GPT instances prompted or trained to have different personalities would choose differently based on their priors and biases.
I'm pretty sure the vast majority of human beings are able to reason without really having the slightest idea as to how exactly they do it. Educated humans can use language to talk about reasoning in terms of logic, and even make logic engines out of doped silicon and whatnot, but as of now, none of us has any idea of how reasoning is implemented in the mass of fat and protein that's inside our heads.
Yet another case, many times we will respond (or not) to a questions with another question, to elicit deeper considerations. In my experience, GPTs will answer any question without thought, pun intended :]
The original question was, given that there were X comments which could be replied to on this page, how and why did the poster decide to respond to the one to which they responded, rather than either "all" (like ChatGPT would) or "none?"
If you mean feeding it the messages one at a time, then, sure, it will respond to each of them because that's what its default "goal function" is. (I assume that we're talking about a single "conversation" here so the context is retained – feeding it each comment out-of-context would be very unfair.)
If you prompt it to only reply to messages it considers "interesting" in some way, and return "I'll skip this" for the others, it would certainly do that.
Yes, GPT is code that only runs when you, erm… run it. It’s not an autonomous being that’s cut loose on the internet and given free reign to use as much compute resources as it wants, a la skynet, replying to random forum posts. But nobody’s really saying that it is, so it seems a strange argument to say “it’s just a next token predictor”, if by that you really mean “it only runs when you give it a prompt.”
The modality of GPT is that it responds to prompts, but the interesting question is “are the responses given a sign of intelligence?”. Saying “no, it’s just a next token predictor” is IMO a sign of a motte-and-bailey fallacy. Your motte is that it only works in response to prompts (trivially true and uninteresting), and your bailey is that it has no intelligence (very debatable and IMO false.)
GPT has no "reason" behind its words. Its actions are not tied to the general betterment of humankind. When we all sit here with our coffees and donuts (and farting) we're generally trying to push others to see our worldview so that maybe, just maybe, the world can change a little bit.
If we eventually have a way to tie an AI to all these moods (hunger, pain, tiredness) and all the other things of the human experience, it would ground the thoughts and text it generates at some point such that it would want to DO things it wasn't doing before. It also can't right now anyway, unless we give it that ability (ReAct pattern could).
We can, of course, simulate all these human experiences, with "best practices" but then the reason won't be really tied to human reasoning. It will be just more text autocomplete with additional instructions.
Sure, LLMs are not inherently goal-oriented. But that doesn't mean they fundamentally cannot exhibit goal-oriented behavior if either prompted or trained to do so. And if we remove the "one-off" restriction and allow the retention of some sort of an efficiently encoded state between invocations, who knows what might happen?
----
EDIT: Tried this with ChatGPT (GPT-4) and a subset of this thread, and the results are pretty good, even though the tree structure was lost in translation (simple copypaste). Certainly its thoughts of the differences and similarities between token-prediction and reasoning are more informed than those of many human commenters. It's also able to seamlessly include references to other comments, for instance (it deduced that "dougwmne" was the nickname of the comment author by itself):
> Additionally, it's worth noting that GPT-4's ability to interact with novel systems, as mentioned by dougmwne, might indicate some form of generalization and adaptation capabilities. Although it may not possess the same kind of reasoning as humans, it demonstrates an ability to understand and apply new information in a useful manner. This could be considered a form of reasoning, albeit a different and less sophisticated one than humans possess.
My prompt was:
> Here is an excerpt from a Hacker News discussion about GPT-4 and whether it can do reasoning. Write a comment that contributes to the discussion, either by bringing up a new idea or deduction, or by replying to another comment and arguing for or against it. You can quote another comment by prefixing it with ">".
followed by the copy-pasted subthread.
So you saw the parent comment about hype and clicked into this comment box without a thought of what you might say other than "When?" Really?
Did you know whether you agreed or disagreed with the parent comment, or did it take until the word "reductive" for you to firm that up?
If you're saying that the part of your brain that controls your fingers is doing so one word at a time, then sure, obviously that's true. But if you're making the opposite claim, that LLMs like Chat-GPT are "deciding" what to say and only then working out the finer details of how to express that decision, that's a claim I'd like to see explicitly.
If you are suggesting that your approach to intelligence is parallel to that of an LLM, then either you're arguing for something far more radical than most as an emergent property of LLMs, or you're suggesting that you only form opinions one word at a time as you type, which I think would make you, well, I won't insult you, but I think you're probably doing a little more than that.
Imagine I lock you in a room filled with french-language magazines and movies and books. Having nothing better to do, you watch and read them (as well as you can). Periodically, I have my confederate, a french man, enter the room and speak a french sentence with one word missing. He asks you to fill in the missing word.
We keep you locked in this room until you've achieved some target benchmark of correctly saying the missing word, say 20% of the time.
Are you then, just a "next-french-word-predicting" robot?
No, obviously not. To have achieved the task and escaped the room, you will have had to have gained a significant understanding of French.
It is the same with ChatGPT.
And this is all that GPT is doing.
To do it it, you need a form of compression: reasoning, intelligence, knowledge.
We already know LLMs tokenize words, so they have definitely generated and stored such a table. The marvelous step forward from previous word-based solutions is that they store not just single words, not just pairs of words, but larger sequences of words, even up to short sentence length. It makes the table pretty large, as you suggest, but that's exactly we've been seeing, and each generation of GPT is that much larger than the one before.
If you lock a reasoning being in a room, then regardless of your further actions with the French language, you'll be dealing with a reasoning being. Your sequence "you will have had to have gained a significant understanding of French" is deprecated on the fact that "understanding" is attributed to the reasoning being capable of it, not to the fact that they were locked in the room and forced to repeat the words.
No, in this scenario the posited person has only gained a significant understanding of patterns of French language, not of the language itself. You may believe that difference to be meaningless but tangibly it is the difference between identifying the next shape in a sequence and understanding that there is some larger purpose at play (context, if nothing else).
What actual understanding of the language they may have gained comes from second order effects -- seeing the language used in a movie for example is going to drastically influence and impart meaning that isn't present in the words alone.
This is somewhat born out in how ChatGPT and other LLMs being hyped are often really aggressively wrong about things. They don't get the correct answer in many circumstances, even when it's obvious to a human. If it truly understood everything it could from what it has been trained on, those gaps would not occur.
I don't think that's true at all. To explain my thinking, let's assume that I am an American, and therefore according to popular understanding, completely incapable of learning any language other than English. I despair! But then I think no, wait, I happen to have a gift for numbers. Like, my mental arithmetic stunned teachers from a young age, and I currently hold the world record for reciting digits of pi from memory. So what I will do, I think, is assign a number to every word I read or hear and where possible, visualize where that chain of numbers exists among the decimal digits of pi.
At some point the person would come in and utter some words I couldn't comprehend, but I would think, aha! That was almost 498153217845632754, except it was missing the second seven, which mapped to "contraire." And I'd do that at least 20% of the time, escaping the room with no understanding of French at all.
If I could map between memorized numbers and words quickly enough, I'd even be able to converse in French given suitable trigger words without a shred of understanding.
Because that's easy for a human, and the mathematical approach you describe is something like impossible.
Even for a machine, the mathematical approach is going to be impossible in practical terms as soon as input gets longer than a few sentences. To build any sort of probability table conditional on more than just a small number of tokens becomes astronomically big.
The intuition I'm trying (not very successfully) to get across is if you did this with a human, the human would end up learning French.
So Consider that as an existence proof that even if you "train to predict" you can still end up with actual understanding.
Beyond that, (and while this is less certain), probably to be able to predict well w/ any accuracy for texts of non-trivial length, you have to be able to understand; probably this is the only way to "compress" the insanely large conditional probabilities.
This is the mistake people keep making, literally anthropomorphizing LLMs! Because humans would find it easier to just learn the meanings of French words than regurgitate from a gigantic table of memoized tokens, these results must reflect understanding!
But of course, it's far, far easier for a computer to index a large number of tokens without any understanding at all than it would be for a person. Computers are not people. They have different strengths and weaknesses, and it is a mistake to forget that. (Ironically, one of our human weaknesses is also a strength: our pattern-matching skills are so strong we sometimes see Jesus in toast, or a face on Mars, when neither are actually there!)
What I described is literally how LLMs works. There is open source code to examine, and what it does is essentially what I said: tokenize words and store multi-word strings and identify their prevalence in training data, where the training data is an incredibly large corpus.
The experiment is about finding one missing word in a sentence, but the same logic is true even for long responses, or as I suggested, conversations.
To over-simplify: humans think in words, computers think in math. But it turns out that if you assigned a number to each word and parse enough of them, you can fake an understanding of words surprisingly well, even though it's still math.
I've given example elsewhere in this thread and on previous ones about the March 14 Chat-GPT giving incontrovertibly false information, even repeatedly in the face of correction, or seeming to correctly identify a popular logic puzzle while breaking the rules of the puzzle three times, then seeming to claim it had followed the rules of the puzzle.
> So Consider that as an existence proof that even if you "train to predict" you can still end up with actual understanding.
As a human, that is often true. Humans have trouble with arbitrary data, and it helps us to connect new data to existing data. In addition, we generally have a curiosity that prompts us to pursue new knowledge and meaning wherever possible, even when it doesn't actually exist. But if you're trying to suggest that because it is possible that humans can develop understanding while trying to only "train to predict," therefore it is probable that computers will develop understanding along the same lines, I suggest that's about three leaps too far.
ChatGPT is a neural network, a design inspired (loosely) by the human brain. How exactly it works is still not known. Like of course the raw details of how the computations work is known, but why the particular trained weights are what they are, and how the thinking actually happens is not known (so far as I know).
But I think it must be more than just something like looking up a conditional probability table -- calculate out the size of a conditional probabiltty of 8000 tokens or whatever and it'll be way way way more than the number of parameters of the model.
And no doubt as it stands ChatGPT is still far inferior to human intellect.
But I would note that making mistakes doesn't mean something like thinking isn't happening -- humans make mistakes all the time, often very dumb ones!
In your analogy, the person never learned the meanings of the French words. Just how they usually go together.
GPT-4 is not a simple Markov chain.
On one hand, it makes this topic difficult to follow since stuff aren't clearly defined anymore and to software folks, I assume this is a major problem. On the other hand, it is fascinating how this has actually resembled biology where literally everything is poorly documented and for every rule, there is a glaring exception or flagrant violation of it, existing simply just to spite that rule and our understanding of it.
I disagree with this conclusion, which some supporters of GPT-4 being a reasoning engine might find distressing.
The AI effect is a real thing. And even real humans had been considered unable to "reason" once upon a time. I will give everything the benefits of the doubt concerning intelligence and sentience. Because history has shown how terrible it would be to get a false negative.
Let A_i be a set of functions that transform sequents in a truth-preserving manner (https://en.wikipedia.org/wiki/Sequent).
Do you have a reason for believing that a transformer couldn't out-perform existing techniques for choosing, given a sequence A_n(...A_1(Prop)...), what token comes next?
There are certain types of arguments that you could make, eg complexity theory does give us some bounds, but these models are large enough that (a) you can trade off enormous amounts of time for space, and (b) even time is questionable since forward passes result in truly enormous amounts of (CPU-indexed) computation.
And of course you can say hand-wavy things like "these models can only at best interpolate", but how many mathematics papers get rejected every year due to lack of novelty? And how much actually useful reasoning is novel, either in its particulars or especially in its generalized schematic form?
The true answer to the question of whether GPT-4 is a decent reasoning engine is, I think, close to your intuition: probably not. But in general, I don't see any good reason to believe that transformers can't do reasoning. In fact, quite the opposite.
If theory of computation has taught us anything, is that complexity very quickly outgrows everything that is even in the grasp of mathematics (e.g. busy beaver).
I just don’t think that a model that fundamentally can only take a fixed number of steps will ever make the jump. A human in theory can “go into a loop” and let’s say apply the same rules ad infinity, which is simply a must to most interesting problems. So - I guess GPT-4 can reason on the “trivial” degree simply by “caching” the output of the real algorithm, but that doesn’t scale at all.
True. But if the science/practice of computation has taught us anything, it is that many problems can be solved for all practical purposes even if they are undecidable or intractable for all theoretical purposes. E.g., the halting problem is undecidable but software verification/static analysis tools that check for termination exist and are even widely used in some industries.
> A human in theory can “go into a loop”
A computer can as well, by identifying loop invariants. And even the discovery of loop invariants can be thought of as a next-token-prediction problem, if characterized in as a sequence of states followed by the question "if x_1=t_1, ..., x_n=t_n, then what are these variables equal to at state N+1?"
> So - I guess GPT-4 can reason on the “trivial” degree simply by “caching” the output of the real algorithm, but that doesn’t scale at all.
I am more-or-less in agreement about GPT-4, but I think claims about the limitations of the architecture/problem statement are over-stated.
Why is treating GPT-4 as a reasoning machine so controversial? And by that, I don’t mean why it is wrong. I am asking why it upsets so many people.
We ignore a whole bunch of lies and half truths every day. What is it about GPT-4 that doesn’t let it get a free pass like UFOs or EmDrives?
What I am really interested in understanding is the possibility that at least some of these strongly dismissive responses are a bit of a threat response.
The other interesting possibility is that for over a decade, we got used to dismissing AI startups - and now, that habit could persist despite a real breakthrough.
Another possibility is that we were all given an overly simplified model/explanation of machine learning and now, those simplifications are being a real hurdle to comprehension.
It’s extremely fascinating to watch this play out. For my generation, this is the first time we’re seeing a brand new fundamental technology get deployed and alter society.
I think what annoys me about claims like this is two things, and it has to do with in-groups and out-groups.
1. I am concerned about how non-developers perceive us developers when they hear claims like this, since I expect that particular hype balloon to pop in the future.
2. I am distressed to see developers (who should know better) make these types of claims, especially without regard to 1.
If I were more into physics, I'd probably get more worked up about em-drives. As it is, I am enjoying LLMs and consider them an incredibly useful tool, possibly up there with things like GUIs and IDEs in terms of productivity enhancement. But it's still just extra-spicy autocomplete, and I think forgetting that could be dangerous.
> and now, that habit could persist despite a real breakthrough
Or the breakthrough has a local maximum, because if LLMs are just statistical token predictors without any real reasoning, it will never achieve AGI level of AI.
Unless, of course, we redefine what an AGI is.
Human brains are known to create "meaning" where there is none and with a perfectly created sentence they will assign meaning (reasoning in this case) where there is none.
One week ago there was a case of a man (in Belgium) who "sacrificed" himself because the AI chatBot convinced him that there was no solution to Enviornment problem. He assigned meaning ("Even the AI could not find a way out") where the chatbot was just parroting what it was reading in the media. Note that in this case both the general media and the AI give the same "result"=no hope, but as long as it was in the general media, he was still hopeful that humans just couldn't find it whereas the AI convinced him.
You may diregard this as an extreme example but we have the general tendency to attribute correctnes to everything "computable". If I can trust the calculator that was created to do math why should I not trust the AI that is created to do Reasoning? People need to be aware that there is no reasoning behind it.
At the risk of attracting down votes, you could ask the same question about the visceral responses to crypto currencies you see here.
It's certainly true they have their problems. Bitcoins 7 transactions per second makes it's chain a no-starter for most uses, the numerous multi million dollar losses due to bugs in Ethereum contracts should scare anybody. But the reaction you see here goes well beyond that. After all they continue exist, even performing the occasional useful task, and they are gradually (glacially?) overcoming their initial shortcomings.
The aversion to everything crypto seems to be more of a guttural reaction to the existence of something that uses algorithms to generate and distribute trust. It challenges the human social trust networks we've been using to do the job for centuries. This aversion to calling the obvious deductive capabilities AI's "thinking" or "intelligence" looks to be similar. I guess it challenges what has been a uniquely human ability.
So? What makes an LLM be incapable of reasoning in your view? It's like saying a cargo ship is massive metal object so it cannot float!
You could easily argue any good language model would need to learn how to reason to be good at next token prediction. Reasoning therefore should, if anything, be an expected emergent property of a sufficiently large LLM.
The fact you can ask GPT problems that require reasoning and it gets them right frequently suggests (although admittedly doesn't prove) some degree of reasoning capability.
I think a lot of the annoyance felt around here is the tendency to apply human attributes to LLMs. It’s not to say that Chat GPT isn’t fulfilling the definition of “reasoning” on some level, even if it was found to fully meeting definition. I think it’s more the leap to the conclusion and that a lot of Hacker News readers are rankled by the leap lacking research and facts to back it up.
Anyway, it’s all really a philosophical discussion anyway since the definition of “intelligence” and “reason” are soft terms even when applied to human beings. We can usually hand wave it away as “intelligence is when humans do it the way humans do it”. Now that we’re on the cusp of creating actual artificial intelligence, I think we’re all finding those soft definitions cumbersome and are struggling to refine them as a society, which I think is a good thing. I also sometimes get rankled at someone asserting that some AI example has some human attribute, but mostly I think it’s an important part of that larger discourse about how we define these things, and that is a good thing in my mind.
Additionally, while we don't know how us humans functionally reason, it's believed that the predictive nature of our brain is central to our reasoning abilities. Maybe in some way, the autoregressive nature of LLMs is similar to our predictive brain.
For a collection of emergent properties in LLMs, I recommend this paper [2]
[1]https://arxiv.org/abs/2201.11903.pdf [2]https://arxiv.org/pdf/2206.07682.pdf
A solid example is the tool use. Package up a brand new, never existed before or seen in the training data plug-in, give a few paragraphs of human readable documentation, and suddenly GPT-4 can interact with other systems, databases, and models. It doesn’t have 10,000 examples of interacting with this novel plugin to predict next word off of, it just has a short English description, enough for a human to be able to use and understand the same plugin effectively.
There’s a reason this is knocking the socks off many smart folks. It’s upending a lot of the sacred cows of human intelligence.
Most people are still talking about an article they read in the New York Times about GPT-3.5 Turbo. Not that 3.5 Turbo didn't exhibit some reasoning abilities, but they were more difficult to suss out. GPT-4 can still seem pretty vacant sometimes and finding "magic" in it has a lot to do with building up a "state" where the model is generating reason. It does not happen by default.
This will all be different in another month or two, as more people have access to the latest models and new models get released (assuming there is no agreement on a pause). Then there will be another lag as laypeople catch up.
There are scenarios that may play out where this speeds up even further and whole generations of AI completely fly under the radar of public acknowledgement.
Or this all collapses in a debris pile of hype. I feel like I am just along for the ride at this point, trying to keep my head above water. I reckon this is going to cause a tremendous amount of stress for a tremendous number of people.
We should be curious about how it does what it does when it seems to be reasoning.
More: https://skybrian.substack.com/p/dont-settle-for-a-superficia...
God of the gaps, reworked. Admittedly, a new claim to me. $10.05.
It’s not clear what the basis for your argument is.
Better read: the original paper demonstrating that prompting chain-of-thought reasoning elicits superior results. TLDR, give examples of working out the problem step-by-step, encourage GPT to emulate, and it's capable of solving challenges it would completely miss otherwise.
https://arxiv.org/abs/2201.11903
I can't try out the API right now so maybe that's working but the chat.openai.com app isn't
The advantage of the cheap API key is also that you can use it with all sorts of tools. CodeGPT for intellij is amazing for example. I have a vs code plugin and I found an extension for Google docs. And I've written a few small integrations myself for some private projects. All good value. And less than a dollar spent on tokens.
Let's say you have an elevator start on floor 11, and another at floor 3. The rules are, they can never occupy the same floor. Is it possible for the first elevator to ever reach floor 2? No, they would have to pass each other.
GPT-4 may be able to come to the conclusion by accident. But not reliably.
At best (and the is a valuable trait) the LLM would be able to reframe your problem in such a way that you could discover the necessary precision in the problem statement. Perhaps it could even encode it in some syntax for another tool.
__user__:
- A system in composed of two elevators. - Each have their own unique velocity parameter. - At each time step in the system, the elevators can move up or down a number of floor equal to their velocity, or change their velocity. - Assume that both elevators move at the same time step. - The elevators may never occupy the same floor.
- Elevator A starts on floor 11, and Elevator B at floor 3. - Both elevator velocities start at 1.
Is it possible for A to ever reach floor 2?
__assistant__:
No, it is not possible for Elevator A to ever reach floor 2 while maintaining the rule that the two elevators may never occupy the same floor.
To see why this is the case, note that Elevator A starts at floor 11 and Elevator B starts at floor 3. Since both elevators move at the same time step, Elevator A can never move to a floor lower than Elevator B without violating the rule that they may never occupy the same floor.
Furthermore, since Elevator A and Elevator B have the same velocity at the start and Elevator B is already on a lower floor, Elevator B will always reach any floor below Elevator A before Elevator A can get there. This means that Elevator A can never occupy any floor that is below the current position of Elevator B, including floor 2.
Therefore, it is not possible for Elevator A to ever reach floor 2 while maintaining the given constraints.
__user__:
what if at the first time step, both velocities became equal to 5?
__assistant__:
If both elevators have their velocities set to 5 at the first time step, then it is possible for Elevator A to reach floor 2.
With both elevators moving at velocity 5, Elevator A will reach floor 6 after the first time step, while Elevator B will reach floor 8. Since Elevator A is now on a lower floor than Elevator B, it is possible for Elevator A to continue moving down to floor 2 without violating the constraint that the elevators may never occupy the same floor.
Specifically, Elevator A can continue moving down at velocity 5, reaching floors 1 and 2 after two more time steps. Elevator B, with velocity 5, will reach floor 13 after the same two time steps.
So, if both elevators have their velocities set to 5 at the first time step, then it is possible for Elevator A to reach floor 2 while maintaining the given constraints.
At the same time, there's been multiple occurrences where it has produced math mistakes, despite the fact that it can tell me that it knows how to do "the right thing" e.g. there's been multiple times where it will incorrectly reduce integrals with Dirac delta functions, or introduce subtle mistakes in analytically known formulas.
The Dirac delta function example was funny to me -- it can tell me what to do when a delta function occurs in an integrand! Yet it makes a mistake reducing the same sort of integral in another context?
So I'm not sure how to take these "GPTX is doing reasoning" claims. On the one hand, maybe? On the other hand, it's not perfect at it -- even when it appears to have knowledge that should ensure that it never gets that wrong.
If I'm going to be using this system to help me do research, I'd love for it to be correct and not contradict itself. That's why I'm happy to have these GPTX layers shell out to something like Wolfram, or another symbolic system. That combination makes perfect sense to me. I'm also much more likely to trust that combination, instead of my current strategy - which is to basically re-process everything that GPTX gives me and assume that there are a few places where I'm likely to find mistakes.