This is an impressive demonstration of GPT-4's logic and reasoning capabilities, but the DM result uses a real-world reward signal (e.g. the list is sorted correctly) to validate its results whereas we can never be sure that GPT-4 outputs are hallucinated or not. Also this experiment would never be done if not for the DM paper in the first place.
Things exist or are created. We discover an existing thing or its application.
Discovery is principally a learning process (that may entail creative effort).
Invention is principally a creative process (that may entail learning effort).
The first statement is either wholehearted agreement with my remark or a rejection of Gödel numbering.
The second statement is similar but highlights the problematic distinction of the terms, since in such a framing they ascribe meaning solely and specifically to human intellectual processes, so one must also show either that we are alone in the universe, or (perhaps equally) that all cognitive entities share the same processes. This is of course merely dancing around questions of the universality of truth. Alas, only Wittgenstein knew the answer.
With Gödel, there is the space of the possible, the invention indeed being the axiomatic foundation that engenders the space of possibilities. If the space of possibilities is disconnected, intuitive jumps are required to arrived at unreacheable truths. So yes, here we are 'discovering' -- learning the membership set of true statements -- but these are existing things only in the context of the invention, which is why it is correct to say they are discovered in a relative sense.
This is hiding the assumption in plain sight in a different form. The question then would be: was x an existing thing that we "uncovered"/discovered (it has merely hidden) or we created it by thinking it into existence?
At least in terms of Mathematics, I do not think there is an objective answer for this question.
The definitions you provided are, IMO, just moving the problem to further concepts, that are also ill-defined. "Discovery mainly a learning process, Invention mainly a creative process", could mean almost anything. Also, as a rigorous definition, "mainly" does not cut it. It needs to be 100%.
Instead of pretending I know it is my opinion to accept they are merely two ways of looking at the same thing. At least, this is my position for now, until a proper argument emerges.
P.S. I see it as arguing which of 0.5 or 1/2 is the "true" number.
The paper shows several distinct improvements to a sorting algorithm, and presents evidence that the process is generally applicable. This tweet points GPT-4 at 20 instructions and asks if any can be removed, and it finds one optimisation.
That's good to see from GPT-4, but the comparison seems disingenuous, and I'd expect more from someone in academia.
I think this is a fair critique! The original tweeter was being condescending, probably to Increase Engagement. This is something social platforms should systemically discourage!
I just tried feeding GPT-4 some assembler as well, and while I did get it to make some optimisations, I also had to point out several mistakes it made (e.g. it moved one instruction to the other side of a label that'd cause a register to have the wrong value if a preceding branch was taken). It made mistakes that I've seen people new to assembler make, but happened to get some things right. When I pointed out an obvious optimisation it had missed, it applied it "half right" (I suggested it might look at collapsing multiple stack allocations because they didn't overlap, but while it collapsed them into a single allocation, it applied the sum of the allocations instead of just the largest one, which was a pointless increase in stack use). Also the kind of misunderstanding I could see with a junior dev with little assembly experience fall for if they recognise what you point to but don't quite understand the implications.
It's "close enough" to be worth trying to see if it hits on something useful, but also bad enough that you really need to carefully scrutinise what it comes up with, and probably need to write a much more comprehensive prompt about strategies it should and shouldn't try to apply to make it more generally usable.
> I just tried feeding GPT-4 some assembler as well, and while I did get it to make some optimisations, I also had to point out several mistakes it made
My point of view on this is that this type of AI cannot reason, so it seems to me quite audacious to use it for programming. Following the AI's suggestion is like examining a PR from someone you don't know: don't they try to insert a backdoor or a subtle bug?
Even if you use it as some kind of "rubber duck", examining its suggestions - doing the necessary reasoning and the double checks - is pretty time consuming. Wouldn't the long term result be better if one spent some time to acquire a deeper knowledge of the field, rather than play the "prompt engineering" game (of which the rules might change next month)?
It feels like we are in the equivalent of programming language's "honey moon" [1] phase with this kind of technology. People will eventually get over the "wow" effect and use it for what it is good at.
For making new stuff, I agree with you that it's pretty nutty. I'd never ask it to try to invent something novel, the way this does, because it's pretty high-noise. I do think it has significant programming advantages insofar as it can analyze existing code and make reasonable permutations of it, though, especially for stuff that's able to effectively analyze your current set of code.
"Would it be better to acquire a deeper knowledge"--for the cases described above, actually no? I already have that deeper knowledge. I don't have the attention span (both because this stuff can be boring and because I am frequently randomized) and for a lot of this stuff the gate is actually not solving the problem but the physical act of typing, or even copying and changing. Copilot, at least, absolutely does create bugs, but in my experience they're mostly bugs of the form "this actually isn't typesafe in the way that it's written" and that makes isolating and fixing them clear. (Rarely are those bugs logic bugs, just a burp in the output.)
There is a very clear future where this turns into incredible copy-paste garbage because it's not capable of refactoring, and that's a different problem. Avoiding it is why they pay us the medium bucks.
But some other models had gotten the right answer.
This points to some evidence of being able to derive logical reasoning. Just because the model is "doing statistical guessing of the next word", doesn't mean this process does not produce logical reasoning as an outcome.
It's also not at all clear that reasoning is anything more than "doing statistical guessing of the next word" or something very similar. We clearly have some other aspects, but it's not clear it's all that much beyond ongoing stimulus and memory in terms of capabilities.
It can only answer questions it has been trained to know the answers to. Ask it an original question that isn't in the training set and it gets it wrong. This is clear evidence that it does not possess logical reasoning skills.
This is simply not true. I've asked it many questions it has not been trained to know the answers to because they involve e.g. code it won't have seen in the wild, and gotten the right answer back.
In one of my other responses I dug into how I'd asked it to analyse a piece of assembly generated with a compiler I've written, from a test program that didn't exist at GPT4's knowledge cutoff, and it did just fine.
I'm not saying it has to be trained on the specific question directly, it has to be trained on similar data. It has a decent knowledge of programming languages because it has been trained on a lot of data surrounding them.
Being capable of logic and reasoning is more of an illusion that arises as a result of the enormous amount of training they put in. You can make simple questions that it can't answer, so you can't claim it has logical reasoning if it fails at simple tasks. It just looks a lot like it has logical reasoning because it has been trained on text displaying logical reasoning.
I've had plenty of luck using it for programming, and getting it to output working code. It's still best for things that is either boilerplate or where it's easy to test, but I've gotten it to write e.g. parts of parsers, a DNS server, and a working web app and many other things. I wouldn't trust complex pieces of code that I can't quickly verify, but I wouldn't do that with a human developer either.
Getting it to symbolically evaluate code it most certainly have never seen before has also worked well for me, as have getting to explain the hypothetical outcome of executing a piece of code written using a made up combination of two languages. Frankly, I've worked with plenty of developers who'd struggle to reason about simpler tasks than some of the stuff I've successfully thrown at GPT4, so I'd argue that if it can't reason then many humans can't either.
It also fails at some very basic things. The biggest challenge is that it's reasoning is different to humans, and it's terrible at some things we consider basic even as it can do other things at a far higher level.
> Wouldn't the long term result be better if one spent some time to acquire a deeper knowledge of the field, rather than play the "prompt engineering" game (of which the rules might change next month)?
"Prompt engineering" works best in fields you do have a deep knowledge of, where you can tell whether you're actually getting better solutions. I'm willing to put effort into that too is acquiring deeper knowledge of the field by exploring dependencies and getting an understanding of ambiguities of language we use to talk about a problem. While you can get bogged down in specific "incantations", the more important part of "prompt engineering" is getting better at recognising under-specified problems. In fact, one of the things I've toyed with is to feed it explanations of a problem and instead of asking it to solve it, I first ask it to suggest improvements and ask questions, and that will often reveal problems with the prompts with little effort.
With respect to assembler, it seems to hit one of the bad spots, in that you get far fewer textual structural hints. E.g. a label can be the start of a function or the start of a basic block, or part of a loop, or part of an if/else block, and you largely need to recognise contextual clues and know the patterns of whatever produced the asm to know which rules applies. E.g. one of the issues I ran into was that the asm I gave it was output from a compiler that used %esi in a very specific way. There was no way for GPT4 to know that from the sample it was given without "thinking it through" (see below), and it made a "guess" that many novice asm programmers also likely would've, but where someone more experienced would've stopped and demanded a confirmation of the calling convention.
That said, to the test I mentioned in the previous comment, I tried giving it the code in a fresh chat, and asked it to reason about the code, and to give me a list of information that I could provide that would make it easier to suggest optimisations.
When I explicitly asked it about the calling convention used for %esi in the code I gave it, it correctly reasoned its way through it by observing how it was treated within the given code. That's something an smart beginner at asm might spot, and an intermediate asm programmer pretty likely would spot, but the important part is that it was clearly able to explain why with references to the code.
It also when asked correctly recognised that the object was tagged (low bit used to indicate it's not a "real" pointer but a value object), and recognised vtable calling pattern that few developers without compiler experience would pick up on.
Frankly, after asking it these follow up questions my opinions on its understanding of asm is way...
Is it possible that GPT-4 already had this new optimization somewhere in its training set? The optimizing patch DeepMind published on has been floating around for several months at this point…
Edit:
I found the merges for sorting (Jan 2022) [1] and hashing (Jan 2023) [2]. Both of these are very plausibly in the training set for GPT-4, which was frozen sometime in March 2023.
No, it's not necessary for it to have had the optimization somewhere in its training set given how it works. LLMs can solve problems they were not trained on, with varying degrees of success.
Wow no reinforcement learning needed! Amazing to think, all they would've had to do instead is spend a decade or two building towards a 1 trillion parameter transformer model and spend 100m or two training it. Then fine-tune the model using.. y'know nevermind : )
Very impressive, though you are using a lot of prior knowledge to guide it towards the solution you already know. The RL version presumably wasn't given the hint that it could remove an instruction.
Still very impressive, but I'd like to see it work on some other assembly for which the "answer" isn't already known.
Since they brute forced all solution programs for this case anyway, Deepmind already admitted that they didn't even need to use AlphaDev in this particular case. GPT-4 being able to work it out is great, but it says nothing about the value of what the AlphaZero/Dev algorithm and the kinds of interesting problems you can solve with it.
> AlphaDev uncovered faster algorithms by starting from scratch rather than refining existing algorithms
Finding that specific optimization, especially when given the comments, seems almost trivial by comparison.
Edit: I tried to understand the optimization in question. This is not the full sort3 algorithm, but only under the assumption that B < C. In that case the GPT-4 answer is actually wrong because it wasn't given that assumption.
I've gotten extremely odd algorithms and solutions that saved me lots of resources.
Lots of bitshifting and unsigned ints and approximations these days. Stuff I would have never thought of, and then I can talk it through how something else only I thought of would be even more applicable, and it refines it for that too! Great pair programmer!
This level of competence will never show up in a live no-resource interview or any (watched) time trial I do.
I think that shared chats all use the green icon. If you check the source code, it says "model_slug":"gpt-4." It will also state that "the previous model used in this conversation is unavailable" if you try to continue the conversation on a free account.
GPT-4's explanation of its optimization does not make sense to me. It writes
"Instead of moving it to P, we can directly use S in the following comparisons, saving one instruction." but then proceeds to use P as if that mov had happened.
AlphaDev's optimization relies on the fact that B and C are already in the correct order. This precondition is missing from the prompt given to GPT-4. It seems that GPT-4 is hallucinating something that only resembles the correct optimization at first glance.
There was already a paper on Transformer models and their inability to redefine variables. They had real problems with swapping variables and would forget that it had happened.
And that's not enough. Think of the swap sequence t=a,a=b,b=t: all three moves are necessary.
While removing moves is a common optimization, it can't be done willy nilly.
As mentioned in the comment you replied to, the algorithm is wrong without the extra precondition B<C that is not included in the prompt, as should be clear from the fact that the largest value is computed as max(A,C) rather than max(A,B,C). This is the same precondition that allows to remove the move, effectively computing the smallest value as min(A,B) instead of min(A,B,C).
So the only correct answer to the prompt is to find a counterexample which does not sort correctly, for example 3 7 5. Not to remove an instruction randomly.
Isn't it something like some interviewer who already knows the answer asks some interviewee a very specific question designed with a certain level of guidance in mind? It's still impressive for LLM, but it's not even apple-to-orange comparison.
I posted a reply, but I think ultimately this is just a coincidence. There's a naïve reason that "mov S P" looks redundant ("just use S instead of P later"), but in typical GPT fashion, this is specious, and can't actually be done. It's essentially trying to swap two variables without using a temporary. If x86 had a conditional swap instruction, it could, but it doesn't, and just doing "cmp,cmov,cmov" can't handle it.
Another giveaway is that removing that line in the real optimization changes the output of the provided snippet if C < B. It feels like a hard sell to say GPT picked this line for that subtle reason based on information not provided, but explained it with something only correct at surface level.
I agree. I don’t think hallucination is the right term. When humans hallucinate, they’re seeing false inputs. When ChatGTP “hallucinates”, it’s giving false outputs. Maybe a better analogy would be a savant with dementia - it knows a lot of things and sounds coherent but it’s inconsistent in keeping track of state and highly variable in the accuracy of its output
Actually no they don't, those neurons are extremely complex machines internally and in their own right. Reducing what they are doing to "propagating electrical impulses" is flat out wrong.
Trying to attribute intelligent intent when there is none. Human brains are just an amalgamation of 20 amino acids and other organic molecules that some bunch of DNA produced to copy themselves. Or: Human brains are ultimately just particles following the unavoidable rules of Quantum Mechanics.
My point being, we are the result of a bunch of DNA molecules selected to make more copies of itself. Our neurons, whatever their internal structure, still reduce to just chemical and physical processes. Yet, here we are. Reductionism is not the way to dismiss emergent phenomena like intelligence, sentience or consciousness.
The problem is that the scale of "computing" going on in that particle soup is EXPONENTIALLY more than what even the largest "neural net" projects can hope to achieve. Internal chemical and signalling gradients are probably an important part of brain functioning.
I have played around with this a bit and it does actually work. I think it is similar to when people tell you to “think creatively”. It doesn’t actually mean anything, but it can change you internal biases for what you say.
ChatGPT-4 “knows” it is an LLM and and that temperature=0 implies a less chaotic more ordered way of thinking. I can easily see that having an effect on what it says
69 comments
[ 3.9 ms ] story [ 136 ms ] threadThis may however be distinction without a difference, since it corrodes the intellectual form of neither, and the processes are indistinguishable.
Discovery is principally a learning process (that may entail creative effort). Invention is principally a creative process (that may entail learning effort).
The second statement is similar but highlights the problematic distinction of the terms, since in such a framing they ascribe meaning solely and specifically to human intellectual processes, so one must also show either that we are alone in the universe, or (perhaps equally) that all cognitive entities share the same processes. This is of course merely dancing around questions of the universality of truth. Alas, only Wittgenstein knew the answer.
With Gödel, there is the space of the possible, the invention indeed being the axiomatic foundation that engenders the space of possibilities. If the space of possibilities is disconnected, intuitive jumps are required to arrived at unreacheable truths. So yes, here we are 'discovering' -- learning the membership set of true statements -- but these are existing things only in the context of the invention, which is why it is correct to say they are discovered in a relative sense.
This is hiding the assumption in plain sight in a different form. The question then would be: was x an existing thing that we "uncovered"/discovered (it has merely hidden) or we created it by thinking it into existence?
At least in terms of Mathematics, I do not think there is an objective answer for this question.
The definitions you provided are, IMO, just moving the problem to further concepts, that are also ill-defined. "Discovery mainly a learning process, Invention mainly a creative process", could mean almost anything. Also, as a rigorous definition, "mainly" does not cut it. It needs to be 100%.
Instead of pretending I know it is my opinion to accept they are merely two ways of looking at the same thing. At least, this is my position for now, until a proper argument emerges.
P.S. I see it as arguing which of 0.5 or 1/2 is the "true" number.
That's good to see from GPT-4, but the comparison seems disingenuous, and I'd expect more from someone in academia.
And delivered with sarcasm and condescension. Not good to see, but par for the course on social media.
It's "close enough" to be worth trying to see if it hits on something useful, but also bad enough that you really need to carefully scrutinise what it comes up with, and probably need to write a much more comprehensive prompt about strategies it should and shouldn't try to apply to make it more generally usable.
My point of view on this is that this type of AI cannot reason, so it seems to me quite audacious to use it for programming. Following the AI's suggestion is like examining a PR from someone you don't know: don't they try to insert a backdoor or a subtle bug?
Even if you use it as some kind of "rubber duck", examining its suggestions - doing the necessary reasoning and the double checks - is pretty time consuming. Wouldn't the long term result be better if one spent some time to acquire a deeper knowledge of the field, rather than play the "prompt engineering" game (of which the rules might change next month)?
It feels like we are in the equivalent of programming language's "honey moon" [1] phase with this kind of technology. People will eventually get over the "wow" effect and use it for what it is good at.
[1] https://earthly.dev/blog/brown-green-language/
"Would it be better to acquire a deeper knowledge"--for the cases described above, actually no? I already have that deeper knowledge. I don't have the attention span (both because this stuff can be boring and because I am frequently randomized) and for a lot of this stuff the gate is actually not solving the problem but the physical act of typing, or even copying and changing. Copilot, at least, absolutely does create bugs, but in my experience they're mostly bugs of the form "this actually isn't typesafe in the way that it's written" and that makes isolating and fixing them clear. (Rarely are those bugs logic bugs, just a burp in the output.)
There is a very clear future where this turns into incredible copy-paste garbage because it's not capable of refactoring, and that's a different problem. Avoiding it is why they pay us the medium bucks.
but it can answer the glass door push or pull question (see https://news.ycombinator.com/item?id=35506666), albeit a bit off (it didn't understand the mirrored writing).
But some other models had gotten the right answer.
This points to some evidence of being able to derive logical reasoning. Just because the model is "doing statistical guessing of the next word", doesn't mean this process does not produce logical reasoning as an outcome.
In one of my other responses I dug into how I'd asked it to analyse a piece of assembly generated with a compiler I've written, from a test program that didn't exist at GPT4's knowledge cutoff, and it did just fine.
Being capable of logic and reasoning is more of an illusion that arises as a result of the enormous amount of training they put in. You can make simple questions that it can't answer, so you can't claim it has logical reasoning if it fails at simple tasks. It just looks a lot like it has logical reasoning because it has been trained on text displaying logical reasoning.
Getting it to symbolically evaluate code it most certainly have never seen before has also worked well for me, as have getting to explain the hypothetical outcome of executing a piece of code written using a made up combination of two languages. Frankly, I've worked with plenty of developers who'd struggle to reason about simpler tasks than some of the stuff I've successfully thrown at GPT4, so I'd argue that if it can't reason then many humans can't either.
It also fails at some very basic things. The biggest challenge is that it's reasoning is different to humans, and it's terrible at some things we consider basic even as it can do other things at a far higher level.
> Wouldn't the long term result be better if one spent some time to acquire a deeper knowledge of the field, rather than play the "prompt engineering" game (of which the rules might change next month)?
"Prompt engineering" works best in fields you do have a deep knowledge of, where you can tell whether you're actually getting better solutions. I'm willing to put effort into that too is acquiring deeper knowledge of the field by exploring dependencies and getting an understanding of ambiguities of language we use to talk about a problem. While you can get bogged down in specific "incantations", the more important part of "prompt engineering" is getting better at recognising under-specified problems. In fact, one of the things I've toyed with is to feed it explanations of a problem and instead of asking it to solve it, I first ask it to suggest improvements and ask questions, and that will often reveal problems with the prompts with little effort.
With respect to assembler, it seems to hit one of the bad spots, in that you get far fewer textual structural hints. E.g. a label can be the start of a function or the start of a basic block, or part of a loop, or part of an if/else block, and you largely need to recognise contextual clues and know the patterns of whatever produced the asm to know which rules applies. E.g. one of the issues I ran into was that the asm I gave it was output from a compiler that used %esi in a very specific way. There was no way for GPT4 to know that from the sample it was given without "thinking it through" (see below), and it made a "guess" that many novice asm programmers also likely would've, but where someone more experienced would've stopped and demanded a confirmation of the calling convention.
That said, to the test I mentioned in the previous comment, I tried giving it the code in a fresh chat, and asked it to reason about the code, and to give me a list of information that I could provide that would make it easier to suggest optimisations.
When I explicitly asked it about the calling convention used for %esi in the code I gave it, it correctly reasoned its way through it by observing how it was treated within the given code. That's something an smart beginner at asm might spot, and an intermediate asm programmer pretty likely would spot, but the important part is that it was clearly able to explain why with references to the code.
It also when asked correctly recognised that the object was tagged (low bit used to indicate it's not a "real" pointer but a value object), and recognised vtable calling pattern that few developers without compiler experience would pick up on.
Frankly, after asking it these follow up questions my opinions on its understanding of asm is way...
Apparently you haven't been following many academics on Twitter.
Edit: I found the merges for sorting (Jan 2022) [1] and hashing (Jan 2023) [2]. Both of these are very plausibly in the training set for GPT-4, which was frozen sometime in March 2023.
[1] https://reviews.llvm.org/D118029
[2] https://github.com/abseil/abseil-cpp/commit/74eee2aff683cc7d...
> GPT-4 generally lacks knowledge of events that have occurred after the vast majority of its data cuts off (September 2021)
https://openai.com/research/gpt-4
Isn't it necessary for GPT-4 to have had the optimization somewhere in its training set given how it works?
Still very impressive, but I'd like to see it work on some other assembly for which the "answer" isn't already known.
> AlphaDev uncovered faster algorithms by starting from scratch rather than refining existing algorithms
Finding that specific optimization, especially when given the comments, seems almost trivial by comparison.
Edit: I tried to understand the optimization in question. This is not the full sort3 algorithm, but only under the assumption that B < C. In that case the GPT-4 answer is actually wrong because it wasn't given that assumption.
Lots of bitshifting and unsigned ints and approximations these days. Stuff I would have never thought of, and then I can talk it through how something else only I thought of would be even more applicable, and it refines it for that too! Great pair programmer!
This level of competence will never show up in a live no-resource interview or any (watched) time trial I do.
AlphaDev's optimization relies on the fact that B and C are already in the correct order. This precondition is missing from the prompt given to GPT-4. It seems that GPT-4 is hallucinating something that only resembles the correct optimization at first glance.
> However, in the lines following the removal, P should be replaced with S to maintain correctness.
While removing moves is a common optimization, it can't be done willy nilly.
As mentioned in the comment you replied to, the algorithm is wrong without the extra precondition B<C that is not included in the prompt, as should be clear from the fact that the largest value is computed as max(A,C) rather than max(A,B,C). This is the same precondition that allows to remove the move, effectively computing the smallest value as min(A,B) instead of min(A,B,C).
So the only correct answer to the prompt is to find a counterexample which does not sort correctly, for example 3 7 5. Not to remove an instruction randomly.
Another giveaway is that removing that line in the real optimization changes the output of the provided snippet if C < B. It feels like a hard sell to say GPT picked this line for that subtle reason based on information not provided, but explained it with something only correct at surface level.
Edit: done. Not sure how I missed that. Thanks!
When it screws up, we call it a hallucination. With careful prompting, you can get it to screw up in a way that works.
Color me unimpressed.
My point being, we are the result of a bunch of DNA molecules selected to make more copies of itself. Our neurons, whatever their internal structure, still reduce to just chemical and physical processes. Yet, here we are. Reductionism is not the way to dismiss emergent phenomena like intelligence, sentience or consciousness.
ChatGPT-4 “knows” it is an LLM and and that temperature=0 implies a less chaotic more ordered way of thinking. I can easily see that having an effect on what it says