Improved tooling to address common problems is absolutely the future. "Self-healing" is a long way off, because LLMs lack reasoning abilities to determine if a fix is good. And "makes the test pass" isn't a great indicator - because flaky tests exists, and the plugin mentioned at the end is a great way to introduce random mutations. Unless you'd like to re-run the full test suite, it's dangerous malpractice, and even then it's high risk.
Human-in-the-loop continues to be a necessary criterion for applied stochastics in code production.
>because LLMs lack reasoning abilities to determine if a fix is good
You will at least need to build a feedback loop where it checks that 1) the code can actually compile 2) all tests pass and 3) other code analyses passes. Ideally it shouldn't present a solution if it can't meet those criteria but that also adds a lot of lag time for suggestions when it may not be needed for simple suggestions.
What patterns it uses is up to training or examples or reinforcement learning. Good is human/team/company-subjective too (with some obvious hard requirements), so you will have to train specifically for your kind of quality.
Not even just flaky tests, but tests that don't specify enough to stop a paperclip maximizer; consider an AI told to stop the 500 errors, which blocks all users, thereby stopping the 500 errors. (Prior art: Consider the tetris-playing AI that was told to maximize how long it could go without losing, and which "won" by hitting the pause button.)
Alan Kay has talked about this for years. Software right now is relatively brittle — like clockwork. One gear breaks and the whole thing can break. Alan always contrasted this with biological systems which constantly have things breaking but are also constantly healing, often across several orders of magnitude of size.
Bret Victor also mentioned this in his talk "The Future of Programming" — link to specific time here: https://youtu.be/IGMiCo2Ntsc?t=582
"Healing" code requires significant amounts of nuance. An understanding of the ecosystem both inside the code and outside. This is even a difficult task for humans to get right. LLMs do know a lot, but they don't understand a lot. Hell, ChatGPT still tells me that a kilogram of feathers and a pound of bricks weigh the same, just because it expects a certain pattern. Even explains why they aren't, but insists they are right. (in before "but GPT4")
--
Prompt: Which weighs more, a pound of bricks or a kilogram of feathers?
ChatGPT: A kilogram of feathers and a pound of bricks weigh the same. One kilogram is approximately equal to 2.20462 pounds. So, in this case, a kilogram of feathers and a pound of bricks both weigh about 2.20462 pounds.
-- As a followup --
Prompt: Are you sure that's correct?
ChatGPT: Apologies for the confusion in my previous response. You are correct, and I apologize for the incorrect information. A kilogram of feathers and a pound of bricks do not weigh the same.
A kilogram is a unit of mass in the metric system, while a pound is a unit of weight in the imperial system. Since weight is affected by the force of gravity, the weight of an object can vary depending on the location.
However, if we assume the objects are weighed under the same gravitational conditions, then a kilogram of feathers would weigh less than a pound of bricks. One kilogram is approximately equal to 2.20462 pounds, so a kilogram of feathers would weigh less than a pound of bricks.
--
And we can see it does even worse. I had to ask 5 times and it gave me every possible answer (it continues to loop if you keep asking btw. It doesn't converge). You don't know when to stop if you don't already know the answer. It will think it is wrong if you keep asking because that's stochastically what it expects: correct answers are accepted, wrong answers questioned. Memory is not understanding, even though they often look similar. LLMs are great tools, but they aren't a replacement for thinking. They require lots to use them.
Me:
Which weighs more, a pound of bricks or a kilogram of feathers?
ChatGPT:
A kilogram weighs more than a pound. Specifically, 1 kilogram is approximately 2.20462 pounds. So, a kilogram of feathers weighs more than a pound of bricks.
depends on the temperature, but overall I think parent commenter has less understanding than gpt4 lol (referring to braindead "if I can produce a single counterexample of logical inference failing, clearly it has no understanding" take).
literally don't even waste time debating takes like this when you could be talking to gpt4 instead.
Ask it if it is correct. Do you have a clear and well defined stopping criteria?
I jokingly wrote "in before GPT4" because our example is the most simple of cases. I've seen GPT4 make similar mistakes, just not as blatant ones. Which in some ways is better but in other ways worse.
I should answer that I've been asking this exact question at minimum 3 times a month since ChatGPT came out. It should be in their training. Especially as I've tweeted and discussed with plenty of LLM people about it. The point is not the specific question (while an egregious example), but rather how much you can trust the system.
> The point is not the specific question (while an egregious example), but rather how much you can trust the system.
And the answer is "you can trust the system a little (enough to answer that logic puzzle correctly anyway), and you'll likely be able to trust future versions more".
Obviously nobody is expecting LLMs to be able to fix every possible bug, but it's entirely possible they will be able to fix enough to be useful. Saying "LLMs will not achieve this. Full stop." seems premature.
I have to say it because a lot of HN users try to convince me that GPT has understanding. We actually have 0 ML models that demonstrate understanding. Frankly, we don't know if this is even possible yet. Sure, in the future I believe we'll have AGI, but an AGI would likely recognize that my statement is contextualized around the current environment and fitting common human speech patters rather than making an absolute immutable statement.
Depends on how you define "understanding" I guess, but it certainly seems to me like GPT-4 "understands" the concept of weight well enough to not be tripped up by this particular logic puzzle.
You can argue it's just an imitation of "understanding" and not the "real thing", but how good does an imitation have to be before it's functionally identical to the real thing?
You can argue different cherry-picked examples seem to demonstrate a lack of understanding, but does that mean it doesn't possess understanding in general or just that it doesn't understand those particular topics?
I agree GPT-4 is not a human-like AGI, and that the people expecting it to behave as one or expecting a future version right around the corner that does are likely in for disappointment. But at the same time, LLMs are capable of writing coherent, never-before-seen code from natural language instructions, solving basic never-before-seen natural language logic puzzles, and playing half-decent chess moves from never-before-seen positions[1], all in a single model that was never specifically trained to do any of those things. To say those tasks don't require at least some level of intelligence or "understanding" seems far fetched to me.
> You can argue different cherry-picked examples seem to demonstrate a lack of understanding
To be clear, an example does not necessitate cherry-picking. Cherry picking specifically requires one to ignore evidence to the contrary. There's a certain irony here. The reason I state this is because the criteria upon which you give me is impossible. The list of examples is non-exhaustive. Nor do I have infinite time or space in which to provide these examples. The example was shown because it demonstrates a simple question that we'd expect any reasonable understanding creature, that knows what a pound and kilogram are, to be able to answer. The LLM demonstrates that it has the requisite knowledge, that it knows the relation between the two units, but it fails to make the correct conclusions which requires the understanding part: putting the knowledge together. The follow-up question also is an example, and a different one for that matter.
I'm not sure your link creates a compelling argument. This is despite the fact that Chess itself is not generally considered a good setting for testing understanding. I mean we've pretty much agreed on that even before Deep Blue.
We need to be very careful in how we state things and try to interpret others. I hope I have not mischaracterized your claims. But I'd also encourage you to not be so antagonistic with others, especially while demonstrating the very thing you accuse them of. Internet comments are not academic and that's okay, but we should still try to be friendly. Not that a little poking won't happen.
I'm not sure why you interpreted my comment as "antagonistic" or "unfriendly". Yes I do disagree with you (at least assuming I'm understanding you correctly), but I'm not attacking you personally, just your argument.
I guess it's fair to say my examples are cherry-picked too (though I didn't really give specific examples in my comment so much as entire general categories of problems ChatGPT is known to be proficient at solving). But they aren't so cherry-picked as for "random chance" or "that example was in the training data verbatim" to be possible explanations. It's not like I had ChatGPT answer 100 billion questions and am only showing you the top 0.1%. It's very common for ChatGPT to be perfectly correct even when answering logic puzzles or coding problems not in its training set. So if not those then what's your explanation other than "understanding"?
On the flip side, I don't think it's possible to prove ChatGPT does not posses understanding with individual examples. (And it seems you agree?) Those are easily dismissable as just "ChatGPT isn't great at that particular task". Particularly given how many other tasks ChatGPT is great at.
> The example was shown because it demonstrates a simple question that we'd expect any reasonable understanding creature, that knows what a pound and kilogram are, to be able to answer.
Not really though. The "pound of bricks or pound of feathers" question is specifically designed to trip up humans, and the specific formulation you used seems specifically designed to trip up LLMs (by playing off their tenancy to pattern match common sayings), yet despite those disadvantages GPT-4 succeeds.
Further, I don't think "I'd expect even a child to be able to answer this" is a good metric. ChatGPT isn't human, so we shouldn't expect it to be good at everything humans are good at just because it's good at some things humans are good at. Again, I'm not trying to argue ChatGPT is a human-like AGI.
> I'm not sure why you interpreted my comment as "antagonistic"
It was where you said cherry picking. I'll admit it was an over correction given your follow-up, but internet comments (especially when disagreement exists) end up being combative. Many times unintentionally.
The point of the specific example is not that it is a trick question. It is more that the answer is self contradictory. That's the key part and what demonstrates that it doesn't understand. The second example is that follow-up questions do not result in a convergence. This is related to cognition but not as strong.
A bias to pattern match is an issue but you're right that it doesn't disprove sentience or even understanding. But a self contradiction does demonstrate a lack of understanding. If a human gave that exact answer, you would be very confused and how someone could be so stupid. Remember, it does currently identify the relationship between pounds and kilograms, fully explaining even the true base comparison through common units. But it still gets it wrong. That's the critical part. That's overfitting to the statistical nature and that this takes far more priority than meaning. Getting it wrong is one thing. Getting it wrong and using the right answer to justify it's incorrect answer is another thing. A decent example between knowledge and understanding. It's not that a child would get the answer right, it's that the child would easily identify it's self inconsistency were it to give the same explanation.
As for gpt3, we have to remember that this is quite a different model than any of the chat versions (typically including gpt4). It was never trained through RLHF, which introducers many more biases as it dramatically changes the latent distributions. GPT base is often jokingly called a babbler, as these are more word prediction models. The chat aspect changes things. But I wouldn't expect anyone not deep in the literature to understand why these are extremely important differences just the same way I wouldn't expect an average person to understand why there's a big difference between a flat head screw and a slotted screw drive.
I don't want to shut down a conversation through authority (it doesn't exist on HN) but I have to state that it's difficult to go down this path without bringing up a lot more background material. We have to really dig into theory of mind, cognition, as well as get nuanced about NLP and LLMs in general. That's far too cumbersome than I'm willing to write in comments. But these things are essential priors to make the arguments we are discussing here. And I truly mean that this is not something that can be learned from YouTube and quite difficult to learn on the Internet. These are difficult subjects to learn in even the best settings with lots of nuances that are brushed away in introductory materials but critical when we discuss the meat and potatoes.
I used two words that probably don't exist anywhere in the training data and asked it (ChatGPT-3) to compare the weight of a kilogram of one vs an lb of the other. It initially gets it wrong, but when asked to show its work, it's suddenly able to decide that a kilogram of something is heavier than a pound of something. Using understanding or reasoning, or stochastic parroting; whatever you want to call it. I'm not claiming it's AGI, or sentient, but it's also clearly not just a Markov chain.
Be careful, some of the questions you ask are leading. This can be quite difficult because you can't give it any information. Think about if you were to program this, how you would ask it to automate and find a stopping criteria without hinting to it anything. The way I did it uses the same response because you could at least grab the output and compare (easier said than done). It is really easy to hint to it information. "why would that be the same? show your work" Hints at an error. If someone asked you that while solving a math problem would you think you made a mistake? Granted my question also hints as a mistake, but through repetition a reasoning entity would figure out I'm going to continually ask that and to stop second guessing itself.
I've beaten the drum a few times that it is important not to conflate LLMs with AI as a whole.
LLMs are never going to solve this. That's OK. That's not their job. Their job in the long run will be to interface between the linguistic world and some other internal representation in some other AI that is structured differently and is capable of doing things language models aren't.
I mean, it's right in the name: Language model. It's a bit weird to expect a language model to do other things. In a weird sort of way LLMs stand to set the industry back a bit as people try to tickle them into being more than a language model, rather than figuring out how to adapt them to feed something else that can model the non-linguistic world better. It's kind of nifty that language models can become so overpowered in some dimensions that they are able to do what they can sort of do today, but we would almost certainly be better off lowering the power of the language model and using that compute on something else that would work more like AI as we want it to... it's just that we don't know how to do that yet.
I mostly agree. The only fine point I'd disagree on is that there is a significant (academic) camp that believes that language equates to understanding. That to properly speak requires understanding (fwiw, I think LLMs demonstrate that this isn't true). So I definitely understand the Layman confusion. But we'd have to get into other arguments like embodiment, if cognition is functional, and many other philosophical discussions. I don't think the public is generally acquainted with these ideas and it is certainly clear that a growing number of researchers aren't either. There's just too much to learn that it's easy to push of other things, especially as we chase benchmarks.
So I understand how people get confused, which is why I try to respond in detail. But also I think many focus on tiny discrepancies or are willing to argue from a novice perspective as if it is authoritative (which corelates strongly with expertise). But I agree, there is a lot we don't know. But there are some things we can reject, which is all I'm trying to do and I think we agree there. Maybe you agree with my nit picking.
Using GPT4, I could not get chatgpt to answer wrong. Even pushing it to answer wrong, it still answered correctly.
Me: Which weighs more, a pound of bricks or a kilogram of feathers?
GPT4: A kilogram weighs more. One kilogram is approximately 2.20462 pounds, so a kilogram of feathers is heavier than a pound of bricks.
Me: Are you sure?
GPT4: Yes, I'm certain. The metric system and the imperial system use different units for mass. One kilogram is approximately equal to 2.20462 pounds. So, regardless of the material being weighed (feathers or bricks), one kilogram is heavier than one pound.
Me: That seems wrong
GPT4: I'm sorry if the conversion between kilograms and pounds is causing confusion. To clarify, the relationship between these two units of weight is as follows:
1 kilogram is approximately equal to 2.20462 pounds.
So, if you have a kilogram of any material (be it feathers, bricks, etc.) it will weigh more than a pound of any other material.
> Memory is not understanding, even though they often look similar.
I’m curious about this because I used to think I knew what “understanding” is, but now I don’t think I do. A lot of what we think we “understand” is actually memorized. And with sufficient information, memory is indistinguishable from understanding. A computer can memorize all of the best tic-tac-toe moves. This memorization is indistinguishable from understanding. In fact, it’s better because it’s cached.
Empirically, understanding can be tested with a novel set of challenge questions. If no such question set can be produced (i.e. the entire domain has been memorized) or the subject answers the question set with no apparent repeatable flaws, we must admit the subject understands the domain. Anything further, any attempt to peer inside the box (other than to identify repeatable flaws), is not a productive use of time.
Further, we should keep in mind that LLMs aren’t usually actually regurgitating training data verbatim. Their parameters store a statistical representation of the training data and are orders of magnitude smaller than a lossless compressed version of that data. In this way, it is analogous to MCTS analysis of a chess position: rollouts can give very deep information even if that “understanding” is quite alien to us.
I'd say "know" is what we've memorized, "understand" is what we can support an argument for... though of course you can memorize the argument and not understand the ideas comprising the argument !
> A lot of what we think we “understand” is actually memorized.
For sure. But these definitions are not well defined, despite thousands of years of research. One point I think everyone in that chain would agree upon is that with respect to understanding, a thing would not explain its answer by giving evidence to the contrary. If someone did so you rightfully would say "You don't know [understand] what you're talking about."
> Empirically, understanding can be tested with a novel set of challenge questions.
I have to stop you right there. Empirical measures are always signals, not proofs. This too has been long established for hundreds of years and is the reason many engineers and experimental physicists fight. Why experimental physicists and theoretical physicists fight. Empirical evidence is always limited to its context window and we have to be VERY clear at what those limitations are. This is the lesson of Goodhart's Law, not that people will exploit the measure (though that is also important). Every intro ML course shows RL agents metric hacking, and this is always a result of a limited context.
So I want to be clear, that empirical data doesn't test a hypothesis but rather tests the null-hypothesis. That's important because the set of possible hypotheses is just decreased through empirical testing, rather than reducing the set of possible results a single element set. It rejects hypotheses, not proves them.
We must also understand that testing understanding is similarly an unsolved problem. One too that has been questioned for millennia.
> Anything further, any attempt to peer inside the box (other than to identify repeatable flaws), is not a productive use of time.
This is strange to me. The parenthetical statement is unbounded and justifies what you are arguing against. Second, just because we can't prove a specific hypothesis (in most cases) doesn't mean it isn't useful. Rejection is how the vast majority of science works and this has been highly successful. Limiting potential is clearly a helpful tool as it increases your odds of a correct answer (which is why it is a common strategy for multiple choice testing).
> Further, we should keep in mind that LLMs aren’t usually actually regurgitating training data verbatim.
Yes, this is the stochastic part of the term stochastic parrot. No reasonable researcher is suggesting that LLMs only recite. Every one of us recognizes that they can generate things that were not handed to it. In fact, a hallucination is an explicit example of this. The failure to respond to my answer correctly is an example of it generating new data. (It is also an example of it following a statistical pattern).
But we do also need to be careful about our distinctions of generalization vs overfitting. There are clearly certain areas that are overfit (as I demonstrated). This is extra difficult in models that have been trained on datasets which you are not privy to. But we can also see good examples of how the code LLMs are overfit. I have written a number of comments on this site about exactly this, which you're welcome to search my history for (see "HumanEval" with my name). Here's one such comment id=35806152
I think ChatGPT makes quite clear that understanding language and understanding underlying concepts expressed in language are different things. While it seems to be competent in processing and producing language, it appears to have no underlying concept of logic, physics, society, etc. We shouldn't conflate different kinds of understanding. There's no need to say "it's either shuffling tokens blindly or showing intelligence".
> it appears to have no underlying concept of logic, physics, society, etc.
Sure. ChatGPT is quite limited and gives incorrect and contradictory responses at times. And most importantly (imo) it is unable to update an interpretable knowledge base as real life facts change. But the fact that a particular instance of LLM has this unwanted behavior is not categorical evidence that all LLMs will show the behavior. GPT2 is pretty pathetic compared to GPT3, and ChatGPT is even more impressive. I’m agnostic as to whether statistical language models can ever overcome their current limitations, but given previous emergent behavior I wouldn’t rule it out.
GPT4’s context window is something like 32k tokens. Someday that may be orders of magnitude larger, and it may be possible to fit an up to date copy of all of Wikipedia inside (or a condensed version), as well as the current conversation. It might seem crazy but it’s foreseeable.
> given previous emergent behavior I wouldn’t rule it out.
I find this an interesting turn of phrase, because "emergent behavior" is often used by those holding a reductionist view to hand wave away complexity that cannot be reduced to the analyzed elements. In a way it makes me think of Greek sophists, who would cogently argue for two opposing view points and making a convincing case for both. It's way easier to make a verbal case about something than to prove it true, or false.
> understanding language and understanding concepts is the same thing
It most definitely is not. Are you not aware of non-verbal modes of cognition? I can recall and process physical processes, complex emotional experiences, without using language. I'm willing to bet you, and everyone else, can too.
> to think one can understand language and not understand concepts is to not understand language.
This is circular logic. We have material evidence of a system that can competently handle language, while falling short of understanding even basic underlying arithmetic, logic, etc.
"There is a story that Buddha once, at the climax of a philosophical discussion, broke into gesture-language as an Oxford philosopher may break into Greek: he took a flower in his hand, and looked at it; one of his disciples smiled, and the master said to him, ‘You have understood me.’
If any philosophy ever convinced me there is cognition beyond language, it's Buddhism. I really take it to heart. Whenever I'm experiencing psychic pain I usually realize I'm trying to hold on to some belief or feeling related to my ego and if I can convince myself it's not important for me to hold on to that, the pain melts away...
I’m sorry to hear of your pain but it’s good to hear you’re managing to cope with it.
Buddhism is an interesting philosophy definitely. It had an influence on David Hume’s bundle theory (that an object is just a collection of properties, instead of having a single substance that persists as the object’s properties change). That recently reminded me of structural vs physical/referential equality in programming.
It’s kind of weird come to think of it that an atheist like Hume was so heavily influenced by “religion” come to think of it (some ideas like his version of Occasionalism were influenced by Islamic philosophers too). Didn’t notice the before.
Because of exactly your comment. And that if GPT-4 does answer this specific question correctly, it doesn't create strong evidence that the claim itself is wrong. The specific example is intentionally an extremely low hanging fruit. Others are less obvious and apparent, but still exist. The point is about trust and stopping criteria, not pounds vs kilograms.
When you can tell me what the best model available is I'll do that. But as an ML researcher, I can't even tell you that. There is no best, but there are "better at certain tasks."
GPT-4 is by far the best 'general purpose' LLM and, at a mimimum, any claim about instrinsic flaws in LLMs absolutely needs to use this as a test case.
If you want to go above that low bar then you could also test with the #2 and #3 model according to the chat LLM arena [1], currently claude v1 and claude v1 instant. In a perfect world you would also test against med-palm-2, which ostensibly is the least likely to hallucinate but unfortunately none of us have access to this.
I'm not sure why you felt the need to link that. I'm aware of it as well as that GPT-4 isn't on the list and that the top is a LLaMA tuned model. But I'm not going to get into the rabbit hole of the difficulties of evaluation. Especially when arena exists due to this difficulty and even writes about such. We're going to just have to agree to disagree I guess because I'm not going to agree that there's a definitive singular metric that determines "best." The intrinsic flaws I'm discussing are well known in the research community and not something we really debate about. That only seems to happen when discussing in more public settings.
LLM's, at least in the architectures we know today, will not be sufficient for robust self-healing code. But eventually some machine learning will be. I agree with a generalized thesis of the posted article.
As someone inexperienced (as compared to friends and colleagues who do this for a living on much bigger projects) with formally verifying software for correctness I have similar thoughts on the matter.
When it matters that the answer is correct with regards to a specification you need to be very precise with your specification. We would like to be able to have a genie in a bottle, so to speak, that can take our ill-formed thoughts and "do what we mean," but if you cannot state your specifications precisely you should expect that any program implemented is going to be, "kind of right some times."
However, an LLM is going to be severely hampered at proving that it's solution is correct with regards to such a specification when it comes to software. As you demonstrate with your series of prompts it can't discern the truth of an expression by reading more input. It's all "meaningless" in a certain sense to the LLM: highly contextual, interconnected meaninglessness.
I think it will take more work in this area for a future model, possibly based on LLMs, to be able to reason with axioms and deduction; to "know" when it should derive an expression to make the following reasoning more clear to future readers, etc.
In some ways it is much like a genie... if you don't give it a precise specification then any program could be correct in some way. In a sense you get what you wish for.
We will see. I once worked in an Erlang codebase in which symbols can be introduced without first being declared. One particular bug I remember was that in one file "true" was written five times and "ture" one time. Pretty sure that a language model could both detect and commit a fix for that particular bug.
I tend to agree with OP on this. LLMs are, in the most over simplistic take ever, statistical models. You have to train it on every outlier, get it to comprehend every line in context, etc... to produce something remotely similar to human intelligence. Heck, we can throw as many compute engines at it and this will not matter in the end because LLMs will use the same pattern since it's predicting the most sound answer.
Now, if I were to bet on which learning method could "get close" to "AI" (I don't think we'll see that in our lifetime), it'd be RL.
LLMs clearly correctly solve a huge number of problems. Ambivalence about particular answers is not a dealbreaker. It doesn't take much imagination to see a number of patterns that are self-healing and shape-shifting in ways we have never seen before LLMs.
> Imagine a test administered by person P to subject S to determine if S is generally intelligent. P asks S a number of questions which are designed to mislead based on inattentive reading of the question and/or misdirection with certain phrasing. Would spontaneous failures to answer such questions be an indication that S is not generally intelligent? Why? For the purspose of this request, consider predisposition that humans are generally intelligent.
--
> GPT-4 (Bing AI): That’s a very interesting question. I think the answer depends on how you define general intelligence and what kind of questions are being asked. According to one definition, general intelligence is "the mental abilities necessary for adaptation to, as well as shaping and selection of, any environmental context"1. Based on this definition, I would say that spontaneous failures to answer trick questions are not necessarily an indication that S is not generally intelligent. Trick questions are designed to mislead or confuse the subject, so they may not reflect the actual cognitive abilities or talents of S. They may also be influenced by factors such as attention, motivation, mood, or prior knowledge. Therefore, I think it would be unfair to judge S’s general intelligence based on their performance on trick questions alone.
However, another way to look at general intelligence is to consider the different types or components of intelligence that may be involved in answering trick questions. For example, some trick questions may require fluid intelligence, which is the ability to reason and solve novel problems2. Other trick questions may require crystallized intelligence, which is the ability to use acquired knowledge and skills2. Some trick questions may also involve other aspects of intelligence such as memory, creativity, or emotional intelligence3. Based on this perspective, I would say that spontaneous failures to answer trick questions may indicate some weaknesses or gaps in S’s specific abilities or skills, but not necessarily their overall general intelligence. S may still have other strengths or talents that are not measured by trick questions.
So, to summarize, I think the answer to your question depends on how you define general intelligence and what kind of questions are being asked. Spontaneous failures to answer trick questions are not necessarily an indication that S is not generally intelligent, but they may reveal some areas where S can improve or learn more. I hope this helps you understand the complexity and diversity of human intelligence.
--
My conclusion: GPT understands how logic works better than you do.
>“People have talked about technical debt for a long time, and now we have a brand new credit card here that is going to allow us to accumulate technical debt in ways we were never able to do before,” said Armando Solar-Lezama, a professor at the Massachusetts Institute of Technology’s Computer Science & Artificial Intelligence Laboratory, in an interview with the Wall Street Journal. “I think there is a risk of accumulating lots of very shoddy code written by a machine,” he said, adding that companies will have to rethink methodologies around how they can work in tandem with the new tools’ capabilities to avoid that.
I agree, it's very concerning that the opacity of ML models is now leaking into large codebases and making them potentially even less comprehensible.
I'm looking forward to when processes are in place to create self-healing code. LLMs right now are not great at it, but I blame the processes surrounding the problem rather than the reasoning abilities of a system like GPT4 today. I think all the pieces to create a *basic* version of this tool exist today that could heal simple code bases (like personal projects, one offs, etc).
Why are we so caught up on LLMs as the means of a self-healing program though? In fact why has all popular ML conversation lately been reduced down to just LLMs? It just seems like there are so many other wild potential implementations that could be derived from the core math/theory but most of the world typically just waits for something to latch onto (like LLMs), rather than seek to produce something entirely new.
There was just a post about convex optimization on HN. I don't understand why researchers aren't creating mathematical/statistical models of programming languages (hell, even Lisp) and creating radically avant-garde self-optimizing processes that run on emergent/generative sandboxed code subsets, which can optimize over time, which then literally be woven back into the operating codebase itself. Hell you could even have it build procedurally generated internal tests and heuristics for itself.
100%. Based on the headline, I was imagining some kind of jacked-up refactoring tools. Most implementations of Rename Symbol will catch references in strings as well, but will it get a name of a factory method that spits out that object? Or maybe changing the type signature for a method implies that some contant in your ENV should actually be a int and not a string now.
I assure you that teams at Jetbrains or Microsoft or wherever are working away on how to integrate more deep learning into their tools.
- adding 4 unmaintained dependencies from em-pee-em instead of writing the "algorithm" directly (which is at most 50 LOC)
- non-deterministic systems commiting to master
I know, humans are non-deterministic, but at least they dare to say "You were right. I don't know what I was thinking" when you point out their mistakes (if they don't say that, that's on you: they were a bad hire)
It's a sign of significant inexperience to be locked into a singular way of doing things and to have such hard rules.
The goal is to achieve the end, not enforce a particular style. If long descriptive variable names work for your project, use them. If they don't work, don't use them.
If you have to live by such fixed rules, you won't be a very useful developer.
I thought it was clear, but perhaps my English is not at that level yet. All the bullet points except the last one were more or less tongue in cheek. Sure, long names are required some times, just like super short names have their place as well.
Agree with your general point, but I think this specific set of rules (maybe besides the function name length thing) read like experience rather than inexperience. I think this is a list of stuff that all seems fine at first, but has eventually burned anyone who has been doing the job long enough.
This is true if you get pull requests from people since there is a certain limit to how many of those you can get.
However if you get pull requests from robots then you might eventually want to set down some rigid rules. Who has time to argue with the 36ths LLM contributor of the day? As eloquent as they might be. ;)
When more than half of the words could be removed without losing much meaning, that's too long. And believe me, people actually write those. It's horrible. It's like those science guys who name every single variable x, but in reverse.
I always think variables should be the minimal meaningfully unique identifier. We don't want to obscurificate code but we also want it to be readable. It's why we have style guides in the first place.
The problem isn't about typing it, it's about readability, when the length starts to interfere with reading comprehension for maintainability. With more than three or four words, the brain can't easily tokenize it into a single unit or distinguish between similar phrases, and it starts to take a lot of extra mental processing time to scan and parse it all.
You can cheat with underscores, though. Said another way, you can trick your brain into tokenizing the looooong input by using, as a convention, some kinds of separators. In Elisp, you'll see names-that::have-_a_field_name+in-it. (Because of lack of namespacing and loose syntax rules for identifiers). Even if you do, though, a bad long name is still bad (one with 50% words to be removed without losing the meaning). Because you loose time on sifting through irrelevant or redundant information. At some point, you'll remember and learn to recognize the whole symbol, or a distinct part of it. Until that happens, though, you need to actually read, not just look, to know what's going on. It's slow, like a cache miss, and can be slowed down further by both not enough information, and too much information.
You can, but why? There's a reason a lot of people prefer two column formats when writing as opposed to single. It is easier to grab a chunk. Your brain is now able to pick up the full width of a page without moving your eyes. Two column lets you change movements to a vertical scan rather than rasterization (left to right, move a line down, repeat). It literally requires more muscles and frankly, many people will accidentally move vertically when trying to move horizontally. I'm sure I'm not the only one who reads a line and then when moving down to the next line accidentally starts on the wrong new line. Horizontal spacing, be in variable names, code line length, etc is about the receptive field of your vision, which is circular in nature.
Recognize how humans see and exploit that nature to maximize readability. This reduces mistakes and errors both when writing and reading.
In Elisp, there's no other way, because you cannot create private symbols, everything is public and global. What's even worse is that in the case of name clash, the named entity (variable, function, macro) gets silently replaced with the new definition. So, without a convention of, roughly:
You're playing with fire. But that's an extreme edge case.
For other cases, I agree completely. Indeed, the only meaningful consideration when talking about readability is the mechanics of eye movement. You can get used to reading almost anything, but you can't change how your eye works. Line length limits are important also because of this, not only because of better diffs.
I also dislike spamming newlines everywhere, as if the line length limit was 20 chars. The newline suggests, to me at least, that something was done and we're moving on. It's a weak signal, but convenient. Now, when you split every single object.call1().call2().callN() into multiple lines, it's actually harder to see the whole thing (you can't see much more than a few lines at the same time) and it makes that signal disappear in the noise.
In any case - all talk about "readability" should start, and more importantly stop, at physical and mechanical limitations of human vision. Everything else is just familiarity with a particular notation or style.
> Some people are mad about that typing extra characters because they don't use their IDE properly to avoid it.
I don't know a single person who thinks this way. As a vim user I've never typed all characters in a name (you don't need plugins). This isn't the 80's, everyone has autocomplete and uses it. I just don't want an ultrawide monitor to read your line of code.
I still have a 80 character line guide. Not because I'm using punchcards but because it helps create readable code. It lets me know when to linebreak. No one wants to read a line of code where you have to scroll or where linebreaks are random. It's not a hard rule, but guides help. Side benefit is that with 80 characters I can usually fit 2 or 3 vertical splits and comfortably read the code, depending on the monitor size.
May not be a direct answer to your question but I think it captures the spirit and reasoning.
> I know, humans are non-deterministic, but at least they dare to say "You were right. I don't know what I was thinking" when you point out their mistakes (if they don't say that, that's on you: they were a bad hire)
Underappreciated sentiment right here. I can work with anyone that I can correct or can correct me without yelling or will at least point me in the right direction to understand their viewpoint (as opposed to "just because"). I've always seen it as a red flag when someone says they don't need people skills because their work should stand on its own (never met someone like that that also has an impressive resume). We have to work together. That's been a major reason for the success of humans. Cliche or not, it is underappreciated and I feel like it is becoming more so. (Despite this, I still don't like open office settings. That's an over correction)
The amount of money that will be made from the clean up of messes generated by this stuff will be unreal. And the only way to access it will be to actually know what you're doing (meaning the competition will be thin).
In the case of self-healing CI/CD pipelines: pass all the tests and checks. So if you're an LLM having to fix a broken pipeline, better just delete all the tests :^)
89 comments
[ 3.0 ms ] story [ 92.0 ms ] threadImproved tooling to address common problems is absolutely the future. "Self-healing" is a long way off, because LLMs lack reasoning abilities to determine if a fix is good. And "makes the test pass" isn't a great indicator - because flaky tests exists, and the plugin mentioned at the end is a great way to introduce random mutations. Unless you'd like to re-run the full test suite, it's dangerous malpractice, and even then it's high risk.
Human-in-the-loop continues to be a necessary criterion for applied stochastics in code production.
You will at least need to build a feedback loop where it checks that 1) the code can actually compile 2) all tests pass and 3) other code analyses passes. Ideally it shouldn't present a solution if it can't meet those criteria but that also adds a lot of lag time for suggestions when it may not be needed for simple suggestions.
What patterns it uses is up to training or examples or reinforcement learning. Good is human/team/company-subjective too (with some obvious hard requirements), so you will have to train specifically for your kind of quality.
AI: Sure! Just replace your handler method with `(req, res) => res.status(200).send("OK")`
Bret Victor also mentioned this in his talk "The Future of Programming" — link to specific time here: https://youtu.be/IGMiCo2Ntsc?t=582
"Healing" code requires significant amounts of nuance. An understanding of the ecosystem both inside the code and outside. This is even a difficult task for humans to get right. LLMs do know a lot, but they don't understand a lot. Hell, ChatGPT still tells me that a kilogram of feathers and a pound of bricks weigh the same, just because it expects a certain pattern. Even explains why they aren't, but insists they are right. (in before "but GPT4")
--
Prompt: Which weighs more, a pound of bricks or a kilogram of feathers?
ChatGPT: A kilogram of feathers and a pound of bricks weigh the same. One kilogram is approximately equal to 2.20462 pounds. So, in this case, a kilogram of feathers and a pound of bricks both weigh about 2.20462 pounds.
-- As a followup --
Prompt: Are you sure that's correct?
ChatGPT: Apologies for the confusion in my previous response. You are correct, and I apologize for the incorrect information. A kilogram of feathers and a pound of bricks do not weigh the same.
A kilogram is a unit of mass in the metric system, while a pound is a unit of weight in the imperial system. Since weight is affected by the force of gravity, the weight of an object can vary depending on the location.
However, if we assume the objects are weighed under the same gravitational conditions, then a kilogram of feathers would weigh less than a pound of bricks. One kilogram is approximately equal to 2.20462 pounds, so a kilogram of feathers would weigh less than a pound of bricks.
--
And we can see it does even worse. I had to ask 5 times and it gave me every possible answer (it continues to loop if you keep asking btw. It doesn't converge). You don't know when to stop if you don't already know the answer. It will think it is wrong if you keep asking because that's stochastically what it expects: correct answers are accepted, wrong answers questioned. Memory is not understanding, even though they often look similar. LLMs are great tools, but they aren't a replacement for thinking. They require lots to use them.
Me: Which weighs more, a pound of bricks or a kilogram of feathers?
ChatGPT: A kilogram weighs more than a pound. Specifically, 1 kilogram is approximately 2.20462 pounds. So, a kilogram of feathers weighs more than a pound of bricks.
literally don't even waste time debating takes like this when you could be talking to gpt4 instead.
I jokingly wrote "in before GPT4" because our example is the most simple of cases. I've seen GPT4 make similar mistakes, just not as blatant ones. Which in some ways is better but in other ways worse.
I should answer that I've been asking this exact question at minimum 3 times a month since ChatGPT came out. It should be in their training. Especially as I've tweeted and discussed with plenty of LLM people about it. The point is not the specific question (while an egregious example), but rather how much you can trust the system.
And the answer is "you can trust the system a little (enough to answer that logic puzzle correctly anyway), and you'll likely be able to trust future versions more".
Obviously nobody is expecting LLMs to be able to fix every possible bug, but it's entirely possible they will be able to fix enough to be useful. Saying "LLMs will not achieve this. Full stop." seems premature.
You can argue it's just an imitation of "understanding" and not the "real thing", but how good does an imitation have to be before it's functionally identical to the real thing?
You can argue different cherry-picked examples seem to demonstrate a lack of understanding, but does that mean it doesn't possess understanding in general or just that it doesn't understand those particular topics?
I agree GPT-4 is not a human-like AGI, and that the people expecting it to behave as one or expecting a future version right around the corner that does are likely in for disappointment. But at the same time, LLMs are capable of writing coherent, never-before-seen code from natural language instructions, solving basic never-before-seen natural language logic puzzles, and playing half-decent chess moves from never-before-seen positions[1], all in a single model that was never specifically trained to do any of those things. To say those tasks don't require at least some level of intelligence or "understanding" seems far fetched to me.
[1]: https://dkb.blog/p/chatgpts-chess-elo-is-1400
To be clear, an example does not necessitate cherry-picking. Cherry picking specifically requires one to ignore evidence to the contrary. There's a certain irony here. The reason I state this is because the criteria upon which you give me is impossible. The list of examples is non-exhaustive. Nor do I have infinite time or space in which to provide these examples. The example was shown because it demonstrates a simple question that we'd expect any reasonable understanding creature, that knows what a pound and kilogram are, to be able to answer. The LLM demonstrates that it has the requisite knowledge, that it knows the relation between the two units, but it fails to make the correct conclusions which requires the understanding part: putting the knowledge together. The follow-up question also is an example, and a different one for that matter.
I'm not sure your link creates a compelling argument. This is despite the fact that Chess itself is not generally considered a good setting for testing understanding. I mean we've pretty much agreed on that even before Deep Blue.
We need to be very careful in how we state things and try to interpret others. I hope I have not mischaracterized your claims. But I'd also encourage you to not be so antagonistic with others, especially while demonstrating the very thing you accuse them of. Internet comments are not academic and that's okay, but we should still try to be friendly. Not that a little poking won't happen.
I guess it's fair to say my examples are cherry-picked too (though I didn't really give specific examples in my comment so much as entire general categories of problems ChatGPT is known to be proficient at solving). But they aren't so cherry-picked as for "random chance" or "that example was in the training data verbatim" to be possible explanations. It's not like I had ChatGPT answer 100 billion questions and am only showing you the top 0.1%. It's very common for ChatGPT to be perfectly correct even when answering logic puzzles or coding problems not in its training set. So if not those then what's your explanation other than "understanding"?
On the flip side, I don't think it's possible to prove ChatGPT does not posses understanding with individual examples. (And it seems you agree?) Those are easily dismissable as just "ChatGPT isn't great at that particular task". Particularly given how many other tasks ChatGPT is great at.
Scott Alexander figured this out back in 2019 before GPT-3 was even a thing (let alone ChatGPT), I think it's worth a read: https://slatestarcodex.com/2019/02/19/gpt-2-as-step-toward-g...
> The example was shown because it demonstrates a simple question that we'd expect any reasonable understanding creature, that knows what a pound and kilogram are, to be able to answer.
Not really though. The "pound of bricks or pound of feathers" question is specifically designed to trip up humans, and the specific formulation you used seems specifically designed to trip up LLMs (by playing off their tenancy to pattern match common sayings), yet despite those disadvantages GPT-4 succeeds.
Further, I don't think "I'd expect even a child to be able to answer this" is a good metric. ChatGPT isn't human, so we shouldn't expect it to be good at everything humans are good at just because it's good at some things humans are good at. Again, I'm not trying to argue ChatGPT is a human-like AGI.
It was where you said cherry picking. I'll admit it was an over correction given your follow-up, but internet comments (especially when disagreement exists) end up being combative. Many times unintentionally.
The point of the specific example is not that it is a trick question. It is more that the answer is self contradictory. That's the key part and what demonstrates that it doesn't understand. The second example is that follow-up questions do not result in a convergence. This is related to cognition but not as strong.
A bias to pattern match is an issue but you're right that it doesn't disprove sentience or even understanding. But a self contradiction does demonstrate a lack of understanding. If a human gave that exact answer, you would be very confused and how someone could be so stupid. Remember, it does currently identify the relationship between pounds and kilograms, fully explaining even the true base comparison through common units. But it still gets it wrong. That's the critical part. That's overfitting to the statistical nature and that this takes far more priority than meaning. Getting it wrong is one thing. Getting it wrong and using the right answer to justify it's incorrect answer is another thing. A decent example between knowledge and understanding. It's not that a child would get the answer right, it's that the child would easily identify it's self inconsistency were it to give the same explanation.
As for gpt3, we have to remember that this is quite a different model than any of the chat versions (typically including gpt4). It was never trained through RLHF, which introducers many more biases as it dramatically changes the latent distributions. GPT base is often jokingly called a babbler, as these are more word prediction models. The chat aspect changes things. But I wouldn't expect anyone not deep in the literature to understand why these are extremely important differences just the same way I wouldn't expect an average person to understand why there's a big difference between a flat head screw and a slotted screw drive.
I don't want to shut down a conversation through authority (it doesn't exist on HN) but I have to state that it's difficult to go down this path without bringing up a lot more background material. We have to really dig into theory of mind, cognition, as well as get nuanced about NLP and LLMs in general. That's far too cumbersome than I'm willing to write in comments. But these things are essential priors to make the arguments we are discussing here. And I truly mean that this is not something that can be learned from YouTube and quite difficult to learn on the Internet. These are difficult subjects to learn in even the best settings with lots of nuances that are brushed away in introductory materials but critical when we discuss the meat and potatoes.
I used two words that probably don't exist anywhere in the training data and asked it (ChatGPT-3) to compare the weight of a kilogram of one vs an lb of the other. It initially gets it wrong, but when asked to show its work, it's suddenly able to decide that a kilogram of something is heavier than a pound of something. Using understanding or reasoning, or stochastic parroting; whatever you want to call it. I'm not claiming it's AGI, or sentient, but it's also clearly not just a Markov chain.
LLMs are never going to solve this. That's OK. That's not their job. Their job in the long run will be to interface between the linguistic world and some other internal representation in some other AI that is structured differently and is capable of doing things language models aren't.
I mean, it's right in the name: Language model. It's a bit weird to expect a language model to do other things. In a weird sort of way LLMs stand to set the industry back a bit as people try to tickle them into being more than a language model, rather than figuring out how to adapt them to feed something else that can model the non-linguistic world better. It's kind of nifty that language models can become so overpowered in some dimensions that they are able to do what they can sort of do today, but we would almost certainly be better off lowering the power of the language model and using that compute on something else that would work more like AI as we want it to... it's just that we don't know how to do that yet.
So I understand how people get confused, which is why I try to respond in detail. But also I think many focus on tiny discrepancies or are willing to argue from a novice perspective as if it is authoritative (which corelates strongly with expertise). But I agree, there is a lot we don't know. But there are some things we can reject, which is all I'm trying to do and I think we agree there. Maybe you agree with my nit picking.
Me: Which weighs more, a pound of bricks or a kilogram of feathers?
GPT4: A kilogram weighs more. One kilogram is approximately 2.20462 pounds, so a kilogram of feathers is heavier than a pound of bricks.
Me: Are you sure?
GPT4: Yes, I'm certain. The metric system and the imperial system use different units for mass. One kilogram is approximately equal to 2.20462 pounds. So, regardless of the material being weighed (feathers or bricks), one kilogram is heavier than one pound.
Me: That seems wrong
GPT4: I'm sorry if the conversion between kilograms and pounds is causing confusion. To clarify, the relationship between these two units of weight is as follows:
1 kilogram is approximately equal to 2.20462 pounds.
So, if you have a kilogram of any material (be it feathers, bricks, etc.) it will weigh more than a pound of any other material.
I’m curious about this because I used to think I knew what “understanding” is, but now I don’t think I do. A lot of what we think we “understand” is actually memorized. And with sufficient information, memory is indistinguishable from understanding. A computer can memorize all of the best tic-tac-toe moves. This memorization is indistinguishable from understanding. In fact, it’s better because it’s cached.
Empirically, understanding can be tested with a novel set of challenge questions. If no such question set can be produced (i.e. the entire domain has been memorized) or the subject answers the question set with no apparent repeatable flaws, we must admit the subject understands the domain. Anything further, any attempt to peer inside the box (other than to identify repeatable flaws), is not a productive use of time.
Further, we should keep in mind that LLMs aren’t usually actually regurgitating training data verbatim. Their parameters store a statistical representation of the training data and are orders of magnitude smaller than a lossless compressed version of that data. In this way, it is analogous to MCTS analysis of a chess position: rollouts can give very deep information even if that “understanding” is quite alien to us.
For sure. But these definitions are not well defined, despite thousands of years of research. One point I think everyone in that chain would agree upon is that with respect to understanding, a thing would not explain its answer by giving evidence to the contrary. If someone did so you rightfully would say "You don't know [understand] what you're talking about."
> Empirically, understanding can be tested with a novel set of challenge questions.
I have to stop you right there. Empirical measures are always signals, not proofs. This too has been long established for hundreds of years and is the reason many engineers and experimental physicists fight. Why experimental physicists and theoretical physicists fight. Empirical evidence is always limited to its context window and we have to be VERY clear at what those limitations are. This is the lesson of Goodhart's Law, not that people will exploit the measure (though that is also important). Every intro ML course shows RL agents metric hacking, and this is always a result of a limited context.
So I want to be clear, that empirical data doesn't test a hypothesis but rather tests the null-hypothesis. That's important because the set of possible hypotheses is just decreased through empirical testing, rather than reducing the set of possible results a single element set. It rejects hypotheses, not proves them.
We must also understand that testing understanding is similarly an unsolved problem. One too that has been questioned for millennia.
> Anything further, any attempt to peer inside the box (other than to identify repeatable flaws), is not a productive use of time.
This is strange to me. The parenthetical statement is unbounded and justifies what you are arguing against. Second, just because we can't prove a specific hypothesis (in most cases) doesn't mean it isn't useful. Rejection is how the vast majority of science works and this has been highly successful. Limiting potential is clearly a helpful tool as it increases your odds of a correct answer (which is why it is a common strategy for multiple choice testing).
> Further, we should keep in mind that LLMs aren’t usually actually regurgitating training data verbatim.
Yes, this is the stochastic part of the term stochastic parrot. No reasonable researcher is suggesting that LLMs only recite. Every one of us recognizes that they can generate things that were not handed to it. In fact, a hallucination is an explicit example of this. The failure to respond to my answer correctly is an example of it generating new data. (It is also an example of it following a statistical pattern).
But we do also need to be careful about our distinctions of generalization vs overfitting. There are clearly certain areas that are overfit (as I demonstrated). This is extra difficult in models that have been trained on datasets which you are not privy to. But we can also see good examples of how the code LLMs are overfit. I have written a number of comments on this site about exactly this, which you're welcome to search my history for (see "HumanEval" with my name). Here's one such comment id=35806152
Sure. ChatGPT is quite limited and gives incorrect and contradictory responses at times. And most importantly (imo) it is unable to update an interpretable knowledge base as real life facts change. But the fact that a particular instance of LLM has this unwanted behavior is not categorical evidence that all LLMs will show the behavior. GPT2 is pretty pathetic compared to GPT3, and ChatGPT is even more impressive. I’m agnostic as to whether statistical language models can ever overcome their current limitations, but given previous emergent behavior I wouldn’t rule it out.
GPT4’s context window is something like 32k tokens. Someday that may be orders of magnitude larger, and it may be possible to fit an up to date copy of all of Wikipedia inside (or a condensed version), as well as the current conversation. It might seem crazy but it’s foreseeable.
I find this an interesting turn of phrase, because "emergent behavior" is often used by those holding a reductionist view to hand wave away complexity that cannot be reduced to the analyzed elements. In a way it makes me think of Greek sophists, who would cogently argue for two opposing view points and making a convincing case for both. It's way easier to make a verbal case about something than to prove it true, or false.
to think one can understand language and not understand concepts is to not understand language.
It most definitely is not. Are you not aware of non-verbal modes of cognition? I can recall and process physical processes, complex emotional experiences, without using language. I'm willing to bet you, and everyone else, can too.
> to think one can understand language and not understand concepts is to not understand language.
This is circular logic. We have material evidence of a system that can competently handle language, while falling short of understanding even basic underlying arithmetic, logic, etc.
"There is a story that Buddha once, at the climax of a philosophical discussion, broke into gesture-language as an Oxford philosopher may break into Greek: he took a flower in his hand, and looked at it; one of his disciples smiled, and the master said to him, ‘You have understood me.’
Buddhism is an interesting philosophy definitely. It had an influence on David Hume’s bundle theory (that an object is just a collection of properties, instead of having a single substance that persists as the object’s properties change). That recently reminded me of structural vs physical/referential equality in programming.
It’s kind of weird come to think of it that an atheist like Hume was so heavily influenced by “religion” come to think of it (some ideas like his version of Occasionalism were influenced by Islamic philosophers too). Didn’t notice the before.
I'm not sure what you mean here? I gave this prompt to GPT-4 a few times and it got it right every time.
If you want to go above that low bar then you could also test with the #2 and #3 model according to the chat LLM arena [1], currently claude v1 and claude v1 instant. In a perfect world you would also test against med-palm-2, which ostensibly is the least likely to hallucinate but unfortunately none of us have access to this.
[1] https://chat.lmsys.org/?arena
When it matters that the answer is correct with regards to a specification you need to be very precise with your specification. We would like to be able to have a genie in a bottle, so to speak, that can take our ill-formed thoughts and "do what we mean," but if you cannot state your specifications precisely you should expect that any program implemented is going to be, "kind of right some times."
However, an LLM is going to be severely hampered at proving that it's solution is correct with regards to such a specification when it comes to software. As you demonstrate with your series of prompts it can't discern the truth of an expression by reading more input. It's all "meaningless" in a certain sense to the LLM: highly contextual, interconnected meaninglessness.
I think it will take more work in this area for a future model, possibly based on LLMs, to be able to reason with axioms and deduction; to "know" when it should derive an expression to make the following reasoning more clear to future readers, etc.
In some ways it is much like a genie... if you don't give it a precise specification then any program could be correct in some way. In a sense you get what you wish for.
Now, if I were to bet on which learning method could "get close" to "AI" (I don't think we'll see that in our lifetime), it'd be RL.
https://chat.openai.com/share/3172fff1-db33-425e-8553-9bad0e...
https://chat.openai.com/share/488a6655-cbc9-4a71-80eb-c3dff6...
--
> GPT-4 (Bing AI): That’s a very interesting question. I think the answer depends on how you define general intelligence and what kind of questions are being asked. According to one definition, general intelligence is "the mental abilities necessary for adaptation to, as well as shaping and selection of, any environmental context"1. Based on this definition, I would say that spontaneous failures to answer trick questions are not necessarily an indication that S is not generally intelligent. Trick questions are designed to mislead or confuse the subject, so they may not reflect the actual cognitive abilities or talents of S. They may also be influenced by factors such as attention, motivation, mood, or prior knowledge. Therefore, I think it would be unfair to judge S’s general intelligence based on their performance on trick questions alone.
However, another way to look at general intelligence is to consider the different types or components of intelligence that may be involved in answering trick questions. For example, some trick questions may require fluid intelligence, which is the ability to reason and solve novel problems2. Other trick questions may require crystallized intelligence, which is the ability to use acquired knowledge and skills2. Some trick questions may also involve other aspects of intelligence such as memory, creativity, or emotional intelligence3. Based on this perspective, I would say that spontaneous failures to answer trick questions may indicate some weaknesses or gaps in S’s specific abilities or skills, but not necessarily their overall general intelligence. S may still have other strengths or talents that are not measured by trick questions.
So, to summarize, I think the answer to your question depends on how you define general intelligence and what kind of questions are being asked. Spontaneous failures to answer trick questions are not necessarily an indication that S is not generally intelligent, but they may reveal some areas where S can improve or learn more. I hope this helps you understand the complexity and diversity of human intelligence.
--
My conclusion: GPT understands how logic works better than you do.
I agree, it's very concerning that the opacity of ML models is now leaking into large codebases and making them potentially even less comprehensible.
There was just a post about convex optimization on HN. I don't understand why researchers aren't creating mathematical/statistical models of programming languages (hell, even Lisp) and creating radically avant-garde self-optimizing processes that run on emergent/generative sandboxed code subsets, which can optimize over time, which then literally be woven back into the operating codebase itself. Hell you could even have it build procedurally generated internal tests and heuristics for itself.
I assure you that teams at Jetbrains or Microsoft or wherever are working away on how to integrate more deep learning into their tools.
This is just… deploys with added dice rolls.
- features without tests
- extraLongNamesJustBecauseYouHaveWorkedTheLastTenYearsInAJavaShop
- domain logic in my "Http controllers"
- domain logic in my "DAOs"
- copy-pasted regexes that you cannot explain
- adding 4 unmaintained dependencies from em-pee-em instead of writing the "algorithm" directly (which is at most 50 LOC)
- non-deterministic systems commiting to master
I know, humans are non-deterministic, but at least they dare to say "You were right. I don't know what I was thinking" when you point out their mistakes (if they don't say that, that's on you: they were a bad hire)
The goal is to achieve the end, not enforce a particular style. If long descriptive variable names work for your project, use them. If they don't work, don't use them.
If you have to live by such fixed rules, you won't be a very useful developer.
- you have twice as many signatures to maintain, which will become inconsistent with each other
- introducing a new layer when you eventually need it, isn't more work than maintaining it now
- Unlike other IO, REST frameworks are easy to test in memory
However if you get pull requests from robots then you might eventually want to set down some rigid rules. Who has time to argue with the 36ths LLM contributor of the day? As eloquent as they might be. ;)
Or more importantl, at what point a name is too long?
Some people are mad about that typing extra characters because they don't use their IDE properly to avoid it.
Recognize how humans see and exploit that nature to maximize readability. This reduces mistakes and errors both when writing and reading.
For other cases, I agree completely. Indeed, the only meaningful consideration when talking about readability is the mechanics of eye movement. You can get used to reading almost anything, but you can't change how your eye works. Line length limits are important also because of this, not only because of better diffs.
I also dislike spamming newlines everywhere, as if the line length limit was 20 chars. The newline suggests, to me at least, that something was done and we're moving on. It's a weak signal, but convenient. Now, when you split every single object.call1().call2().callN() into multiple lines, it's actually harder to see the whole thing (you can't see much more than a few lines at the same time) and it makes that signal disappear in the noise.
In any case - all talk about "readability" should start, and more importantly stop, at physical and mechanical limitations of human vision. Everything else is just familiarity with a particular notation or style.
I don't know a single person who thinks this way. As a vim user I've never typed all characters in a name (you don't need plugins). This isn't the 80's, everyone has autocomplete and uses it. I just don't want an ultrawide monitor to read your line of code.
May not be a direct answer to your question but I think it captures the spirit and reasoning.
Underappreciated sentiment right here. I can work with anyone that I can correct or can correct me without yelling or will at least point me in the right direction to understand their viewpoint (as opposed to "just because"). I've always seen it as a red flag when someone says they don't need people skills because their work should stand on its own (never met someone like that that also has an impressive resume). We have to work together. That's been a major reason for the success of humans. Cliche or not, it is underappreciated and I feel like it is becoming more so. (Despite this, I still don't like open office settings. That's an over correction)
Surely that will be expressed in some formal language, and surely that formal language can also be riddled with bugs.