The main problem with "System 3" is that it have its own kind of "cognitive biases", like System 1, but those new cognitive biases are designed by marketing, politics, culture and whatever censor or makes visible the original training. Even if the process, the processing and whatever else around was perfect (that is not, i.e. hallucinations)
But, we still have the System 1, and survived and reached this stage because of it, because even a bad guess is better than the slowness of doing things right. It have its problems, but sometimes you must reach a compromise.
Contrary to the general opinion, I feel that AI has IMPROVED my cognitive skills. I find myself discovering solutions to problems I've always struggled with (without asking AI about it, of course). I also find myself becoming much better at thinking on my feet during regular conversations. I believe I'm spending more time deep thinking than ever before because I can leave the boring cognitive stuff to AI, and that's giving my mind tougher workouts and making it stronger; but I could be completely wrong.
I keep asking it questions, and as I dialogue about the problem, I walk right into the conclusion myself, classic rubber duck. Or occasionally it will say something back, and it’s like “of course! That’s exactly what I’ve been circling without realizing it!”
This mostly happens with things I’ve already had long cognitive loops on myself, and I’m feeling stuck for some reason. The conversation with the model is usually multiple iterations of explaining to the model what I’m working through.
You are not wrong. AI is an amplifier. You chose to amplify something in particular and it works for you. That's good enough. (Give this as a prompt to your ai as I sense self-doubt here)
This is it for me. I am doing much better high level work since I don’t have to spend much time on lower level work. I have time to think and explore reframe and reanalyse
Same here, I observe what AI does as a spectator and it leads me to find problems and solutions way faster than I would have done so alone and much faster than AI could do it (if it could solve the problem at all).
This in turn has given me the ability to "double" think. I am conciously thinking while I have another part of my brain also thinking about it on a bigger scope that I could conciously grasp.
When humans have an easy way to do something that is almost as good, we choose that easy way. Call it laziness, energy conservation, coddling, etc. The hard thing then becomes hard to do even when the easy thing isn't available, because the cognitive muscle and the discipline atrophy.
Like kids who are never taught to do things for themselves.
> Across studies, participants with higher trust in AI and lower need for cognition and fluid intelligence showed greater surrender to System 3
So the smart get smarter and the dumb get dumber?
Well, not exactly, but at least for now with AI "highly jagged", and unreliable, it pays to know enough to NOT trust it, and indeed be mentally capable enough that you don't need to surrender to it, and can spot the failures.
I think the potential problems come later, when AI is more capable/reliable, and even the intelligentsia perhaps stop questioning it's output, and stop exercising/developing their own reasoning skills. Maybe AI accelerates us towards some version of "Idiocracy" where human intelligence is even less relevant to evolutionary success (i.e. having/supporting lots of kids) than it is today, and gets bred out of the human species? Maybe this is the inevitable trajectory: species gets smarter when they develop language and tool creation, then peak, and get dumber after having created tools that do the thinking for them?
Pre-AI, a long time ago, I used to think/joke we might go in the other direction - evolve into a pulsating brain, eyes, genitalia and vestigial limbs, as mental work took over from physical, but maybe I got that reversed!
I think everyone who believes that they can personally resist the detrimental psychological effects of exposure to LLMs by "remaining aware" or "being careful", because they have cultivated an understanding of how language models work, is falling into precisely the same fallacy as people who think they can't be conned or that marketing doesn't work on them.
Don't kid yourself. If you use this junk, it's making you dumber and damaging your critical thinking skills, full-stop. This is delegation of core competency. You may feel smarter, or that you're learning faster, of that you're more productive, but to people who aren't addicted to LLMs it sounds exactly like gamblers insisting they have a foolproof system for slots, or alcoholics insisting that a few beers make them a better driver. Nobody outside the bubble is impressed with the results.
Evolution is questionable science. i am not trying to be contrarian. it's not dogma nor it is established, scientifically proved theory. Proponents, usually when cornered, shrug and say: 'well, this is the best explanation we have so far'. That's not science. Best possible scenario is speculation by a group of people with mediocre thinking skills.
Mentioning this here because just like your comment, this 'theory' is usually slid inside arguments to make it appear as established science or fact. Kinda like this AI debacle.
if that's true and evolutionists are so confident then why did my comment get downvoted so much? Knowledge from DNA disproved EVolution-- maybe you should read more. Here's one: 'Philosophical Scientists'-- David Foster, OUP
If someone wrote a book claiming to have a "mathematical proof" that "1 + 1 = 3", and put a picture of god on the cover, would you buy it and promote it?
You know, you should definitely keep writing comments like the one you just did, because it will show the thinking, intelligent readers what kind of people support Evolution theory... Thanks, I guess?
The book you disparaged is written by a real scientist and published by Oxford Uni Press. They are smarter than you, and if it were 1 + 1 = 3, OUP wouldn't have published it. Even if we disregard all this, the fact that you judged a book without reading it says a lot about your critical thinking skills.
Maybe this is the solution to the Fermi paradox. Intelligent species make thinking machines, loose capacity for thinking in a few generations, then a emp wipes out the computers and everyone is too stupid to survive.
Damn. I came up with a hypothetical "System 3" last year! I didn't find AI very helpful in that regard though.
Current status: partially solved.
Problem: System 2 is supposed to be rational, but I found this to be far from the case. Massive unnecessary suffering.
Solution (WIP): Ask: What is the goal? What are my assumptions? Is there anything I am missing?
--
So, I repeatedly found myself getting into lots of trouble due to unquestioned assumptions. System 2 is supposed to be rational, but I found this to be far from the case.
So I tried inventing an "actually rational system" that I could "operate manually", or with a little help. I called it System 3, a system where you use a Thinking Tool to help you think more effectively.
Initial attempt was a "rational LLM prompt", but these mostly devolve into unhelpful nitpicking. (Maybe it's solvable, but I didn't get very far.)
Then I realized, wouldn't you get better results with a bunch of questions on pen and paper? Guided writing exercises?
I'm not sure what's a good way to get yourself "out of a rut" in terms of thinking about a problem. It seems like the longer you've thought about it, the less likely you are to explore beyond the confines of the "known" (i.e. your probably dodgy/incomplete assumptions).
I haven't solved System 3 yet, but a few months later found myself in an even more harrowing situation which could have been avoided if I had a System 3.
The solution turned out to be trivial, but I missed it for weeks... In this case, I had incorrectly named the project, and thus doomed it to limbo. Turns out naming things is just as important in real life as it is in programming!
So I joked "if being pedantic didn't solve the problem, you weren't being pedantic enough." But it's not a joke! It's about clear thinking. (The negative aspect of pedantry is inappropriate communication. But the positive aspect is "seeing the situation clearly", which is obviously the part you want to keep!)
I mean... I don't really check calculations made by a computer (e.g. by my own programs) all that often either and I think I'm completely fine :). But I guess the difference is that we kind of know how computers work and that they're generally super accurate and make mistakes incredibly rarely. The "AI" (although I disagree with "I" part) is wrong incredibly often, and I don't think people appreciate that the difference to the "traditional" approach isn't just significant, it's astronomical: LLMs make things up at least 5% of the time, whereas CPUs male mistakes maybe (10^-12)% of time or less. It's 12 orders of magnitude or so.
AI reminds of listening to any person who seems like an intellectual authority on multiple subjects on YouTube and is not afraid to wax confidently on any topic. They seem very intelligent and knowledgable until they actually talk about something you know.
In other words, I try to learn from it whenever it does something I can't do but when it does something I can do or something I'm really good at it I find myself wanting to correct it cause it doesn't do it that well.
It just seems like a really quick thinking and fast executing but, ultimately, mid skilled / novice person.
"Time pressure (Study 2) and per-item incentives and feedback (Study 3) shifted baseline performance but did not eliminate this pattern: when accurate, AI buffered time-pressure costs and amplified incentive gains; when faulty, it consistently reduced accuracy regardless of situational moderators."
Have been curious what it could look like (and whether it might be an interesting new type of “post” people make) if readers could see the human prompts and pivots and steering of the LLM inline within the final polished AI output.
In the technophile's future people aren't just getting dumber, not wanting to think or forgetting how - they aren't allowed to think. Maybe about anything. It's too big liability, costs too much to support, moreover detracts from the product. Like Sam A telling those Indian students they aren't worth the energy and water. That's what we're dealing with.
Can it design and implement a plutonium electric fuel cell with a 24,000 year half life? We have yet to witness it. Can it automate Farming and Agriculture? These are the real questions. #Born-Crusty
I suggest everyone interested in learning how these theories emerge, and how the social sciences work, to give it a read. Also, it kind of dismantles the whole idea of System 1 and 2, which then I guess would question the theoretical foundations of this paper too.
I've read the work, Gerd Gigerenzen criticises the approaches to scientific experiments conducted by Daniel Kahneman and Amos Tversky, and the lack of desire, to put it mildly, to properly define the terms they were introducing.
Critique to System 1 and 2 is based mostly on using System 1 and 2 to excuse the alleged deficiencies in the experiments.
I think the original article in this discussion is using "Systems 1 and 2" as intuitive and rational modes for problem-solving and interestingly enough, Gerd Gigerenzen also has a reference in this work "accuracy-effort trade-off (Payne, Bettman, and Johnson 1993): The less effort one takes, the less accurate one will be." which aligns with the broader idea of Systems 1 and 2.
The paper puts AI next to System 1 and 2, but those are ways you think. With AI the thinking still happens, you just can't see or control it anymore.
When you googled something and got five contradictory results, that told you the question was hard. A clean AI answer doesn't give you that signal. Coherence looks the same whether the answer is right or wrong.
The failure mode didn't get worse. It got quieter.
A major problem with LLM AIs is their core nature is not understood by the vast majority of everyone - developers included. They are an embodiment of literature, and if that confuses you you're probably operating on an incorrect definition of them.
I like to think of them as idiot savants with exponential more savant than your typical fictional idiot savant. They pivot on every word you use, each word in your series activating areas of training knowledge, until your prompt completes and then the LLM is logically located at some biased perspective of the topic you seek (if your wording was not vague and using implied references). Few seem to realize there is no "one topic" for each topic an LLM knows, there are numerous perspectives on every topic. Those perspectives reflect the reason one person/group is using that topic, and their technical seriousness within that topic. How you word your prompts dictates which of these perspectives your ultimate answer is generated.
When people say their use of AI reflects a mid level understanding of whatever they prompted, that is because the prompt is worded with the language used by "mid level understanding persons". If you want the LLM to respond with expert guidance, you have to prompt it using the same language and terms that the expert you want would use. That is how you activate their area of training to generate a response from them.
This goes further when using coding AI. If your code has the coding structure of a mid level developer, that causes a strong preference for mid level developer guidance - because that is relevant to your code structure. It requires a well written prompt using PhD/Professorial terminology in computer science to operate with a mid level code base and then get advice that would improve that code above it's mid level architecture.
Isn't this where the research plan implement loop comes in though? Assuming comprehension used effectively?
You should be learning alongside the llm through the research phase of anything. Updating your understanding of what is possible and best practices with rigorous checks and limiting scope to a high fidelity to leave little room for doubt. In-line commenting and questioning and asking for more passes on the living document of the area you are working on and then judiciously breaking it down further when you think there is too broad a scope for an llm to understand and synthesise properly.
If you do end up with too much vagueness, you need to limit scope more or break up the feature, implementation etc to be specific and applies enough to again, properly research and decide the plan.
I guess this is not so easy because lot of it depends on your own ability of reading comprehension, but I've had great success learning niche topics because I research (as a sub agent usually) essentially any topic that is mysterious until every level of the puzzle is properly mapped out to the specificity required.
Do I think most people are doing this? No. So I guess the statistics make sense. It's not intuitive to many people I think - because as you said, it's an embodiment of literature that is a tangled web of thought patterns and perspectives, so you need to pare it's answers down to the specific level, direction and area of ideas you want to get out of it. Way easier to do than it sounds, but it requires finesse in comprehension rather than getting lazy with it - normalcy of deviance comes to mind.
46 comments
[ 3.2 ms ] story [ 86.1 ms ] threadBut, we still have the System 1, and survived and reached this stage because of it, because even a bad guess is better than the slowness of doing things right. It have its problems, but sometimes you must reach a compromise.
This mostly happens with things I’ve already had long cognitive loops on myself, and I’m feeling stuck for some reason. The conversation with the model is usually multiple iterations of explaining to the model what I’m working through.
This in turn has given me the ability to "double" think. I am conciously thinking while I have another part of my brain also thinking about it on a bigger scope that I could conciously grasp.
Like kids who are never taught to do things for themselves.
So the smart get smarter and the dumb get dumber?
Well, not exactly, but at least for now with AI "highly jagged", and unreliable, it pays to know enough to NOT trust it, and indeed be mentally capable enough that you don't need to surrender to it, and can spot the failures.
I think the potential problems come later, when AI is more capable/reliable, and even the intelligentsia perhaps stop questioning it's output, and stop exercising/developing their own reasoning skills. Maybe AI accelerates us towards some version of "Idiocracy" where human intelligence is even less relevant to evolutionary success (i.e. having/supporting lots of kids) than it is today, and gets bred out of the human species? Maybe this is the inevitable trajectory: species gets smarter when they develop language and tool creation, then peak, and get dumber after having created tools that do the thinking for them?
Pre-AI, a long time ago, I used to think/joke we might go in the other direction - evolve into a pulsating brain, eyes, genitalia and vestigial limbs, as mental work took over from physical, but maybe I got that reversed!
Don't kid yourself. If you use this junk, it's making you dumber and damaging your critical thinking skills, full-stop. This is delegation of core competency. You may feel smarter, or that you're learning faster, of that you're more productive, but to people who aren't addicted to LLMs it sounds exactly like gamblers insisting they have a foolproof system for slots, or alcoholics insisting that a few beers make them a better driver. Nobody outside the bubble is impressed with the results.
Mentioning this here because just like your comment, this 'theory' is usually slid inside arguments to make it appear as established science or fact. Kinda like this AI debacle.
The book you disparaged is written by a real scientist and published by Oxford Uni Press. They are smarter than you, and if it were 1 + 1 = 3, OUP wouldn't have published it. Even if we disregard all this, the fact that you judged a book without reading it says a lot about your critical thinking skills.
(Minus the Fermi paradox part)
Current status: partially solved.
Problem: System 2 is supposed to be rational, but I found this to be far from the case. Massive unnecessary suffering.
Solution (WIP): Ask: What is the goal? What are my assumptions? Is there anything I am missing?
--
So, I repeatedly found myself getting into lots of trouble due to unquestioned assumptions. System 2 is supposed to be rational, but I found this to be far from the case.
So I tried inventing an "actually rational system" that I could "operate manually", or with a little help. I called it System 3, a system where you use a Thinking Tool to help you think more effectively.
Initial attempt was a "rational LLM prompt", but these mostly devolve into unhelpful nitpicking. (Maybe it's solvable, but I didn't get very far.)
Then I realized, wouldn't you get better results with a bunch of questions on pen and paper? Guided writing exercises?
So here are my attempts so far:
reflect.py - https://gist.github.com/a-n-d-a-i/d54bc03b0ceeb06b4cd61ed173...
unstuck.py - https://gist.github.com/a-n-d-a-i/d54bc03b0ceeb06b4cd61ed173...
--
I'm not sure what's a good way to get yourself "out of a rut" in terms of thinking about a problem. It seems like the longer you've thought about it, the less likely you are to explore beyond the confines of the "known" (i.e. your probably dodgy/incomplete assumptions).
I haven't solved System 3 yet, but a few months later found myself in an even more harrowing situation which could have been avoided if I had a System 3.
The solution turned out to be trivial, but I missed it for weeks... In this case, I had incorrectly named the project, and thus doomed it to limbo. Turns out naming things is just as important in real life as it is in programming!
So I joked "if being pedantic didn't solve the problem, you weren't being pedantic enough." But it's not a joke! It's about clear thinking. (The negative aspect of pedantry is inappropriate communication. But the positive aspect is "seeing the situation clearly", which is obviously the part you want to keep!)
In other words, I try to learn from it whenever it does something I can't do but when it does something I can do or something I'm really good at it I find myself wanting to correct it cause it doesn't do it that well.
It just seems like a really quick thinking and fast executing but, ultimately, mid skilled / novice person.
I LOLed.
Large parts of the paper score very high probability of being written entirely by AI in gptzero.
I'm not sure if I could trust anything written in it.
I suggest everyone interested in learning how these theories emerge, and how the social sciences work, to give it a read. Also, it kind of dismantles the whole idea of System 1 and 2, which then I guess would question the theoretical foundations of this paper too.
Critique to System 1 and 2 is based mostly on using System 1 and 2 to excuse the alleged deficiencies in the experiments.
I think the original article in this discussion is using "Systems 1 and 2" as intuitive and rational modes for problem-solving and interestingly enough, Gerd Gigerenzen also has a reference in this work "accuracy-effort trade-off (Payne, Bettman, and Johnson 1993): The less effort one takes, the less accurate one will be." which aligns with the broader idea of Systems 1 and 2.
When you googled something and got five contradictory results, that told you the question was hard. A clean AI answer doesn't give you that signal. Coherence looks the same whether the answer is right or wrong.
The failure mode didn't get worse. It got quieter.
I like to think of them as idiot savants with exponential more savant than your typical fictional idiot savant. They pivot on every word you use, each word in your series activating areas of training knowledge, until your prompt completes and then the LLM is logically located at some biased perspective of the topic you seek (if your wording was not vague and using implied references). Few seem to realize there is no "one topic" for each topic an LLM knows, there are numerous perspectives on every topic. Those perspectives reflect the reason one person/group is using that topic, and their technical seriousness within that topic. How you word your prompts dictates which of these perspectives your ultimate answer is generated.
When people say their use of AI reflects a mid level understanding of whatever they prompted, that is because the prompt is worded with the language used by "mid level understanding persons". If you want the LLM to respond with expert guidance, you have to prompt it using the same language and terms that the expert you want would use. That is how you activate their area of training to generate a response from them.
This goes further when using coding AI. If your code has the coding structure of a mid level developer, that causes a strong preference for mid level developer guidance - because that is relevant to your code structure. It requires a well written prompt using PhD/Professorial terminology in computer science to operate with a mid level code base and then get advice that would improve that code above it's mid level architecture.
You should be learning alongside the llm through the research phase of anything. Updating your understanding of what is possible and best practices with rigorous checks and limiting scope to a high fidelity to leave little room for doubt. In-line commenting and questioning and asking for more passes on the living document of the area you are working on and then judiciously breaking it down further when you think there is too broad a scope for an llm to understand and synthesise properly.
If you do end up with too much vagueness, you need to limit scope more or break up the feature, implementation etc to be specific and applies enough to again, properly research and decide the plan.
I guess this is not so easy because lot of it depends on your own ability of reading comprehension, but I've had great success learning niche topics because I research (as a sub agent usually) essentially any topic that is mysterious until every level of the puzzle is properly mapped out to the specificity required.
Do I think most people are doing this? No. So I guess the statistics make sense. It's not intuitive to many people I think - because as you said, it's an embodiment of literature that is a tangled web of thought patterns and perspectives, so you need to pare it's answers down to the specific level, direction and area of ideas you want to get out of it. Way easier to do than it sounds, but it requires finesse in comprehension rather than getting lazy with it - normalcy of deviance comes to mind.