Microsoft Study Finds AI Makes Human Cognition “Atrophied and Unprepared”
“[A] key irony of automation is that by mechanising routine tasks and leaving exception-handling to the human user, you deprive the user of the routine opportunities to practice their judgement and strengthen their cognitive musculature, leaving them atrophied and unprepared when the exceptions do arise,” the researchers wrote.
Certainly a common enough concern in people critiquing use of ChatGPT here. I'm more worried about "softer" problems, though—morality, values, persuasion, including deciding which of two arguments is more convincing and why.
But these have always been issues that humans commonly struggle with so idk.
So it does what Google searching did: it made retaining of information an optional cognitive burden, and optional cognitive burdens are usually jettisoned.
Fortunately, my ADHD-addled brain doesn't need some fancy AI to make its cognition "Atrophied and Unprepared"; I can do that all on my own, thank you very much.
It really doesn't though. Even when google was at its best, and showed you relevant non-spammy results, a degree of critical thinking was required when sifting through the results, evaluating the credibility of the author, website, etc. Do you fact check everything the AI spits out? The ability for people to critically think is basically gone. That's been trending since before AI, but it's really clear to me at this moment in time how bad it has gotten. It's a laziness of thinking that I don't think was the same with Google.
> It really doesn't though. Even when google was at its best, and showed you relevant non-spammy results, a degree of critical thinking was required when sifting through the results, evaluating the credibility of the author, website, etc. Do you fact check everything the AI spits out? That's been trending since before AI, but it's really clear to me at this moment in time how bad it has gotten. It's a laziness of thinking that I don't think was the same with Google.
Nah, it was already at zero before ChatGPT came to public attention.
may not have been the same degree but reducing cognitive burden trends in the same direction. This might be bad but it might be very good. Is the AI competing with you to get a promotion and replace you? Is it going to lie to you knowingly cause it doesn't like you?
> Google helps you find things that you process later on with your brain.
I'm willing to bet that there were a lot of Google searches, pre-ChatGPT, that effectively were questions. Lots of "huh, I wonder" during conversations and the first result was taken as "the truth".
>> Moreover, participants perceived it to be more effort to constantly steer AI responses (48/319), which incurs additional Synthetic thinking effort due to the cost of developing explicit steering prompts. For example, P110 tried to use Copilot to learn a subject more deeply, but realised: “its answers are prone to several [diversions] along the way. I need to constantly make sure the AI is following along the correct ‘thought process’, as inconsistencies evolve and amplify as I keep interacting with the AI.
While much is made of the 'diminished skill for independent problem-solving' caused by over-reliance, is there a more salient KPI than some iteration of this 'Synthetic Thinking Effort' by which to baseline and optimise the cost/benefit of AI usage versus traditional cognition?
One thing I've tried using Gemini for, and been really impressed with, is practicing languages. I find Duolingo doesn't really translate to fluency, because it doesn't really get you to struggle to express yourself - the topics are constrained.
Whereas, you can ask an LLM to speak to you in e.g. Spanish, about whatever topic you're interested in, and be able to stop and ask it to explain any idioms or vocabulary or grammar in English at any time.
I found this to be more like a "cognitive gym". Maybe we're just not using the tools beneficially.
I remain perplexed that everyone is so focused on using LLMs to automate software engineering, when there are language-based professions (like Spanish tutor, in your example) that seem more directly threatened by language models. The only explanation I've heard is that the industry is so excited about reducing spend on software engineering salaries that they're trying to fit a square peg into a round hole, and largely ignoring the square holes.
> The only explanation I've heard is that the industry is so excited about reducing spend on software engineering salaries that they're trying to fit a square peg into a round hole, and largely ignoring the square holes.
I think that's really just it, and I agree with you. There are many other areas LLMs can, and should, be more useful and effort put toward both assisting and automating.
Instead, the industry is focusing on creative arts and software development because human talent for that is both limited and expensive, with a third factor of humans generally being able to resist doing morally questionable things (e.g., what if hiring for weapons systems software becomes increasingly difficult due to a lack of willingness to work on those projects, likewise for increasingly invasive surveillance tech, etc.)
We're rushing into basically the opposite of what AI should do for us. Automation should work to free us up to focus more on the arts and sciences, not take it away from us.
I think it's because software engineers are the only group that can unanimously operate LLMs effectively and build them into larger systems. They'll automate their own jobs first and move on to building the toolkits to automate the others.
language-based professions like translation have been dying for years and no one has cared; they're not about to start now that the final nail's been put in the coffin.
This isn't a new thing. I noticed it in the 1990s in bank employees as their work became increasingly automated. As the software became better at handling exceptions, their skills atrophied further and they became even worse at handling the harder exceptions that remained.
As our grade school teachers warned, my arithmetic skills are indeed brutally stunted thanks to calculators. The implications do seem even worse with folks using AI to do their technical debugging and decision making, though.
I've been calling this out since ChatGPT went mainstream.
The seductive promise of solving all your problems is the issue. By reaching for it to solve any problem at an almost instinctual level you are completely failing to cultivate an intrinsically valuable skill - that of critical reasoning.
That act of manipulating the problem in your head—critical thinking—is ultimately a craft. And the only way to become better at it is by practicing it in a deliberate, disciplined fashion.
This is why it's pretty baffling to me when I see attempts at comparing LLMs to the invention of the calculator. A calculator is still used IN SERVICE of a larger problem you are trying to solve.
Yeah, but the calculator analogy is apt. In the past, anyone who went to grade school could answer 6 * 7 off the top of their head, and do basic mental math. We've pretty much lost that.
With that said, I do worry that losing the ability to craft sentences (or code) is more problematic than losing the ability to do mental math.
Why? I'm pretty sure my public school education prepared me for all of these questions by 8th grade, excepting some notation that we no longer use and some specific history questions that are now less relevant.
Losing the ability to do mental math is probably not actually a big deal
Losing the ability to do calculations by hand on a piece of paper with a pencil probably actually is a big deal
When I went to school we still had to do a lot of calculations by hand on paper. Thus, if I use a calculator to get an answer, I'm capable of reproducing the answer by hand if necessary
With math, at least when I was learning it, we seemed to understand that the calculator is a useful tool that doesn't replace the need to develop underlying skills
I'm seeing the exact opposite behavior and mentality from the AI crowd. "You don't need to learn how to do that correctly anymore, you can just have the AI do it"
"Vibe Coding", literally the attitude that you don't need to understand your code anymore. You just have the AI generate it and go off vibes to decide if it's right or not
Yeah, I don't know how my car engine works. But I trust that the people who engineered it do, and the mechanics that fix it when it breaks do. There's no room for "Vibe Bridge Building" in reality
Anyone advocating for "Vibe coding" is an admission that it doesn't actually matter if the thing they build works or not
Unfortunately that seems to be a growing portion of software
Not to mention the absolute pile of "mathematics problems" that can't be solved except by pushing symbols around a page, which a calculator is absolutely useless at. So sure I can have a calculator "calculate" an approximation for 4/3 but it can't help me manipulate the symbols around the improper fraction that i need to manipulate to calculate the radius given the surface area of a sphere. And it's of zero help in understanding the relevance of that "pattern" to whatever phenomenon I'm using the mathematics to reason about. That all requires human intelligence.
There are a lot of calculators and other tools that can push the symbols around and many that can even apply some special high-level mathematical rules to the symbols. But whether or not those rules are relevant to the task at hand is entirely a matter for a human to decide.
My feeling is… just give it 6-12 months. All the low-quality apps that were "vibe-coded" will start to break down, have massive security breaches, or generally fall apart as new features are added.
Brace yourself for a wave of think pieces a year from now, all wringing their hands about the death of vibe coding, and the return of artisanal handmade software.
I have a similar feeling, but there's also a little voice in the back of my head trying to convince me that I'm just trying to cope with the fact I'm going to be made obsolete by a software industry that actually doesn't care if software breaks down and has massive security breaches as long as the AI furnace keeps getting coal
I think both are true. The vibe-coders will run into massive problems. But AI will also improve a lot and probably be better than humans at some point.
“ Losing the ability to do mental math is probably not actually a big deal”
I think it’s a huge deal. I see a lot of people do some financing stuff and they have no idea what a 20% interest rate really means. So they just go ahead and do it because taking out a calculator is too tedious. I find it pretty crazy how many can’t figure how much a 20% or even 10% is. A lot of financial offerings take advantage of the fact that people can’t do even basic math.
If taking out a calculator is too much of a burden when we all have smartphones with built in calculator apps, then the problem is not the lack of mental math skill.
Edit:
At best, it shows that the person doesn't really know how to calculate that even with a calculator
At worst, it shows that the person lacks of any kind of give a fuck at all
Either way, they probably would not likely have learned the mental math required to do this regardless of if it were being taught at school or not
> At best, it shows that the person doesn't really know how to calculate that even with a calculator
What it shows is that 10% means nothing to them. It's just another number that a calculator will spit out. It's not "one out of ten" or "shift everything to the right." They have no ability to evaluate what the number means on the fly. They just plug it into the calculator and check if they have that much. They can't have a reasonable discussion about the number without stopping and calculating every interest rate they can think of, writing them down on a piece of people, and writing the budget next to them.
A calculator here, of course, leads to more work, not less. I bet the vast majority of the time that somebody too lazy or anxiety-ridden to learn how to calculate 20% in their head is also going to be too lazy and bothered to take out a calculator every time they need to know, rather than just trying to bluff
their way through conversations until they can get to a calculator.
Calculators are of no benefit to students who have no clear concept of the calculation. Just learn your times tables, they're one of the first things we teach kids.
The pragmatic pro-calculator "nobody needs to calculate those numbers in real life" school I think gets most of its support because constructivists often fail entirely to teach things that have been until now best learned by rote, hoping that an instinct for multiplication and division will naturally and precisely arise from children's mathematical souls. The fact that it doesn't is proof that doing arithmetic is bad and a waste of time actually. They love abstractions, but only vague ones that can't be reliably tested.
> Either way, they probably would not likely have learned the mental math required to do this regardless of if it were being taught at school or not
I wonder if people that were writing code in assembly complained that people learning more modern languages didn't really know how the 0s and 1s work.
I'm not sure where the line is, but there is a point where the abstraction works so well you really don't need to know how it works underneath.
I'm also not sure if a car mechanic needs to know how an engine works. I'm assuming almost none of them could design a car engine from scratch. They know just enough to know which parts needs to be replaced.
"I'm also not sure if a car mechanic needs to know how an engine works. I'm assuming almost none of them could design a car engine from scratch. They know just enough to know which parts needs to be replaced."
That's why, when your car has a problem, a lot of mechanics just blindly replace parts with the hope that something will fix it. You are much better off with a mechanic that understands how the car works. And you will save a lot of money.
I disagree that any car mechanic working a local auto body shop knows engines well enough to design one. They just know which parts are broken.
Similarly we reach a point in coding where you don't really need to know how every API or language you use operates beneath the hood, you just need to be able to see where its broken.
And those skills are entirely context dependent. You're likely saying this from a SW engineer's point of view. Whereas I've worked in teams with physicists and electrical engineers. When you're in a meeting, and there is a technical discussion, and everyone can calculate in their head the effects after integrating a well known function and how that will impact the physical system, while you have to pull out a calculator/computer, you'll be totally lost.
You can argue that you could be as productive as the others if you were given the time (e.g. doing this on your own at your own cubicle), but no one will give you that time in meetings.
I love doing all aspects of building software. However, I’ve noticed when I am feeling lazy I’ll just copy pasta a stack trace into an LLM and just trust what is says is wrong. I won’t even read the stack trace.
I only tend to do that when I am tired or annoyed, but when I do it I can feel myself getting dumber. And it’s a weirdly satisfying feeling.
I just need a chair that doubles as a toilet and I’ll be all set.
You can't delegate understanding. I don't mean you shouldn't, you can't.
If you don't understand what's happening, you have no way to know if the system is working as intended. And understanding (and deciding) exactly how the system works is the really hard part for any sufficiently complex project.
Then prompt the AI to provide its outputs in a way that keeps the human user engaged and aware of where they are in the thought process: maps, diagrams, repetition summaries.
We have the cognition science to make it happen - or at least learn how to structure it.
> Abstract The rise of Generative AI (GenAI) in knowledge workflows raises questions about its impact on critical thinking skills and practices. We survey 319 knowledge workers to investigate 1) when and how they perceive the enaction of critical thinking when using GenAI, and 2) when and why GenAI affects their effort to do so. Participants shared 936 first-hand examples of using GenAI in work tasks. Quantitatively, when considering both task- and user-specific factors, a user’s task-specific self-confidence and confidence in GenAI are predictive of whether critical thinking is enacted and the effort of doing so in GenAI-assisted tasks. Specifically, higher confidence in GenAI is associated with less critical thinking, while higher self-confidence is associated with more critical thinking. Qualitatively, GenAI shifts the nature of critical thinking toward information verification, response integration, and task stewardship. Our insights reveal new design challenges and opportunities for developing GenAI tools for knowledge work.
A good model for understanding what happens to people as they delegate tasks to AI is to think about what happens to managers who delegate tasks to their subordinates. Sure, there are some managers who can remain sharp, hands-on and relevant, but many gradually lose their connection to the area they're managing and become pure process/project/people managers and politicians.
Ie. most managers can't help their team find a hard bug that is causing a massive outage.
Note: I'm a manager, and I spend a lot of time pondering how to spend my time, how to be useful, how to remain relevant, especially in this age of AI.
I think this is a really good analogy. Delegating problems to others is nothing novel or new to human experience.
Perhaps the biggest difference is the lack of feedback AI gives that humans can give: a subordinate can communicate if they feel like their manager is being too hands off. AI never questions management style.
I think this is the primary reason I have difficulty with management: my brain simply doesn't really work out (in the fitness sense) in the cognitive way it is used to, but instead has to track all sorts of problems, many of which are emotional/political. Those problems are often emotional/political and thus not really something I can readily solve by thinking real hard about things.
What if you use it to handle boring BS work you don't care about and focus on what you actually want to do? I offload my work tasks to GPT and do other stuff during work hours. Play with my dog. Stretch. Paint. Work on other creative projects. I don't give a fuck about work as long as they keep sending me a paycheck. Oh no! My brain is going to atrophy from not manually synthesizing info from this report that no one reads anyway.
> Ie. most managers can't help their team find a hard bug that is causing a massive outage.
Strategy vs tactics. Managers aren't there to teach their reports skills, they hire them because they already have them. They're there to set priorities and overall direction.
He's not disputing that. The difference, though, is that if you're using LLMs as code assistants, your skills will atrophy the way the manager's skills will, but you are still a SW engineer.
While the manager doesn't need those skills, you still do.
LLMs and agents are going to make labels like 'sw engineer', 'qa tester', marketer, CFO, and CEO very squishy.
I think in the coming decade if you put yourself in a box and do a specialized task, and nothing more, you'll have a bad time. This is going to be an era where strategy is far more important than tactics.
I can't disagree more strongly. If you're a specialized person with expertise and ability to perform outside the knowledge of all possible interns because you're developing something novel and not covered by standard materials, you'll be hard to compete with using AI because AI will direct people to standardized methods.
Granted, if you do a specialized task that is taught in schools and that anyone can do, that's trouble. But that's not tactics either, that's clock-punching. That's replaceable. You can talk 'strategy' all you like but if you're only exploiting the AI's ability to do what people already know, that's another box to be stuck in.
I think we're in agreement then, I'm saying the specialists of this stripe:
> if you do a specialized task that is taught in schools
are in trouble, and so are you.
> perform outside the knowledge of all possible interns
This is the strategic level thinking I'm referring to. Frankly this role might not be safe either, but if that's the case then we're probably headed for fully automated luxury communism no matter what we do.
Indeed. I started using Sonnet for coding only about a month ago. It's been great in that I've finally written scripts I had floating around my brain for years, and very rapidly.
But the feeling of skill atrophy is very real. The other day I needed to write a function that recursively traverses a Python dictionary, making modifications along the way. It's the perfect task to give an LLM. I can easily see that if I always give this task to an LLM, a few years down the road I'll be really slow in writing it on my own, and will fail most coding interviews.
Also, while there is a high in producing a lot in a short amount of time, there is no feeling of satisfaction that you get from programming. And debugging its bugs is as much fun as debugging your coworkers' bugs - except at least with the coworkers you can have a more intelligent conversation on what they were trying to do.
This is exactly why I still play Go, practice martial arts, archery and use the command line for my dev workflow. Those are all arguably less efficient and obsolete. The AlphaGo series can defeat the strongest human Go players and firearms are more effective than unarmed martial arts and archery. GUI is easier for most people than the command line. Yet, I practice all these to develop my mind and body. I don't have to be a world-class Go player to benefit from learning to play Go.
This happens a lot in the natural world in ecosystems. For example, many people plant trees and add a drip system. The trees grow to depend on the drip system, and never stretch and develop their roots -- and the relationship they have with the soil microbiome. They make the trees prone to get knocked down when an unusually strong gust of wind come through.
> This is exactly why I still play Go, practice martial arts, archery and use the command line for my dev workflow.
And this is what worries me. Pre-LLM, I'd get all my practice done during work hours. I fear that with LLMs, I'll need to do all this practice on my free time, effectively reducing my truly "free" time.
> Pre-LLM, I'd get all my practice done during work hours. I fear that with LLMs, I'll need to do all this practice on my free time
If you can attach a valid business reason/excuse then it's fairly easy to get time dedicated to experimentation and learning.
I haven't been a coder for several years now, but I still try to justify a personal lab environment in order to understand the market and customer problems.
If you're an Engineer or EM, I think it would be even easier (attach experimentation to some sort of critical engineering initiative).
Even with Code Gen Copilots becoming a thing, between the lines most of us in engineering and business leadership recognize it cannot replace architecture or design. If it can, then you've solved the Turing or Chinese Room problem and that's a whole other story.
That said, if your day job isn't only "write this terraform/python script" you shouldn't be at risk.
> If you can attach a valid business reason/excuse then it's fairly easy to get time dedicated to experimentation and learning.
This has the same problem as "doing it in free time". I have a finite number of goodie points with management, and I'd be using some when I request this - which means I have fewer things I can request from them in the future.
> That said, if your day job isn't only "write this terraform/python script" you shouldn't be at risk.
I'm not worried about the present, or even the near future. The reason I have other valuable skills is because of the years I spent on writing code. Eventually, I'll stop growing if I rely too much on LLMs. I may have to rely on them if everyone else is and being more productive.
The Turing test can be considered well and truly passed at this point, partially because it has always said more about the human taking the test than it did about the machine. As for the Chinese Room, it was never anything but a pointless exercise in question-begging. If anyone ever considered it relevant, the advent of LLMs should have immediately convinced them otherwise. Searle knew nothing about embedding vectors, never mind attention.
Great points otherwise, though. A strong focus on identifying, understanding, and solving customer problems is, in modern parlance, all you need.
I had someone give me a good use-case for LLMs -- such as, doing the CRUD work that I really don't want to do, if you are short on funding.
On the other hand, there is something I learned from permaculture design. You need to allow what you want to grow in the ecosystem if you want them present. If I wanted to develop the next generation of developers, then I'd be giving that CRUD stuff I've done many times to a junior developer instead. It obviously depends on funding, and whether your investors are on board with that.
As far as keeping up and remaining competitive ... my suggestion is to learn these things in a way where you can broadly apply the underlying principles in other domains, especially that, as you age, your working memory will shrink. This isn't accumulating more of the same thing, but gaining deeper insights that can only be found when you've developed a sufficient level of skill. There are lots of ways this can be accomplished.
> You need to allow what you want to grow in the ecosystem if you want them present. If I wanted to develop the next generation of developers, then I'd be giving that CRUD stuff I've done many times to a junior developer instead.
Indeed, one of my fears is that people won't develop those skills to begin with. In my team, the few senior people can pretty much be trusted with using LLMs and still producing quality code. But we decided against evangelizing it to the junior folks. They still have access via Copilot, but we won't gently suggest they try it out.
Coding is actually a hard skill which requires practice. Regular litcoding for the sake of it should help. The problem with that is it takes the whole brain and breaks other thoughts chain. I'm thinking about to dedicate full days for small tasks like this.
> Regular litcoding for the sake of it should help.
I got where I am without regular leetcoding. For me, regular leetcoding will only marginally improve my skills. And I definitely don't want to do regular leetcoding just to maintain my skills.
You know how many people love jogging outdoors but hate treadmills? Leetcoding is like treadmills. You may have to do it, but it sucks. Yes, some people love treadmills, but they're in the minority.
There’s two kinds of leetcoding. Leetcoding via memorization obviously sucks, but leetcoding by clicking next problem until you get one thats totally alien and then letting it mull in the back of the head for a few days to crack it without reference material is pretty fun
After being a professional programmer for ~20 years, and recently playing around with leetcode - my main issue with leetcode is that there's almost no overlap between leetcode problems and the problems I actually encounter in the wild. The validation tests often have silly corner cases that force you into a single answer to avoid timing out. It's frequently as much work to understand what the problem is actually asking you as it is to implement a solution. Just like I've found ChatGPT to be pretty mediocre at writing the sort of code I work on, but others swear by it, maybe some peoples' dayjob actually looks like writing leetcode all day? I know a lot of interviewers use it, but it feels so disconnected from actual engineering work.
My line of work (ML for medical imaging) is pretty dense with leetcodelikes, especially the classic “what’s the best time complexity? Great now whats the best space conplexity”
I'm working on sort of 'graph' library. It's litcoding all the way. There are many separate containers and algorithms. The problem to a) write them b) optimize for memory c) optimize for performance d) find a 'good' balance where 'good' is undefined. But it starts with architecture which is based one some estimates of achievable functionality/performance.
the flip side to that “high” that comes with working super quickly for me has been a crash associated with a sinking feeling that I’ve outsourced too much.
I started using a C compiler for coding about 30 years ago. It's been great, but the feeling of skill atrophy is very real. I probably couldn't write useful code in x86 assembly any more, at least not without refreshing my memory first.
And you know what? That's just peachy keen. I don't need to write x86 assembly anymore. In fact, these days I do a lot of coding for ARM platforms, but I never learned ARM assembly at all. So it would take more than just a refresher course to bring me up to speed in that. I don't anticipate any such need, fortunately.
So... if I still need to write C in 10 years, why in the world would I consider that a good thing? Things in this business are supposed to progress toward higher and higher levels of abstraction.
What are some examples of prompts and responses that have made you pessimistic about the technology's ability to improve? Reliability has gotten insanely better over the past couple of years, and it's still improving.
> What are some examples of prompts and responses that have made you pessimistic about the technology's ability to improve?
I'm not saying it won't improve. I think its lack of determinism puts a natural upper bound on reliability - something we don't have to worry about with compilers.
As for examples: Oh wow. As much as I love and use them for coding, almost every project I work on has cases of coming up with complex solutions to a task that do not work, where 1-2 lines of (simple!) code would have solved the problem. It's not just that it fails, but it kept failing even after I told it how to do it right.
Sometimes it does a great job and solves a hard problem for me. Other times I lose a lot of time getting it to do beginner level stuff. It has huge gaps/blind spots.
This is a good analogy, and not just because of skills atrophy.
Managers grow the skills needed for their organization. Their team affects them.
A process-oriented team with quality/validation mindset has replaceable roles; the action is in the process. An expert team has people with tremendous skills and discretion doing what needs doing; the action is in selection and incentives. Managers adapt to their team requirements, in ways positive and negative: becoming rule-bound, privileging favorites, etc.
With AI this might be a positive insofar as it forces people to state the issues clearly, identifying relevant context, constraints, objectives, etc.
Agile benefitted software development by changing the granularity of delivery and planning -- essentially, helping people avoid getting lost in planning fantasies.
Similarly, I believe that the winner of the AI-for-development race (copilot et al) will not just produce good code, but build good developers by driving them to state requirements clearly and simply. A good measure here is the number of iterations to complete code.
An anti-pattern here, as with agile, is where planning devolves into exploring and exploring into incremental changes for no real benefit - polishing turds. Again, a good measure is the number of sessions to completion; too many and you know you don't know what you're doing, or the AI cannot grasp the trees for the forest.
I'm often surprised that a study like this is even needed, the result seems obvious.
Critical thinking is a skill that requires practice to improve at and maintain it. Using LLMs pushes the task that would require critical thinking off to something/someone else. Of course the user will get worse at critical thinking when they try to do it less often.
It seems like something like medical/legal professionals’ annual/otherwise periodic credential exams might make sense in fields where AI is very usable.
Basically, we might need to standardize 10-20% of work time being used to “keep up” automatable skills that once took up 80+% of work time in fields where AI-based automation is making things more efficient.
This could even be done within automation platforms themselves, and sold to their customers as an additional feature. I suspect/hope that most employers do not want to see these automatable skills atrophy in their employees, for the sake of long-term efficiency, even if that means a small reduction in short-term efficiency gains from automation.
> suspect/hope that most employers do not want to see these automatable skills atrophy in their employees, for the sake of long-term efficiency, even if that means a small reduction in short-term efficiency gains from automation.
I wish you were right, but I don't think any industry is realistically trending towards thinking about long term efficiency or sustainability.
Maybe it's just me, but I see the opposite, constantly. Everything is focused on the next quarter, always. Companies want massive short term gains and will trade almost anything for that.
And the whole system is set up to support this behavior, because if you can squeeze enough money to retire out of a company in as short a time as possible, you can be long gone before it implodes
Sort of like once you get used to GPS to get anywhere, you stop developing any further directional sense but even existing capabilities start withering away
This is interesting because I don't feel like my directional sense has withered at all because of GPS, but I do think it was important that I develop a sense of how to navigate and use maps before I introduced GPS.
I find this is similar in my experience with AI: I pick up tidbits and tricks from AI when it's doing something I'm familiar with, but if I have it working with a completely novel framework or language it quickly races ahead and I'm essentially steering it blind, which inevitably fails.
Usain Bolt didn't walk around on crutches all day.
Comedians' ability diminishes as they take time off.
Ahnold wasn't lounging around all day.
We should understand that fixing crap, unsensible code is not a productive skillset. As Leslie Lamport said the other day, logically developing and coding out proper abstractions is the core skillset, and not one to be delegated to just anything or anyone.
It's ok; the bright side for folks like me is that you're just happily hamstringing yourselves. I've been trying to tell y'all, but I can only show y'all the water, not make you drink.
Is this any different than saying that nowadays most people in the USA are physically weaker and less able to work on a farm than their predecessors? Sure, it's not optimal through certain lenses, but through other lenses it is an improvement. We are by any rights dependent on new systems to procure food, which is even more fundamental than other types of human cognition being preserved.
> Is this any different than saying that nowadays most people in the USA are physically weaker and less able to work on a farm than their predecessors?
Yes, far different, because we can still go to the gym and throw medicine balls around or swing kettle bells and do dead lifts and squats if we want to stay fit.
There is no substitute for exercising our ability to logically construct deterministic, hardened, efficient data flow networks that process specific inputs in specific environments to produce specific changes and outputs.
Maybe I'm the only one who understood the most important factor the eminent Leslie Lamport explained in grisly detail the other day, that, namely, logical thinking is both irreplaceable and essential. I'll add that that nerdiest of skillsets is also withering on the vine.
There is no substitute for exercising our ability to logically construct deterministic, hardened, efficient data flow networks that process specific inputs in specific environments to produce specific changes and outputs.
And every single microprocessor and their encompassing support systems and the systems they host and execute.
Every single system, even analog ones, because it's all just information flowing through switched systems, even if it's solely measured in something involving coulombs.
Also, fundamentally, living cells and the organisms that encompass them, because they all have a logically variable information flow both within them and between them, measured in molecules and energy.
This would be a valid POV if there was any solid evidence that LLMs truly increased worker productivity or reliability - at best it is a mixed bag. To stretch the food analogy, it seems like LLMs could be pure corn syrup, without any disease-resistant fruits and unnaturally plump chickens that actually make modern agriculture worthwhile.
Or, since LLMs seem to be addictive, it's like getting rid of the spinach farms and replacing them with opium poppies. (I really hate this tech.)
People pay a fortune and expend endless hours to replace the basic physical activity that used to be a default part of the human experience. Also a huge chunk of the population that doesn't suffers from life-altering metabolic disorders.
Just today had Gemini write a shell spot for me that had to generate a relative symlink..
Getting it to work xplat on linux & mac took more than ten tries and I stopped reading after the second
At the end, I spent probably more time and learnt nothing..
My initial take was that this is the kind of thing I don't care much for so giving it to a llm is OK... However, by the end of it I ended up more frustrated and lost it in the simulation of working things out aa well
I feel like my critical thinking has taken a nosedive recently, I changed jobs and the work in the new job is monotonous and relies on automation like copilot. Most of my day is figuring out why the ai code didn't work this time rather than solving actually problems. It feels like we're a year away from the me part being obsolete.
I've also turned to AI in side projects, and it's allowed me to create some very fast MVPs, but the code is worse than spaghetti - it's spaghetti mixed with the hair from the shower drain.
None of the things I've built are beyond my understanding, but I'm lazy and it doesn't seem worth the effort to use my brain to code.
Probably the most use my brain gets every day is wordle
144 comments
[ 5.7 ms ] story [ 191 ms ] thread“[A] key irony of automation is that by mechanising routine tasks and leaving exception-handling to the human user, you deprive the user of the routine opportunities to practice their judgement and strengthen their cognitive musculature, leaving them atrophied and unprepared when the exceptions do arise,” the researchers wrote.
But these have always been issues that humans commonly struggle with so idk.
[1]: https://en.wikipedia.org/wiki/Ironies_of_Automation
Submitters: please don't post paywalled articles unless there are workarounds (such as archived copies).
Fortunately, my ADHD-addled brain doesn't need some fancy AI to make its cognition "Atrophied and Unprepared"; I can do that all on my own, thank you very much.
Nah, it was already at zero before ChatGPT came to public attention.
Google helps you find things that you process later on with your brain.
With AI your brain shuts off as you offload all thinking to asking questions. And asking questions is not thinking. Answering them is.
I'm willing to bet that there were a lot of Google searches, pre-ChatGPT, that effectively were questions. Lots of "huh, I wonder" during conversations and the first result was taken as "the truth".
Now you are being spoon-fed.
I also believe, however, that humans who are able to reason properly would become much more valuable, because of this same thing.
While much is made of the 'diminished skill for independent problem-solving' caused by over-reliance, is there a more salient KPI than some iteration of this 'Synthetic Thinking Effort' by which to baseline and optimise the cost/benefit of AI usage versus traditional cognition?
Whereas, you can ask an LLM to speak to you in e.g. Spanish, about whatever topic you're interested in, and be able to stop and ask it to explain any idioms or vocabulary or grammar in English at any time.
I found this to be more like a "cognitive gym". Maybe we're just not using the tools beneficially.
I think that's really just it, and I agree with you. There are many other areas LLMs can, and should, be more useful and effort put toward both assisting and automating.
Instead, the industry is focusing on creative arts and software development because human talent for that is both limited and expensive, with a third factor of humans generally being able to resist doing morally questionable things (e.g., what if hiring for weapons systems software becomes increasingly difficult due to a lack of willingness to work on those projects, likewise for increasingly invasive surveillance tech, etc.)
We're rushing into basically the opposite of what AI should do for us. Automation should work to free us up to focus more on the arts and sciences, not take it away from us.
Greed at its finest.
Some discussion on the study: https://news.ycombinator.com/item?id=43057907
The seductive promise of solving all your problems is the issue. By reaching for it to solve any problem at an almost instinctual level you are completely failing to cultivate an intrinsically valuable skill - that of critical reasoning.
That act of manipulating the problem in your head—critical thinking—is ultimately a craft. And the only way to become better at it is by practicing it in a deliberate, disciplined fashion.
This is why it's pretty baffling to me when I see attempts at comparing LLMs to the invention of the calculator. A calculator is still used IN SERVICE of a larger problem you are trying to solve.
With that said, I do worry that losing the ability to craft sentences (or code) is more problematic than losing the ability to do mental math.
https://www.reddit.com/r/interestingasfuck/comments/13jhckh/...
But I am older than many. :-)
Losing the ability to do calculations by hand on a piece of paper with a pencil probably actually is a big deal
When I went to school we still had to do a lot of calculations by hand on paper. Thus, if I use a calculator to get an answer, I'm capable of reproducing the answer by hand if necessary
With math, at least when I was learning it, we seemed to understand that the calculator is a useful tool that doesn't replace the need to develop underlying skills
I'm seeing the exact opposite behavior and mentality from the AI crowd. "You don't need to learn how to do that correctly anymore, you can just have the AI do it"
"Vibe Coding", literally the attitude that you don't need to understand your code anymore. You just have the AI generate it and go off vibes to decide if it's right or not
Yeah, I don't know how my car engine works. But I trust that the people who engineered it do, and the mechanics that fix it when it breaks do. There's no room for "Vibe Bridge Building" in reality
Anyone advocating for "Vibe coding" is an admission that it doesn't actually matter if the thing they build works or not
Unfortunately that seems to be a growing portion of software
There are a lot of calculators and other tools that can push the symbols around and many that can even apply some special high-level mathematical rules to the symbols. But whether or not those rules are relevant to the task at hand is entirely a matter for a human to decide.
Brace yourself for a wave of think pieces a year from now, all wringing their hands about the death of vibe coding, and the return of artisanal handmade software.
I think it’s a huge deal. I see a lot of people do some financing stuff and they have no idea what a 20% interest rate really means. So they just go ahead and do it because taking out a calculator is too tedious. I find it pretty crazy how many can’t figure how much a 20% or even 10% is. A lot of financial offerings take advantage of the fact that people can’t do even basic math.
Edit: At best, it shows that the person doesn't really know how to calculate that even with a calculator
At worst, it shows that the person lacks of any kind of give a fuck at all
Either way, they probably would not likely have learned the mental math required to do this regardless of if it were being taught at school or not
What it shows is that 10% means nothing to them. It's just another number that a calculator will spit out. It's not "one out of ten" or "shift everything to the right." They have no ability to evaluate what the number means on the fly. They just plug it into the calculator and check if they have that much. They can't have a reasonable discussion about the number without stopping and calculating every interest rate they can think of, writing them down on a piece of people, and writing the budget next to them.
A calculator here, of course, leads to more work, not less. I bet the vast majority of the time that somebody too lazy or anxiety-ridden to learn how to calculate 20% in their head is also going to be too lazy and bothered to take out a calculator every time they need to know, rather than just trying to bluff their way through conversations until they can get to a calculator.
Calculators are of no benefit to students who have no clear concept of the calculation. Just learn your times tables, they're one of the first things we teach kids.
The pragmatic pro-calculator "nobody needs to calculate those numbers in real life" school I think gets most of its support because constructivists often fail entirely to teach things that have been until now best learned by rote, hoping that an instinct for multiplication and division will naturally and precisely arise from children's mathematical souls. The fact that it doesn't is proof that doing arithmetic is bad and a waste of time actually. They love abstractions, but only vague ones that can't be reliably tested.
> Either way, they probably would not likely have learned the mental math required to do this regardless of if it were being taught at school or not
That's the law of averages. It is not a law.
Compound interest is hard to do with mental math, but what you're actually seeing is a behavioral economics phenomenon called hyperbolic discounting.
I'm not sure where the line is, but there is a point where the abstraction works so well you really don't need to know how it works underneath.
I'm also not sure if a car mechanic needs to know how an engine works. I'm assuming almost none of them could design a car engine from scratch. They know just enough to know which parts needs to be replaced.
There is a point where most people might not need to know
There is never a point where no one needs to know
That's why, when your car has a problem, a lot of mechanics just blindly replace parts with the hope that something will fix it. You are much better off with a mechanic that understands how the car works. And you will save a lot of money.
Similarly we reach a point in coding where you don't really need to know how every API or language you use operates beneath the hood, you just need to be able to see where its broken.
In the US :-)
And those skills are entirely context dependent. You're likely saying this from a SW engineer's point of view. Whereas I've worked in teams with physicists and electrical engineers. When you're in a meeting, and there is a technical discussion, and everyone can calculate in their head the effects after integrating a well known function and how that will impact the physical system, while you have to pull out a calculator/computer, you'll be totally lost.
You can argue that you could be as productive as the others if you were given the time (e.g. doing this on your own at your own cubicle), but no one will give you that time in meetings.
Source?
I only tend to do that when I am tired or annoyed, but when I do it I can feel myself getting dumber. And it’s a weirdly satisfying feeling.
I just need a chair that doubles as a toilet and I’ll be all set.
If you don't understand what's happening, you have no way to know if the system is working as intended. And understanding (and deciding) exactly how the system works is the really hard part for any sufficiently complex project.
We have the cognition science to make it happen - or at least learn how to structure it.
> Abstract The rise of Generative AI (GenAI) in knowledge workflows raises questions about its impact on critical thinking skills and practices. We survey 319 knowledge workers to investigate 1) when and how they perceive the enaction of critical thinking when using GenAI, and 2) when and why GenAI affects their effort to do so. Participants shared 936 first-hand examples of using GenAI in work tasks. Quantitatively, when considering both task- and user-specific factors, a user’s task-specific self-confidence and confidence in GenAI are predictive of whether critical thinking is enacted and the effort of doing so in GenAI-assisted tasks. Specifically, higher confidence in GenAI is associated with less critical thinking, while higher self-confidence is associated with more critical thinking. Qualitatively, GenAI shifts the nature of critical thinking toward information verification, response integration, and task stewardship. Our insights reveal new design challenges and opportunities for developing GenAI tools for knowledge work.
It is be presented at CHI Conference https://chi2025.acm.org/
https://en.wikipedia.org/wiki/Conference_on_Human_Factors_in...
https://www.semanticscholar.org/paper/The-Impact-of-Generati...
Ie. most managers can't help their team find a hard bug that is causing a massive outage.
Note: I'm a manager, and I spend a lot of time pondering how to spend my time, how to be useful, how to remain relevant, especially in this age of AI.
Perhaps the biggest difference is the lack of feedback AI gives that humans can give: a subordinate can communicate if they feel like their manager is being too hands off. AI never questions management style.
Do all the work Farts_Mckensy does at half the price using AI.
Strategy vs tactics. Managers aren't there to teach their reports skills, they hire them because they already have them. They're there to set priorities and overall direction.
While the manager doesn't need those skills, you still do.
I think in the coming decade if you put yourself in a box and do a specialized task, and nothing more, you'll have a bad time. This is going to be an era where strategy is far more important than tactics.
I can't disagree more strongly. If you're a specialized person with expertise and ability to perform outside the knowledge of all possible interns because you're developing something novel and not covered by standard materials, you'll be hard to compete with using AI because AI will direct people to standardized methods.
Granted, if you do a specialized task that is taught in schools and that anyone can do, that's trouble. But that's not tactics either, that's clock-punching. That's replaceable. You can talk 'strategy' all you like but if you're only exploiting the AI's ability to do what people already know, that's another box to be stuck in.
> if you do a specialized task that is taught in schools
are in trouble, and so are you.
> perform outside the knowledge of all possible interns
This is the strategic level thinking I'm referring to. Frankly this role might not be safe either, but if that's the case then we're probably headed for fully automated luxury communism no matter what we do.
But the feeling of skill atrophy is very real. The other day I needed to write a function that recursively traverses a Python dictionary, making modifications along the way. It's the perfect task to give an LLM. I can easily see that if I always give this task to an LLM, a few years down the road I'll be really slow in writing it on my own, and will fail most coding interviews.
Also, while there is a high in producing a lot in a short amount of time, there is no feeling of satisfaction that you get from programming. And debugging its bugs is as much fun as debugging your coworkers' bugs - except at least with the coworkers you can have a more intelligent conversation on what they were trying to do.
This happens a lot in the natural world in ecosystems. For example, many people plant trees and add a drip system. The trees grow to depend on the drip system, and never stretch and develop their roots -- and the relationship they have with the soil microbiome. They make the trees prone to get knocked down when an unusually strong gust of wind come through.
Just beautiful concepts, inspiring, and well-represented in this comment.
Peace be with you, my friend.
And this is what worries me. Pre-LLM, I'd get all my practice done during work hours. I fear that with LLMs, I'll need to do all this practice on my free time, effectively reducing my truly "free" time.
If you can attach a valid business reason/excuse then it's fairly easy to get time dedicated to experimentation and learning.
I haven't been a coder for several years now, but I still try to justify a personal lab environment in order to understand the market and customer problems.
If you're an Engineer or EM, I think it would be even easier (attach experimentation to some sort of critical engineering initiative).
Even with Code Gen Copilots becoming a thing, between the lines most of us in engineering and business leadership recognize it cannot replace architecture or design. If it can, then you've solved the Turing or Chinese Room problem and that's a whole other story.
That said, if your day job isn't only "write this terraform/python script" you shouldn't be at risk.
This has the same problem as "doing it in free time". I have a finite number of goodie points with management, and I'd be using some when I request this - which means I have fewer things I can request from them in the future.
> That said, if your day job isn't only "write this terraform/python script" you shouldn't be at risk.
I'm not worried about the present, or even the near future. The reason I have other valuable skills is because of the years I spent on writing code. Eventually, I'll stop growing if I rely too much on LLMs. I may have to rely on them if everyone else is and being more productive.
Great points otherwise, though. A strong focus on identifying, understanding, and solving customer problems is, in modern parlance, all you need.
On the other hand, there is something I learned from permaculture design. You need to allow what you want to grow in the ecosystem if you want them present. If I wanted to develop the next generation of developers, then I'd be giving that CRUD stuff I've done many times to a junior developer instead. It obviously depends on funding, and whether your investors are on board with that.
As far as keeping up and remaining competitive ... my suggestion is to learn these things in a way where you can broadly apply the underlying principles in other domains, especially that, as you age, your working memory will shrink. This isn't accumulating more of the same thing, but gaining deeper insights that can only be found when you've developed a sufficient level of skill. There are lots of ways this can be accomplished.
Indeed, one of my fears is that people won't develop those skills to begin with. In my team, the few senior people can pretty much be trusted with using LLMs and still producing quality code. But we decided against evangelizing it to the junior folks. They still have access via Copilot, but we won't gently suggest they try it out.
I got where I am without regular leetcoding. For me, regular leetcoding will only marginally improve my skills. And I definitely don't want to do regular leetcoding just to maintain my skills.
You know how many people love jogging outdoors but hate treadmills? Leetcoding is like treadmills. You may have to do it, but it sucks. Yes, some people love treadmills, but they're in the minority.
And you know what? That's just peachy keen. I don't need to write x86 assembly anymore. In fact, these days I do a lot of coding for ARM platforms, but I never learned ARM assembly at all. So it would take more than just a refresher course to bring me up to speed in that. I don't anticipate any such need, fortunately.
So... if I still need to write C in 10 years, why in the world would I consider that a good thing? Things in this business are supposed to progress toward higher and higher levels of abstraction.
I am extremely pessimistic that LLMs will ever reach that level of reliability. Or even close. They are a great helper, but that's all they'll be.
I'm not saying it won't improve. I think its lack of determinism puts a natural upper bound on reliability - something we don't have to worry about with compilers.
As for examples: Oh wow. As much as I love and use them for coding, almost every project I work on has cases of coming up with complex solutions to a task that do not work, where 1-2 lines of (simple!) code would have solved the problem. It's not just that it fails, but it kept failing even after I told it how to do it right.
Sometimes it does a great job and solves a hard problem for me. Other times I lose a lot of time getting it to do beginner level stuff. It has huge gaps/blind spots.
Managers grow the skills needed for their organization. Their team affects them.
A process-oriented team with quality/validation mindset has replaceable roles; the action is in the process. An expert team has people with tremendous skills and discretion doing what needs doing; the action is in selection and incentives. Managers adapt to their team requirements, in ways positive and negative: becoming rule-bound, privileging favorites, etc.
With AI this might be a positive insofar as it forces people to state the issues clearly, identifying relevant context, constraints, objectives, etc.
Agile benefitted software development by changing the granularity of delivery and planning -- essentially, helping people avoid getting lost in planning fantasies.
Similarly, I believe that the winner of the AI-for-development race (copilot et al) will not just produce good code, but build good developers by driving them to state requirements clearly and simply. A good measure here is the number of iterations to complete code.
An anti-pattern here, as with agile, is where planning devolves into exploring and exploring into incremental changes for no real benefit - polishing turds. Again, a good measure is the number of sessions to completion; too many and you know you don't know what you're doing, or the AI cannot grasp the trees for the forest.
Critical thinking is a skill that requires practice to improve at and maintain it. Using LLMs pushes the task that would require critical thinking off to something/someone else. Of course the user will get worse at critical thinking when they try to do it less often.
Basically, we might need to standardize 10-20% of work time being used to “keep up” automatable skills that once took up 80+% of work time in fields where AI-based automation is making things more efficient.
This could even be done within automation platforms themselves, and sold to their customers as an additional feature. I suspect/hope that most employers do not want to see these automatable skills atrophy in their employees, for the sake of long-term efficiency, even if that means a small reduction in short-term efficiency gains from automation.
I wish you were right, but I don't think any industry is realistically trending towards thinking about long term efficiency or sustainability.
Maybe it's just me, but I see the opposite, constantly. Everything is focused on the next quarter, always. Companies want massive short term gains and will trade almost anything for that.
And the whole system is set up to support this behavior, because if you can squeeze enough money to retire out of a company in as short a time as possible, you can be long gone before it implodes
I find this is similar in my experience with AI: I pick up tidbits and tricks from AI when it's doing something I'm familiar with, but if I have it working with a completely novel framework or language it quickly races ahead and I'm essentially steering it blind, which inevitably fails.
Comedians' ability diminishes as they take time off.
Ahnold wasn't lounging around all day.
We should understand that fixing crap, unsensible code is not a productive skillset. As Leslie Lamport said the other day, logically developing and coding out proper abstractions is the core skillset, and not one to be delegated to just anything or anyone.
It's ok; the bright side for folks like me is that you're just happily hamstringing yourselves. I've been trying to tell y'all, but I can only show y'all the water, not make you drink.
Yes, far different, because we can still go to the gym and throw medicine balls around or swing kettle bells and do dead lifts and squats if we want to stay fit.
There is no substitute for exercising our ability to logically construct deterministic, hardened, efficient data flow networks that process specific inputs in specific environments to produce specific changes and outputs.
Maybe I'm the only one who understood the most important factor the eminent Leslie Lamport explained in grisly detail the other day, that, namely, logical thinking is both irreplaceable and essential. I'll add that that nerdiest of skillsets is also withering on the vine.
"Enjoy." --Daniel Tosh
Factorio?
And every single microprocessor and their encompassing support systems and the systems they host and execute.
Every single system, even analog ones, because it's all just information flowing through switched systems, even if it's solely measured in something involving coulombs.
Also, fundamentally, living cells and the organisms that encompass them, because they all have a logically variable information flow both within them and between them, measured in molecules and energy.
They're extraordinary and beautiful.
Or, since LLMs seem to be addictive, it's like getting rid of the spinach farms and replacing them with opium poppies. (I really hate this tech.)
Let's... not do that for brainrot.
At the end, I spent probably more time and learnt nothing.. My initial take was that this is the kind of thing I don't care much for so giving it to a llm is OK... However, by the end of it I ended up more frustrated and lost it in the simulation of working things out aa well
I've also turned to AI in side projects, and it's allowed me to create some very fast MVPs, but the code is worse than spaghetti - it's spaghetti mixed with the hair from the shower drain.
None of the things I've built are beyond my understanding, but I'm lazy and it doesn't seem worth the effort to use my brain to code.
Probably the most use my brain gets every day is wordle