IDEs, static linters, and syntax highlighters aren't designed to replace any part of the problem solving process, but rather augment them. It's impossible to become dependent on any of these tools to the point where your cognitive skills begin to atrophy, unlike LLMs [1].
IDEs, static linters, and syntax highlighters aren't the cause of a global microchip shortage, immensely increased energy demand, or the destruction of rare works. They have not, at any point, been the cause of effectively DDoS attacks in an effort to slurp up all of the internet's data, or the extensive proliferation of hallucinated content on the internet.
The development of IDEs, static linters, and syntax highlighters has at no point employed low-paid labor from third world countries for data labelling or safety filtering, exposing thousands of people to violent and abusive content for multiple hours a day.
Any of these issues alone would be a reasonable cause for adopting a policy to avoid using such tools. Calling it equivalent to not using IDEs or linters is bizarre to me.
> AI will lead to a tyrannical police state
> What will happen once the rug invariably gets pulled? Those data centers will start serving another purpose.
> All around the world, governments are assembling massive networks of cameras, “license plate readers” that are suspiciously equipped with microphones sensitive enough to listen in on your conversations, and other unimaginable surveillance devices.
> All that computing power built up under the guise of bringing about the “AI future” will start being used to spy on you, follow your every movement, listen to all your calls, and more.
Isn't there a case to be made that this is/already has been the case and is not uniquely an AI thing? The governements of the world weren't waiting for ChatGPT to go viral before deciding to spy on their citizens so why we blame AI for this?
Are you saying large language models can better for surveillance? The article says all the data center power will be used to spy on you but that makes 0 sense, why would you need that much power to do what you have already done for so long so well? What is the LLM adding here that makes us scale this to previously impossible levels?
This is the truth! We are living in a surveillance state that would make Stalin green with envy.
We are also seeing daily examples of this surveillance state being used against political opponents, powerful bloggers and journalists among the alternative media, and regular citizens who said the wrong thing at the wrong time.
Since human beings are easily corrupted by power, this system will be weaponized more and more, and democracy will be hacked away at as well, until we are living in an authoritarian society with a democratic fig leaf, as we can see among the dirka-dirka-stans of the world.
> All that computing power built up under the guise of bringing about the “AI future” will start being used to spy on you, follow your every movement, listen to all your calls, and more.
This post sounds like someone who lives in deep in an echo chamber and has a warped reality because of it. That said I read this point and was like yup they’re right about one thing anyway.
Believe it or not, the very first LLM was actually invented by Nazis who live on the dark side of the moon. The A in AI? Adolf.
How many articles do we really need about how AI will break into your house and steal your TV, narrated from the perspective of someone who thinks Terminator was realistic? This in particular feels like a shallow marketing ploy to appeal to luddites; especially since as simonw has discovered, it depends on code produced by Claude anyway.
A lot of these criticisms seem to be very 2023-coded.
There have been a lot of hiccups to the transition towards using AI-assisted code but, ignoring some of that sloppiness, I've really enjoyed the product velocity from some of the services/apps I know that heavily use coding agents.
It occurred to me as well that this is ragebait trolling too since being disconnected this much from reality would have been concerning. The author's app can be rewritten with the use of AI in one day (pessimistically speaking).
That certainly couldn't be why they feel that threatened. But just think, using AI to build it will literally destroy Bambi's entire forest. Just that one prompt. Think about it.
Handcrafted software will find it's own market the same way that handcrafted clothing, furniture and other previously industrialized goods that became commodities did. They'll be more expensive generally and it'll be a smaller market overall, you'll always have some people who will love them all the same. Not my choice, but I can understand why they would want it.
The difference is that one is physical and the other digital. Digital art, even if exceptionally high quality, has been commoditized by artists from developing countries asking dumping wages, and not recovered since. People who have enough money to buy for the "soul" of an object, and the labor behind it, usually buy physical.
I agree with you but I believe the definition of handcrafted will shift to include software produced partly by AI, just as most handcrafted things involve industrial machinary at some point.
But then, I can also imagine a world where nobody cares enough about writing code and instead we have "human designed software" just as we currently have human designed cars and fridges but few made by hand (and no market for them). Then, handwritten software becomes an impressive curiosity like Chris Sawyer's original Rollercoaster Tycoon written entirely in ASM.
I doubt it. Software is, well, software. Most people have their aesthetic preferences in regard to clothing or furniture, and aesthetics of these are obvious to anyone. Code that runs on a device and implements some functionality is not like it. It is more like infrastructure - nobody cares about handcrafted sewage pipes. Plumbers possibly can have strong preferences regarding plumbing, but other people don't.
Whether or not this person uses AI is their concern, but I see this get bandied about a lot and I think it's worth talking about,
Large Language Models are nothing but a fancy autocorrect. Where they differ from your phone’s keyboard is the size of their memory, which they can use to predict the possible next word, based on a huge database of previous words. But that’s all it is, a prediction.
The AI doesn’t know or understand what it’s writing. Give the same AI the same prompt, and watch it give you two completely different answers. It can guess, but it will never know. Fixing this would require a complete rebuilding of the way the Large Language Model operates.
To plagiarize myself, it is indeed true that all models are, at their core, predictors of what occurs next in a sequence. But I think it's worth exploring the implication of what that means.
If you've tried making one yourself, you'll find that when fed tiny pieces of information for a few tasks at a small scale, this technique results in something that sorta, kinda works. Or, works surprisingly well.
But when scaled... When the amount of information starts approaching the sum of all human knowledge, the tasks start approaching all useful applications of that human knowledge, and the fidelity of the predictor approaches incomprehensible sizes, the starts encodes / becomes (I'd argue it becomes) something that can model all human knowledge.
It feels wrong to say that, but let me explain, what is the best way to predict the behavior of a ball constrained in two directions that bounces with initial vertical velocity v(y) (y is up / down axis) and horizontal velocity v(x) (x is side by side in 1d) ?
If we purely look at it via a graph, it's by modelling the function of acceleration under earth's gravity.
If only a few points are given to you for this and you can't make something really sophisticated, then you'll make something that's rough that kinda sorta works and then call it a day.
But... if the number of points keeps increasing in number, precision and accuracy as well as the number of examples (assumed that data about air pressure, velocity and all other factors is included alongside these points), the fidelity with which you can replay / tweak the function keeps improving, and the number of times you can iterate keeps increasing, you'll eventually create a function that models that process so well that it intrinsically contains a good enough model of the deformation of the ball (provided the dataset contains information about elasticity of the ball's material, its dimensions and mass etc..), the nearly negligible (under normal conditions) effects of the ambient environment (provided there's diversity in the number of environments supplied), the oblateness of the Earth and minute changes in the gravitational field (the length of a seconds pendulum varies depending on where the experiment happens. It's presumed that all of the prior set of experiments were repeated across the Earth and the subtle, but real deviations were faithfully recorded)... and so much more.
A machine trained on the above with a large number of parameters, measures to prevent "laziness" and enough reps for high fidelity across a large enough dataset would start to approach a simulation of the ball falling. Because to predict what happens next in the sequence, you must model what's occurring in the sequence.
Now imagine doing that for other tangible and intangible things in this world. For all of human knowledge across all fields of endeavor. All experiences. No matter how noble, ignoble, notable or ignorable. But putting all of it into the soup that's this machine. Then at larger and larger scales, you eventually start encountering "good enough" models (in modelling the falling ball sense) for even the most hard to quantify / qualify things like grief and joy. At some point, by simply trying to predict wha...
I appreciate the example you’re using of the bouncing ball because it shows how additional data helps more accurately model the event and thus offers a more precise and correct prediction. However, that better modeling is based on the original data being correct, true, or factual. If LLMs are ingesting all written human knowledge (theoretically), then they are ingesting “truth” as well as errors, falsehoods, and subjective beliefs that don’t necessarily reflect accepted practices. Right now, they don’t seem fully capable of differentiating between these, which is why they sometimes produce bizarre inaccuracies for things like world events, history, culture, etc.
Just because LLMs are (or can be) reliable predictors in some domains does not mean they are (or can be) in all domains.
Not necessarily. No sensor is reliable, all sensors have noise and they drift. It's why you use something like a kalman filter.
It's worth doing it over an afternoon, but you can do very simple curve fitting and you'll see it naturally starts to ignore extremes / becomes accurate.
As long as there are enough correct data points, the incorrect data shouldn't matter. How much is enough is a black art though.
They're right about everything, but at the same time it actually has become useful.
Sensible not to depend on it and create a vibe-coded mess, but keep everything modular & review the generated code closely and you can go back to humans-only in future with no problems.
If someone offered a whole team of devs in India for an unknown number of months basically for free, one would be a fool to turn that down even knowing it wouldn't last forever. Claude etc feels like the same kinda deal.
> If someone offered a whole team of devs in India for an unknown number of months basically for free, one would be a fool to turn that down even knowing it wouldn't last forever. Claude etc feels like the same kinda deal.
I've made a lot of money fixing software where people paid for those offshore teams and they would have been better off having not done that and writing maintainable software from the start. They were fools paying for it and they still would have been fools if it had been free.
Sure, I've been part of "in-sourcing" failing projects too, but it was always a sign of management incompetence rather than anything else. It's easy to shift the blame to an outsourcer when it's been GIGO; constantly changing requirements, feature creep, no budget for maintenance, the usual stuff that kills codebases.
This feels like saying you'll never use self-driving cars.
Sure, LLM effectiveness might seem debatable to some (not for me), but these things improve frequently. So saying you'll never use it, is a red flag to me and I question what other technologies and advancements you ignore.
I use it for new code base summary, documentation lookup, code review, improvement suggestions, security audit, everything but writing code, I still do that by hand. I'm more productive and my skills improve. Reviewing slop burns me out. I can't be the only one.
I think there's merit to it. It's not as if doing it exclusively with AI is all sunshine and rainbows. As someone doing agentic engineering at work I'm actually starting to sour on it after seeing what it's doing to people's ability to think for themselves and even do basic software engineering tasks.
I do not really disagree with the overall claims but the extent is IMO exaggerated.
Yes, the billionaire nerds were parading around hoping to replace me with a loop, yes, your skill atrophy, yes, you need to pay attention to details because it will happily lie to your face, yes, sometimes it hallucinates, yes it fails miserably on the most trivial tasks...
But also! Sometimes a bug fix is a prompt and me getting a coffee in the meantime. Sometimes it catches stuff no human would. Sometimes a task that would have been a week is ready in an hour. Sometimes the free models are plenty enough, with more and more providers expected to spawn.
I do not feel the need to think so black and white. I feel I need to be careful how I continue, what I focus on, what I learn and what I delegate.
Is it just me or if you replace in the article "A.I." with "Humans", it would still be plausible? Don't get me wrong, I think those are fine points, but I see as many applications of the same situations that would be generated by A.I. that were and are actively generated by humans.
"AI is designed to make you dependent on it" - pre-AI service subscriptions - music, movies, you name it.
"AI is not intelligent, and hallucinations are a feature" - yes, pretty much the same way most (if not all) humans I ever interacted with hallucinate some times; me included. And I don't think it needs fixing (it's not a disease, it's a feature for us also), it just needs dealing-with.
"AI is great at lying. And that’s it." - Where shall I start about humans lying and being great at it? I just can't seem to find a good example....
"AI is killing the environment and destroying people’s lives" - pretty much humans have been inflicting this kind of damage onto themselves and our environment since the start of time, and even more so once doing this accelerated due to heavy machinery that can do what humans never could.
"AI will lead to a tyrannical police state" - Again, don't get me started on the ingenuity of humans building tyrannical police states.
69 comments
[ 2.0 ms ] story [ 3.2 ms ] threadAlso there are already non-VC subsidized hosted open weights LLM providers which should give you a very good sense of the marginal cost.
IDEs, static linters, and syntax highlighters aren't the cause of a global microchip shortage, immensely increased energy demand, or the destruction of rare works. They have not, at any point, been the cause of effectively DDoS attacks in an effort to slurp up all of the internet's data, or the extensive proliferation of hallucinated content on the internet.
The development of IDEs, static linters, and syntax highlighters has at no point employed low-paid labor from third world countries for data labelling or safety filtering, exposing thousands of people to violent and abusive content for multiple hours a day.
Any of these issues alone would be a reasonable cause for adopting a policy to avoid using such tools. Calling it equivalent to not using IDEs or linters is bizarre to me.
* [1]: https://www.media.mit.edu/publications/your-brain-on-chatgpt...
Isn't there a case to be made that this is/already has been the case and is not uniquely an AI thing? The governements of the world weren't waiting for ChatGPT to go viral before deciding to spy on their citizens so why we blame AI for this?
Because AI can do this at a scale previously impossible.
We are also seeing daily examples of this surveillance state being used against political opponents, powerful bloggers and journalists among the alternative media, and regular citizens who said the wrong thing at the wrong time.
Since human beings are easily corrupted by power, this system will be weaponized more and more, and democracy will be hacked away at as well, until we are living in an authoritarian society with a democratic fig leaf, as we can see among the dirka-dirka-stans of the world.
This post sounds like someone who lives in deep in an echo chamber and has a warped reality because of it. That said I read this point and was like yup they’re right about one thing anyway.
How many articles do we really need about how AI will break into your house and steal your TV, narrated from the perspective of someone who thinks Terminator was realistic? This in particular feels like a shallow marketing ploy to appeal to luddites; especially since as simonw has discovered, it depends on code produced by Claude anyway.
> AI is not intelligent, and hallucinations are a feature
How the times have changed... I would have agreed two years ago.
There have been a lot of hiccups to the transition towards using AI-assisted code but, ignoring some of that sloppiness, I've really enjoyed the product velocity from some of the services/apps I know that heavily use coding agents.
But maybe it was written years ago idk.
But then, I can also imagine a world where nobody cares enough about writing code and instead we have "human designed software" just as we currently have human designed cars and fridges but few made by hand (and no market for them). Then, handwritten software becomes an impressive curiosity like Chris Sawyer's original Rollercoaster Tycoon written entirely in ASM.
If you've tried making one yourself, you'll find that when fed tiny pieces of information for a few tasks at a small scale, this technique results in something that sorta, kinda works. Or, works surprisingly well.
But when scaled... When the amount of information starts approaching the sum of all human knowledge, the tasks start approaching all useful applications of that human knowledge, and the fidelity of the predictor approaches incomprehensible sizes, the starts encodes / becomes (I'd argue it becomes) something that can model all human knowledge.
It feels wrong to say that, but let me explain, what is the best way to predict the behavior of a ball constrained in two directions that bounces with initial vertical velocity v(y) (y is up / down axis) and horizontal velocity v(x) (x is side by side in 1d) ?
If we purely look at it via a graph, it's by modelling the function of acceleration under earth's gravity.
If only a few points are given to you for this and you can't make something really sophisticated, then you'll make something that's rough that kinda sorta works and then call it a day.
But... if the number of points keeps increasing in number, precision and accuracy as well as the number of examples (assumed that data about air pressure, velocity and all other factors is included alongside these points), the fidelity with which you can replay / tweak the function keeps improving, and the number of times you can iterate keeps increasing, you'll eventually create a function that models that process so well that it intrinsically contains a good enough model of the deformation of the ball (provided the dataset contains information about elasticity of the ball's material, its dimensions and mass etc..), the nearly negligible (under normal conditions) effects of the ambient environment (provided there's diversity in the number of environments supplied), the oblateness of the Earth and minute changes in the gravitational field (the length of a seconds pendulum varies depending on where the experiment happens. It's presumed that all of the prior set of experiments were repeated across the Earth and the subtle, but real deviations were faithfully recorded)... and so much more.
A machine trained on the above with a large number of parameters, measures to prevent "laziness" and enough reps for high fidelity across a large enough dataset would start to approach a simulation of the ball falling. Because to predict what happens next in the sequence, you must model what's occurring in the sequence.
Now imagine doing that for other tangible and intangible things in this world. For all of human knowledge across all fields of endeavor. All experiences. No matter how noble, ignoble, notable or ignorable. But putting all of it into the soup that's this machine. Then at larger and larger scales, you eventually start encountering "good enough" models (in modelling the falling ball sense) for even the most hard to quantify / qualify things like grief and joy. At some point, by simply trying to predict wha...
Just because LLMs are (or can be) reliable predictors in some domains does not mean they are (or can be) in all domains.
It's worth doing it over an afternoon, but you can do very simple curve fitting and you'll see it naturally starts to ignore extremes / becomes accurate.
As long as there are enough correct data points, the incorrect data shouldn't matter. How much is enough is a black art though.
Sensible not to depend on it and create a vibe-coded mess, but keep everything modular & review the generated code closely and you can go back to humans-only in future with no problems.
If someone offered a whole team of devs in India for an unknown number of months basically for free, one would be a fool to turn that down even knowing it wouldn't last forever. Claude etc feels like the same kinda deal.
I've made a lot of money fixing software where people paid for those offshore teams and they would have been better off having not done that and writing maintainable software from the start. They were fools paying for it and they still would have been fools if it had been free.
Sure, LLM effectiveness might seem debatable to some (not for me), but these things improve frequently. So saying you'll never use it, is a red flag to me and I question what other technologies and advancements you ignore.
It depends on Factory: https://github.com/hmlongco/Factory/blob/main/CLAUDE.md
Sticking to a strident "No AI" policy developing software is going to become increasingly difficult.
Yes, the billionaire nerds were parading around hoping to replace me with a loop, yes, your skill atrophy, yes, you need to pay attention to details because it will happily lie to your face, yes, sometimes it hallucinates, yes it fails miserably on the most trivial tasks...
But also! Sometimes a bug fix is a prompt and me getting a coffee in the meantime. Sometimes it catches stuff no human would. Sometimes a task that would have been a week is ready in an hour. Sometimes the free models are plenty enough, with more and more providers expected to spawn.
I do not feel the need to think so black and white. I feel I need to be careful how I continue, what I focus on, what I learn and what I delegate.
If you’ve done a proper cost-benefit analysis and truly found that it’s not yet worth it for your use case, great.
But if you’ve decided it _would_ be helpful but then decided ostentatiously not to use it so you can farm clout on Twitter… not so good.
"AI is designed to make you dependent on it" - pre-AI service subscriptions - music, movies, you name it.
"AI is not intelligent, and hallucinations are a feature" - yes, pretty much the same way most (if not all) humans I ever interacted with hallucinate some times; me included. And I don't think it needs fixing (it's not a disease, it's a feature for us also), it just needs dealing-with.
"AI is great at lying. And that’s it." - Where shall I start about humans lying and being great at it? I just can't seem to find a good example....
"AI is killing the environment and destroying people’s lives" - pretty much humans have been inflicting this kind of damage onto themselves and our environment since the start of time, and even more so once doing this accelerated due to heavy machinery that can do what humans never could.
"AI will lead to a tyrannical police state" - Again, don't get me started on the ingenuity of humans building tyrannical police states.
[Edit: typos]