I've noticed several companies replacing deterministic systems in their support flows with a LLM version that is slower and worse. Many interfaces simply aren't better with AI added
Anyone who does a Google search gets a satisfactory looking answer as the very first entry. I daresay most people don't go beyond that, not even the entries on the first page, let alone go to the next.
I argue that this is at the level of everyone for everything.
A "satisfactory looking answer" was what I got yesterday when I queried Google about a Pyodide question. It produced some code in html format that was supposed to work, but failed on execution. The AI generated result was incorrect and it was only 30ish lines of code that was supposed to print "hello world" to the console.
It's been difficult to do, but I have recently picked up the habit of scrolling directly past the AI overview. It's not what I wanted, so it's useless to me (regardless of whether or not it's correct). I wanted search results. Sometimes that's a Wikipedia page, documentation, video results. But, web search results.
If I wanted to read an AI response to a query, I would use that tool specifically - I consider them different tools entirely.
It's definitely taken some time to get into the habit - perhaps defaulting to Google as the search engine was my first mistake.
However, most people I've seen run a quick search DO in fact just read the AI response (often with a caveat of "this is just from AI but..." which I appreciate), because for a lot of things it IS good enough or exactly what they needed.
The numbers given in the article are actually consistent with what is usually meant by “everyone” in such statements. Sure, it’s not literally everyone. But it’s a very significant percentage, especially given how quick the adoption has been.
Software engineers are definitely in a bit of a bubble here. Are we just early adopters who see the value sooner, or does it uniquely benefit software engineering, or do we just like cool automation and we're deluding ourselves that this adds value beyond the cost?
> AI has gotten so good that despite any misgivings, “everyone is using A.I.”
In my experience, it's a mixed bag. I wrote this comment[0], yesterday. It reflects my current work, and how I am integrating an LLM.
I have used it for two parts of my project:
1) The backend (PHP), and
2) The frontend (Swift)
It has been a huge help, in both, but #2 is a cautionary tale. It really needs adult supervision, in developing native UIKit Swift apps. I'm realizing how truly bad the code it wrote was. I mean, terrible.
That's jarring, because it did a great job with #1. It made sound, reasonable design decisions, and provided code that is better than what I would write.
With #2, it behaved exactly like an inexperienced engineer, panicking, when confronted with real-world problems. My rewrite is going to feature a much simpler, sound approach.
All that said, it has been a net positive, and has increased my productivity by a large margin.
I guess the lesson I needed to get from this, is that it is good at helping me to find problems, but maybe not so good at fixing them.
There's probably a lot more PHP code to regurgitate, while I'm guessing a lot of Swift apps are proprietary. That's the problem.. they aren't coding from principles, they're pattern matching their training data
I only use AI for software development. For writing, I don't use it at all except to translate source materials. So yes, AI is only for software development in my case.
The real question is whether I have any value outside of software development. Sometimes I get the feeling that AI is replacing the value I have in society.
I honestly just use it as a search engine to get around SEO garbage and ads.
My wife uses it for a (non-computer related) business though and it's great for all sorts of normally tedious marketing/social media type jobs though. Stuff that doesn't really require accuracy just needs text on pictures that looks good quickly.
I think everyone just has FOMO and doesn't want to lose to competitors. Eventually it'll die down.
One thing I'd personally like to see a little more discussion of (at least within my social circles) is.. what exactly does "using AI" mean?
How does this connect to everyone's high level ideas/thoughts about "tech", "AI" and "morals and feels" etc. These lines can start to seem a little blurry, at least for me.
For example, would we say my partner is "using AI" (for all intents and purposes), if she's frequently using Google.com throughout the day, and then ends up picking and believing the AI generated answer overview at the top of the SERPs almost every time?
Or do we feel "uses AI", is more along the lines of the vampire kids running 1000 sub-agents on a mattress floor in SF?
I kind of find the whole spectrum really interesting because even basic phone use is now stuffed with AI, whether we choose to label it or not.
Everyone is using AI, issue is not just everyone recognizes what AI actually is, how broadly it's used.
Looking things up and asking questions was always something for a minority of the population so the language model usage being relatively low isn't a surprise.
Problem arises if the non-AI segment is leveraged to create regulations that impact the AI using segment negatively.
For example; ChatGPT is replacing my Google searching. Not necessarily because it's better, or because it's summaries are better than Google (I find them subjectively better but it's not clear cut).
But because the app has a nice history; can ask a relatively complicated question and go do something else and then come back to it, ask a follow up. Etc.
None of that is specifically an AI benefit, but it's a workflow that really helps, well, flow.
On the post-grad job hunt right now - I note that most employers will ask in a technical interview or whiteboard interview "how are you using LLMs?"
It's tough to answer because you want to hedge for both an AI enthused employer and an AI hesitant employer with limited information about who they are and how they personally use these products. I've been responding with a sort of long winded answer about how 'there is clearly a learning curve for how this technology fits into any process and how I always always always double double double check yadayadayada'
I'm probably using the chat/ask functionality on a daily basis for quick debugging / new technology learning questions but I have yet to really use the fully agent or computer-use products because I've had more bad results than good the few times I've tried them (re-factoring a big repo of decades old fortran+C code for modern compiler/OS some things started to work but ultimately I abandoned that effort).
> It's tough to answer because you want to hedge for both an AI enthused employer and an AI hesitant employer
That this doesn't have a clear and obvious answer one can expect shows how the issue is politics, not strategy.
When you apply as a mechanic, there is no such weird political debates about certain power tools where people have passionate opinions on which tool to use.
I am also on the job market, but as a Senior. Pro-tip: ask them this question before they ask you. “One quick question I have about the company culture, …”
The reason you've had more bad results than good is because you haven't fully learned how to use LLMs yet. They are not as simple as they first appear. I think a lot of people think using a coding agent is just a case of firing it up and telling it what to do and expecting to get it right first time. When it doesn't they just think it's no good and like you abandon the effort.
The reason a technical interviewer will be asking this question is because they want to see how you adapt to using new technologies, LLMs being one of the most disruptive technology that has hit the tech industry since at least the internet. You will likely be expected to use LLMs and they will want to know that you are someone who truly understands the capabilities of them - upsides and downsides, where to use them, what guardrails you need to put in place.
I'd encourage you to revisit the re-factoring task you worked on. Work out why it didn't work, work out what didn't work about it and if you have the chance try again, but use different techniques, there's a lot of conversations going on about what people find working and not working - try to join that conversation. Try to document what you learn. Then in the interview discuss these rather than just saying you gave up. The interviewer isn't going to check up on how successful your project was, they just want to know how you think and how you approach problems.
You should just be honest. If you're not a good fit for the company then you should honestly be eager to discover this.
> I've been responding with a sort of long winded answer
"I don't. I personally don't find value in them for the type of work I do. I am also uncomfortable with using their outputs under the current copyright regime. I also question how competitive any organization can possibly be if LLMs become the main driver of their work products."
> I've had more bad results than good the few times I've tried them
"I prefer to write correct code rather than debug bad code generated from a limited context window."
I think what would be great is to have eg a concise example where it works well for you and a concise example where it doesn’t. This shows you have explored it and thought about it enough to explain interesting observations. It’s good to then be ready to go deeper if of interest.
I'd take a candidate who fights for a view I disagree with over one who hedges to please everyone (Strong opinions weakly held, but willing to change their mind if/when the data tells a different story)
I've personally started to dislike agentic coding, not because of the LLM side of things, but rather, because the non-LLM workflow itself is stupid.
The UI/UX is built around one-shotting the code as a single fully baked unit. There should be intense competition around having the best code review tools to adjust the fine details and yet everything is devolving into a text prompt, even CAD tools where being off by a millimeter or less can make a part completely useless even if it is 99% of the way there.
All of these tools are assuming that the LLM already achieved superhuman AGI capable of reading your mind while making zero mistakes.
I think you will find that both AI-pilled as well as AI-sceptics will actually agree on that approach, because it rejects the question of "how much" and rather reflects it towards the "when to use AI".
Depending on who it is, you can then talk more about the beginning, end or center piece of it.
It's hard right now.
I've been interviewing some folks for a while.
I am really interested how someone can make a 100x of themselves when working.
This often means "how are you using claude code", "what is your coding setup", "how do you decide if your agent is going in the right direction or not w/o looking directly at the massive amount of code generated".
This is not technical, but it measures how much we would have to teach someone that comes from a non-industry background.
Remember that companies have money, universities dont pay for a 200 USD Codex+Claude+Grok+Cursor per employee.
That said, I also had to ask the same stuff to folks from big techs that dont actually are software engineers. How are data scientist leveraging LLMs to automate experiments? Are they using it for something other than data viz? Are they using it for /loop exploration? how trustworthy are the agents right now for this?
Having someone with this grasp nowadays might be very very valuable IMO
everyone might not be using ai.
but i see myself reaching for it for every small thing these days.
it's like every curiousity or lifestyle choice or optimization is something ai can help research.
i am not saying it's really powerful or great.
but the lure is undeniable. because of how low friction it has become.
So true, just built a deterministic system to identify duplicated code. It's offline and doesn't use AI on purpose, since a gate that blocks your CI has to give the exact same answer every time, and finding dupes means comparing every function against every other (that's index work). It does NOT use AI. But ironically, I used AI to build it (https://github.com/Rafaelpta/dupehound )
I'm using AI for most things. It has been an incredible improvement to both my quality of life and my wallet. Some of the most high profile items from just the past three months:
- I'm getting my roof replaced due to hail damage. Insurance originally covered only $5k due to depreciation. I fed the insurance policy to AI. I learned about the appraisal clause and invoked it. At the end, I got another $6,500 back.
- I was having issues with plumbing. Four different plumbers came, they all said the cast iron pipes under the house need to change. Quotes ranged from $35k to $55k. I had AI walk me through the process. It taught me about the yard line vs. under-slab distinction, and suggested getting just the yard line replaced first because it's much cheaper and can fix the issue. I did that and spent $6k. The issue was fixed. I "saved" $30k for now by deferring that massive month-long project. (For brevity, I'm omitting a ton of boring technical stuff I learned about plumbing that helped me make the optimal decision - none of the contractors bothered explaining any of it.)
- My 2010 Hyundai Santa Fe is starting to show its age. I've taken it to multiple different repair shops, then fed their diagnoses and recommendations to AI and figured out which ones are trying to fleece me and which ones are being more careful and conservative with their repair recommendations. Probably saved several thousand dollars there. Learned a lot about cars too!
- My partner and I are converting the backyard to a wildlife sanctuary. The AI helped us plan what to plant where (depending on lots of factors like sunlight location, irrigation access, etc.) and it has been going really well. Also planned out a dragonfly pond to deal with mosquitoes. AI created a project plan, including schematics, material purchase list and step-by-step instructions.
- I've been wanting to do various other home improvement projects, but only ones that make financial sense. I took photos of my house, both inside and outside, and fed them to AI, and said "give me a list of projects I can do that will have high ROI for when I decide to sell this house". It spent 15 mins doing deep research, then came back with a long, prioritized list. If I do all the projects, I'd be spending about $40k and it would improve the house valuation by about $90k.
I can go on. There's probably dozens of stuff that I've used it for over the past year that led to massive time and money savings, and I've learned a ton as well about topics I normally would not have been exposed to or bothered to research myself. And I'm not even including all the work-related usage, both for my employer and my side business. That would be its own very long list.
> People are consuming AI like they eat meat: some are embracing it, some are limiting their use of it, and some are avoiding it altogether.
That's an interesting analogy as, despite the real ecological issues with it and principled arguments against meat eating, in general meat consumption has trended upward globally in country after country for decades.
They are great on exploring, understanding and finding bugs in existing codebase.
They are great for simple or one time scripts/programs.
They are terrible, really terrible coders. The overengineering is so deep in their training that no matter what is your prompt, your skills or agents.md/claude.md, if you don't babysit them continuously, at some point they will just fuck up your codebase.
98 comments
[ 4.1 ms ] story [ 123 ms ] threadIf I wanted to read an AI response to a query, I would use that tool specifically - I consider them different tools entirely.
It's definitely taken some time to get into the habit - perhaps defaulting to Google as the search engine was my first mistake.
However, most people I've seen run a quick search DO in fact just read the AI response (often with a caveat of "this is just from AI but..." which I appreciate), because for a lot of things it IS good enough or exactly what they needed.
Software engineers are definitely in a bit of a bubble here. Are we just early adopters who see the value sooner, or does it uniquely benefit software engineering, or do we just like cool automation and we're deluding ourselves that this adds value beyond the cost?
That aside, this piece is interesting and ties together some useful numbers and studies.
I hadn't seen the recent Microsoft paper showing:
> 30 percent of the US working-age population is using AI [...] with at least 90 minutes of usage time in a given month.
I'm honestly impressed at how high that number is! That's a lot of adoption for a technology (LLM chatbots) that didn't exist four years ago.
Actually anything that is about 90% great and 10% disastrously wrong is utter crap given the way people want and do use AI models.
They are great tools in the right hands and awful in the wrong.
In my experience, it's a mixed bag. I wrote this comment[0], yesterday. It reflects my current work, and how I am integrating an LLM.
I have used it for two parts of my project:
1) The backend (PHP), and
2) The frontend (Swift)
It has been a huge help, in both, but #2 is a cautionary tale. It really needs adult supervision, in developing native UIKit Swift apps. I'm realizing how truly bad the code it wrote was. I mean, terrible.
That's jarring, because it did a great job with #1. It made sound, reasonable design decisions, and provided code that is better than what I would write.
With #2, it behaved exactly like an inexperienced engineer, panicking, when confronted with real-world problems. My rewrite is going to feature a much simpler, sound approach.
All that said, it has been a net positive, and has increased my productivity by a large margin.
I guess the lesson I needed to get from this, is that it is good at helping me to find problems, but maybe not so good at fixing them.
[0] https://news.ycombinator.com/item?id=48515217
My wife uses it for a (non-computer related) business though and it's great for all sorts of normally tedious marketing/social media type jobs though. Stuff that doesn't really require accuracy just needs text on pictures that looks good quickly.
I think everyone just has FOMO and doesn't want to lose to competitors. Eventually it'll die down.
How does this connect to everyone's high level ideas/thoughts about "tech", "AI" and "morals and feels" etc. These lines can start to seem a little blurry, at least for me.
For example, would we say my partner is "using AI" (for all intents and purposes), if she's frequently using Google.com throughout the day, and then ends up picking and believing the AI generated answer overview at the top of the SERPs almost every time?
Or do we feel "uses AI", is more along the lines of the vampire kids running 1000 sub-agents on a mattress floor in SF?
I kind of find the whole spectrum really interesting because even basic phone use is now stuffed with AI, whether we choose to label it or not.
EG: That the entire conversation is juggled by the token selling shills, per usual.
Looking things up and asking questions was always something for a minority of the population so the language model usage being relatively low isn't a surprise.
Problem arises if the non-AI segment is leveraged to create regulations that impact the AI using segment negatively.
For example; ChatGPT is replacing my Google searching. Not necessarily because it's better, or because it's summaries are better than Google (I find them subjectively better but it's not clear cut).
But because the app has a nice history; can ask a relatively complicated question and go do something else and then come back to it, ask a follow up. Etc.
None of that is specifically an AI benefit, but it's a workflow that really helps, well, flow.
It's tough to answer because you want to hedge for both an AI enthused employer and an AI hesitant employer with limited information about who they are and how they personally use these products. I've been responding with a sort of long winded answer about how 'there is clearly a learning curve for how this technology fits into any process and how I always always always double double double check yadayadayada'
I'm probably using the chat/ask functionality on a daily basis for quick debugging / new technology learning questions but I have yet to really use the fully agent or computer-use products because I've had more bad results than good the few times I've tried them (re-factoring a big repo of decades old fortran+C code for modern compiler/OS some things started to work but ultimately I abandoned that effort).
That this doesn't have a clear and obvious answer one can expect shows how the issue is politics, not strategy.
When you apply as a mechanic, there is no such weird political debates about certain power tools where people have passionate opinions on which tool to use.
The reason a technical interviewer will be asking this question is because they want to see how you adapt to using new technologies, LLMs being one of the most disruptive technology that has hit the tech industry since at least the internet. You will likely be expected to use LLMs and they will want to know that you are someone who truly understands the capabilities of them - upsides and downsides, where to use them, what guardrails you need to put in place.
I'd encourage you to revisit the re-factoring task you worked on. Work out why it didn't work, work out what didn't work about it and if you have the chance try again, but use different techniques, there's a lot of conversations going on about what people find working and not working - try to join that conversation. Try to document what you learn. Then in the interview discuss these rather than just saying you gave up. The interviewer isn't going to check up on how successful your project was, they just want to know how you think and how you approach problems.
You should just be honest. If you're not a good fit for the company then you should honestly be eager to discover this.
> I've been responding with a sort of long winded answer
"I don't. I personally don't find value in them for the type of work I do. I am also uncomfortable with using their outputs under the current copyright regime. I also question how competitive any organization can possibly be if LLMs become the main driver of their work products."
> I've had more bad results than good the few times I've tried them
"I prefer to write correct code rather than debug bad code generated from a limited context window."
personally i find this offensive and would disqualify the candidate.
The UI/UX is built around one-shotting the code as a single fully baked unit. There should be intense competition around having the best code review tools to adjust the fine details and yet everything is devolving into a text prompt, even CAD tools where being off by a millimeter or less can make a part completely useless even if it is 99% of the way there.
All of these tools are assuming that the LLM already achieved superhuman AGI capable of reading your mind while making zero mistakes.
I think you will find that both AI-pilled as well as AI-sceptics will actually agree on that approach, because it rejects the question of "how much" and rather reflects it towards the "when to use AI". Depending on who it is, you can then talk more about the beginning, end or center piece of it.
I am really interested how someone can make a 100x of themselves when working. This often means "how are you using claude code", "what is your coding setup", "how do you decide if your agent is going in the right direction or not w/o looking directly at the massive amount of code generated". This is not technical, but it measures how much we would have to teach someone that comes from a non-industry background. Remember that companies have money, universities dont pay for a 200 USD Codex+Claude+Grok+Cursor per employee.
That said, I also had to ask the same stuff to folks from big techs that dont actually are software engineers. How are data scientist leveraging LLMs to automate experiments? Are they using it for something other than data viz? Are they using it for /loop exploration? how trustworthy are the agents right now for this?
Having someone with this grasp nowadays might be very very valuable IMO
i am not saying it's really powerful or great. but the lure is undeniable. because of how low friction it has become.
"No, everyone is not using the internet for everything."
Which would have been entirely true when written, and entirely false a relatively short time later.
Everyone does use the internet for everything today, and everyone will use AI for everything soon.
- I'm getting my roof replaced due to hail damage. Insurance originally covered only $5k due to depreciation. I fed the insurance policy to AI. I learned about the appraisal clause and invoked it. At the end, I got another $6,500 back.
- I was having issues with plumbing. Four different plumbers came, they all said the cast iron pipes under the house need to change. Quotes ranged from $35k to $55k. I had AI walk me through the process. It taught me about the yard line vs. under-slab distinction, and suggested getting just the yard line replaced first because it's much cheaper and can fix the issue. I did that and spent $6k. The issue was fixed. I "saved" $30k for now by deferring that massive month-long project. (For brevity, I'm omitting a ton of boring technical stuff I learned about plumbing that helped me make the optimal decision - none of the contractors bothered explaining any of it.)
- My 2010 Hyundai Santa Fe is starting to show its age. I've taken it to multiple different repair shops, then fed their diagnoses and recommendations to AI and figured out which ones are trying to fleece me and which ones are being more careful and conservative with their repair recommendations. Probably saved several thousand dollars there. Learned a lot about cars too!
- My partner and I are converting the backyard to a wildlife sanctuary. The AI helped us plan what to plant where (depending on lots of factors like sunlight location, irrigation access, etc.) and it has been going really well. Also planned out a dragonfly pond to deal with mosquitoes. AI created a project plan, including schematics, material purchase list and step-by-step instructions.
- I've been wanting to do various other home improvement projects, but only ones that make financial sense. I took photos of my house, both inside and outside, and fed them to AI, and said "give me a list of projects I can do that will have high ROI for when I decide to sell this house". It spent 15 mins doing deep research, then came back with a long, prioritized list. If I do all the projects, I'd be spending about $40k and it would improve the house valuation by about $90k.
I can go on. There's probably dozens of stuff that I've used it for over the past year that led to massive time and money savings, and I've learned a ton as well about topics I normally would not have been exposed to or bothered to research myself. And I'm not even including all the work-related usage, both for my employer and my side business. That would be its own very long list.
That's an interesting analogy as, despite the real ecological issues with it and principled arguments against meat eating, in general meat consumption has trended upward globally in country after country for decades.
They are great on exploring, understanding and finding bugs in existing codebase.
They are great for simple or one time scripts/programs.
They are terrible, really terrible coders. The overengineering is so deep in their training that no matter what is your prompt, your skills or agents.md/claude.md, if you don't babysit them continuously, at some point they will just fuck up your codebase.