My theory... Being able to code well or fast doesn't one to one translate to a good end user experience. The strength of your org's ability to determine good features and iterate on them from a product perspective is what matters, and that /can/ potentially happen faster if AI is enabling faster development, but it's not guaranteed.
Even if we had a magic box that results in perfect code coming out every time for a given feature description, that doesn't mean the feature itself is good or well thought out.
My guess would be that the hurdle in creating better products isn't in code completion.
User research, UX improvements, feature ideation and creation, etc, are all the same as they have always been. Getting the code out faster doesn't help if its in service of a bad feature.
Writing the same code 10% faster isn't necessarily going to make it better. Also the biggest improvements have been among novices, and the products you regularly use were predominantly written (or at least reviewed) by more experienced people.
This was my first reaction. It hasn't been that long and giant ships can't change course quickly.
And for much of that year there were a lot of questions around the ownership of AI generated code as well as information security. In fact, most of those questions are still not satisfactorily solved. So I'm not so surprised that we haven't seen massive results "yet"
This is a personal anecdote, and just one data point, but still. I work on websites for customers, and frequently I'll need to (for example) iterate through some spreadsheet data, or convert a big object of this format to some other format. These are tedious tasks that don't take a _huge_ amount of time, but instead of grinding on a particular function for 30 minutes, I have a workable thing I can tweak in five minutes. I'd say this helps me "code faster and better."
Does it make the end product better? Not really: I would have gotten there with a function written by me or some LLM. But like everything I've been asked to do my professional career, it allows me to do more with less. More dumb functions in less time.
I imagine for first pass prototypes, AI will greatly accelerate the process. But getting to the fine details and getting things done well will still take same amount of time.
AI-guided coding will help code up “good enough” implementations, which is great for research and testing ideas but not for production.
Copilot mostly helps people code faster, or with less knowledge required. I'd expect output quality to go down, not up.
And it's like anything in capitalism. Companies could choose higher quality, but instead they do whatever gives the highest profit. which is usually adequate quality at low cost.
Code AI tools today absolutely crush at creating proof of concept apps. You can test your idea and get market validation in days vs months.
They are getting better at medium/large codebases, but still have a ways to go before being super useful and it translating to a huge increase in productivity. Currently it's really good for helping with the menial tasks (creating docs, unit tests, understanding and onboarding) but not quite there yet when it comes to integrating gen ai code in large codebases, but it's only a matter of time.
I run DevRel at Sourcegraph and our AI coding assistant, Cody, is used by tons of individuals, small business, and large enterprises. I get to talk to a ton of customers and see how their adoption of AI is going. And it's certainly increasing and developers are finding a ton of value.
Finding value in a product does make development go from months to days , which is the unsubstantiated claim. Even your customers can talk shit to sometimes in order to curry favour for a discount.
Since gpt came out I have built tons of throwaway apps, plenty of specialized apps for side projects, and experimented with tons of ideas that I likely wouldn’t have if I didn’t have access to a tool to build it for me from just asking it to do it and explain what it did. Claude artifacts has been awesome for this. Cody when I actually want to build it out. I recommend trying it before you knock it.
> Since gpt came out I have built tons of throwaway apps, plenty of specialized apps for side projects, and experimented with tons of ideas that I likely wouldn’t have if I didn’t have access to a tool to build it for me from just asking it to do it and explain what it did.
GitHub Copilot, ChatGPT and Phind are all a bit like this for me - they both lower the barrier of entry and save me a lot of time for trivial algorithms and boilerplate code, in addition to helping me find things better than search engines sometimes do, especially when given a look at the code that I'm working with.
It might not be an order of magnitude difference in my case, but things that wouldn't have happened with the higher barrier of entry are now happening and that's quite the difference in of itself! I'm cautiously optimistic about LLMs and other forms of "AI". If nothing else, so far we basically have a more versatile form of IntelliSense, even if it's not always going to output correct code.
I wonder if some day it'll be feasible to feed in the entirety of a larger codebase and reason about it better than people who only know a part of it could.
> If nothing else, so far we basically have a more versatile form of IntelliSense, even if it's not always going to output correct code.
That's the real issue for me. I remember learning programming and I either had not so good intelligence (Codeblocks, IDLE, Netbeans) or none at all (notepad++,...). This forces me to either follow the book attentively (and hunting down errata) or read the manual and getting explanations from forums or friends. When you're a beginner, uou need a good source of truth, not something that can be subtly wrong.
Since gpt came out I have built tons of throwaway apps,
Throw away apps are the easiest to make. No scaling, no bug fixing, no long term maintenance considerations, no consequences for poor architecture, no need to consider data models, you just write some shit. Bravo.
Many services that power https://videotap.com/ have recently been rewritten with the help of AI.
Recently I used Cody to rebuild the entire video processing pipeline and made it much more efficient and scalable, and I actually learned a ton about ffmpeg by pair programming with the AI. Now I'm building additional features into this app, mostly w/ just iterative prompting or chat-oriented programming to replace 3rd party services that are still in this pipeline and it's been a blast.
I've also used AI tools to really brush up on frameworks, like Laravel, that I haven't touched in a while, and it's been a great experience. Also started building a game w/ Godot and found AI super helpful there in walking me step by step. So for me it's been great.
I believe o1's context window will increase over time as well. It's a new model that takes a different approach compared to other ones, so testing it with less context seems logical and expanding the context window as it's quality is validated.
Even in a large code-base, you don't have to have the full context of every single file for every single question, can usually get away with half a dozen files, it's figuring out which ones to provide to the LLM to get the best response.
Not discounting your claim, but the fact that you work at a company (Sourcegraph) whose business is literally AI coding / coding assistants calls your objectivity somewhat into question.
I will agree that they're very useful for boilerplate tasks particularly around deployment (cloudformation, github actions, etc.)
Def don't take my word at face value. I'm just sharing my experience and knowledge, but the great thing about most of these tools is that many offer usable free tiers or very low-cost plans that don't require any sort of lock in. So try the tools yourself, for your use cases, and see if it's a fit or not.
Personally, I have seen a ton of change in the last 4-5 months, so if you haven't tried these tools recently, I encourage you to try them today and see what's possible.
I signed into Zapier yesterday for the first time in a while. You can seemingly run their entire UI right now via AI. I typed a simple idea into their AI box, and it created a multi-step "zap" for me that was more-or-less what I wanted.
Is there any evidence for any of those hypotheses? More apps in app stores? Mass layoffs? How could the "benefits be too recent to be measured" if everyone is arguing that they are already 10x more productive?
If you accept that premise the only conclusion left is that it has made developer's work much less but their bosses haven't noticed yet.
But you would still expect that some people are working for themselves and continue to work the same amount of hours so should be producing 10 apps a year instead of 1. Are there any examples of that?
If "everyone is arguing that they are already 10x" since GPT4 18mo ago, why aren't they 100x with _o1 and 3.5 Sonnet? Maybe it's bs and they've been spending actual hours getting maybe 2x more productive for the last year. None of it has kicked in yet? It might continue to improve? Who knows.
Actual productivity takes time, and actual products take work. Talk is cheap, show me the code.
I am leaning towards fewer developers. There is a crunch in the job market and companies are finding rate of progress as being similar before, so don't feel inclined to hire more
Most companies I know are turning out less features, they are just more tuned features towards what they believe will provide value and less "moon shots".
To me it looks like "usability" never gets on top of that value list. Yet the seminars and courses and LinkedIn posts season everything with that word.
The highest quality product is finished first, yeah.
Often the difference is between finishing within an order of magnitude of the predicted time versus not finishing at all. But the highest quality product is almost always finished first.
There are exceptions, and AFAIK, those are all for very small projects. In my experience, high-quality takes about a week to be paid back. So if you have something smaller that you'll throw away after a use, you may want to cut corners.
I can think of several ways in which faster development could lead to better quality. Let's say you have a fixed time to come up with a solution to a problem. If you are able to come up with one candidate and evaluate it in that time, you have to live with what you get. If you do two candidates, you can select the one that is better.
Another aspect of quality is "polish". A team that can get a UI in front of QA twice in a development cycle instead of only once will benefit from more fault-finding.
Parkinson's law tells us that work expands so as to fill the time available for its completion.
I think the more likely outcome would either be it still taking the same time to deliver, with extra fluff in the middle, or the time simply shrinks. One thing is still evaluated and shipped, but slight faster.
And does it matter? The broader economy is whats is killing me and everyone one else. Healthcare and education are luxuries i dump tons of money into providing healthcare and education for my kid. Healthcare for myself.
the efficiency gains are resulting in more
products being created rather than existing
products being improved
This has been perhaps the only constant in the history of this industry. As software tooling and hardware get better, we never really feel or see the gains because companies and individual developers are pressed to do more.
If a tool makes my job 2x easier, then I'm simply expected to have 2x more output. Not 2x "better" output.
Because the quality of the product is not solely in engineering's hand. We build the products, and have a lot of impact on the microdecisions, but larger feature planning usually comes from product management, who ultimately should be the ones deciding what features to build to make the customer experience better
I think that at every stage of software development, a new tool comes along to help manage complexity. This is true from punchcards all the way to LLMs.
My experience with AI as a coding parter is yes, great at just doing boring things like take this list and give me an enum or add a form for this class etc .. but when I do anything remotly advanced it breaks apart, especially bad at dotnet where there are 25 years worth of history it source code from .. often it creates somthing that is long out of date, and jesus it tried to rewrite the same code ten times to solve a problem that was not supported by framework and could not be solved. So yea .. give it some years I guess. Still use it daily and I still need to fix Ai generated bugs ..
I'm a designer=engineer, and I feel like AI has given me super powers. Design has always been easy for me--it was just the incredible grind of coding that made creating my own thing difficult.
All of the products you are using today were likely created before ChatGPT was released. For those products, you are not going to see any visible improvements because many of them will suffer from poor implementation due to lack of adequate knowledge of code/frameworks. Most software best practices are learned after the software is created and release. The code is probably very spaghetti and hard to maintain. Refactoring is still hard for AI, but writing from scratch is much easier with AI. For the software that is currently out, bugs will be fixed faster, and features might be added sooner.
The real explosion of great software will happen in 3-5 years. AI is huge for the beginning of projects. You know _what_ you want the app to do but you don't know _how_. That's where AI adds huge value. People are now starting new projects with AI help, and they are building foundations of codebases that will be much more maintainable and sustainable as development continues compared to the current suite of software products we interact with today.
> You know _what_ you want the app to do but you don't know _how_. That's where AI adds huge value. People are now starting new projects with AI help, and they are building foundations of codebases that will be much more maintainable and sustainable
I'm not following. How are these codebases going to be more maintainable and sustainable if the developers are committing code they don't even understand?
I think you misinterpret the 'don't know how' part. LLM's are fantastic for boilerplate (which is a big part of getting things started). From there, I might not know the right incantation for (say) handling a button click, but it's not too hard to validate that an LLM-generated handler a) works, b) looks sensible, and c) fits reasonably in the codebase. In fact, most of my use of LLMs for coding is about generating snippets of functionality, which go into a codebase that I'm maintaining the shape of... which helps maintainability.
You can also generate tests more efficiently, meaning you can get better test coverage cheaper. This leads to better maintainability as well, as you know more quickly when you've broken things with a change.
Yeah precisely. In my case I’ve been building a Django rest app over the last few years. I started off writing way too much of my own code rather than using plugins I had no idea existed. After finally getting ahold of ChatGPT I was able to expand my knowledge of Django tenfold, and was able to rewrite the app from the ground up using proven libraries and design decisions.
Digital distribution and cloud apps lowered the cost to correct mistakes and therefore lowered the barrier to release.
No longer are developers bound by physical media, or have to force clients to troubleshoot, manually download updates on their website and install them.
... and as others said, LLMs impact is greater on junior developers, and at that, more on speed than quality. For experienced developers, the impact is greater on speed.
Maybe things that weren't being built (available capacity to develop) is increasing, thereby lifting up the floor of things that never get built.
Further, non-coders being able to become equivalent to a junior developer is a huge leap.
What active developers do with AI remains to be seen. It really could 20x the average developer, but it doesn't seem like a huge chunk of developers are really using AI in a way that it's the rage on the developer level broadly.
Maybe that's why Cursor going "viral" on youtube seems different when it was known to some, and not others.
Does anyone know if Github has any sort of public telemetry data (of if anyone from GH is around here somewhere)? There was a ChatGPT outage about a month ago and I'm DEEPLY curious if there was an overall drop in commit volume.
Better products require better features and better quality. Features are defined by managers, quality is controlled by testing. These haven't benefited from AI as much as coding did.
172 comments
[ 5.7 ms ] story [ 204 ms ] threadEven if we had a magic box that results in perfect code coming out every time for a given feature description, that doesn't mean the feature itself is good or well thought out.
User research, UX improvements, feature ideation and creation, etc, are all the same as they have always been. Getting the code out faster doesn't help if its in service of a bad feature.
And for much of that year there were a lot of questions around the ownership of AI generated code as well as information security. In fact, most of those questions are still not satisfactorily solved. So I'm not so surprised that we haven't seen massive results "yet"
Does it make the end product better? Not really: I would have gotten there with a function written by me or some LLM. But like everything I've been asked to do my professional career, it allows me to do more with less. More dumb functions in less time.
I imagine for first pass prototypes, AI will greatly accelerate the process. But getting to the fine details and getting things done well will still take same amount of time.
AI-guided coding will help code up “good enough” implementations, which is great for research and testing ideas but not for production.
Copilot mostly helps people code faster, or with less knowledge required. I'd expect output quality to go down, not up.
And it's like anything in capitalism. Companies could choose higher quality, but instead they do whatever gives the highest profit. which is usually adequate quality at low cost.
Code AI tools today absolutely crush at creating proof of concept apps. You can test your idea and get market validation in days vs months.
They are getting better at medium/large codebases, but still have a ways to go before being super useful and it translating to a huge increase in productivity. Currently it's really good for helping with the menial tasks (creating docs, unit tests, understanding and onboarding) but not quite there yet when it comes to integrating gen ai code in large codebases, but it's only a matter of time.
I run DevRel at Sourcegraph and our AI coding assistant, Cody, is used by tons of individuals, small business, and large enterprises. I get to talk to a ton of customers and see how their adoption of AI is going. And it's certainly increasing and developers are finding a ton of value.
GitHub Copilot, ChatGPT and Phind are all a bit like this for me - they both lower the barrier of entry and save me a lot of time for trivial algorithms and boilerplate code, in addition to helping me find things better than search engines sometimes do, especially when given a look at the code that I'm working with.
It might not be an order of magnitude difference in my case, but things that wouldn't have happened with the higher barrier of entry are now happening and that's quite the difference in of itself! I'm cautiously optimistic about LLMs and other forms of "AI". If nothing else, so far we basically have a more versatile form of IntelliSense, even if it's not always going to output correct code.
I wonder if some day it'll be feasible to feed in the entirety of a larger codebase and reason about it better than people who only know a part of it could.
That's the real issue for me. I remember learning programming and I either had not so good intelligence (Codeblocks, IDLE, Netbeans) or none at all (notepad++,...). This forces me to either follow the book attentively (and hunting down errata) or read the manual and getting explanations from forums or friends. When you're a beginner, uou need a good source of truth, not something that can be subtly wrong.
Throw away apps are the easiest to make. No scaling, no bug fixing, no long term maintenance considerations, no consequences for poor architecture, no need to consider data models, you just write some shit. Bravo.
Recently I used Cody to rebuild the entire video processing pipeline and made it much more efficient and scalable, and I actually learned a ton about ffmpeg by pair programming with the AI. Now I'm building additional features into this app, mostly w/ just iterative prompting or chat-oriented programming to replace 3rd party services that are still in this pipeline and it's been a blast.
I've also used AI tools to really brush up on frameworks, like Laravel, that I haven't touched in a while, and it's been a great experience. Also started building a game w/ Godot and found AI super helpful there in walking me step by step. So for me it's been great.
Everything old is new again
If we are strictly speaking about the cutting-edge models like OpenAI's o1, their context is getting smaller, not larger.
Even in a large code-base, you don't have to have the full context of every single file for every single question, can usually get away with half a dozen files, it's figuring out which ones to provide to the LLM to get the best response.
I will agree that they're very useful for boilerplate tasks particularly around deployment (cloudformation, github actions, etc.)
Personally, I have seen a ton of change in the last 4-5 months, so if you haven't tried these tools recently, I encourage you to try them today and see what's possible.
1. The products you use are not developed by people using LLMs.
2. The products you use may be using LLMs in development, but only recently so you'll see a delay before any improvement.
3. The products you use are using it, and maybe it's helping with quality, but not anywhere that users care about or notice.
4. The products you use are using it, and it's not helping with quality, just churning out more code.
So at least some software is getting better.
Assuming that AI is helping developers to write more code, it could mean:
* there are fewer developers
* developers are working less
* the efficiency gains are resulting in more products being created rather than existing products being improved
* AI isn't widely enough adopted or used to make enough of a difference
* the benefits are too recent to be measured
If you accept that premise the only conclusion left is that it has made developer's work much less but their bosses haven't noticed yet.
But you would still expect that some people are working for themselves and continue to work the same amount of hours so should be producing 10 apps a year instead of 1. Are there any examples of that?
Actual productivity takes time, and actual products take work. Talk is cheap, show me the code.
India have had massive layoffs in tech sector.
Europe and US will follow suit once companies layoff dead weight in outsourcing locations first.
I've found that for all but the smallest and tightest of teams, there definitely is a correlation... an inverse correlation.
Often the difference is between finishing within an order of magnitude of the predicted time versus not finishing at all. But the highest quality product is almost always finished first.
There are exceptions, and AFAIK, those are all for very small projects. In my experience, high-quality takes about a week to be paid back. So if you have something smaller that you'll throw away after a use, you may want to cut corners.
Another aspect of quality is "polish". A team that can get a UI in front of QA twice in a development cycle instead of only once will benefit from more fault-finding.
I think the more likely outcome would either be it still taking the same time to deliver, with extra fluff in the middle, or the time simply shrinks. One thing is still evaluated and shipped, but slight faster.
If a tool makes my job 2x easier, then I'm simply expected to have 2x more output. Not 2x "better" output.
Software is easier than ever to write, but now the average AAA video game is 100+ GB and most popular software is browser-based.
This chart illustrates it better: https://x.com/feketegy/status/1809173358279311672/photo/1
The real explosion of great software will happen in 3-5 years. AI is huge for the beginning of projects. You know _what_ you want the app to do but you don't know _how_. That's where AI adds huge value. People are now starting new projects with AI help, and they are building foundations of codebases that will be much more maintainable and sustainable as development continues compared to the current suite of software products we interact with today.
I'm not following. How are these codebases going to be more maintainable and sustainable if the developers are committing code they don't even understand?
You can also generate tests more efficiently, meaning you can get better test coverage cheaper. This leads to better maintainability as well, as you know more quickly when you've broken things with a change.
You can be above-average in your favourite programming language, but suck at the system or library that will be required in your next project.
AI will help you get up to speed.
No longer are developers bound by physical media, or have to force clients to troubleshoot, manually download updates on their website and install them.
... and as others said, LLMs impact is greater on junior developers, and at that, more on speed than quality. For experienced developers, the impact is greater on speed.
I have no data, only sense to make.
Further, non-coders being able to become equivalent to a junior developer is a huge leap.
What active developers do with AI remains to be seen. It really could 20x the average developer, but it doesn't seem like a huge chunk of developers are really using AI in a way that it's the rage on the developer level broadly.
Maybe that's why Cursor going "viral" on youtube seems different when it was known to some, and not others.
Sources:
https://openrouter.ai/models/anthropic/claude-3.5-sonnet:bet... - claude-dev: 5 billion tokens this week
https://openrouter.ai/models/anthropic/claude-3.5-sonnet/app... - aider - 325m tokens this week
Also see aider's leaderboard for models: https://aider.chat/docs/leaderboards/