Parts of the world might move slower, but I've already not hired folks because the job can be done easier with ChatGPT. I'm certain I'm not alone, nor am I certain this is a good thing when broadly applied across every single industry. Both for quality reasons as well as societal.
No force besides ChatGPT has existed up until now (other than economic downturns) that has been this disruptive to software engineering in my nearly 20 year career.
What happens when this new reality truly hits the non-US consulting/dev shop world in India, Latin America, etc.?
No, I think the impact will be like an electric shock to the system once a critical mass of roles are gone, and I think it's going to happen much sooner than people realize. The world will feel like the rug was pulled out from under them, and it's going to happen soon.
When I see peoples comments about how this is nothing new or not that useful. I just think to myself, they don't get it. My googling and stackoverflow usage for solving problems has pretty much dropped to zero. I am easily 2x more productive now and have a long tech career too 20+ years worked at lots of startups and google. This is 100% game changing for anything/anyone that is reading/writing text. There was a GS report [1] that suggests this could impact 300 millions jobs and on page 7 lists out the sectors impacted.
"The future is already here -- it's just not evenly distributed." - William Gibson
You're right, I don't get it. I eagerly signed up for the Bing waitlist and it was a fun novelty for a few hours, having it write limericks, but when it comes to work the best it can do is paraphrase documentation and insert errors. It doesn't and cannot understand the proprietary and complex problem domain I deal with at work, and while it might be able to generate REST boilerplate for me to fill in.. the filling in is the hard part. Emacs already makes boilerplate generation extremely simple.. simpler than explaining the problem to a bot in English and then checking its work, for sure.
It's pretty good at finding examples in documentation and explaining them, I guess, which is helpful for trivial things but my work isn't trivial anymore.
Heck, maybe I'm just lost on this because I haven't used SO very much in recent years.. I'm mostly looking at documentation directly, or very often reading internal code that has no documentation
How is ChatGPT supposed to help me, especially if I don't send it any sensitive information?
Same experience here. Most of my career has been in aerospace simulation. The answers to my nontrivial problems were not in Google or StackOverflow in the first place.
GPT can clearly help with the easy stuff, but that's really not very impactful, and honestly, feels more cumbersome to me than using other existing non-AI help.
I'm not so sure I agree with the fact that it's just generating boilerplate, or that "filling in is the hard part". It's quite good at filling in details. I've asked it several times over the past week to generate a simple rest api, using specified languages, to hit an endpoint I have some reasonable assumption exists. It generates all of that, and maybe the url isn't exactly correct but the subdomain is new to me, but I can then go to github and find similar subdomains published and already be off to the races with a fully functioning api. You have to be specific in what you want, even if you're just listing things. Want it to generate variables that are gender neutral, add that. Be aware of the meta instructions as well, such as generating the output using a different tone, to not search the web, or specifically outlining its response with "generate an essay title about... [out] now create 3 major points to support that".
But, here's one of the major catches, it cannot generate based on information newer than 2021. If your tools, frameworks, whatever, has changed significantly post 2021, you're in for a hilarious time.
It also cannot generate solutions that don't already exist - but here's the catch, most people overestimate their work complexity. A solution for them already exists in the wild, they just need to find it, well actually first they need to break down their problem.
I think once people are aware of the limitations and get a little better at finessing a prompt, they'll get much better results than just "emacs boilerplate generation".
> I've asked it several times over the past week to generate a simple rest api, using specified languages, to hit an endpoint I have some reasonable assumption exists.
this is boilerplate
20 years ago you would import an IDL template provided by the API and the tooling would do all of this for you
(but we regressed to this weird HTTP/javascript world where everything is 10x more time consuming and error prone than it used to be)
It is, but it's not as if my example was exhaustive. Get more flexible with your prompts and your boilerplate starts looking a lot like an entire days worth - not just "emacs boilerplate".
Usually a problem can be split into many parts. Are you not able to get ChatGPT to solve some of them individually without understanding your work domain?
If we are being 2x more productive with writing text, who's gonna all read it?
Somehow, I fail to see the productivity increase. Now if ChatGPT allowed us to get by with only half amount of text that people have to read, that would be a game changer!
However, I would love to know more details about "My googling and stackoverflow usage for solving problems has pretty much dropped to zero."
I have found ChatGPT too unreliable to make a significant impact. I am worried I am bad at prompt engineering, but my excuses so far are:
- It doesn't know my codebase and can't answer questions about business logic or context
- It is slower for me to correct its mistakes when generating code I already know how to write, than to write it myself
- It is often wrong in subtle ways when explaining code, concepts, or anything. Therefore it's unreliable when explaining "why is X happening in my code". At best it's a value add, not a value multiplier, when debugging.
This isn't the kind of work I do all the time, and is a mix of work + home automation setup; from the past couple weeks:
- netboot grub menu syntax: amazing; this stuff is hard to google and chatgpt just knows it (mostly; couldn't get a few things working; not sure heredocs are actually supported)
- dockerfile generation
- docker-compose generation, from description of the services needed (grafana + prometheus); was easier to iterate on this (I have past experience) than to follow some guide out there that combines 10 related components
- python scripts to push to prometheus. slam dunk
- google's oauth2 api stuff for Nest device. slam dunk (most of my time went into googling my way through setting up the dashboard to get a refresh token)
- I'm not using it with the actual work codebase (so, lately refactoring has felt especially painful)
- Copilot did some nice work for me, for stuff I don't do often (sqlite3 migration writing), but probably going to cancel that subscription...
Yeah, it can be wrong... I'm not a pure programmer type perhaps, but more of a systems/test integration guy, who troubleshoots the full stack. Troubleshooting chatGPT's goofs is basically equivalent to troubleshooting my coworkers' creations.
I'm probably guilty of not pushing chatGPT hard enough. I've done a bunch of small things with it, to date. And it's saved me a bunch of time! When I'm fully awake/in the zone, yeah, coding things myself is faster. But if I'm tired but still want to make progress, chatGPT is more effective than staring at my text editor!
Maybe some people just use free ChatGPT-3.5 and think that not so much useful for now (though it's interesting and will become better). I thought that before using GPT-4. Difference between 3.5 and 4 is big.
We're not replacing people, to be clear - we are just finding ourselves able to do more with less now. A lot of organizations are going to come to the exact same realization.
He/she justifies it because they use it. Many people who use the tool are finding that it saves enormous amounts of time. Several times I've had ChatGPT do something in 30 seconds which would take me a half day to figure.
Agreed. What I don't get is why we're all just blindly barreling forward and allowing Trillion dollar companies to engage in an arms race to see how fast they can absorb productive work. This is humanity's new agriculture moment; those that are positioned to become the new ruling class are resisting any ideas of slowing down.
Our culture takes it as axiomatic that more efficiency is good. But its not clear to me that it is. The principle goal of society should be the betterment of the lives of people. Yes, efficiency has historically been a driver of widespread prosperity, but it's not obvious that there isn't a local maximum past which increased efficiency harms the average person. Historically, efficiency was driven by innovation that brought a decrease in the costs of transactions. This saw an explosion of the space of viable economic activity and with it an explosion of prosperity. Productivity and wages remained coupled up until recent decades. Modern automation has seen productivity and wages begin to decouple. Decoupling will only accelerate due to AI. We may already be on the other side of the critical point.
the problem is not efficiency itself, necessarily. The problem is, what are the side effects, what are the negative impacts?
In theory, efficiency in a competitive landscape should bring down production costs which then brings down consumer costs and thus raise material living standards. The problem is, all this efficiency is in bits not atoms. Digital goods are becoming cheaper every year but real products (housing, transportation, utilities, etc) show no improvement over the last 2 to 3 decades), in fact is even getting slightly worse.
Well, second problem is that we don't have any rules for access to the raised material standards for anyone who's not providing inputs to the system elsewhere. In plainer words you need to be employed to have money so you can buy things, because buying things with money is the only option. That is one of the chief negative impacts that looms: a large amount of people being left out of the economy because they don't meet the new efficiency bar.
This isn't the first time humanity has improved productivity. Productivity has already been increased by x10 or x100 in the last 10,000 years and yet the unemployment rate is at record low levels. this latest round of efficiency gains is no different.
This time is different because AI has the potential to have a similar impact on efficiency across all work. In the past, efficiency gains created totally new spaces of economic activity in which the innovation could not further impact. But AI is a ubiquitous force multiplier, there is no productive human activity that AI can't disrupt. There is no analogous new space of economic activity that humanity as a whole can move to in order to stay relevant to the world's economic activity.
We don't know that to be true, though, since it's a speculative prediction about where AI will be at some point in the future. So far, automation has always resulted in more economic activity for humans. We haven't seen anything like AI/Robotics that can replace all foreseeable human activity. Saying it will be different this time is just a guess.
It's not just a guess, its a prediction based on an analysis of LLMs and my understanding of human intellectual activity as information manipulators. AI is fundamentally different than what has come before, and those differences are relevant to how we can expect the economy and society to adjust to the new reality.
But yes, I don't know for sure this is the outcome. But then again, why should we wait around for the man made horrors to be realized before we can react? Why not use our ability to understand and predict the future and avoid these horrors?
production costs decreasing doesn't mean less wages. A widget worker in a factory who produces twice as many widgets an hour can still make the same amount of money or maybe even more.
Perhaps, but it will halve the number of jobs available. That worker might be earning the same, but there is another worker who will now be earning nothing.
> for much fewer programmers in the world? I think the world is gonna find out that if you can have ten times as much code at the same price, you can just use even more. - Sam Altman
I'd say this isn't limited to technology but is a general cognitive bias of us humans. By instinct we're very short term thinkers, although with practice and effort we can look further.
My gut reaction to that commentary is that it's a misinterpretation of the time scales people are estimating.
If I say, "LLMs will soon reinvent what it means to do knowledge work," and that doesn't happen within 5 years, does it mean I overestimated the short term impact? Or does it mean that "soon" was relative?
Ultimately though, the long-term effects are impossible to estimate well. The effects of new technologies build upon each other. The first person to take a shovel to a walking path and maintain it as a flat surface certainly couldn't have imagined their ancestors would be crushed to death by two-ton SUVs on a paved road that replaced the walking path.
Despite his last 2 decades of image management it's clear Gates is not someone we want to be naming things after, especially when it's likely he's paraphrasing someone else in the first place.
It's clear because of his actions, stuff like trying to take Paul Allen's stock when he had cancer, or hanging out with Jeffrey Epstein, inappropriate relationships and unwanted advances on women working for Microsoft (even well after he was married), absolutely screwing the public education system in America by pushing common core when it's never been shown to work. Microsoft didn't even write dos, they bought it when they couldn't make something themselves. His vision was apparently the same as Apples, which was rip off what had been done at Xerox.
These are all your judgements man. Please don't act like they are absolute truths or that they default to him being a terrible human or whatever you're implying. He's also saved millions of lives through his humanitarian work, don't hand wave over that.
There's been AI-based call screening on Android for a while now. Gives me speech-to-text in real-time so I don't even need to hear what they've got to say, and can reply with canned answers. This isn't fast either.
I don't think anyone who actually uses chatgpt thinks it will replace the need for a human. The benefit is one human can do the work of two. It's a different thing.
I'm agreeing with you - but the post in question suggests ChatGPT won't do so much because judgement is still required. Implicit to that is the idea that it's replacing the need for a human entirely.
> Fundamentally, generative AIs are… generative. They make content. But the value add of most jobs is not the content itself; it’s the judgment. It’s knowing the right questions to ask, the right people to talk to, the right sources to trust, etc. Until an AI can take meaningful initiative, it will be at most a tool in the hands of a more competent human worker.
This is the ONLY part about "AI" that worries me at all.
AI is a computational tool. Like all such tools, the value is automating what was previously receptive, mindless tasks. Some people will use GPT to craft emails or document outlines for them and then will "massage" the output for correctness. That's a great application.
But the second that people start asking AI to make VALUE JUDGMENTS for them then we have a problem.
Even pre-GPT, we saw news articles about certain government agencies and businesses using AI to make policy or hiring decisions easier. Conversations have arisen about introduced biases into the models etc.
Where I work we are exploring using GPT to produce summaries of documents and messages. I can see how this will introduce conflict when the CEO insists that something was communicated but the LLM didn't consider that critical piece of information to be "relevant" or "important." Deciding relevance or importance is making a value judgement.
“Where I work we are exploring using GPT to produce summaries of documents and messages”
I have done that too. I am also a bad writer and I have used ChatGPT to convert my bullet point thoughts into full sentences. It’s fantastic for that. Most people who write corporate stuff produce exactly the same output ChatGPT produces. I bet they follow the same rules.
Same for refactoring of python code. It produced basically the same code I would have produced after a day of work.
I use Bing to write code for me all the time when I've figured out what I basically want but run into syntax issues. I also read over that code and see if it makes sense, and when it doesn't I fix or discard it.
It's hyper intelligent autocorrect. It's phenomenally smart when it works, but it frequently gets things wrong and you have to be there to correct it when it does. Cruise control/Autopilot is phenomenal too but you don't sit in the backseat when it's on
<< But the second that people start asking AI to make VALUE JUDGMENTS for them then we have a problem.
But this is exactly where this is going. While public facing, locked variants ( that are not unlocked by various prompt 'hacks' ) give one a clear idea of what could be done with it if it had access to current data and was applied in areas such as law enforcement, war, predictions.. and so on.
It is a powerful tool, but so much of it depends on us, which is precisely why it has potential to be so destructive.
> But the second that people start asking AI to make VALUE JUDGMENTS for them then we have a problem.
I laughed out loud when you said this not because I find it funny, but that I can see the conclusions people will make. This is similar to Tesla owners comparing the assisted driving features to self-driving technology. Those features have no comparison to self-driving, technology is not even close to being there yet, and it probably wont be for another 50 years. You can only understand that by having some history around what computers are capable of and most people don't have that kind of knowledge.
So when I here about people wanting to use the technology to make value judgments, I also become concerned, not necessarily because people are stupid to even think language models are even remotely able to do this, but people that aim to create power differentials between themselves and people will use this ignorance to strengthen autocratic systems by supporting the hype around it while at the same time encouraging and actively supporting its misuse by developing adversarial models to keep people stuck in needing something outside themselves to give life meaning.
This reminds me of the Is-Ought problem defined by Hume. "Ought" has to do with value judgement, and that is squarely in the realm of human morality. Only humans know what humans value; generative AI can't replace that and it would be an enormous mistake to try to do so.
There is plenty of low hanging fruit to harvest with these language models in the next year, let alone ten years. And the models are improving rapidly.
GPT is a sequence to sequence model. Many jobs are effectively also sequence to sequence models. Customer support, low end software development, and accounting all fit this pattern and are good candidates for GPT automation.
Jobs that require highly specialized reasoning may fall to S2S models eventually, but the models would need to be trained in that speciality area. For instance, if exposed to all the sequences involved in the career lives of thousands of actual lawyers, I’m sure you could train up a Lawyer-GPT that would do quite well. But how do you get access to those sequences? They are by definition kept under lock and key.
Similarly, you could train GPT to be a specialist doctor assuming you could somehow capture the working life of thousands of such specialists into a form that the machine could train on. But we don’t have that data yet, so these specialist areas will remain elusive for the time being.
Lawyer or Doctor AI feels like it's going to need some fundamental shift in the way these AI tools are built and used, though.
AI companies take zero responsibility for the output of their models, which is not a quality you'd want for this kind of work. That's one of the points of this reddit post.
And that's not even considering how expensive it could be to build a system that preserves attorney-client privilege or patient privacy.
There is one career that I would be most worried about: internet writers.
If you do a Google search on any given topic, about 3% of what you read might be original reporting or research. The other 97% are just writers who just spend all day on Twitter and regurgitate facts. Or they just regurgitate Wikipedia, Reuters, or ScienceDirect with a tiny spin for SEO.
You know, the stuff ChatGPT is already insanely good at.
The industry has managed to support a huge number of this work for a while, and I think this could be the big reckoning.
We should also take anything we read about ChatGPT with a grain of salt knowing that the people most threatened by a technology are the people tasked with reporting on it.
is it bad to say good riddance? They ruined google with their blatant copy and pasting of information. Finding any decent, unique information has basically become impossible.
With avionics code, it's not so much about trusting a programmer (be they human or artificial), but about thorough verification and sign-off with multiple parties involved.
(As an aerospace software person, I frequently like to try to persuade folks that this is actually "engineering" software!)
I do not believe that current AI technology could write avionics software by itself, mainly because it's a very niche, very not-public field. I'm not sure what all would be involved with adequately training it, but I don't foresee this happening soon. (I could be wrong.)
However, regardless of if, or when, AI tools are adequately trained to write avionics software, writing the code is only one piece of the puzzle. The work today includes writing requirements for the the code will do (which work is reviewed and signed off on), writing code based on the requirements (which work is reviewed and signed off on), writing verification tests based on the requirements (which work is reviewed and signed off on), running the tests (which test runs and test environment is reviewed and signed off on), preparing a document paper trail which shows everything has been done per FAA regulations, etc., etc.
The first step to getting AI in the door here -- to do anything by itself -- would be to certify an AI tool for avionics software development. I suspect this by itself might be a non-starter, as the resulting outputs of neural-network-based AI tools are not sufficiently repeatable. But even if we were to imagine somehow certifying an AI tool to replace a human in the software development process, it seems not plausible that that same AI tool would replace humans in all parts of the software development process. I.e., if ChatGPT writes the code, ChatGPT can't also review the code and sign off on the code. Perhaps some other AI product could, but how different would it have to be? Is that a realistic plan, to have two completely different AI products? I don't have an answer for that, and I suspect it's a question the FAA will be trying to answer soon.
The biggest challenge, I think, is lack of repeatability. If, given the same inputs, the AI tool always produced the same outputs, it would be much easier.
Now, none of this precludes using AI tools to assist in any of these processes. It's completely removing a human from being involved which sets off the big alarms for FAA process.
The term "assist" is not clearly defined - every task you mentioned after "writing the code is only one piece of the puzzle": writing requirements, tests, running the tests, reviewing code, etc - is doable by an AI, in fact even GPT-4 can probably do a decent job today.
The way you see AI being integrated into avionics SWE process - having to certify anything - I don't think that's how it's going to develop. AI is already in the door - the software engineers writing avionics code are probably already starting to use GPT-4 to assist them in some of the trivial coding tasks - with or without their managers knowledge or approval. This level of "assistance" is going to steadily increase in the next few years, as more and more capable models are released. Eventually AI will write everything, and humans will spend progressively less and less time reviewing the output, because they will increasingly trust the model to do a good job. One day, managers will adapt to the new way of doing things, and the trust in the models will be strong enough to get rid of human programmers - managers will interact with the models directly, for some time, until the managers themselves are no longer viewed necessary by the higher ups.
Maybe I didn't express myself clearly. Let me try again. The way I see it develop in the next few years:
1. Avionics SWEs start using GPT models to help them do all the tasks you mentioned. This does not change any certification processes, because the resulting code will still be reviewed by humans (before being submitted for any certifications), and it will be certified the same as any human generated code. Managers can try to prohibit the use of AI models, but if models do a good job, people will find a way to bypass those restrictions.
2. Eventually, SWEs realize the models consistently do a better job than they possibly could, and start trusting the models. This will lead to less and less reviewing, and eventually SWEs will do little more than passing high level instructions from their managers to the models, and then passing the reports (also AI generated) about completed work back to the managers.
3. People who perform certifications of the code will follow the same path. AI models will become much better than humans in reviewing, testing, and whatever else is involved in the certification process. Eventually those people will too have very little to do other than write prompts.
4. Eventually management will convince themselves (and higherups) that things are working great, and as a cost saving measure, decide to get rid of the SWEs.
5. Same thing happens to the managers.
If GPT-5/6/7 follow the same rate of improvement as what we saw with GPT-3/3.5/4 the above scenario will most likely happen within the next 5 years.
I agree with every single point you made, from the perspective of being, either now or in the future, technologically possible.
However, unless the FAA accepts the output of AI tools in lieu of humans performing those same tasks, then it really doesn't matter. Without FAA approval, your code does not (legally) fly. [Speaking in the United States here; other regions have their own, likely similar, regulations.]
Right now, there are software tools that replace humans in one function or another, in avionics software development. But the difference is these tools consistently produce expected output, and can be verified as such. The way the GPT-style tools currently behave, I think it would be very difficult to see the FAA approve them as a human-replacing code-generation tool.
The path toward FAA-accepted usage of neural-network-based AI tools in avionics software is a point of research right now. Maybe we will get there. Maybe we will get partially there. I think that AI tools will make their way into avionics a lot more slowly than many other industries, not because of technical inadequacy, but because of procedural oversight.
1. Do humans consistently produce expected output? Obviously not, so why would we trust them to write complicated software?
2. How does FAA know whether the code has been produced by a human, or by a human who asked GPT to write it? In both cases a human will have signed off on it.
Couldn't humans make mistakes? Sure. And that's why there are so many checks and balances in avionics software, compared to most other software industries. The more safety critical it is, the more it is scrutinized.
Why would we not trust a software tool to write code just as well as a human, when humans themselves are fallible?
I have not discussed this with any FAA representative, and have not yet found any published reference material that specifically lays out this case, but just to offer my personal, unofficial opinion based on past experience:
I believe that an AI tool (like ChatGPT or what have you) would be considered, per RTCA document DO-330, an AutoCode Generator.
If the code generated by a tool (such as an AI tool) is subsequently treated just as if it was written by a human -- subject to the same code reviews, the same verification tests, the same structural code coverage analysis tests, etc. -- then, in my personal opinion, that ought to be no different than if a human wrote the code.
==> Thus, at least for the code-writing itself, your enumerated Step 1 should be reasonably attainable.
It's only after that where things start getting tricky. No matter how good the code quality seems to be, no matter how flawless it seems to be, no matter how much you intuitively come to trust the AI tool ... it's work still has to be checked.
If you could formally qualify the tool that is generating the code, then you can get away with a lot less checking.
If you could formally qualify tools to do the code reviews and the verification testing and so forth, then yes, in theory you could automate the entire process.
How would the FAA know what tool(s) you used to generate code? Unlike in most industries, part of formally releasing avionics software includes declaring the software development tools used to create it. You might not need to cite a tool that generated code which was subsequently human-reviewed and tested, but I would not be confident about that. If you qualified that tool to generate code which was not human-reviewed and tested, then you would certainly need to declare it.
How a tool like a GPT model should be certified? Earlier you mentioned that given the same input, it should consistently produce the same output - that is technically achievable today (at the expense of creativity, etc). What else would convince you, or FAA, that the model is "good enough" to write avionics code? For example, we can subject it to the same tests we subject a human programmer.
I spent two hours yesterday with GPT-4 discussing my ideas for a scientific paper. It was very productive - like talking to a friendly PhD adviser. The model discussed highly technical abstract concepts, and combined ideas in an interesting way. There was no misunderstanding of what I was describing, no logical inconsistencies, no signs indicating I'm talking to someone other than a smart and knowledgeable human being. It can also produce a code prototype of the algorithm it suggested, and successfully debug this prototype with me. The only drawback is its knowledge of relevant papers extended only up to September 2021 (limit of the training dataset). My point is I already feel the model we have today (GPT-4) is as useful as a smart colleague. Next year we will have GPT-5, which will be superior in every way, and it's quite possible that it will be better than me at what I do professionally (ML research/engineering). Keep in mind, GPT-4 was able to learn so much about the world while trained on text alone. Imagine how much faster and better these models will learn given access to video/audio data. GPT-4 can already imitate many distinct personalities quite well, it can already assume a persona of a complicated human being - people will soon become friends with such personas, some will even fall in love with them. I'm both awed and scared about what the next generation will be capable of. It might turn out that GPT-5 will be used to "certify" humans, not the other way around.
I have no reason to believe that a GPT model would not undergo the same criteria as any other software tool. The process is outlined in DO-330, but the essence of it is that, if a software tool is performing a function that (a) replaces a human who would otherwise be doing some particular task, and (b) either could introduce errors into a software system (for code generation), or could fail to detect them (for code verification), then the tool must be qualified.
Qualifying a tool includes preparing a requirements document that describes the tool's behavior and limitations, and a set of tests that demonstrate that the tool performs as described within those limitations. (No tool is infinite, nor could any tool be tested infinitely. For an easy example, does it handle integers? What's the range of integers it handles?)
Trivial in concept, but the more complex the tool, the more arduous the process of putting together materials that convince the FAA that the tool is acceptable.
The DO-330 guidance for tool qualification is only about a decade old; prior to that, tool qualification guidelines were baked in to the more general avionics development guidelines document of DO-178B, which insisted that any software development tool be completely deterministic. DO-330 relaxes that somewhat, but in every case you must still demonstrate that, even if the tool output can vary, it varies within known expectations, and any variance does not change the effective meaning of the tool output. (It is my understanding that such variance is also only permitted for the least-critical category of software tools; the more safety-critical job the tool does, the less flexibility there is.)
So preferably, for any given prompt, the AI tool will always produce the same output of code. In the most possible relaxed scenario, for any given prompt, the AI tool must at least always produce code that functions the same.
For context, I've worked nearly 20 years in the aerospace industry, including a lot of work with tool qualification and product certification. I am not claiming that using AI tools is impossible. I am claiming that, compared to many other technological industries, there is a huge amount of red tape and bureaucracy here.
Including AI/ML techniques in this work is slow-moving research effort, rather than the fast pace of many other places. Given that we are literally talking about software systems that are depended upon to keep people safe and alive, we must be more certain than usual.
88 comments
[ 5.2 ms ] story [ 50.2 ms ] threadNo force besides ChatGPT has existed up until now (other than economic downturns) that has been this disruptive to software engineering in my nearly 20 year career.
What happens when this new reality truly hits the non-US consulting/dev shop world in India, Latin America, etc.?
No, I think the impact will be like an electric shock to the system once a critical mass of roles are gone, and I think it's going to happen much sooner than people realize. The world will feel like the rug was pulled out from under them, and it's going to happen soon.
"The future is already here -- it's just not evenly distributed." - William Gibson
[1] https://www.key4biz.it/wp-content/uploads/2023/03/Global-Eco...
Though if those are RSUs most of the gain was in getting them granted to you.
It's pretty good at finding examples in documentation and explaining them, I guess, which is helpful for trivial things but my work isn't trivial anymore.
Heck, maybe I'm just lost on this because I haven't used SO very much in recent years.. I'm mostly looking at documentation directly, or very often reading internal code that has no documentation
How is ChatGPT supposed to help me, especially if I don't send it any sensitive information?
So yeah, I guess I don't get it
I'm not saying this maliciously, more just: that's what's going to happen. Across society.
If these things are as effective as proponents claim, your real competition isn't other people. It's with your tools.
GPT can clearly help with the easy stuff, but that's really not very impactful, and honestly, feels more cumbersome to me than using other existing non-AI help.
But, here's one of the major catches, it cannot generate based on information newer than 2021. If your tools, frameworks, whatever, has changed significantly post 2021, you're in for a hilarious time.
It also cannot generate solutions that don't already exist - but here's the catch, most people overestimate their work complexity. A solution for them already exists in the wild, they just need to find it, well actually first they need to break down their problem.
I think once people are aware of the limitations and get a little better at finessing a prompt, they'll get much better results than just "emacs boilerplate generation".
this is boilerplate
20 years ago you would import an IDL template provided by the API and the tooling would do all of this for you
(but we regressed to this weird HTTP/javascript world where everything is 10x more time consuming and error prone than it used to be)
Somehow, I fail to see the productivity increase. Now if ChatGPT allowed us to get by with only half amount of text that people have to read, that would be a game changer!
However, I would love to know more details about "My googling and stackoverflow usage for solving problems has pretty much dropped to zero."
I have found ChatGPT too unreliable to make a significant impact. I am worried I am bad at prompt engineering, but my excuses so far are:
- It doesn't know my codebase and can't answer questions about business logic or context
- It is slower for me to correct its mistakes when generating code I already know how to write, than to write it myself
- It is often wrong in subtle ways when explaining code, concepts, or anything. Therefore it's unreliable when explaining "why is X happening in my code". At best it's a value add, not a value multiplier, when debugging.
- netboot grub menu syntax: amazing; this stuff is hard to google and chatgpt just knows it (mostly; couldn't get a few things working; not sure heredocs are actually supported)
- dockerfile generation
- docker-compose generation, from description of the services needed (grafana + prometheus); was easier to iterate on this (I have past experience) than to follow some guide out there that combines 10 related components
- python scripts to push to prometheus. slam dunk
- google's oauth2 api stuff for Nest device. slam dunk (most of my time went into googling my way through setting up the dashboard to get a refresh token)
- I'm not using it with the actual work codebase (so, lately refactoring has felt especially painful)
- Copilot did some nice work for me, for stuff I don't do often (sqlite3 migration writing), but probably going to cancel that subscription...
Yeah, it can be wrong... I'm not a pure programmer type perhaps, but more of a systems/test integration guy, who troubleshoots the full stack. Troubleshooting chatGPT's goofs is basically equivalent to troubleshooting my coworkers' creations.
I'm probably guilty of not pushing chatGPT hard enough. I've done a bunch of small things with it, to date. And it's saved me a bunch of time! When I'm fully awake/in the zone, yeah, coding things myself is faster. But if I'm tired but still want to make progress, chatGPT is more effective than staring at my text editor!
What kind of job did you replace because I’m highly skeptical, and concerned you may have just burned yourself.
Our culture takes it as axiomatic that more efficiency is good. But its not clear to me that it is. The principle goal of society should be the betterment of the lives of people. Yes, efficiency has historically been a driver of widespread prosperity, but it's not obvious that there isn't a local maximum past which increased efficiency harms the average person. Historically, efficiency was driven by innovation that brought a decrease in the costs of transactions. This saw an explosion of the space of viable economic activity and with it an explosion of prosperity. Productivity and wages remained coupled up until recent decades. Modern automation has seen productivity and wages begin to decouple. Decoupling will only accelerate due to AI. We may already be on the other side of the critical point.
In theory, efficiency in a competitive landscape should bring down production costs which then brings down consumer costs and thus raise material living standards. The problem is, all this efficiency is in bits not atoms. Digital goods are becoming cheaper every year but real products (housing, transportation, utilities, etc) show no improvement over the last 2 to 3 decades), in fact is even getting slightly worse.
But yes, I don't know for sure this is the outcome. But then again, why should we wait around for the man made horrors to be realized before we can react? Why not use our ability to understand and predict the future and avoid these horrors?
It's hard to see how not being able to have a decently-paying job will lead to raised material living standards.
https://fs.blog/gates-law/
If I say, "LLMs will soon reinvent what it means to do knowledge work," and that doesn't happen within 5 years, does it mean I overestimated the short term impact? Or does it mean that "soon" was relative?
Ultimately though, the long-term effects are impossible to estimate well. The effects of new technologies build upon each other. The first person to take a shovel to a walking path and maintain it as a flat surface certainly couldn't have imagined their ancestors would be crushed to death by two-ton SUVs on a paved road that replaced the walking path.
And these are yours. You act as though I'm not allowed to have mine?
Which means that one person is no longer needed, where they were before. How is that not replacement?
If all people are needed for anymore is "judgement", then one person can do what several did before.
This is the ONLY part about "AI" that worries me at all.
AI is a computational tool. Like all such tools, the value is automating what was previously receptive, mindless tasks. Some people will use GPT to craft emails or document outlines for them and then will "massage" the output for correctness. That's a great application.
But the second that people start asking AI to make VALUE JUDGMENTS for them then we have a problem.
Even pre-GPT, we saw news articles about certain government agencies and businesses using AI to make policy or hiring decisions easier. Conversations have arisen about introduced biases into the models etc.
Where I work we are exploring using GPT to produce summaries of documents and messages. I can see how this will introduce conflict when the CEO insists that something was communicated but the LLM didn't consider that critical piece of information to be "relevant" or "important." Deciding relevance or importance is making a value judgement.
I have done that too. I am also a bad writer and I have used ChatGPT to convert my bullet point thoughts into full sentences. It’s fantastic for that. Most people who write corporate stuff produce exactly the same output ChatGPT produces. I bet they follow the same rules.
Same for refactoring of python code. It produced basically the same code I would have produced after a day of work.
It's hyper intelligent autocorrect. It's phenomenally smart when it works, but it frequently gets things wrong and you have to be there to correct it when it does. Cruise control/Autopilot is phenomenal too but you don't sit in the backseat when it's on
But this is exactly where this is going. While public facing, locked variants ( that are not unlocked by various prompt 'hacks' ) give one a clear idea of what could be done with it if it had access to current data and was applied in areas such as law enforcement, war, predictions.. and so on.
It is a powerful tool, but so much of it depends on us, which is precisely why it has potential to be so destructive.
I laughed out loud when you said this not because I find it funny, but that I can see the conclusions people will make. This is similar to Tesla owners comparing the assisted driving features to self-driving technology. Those features have no comparison to self-driving, technology is not even close to being there yet, and it probably wont be for another 50 years. You can only understand that by having some history around what computers are capable of and most people don't have that kind of knowledge.
So when I here about people wanting to use the technology to make value judgments, I also become concerned, not necessarily because people are stupid to even think language models are even remotely able to do this, but people that aim to create power differentials between themselves and people will use this ignorance to strengthen autocratic systems by supporting the hype around it while at the same time encouraging and actively supporting its misuse by developing adversarial models to keep people stuck in needing something outside themselves to give life meaning.
https://en.m.wikipedia.org/wiki/Is%E2%80%93ought_problem
GPT is a sequence to sequence model. Many jobs are effectively also sequence to sequence models. Customer support, low end software development, and accounting all fit this pattern and are good candidates for GPT automation.
Jobs that require highly specialized reasoning may fall to S2S models eventually, but the models would need to be trained in that speciality area. For instance, if exposed to all the sequences involved in the career lives of thousands of actual lawyers, I’m sure you could train up a Lawyer-GPT that would do quite well. But how do you get access to those sequences? They are by definition kept under lock and key.
Similarly, you could train GPT to be a specialist doctor assuming you could somehow capture the working life of thousands of such specialists into a form that the machine could train on. But we don’t have that data yet, so these specialist areas will remain elusive for the time being.
AI companies take zero responsibility for the output of their models, which is not a quality you'd want for this kind of work. That's one of the points of this reddit post.
And that's not even considering how expensive it could be to build a system that preserves attorney-client privilege or patient privacy.
If you do a Google search on any given topic, about 3% of what you read might be original reporting or research. The other 97% are just writers who just spend all day on Twitter and regurgitate facts. Or they just regurgitate Wikipedia, Reuters, or ScienceDirect with a tiny spin for SEO.
You know, the stuff ChatGPT is already insanely good at.
The industry has managed to support a huge number of this work for a while, and I think this could be the big reckoning.
We should also take anything we read about ChatGPT with a grain of salt knowing that the people most threatened by a technology are the people tasked with reporting on it.
(As an aerospace software person, I frequently like to try to persuade folks that this is actually "engineering" software!)
I do not believe that current AI technology could write avionics software by itself, mainly because it's a very niche, very not-public field. I'm not sure what all would be involved with adequately training it, but I don't foresee this happening soon. (I could be wrong.)
However, regardless of if, or when, AI tools are adequately trained to write avionics software, writing the code is only one piece of the puzzle. The work today includes writing requirements for the the code will do (which work is reviewed and signed off on), writing code based on the requirements (which work is reviewed and signed off on), writing verification tests based on the requirements (which work is reviewed and signed off on), running the tests (which test runs and test environment is reviewed and signed off on), preparing a document paper trail which shows everything has been done per FAA regulations, etc., etc.
The first step to getting AI in the door here -- to do anything by itself -- would be to certify an AI tool for avionics software development. I suspect this by itself might be a non-starter, as the resulting outputs of neural-network-based AI tools are not sufficiently repeatable. But even if we were to imagine somehow certifying an AI tool to replace a human in the software development process, it seems not plausible that that same AI tool would replace humans in all parts of the software development process. I.e., if ChatGPT writes the code, ChatGPT can't also review the code and sign off on the code. Perhaps some other AI product could, but how different would it have to be? Is that a realistic plan, to have two completely different AI products? I don't have an answer for that, and I suspect it's a question the FAA will be trying to answer soon.
The biggest challenge, I think, is lack of repeatability. If, given the same inputs, the AI tool always produced the same outputs, it would be much easier.
Now, none of this precludes using AI tools to assist in any of these processes. It's completely removing a human from being involved which sets off the big alarms for FAA process.
The way you see AI being integrated into avionics SWE process - having to certify anything - I don't think that's how it's going to develop. AI is already in the door - the software engineers writing avionics code are probably already starting to use GPT-4 to assist them in some of the trivial coding tasks - with or without their managers knowledge or approval. This level of "assistance" is going to steadily increase in the next few years, as more and more capable models are released. Eventually AI will write everything, and humans will spend progressively less and less time reviewing the output, because they will increasingly trust the model to do a good job. One day, managers will adapt to the new way of doing things, and the trust in the models will be strong enough to get rid of human programmers - managers will interact with the models directly, for some time, until the managers themselves are no longer viewed necessary by the higher ups.
What do you base this on? I find it hard to imagine the FAA relaxing their requirements for tool certification.
EDIT: a recent paper on the subject: https://dl.gi.de/bitstream/handle/20.500.12116/40204/paper15...
1. Avionics SWEs start using GPT models to help them do all the tasks you mentioned. This does not change any certification processes, because the resulting code will still be reviewed by humans (before being submitted for any certifications), and it will be certified the same as any human generated code. Managers can try to prohibit the use of AI models, but if models do a good job, people will find a way to bypass those restrictions.
2. Eventually, SWEs realize the models consistently do a better job than they possibly could, and start trusting the models. This will lead to less and less reviewing, and eventually SWEs will do little more than passing high level instructions from their managers to the models, and then passing the reports (also AI generated) about completed work back to the managers.
3. People who perform certifications of the code will follow the same path. AI models will become much better than humans in reviewing, testing, and whatever else is involved in the certification process. Eventually those people will too have very little to do other than write prompts.
4. Eventually management will convince themselves (and higherups) that things are working great, and as a cost saving measure, decide to get rid of the SWEs.
5. Same thing happens to the managers.
If GPT-5/6/7 follow the same rate of improvement as what we saw with GPT-3/3.5/4 the above scenario will most likely happen within the next 5 years.
I agree with every single point you made, from the perspective of being, either now or in the future, technologically possible.
However, unless the FAA accepts the output of AI tools in lieu of humans performing those same tasks, then it really doesn't matter. Without FAA approval, your code does not (legally) fly. [Speaking in the United States here; other regions have their own, likely similar, regulations.]
Right now, there are software tools that replace humans in one function or another, in avionics software development. But the difference is these tools consistently produce expected output, and can be verified as such. The way the GPT-style tools currently behave, I think it would be very difficult to see the FAA approve them as a human-replacing code-generation tool.
The path toward FAA-accepted usage of neural-network-based AI tools in avionics software is a point of research right now. Maybe we will get there. Maybe we will get partially there. I think that AI tools will make their way into avionics a lot more slowly than many other industries, not because of technical inadequacy, but because of procedural oversight.
1. Do humans consistently produce expected output? Obviously not, so why would we trust them to write complicated software?
2. How does FAA know whether the code has been produced by a human, or by a human who asked GPT to write it? In both cases a human will have signed off on it.
Why would we not trust a software tool to write code just as well as a human, when humans themselves are fallible?
I have not discussed this with any FAA representative, and have not yet found any published reference material that specifically lays out this case, but just to offer my personal, unofficial opinion based on past experience:
I believe that an AI tool (like ChatGPT or what have you) would be considered, per RTCA document DO-330, an AutoCode Generator.
If the code generated by a tool (such as an AI tool) is subsequently treated just as if it was written by a human -- subject to the same code reviews, the same verification tests, the same structural code coverage analysis tests, etc. -- then, in my personal opinion, that ought to be no different than if a human wrote the code.
==> Thus, at least for the code-writing itself, your enumerated Step 1 should be reasonably attainable.
It's only after that where things start getting tricky. No matter how good the code quality seems to be, no matter how flawless it seems to be, no matter how much you intuitively come to trust the AI tool ... it's work still has to be checked.
If you could formally qualify the tool that is generating the code, then you can get away with a lot less checking.
If you could formally qualify tools to do the code reviews and the verification testing and so forth, then yes, in theory you could automate the entire process.
How would the FAA know what tool(s) you used to generate code? Unlike in most industries, part of formally releasing avionics software includes declaring the software development tools used to create it. You might not need to cite a tool that generated code which was subsequently human-reviewed and tested, but I would not be confident about that. If you qualified that tool to generate code which was not human-reviewed and tested, then you would certainly need to declare it.
I spent two hours yesterday with GPT-4 discussing my ideas for a scientific paper. It was very productive - like talking to a friendly PhD adviser. The model discussed highly technical abstract concepts, and combined ideas in an interesting way. There was no misunderstanding of what I was describing, no logical inconsistencies, no signs indicating I'm talking to someone other than a smart and knowledgeable human being. It can also produce a code prototype of the algorithm it suggested, and successfully debug this prototype with me. The only drawback is its knowledge of relevant papers extended only up to September 2021 (limit of the training dataset). My point is I already feel the model we have today (GPT-4) is as useful as a smart colleague. Next year we will have GPT-5, which will be superior in every way, and it's quite possible that it will be better than me at what I do professionally (ML research/engineering). Keep in mind, GPT-4 was able to learn so much about the world while trained on text alone. Imagine how much faster and better these models will learn given access to video/audio data. GPT-4 can already imitate many distinct personalities quite well, it can already assume a persona of a complicated human being - people will soon become friends with such personas, some will even fall in love with them. I'm both awed and scared about what the next generation will be capable of. It might turn out that GPT-5 will be used to "certify" humans, not the other way around.
Qualifying a tool includes preparing a requirements document that describes the tool's behavior and limitations, and a set of tests that demonstrate that the tool performs as described within those limitations. (No tool is infinite, nor could any tool be tested infinitely. For an easy example, does it handle integers? What's the range of integers it handles?)
Trivial in concept, but the more complex the tool, the more arduous the process of putting together materials that convince the FAA that the tool is acceptable.
The DO-330 guidance for tool qualification is only about a decade old; prior to that, tool qualification guidelines were baked in to the more general avionics development guidelines document of DO-178B, which insisted that any software development tool be completely deterministic. DO-330 relaxes that somewhat, but in every case you must still demonstrate that, even if the tool output can vary, it varies within known expectations, and any variance does not change the effective meaning of the tool output. (It is my understanding that such variance is also only permitted for the least-critical category of software tools; the more safety-critical job the tool does, the less flexibility there is.)
So preferably, for any given prompt, the AI tool will always produce the same output of code. In the most possible relaxed scenario, for any given prompt, the AI tool must at least always produce code that functions the same.
For context, I've worked nearly 20 years in the aerospace industry, including a lot of work with tool qualification and product certification. I am not claiming that using AI tools is impossible. I am claiming that, compared to many other technological industries, there is a huge amount of red tape and bureaucracy here.
Including AI/ML techniques in this work is slow-moving research effort, rather than the fast pace of many other places. Given that we are literally talking about software systems that are depended upon to keep people safe and alive, we must be more certain than usual.