I knew a few people who thought they could cheat their way through a CS degree. They all dropped out or changed majors within a few semesters.
It’s generally a problem that solves itself, in my experience. Perhaps a benefit of these tools is that we stop the obsession over cheating, which inconveniences honest students in many ways. Cheating has always occurred, but now we can’t even pretend that it’s preventable.
Would you be able to elaborate on why they dropped out? did they hit a point where cheating wasn't possible? or maybe realized it was pointless and unhelpful once they need to do a job?
In my experience it's always the former. Back in university the surest sign of a good class was that you were allowed to "cheat". The professor encouraged using any and all outside materials for your assignments. You were allowed to work with other students on projects. Exams were all open book. You could use open source code. Such classes always led to the best outcomes for students who were truly interested and engaged, and those who weren't had no way of faking it. On the other hand classes where professors spent more effort on detecting cheating than teaching the material were always duds.
Huge +1 to this. I had one lecturer who would essentially rant at us about security vulnerabilities every week for a semester, and then write a paper based on his rants.
Some students ignored his rants because they didn't find the subject interesting, did the past papers, and then got to the exam and did terribly, then complained about the fact that "it's all based on his rants".
Others found the ranting engaging because it was deep dives on obscure bits of computer security history. My year he spent ~4 weeks going on about Stuxnet, including deep dives into the wider political context. When I got to the paper, one of the few questions we could choose, for 50% of the paper, was just "What was Stuxnet". I wrote pages and pages. Figured out from marking that I got full marks on that one. I always did great in that lecturer's modules despite never taking notes and rarely doing any targeted reading.
To show the level of engagement, I once turned up to one of these lectures 10 mins late, with most of the class already there, and the lecturer said "oh I wondered where you guys were, I guess I'll start again". He knew we were the only ones who cared about the class.
Even better was the second year of the class (this lecturer lead the security side of the CS degrees). I took it in autumn 2013, just after the summer of Snowden. The class was just week after week of "I TOLD YOU SO! Look at these leaks! Look at these NSA details! Look at what we happens if we take this bit and stick it into LinkedIn and Google Street View and see where the developer of XKeyscore lives! Look at where this undersea cable goes, isn't that a bit close to Diego Garcia?! Look at this submarine and the bit carved out for an undersea cable splicing facility! I'VE BEEN SAYING THIS ALL FOR YEARS!"
As much as it sounded like the rantings of a tin-foil hat wearing madman, he was in fact quite accurate through a combination of having worked in government in computer security at a fairly high level for a while, and having deep technical knowledge about what was possible.
I miss his lectures, they were excellent, but I hear he's still going strong scaring freshers and running the on-campus teaching union, which is nice to hear.
What I noticed was the lack of motivation to study, and constant difficulty, eventually sapped the motivation to continue slogging through a degree and subsequent career just for the money. "I want to work in this field and make lots of money, but I hate it, and can't get through the coursework" is actually a pretty rare combination of talents.
The path of being interested enough in a field that the easy work is easy, and the hard work is at least interesting, actually turns out to be the path of least resistance.
I think LLMs are wonderful study aids and can bridge gaps in understanding by providing collaborative discovery with a “copilot,” but absolutely, if you cheat off of it, things catch up quickly. First, the sort of stuff ChatGPT can do out of the box is introductory course material. More advanced material requires a deep semantic understanding of the domain. Even if you use ChatGPT to produce boilerplate and syntax, or ask it for semantic advice, it won’t solve the crucial nuggets for you. Second, there is always a “test” component where you are required to remember what you learned impromptu without aids. Third, a lot of cs programs require collaborative work in projects. It’ll become very obvious who doesn’t know anything. Finally, even if you manage to graduate cheating through the entire program, when you interview for work, it’ll be transparent you learned nothing. A degree is an end, it’s a means to an end - be it research of industry, and experts will always be able to sniff out dilettantes.
By the time the kids are done their CS degree (4 years from now), are the LLMs going to be writing all the code anyways?
If I were 18 again, I am not sure what the answer is - would it even be a good idea to pursue this field of study if there aren't going to be many jobs post graduation...
I think humans need to move up the levels of abstraction and master specific domain knowledge. Barring AGI, there will still be need for humans to elicit and learn the requirements from customers and other stakeholders, then translate those into more concrete specifications.
Future LFMs (Large Foundation Models, since they will be multimodal) can help automate some or much work from specifications onward. Humans will need to validate and revise the results by working with an LFM-powered system (Ref: the spiral model in software engineering).
Lots of new projects involve shitting out a massive amount of code though. I've certainly worked on stuff where I had precise clarity on what needed to be done and typing speed was practically the limiter. But for some other things it wasn't.
I dunno. I go back and forth on the effects these things will have on programming.
I think there will be more jobs making software 4 years from now. But whether most CS degree programs will be teaching the most useful skills to do those jobs, I really have no idea.
We are talking AGI. It is funny people think programmers will go first because AI can generate code. It can also generate words. Why do you need Lawyers? Politicians? Doctors? etc. when you have a knowledgable word predictor.
models are still having trouble reasoning – current tricks still depend of feeding back an intermediate answer with a different prompt, but it is still an LLM, and most times the same LLM
Yeah, it's a large language model not a large reasoning model. I don't really have any faith in what seems to me like attempts to make LLMs act like LRMs.
I think it's always going to be a bit shit, no matter how large they make them it's always going to be an obvious faker, always going to confabulate and lie etc.
As long as we are using a technology that merely fakes intelligence (using statistics to generate the most likely next token based on training data is not intelligence), it is going to be obvious that it's fake. This whole bubble is going to burst when people start realizing how overrated it is.
People thought the same thing many times, for many different reason, over the last decades. I'm sure they will continue to think the same thing for many more.
Building an entire application is way beyond what any LLM can do today and probably for years, if ever. Writing a method is basically all they can do, and even that is often unreliable. Maybe some companies will try to replace junior engineers with AI, but will learn really quickly that it doesn't work. The bad thing is that companies will do it anyway, meaning the juniors being replaced will never learn all the skills necessary to build applications or lead teams, so in the future getting anything built will become less and less likely.
I've always said that if AI had to deal with some of the executives and product teams I have, with constant insane changes and shipping demands, the AI will more likely figure out how to eliminate people than deal with them.
Compare that to the whole genAI art thing and people here sneering that being an "Artist" is dead as a career. I wouldn't feel a lot of sympathy if such CSbros found themselves out of a job.
GPT is going to change everything, it’s not even a year in and most code is already written by AI.
It save a lot of time but write a lot of local problem while ignoring the bigger picture. (for now)
The only thing valuable is deep understanding of architecture and of the limitation of hardware.
Know both end of the system, let the AI create all the glue in between.
I would be skeptical that "most" code is written by AI. Some code, certainly - but major enterprises are still terrified of the copyright outcome for existing non-local GPT implementations.
I noticed stable diffusion's CEO made a similarly fantastical claim, @13:33, "41 percent of all code on GitHub right now is AI generated" [0]
Some redditors put together where the claim came from, it's a bad interpretation of Microsoft's Scott Guthrie, should be "40% of code checked in by users of github co-pilot" [1][2]
If most of your code is being written by AI, you need a more interesting/higher value job.
I wish AI could write more of my code but it really is much much faster if I just write it myself. The context and curiosity necessary to even know what questions to ask so you know what code to write is far beyond the scope of any AI I’ve seen.
I agree with your basic premise that AI is, at the moment, mostly only good for writing pretty simple code for pretty simple problems. The few things I've had it do for me, even though simple, still needed a bit of massaging (although I haven't been using GPT 4). But to your final point, that sort of seems like the way it's going to go in the mid-term. The human skill is going to be in figuring out what needs to be done/how to ask AI for it, and the AI will do the nitty gritty of writing it. Right now, for more complex tasks, the human skill is _both_ in recognizing what needs to be done _and_ in doing it. The second part is going to move to AI probably quite a ways before the first part.
> The human skill is going to be in figuring out what needs to be done/how to ask AI for it, and the AI will do the nitty gritty
This is great news if your job is to do the thinking. You will have more time for better thinking. Awful news if your job is hand-on-keyboard typing with others telling you what to do.
Whose code? I simply don't understand this. Are you writing code with GPT? I have simply been unable to get it to solve a technical problem in a production level application AT ALL. The only solutions that work for me are the type of solutions that might already have a stack overflow answer. Being generous, it MIGHT improve my productivity by a percentage point in a good day.
I go to ChatGPT sometimes when I'm really banging my head against a wall and Copilot fails to generate anything useful to even get me moving in the right direction, which is rare. I've hit it up probably let's guess 3-4 times in the last 2 or 3 months. Probably 2 of those times it was anti-useful, because the garbage it put out only wasted my time.
It's good to know I'm not the only one. Whenever I've had an occasion recently where I thought an LLM might be useful (either I didn't know the framework or I didn't quite understand the subject matter) and asked ChatGPT or Bard, they always just hallucinated crap that didn't work and couldn't be made to work. Just yesterday Bard hallucinated a method that was called `something_that_exists_somethingIAskedAbout()`, switching to camel case halfway through, that obviously didn't exist. For the same problem, ChatGPT came up with an existing method, but modified its parameter list to make it do things it didn't do.
I struggle to think of a single instance where it's been helpful for me, and I keep trying, because I don't enjoy doing useless grunt work any more than the next guy, and I want to at least try to learn to ask it the "right" questions.
I find it mostly useful precisely for SO questions - where I know the answer exists, but I don’t want to try to find it via Google.
The other subject I give it high praise is shell one-liners. It can produce some hideously abstruse chains that, by and large, do precisely what I asked. It’s not as good with awk, I’ve noticed, but maybe that’s because awk is much more akin to an actual general purpose programming language than, say, sed.
It's not even close to "most code". Most AI generated code isn't even correct and needs the human to give it some adjustments before it works as intended.
It depends on the bugs. If you have an error message or a wrong behaviour you can explain, it’s pretty good at suggesting solutions and changes. It’s not always working from the first try but it helps.
Really surprisingly good. Have you seen ChatGPT Code Interpreter debug its own code? I've seen it loop 4 or 5 times writing code, running it, getting an error message, tweaking the code and running it again - all without any input from me.
I have deployed LLM-based workflows into several production environments so far this year to great effect and worked extensively with engineering teams. There is no way it's true that "most code is already written by AI".
People I know that use ChatGPT for coding ask the occasional question to simplify boilerplate or similar to stack overflow. Sometimes they'll "see if it can do X" (e.g. build a crud app) and generally it's horribly painful.
I have seen no evidence of "most code [being] written by AI" and believe this to be completely false.
It is not. 90% of the Job a SW dev is doing is impossible to do for ChatGPT. Writing 15 line snippets from a CS101 course is useful to reduce boring repeated tasks, but nothing more.
I happen to be teaching a programming course currently, though it's not in English and the language I'm teaching is C. My current experience is that it does not seem like a majority of the students are using ChatGPT at all, even though I did encourage the use of it at the beginning of the course.
For my own course, I think several factors contributed to students not utilizing ChatGPT as much:
- The assignments are not in English, and performance of ChatGPT in languages other than English is subpar.
- The programming language that I'm teaching is C, I'd imagine Python/Javascript and other more popular languages might lead to different outcomes
- I did specifically design the assignments so that copy/pasting the assignment to ChatGPT does not lead to a usable answer (by restricting use of certain standard library functions, making the assignment more complicated)
- The course is not introductory, i.e. a previous course already taught the basic syntax of C and basics of programming, so I can make my assignments much more advanced
It's difficult to say if advancements in LLMs will make my job harder, where say copy/pasting my more complicated assignments can lead to correct results. But from what I can see right now, LLMs still have trouble solving novel problems, so it's probably always possible to come up with assignments that's difficult for them to solve.
I have been using 3.5, but when I was designing the assignments, I asked a friend who had access to 4 to check it, and the code it produced was still incorrect.
I've had a few online code assessments that appear to have been hardened against ChatGPT "attacks". I failed at solving a problem that was just "Compute values of the Collatz conjecture for input n" because they wrote it to sound like an extremely difficult graph problem about being lost in a forest but being able to enter a "magic door".
I fed the problem into ChatGPT later and it was utterly unable to comprehend it, but confidently gave wrong answer after wrong answer.
Interesting, I'd suppose it makes sense that rephrasing the problem in a different way and also adding a bunch of nouns that have no relations with the problem at hand will definitely confuse LLMs. It will be interesting to see how LLMs will adapt to these as more and more of these techniques develop.
In my own assignments however, I focus less on algorithmic stuff but more on adding and mixing several things together. E.g. instead of just sorting, do group & sort, and a combination of a bunch of other practical stuff like reading big-endian binary files.
I've been using ChatGPT and Bard with C++ and it is quite helpful for boilerplate and reference (replacing Google/Stackoverflow).
Just for fun I asked ChatGPT 4 to calculate the RMSE between two vectors both in English and Portuguese (also translating RMSE to Portuguese) and it gave me the same code for both questions (asked in separate instances). It would be interesting to know what restrictions you applied.
It definitely depends on the task at hand, but when you're teaching programming you don't teach stuff with boilerplate. Using ChatGPT for reference to replace Google/Stackoverflow was definitely one of the ways I'd expected the students to utilize it, but it probably wasn't providing answers in ways a beginner/novice could understand.
I'd expect simple tasks like calculating RMSE to definitely be within the abilities of LLM, you might combine things like actually reading the vectors from a CSV file (or a custom format) and calculating RMSE then sorting them etc to see the limitations of LLMs. Most students have no issues with calculating RMSE, they have issues with trying to do all the other stuff that leads to it, and then the combination of sorting and other tasks.
Regarding the restrictions, most of them are just don't use itoa/strtod or strcpy or some other standard library functions.
Thanks, yes, RMSE is a simple task, I was focusing on its ability to translate the name (raiz quadrada do erro quadrático médio) correctly. It is funny that the Portuguese code has Portuguese comments but the name of the function is calculateRMSE even though I don't mention RMSE in the prompt.
I agree with you, in my experience, ChatGPT is a better search engine but it is not capable of composing the various parts of an application in a cogent manner. I also think that the current UI is not appropriate for software development and I am sure there are efforts going on to create something closer to Jupyter notebooks for programming. That may be a game changer for your students (and you).
True that based on my experience the variable names and function names remains in English despite the prompt, maybe its just the convention overall in the programming world, or maybe ChatGPT is finetuned to do so.
I don't think Jupyter notebooks or like similar REPL interfaces will help too much for my course, at least in the current syllabus. I'm aiming to teach about pointers, memory management etc, the more fundamental parts of how to interact with computers instead of a high level language. Though I would agree that the current UI is suboptimal, some improvements in allowing students to visualize memory layouts and see how their code manipulates memory will help a lot.
A much more interesting consideration would be "Teaching Programming in the Age of Stack Overflow", which involves a well-established, gigantic resource for in-the-trenches programmers that generally provides functional, peer-reviewed and expansively commented solutions to the most common and sometimes most interesting problems.
What programming will be like in an age where LLM's can correctly code is something we won't have any idea about for some time yet (if ever).
Huh? I have personally committed production code that was 100% generated by an LLM. Simply because the code was correct and written more quickly than I could have physically typed it out. Does that not count as correct code to you?
Congratulations on being the exception that proves the rule, or rather, the minority situation that is being used to preemptively make claims about a technology that are not yet (and may never be) true.
I tried it on my/our project 600K LOC (C++), all open source and almost certainly in the training set. Not only could it not explain the questions I asked it correctly, its answers and generated code were absolute jibberish, the sort of thing that would lead to you likely firing the person who gave them to you.
I have no idea what the "age of chatgpt" is supposed to be, it changes nothing in regards to cheating. What my uni professor told us when I did my CS degree was. "You're not in kindergarten, if you want to copy your exercises you can copy them, but you have to do the exam at the end of the semester by hand without assistance and if you don't understand the exercises and do them yourself you're screwed". It's not like profs even bother to invent entirely new exercises every year so you could always just go through old course notes if you really wanted it easy.
Doesn't matter if people Google, Stackoverflow, use ChatGpt, ask their wizkid neighbor or what have you. You don't need to resist cheaters, especially not when we're talking about adults, it's their responsibility. Cheaters don't learn so when they're tested they sooner or later flunk out.
I have the extremely fortunate scenario of watching my wife just begin a computer science degree now, with no prior experience programming whatsoever. This is not hyperbole: ChatGPT is an absolute game-changer when it comes to getting through the basics.
It is simply far, far easier for a newbie to see a computer generate 1 / 3 / 5 slightly-different attempts to solve a specific problem they have, and then to pattern match sufficiently to be able to solve the problem themselves, than it is to muddle around by yourself with it for hours and hours with no end in sight.
I thought she would have to ask me, someone who has been slinging Python in some form since 2008, a question at least once a week. In reality it's been about 3 times over the last 9 months.
ChatGPT doesn't do much for experienced SWEs, but it demolishes the difficulty curve for newbies. I wish I had this when I was learning.
> ChatGPT doesn't do much for experienced SWEs, but it demolishes the difficulty curve for newbies.
Interesting, my impression was that ChatGPT helps experienced SWEs by filling out large amounts of boilerplate-ish code. It sounds like that's not your experience, though?
Which might make you, what, maybe 10% more productive overall? The more experienced you are, the less time you spend writing boilerplate code, so it's not that huge of a win in the grand scheme of things compared to what it provides when you're learning something new.
It makes us far more productive; all of us, so we just do more as a team rather than fire someone. For me, copilot writing out if / then / else stuff like it's reading my mind, so tab -> 15 lines, all correct. Or mappings (.map .filter etc) which it completely does within milliseconds. As an experienced programmer, I cannot avoid these things and they do save me a lot of time and debugging. Oh yes, and tests. Going from a jira task to a bunch of jest tests for code that doesn't exist yet is a pretty big win. Very much more than 10% of the average day. You have to be sharp, be a good code reader and writer and it cannot do anything 'alone', but it makes us vastly more efficient.
> Which might make you, what, maybe 10% more productive overall? The more experienced you are...
But the more exeperienced you are, more valuable your output is (in most cases). So the 10% of your productivity could be more valuable than a novice's 50%.
The more experience you are, the less the value is in the output objective productivity. The ability to write self descriptive, maintainable and bitrot resistant code in one pass goes way above feature delivery.
I think it works well in both ways. I've been developing software professionally for the past ~20 years and there's no doubt that the combination of ChatGPT and CoPilot greatly reduces the amount of time I would usually spend on boilerplate code. Now I can more or less just review the suggestions, review tests and progress with my solutions a lot faster.
At the same time, I see my 9 year old girl using ChatGPT to create Minecraft extensions; she has no prerequisites for actually developing software, but she manages to make some small extensions and have them work either way.
Acknowledging that she doesn't actually learn much from this, I think one can assume that everything in between (e.g. using ChatGPT for pair learning) is definitely also possible.
But whatever AI is outputting, there's no way these qualified guesses can beat experience from a professional developer. I know that the output will never be better than the prompt, but sometimes the output - no matter how much effort you put into the prompt - is visibly just a qualified guess.
While I am not the most experienced, I did recently finish a CS a degree and find chatgpt a great resource for getting a general understanding to go off of when approaching something outside my domain of experience. Having a core understanding of things in general makes it easier to identify when it's wrong about things, and the majority of time I am not using it to generate code, outside of asking for an example. Being able to ask a few direct question on a topic "demolishes" the time spent to become familiar with something, as most of the time I am just looking for answer about some specifics or to clear up some ambiguity in my understanding of a new concept.
I would say with someone new to coding it can be bad and good, as a lot of times it glosses over things, or can be slightly incorrect as it makes assumptions (or more so just answers in a more general context, and when asked to elaborate, or challenged on specifics it will reformat/improve it's answer, but without knowing you need to do so, I could see it easily see it providing half-baked foundational knowledge. You can ask it "x" and it will give a answer, but then if you ask it I am trying to do "y" with "x" and isn't "z" an issue or area of concern with its answer it will reformulate the information provided as its original response was flawed, but if you don't know exactly what the "y" you want to do is, or the "z" being foundational knowledge to challenge it on, you can easily get a whole wall of text that is out of context with what you are actually trying to learn.
> find chatgpt a great resource for getting a general understanding to go off of when approaching something outside my domain of experience.
This is my hang up with LLMs … I don’t trust them.
Frankly it seems just as easy if not easier to just google keywords and read sites.
Google vs LLM is like asking random people on the internet (some are brilliant, some don’t know anything, some are nuts, … etc.) vs asking random people on the internet but all of them have a history of suffering from hallucinations and are routine liars with a compulsive need to answer confidently even when they know nothing.
Especially with an Idea of how they work, I find it very hard to trust them too. Doesn't help that the couple of times I asked them about something subtle, GPT gave me code with the same bugs I was trying to fix. I can see how a beginner would find it useful though.
Plus, with Wikipedia, you can clearly see the sources or the lack of them. I tried asking ChatGPT for a source twice. One time, it gave a source. Another time, it gave the "as a language model I can't" spiel.
> suffering from hallucinations and are routine liars with a compulsive need to answer confidently even when they know nothing.
Sounds like a description of narcissistic personality disorder or schizoaffective. Of course proto ai would have a personality disorder, go figure.
In the future, the job of an ai psychologist will be to certify the personality of ai products. Gotta make sure you’re not shipping a shrink wrapped psychopath.
I tried that a few times, but by the time I've explained enough for the code to be close to what I want i could've just implemented it myself twice over... And that's ignoring the fact that it's always generating subtly wrong or just poorly performing code.
It might change eventually as ide integrations improve, but for now it's a novelty for me.
Yeah, my design problem is the language I'm using, the language version I'm using, the outdated framework I'm using, and the 3! frontend frameworks I'm using on my job.
Both this post and the one two levels back honestly sound like someone who's never worked in a SW company. Craft my own tools? When? Who's paying for that? Too much boilerplate? Tell that to the 20+ people who built this thing and are long gone. You really have to work for a specific kind of company to be able to change any of that.
I have about 15 years experience professionally as a software engineer, in all kinds of companies, at all kinds of different seniority levels.
I call bullshit. Your task is to deliver a project that does what it's supposed to, all in a timely fashion. You do whatever it takes, including getting rid of old code and rewriting stuff if it makes you more productive overall.
Writing code that does what you want is much faster than taking someone else's code and twist it to do something it was not written to do.
You should always be very careful about what dependencies you use, as most often you will have to deal with their limitations and bugs, and they could kill your project.
If you had 20 separate people who contributed to the same thing and left then you most likely have huge code debt. Managing that sort of thing is a daily task for any software engineer. Apparently your strategy is to just add debt onto the pile and put your head in the sand pretending it's not your problem. Instead you should plan tactical refactoring moves to make the codebase leaner and better able to adapt to the business needs, all without breaking any existing functionality and workflows.
You're asking "when". That's the wrong question. It's up to you to manage your time and invest it into tasks that help you save time later on. A software engineer is usually not paid by the hour either, so you just do the work until you're satisfied with the effort you've put in.
A full canoe can take him up to a year, depending on the wood species. He made one once from ebony, which is extremely difficult to work. Also if it wasn’t clear, these aren’t intended for use. They all do (except for the ebony - he said the heat absorption from the dark color would delaminate the fiberglass coat within hours) float and work extremely well if you want, but given the cost, most people use them as art.
He also makes paddles, which are still expensive (I think the cheapest starts at about $1000), but some people actually use those.
That depends what your definition of tool is. He custom-makes jigs all the time, as those are bespoke for whatever particular shape of thing he’s making.
But saws, blades, chisels… maybe he could make them, but why? Metalworking is its own subject of mastery.
AFAIK, he doesn’t often modify any tools of that sort. He spends time selecting the one he wants, yes.
Don’t hesitate to be more specific on your boiler plate problem, because after 19 years career I’m yet to have or see, or even being counted a case where it has any significance on productivity loss. It is at worse questionable yet somewhat healthy warm up.
In my experience, the key is that it can also provide good explanations for the basics as well. You can fairly easily overwhelm it, but with some practise of prompting, learning new language/domain basics it is pretty good, and much better than googling to learn the same. The best of all is that you can query your understanding and get decent feedback.
As an SWE, my problem isn’t boilerplate. It’s architecture and orchestration. I cancelled my Copilot subscription because it just doesn’t help with this. Let alone ChatGPT which has zero context about the application.
I would like an AI to tell me if I’m reinventing the wheel or if something similar can be abstracted. Whether what I’m doing aligns with existing patterns in the codebase. How the application as a whole could be refactored to improve maintainability, performance, or both.
This can all already be done by a human with experience and context, so it must be possible with AI. This will be the biggest game changer for me.
I can’t believe you canceled copilot, at $10 a month it pays for itself every single day for me. I’m just baffled how it can be so useful for me and not useful for you.
I've found it extremely similar to IntelliCode for C#, which is not very impressive. Surprisingly I've found it does a better job of writing accurate comments than generating useful code.
Maybe it's good for some languages, or for code that's doing the exact stuff a hundred people have already written. Copilot X certainly looks cool, but who knows when that'll be ready.
I use it the other way around. I write the comments first and then I find it can often fill in the code.
I find it especially helpful with SQL queries, and with boilerplate type code. It’s also very helpful when I’d need to lookup the usage of various methods. I just start with the comments and most of the time it saves me the context switch to the browser to lookup the usage.
Also as a learner - what i've found incredible is the ability to get AN ANSWER...without context switching. 1 window, 1 result location, no fishing...its trivial to test that given solution...as a beginner...my problems are fairly one dimensional.
Not to mention...things like "cool want to learn python"..."wait - wtf..how do i setup a venv in python!!"
The good thing about ChatGPT is that you can keep asking it to explain something in a different way until it explains it in a way that “clicks” with you.
Of course you then have to compare this explanation against all others and see if it fits or if it’s not a valid explanation.
It’s also great for exploring topics which have polluted namespaces on search engines.
Overall though I think the majour benefit of ChatGPT is it just teaches people to clearly define problems as natural language questions. Developing this ability helps the subconscious mind solve problems when the user is away from screens.
+1 for this, I think this is one of the most useful and underrated things enabled by ChatGPT for effective learning.
One of the types of teaching video that I've found very helpful is where an expert, usually a professor explaining a concept (e.g zero knowledge proof) to five different audiences starting from school kids up to fellow researchers or professors. You can basically ask many different targeted levels of questions to ChatGPT for example, explain the concept as I am five years old, etc.
Yes, but my impression is that chatGPT returns are really linked to the intelligence of the user. How good are to get things out of it. For a clever person starting to code you can easily jumps all the technical hops of the base knowdledge needed to start creating. In every discipline you need base knowdledge that allow you to be able to make the connections.
> Overall though I think the majour benefit of ChatGPT is it just teaches people to clearly define problems as natural language questions. Developing this ability helps the subconscious mind solve problems when the user is away from screens.
Yeah we've had them for a while, it was called the rubber duck, ;)
> It is simply far, far easier for a newbie to see a computer generate 1 / 3 / 5 slightly-different attempts to solve a specific problem they have, and then to pattern match sufficiently to be able to solve the problem themselves, than it is to muddle around by yourself with it for hours and hours with no end in sight.
That's because it is easier. Multiple choice is easier than open questions.
It's also a good way to not really grasp anything deeply.
>
When we teach math to children we don’t start with algebraic structures and proofs. We give them simple recipes.
This is not mathematics, but arithmetic/calculating. When one starts teaching not-children-anymore mathematics (typically at the university), one indeed starts with something of the kind of algebraic structures and proofs.
The GP mentioned that their wife is entirely new to programming but starting a CS degree.
She will certainly learn the deeper concepts in university, but catching up with coding quickly, even in a shallow manner, is super beneficial. I think it's a huge plus.
School has always been exactly the same way. Students who want to learn will learn. Students who do not will not. ChatGPT changes nothing for the latter, it is a tool for the former.
It's like learning from docs examples (guides), tutorials, stack overflow questions etc. But on steroids. We all have learned this way and still do when we quickly want to grasp the surface level. There's nothing wrong with that.
True, but that’s not a good comparison. If we would compare this with arithmetic it would be like handing out a calculator and claiming this is way faster without “muddling around” (also known as “learning things”).
You teach programming by giving simple assignments, not by handing out “calculators”.
> It's also a good way to not really grasp anything deeply.
Having recently done a screenshare with a fresh CS grad as they developed a simple program and seen how much Copilot was doing for them, I was both impressed and appalled. Yes, it's able to do a ton of that simple stuff, but this new grad has no idea _why_ it's doing what it's doing. They are more concerned with getting it done quickly due to not working on the problem early enough (that's a whole other issue).
It genuinely concerns me for the next generation of CS grads and how well they are going to understand the systems they are building. They are already coming out of school unprepared for the professional development world, this is just going to highly exacerbate the problems.
> ChatGPT doesn't do much for experienced SWEs, but it demolishes the difficulty curve for newbies. I wish I had this when I was learning.
Yes and no. It doesn't help me that much with technologies that I'm senior within, but it has allowed me to work with a wider tech-stack. For example, I can now confidently write any SQL querys I want (so far anyway) although my prior knowledge in SQL isn't that deep.
Experienced SWE here. ChatGPT is awesome for me. It's great for learning new technologies and it's great for writing programs that I'm too lazy to write myself.
I mostly use ChatGPT to write code while I'm playing video games. Cutscene/ load screen comes up and I review what it sent and have it iterate. Then back to games.
It's much slower at producing good code than I am but it's much faster for me to write a prompt than a program, so ultimately it allows me to do two things at once because 'prompt time' is so short I can fit it in between other things.
I don't think of programming as a grind unless I'm working in a shitty codebase, which does happen, certainly.
Graying SWE here. ChatGPT is quite helpful, and so is Copilot. I mostly use ChatGPT for well defined functions that I need to implement that don’t rely on too much external context. I love it for writing unittests. It frequently makes mistakes, but it’s easier than starting from zero.
Perusing the comments here, it's interesting how short-sighted HN can be when it comes to LLMs. Generative "AI" is only going to get better, and the question of whether or not ChatGPT can write FizzBuzz, or avionics firmware, or a CRUD app, seems to be missing the forest for the trees.
Even if LLMs or some successor technology never ends up supplanting programmers entirely, it is guaranteed we will see a mixture of deskilling (certain skills no longer being required, like spelling in the age of autocorrect) and massive productivity gains (meaning n-m workers can now do the job of n). These tools aren't going away, and they're only going to improve.
Thus the market for programmers will shrink henceforth. There are no doubts about it. Maybe slowly, maybe quickly, but shrink it will, for the same reason that the market for radiologists is shrinking, or the reason that engineering firms no longer employ whole floors of draftsmen.
Assuming LLM improve productivity why do you think the market will shrink instead of getting even more stuff done? Say if some actually smart LLM could improve my team productivity with 25% then
1 we could fire a person
2 we could handle 25% more tasks, features and bugs
So I think 2 will happen.
My latest example of LLM issues,|
I asked ChatGPT and competition to convert a line of jQuery into native JS, they all got it wrong because jQuery has some selectors like ":header" that is native to JS but the LLM used it anyway though somewhere in it's big memory it has the information that his is not native and if you prompt it right it will fix the issue. So seems to me the LLM are focusing too much on the prompt and failing to use it's full memory on the problem, so it can fix small individual micro tasks but is a complete waste of time for something a bit more advanced ,
my conclusion you can't have a manager or an artist armed with ChatGPT and create a full project without actually learning to code, at best they will learn to code from the LLM but the hard way and probably using outdated code and inefficient ways.
>Assuming LLM improve productivity why do you think the market will shrink instead of getting even more stuff done?
Because this is what always happens when tools cause productivity gains. Budgets aren't infinite.
- Computerized avionics dramatically cut demand for flight crews. It used to require a dedicated "flight engineer" to literally run the engines (like a train engineer) in a WWII-era bomber, and a dedicated navigator. Now airliners barely need a co-pilot.
- As I mentioned previously, computerized medical imaging has caused demand for radiologists to go down, because one radiologist today can do the work of multiple radiologists of yesteryear.
- In software in particular, if engineering and development is more productive, companies can cut costs and still get the same amount of work done, or possibly more. It's not about firing people immediately, it's about the next company hiring fewer. And the one after that hiring even fewer. Generative coding is going to make software development labs leaner.
>my conclusion you can't have a manager or an artist armed with ChatGPT and create a full project without actually learning to code, at best they will learn to code from the LLM but the hard way and probably using outdated code and inefficient ways.
That's the current state of affairs. Why do you think the technology will not improve beyond where it is now?
I don't care about the rest of the post, or about arguing with you. I'm publicly correcting a specific, widely repeated, false claim. One which, if you take the time to read the links I posted, you will see is having a concrete negative effect on an industry responsible for real people's wellbeing. In particular (see the third link), ~50% of students considering radiology as a specialisation express concerns because of false claims its future as a job is under threat. These concerns contribute to hiring problems in that profession, which exacerbate a global staff shortage (first link) which results in worsened healthcare outcomes.
There's a non-zero chance that correcting your statement will influence somebody considering whether to specialise in this field, and replying to your repeating of the claim as well as to the original instance increases that chance. I don't consider it pedantry.
I think the misconception is that there's a finite amount of stuff to do. Every place i've worked at there's been a near infinite backlog. Lots of stuff that would be done if they could. It's always been a constraint on the number of people hired (budget and availability).
I don't see engineers not being part of the process for many years, even if most of the work is done for them. For most companies, if they had their engineers suddenly become 10x more productive, they'd want them to do 10x as much, not scale down to 10% of the staff. Not to say some companies wouldn't.
There are certainly more programmers now than when you had to build a program by wiring together vacuum tubes, or even manually manage memory. Hollywood definitely didn't employ 2.5 million people back in the silent film era when film-making tools were much less productive. That's Jevon's Paradox, increased efficiency increases demand. Labor is like 30% of an airline ticket, and that counts not just the flight crew, so efficiency doesn't do much much for demand when jet fuel is pricey. Likewise there's only so many Xrays that are justified to give out. There's an infinite amount of software crud to be created, and it's primary input is programmers.
>That's the current state of affairs. Why do you think the technology will not improve beyond where it is now?
To replace a developer you would need a human level AI, that is impossible for now.
You can improve the tech to do simple jobs and help me like
- review this code I wrote for potential corner cases
- generate some unit tests for this functions
- find some code in this giant project that does X . I know I wrote it 3 years ago but I have no idea how I named the function or maybe it was a different project.
- rewrite this old IE6 JS/jQuery into modern JAvascript
But IMO a tool that can do this would use LLM but you need an actual
valid code checker/interpretor to do a production ready code. I can't trust an LLM to update a giant piece of code without chainging something, I prefer a tool like Intellj refactoring that actual understand code.
>Because this is what always happens when tools cause productivity gains. Budgets aren't infinite.
I think you'd actually struggle to find an example in history where productivity gains have resulted in prolonged, and industry-wide unemployment for those trained in that area.
I completely agree that technology can make individuals redundant, as in your flight crew example. And yet we are seeing a global shortage of airline pilots, how do you explain that? If technology makes people unemployed (across an economy, over the medium-long term), why aren't there piles of pilots working at McDonald's?
Is not that as technology improves, industries which were once small/unprofitable can balloon into hugely lucrative industries?
On the contrary, it's expanding and there's a global shortage of them [1].
AI has been changing the role of radiologists but not replacing it [2] and radiologists are needed more than ever, but the industry has been struggling to communicate this to students worried by ill-informed claims the profession is under threat from AI [3].
I think it's two sides of the same coin. Demand for medical imaging is not necessarily the same as demand for human radiologists. One radiologist can service 2 or 3 hospitals thanks to the internet, and "AI" is getting better at radiology too. Human-AI Augmentation is happening right now.
When steam drills were becoming the norm I'm sure some mining firms were struggling to hire human diggers too.
The demand for human radiologists never actually went down, though[1]. There is improved efficiency due to technology, but it hasn't kept up with increased demand for scanning and increased patient volumes. There's also induced demand caused by efficiency increasing. Diagnostic AI tends to make different kinds of errors than a human, so even if it were better, both together is even stronger.
The "Radiologists are being replaced by technology" story has been repeated so many times by uninformed software developers that it has become popular wisdom, but the reality so far is that we need more radiologists than ever.
Ironically, radiology might be a decent proxy for what could happen with software engineering, but in the opposite way you intend.
Assuming that HumanEval is a good benchmark (it's not) and assuming you can naively scale under fat-tails (you can't) then according to OpenAI's own gpt4 report (someone did the math on hn/reddit where they reproduced the curve but i can't find the link) at 100x the training cost of gpt4 you will still have a 15% error rate on medium difficulty tasks.
"Programmers" is ill defined. Once it meant translating maths and engineering specs for assemblers. Now it can mean anything from someone writing verified and audited device drivers for robotics to designers a/b testing retail workflows. In the last century, 'programming' became a major economic and technological field of activity.
To think some new 'programming' tool will make it shrink is not at all warranted. Tools that make computer programming more accessible, more flexible and more powerful will increase demand for people who can reason about it, work on it, teach it, evaluate it etc etc.
It may get better but I'm not convinced it will get good. Currently I am not impressed. By which I mean I'm really impressed, it's pretty cool stuff but also hugely overrated.
I'm not even slightly worried about my job security. I tried Copilot and it sucked.
At the end of the day, LLMs are fakers. They do not possess real intelligence, they merely fake intelligence using statistics and training data. There comes a point where you can't fake it any more.
> meaning n-m workers can now do the job of n... Thus the market for programmers will shrink henceforth.
Maybe in the far future when we have programmed nearly everything there is to program... Until then companies will just produce n*(n/(n-m)) more output.
Actually, the reduction of the cost of programming output will logically lead to more demand as solutions that were previously deemed too expensive to produce will become viable.
Not just programming. If you are interested in any kind of topic, be it scientific, or humanistic, GPT is an amazing tool to slash that learning curve.
Just as an example, try using ChatGPT to explain an article in a foreign language. Can even go as low as a letter-by-letter break down of each word. No private tutor will ever have the patience to teach people at this level.
Not using chatGPT to code is like not using antibiotics to treat infection. Sure, it can be done but why go back to the old ways? How many people know how to treat infections without antibiotics? How many people know how the antibiotics they use actually work ?
Coding will become like the liturgical exercises of monks toiling in isolated monasteries while the rest of the world will move on.
I cant wait for a chatGPT decompiler. IDA, watch out!
Disappointing. I cannot evaluate claims about GPT when people say ChatGPT instead of GPT4, if they actually mean the latter.
And if someone really is discussing the usefulness of AI while only having used ChatGPT (GPT3.5), then they’re missing out on a major improvement and their input is less valuable than those discussing GPT 4.
Of course GPT-4 has plenty of limitations but people just need to be clear that they’re familiar with the state of the art.
Also, statements to the effect of “it just predicts the next word” do not appreciate the major difference in capacity for learned abstractions between GPT 3.5 vs GPT 4. So to me it’s just not a useful way of thinking about LLMs. It may technically be true, but at some level that can also be said of human beings. In other news, an airplane “just” flies.
I see ChatGPT as the next “layer” on top of google search in the greater scheme of information. Before google, one had to spend a lot of time perusing the internet manually. And before that, people had to read physical books or call someone up and have a conversation over the phone. Google drastically sped up that process by giving you the information in 2 seconds, for what would normally take a few hours.
ChatGPT is a game changer for this same process. It speeds up the googling that would normally take a few hours and gives me my information in a few queries.
There have been so many times where I’m reading a book and need to research a bunch of technical terms it throws at me; which then consists of wading through blog posts or documentation, with varying levels of difficulty and quality.
This process is sped up 100x because of chat GPT, because I can followup with questions and customize them to my specific application. Sometimes I need things explained like I’m five; others, a deep technical deep dive.
Point is, it allows for dynamic interaction with content; it helps when I’m struggling with a bug, documentation, a book I’m reading, or pieces of code myself or someone else has written. It’s been an absolute game changer.
167 comments
[ 3.9 ms ] story [ 259 ms ] threadIt’s generally a problem that solves itself, in my experience. Perhaps a benefit of these tools is that we stop the obsession over cheating, which inconveniences honest students in many ways. Cheating has always occurred, but now we can’t even pretend that it’s preventable.
Some students ignored his rants because they didn't find the subject interesting, did the past papers, and then got to the exam and did terribly, then complained about the fact that "it's all based on his rants".
Others found the ranting engaging because it was deep dives on obscure bits of computer security history. My year he spent ~4 weeks going on about Stuxnet, including deep dives into the wider political context. When I got to the paper, one of the few questions we could choose, for 50% of the paper, was just "What was Stuxnet". I wrote pages and pages. Figured out from marking that I got full marks on that one. I always did great in that lecturer's modules despite never taking notes and rarely doing any targeted reading.
To show the level of engagement, I once turned up to one of these lectures 10 mins late, with most of the class already there, and the lecturer said "oh I wondered where you guys were, I guess I'll start again". He knew we were the only ones who cared about the class.
As much as it sounded like the rantings of a tin-foil hat wearing madman, he was in fact quite accurate through a combination of having worked in government in computer security at a fairly high level for a while, and having deep technical knowledge about what was possible.
I miss his lectures, they were excellent, but I hear he's still going strong scaring freshers and running the on-campus teaching union, which is nice to hear.
The path of being interested enough in a field that the easy work is easy, and the hard work is at least interesting, actually turns out to be the path of least resistance.
If I were 18 again, I am not sure what the answer is - would it even be a good idea to pursue this field of study if there aren't going to be many jobs post graduation...
I hope I am wrong...
Future LFMs (Large Foundation Models, since they will be multimodal) can help automate some or much work from specifications onward. Humans will need to validate and revise the results by working with an LFM-powered system (Ref: the spiral model in software engineering).
Yes and that's what code is: most concrete specifications :P
Figuring out what to code is much harder than coding it, like 99.99% of the time.
People worrying about ChatGPT taking away their job frankly do not understand this: my job is to think. I type very little code.
I dunno. I go back and forth on the effects these things will have on programming.
No, because that's not an expectation based in reality.
I think it's always going to be a bit shit, no matter how large they make them it's always going to be an obvious faker, always going to confabulate and lie etc.
As long as we are using a technology that merely fakes intelligence (using statistics to generate the most likely next token based on training data is not intelligence), it is going to be obvious that it's fake. This whole bubble is going to burst when people start realizing how overrated it is.
I've always said that if AI had to deal with some of the executives and product teams I have, with constant insane changes and shipping demands, the AI will more likely figure out how to eliminate people than deal with them.
Some redditors put together where the claim came from, it's a bad interpretation of Microsoft's Scott Guthrie, should be "40% of code checked in by users of github co-pilot" [1][2]
[0] https://youtu.be/ciX_iFGyS0M
[1] https://www.reddit.com/r/github/comments/14qwhem/how_does_gi...
[2] https://www.microsoft.com/en-us/Investor/events/FY-2023/Morg...
I wish AI could write more of my code but it really is much much faster if I just write it myself. The context and curiosity necessary to even know what questions to ask so you know what code to write is far beyond the scope of any AI I’ve seen.
This is great news if your job is to do the thinking. You will have more time for better thinking. Awful news if your job is hand-on-keyboard typing with others telling you what to do.
I struggle to think of a single instance where it's been helpful for me, and I keep trying, because I don't enjoy doing useless grunt work any more than the next guy, and I want to at least try to learn to ask it the "right" questions.
The other subject I give it high praise is shell one-liners. It can produce some hideously abstruse chains that, by and large, do precisely what I asked. It’s not as good with awk, I’ve noticed, but maybe that’s because awk is much more akin to an actual general purpose programming language than, say, sed.
I have seen no evidence of "most code [being] written by AI" and believe this to be completely false.
It is not. 90% of the Job a SW dev is doing is impossible to do for ChatGPT. Writing 15 line snippets from a CS101 course is useful to reduce boring repeated tasks, but nothing more.
For my own course, I think several factors contributed to students not utilizing ChatGPT as much:
It's difficult to say if advancements in LLMs will make my job harder, where say copy/pasting my more complicated assignments can lead to correct results. But from what I can see right now, LLMs still have trouble solving novel problems, so it's probably always possible to come up with assignments that's difficult for them to solve.Have you been using 3.5 or 4?
I fed the problem into ChatGPT later and it was utterly unable to comprehend it, but confidently gave wrong answer after wrong answer.
In my own assignments however, I focus less on algorithmic stuff but more on adding and mixing several things together. E.g. instead of just sorting, do group & sort, and a combination of a bunch of other practical stuff like reading big-endian binary files.
Just for fun I asked ChatGPT 4 to calculate the RMSE between two vectors both in English and Portuguese (also translating RMSE to Portuguese) and it gave me the same code for both questions (asked in separate instances). It would be interesting to know what restrictions you applied.
I'd expect simple tasks like calculating RMSE to definitely be within the abilities of LLM, you might combine things like actually reading the vectors from a CSV file (or a custom format) and calculating RMSE then sorting them etc to see the limitations of LLMs. Most students have no issues with calculating RMSE, they have issues with trying to do all the other stuff that leads to it, and then the combination of sorting and other tasks.
Regarding the restrictions, most of them are just don't use itoa/strtod or strcpy or some other standard library functions.
I agree with you, in my experience, ChatGPT is a better search engine but it is not capable of composing the various parts of an application in a cogent manner. I also think that the current UI is not appropriate for software development and I am sure there are efforts going on to create something closer to Jupyter notebooks for programming. That may be a game changer for your students (and you).
I don't think Jupyter notebooks or like similar REPL interfaces will help too much for my course, at least in the current syllabus. I'm aiming to teach about pointers, memory management etc, the more fundamental parts of how to interact with computers instead of a high level language. Though I would agree that the current UI is suboptimal, some improvements in allowing students to visualize memory layouts and see how their code manipulates memory will help a lot.
I can't recall exactly but I think https://godbolt.org/ might do that for example?
Godbolt is a compiler explorer, it shows disassembly of a code but there's nothing to visualize each step in the process.
A much more interesting consideration would be "Teaching Programming in the Age of Stack Overflow", which involves a well-established, gigantic resource for in-the-trenches programmers that generally provides functional, peer-reviewed and expansively commented solutions to the most common and sometimes most interesting problems.
What programming will be like in an age where LLM's can correctly code is something we won't have any idea about for some time yet (if ever).
I tried it on my/our project 600K LOC (C++), all open source and almost certainly in the training set. Not only could it not explain the questions I asked it correctly, its answers and generated code were absolute jibberish, the sort of thing that would lead to you likely firing the person who gave them to you.
Doesn't matter if people Google, Stackoverflow, use ChatGpt, ask their wizkid neighbor or what have you. You don't need to resist cheaters, especially not when we're talking about adults, it's their responsibility. Cheaters don't learn so when they're tested they sooner or later flunk out.
It is simply far, far easier for a newbie to see a computer generate 1 / 3 / 5 slightly-different attempts to solve a specific problem they have, and then to pattern match sufficiently to be able to solve the problem themselves, than it is to muddle around by yourself with it for hours and hours with no end in sight.
I thought she would have to ask me, someone who has been slinging Python in some form since 2008, a question at least once a week. In reality it's been about 3 times over the last 9 months.
ChatGPT doesn't do much for experienced SWEs, but it demolishes the difficulty curve for newbies. I wish I had this when I was learning.
Interesting, my impression was that ChatGPT helps experienced SWEs by filling out large amounts of boilerplate-ish code. It sounds like that's not your experience, though?
But the more exeperienced you are, more valuable your output is (in most cases). So the 10% of your productivity could be more valuable than a novice's 50%.
At the same time, I see my 9 year old girl using ChatGPT to create Minecraft extensions; she has no prerequisites for actually developing software, but she manages to make some small extensions and have them work either way.
Acknowledging that she doesn't actually learn much from this, I think one can assume that everything in between (e.g. using ChatGPT for pair learning) is definitely also possible.
But whatever AI is outputting, there's no way these qualified guesses can beat experience from a professional developer. I know that the output will never be better than the prompt, but sometimes the output - no matter how much effort you put into the prompt - is visibly just a qualified guess.
I would say with someone new to coding it can be bad and good, as a lot of times it glosses over things, or can be slightly incorrect as it makes assumptions (or more so just answers in a more general context, and when asked to elaborate, or challenged on specifics it will reformat/improve it's answer, but without knowing you need to do so, I could see it easily see it providing half-baked foundational knowledge. You can ask it "x" and it will give a answer, but then if you ask it I am trying to do "y" with "x" and isn't "z" an issue or area of concern with its answer it will reformulate the information provided as its original response was flawed, but if you don't know exactly what the "y" you want to do is, or the "z" being foundational knowledge to challenge it on, you can easily get a whole wall of text that is out of context with what you are actually trying to learn.
This is my hang up with LLMs … I don’t trust them.
Frankly it seems just as easy if not easier to just google keywords and read sites.
Google vs LLM is like asking random people on the internet (some are brilliant, some don’t know anything, some are nuts, … etc.) vs asking random people on the internet but all of them have a history of suffering from hallucinations and are routine liars with a compulsive need to answer confidently even when they know nothing.
Sounds like a description of narcissistic personality disorder or schizoaffective. Of course proto ai would have a personality disorder, go figure.
In the future, the job of an ai psychologist will be to certify the personality of ai products. Gotta make sure you’re not shipping a shrink wrapped psychopath.
It might change eventually as ide integrations improve, but for now it's a novelty for me.
A good craftsman crafts his own tools.
I call bullshit. Your task is to deliver a project that does what it's supposed to, all in a timely fashion. You do whatever it takes, including getting rid of old code and rewriting stuff if it makes you more productive overall.
Writing code that does what you want is much faster than taking someone else's code and twist it to do something it was not written to do. You should always be very careful about what dependencies you use, as most often you will have to deal with their limitations and bugs, and they could kill your project.
If you had 20 separate people who contributed to the same thing and left then you most likely have huge code debt. Managing that sort of thing is a daily task for any software engineer. Apparently your strategy is to just add debt onto the pile and put your head in the sand pretending it's not your problem. Instead you should plan tactical refactoring moves to make the codebase leaner and better able to adapt to the business needs, all without breaking any existing functionality and workflows.
You're asking "when". That's the wrong question. It's up to you to manage your time and invest it into tasks that help you save time later on. A software engineer is usually not paid by the hour either, so you just do the work until you're satisfied with the effort you've put in.
He does not make his own tools. He buys them, because there are already extremely talented tool-makers who are very good at their craft.
He also makes paddles, which are still expensive (I think the cheapest starts at about $1000), but some people actually use those.
Certainly a lot of youtube woodworkers do that, can't say I work wood myself.
But saws, blades, chisels… maybe he could make them, but why? Metalworking is its own subject of mastery.
AFAIK, he doesn’t often modify any tools of that sort. He spends time selecting the one he wants, yes.
Imagine, a vision of the future... A thousand million chatbots, all outputting 2018 JavaScript
I would like an AI to tell me if I’m reinventing the wheel or if something similar can be abstracted. Whether what I’m doing aligns with existing patterns in the codebase. How the application as a whole could be refactored to improve maintainability, performance, or both.
This can all already be done by a human with experience and context, so it must be possible with AI. This will be the biggest game changer for me.
Maybe it's good for some languages, or for code that's doing the exact stuff a hundred people have already written. Copilot X certainly looks cool, but who knows when that'll be ready.
I find it especially helpful with SQL queries, and with boilerplate type code. It’s also very helpful when I’d need to lookup the usage of various methods. I just start with the comments and most of the time it saves me the context switch to the browser to lookup the usage.
Not to mention...things like "cool want to learn python"..."wait - wtf..how do i setup a venv in python!!"
Of course you then have to compare this explanation against all others and see if it fits or if it’s not a valid explanation.
It’s also great for exploring topics which have polluted namespaces on search engines.
Overall though I think the majour benefit of ChatGPT is it just teaches people to clearly define problems as natural language questions. Developing this ability helps the subconscious mind solve problems when the user is away from screens.
One of the types of teaching video that I've found very helpful is where an expert, usually a professor explaining a concept (e.g zero knowledge proof) to five different audiences starting from school kids up to fellow researchers or professors. You can basically ask many different targeted levels of questions to ChatGPT for example, explain the concept as I am five years old, etc.
Yeah we've had them for a while, it was called the rubber duck, ;)
That's because it is easier. Multiple choice is easier than open questions.
It's also a good way to not really grasp anything deeply.
When we teach math to children we don’t start with algebraic structures and proofs. We give them simple recipes.
Later we show them problems that the recipes can’t solve and dig deeper. „By the way, the thing we did last year, it actually works like this…“
This is not mathematics, but arithmetic/calculating. When one starts teaching not-children-anymore mathematics (typically at the university), one indeed starts with something of the kind of algebraic structures and proofs.
She will certainly learn the deeper concepts in university, but catching up with coding quickly, even in a shallow manner, is super beneficial. I think it's a huge plus.
Well it’s right, but you can turn a student from one who doesn’t want to learn into one who does.
You can absolutely motivate unwilling or uncooperative students to learn. I have done it many times.
It has greatly accelerated the process of building prototypes in multiple languages and building various permutations of solutions.
Doing that has allowed me to study various problems both from the bottom up and the top down. I think that has been useful for understanding concepts.
I could have done the same things without AI, but it would have taken a lot longer.
I still think it’s better to learn without too much tooling of any kind, but that’s a topic for another day.
It may very well be a trivial and uninteresting part of mathematics compared to the rest, but it’s still mathematics nonetheless.
Depends. In my high school, math was tought not by telling us the proofs, but by asking us to provide them.
You teach programming by giving simple assignments, not by handing out “calculators”.
Having recently done a screenshare with a fresh CS grad as they developed a simple program and seen how much Copilot was doing for them, I was both impressed and appalled. Yes, it's able to do a ton of that simple stuff, but this new grad has no idea _why_ it's doing what it's doing. They are more concerned with getting it done quickly due to not working on the problem early enough (that's a whole other issue).
It genuinely concerns me for the next generation of CS grads and how well they are going to understand the systems they are building. They are already coming out of school unprepared for the professional development world, this is just going to highly exacerbate the problems.
This fresh grad happens to already be working for us and I was mentoring him.
Yes and no. It doesn't help me that much with technologies that I'm senior within, but it has allowed me to work with a wider tech-stack. For example, I can now confidently write any SQL querys I want (so far anyway) although my prior knowledge in SQL isn't that deep.
I'm having a harder and harder time justifying writing a lot of code myself when faster tools are right at my fingertips
Would love to hear how other SWE's who are using GPT feel about this
It's much slower at producing good code than I am but it's much faster for me to write a prompt than a program, so ultimately it allows me to do two things at once because 'prompt time' is so short I can fit it in between other things.
I don't think of programming as a grind unless I'm working in a shitty codebase, which does happen, certainly.
Even if LLMs or some successor technology never ends up supplanting programmers entirely, it is guaranteed we will see a mixture of deskilling (certain skills no longer being required, like spelling in the age of autocorrect) and massive productivity gains (meaning n-m workers can now do the job of n). These tools aren't going away, and they're only going to improve.
Thus the market for programmers will shrink henceforth. There are no doubts about it. Maybe slowly, maybe quickly, but shrink it will, for the same reason that the market for radiologists is shrinking, or the reason that engineering firms no longer employ whole floors of draftsmen.
1 we could fire a person
2 we could handle 25% more tasks, features and bugs
So I think 2 will happen.
My latest example of LLM issues,| I asked ChatGPT and competition to convert a line of jQuery into native JS, they all got it wrong because jQuery has some selectors like ":header" that is native to JS but the LLM used it anyway though somewhere in it's big memory it has the information that his is not native and if you prompt it right it will fix the issue. So seems to me the LLM are focusing too much on the prompt and failing to use it's full memory on the problem, so it can fix small individual micro tasks but is a complete waste of time for something a bit more advanced , my conclusion you can't have a manager or an artist armed with ChatGPT and create a full project without actually learning to code, at best they will learn to code from the LLM but the hard way and probably using outdated code and inefficient ways.
Because this is what always happens when tools cause productivity gains. Budgets aren't infinite.
- Computerized avionics dramatically cut demand for flight crews. It used to require a dedicated "flight engineer" to literally run the engines (like a train engineer) in a WWII-era bomber, and a dedicated navigator. Now airliners barely need a co-pilot.
- As I mentioned previously, computerized medical imaging has caused demand for radiologists to go down, because one radiologist today can do the work of multiple radiologists of yesteryear.
- In software in particular, if engineering and development is more productive, companies can cut costs and still get the same amount of work done, or possibly more. It's not about firing people immediately, it's about the next company hiring fewer. And the one after that hiring even fewer. Generative coding is going to make software development labs leaner.
>my conclusion you can't have a manager or an artist armed with ChatGPT and create a full project without actually learning to code, at best they will learn to code from the LLM but the hard way and probably using outdated code and inefficient ways.
That's the current state of affairs. Why do you think the technology will not improve beyond where it is now?
And as I said in my reply, this isn't true.
Do you have anything useful to add to the discussion?
There's a non-zero chance that correcting your statement will influence somebody considering whether to specialise in this field, and replying to your repeating of the claim as well as to the original instance increases that chance. I don't consider it pedantry.
I don't see engineers not being part of the process for many years, even if most of the work is done for them. For most companies, if they had their engineers suddenly become 10x more productive, they'd want them to do 10x as much, not scale down to 10% of the staff. Not to say some companies wouldn't.
To replace a developer you would need a human level AI, that is impossible for now.
You can improve the tech to do simple jobs and help me like
- review this code I wrote for potential corner cases
- generate some unit tests for this functions
- find some code in this giant project that does X . I know I wrote it 3 years ago but I have no idea how I named the function or maybe it was a different project.
- rewrite this old IE6 JS/jQuery into modern JAvascript
But IMO a tool that can do this would use LLM but you need an actual valid code checker/interpretor to do a production ready code. I can't trust an LLM to update a giant piece of code without chainging something, I prefer a tool like Intellj refactoring that actual understand code.
I think you'd actually struggle to find an example in history where productivity gains have resulted in prolonged, and industry-wide unemployment for those trained in that area.
I completely agree that technology can make individuals redundant, as in your flight crew example. And yet we are seeing a global shortage of airline pilots, how do you explain that? If technology makes people unemployed (across an economy, over the medium-long term), why aren't there piles of pilots working at McDonald's?
Is not that as technology improves, industries which were once small/unprofitable can balloon into hugely lucrative industries?
On the contrary, it's expanding and there's a global shortage of them [1].
AI has been changing the role of radiologists but not replacing it [2] and radiologists are needed more than ever, but the industry has been struggling to communicate this to students worried by ill-informed claims the profession is under threat from AI [3].
[1] https://www.rsna.org/news/2022/may/global-radiologist-shorta...
[2] https://www.hcinnovationgroup.com/imaging/radiology/article/...
[3] https://www.ncbi.nlm.nih.gov/pmc/articles/PMC10131993/
When steam drills were becoming the norm I'm sure some mining firms were struggling to hire human diggers too.
The "Radiologists are being replaced by technology" story has been repeated so many times by uninformed software developers that it has become popular wisdom, but the reality so far is that we need more radiologists than ever.
Ironically, radiology might be a decent proxy for what could happen with software engineering, but in the opposite way you intend.
[1] https://marvel-b1-cdn.bc0a.com/f00000000046012/info.vrad.com...
To think some new 'programming' tool will make it shrink is not at all warranted. Tools that make computer programming more accessible, more flexible and more powerful will increase demand for people who can reason about it, work on it, teach it, evaluate it etc etc.
I'm not even slightly worried about my job security. I tried Copilot and it sucked.
At the end of the day, LLMs are fakers. They do not possess real intelligence, they merely fake intelligence using statistics and training data. There comes a point where you can't fake it any more.
Maybe in the far future when we have programmed nearly everything there is to program... Until then companies will just produce n*(n/(n-m)) more output.
Actually, the reduction of the cost of programming output will logically lead to more demand as solutions that were previously deemed too expensive to produce will become viable.
Just as an example, try using ChatGPT to explain an article in a foreign language. Can even go as low as a letter-by-letter break down of each word. No private tutor will ever have the patience to teach people at this level.
Coding will become like the liturgical exercises of monks toiling in isolated monasteries while the rest of the world will move on.
I cant wait for a chatGPT decompiler. IDA, watch out!
Hopefully our bugs aren't evolving though.
Disappointing. I cannot evaluate claims about GPT when people say ChatGPT instead of GPT4, if they actually mean the latter.
And if someone really is discussing the usefulness of AI while only having used ChatGPT (GPT3.5), then they’re missing out on a major improvement and their input is less valuable than those discussing GPT 4.
Of course GPT-4 has plenty of limitations but people just need to be clear that they’re familiar with the state of the art.
Also, statements to the effect of “it just predicts the next word” do not appreciate the major difference in capacity for learned abstractions between GPT 3.5 vs GPT 4. So to me it’s just not a useful way of thinking about LLMs. It may technically be true, but at some level that can also be said of human beings. In other news, an airplane “just” flies.
ChatGPT is a game changer for this same process. It speeds up the googling that would normally take a few hours and gives me my information in a few queries.
There have been so many times where I’m reading a book and need to research a bunch of technical terms it throws at me; which then consists of wading through blog posts or documentation, with varying levels of difficulty and quality.
This process is sped up 100x because of chat GPT, because I can followup with questions and customize them to my specific application. Sometimes I need things explained like I’m five; others, a deep technical deep dive.
Point is, it allows for dynamic interaction with content; it helps when I’m struggling with a bug, documentation, a book I’m reading, or pieces of code myself or someone else has written. It’s been an absolute game changer.