Ask HN: How are you using LLMs or ML to improve your SWE?
The other day I needed to write a polyfill for a browser API. I used ChatGPT (in browser) to do it and it got me mostly there pretty quickly.
But I don't yet use any other LLM tooling like Copilot, WizardCoder etc. and I wonder what I'm missing out on.
20 comments
[ 4.0 ms ] story [ 55.1 ms ] threadFor a software engineer not to use LLMs now strikes me as basically "malpractice"
If you think it's the same as using SO, I don't think you have used LLMs enough. For example, I have countless times just dumped errors from my terminal into GPT-4 and it gave me steps to resolve the issues. A number of times, Googling the same issues didn't provide any answers.
Can "copy/paste from stackoverflow" fix and update config files you paste into it? No.
Can it explain to you the changes it made and answer your specific questions about said changes immediately? No.
Can it give you all options for libraries in your language that do x, tell you the pros and cons, relating them to your use-case, and integrate said library into a code snippet you provide? No.
Right now, GPT-4 is tutoring me in Haskell, Elasticsearch, and AWS. Can "copy/paste from stackoverflow" do that? No.
That's fine, but a pretty poor basis of comparison for what StackOverflow can do. If you identify the part that is broken (which most linters will do for you) and paste that into Google, you will get what you want roughly as often as ChatGPT furnishes a proper answer.
I disagree with the parent's pointless moralizing of the situation, but I'm also pretty disenfranchised with the state of GPT-enabled search and programming. When I ask ChatGPT to write something I've already done myself, it will often do things in roundabout or outdated ways. When you play the role of a proctor rather than a student, you can really identify it's shortcomings.
That's not to say AI-enabled coding isn't helpful. But past a certain watermark of trivial problem-solving, AI literally lacks the context to assist in certain codebases where human programmers could easily grok what's happening. Maybe my answer isn't based off the state of the art, but nothing I've tried so far has really gotten everything right. Claiming that it's "malpractice" to avoid it is a plainly inflammatory claim.
Well, I'll certainly accept the critique on that part, lol
I think that the current frontier is the completion of tasks that require large chunks of a codebase as context/knowledge.
I think one major area where it's useful is that it "knows" how to use sufficiently common libraries in your language and is simply a superior form of documentation for a lot of stuff.
For example, I wanted to do stuff with PDFs in Python, both ingesting PDFs and modifying them. For the latter functionality, I asked GPT-4 what my options were for modifying PDFs in Python in xyz manner; it gave some options, I went with ReportLab; I asked it how specifically to do what I wanted in ReportLab, telling it to fill in some half-baked function I copied and pasted in. It filled it in, gave me the imports, etc. This all took about 5-10m. There is no way it would have been efficient for me to research the original question, and no way I could have derived the same functionality from ReportLab's documentation as quickly.
I use it a lot as a replacement for "documentation" in that sense; for language-level libs, for stuff like Docker (whose docs are pretty bad). It's just a waste of time, IMO, to read people's (usually) poorly written documentation or a SO thread when the LLM can just cogently distill the information for you AND answer your specific questions about the issue.
Similarly, it's been pretty invaluable for me in learning Haskell. Syntax and other errors in Haskell provided through VSCode Haskell extension via GHC are comically bad, IMO, often amounting to a small red squiggly that's not even where the actual error is, with the message "parse error". When I have put the code snippets into GPT-4, it has identified and corrected the issue 99% of the time, and I can ask follow up questions of my own and get explanations. It's like having a senior Haskell dev at my beck and call to badger, and I think that's the general vibe of my usage.
I consider myself a pretty competent mid-level dev who can ask good questions and has a reasonable estimation of what he doesn't know, and I think that's a good Goldilocks position to make good use of the tool.
Accordingly, I think if people who are working with the same stuff in the same codebase who have themselves become relatively expert in what they need to know won't find LLMs as useful here, but if you're in a situation where you have to get familiar with a lot of new stuff, they are pretty incomparable.
> If you identify the part that is broken (which most linters will do for you) and paste that into Google, you will get what you want roughly as often as ChatGPT furnishes a proper answer.
Well most of the time, I don't need to identify the part that is broken or devote any mental energy to combing through Google search results. I just copy and paste errors into GPT-4, often without any context, and it tells me what's wrong, and I take the fix and do it, only stopping to comprehend what actually went wrong if I judge that it's worthwhile.
The UX of traditional internet search, with its multiple results of unguaranteed quality, is, I think, kind of obsolete now in many contexts. I have questions to which I want answers. I have patience for reading actual long-form content, such as good dev blog posts, books on stuff like Docker, cloud providers, "how to build an interpreter", etc. What I do not have patience for is combing a bunch of random internet Q&A stuff that is about an error that might not even be the exact error I have.
If you want to be an autodidact, seek out the information from primary sources, study it and organize it yourself. With GPT-4, you're essentially reading a sloppy cliffs notes of the topics. You'll be much better informed and be able to think more deeply about the subject matter if you adopt some ideas from Richard Feynman, wherein you try to explain what you think you know to a child, or at least someone with no knowledge of the subject. That will highlight the gaps in your understanding, and you can return to the source material to fill in the gaps.
You're right about the enshittification of search, though. Maybe ask GPT-4 for a solution to that.
> If you want to be an autodidact, seek out the information from primary sources, study it and organize it yourself. With GPT-4, you're essentially reading a sloppy cliffs notes of the topics. You'll be much better informed and be able to think more deeply about the subject matter if you adopt some ideas from Richard Feynman, wherein you try to explain what you think you know to a child, or at least someone with no knowledge of the subject. That will highlight the gaps in your understanding, and you can return to the source material to fill in the gaps.
The most immediate and fundamental problem with what you are suggesting is that it is infeasible. Nobody has time to learn everything they wish to at the level of depth they would ideally want. Nobody has time to read all the primary sources on which GPT-4 was trained. I find this so obvious I won't belabor the point.
I don't think any other tool can approximate the personalized "per-unit-of-time learning" afforded by a good LLM.
The alternative is whatever one can learn without GPT-4 in whatever amount of time is reasonable/allocated for it at the moment.
Like, take Docker. Yes, you would not be wrong to say that "by getting help from GPT-4 to do stuff, you won't learn it as well as people who didn't get that help and slogged through it over the course of 2 years at one of their last job", but this ignores that I probably will, over the course of the next several months, learn a lot about Docker in a fraction of the time it took the hypothetical other person.
I don't doubt that leaning too much on LLMs as a crutch may have negative effects, but I find this to be a surprising line of argument on a tech forum; there is such an obvious difference in efficiency between learning using an LLM versus the slog of other materials--again, aside from books, though I think they offer, ideally, a different kind of learning.
Now, zoom out to what people actually work on. Typescript projects careening past 10k SLOC, microservice architectures with a mile-wide helm chart, and highly situational firmware programming that depends on proprietary datasheets. Unless you're an utterly disenfranchised CEO (see: Elon Musk, et. al), you will quickly realize that nobody gets paid to work on anything simple. Past a certain point, surface-level "copy and paste" answers stop being helpful. This is where I'd consider that a developer relying on ChatGPT becomes negligent; many projects force you to trade in your "let's try this snippet" instinct for investigative problem-solving. Hitting the reroll button on ChatGPT won't get you any closer to understanding the issue or preventing it from happening in the future.
I'm not entirely bearish on AI or LLMs, but if you can't see their limitations then I recommend pushing them to their limits. Ask ChatGPT something impossible (write an OS) and work your way down to more simpler requests. In my experience, every current AI model gives up long before writing a complete piece of software. Even working on smaller apps and programs (>500 lines) feels like a struggle, with current context limitations.
> In my experience, every current AI model gives up long before writing a complete piece of software.
I don't get why you want to get it to write a complete piece of software.
> I feel you are viewing it as very much an either-or proposition of "it either writes code for me or tells me exactly what's wrong" or it's unusable.
Yeah, kinda. If ChatGPT spends it's output tokens telling you wrong, unnecessary or unhelpful information, it's not usable. Once it passes a certain bar of unreliability, it becomes useless. I'm not sure any place I've worked at in the past would feel comfortable paying me to roll those dice with ChatGPT on their systems.
I'm not against people empowering themselves by delegating AI for programming. Knock yourself out, if you want to develop like that it is entirely your prerogative.
Wow, GPT-4 can correct whitespace errors in whitespace-sensitive languages? That's pretty astonishing.
As a software engineer in aerospace, I have not really found it helpful. But neither has StackOverflow been as helpful to me as it clearly is to many. I think that what kinds of problems you're working on -- more specific than "software" -- likely plays into how helpful such resources can be.
In this context, I have used the LLMs to learn/fix/write about Docker, Google Cloud, Haskell, Nim, various Python libs and frameworks and language features, and various other sundry and misc topics.