I agree with your sentiment. It's like asking a 3 year old to compose a symphony because they learned to play twinkle twinkle little star on a casio keyboard reasonably well.
I don't see it as a "judgemental piece to anyone that i am alluring to" (I think you mean alluding) but I do think it's an honest assesment of those who are attempting to rely heavily on AI.
It is a judgemental piece because the writer is putting a value judgement on people who use LLMs to code and those who don't. Its right after the piece you quoted.
> please don’t take this as a judgemental piece to anyone that i am alluring to. it’s fine to not find programming enjoyable. it’s fine to just want things to work. i am just disappointed at how the ones who care appear to be an ever dying breed.
I'm not saying the author shouldn't write that, write whatever you want. But you should own what you write.
To be fair, we're almost 2 years from the release of ChatGPT and the "demise of the programmer".
We've already did the speedrun from people styling themselves "prompt engineers" to people openly mocking people styling themselves "prompt engineers".
And now it seems we're going to be "6 months to a year away from replacing programmers" for as long as it generates more funding.
Exactly, now replace code with AI generated art, photos, drawings, videos, music. Your employers couldn't care less if its convincing enough to ship. even better now that it only takes seconds to minutes.
We are at the cusp of creative destruction and we are only getting started. Ironically, blue collar jobs seem safe as there hasn't been a humanoid revolution and what I see in the white collar field is what blue collar workers experienced before the automation and offshoring of jobs
I think AI art is actually an interesting example. It’s mostly run of the mill schlock to replace clip art / stock images.
Adequate but not enjoyable. Lorem ipsum of visual art. Probably kills some basic graphic design jobs at the margin working on low budget projects.
Meanwhile big brands using big agencies will just incorporate it into their design process. McD Japan is a recent example. You still need a human with an eye and taste to be the editor.
But no one is reading or viewing AI art for pleasure. It’s all “that’s neat” (continues scrolling).
The chances of landing on an IT job for which I can take pride in, is 100% remote and that pays well is very low. To begin with, I dislike all the faang-like companies.
Besides, being good at programming makes it easier to deal with BS jobs that pay well, so it’s not that I suffer 40h/week.
> it is sad to me just how much people are trying to automate away programming and delegating it to a black box
I take it you're not using a compiler to generate machine code, then?
Scratch that, I guess you're not using a modern microprocessor to generate microcode from a higher-level instruction set either?
Wait, real programm^Wartists use a magnetised needle and a steady hand.
Programming has always been about finding the next black box that is both powerful and flexible. You might be happy with the level of abstraction you have settled on, but it's just as arbitrary as any other level.
Even the Apollo spacecraft programmers at MIT had a black box: they offloaded the weaving of core rope memory to other people. Programming is not necessarily about manually doing the repetitive stuff. In some sense, I'd argue that's antithetical to programming -- even if it makes you feel artistic!
Thank you for saying this. It's always baffled me that people will decry ChangeX as unnatural and wrong when it happens in their lifetime, but happily build their lives upon NearlyIdenticalChangeY so long as it came before them.
how do you figure? people were making directly analogous arguments about compilers back in the day. (not trying to argue that they are 'the same', but their is definitely a spectrum of code generation methods, with widely varying genres of guarantees, suiting a widely varying range of use cases)
They aren't saying they're the same, I'm not sure how you got that interpretation. It's very clear they're highlighting the hypocrisy that arises from claiming to be against automating away aspects of programming while relying on tools that do exactly that for you - only being Ok with it as long as they aren't called "AI".
The crux of why this is a bad analogy is that everyone talking about "automating" things with LLMs is misusing the word "automation". A machine can automate a repetitive manual task. A computer can automate the operation of machinery. A machine instruction set is an abstraction on top of circuitry that can automate the labor of extrapolating the logic physically executed by that circuitry into human-comprehensible routines. In the same way, a programming language implementation (e.g. a compiler) can somewhat automate programming (in the sense that it uses higher levels of abstraction to describe the same thing, saving labor while keeping determinism). What do these things have in common? We can reliably make them approach deterministic behavior. In the case of compilers, completely and reliably and transparently so. Just because you haven't bothered to read what a compiler is doing doesn't mean someone can't verify what it's doing. Physical machines are less reliable, but we have reliable ways to test them, reliable error margins, reliable failure modes, reliable variance. When you are on a stack of abstractions like a programming language on top of a compiler on top of transistors on top of a machine, an error at the top of that stack can have a lot of implications. A tool that probabilistically generates code is not automation. We have no guarantees about how and when it will get things wrong, how much this will happen, and what kinds of things it will get wrong. We have no way to audit their results that will generalize to every problem upstream of them. We have no way to reliably measure improvement in consistency, let alone improve that margin of error reliably. The entire idea that this is an automation at all is nonsense.
I get the point that they are in different magnitudes of unknown but the analogy is still pretty good when it comes to the median programmer, who has no idea what goes on within either one. And if you argue that compilers are ultimately deterministic, that same argument technically holds for an LLM as well.
The biggest difference to me is that we have humans that claim they can explain why compilers work the way they do. But I might as well trust someone who says the same about LLMs, because honestly I have no way to verify if they speak the truth. So I am already offloading a lot of burden of proof about the systems I work on to others. And why does this ”other” need to be a human.
This is like saying “I don’t understand how airplanes fly, so I’ll happily board an airplane designed by an LLM. The reality is determined by how much I know about it.”
No, the other way around. I am saying it is not a smart take to say ”a safe airplane cannot be built if LLMs were used in the process in any way, because reasons”. The safety of the airplane (or more generally the outcome of any venture) can be measured in other ways than leaning on some rule that you cannot use an LLM for help at any stage because they are not always correct
While the base of your argument is true, it’s also a bit dishonest. LLMs are significantly different than any of these other abstractions because they can’t be reasoned about or meaningfully examined/debugged. They’re also the first of these advances which anyone has claimed would eliminate the need for programmers at all. I don’t believe the C compiler was meant to do my whole job for me.
I don't buy that you're actually examining compiled programs. Very few people do. Theoretically you could, but the whole point of the compiler is to find optimizations that you wouldn't think of yourself.
A big feature of compilers is to find optimizations you wouldn't think of. I tried to make the point that compiled output is typically not read by humans
The point of an optimizing compiler is to find optimizations which, crucially, are semantics-preserving. This is the contract that we have with compilers, is the reason that we trust them to transform our code, and is the reason why people get up in arms every time some C compiler starts leveraging undefined behavior in new and exciting ways.
We have no such contract with LLMs. The comparison to compilers is highly mistaken, and feels like how the cryptocurrency folks used to compare cryptocurrency to gestures vaguely "the internet" in an attempt to appropriate legitimacy.
> I don't buy that you're actually examining compiled programs. Very few people do
I take it you don't write C, C++, or any language at that level? It is very common to examine compiled programs to ensure the compiler made critical optimizations. I have done that many times, there are plenty of tools to help you do that.
If using an LLM meant carefully crafting a complex, precise, formal prompt that specified only one possible output, I might be interested. But then I wonder if the prompt would be very much shorter.
Thinking about it, this depends on which differences we consider aspects of the output program, and which ones we consider trivial differences that don't count. If you say "build an RPG about dragons with a party of magic using heroes" and the LLM spits one out, you reached a level of abstraction where many choices relating to taste and feeling and atmosphere (and gameplay too) are waved aside as trivial details. You might extend the prompt to add a few more, but the whole point of creating a program this way is not to care about most of the details of the resulting experience. Those can be allowed to be generic and bland, right? Unless you care about leaving your personal touch on, say, all of them.
But this localization makes it computationally possible, and has limits.
The qualification and frame problems, combined with the very limited computational power of transformers is another lens.
LLMs being formalized doesn't solve the problem. Fine tuning and RAG can help with domain specificity, but hallucinations are a fundamental feature of LLMs, not a bug.
Either a use case accepts the LLM failure mode (competent, confident, and inevitably wrong) or another model must be found.
Gödel showed us the limits of formalization, unless we find he was wrong, that won't change.
Thanks for your insightful comment. I'll read the links later.
I had just assumed that RNNs were TC, didn't think of limitation put on by bounded precision since I assumed that any bounded precision could be compensated by growing memory module.
> As discussed in the paper and pointed out by the reviewer, the growing memory module is non-differentiable, and so it cannot be trained directly by SGD. We acknowledge this observation.
Two stack FSA/RNN are interesting, but as of now, not usable in practice.
I think you’re assuming your reference is the correct one. I can’t reason about the assembly language that the compiler spits out, the microcode in the CPU kernel or any of the electronics on the motherboard. That anyone can or not doesn’t change things in my opinion. It’s an arbitrary distinction to say _this_ abstraction is uniquely different in this very specific way.
LLMs are deterministic if you force a seed or disable sampling. They however do not guarantee that small input changes will cause small output changes.
So as the OP said, all the parts are deterministic in this stack. Their behavior is fixed for a given input, and all the parts are interpretable, readable, verifiable and observable.
This is entirely different from LLMs which are opaque even to their designers, and have unpredictable flaws and hallucinations, they are probability machines based on what data they have been exposed to, which means they are not a reliable way to generate programs.
Maybe one day we'll fix this, but the current generation is not very useful for programming because of this.
Cobol and other early high-level languages were designed with the intention of allowing businesspeople to write their own programs so programmers wouldn't be needed. Some people really believed that!
I'd really like to have everything written in Rust, not C. Rust does a lot of verification, verification that is very hard to understand. I'd like to be able to specify a function with a bunch of invariants about the inputs and outputs and have a computer come up with some memory-safe code that satisfies all those invariants and is very optimized, and also have a list of alternative algorithms (maybe you discard this invariant and you can make it O(nLog(n)) instead of O(n^2), maybe you can make it linear in memory and constant in time or vice versa...)
Maybe you can't examine what the LLM is doing, but as things get more advanced we can generate code to do things, and also have it generate executable formal proofs that the code works as advertised.
I agree with the second part of your argument, regarding the assertion that LLMs may eventually replace programmers.
However, I don't understand your claim that an LLM acting as a programming assistant "...can’t be reasoned about or meaningfully examined/debugged."
I type something, and Copilot or whatever generates code which I can then examine directly, and choose to accept or reject. That seems much easier to reason about than what's happening inside a compiler, for example.
We can argue the same thing for artists. Who wrote the algorithms for your favorite Photo/Image editor? Who created the image formats and standards, infrastructures for you to be able to push binary files to millions of people?
Which would be relevant if either side respected it.
In practice, compilers frequently have bugs and programmers even more frequently make use of "what the compiler actually does" rather than adhering to the language specification -- to the point where the de facto spec for many languages is "what the canonical implementation does".
The frequency with which compiler implementations functionally diverge from language specifications is dwarfed, by many orders of magnitude, by the frequency with which LLMs generate provably nonsensical code in response to a prompt.
To wit, a compiler diverging from the specification is so relatively rare that people will get angry about it and demand that it be fixed, while an LLM spewing creative nonsense is so accepted and par for the course that complaining about that fact is met with a shrug and "well, what did you expect?"
LLMs don’t have a canonical output that could serve as a specification. And if they had, we wouldn’t consider that a satisfactory specification at all.
You can learn how compilers work and understand how they do what they do. Nobody understands what’s in those billions of parameters, and no one ever will.
why are there so many parameters in the first place? and was it humans who generated so many? seems like a very big job for a human to do, or even a team of humans to do.
disclaimer: I know next to nothing about llms. and I'm not that interested to learn about them. just asking casually.
They're not manually generated or anything, it's just a setting. Too few and the model doesn't have enough flexibility to capture complex patterns. Too many and the model can just memorize the data you train it on rather than capturing the patterns driving it.
> why are there so many parameters in the first place?
Because parsing and writing human language in a natural way is extremely complex.
> and was it humans who generated so many?
No, it is generated using an algorithm that tries to predict the next word in human written text using the words that comes before it. It ingests basically all the text on the internet to do this, without that much text the LLM performs horribly.
I don't think that this is a fair comparison because at some point the nature of the craft actually does change.
To give an analogy, a carpenter might be happy with hand tools, happy with machine tools, happy with plywood, and happy with MDF. For routine jobs they may be happy to buy pre-fabbed cabinets.
But for them to employ an apprentice (AI in this example) and outsource work to them - suddenly they are no longer really acting as a carpenter, but a kind of project manager.
edit: I agree that LLMs in their current state don't really fundamentally change the game - the point I am trying to make is that it's completely understandable that everyone has their own "stop" point. Otherwise, we'd all live in IKEA mansions.
Running state of the art LLMs for programming is nowhere near project management. At least in my experience, all LLMs are really good at is dumping plausible tokens quickly. They can't think, design, or handle tradeoffs intelligently.
They help me with the keyboard work, not any of the actual programming.
An apprentice is another person performing the same kind of work as the carpenter. That's fundamentally different from using an LLM, which is not a person and does not function like a person.
Whether you think LLMs are spectacularly worthwhile or odious and destructive, it's crucial not to classify them as being a person instead of a software tool.
This is a very poor analogy. It's not a matter of abstractions, it's a matter of getting someone or something else to do the work, while you mostly watch and fix any errors you're able to catch.
HN commenter: Samuel, why don't you use an LLM to write this play you are working on?
Beckett: What?
HN commenter: Well it's just like when you decided to work in French instead of English. Your art was no less because of it. Now you can use an LLM instead of French. It will be so much quicker.
>> I take it you're not using a compiler to generate machine code, then?
The dismissive glibness of your comment makes me wonder if it's worth it trying to point out the obvious error in the analogy you're making. Compilers translate, LLMs generate. They are two completely different things.
When you write a program in a high-level language and pass it to a compiler, the compiler translates your program to machine code, yes. But when you prompt an LLM to generate code, what are you translating? You can pretend that you are "translating natural language to code" but LLMs are not translators, they're generators, and what you're really doing is providing a prefix for the generated string. You can generate strings form an LLM with an empty prefix; but try asking a compiler to compile an empty program.
>> Even the Apollo spacecraft programmers at MIT had a black box: they offloaded the weaving of core rope memory to other people.
There is no "black box" here. Programmers created the program and handed it over to others to code it up. That's like hiring someone to type your code for you at a keyboard, following your instructions to do so. You have to stretch things very far to see this as anything like compilation.
Also, really, compilers are not black boxes. Just because most people treat them as a scary unknowable thing doesn't mean that's what they are. LLms are "black boxes" because no matter how much we peer at their weights, arrays of numerical values, there's nothing we can ... er ... glean from them. They're incomprehensible to humans. Not so the code of a compiler. Even raw binary is comprehensible, with some experience.
I recently had used an LLM to convert a lot of Python to Rust. It got it 99% right, and it took me a short while to fix the compile-time errors, and carefully check the tests weren't broken (as I trusted the code worked when the tests passed).
Is that "compiling" or "translating"? Lots of people use language to C "compilers".
I get what you're trying to say, but I don't entirely agree. Raising levels of abstraction is generally a good thing. But up until now, those have mostly been deterministic. We can be mostly confident that the compiler will generate correct machine code based on correct source code. We can be mostly confident that the magnetised needle does the right thing.
I don't think this is true for LLMs. Their output is not deterministic (up for discussion). Their weights and the sources thereof are mostly unknown to us. We cannot really be confident that an LLM will produce correct output based on correct input.
Most frequent output does not imply correctness, LLMs often are confidently wrong.
They can't even perform basic arithmetic (which is not surprising since they operate at the syntactic level, oblivious to any semantic rules), yet people seem to think offloading more complex tasks with strict correctness requirements is a good idea. Boggles the mind tbh.
For the average programmer the infinite layers of abstraction, libraries and middleware isn't deterministic either. The LLMs actually honest to god being probabilistic estimators doesn't change anything about what they produce or how they see their own stuff.
I’d say it’s not only determinism, but also the social contract that’s missing.
When I’m calling ‘getFirstChar’ from a library, me and the author have a good understanding of what the function does based on a shared context of common solutions in the domain we’re working in.
When you ask ChatGPT to write a function that does the same, your social contract is between you and untold billions of documents that you hope the algorithm weights correctly according to your prompt (we should probably avoid programming by hope).
You could probably get around this by training on your codebase as the corpus, but until we answer all the questions about what that entails it remains, well, questionable.
I use Cursor at work, which is basically VSCode + LLM for code generation. It's a guess and check, basically. Plenty of people look up StackOverflow answers to their problem, then verify that the answer does what they want. (Some people don't verify but those people are probably not good programmers I guess.) Well, sometimes I get the LLM to complete something, then verify the code is completed is what I would have written (and correct it if not). This saves time/typing for me in the long run even if I have to correct it at times. And I don't see anything wrong with this. I'm not programming by hope, I'm just saving time.
This increases the time you spend proofing other’s work (tedious) versus time you spend developing a solution in code (fun). Also, if the LLM output is correct 95% of the time, one tends to get more sloppy with the checking, as it will feel unnecessary most of the time.
Just what I was thinking about lately, what if LLMs are not 95% precise, but 99,95%. After like 50-100 checks you find nothing, and you just dump the whole project to be implemented - and there come the bugs.
However ... your colleagues just do the same.
We'll see how this unfolds. As for now the industry seems to be a bit stuck at this level. Big models too expensive to train for marginal gains, smaller are getting better but doesn't help this. Until some one new idea comes in how LLMs should work, we won't see the 99.95% anyway.
one idea is obvious: multi-model approach. it partially done today for safety checks. the same can be done for correctness. one model produces result, different model only checks the correctness. optionally several results, second model checks correctness and selects the best. this is more expensive, but should give better final output. not sure, this may have been already done.
> This increases the time you spend proofing other’s work (tedious) versus time you spend developing a solution in code (fun).
I find that I don't use it as much for generating code as I do for automating tedious operations. For example, moving a bunch of repeating-yourself into a function, then converting the repeating blocks into function calls. The LLM's really good at doing that quickly without requiring me to perform dozens of copy-paste operations, or a bunch of multi-cursor-fu.
Also, I don't use it to generate large blocks of code or complicated logic.
I agree with you but I want to try to define the language better.
It's not that LLMs aren't deterministic, because neither are many compilers.
It's also not that LLMs produce incorrect output, because compilers do that to, sometimes.
But when a compiler produces the wrong output, it's because either (1) there's a logic error in my code, or (2) there's a logic error in the compiler†, and I can drill down and figure out what's going on (or enlist someone to help me) to fix the problem.
Let's say I tell an LLM to write a algorithm, and it produces broken code. Why didn't my prompt work? How do I fix it? Can anyone ever actually know? And what did I learn from the experience?
---
† Or I guess there could be a hardware bug. Whatever. I'm going to blame the compiler because it needs to produce bytes that work on my silicon regardless of whether the silicon makes sense.
This is in general only true for either trivial toy compilers or ones which have gone to lengths to have reproducible builds. GCC for instance uses a randomised branch prediction model in some circumstances.
Ok, but my understanding is that that they are mostly deterministic. And that there are initiatives like Reproducible Builds (https://reproducible-builds.org) that try to move even more in that direction.
But what does "mostly" mean? You can compile the same code twice and literally get two different binaries. The bits don't match.
Sure, those collections of bits tend to do exactly the same thing when executed, but that's is in some sense a subjective evaluation.
---
Szundi said in a sibling comment that I was "completely [missing] the point on purpose" by bringing up compiler determinism. I think that's fair, but it's also why I opened my post by saying "I agree [with the parent], but I want to try to define the language better." Most compilers in use today are literally not deterministic, but they are deterministic in a different sense, which is useful as a comparison point to LLMs. Well, which sense? What is the fundamental quality that makes a compiler more predictable?
I'd like to try to find the correct words, because I don't think we have them yet.
I'm not an compiler expert, not by far. But my understanding is that if you compile the same code on the same machine for the same target, you'll get the same bits. Only minor things like timestamps that are sometimes introduced might differ. In this sense, maybe they are not deterministic. But I think it's fair to classify them as "determinstic" compared to LLMs.
> We cannot really be confident that an LLM will produce correct output based on correct input.
There are 2 things at play here, one is LLM with human in the loop, in which it's just a tool for programmers to do the same thing they have been doing, and the other is LLM as black box automaton. For the former, it's not a problem that the tool is undeterministic, we are double checking the results and add our manual labour anyway. The fact that a tool can fail sometimes is an unsurprising fact of engineering.
I think the criticism in this chain of comment applies more to the latter, but even it always has values to non-tech people, just like how no-code approaches are, however shitty it looks to us software enfineers.
I don't know. Programming with an LLM turns every line of code into legacy code that you have to maintain and debug and don't fully grok because you didn't write it yourself.
If it's in your PR then you wrote it, no one should be approving code they do not understand whether that's from AI or googling. Nothing changes there.
> We can be mostly confident that the compiler will generate correct machine code based on correct source code.
Recently got email about gcc 14.2, they fixed some bugs in it. Can we trust it now, these could be the last bugs. But before that it was probably a bad idea to trust. No, even compiler's output requires extensive testing. Usually it's done at once, just final result of coding and compilation.
> Their output is not deterministic
yes.
> Their weights and the sources thereof are mostly unknown to us
Some of them are known. Does it make you feel better. There are too many weights, so you are not able to track its 'thinking' anyway. There are some tools which sort of show something. Still doesn't help much.
> We cannot really be confident that an LLM will produce correct output based on correct input
No, we can't. But it's so useful when it works. I'm using it regularly for small utilities and fun pictures. Even though it can give outright wrong answers for relatively simple math questions. With explanations and full confidence.
This is a qualitatively different kind of abstraction. All other abstractions still require the programmer to express the solution in a formal language, while LLMs are allowing the user to express the solution in natural language. It's no longer programming, but much more like talking to a programmer as a manager.
By that light, the artist that is painting using store-bought pigments instead of hand made, or brushing paints instead of reallocating molecules, or even better, atoms, is also using a black box.
I think OP has a point, and is about guiding the design, then overall structure and dynamic of the code. Nobody expects to write in Assembler or not using libraries, but making a concious decision on the design.
I have met few programmers, but many coders. For them coding is a job, and generally they don’t care about overall architecture, algorithm efficiency, and code elegance is limited to syntax-coloring-themes in their editor. I respect their existence, but generally they are building on top off somobody else’s effort.
I guess we are currently in the special situation, that we as human programmers can understand the output of a coding LLM. That's because programming languages are designed to be human readable. And we had an incentive to learn those languages.
I imagine that machine learning powered coding will evolve to an even blacker box, than it is today: It will transform requirements to CPU instructions (or GPU instructions, netlists, ...). Why bother to follow those indirections, that are just convenience layers for those weak carbon units (urgh)?
Simultaneously, automation will likely lead to fewer skilled programmers in the future, because there will be fewer incentives to become one.
Together those effects could lead to a situation where we are condemned to just watch.
Your argument is a false dichotomy and thus a logical fallacy. Not all steps forward in abstraction or code generation are necessary good steps and have to be considered on their own merits. If LLMs are indeed superior, then you should be able to articulate their merits without condescending to fallacious attacks.
The transformations you're referring to are fully deterministic and guaranteed to be correct.
LLMs provide statistically probable answers with no guarantee of correctness. It takes more time to review LLM code than it takes to write it correctly from scratch.
I 100% agree with you. Whenever I see artists buy their brushes I cringe. Real artists don't draw anything until they've grown a tree and raised horses to obtain the raw materials (wood and horse hair) to make their first brush.
Using a bought brush to paint and generating a painting via a prompt are basically identical.
The thing is, all these stacks are built by people and verified against specifications. When they failed to perform the way they should, we fixed them.
Plus, all the parts are deterministic in this stack. Their behavior is fixed for a given input, and all the parts are interpretable, readable, verifiable and observable.
LLMs are none of that. They are stochastic probability machines, which are nondeterministic. We can't guarantee their output's correctness, and we can't fix them to guarantee correct output. They are built on tons of (unethically sourced) data, which has no correctness and quality guarantees.
Some people will love LLMs, and/or see programming as a task/burden they have to complete. Some of us love programming for the sake of it, and earn money by doing it that way, too.
So putting LLMs to the same bucket with a deterministic, task specific programming tool is both wrong, and disservice to both.
I'm also strongly against LLMs, not because of the tech, but because of how they are trained and how their shortcomings are hid and they're put forward as the "oh the savior of the woeful masses, and the silver bullet of all thy problems", and it's neither of them.
LLMs are just glorified tech demos which shows what stochastic parrots can pose as accomplishing when you feed the whole world to them.
We can (for programming, at least): run the output thru theorem prover, ensure that proof is constructive, the Curry-Howard correspondence guarantees that you can turn the output into a correct program. It doesn't guarantee that formal properties of the program correspond to the informal problem statement. But even people occasionally make such errors (a provably correct program doesn't do what we wanted it to do).
> and we can't fix them to guarantee correct output
Same thing with other systems capable of programming. That is people.
You just can't make a system that guarantees correct transformation from an informal problem statement into a formally correct implementation. "Informal" implies that there's wiggle room for interpretation.
No, it doesn't mean that current LLMs are ready to replace programmers, it also doesn't mean that ML models of 2030s will not be able to.
Indeterminism is not always bad. A probabilistic Turing machine is more powerful than a Turing machine, for example (BPP complexity class is a superset of P).
>> We can (for programming, at least): run the output thru theorem prover, ensure that proof is constructive, the Curry-Howard correspondence guarantees that you can turn the output into a correct program. It doesn't guarantee that formal properties of the program correspond to the informal problem statement. But even people occasionally make such errors (a provably correct program doesn't do what we wanted it to do).
That sounds very ambitious. Automated theorem provers are real sticklers for complete specifications in a formal language and can't parse natural language at all, but when you generate code with an LLM all you have in terms of a specification is a natural language prompt (that's your "informal problem statement"). In that case what exactly is the prover going to prove? Not the natural language prompt it can't parse!
The best you can do if you start with a natural language specification, like an LLM prompt, is to verify that the generated program compiles, i.e. that it is correct syntactically. As to semantic correctness, there, you're on your own.
Edit: I'm not really sure whether you're talking about syntactic or semantic correctness after all. Which one do you mean?
>> You just can't make a system that guarantees correct transformation from an informal problem statement into a formally correct implementation. "Informal" implies that there's wiggle room for interpretation.
Note that in program synthesis we usually make a distinction between complete and incomplete specifications ("problem statements") not formal and informal. An incomplete specification may still be given in a formal language. And, for the record, yes, you can make a system that guarantees that an output program is formally consistent with an incomplete specification. There exist systems like that already. You can find a bit about this online if you search for "inductive program synthesis" but the subject is spread over a wide literature spanning many fields so it's not easy to get a clear idea about it. But, in general, it works and there are approaches that give you strong theoretical guarantees of semantic correctness.
Ah, I said "theorem prover", I should have said "proof verifier". What I meant is something like DeepMind's AlphaProof with an additional step of generating a formal specification from a natural language description of the problem. In this way we get a semantically correct program wrt the formal specification. But with current generation of LLMs we probably won't get anything for non-trivial problems (the LLM won't be able to generate a valid proof).
> Note that in program synthesis we usually make a distinction between complete and incomplete specifications
Program synthesis begins after you can coherently express an idea of what you want to do. And getting to this point might involve a ton of reasoning that will not go into a program synthesis pipeline. That's what I mean when I say an "informal problem statement": some brain dumps of half-baked ideas that doesn't even constitute an incomplete specification because they are self-contradictory (but you haven't noticed it yet).
LLMs can help here by trying to generate some specification based on a brain dump.
>> What I meant is something like DeepMind's AlphaProof with an additional step of generating a formal specification from a natural language description of the problem.
That's even more ambitious. From DeepMind's post on AlphaProof:
First, the problems were manually translated into formal mathematical language for our systems to understand.
DeepMind had to resort to this manual translation because LLMs are not reliable enough, and natural language is not precise enough, to declare a complete specification of a formal statement, like a program or a mathematical problem (as in AlphaProof) at least not easily.
I think you point that out in the rest of your comment but you say "the LLM won't be able to generate a valid proof" where I think you meant to say "a valid specification". Did I misunderstand?
>> Program synthesis begins after you can coherently express an idea of what you want to do.
That's not exactly right. There are two kinds of program synthesis. Deductive program synthesis is when you have a complete specification in a formal language and you basically translate it to another language, just like with a compiler. That's when you "coherently express an idea of what to do". Inductive program synthesis is when you have an incomplete specification, consisting of examples of program behaviour, usually in the form of example pairs of the inputs and outputs of the target program, but sometimes program traces (like debug logs), abstract syntax trees, program schemas (a kind of rich program template) etc.
Input-output examples are the simplest case. Today, if you can express your problem in terms of input-output examples there are approaches that can synthesize a program that is consistent with the examples. You don't even need to know how to write that program yourself.
> where I think you meant to say "a valid specification". Did I misunderstand?
What do you mean when you say "a valid specification"? There are known algorithms to check validity of a proof. How do you check that specification is valid? People are inspecting it and agree that "yes, it seems to be expressing what was intended to be expressed in the natural language", or, "no, this turn of phrase needs to be understood in a different way" or some such. Today there's no other system that can handle this kind of a task besides humans (who are fallible) and LLMs (that are much more fallible).
That is deciding that specification is valid cannot be done without human involvement. I left that part out and focused on what we can mechanistically check (that is validity of a proof).
So, no, I didn't mean "a valid specification". And, yes, I don't think that today's LLMs would be good at producing specifications that would be deemed valid by a consensus of experts.
> Today, if you can express your problem in terms of input-output examples there are approaches that can synthesize a program that is consistent with the examples
In a limited domain with agreed-upon rules of generalization? Sure. In general? No way. The problem of generalizing from a limited number of examples with no additional restrictions is ill-defined.
And the problem "generalize as an expert would do" is in the domain of AI.
> "oh the savior of the woeful masses, and the silver bullet of all thy problems"
Who said that? Everyone I've talked to warns about their shortcomings (including their creators) and even the platform where I use them has a warning plastered right under the input box saying "ChatGPT can make mistakes. Check important info."
I would argue thats an LLM spec --> Generate probabilistic output with a degree of confidence on the output nearing p(1). IMHO End users are supposed to not take the output of these machines as is but rather iterate on top and finish their task in lesser time.
Compilers are complex programs fraught with bugs. Modern microprocessors are hideously complex devices fraught with bugs. But at least we understand them in principle and practice.
LLMs are nonsense generators, you need a second device that can recognize correct programs to use them effectively. Only humans can do that all-important second part.
> Programming has always been about finding the next black box that is both powerful and flexible.
That's the opposite of programming. Programming is the art and science of developing reliable algorithms. You can treat programs as black boxes only after you're sure that they work correctly. Otherwise you're just engaged in a kind of cargo cult.
That the scary thing about the LLM fad: so many people seem so willing to abdicate their responsibility to actually think.
Reminds me long ago of working with a guy that did everything in C when we were rewriting things in Perl . Yes, his stuff was faster. Yes, it was also buggier, harder to debug, and took 3x as long to write for similar levels of functionality (it wasn't speed dependent code by any stretch.)
The question is about the abstraction being understandable and predictable. All the examples you have follow that, LLMs throw that out of the window.
>> Scratch that, I guess you're not using a modern microprocessor to generate microcode from a higher-level instruction set either?
Hell, I design gate level logic -> map it to instructions -> use them in C for the very LLMs and can fully understand[0] every aspect of it (if it doesn't behave as expected, that is a bug) but I cannot fathom or predict how the LLMs behave when i use them even though I know their architecture and implementation.
[0] Admittedly I treat the tools I use during the process, like cad tools, compiler, as black boxes, however I know that if I want to or the need arises, I can debug/understand them.
Actually the opacity of the abstraction layer is the core of the issue. First we note that opacity is a measure in both the inner-workings of the 'box' and orthogonally (in context of LLMs) a measure of deterministic outcome.
Programming, it is asserted by some of us, is exactly the act of instructing a deterministic 'black' box.
Peer-coding with an LLM is the act of cajoling a mechanism to hopefully consistently produce the input to a sensible "blackbox". It is not programming, it is getting help. Now if the help was 100% reliable, we could discuss programming the helper.
The other day I had a vision of the future AI whisperer in the corporate setting. They wear capes of varied colors and possibly sport a wand. "It's an art you see".
This feels exactly like the manifesto that artists on X started defending that Ai-generated Art isn't Art but look where we are now, companies are able to generate not only quality Art assets but photos too but amateur consumers creating songs, videos, pictures that they couldn't have previously.
Unlike art where you can tell if the end product is AI generated even, software is not. The end users do not care if you used AI to generate your react front-end or back-end, your employers dont care who wrote the code as long as its bug free and works like in the scope document.
Ironically, I don't see software developers writing manifestos or complaining that AI generated software isn't software or that they are stealing from them.
Seems to me like art is art if the person who made it feels that it's art, and that may or may not be a component of the process that went into an apparently artistic product. Artistic products seem more likely to be "art" of there's no specific value to them, and they exist to exist or to add some abstract element of decoration to world. A former theatre friend of mine once got mad when I described art as having no intrinsic value, but that doesn't mean it has no value, and I don't think it fundamentally changes the non-art noun of what's produced; a painting/print can be just a decoration, and/or it can be art, but the fact that it's art doesn't change whether it's a painting or a decoration or the type of object. That may not be the case of course if the resulting object is an artistic illusion, such as a table that's not a functional table because it's made of cake, but again I don't think it needs to get that deep for the point to be true.
Likewise, software that was produced artistically may or may not be better, or more valuable, or distinguishable in any way from other software products, but if the author feels like it's art it may be art. That may be because there's no tangible reason for it to exist other than a creative endeavor, in which case maybe it's not actually software.
You presented 2 broad generalizations about "software developers" and "artists" along with your statement about this specific "manifesto", so that's what I was referring to.
Artistically produced software though could certainly be a CRUD app just as much as it could be a photo that you pulled off the shelf at Target, I'm arguing that the type of product it is doesn't necessarily have a bearing on whether it can be art or have elements of art in it.
Many people use the term in different ways to describe a kind of abstract ambiguously valuable process of creation, or the product of it. I think it's possible to interpret software as art, but it's not a necessary quality for the software to exist.
For purely artistic works of software, my opinion is that they'd pretty much serve no explicit purpose at all other than as a creative exploration of some sort, completely open to interpretation with no success or failure case. Like the webfont Candy which was made with CSS shapes, creative coding, or various kinds of digital illustrations. Exploring what you can do with ai generated imagery is surely one of them, but not necessarily so if it's just the solution to a problem.
Most things people would describe as "having an art to it" or "art" have nuanced colloquial interpretations, but it's usually just an aspect to the creation process that embodies some of these qualities. Someone could say "there's an art to sucking as bad as you do at X" and although it's meant to be figuratively derisive rather than literally artistic, it could also refer to the abstract nebulous means by which someone fails to be good. Likewise, it could mean the abstract nebulous process by which someone makes CRUD apps good/bad.
I tend to agree with this post. I am the only person on my team and one of the few at my company not using LLMs to assist my programming. So far I don’t think I notice any difference (better or worse) … we’ll see how this experiment plays out I guess.
I find the post confusing. The author notes two aspects of programming and then seems to conflate LLMs as doing both when they only do the second. It’s sort of like saying directing a movie isn’t art or creative because the director is not every actor in the movie?
I suppose I get where the sentiment is coming from and anybody can use whichever tools make them happy but I feel like the comparison doesn’t make too much sense to me. As programmers we leverage so many levels of abstraction that help us write better code. It feels similar to saying if you use some package or library you’re letting some library author do the painting for you. Or if you use a high level language instead of assembly you’re letting a compiler do the painting for you.
I think of myself as a code artist too but definitely not leaving the productivity boost of an LLM behind. The thing just helps me write stuff that I was going to write anyways!
I still very much felt like I was creatively crafting this [0] project even though the entire approach used the Claude project feature. I had to hand-write some sections but for the most part I was just instructing, reading, refining, and then copying and pasting. I was the one who instructed the use of a bash parser and operating on the AST for translation between text and GUI. I was the one who instructed the use of a plugin architecture to enforce decoupling. I was the one who suggested every feature and the look of the GUI. The goal was to create an experimental UI for creating and analyzing bash pipelines. The goal was not to do a lot of typing!
These high level abstractions are where I find the most joy from programming. Perhaps for some there is still some modicum of enjoyment from writing a for loop but for most people twenty years into a career there's nothing but the feeling of grinding out the minutia.
There's still a lot of room for better abstractions when it comes to interfacing with computing devices. I'd love to write my own operating system, CLI interface, terminal, and scripting language, etc from scratch and to my own personal preferences. I don't imagine I could ever have the time to handcraft such a vast undertaking. I do imagine that within a few decades I will be able to guide a computing assistant through the entire process and with great joy!
For me LLMs are like programming power tools. Use them wrong and you can hurt yourself. Use them right and you can accomplish far more in the same amount of time.
People that refuse to program with AI or intellisense or any other assistance are like carpenters who refuse to build furniture with power saws and power drills. Which is perfectly fine, but IMO that choice doesn't really affect the artistry of the final product
> For me LLMs are like programming power tools. Use them wrong and you can hurt yourself. Use them right and you can accomplish far more in the same amount of time.
Fun analogy because if you're especially negligent you can injure yourself so badly you'll make programming forever more difficult than it needs to be or end your career altogether - like with a tablesaw cutting off fingers.
Seeing luddism in programming is hilarious to me. Keep writing your machine code old man, we'll pick up the slack for you while you fade into obscurity.
Author needs to get into Bret Victor. Has no idea how much more fun he could be having.
Programming is a step on the way to access to the state space of information. When we get to that stage, programming will seem like a maze of syntax, that has its own idiosyncrasies that force you into corners or regions in the state space, just like any DAW plugin or 3D tool, or any tool at all that exists.
The six year old English took me more effort to read than it should. The point of language is to communicate. Write in a way that people can read.
But to the point of the post, yeah, I agree. The people using LLMs are just using it to compensate for a horrible toolkit. It doesn't help them think or break down problems that nobody has seen before. If it makes them quicker it's because they were using caveman-level tools before. I haven't typed every character in the code I produce for at least a decade at this point. Most of my time and effort is in thinking. But for braindead code monkeys, sure, anything will make you quicker.
I have wondered whether one day people might attempt to prove their "realness" by writing in an intentionally stupid way that an LLM could never write.
I do sometimes get the impression that there will be a generational gap in ability to code between millennials and zoomers.
We had an overdemand for devs during late ZIRP early COVID leading to bootcamps and self taught pulling a lot of untrained into the industry. Many of them have left the industry.
Add to that the whole data science bubble and it’s bursting where we had tons of degrees and job openings for sort-of-devs. Lot of those jobs are gone now too.
Don’t forget the pull of “product management” and its demise outside big tech.
Now we have hiring freezes and juniors leaning on LLMs instead of actually spending an hour trying to solve problems.
Lol, yeah I was being cheeky here but when I see so many spelling / grammar related mistakes, I wonder can they not at least put the text through Microsoft Word or something?
Do we have grammar/spelling checkers for markdown editors or whatever most are using to write their blogs?
I have noticed something recently in developer blogs. I only know it's not AI generated if it contains spelling mistakes. It's 99% not AI generated spam if it is completely uncapitalized. If the author fixed these "errors", my certainty that it's not AI generated garbage drops to 70%. For this reason, I may adopt an uncapitalized style in my own blogs, though I'm sure it will annoy many people.
What is up with these blogs that are apparently not made with the purpose of being read? Having sentences not start with uppercase letters really makes it one big mesh of letters. Certainly when they are run-on sentences only separated by commas.
Not helped by the choice of using a monospaced font. I get that it is often an aesthetic choice, but given that a blog post is written with the idea to be read, one I don't think is a particularly good one. Although the last time I made a remark about that on HN it became clear to me that a lot of people don't see the issue. Even if there are decades worth (at this point) of research that makes it clear that a sans serif font (or even a serif font on modern displays) works better for readability. ¯\_(ツ)_/¯
In this case though, the combination of the monospaced font, everything being in lowercase and the run-on sentences I really am scratching my head here.
Are you trying to get a message out there? Or are you mostly going for aesthetics?
Below you'll find the fixed version an LLM provided. I simply asked it to add uppercase letters where appropiate and split up sentences where needed. You are welcome:
---
As LLMs get better and better at writing code, more and more people, at least on twt, have started to incorporate LLMs into their workflow. Most people seem to agree that LLMs have been a game changer for coding. They praise them for how much they have improved their productivity. They also mention how much easier it is to write code. Some claim that programmers who refuse to use them are "not using them correctly" and will eventually get left behind.
In my opinion, the effectiveness of LLMs in coding at their current state is vastly overblown. Even if LLMs were as good as what avid users of them claim, I still won't see myself using it in any meaningful capacity.
## The art of programming
Programming can be broken down into two parts. The first is solving problems algorithmically, breaking problems into steps that computers can follow within some constraints, thus forming a solution to the original problem. The second is expressing the solution in a way that the computer can understand.
Both parts provide the programmers with an infinite canvas on which they can express their creativity. There are practically limitless ways to approach and solve a problem, and a practically infinite way to express a solution to the problem. Hence, programming is a form of self-expression - it is an art form. What is produced through programming is a kind of art - an art few appreciate.
## I am a programming artist
In that sense, I see myself as an artist, one that expresses his creative self through programming. I enjoy creating programming art, because only through it do I find my true self, one who has a burning passion to create and build things.
## LLM is not for me
Using LLM to write code is like asking an artist to paint for you. If you only want the end result, by all means! If you are like me who enjoy the process of painting, then why would you bother automating the fun part away? One may say, "But I am only using LLM to write code. I am still doing the problem solving myself!". To me, programming isn't complete if I don't get to express the solution in code myself. It isn't my art if I don't create it myself.
## A sad reality
It is sad to me just how much people are trying to automate away programming and delegating it to a black box that can't even count letters in a word sometimes. They are even going as far as trying to emulate a software engineer on top of the black box. Does no one not find programming fun anymore? Does no one care enough about programming to go further beyond getting things working "well enough"? Is this just another case of availability bias?
Please don't take this as a judgmental piece to anyone that I am alluding to. It's fine to not find programming enjoyable. It's fine to just want things to work. I am just disappointed at how the ones who care appear to be an ever dying breed.
It seems that they are doing it everywhere on their website, also in most of their recently started projects.
Ironically, while looking into it, I found that one of their projects seems to use generative AI to make it work.
It might be because of what you said, though the same way I asked an LLM to insert proper uppercasing I can also ask it to remove it. So it would be more for the "vibes" more than anything.
293 comments
[ 1.9 ms ] story [ 353 ms ] threadI don't see it as a "judgemental piece to anyone that i am alluring to" (I think you mean alluding) but I do think it's an honest assesment of those who are attempting to rely heavily on AI.
> please don’t take this as a judgemental piece to anyone that i am alluring to. it’s fine to not find programming enjoyable. it’s fine to just want things to work. i am just disappointed at how the ones who care appear to be an ever dying breed.
I'm not saying the author shouldn't write that, write whatever you want. But you should own what you write.
We've already did the speedrun from people styling themselves "prompt engineers" to people openly mocking people styling themselves "prompt engineers".
And now it seems we're going to be "6 months to a year away from replacing programmers" for as long as it generates more funding.
For work stuff? I couldn’t care less if the code comes from me, my colleagues or an LLM. As long as it works and it’s secure, we’ll ship it.
Folks, career != job.
We are at the cusp of creative destruction and we are only getting started. Ironically, blue collar jobs seem safe as there hasn't been a humanoid revolution and what I see in the white collar field is what blue collar workers experienced before the automation and offshoring of jobs
Adequate but not enjoyable. Lorem ipsum of visual art. Probably kills some basic graphic design jobs at the margin working on low budget projects.
Meanwhile big brands using big agencies will just incorporate it into their design process. McD Japan is a recent example. You still need a human with an eye and taste to be the editor.
But no one is reading or viewing AI art for pleasure. It’s all “that’s neat” (continues scrolling).
AI art is very much already being consumed and sold. It's not just "mill schlock" anymore. You won't even know you are consuming AI art.
he needs to prove that point
Besides, being good at programming makes it easier to deal with BS jobs that pay well, so it’s not that I suffer 40h/week.
I take it you're not using a compiler to generate machine code, then?
Scratch that, I guess you're not using a modern microprocessor to generate microcode from a higher-level instruction set either?
Wait, real programm^Wartists use a magnetised needle and a steady hand.
Programming has always been about finding the next black box that is both powerful and flexible. You might be happy with the level of abstraction you have settled on, but it's just as arbitrary as any other level.
Even the Apollo spacecraft programmers at MIT had a black box: they offloaded the weaving of core rope memory to other people. Programming is not necessarily about manually doing the repetitive stuff. In some sense, I'd argue that's antithetical to programming -- even if it makes you feel artistic!
1. Anything that is in the world when you’re born is normal and ordinary and is just a natural part of the way the world works.
2. Anything that's invented between when you’re fifteen and thirty-five is new and exciting and revolutionary and you can probably get a career in it.
3. Anything invented after you're thirty-five is against the natural order of things.”
― Douglas Adams, The Salmon of Doubt: Hitchhiking the Galaxy One Last Time
https://ai.meta.com/research/publications/meta-large-languag...
The biggest difference to me is that we have humans that claim they can explain why compilers work the way they do. But I might as well trust someone who says the same about LLMs, because honestly I have no way to verify if they speak the truth. So I am already offloading a lot of burden of proof about the systems I work on to others. And why does this ”other” need to be a human.
Yes, it can optimize things for you, but that is not its purpose.
A big feature of compilers is to find optimizations you wouldn't think of. I tried to make the point that compiled output is typically not read by humans
We have no such contract with LLMs. The comparison to compilers is highly mistaken, and feels like how the cryptocurrency folks used to compare cryptocurrency to gestures vaguely "the internet" in an attempt to appropriate legitimacy.
I take it you don't write C, C++, or any language at that level? It is very common to examine compiled programs to ensure the compiler made critical optimizations. I have done that many times, there are plenty of tools to help you do that.
Thinking about it, this depends on which differences we consider aspects of the output program, and which ones we consider trivial differences that don't count. If you say "build an RPG about dragons with a party of magic using heroes" and the LLM spits one out, you reached a level of abstraction where many choices relating to taste and feeling and atmosphere (and gameplay too) are waved aside as trivial details. You might extend the prompt to add a few more, but the whole point of creating a program this way is not to care about most of the details of the resulting experience. Those can be allowed to be generic and bland, right? Unless you care about leaving your personal touch on, say, all of them.
1) Prove Addition of natural numbers.
2) Prove two real numbers are equal.
RNNs are only TC with infinite precision and unlimited resources, once you have finite precision they are very limited.
LLMs do not have recursion at all and can't even emulate finite automations. In fact soft attention can only emulate TC_0
Feed forward networks are effectively DAGs and with soft attention, DAGs built with AND,OR, NOT, and threshold circuits.
One of the state of the art inference in code methods is bi-abduction, probably best described here.
https://fbinfer.com/docs/separation-logic-and-bi-abduction/
But this localization makes it computationally possible, and has limits.
The qualification and frame problems, combined with the very limited computational power of transformers is another lens.
LLMs being formalized doesn't solve the problem. Fine tuning and RAG can help with domain specificity, but hallucinations are a fundamental feature of LLMs, not a bug.
Either a use case accepts the LLM failure mode (competent, confident, and inevitably wrong) or another model must be found.
Gödel showed us the limits of formalization, unless we find he was wrong, that won't change.
I had just assumed that RNNs were TC, didn't think of limitation put on by bounded precision since I assumed that any bounded precision could be compensated by growing memory module.
So, after your comment, I went literature searching and I found this: https://papers.nips.cc/paper_files/paper/2021/hash/ef452c63f...
I haven't read it yet. But if it is true, then RNN would seem to be TC
> As discussed in the paper and pointed out by the reviewer, the growing memory module is non-differentiable, and so it cannot be trained directly by SGD. We acknowledge this observation.
Two stack FSA/RNN are interesting, but as of now, not usable in practice.
LLMs are not.
They seem that way, until you're tasked with getting a repeatable, idempotent build out of a non-trivial build system.
Not with todays GPU's, you would need to run it locally with special GPU settings or run it on your CPU to ensure it is deterministic.
This is entirely different from LLMs which are opaque even to their designers, and have unpredictable flaws and hallucinations, they are probability machines based on what data they have been exposed to, which means they are not a reliable way to generate programs.
Maybe one day we'll fix this, but the current generation is not very useful for programming because of this.
Maybe you can't examine what the LLM is doing, but as things get more advanced we can generate code to do things, and also have it generate executable formal proofs that the code works as advertised.
However, I don't understand your claim that an LLM acting as a programming assistant "...can’t be reasoned about or meaningfully examined/debugged."
I type something, and Copilot or whatever generates code which I can then examine directly, and choose to accept or reject. That seems much easier to reason about than what's happening inside a compiler, for example.
In practice, compilers frequently have bugs and programmers even more frequently make use of "what the compiler actually does" rather than adhering to the language specification -- to the point where the de facto spec for many languages is "what the canonical implementation does".
And the specifications change over time.
To wit, a compiler diverging from the specification is so relatively rare that people will get angry about it and demand that it be fixed, while an LLM spewing creative nonsense is so accepted and par for the course that complaining about that fact is met with a shrug and "well, what did you expect?"
disclaimer: I know next to nothing about llms. and I'm not that interested to learn about them. just asking casually.
Because parsing and writing human language in a natural way is extremely complex.
> and was it humans who generated so many?
No, it is generated using an algorithm that tries to predict the next word in human written text using the words that comes before it. It ingests basically all the text on the internet to do this, without that much text the LLM performs horribly.
To give an analogy, a carpenter might be happy with hand tools, happy with machine tools, happy with plywood, and happy with MDF. For routine jobs they may be happy to buy pre-fabbed cabinets.
But for them to employ an apprentice (AI in this example) and outsource work to them - suddenly they are no longer really acting as a carpenter, but a kind of project manager.
edit: I agree that LLMs in their current state don't really fundamentally change the game - the point I am trying to make is that it's completely understandable that everyone has their own "stop" point. Otherwise, we'd all live in IKEA mansions.
They help me with the keyboard work, not any of the actual programming.
Whether you think LLMs are spectacularly worthwhile or odious and destructive, it's crucial not to classify them as being a person instead of a software tool.
Beckett: What?
HN commenter: Well it's just like when you decided to work in French instead of English. Your art was no less because of it. Now you can use an LLM instead of French. It will be so much quicker.
The dismissive glibness of your comment makes me wonder if it's worth it trying to point out the obvious error in the analogy you're making. Compilers translate, LLMs generate. They are two completely different things.
When you write a program in a high-level language and pass it to a compiler, the compiler translates your program to machine code, yes. But when you prompt an LLM to generate code, what are you translating? You can pretend that you are "translating natural language to code" but LLMs are not translators, they're generators, and what you're really doing is providing a prefix for the generated string. You can generate strings form an LLM with an empty prefix; but try asking a compiler to compile an empty program.
>> Even the Apollo spacecraft programmers at MIT had a black box: they offloaded the weaving of core rope memory to other people.
You're referring to core rope memory:
https://en.wikipedia.org/wiki/Core_rope_memory
There is no "black box" here. Programmers created the program and handed it over to others to code it up. That's like hiring someone to type your code for you at a keyboard, following your instructions to do so. You have to stretch things very far to see this as anything like compilation.
Also, really, compilers are not black boxes. Just because most people treat them as a scary unknowable thing doesn't mean that's what they are. LLms are "black boxes" because no matter how much we peer at their weights, arrays of numerical values, there's nothing we can ... er ... glean from them. They're incomprehensible to humans. Not so the code of a compiler. Even raw binary is comprehensible, with some experience.
Is that "compiling" or "translating"? Lots of people use language to C "compilers".
LLMs are great, but they are the opposite of a compiler (in a good way)
I don't think this is true for LLMs. Their output is not deterministic (up for discussion). Their weights and the sources thereof are mostly unknown to us. We cannot really be confident that an LLM will produce correct output based on correct input.
They can't even perform basic arithmetic (which is not surprising since they operate at the syntactic level, oblivious to any semantic rules), yet people seem to think offloading more complex tasks with strict correctness requirements is a good idea. Boggles the mind tbh.
So other path forward could very well be LLMs as they can save lot of time with writing boilerplate code
When I’m calling ‘getFirstChar’ from a library, me and the author have a good understanding of what the function does based on a shared context of common solutions in the domain we’re working in.
When you ask ChatGPT to write a function that does the same, your social contract is between you and untold billions of documents that you hope the algorithm weights correctly according to your prompt (we should probably avoid programming by hope).
You could probably get around this by training on your codebase as the corpus, but until we answer all the questions about what that entails it remains, well, questionable.
I use Cursor at work, which is basically VSCode + LLM for code generation. It's a guess and check, basically. Plenty of people look up StackOverflow answers to their problem, then verify that the answer does what they want. (Some people don't verify but those people are probably not good programmers I guess.) Well, sometimes I get the LLM to complete something, then verify the code is completed is what I would have written (and correct it if not). This saves time/typing for me in the long run even if I have to correct it at times. And I don't see anything wrong with this. I'm not programming by hope, I'm just saving time.
However ... your colleagues just do the same.
We'll see how this unfolds. As for now the industry seems to be a bit stuck at this level. Big models too expensive to train for marginal gains, smaller are getting better but doesn't help this. Until some one new idea comes in how LLMs should work, we won't see the 99.95% anyway.
I find that I don't use it as much for generating code as I do for automating tedious operations. For example, moving a bunch of repeating-yourself into a function, then converting the repeating blocks into function calls. The LLM's really good at doing that quickly without requiring me to perform dozens of copy-paste operations, or a bunch of multi-cursor-fu.
Also, I don't use it to generate large blocks of code or complicated logic.
It's not that LLMs aren't deterministic, because neither are many compilers.
It's also not that LLMs produce incorrect output, because compilers do that to, sometimes.
But when a compiler produces the wrong output, it's because either (1) there's a logic error in my code, or (2) there's a logic error in the compiler†, and I can drill down and figure out what's going on (or enlist someone to help me) to fix the problem.
Let's say I tell an LLM to write a algorithm, and it produces broken code. Why didn't my prompt work? How do I fix it? Can anyone ever actually know? And what did I learn from the experience?
---
† Or I guess there could be a hardware bug. Whatever. I'm going to blame the compiler because it needs to produce bytes that work on my silicon regardless of whether the silicon makes sense.
Sure, those collections of bits tend to do exactly the same thing when executed, but that's is in some sense a subjective evaluation.
---
Szundi said in a sibling comment that I was "completely [missing] the point on purpose" by bringing up compiler determinism. I think that's fair, but it's also why I opened my post by saying "I agree [with the parent], but I want to try to define the language better." Most compilers in use today are literally not deterministic, but they are deterministic in a different sense, which is useful as a comparison point to LLMs. Well, which sense? What is the fundamental quality that makes a compiler more predictable?
I'd like to try to find the correct words, because I don't think we have them yet.
There are 2 things at play here, one is LLM with human in the loop, in which it's just a tool for programmers to do the same thing they have been doing, and the other is LLM as black box automaton. For the former, it's not a problem that the tool is undeterministic, we are double checking the results and add our manual labour anyway. The fact that a tool can fail sometimes is an unsurprising fact of engineering.
I think the criticism in this chain of comment applies more to the latter, but even it always has values to non-tech people, just like how no-code approaches are, however shitty it looks to us software enfineers.
Recently got email about gcc 14.2, they fixed some bugs in it. Can we trust it now, these could be the last bugs. But before that it was probably a bad idea to trust. No, even compiler's output requires extensive testing. Usually it's done at once, just final result of coding and compilation.
> Their output is not deterministic
yes.
> Their weights and the sources thereof are mostly unknown to us
Some of them are known. Does it make you feel better. There are too many weights, so you are not able to track its 'thinking' anyway. There are some tools which sort of show something. Still doesn't help much.
> We cannot really be confident that an LLM will produce correct output based on correct input
No, we can't. But it's so useful when it works. I'm using it regularly for small utilities and fun pictures. Even though it can give outright wrong answers for relatively simple math questions. With explanations and full confidence.
I think OP has a point, and is about guiding the design, then overall structure and dynamic of the code. Nobody expects to write in Assembler or not using libraries, but making a concious decision on the design.
I have met few programmers, but many coders. For them coding is a job, and generally they don’t care about overall architecture, algorithm efficiency, and code elegance is limited to syntax-coloring-themes in their editor. I respect their existence, but generally they are building on top off somobody else’s effort.
I imagine that machine learning powered coding will evolve to an even blacker box, than it is today: It will transform requirements to CPU instructions (or GPU instructions, netlists, ...). Why bother to follow those indirections, that are just convenience layers for those weak carbon units (urgh)?
Simultaneously, automation will likely lead to fewer skilled programmers in the future, because there will be fewer incentives to become one.
Together those effects could lead to a situation where we are condemned to just watch.
> I take it you're not using a compiler to generate machine code, then?
An LLM is much, much closer to a "black box" than a compiler ...
In fact, an LLM is pretty much a black box even to the people who created it.
LLMs provide statistically probable answers with no guarantee of correctness. It takes more time to review LLM code than it takes to write it correctly from scratch.
Using a bought brush to paint and generating a painting via a prompt are basically identical.
/s
Plus, all the parts are deterministic in this stack. Their behavior is fixed for a given input, and all the parts are interpretable, readable, verifiable and observable.
LLMs are none of that. They are stochastic probability machines, which are nondeterministic. We can't guarantee their output's correctness, and we can't fix them to guarantee correct output. They are built on tons of (unethically sourced) data, which has no correctness and quality guarantees.
Some people will love LLMs, and/or see programming as a task/burden they have to complete. Some of us love programming for the sake of it, and earn money by doing it that way, too.
So putting LLMs to the same bucket with a deterministic, task specific programming tool is both wrong, and disservice to both.
I'm also strongly against LLMs, not because of the tech, but because of how they are trained and how their shortcomings are hid and they're put forward as the "oh the savior of the woeful masses, and the silver bullet of all thy problems", and it's neither of them.
LLMs are just glorified tech demos which shows what stochastic parrots can pose as accomplishing when you feed the whole world to them.
We can (for programming, at least): run the output thru theorem prover, ensure that proof is constructive, the Curry-Howard correspondence guarantees that you can turn the output into a correct program. It doesn't guarantee that formal properties of the program correspond to the informal problem statement. But even people occasionally make such errors (a provably correct program doesn't do what we wanted it to do).
> and we can't fix them to guarantee correct output
Same thing with other systems capable of programming. That is people.
You just can't make a system that guarantees correct transformation from an informal problem statement into a formally correct implementation. "Informal" implies that there's wiggle room for interpretation.
No, it doesn't mean that current LLMs are ready to replace programmers, it also doesn't mean that ML models of 2030s will not be able to.
That sounds very ambitious. Automated theorem provers are real sticklers for complete specifications in a formal language and can't parse natural language at all, but when you generate code with an LLM all you have in terms of a specification is a natural language prompt (that's your "informal problem statement"). In that case what exactly is the prover going to prove? Not the natural language prompt it can't parse!
The best you can do if you start with a natural language specification, like an LLM prompt, is to verify that the generated program compiles, i.e. that it is correct syntactically. As to semantic correctness, there, you're on your own.
Edit: I'm not really sure whether you're talking about syntactic or semantic correctness after all. Which one do you mean?
>> You just can't make a system that guarantees correct transformation from an informal problem statement into a formally correct implementation. "Informal" implies that there's wiggle room for interpretation.
Note that in program synthesis we usually make a distinction between complete and incomplete specifications ("problem statements") not formal and informal. An incomplete specification may still be given in a formal language. And, for the record, yes, you can make a system that guarantees that an output program is formally consistent with an incomplete specification. There exist systems like that already. You can find a bit about this online if you search for "inductive program synthesis" but the subject is spread over a wide literature spanning many fields so it's not easy to get a clear idea about it. But, in general, it works and there are approaches that give you strong theoretical guarantees of semantic correctness.
Ah, I said "theorem prover", I should have said "proof verifier". What I meant is something like DeepMind's AlphaProof with an additional step of generating a formal specification from a natural language description of the problem. In this way we get a semantically correct program wrt the formal specification. But with current generation of LLMs we probably won't get anything for non-trivial problems (the LLM won't be able to generate a valid proof).
> Note that in program synthesis we usually make a distinction between complete and incomplete specifications
Program synthesis begins after you can coherently express an idea of what you want to do. And getting to this point might involve a ton of reasoning that will not go into a program synthesis pipeline. That's what I mean when I say an "informal problem statement": some brain dumps of half-baked ideas that doesn't even constitute an incomplete specification because they are self-contradictory (but you haven't noticed it yet).
LLMs can help here by trying to generate some specification based on a brain dump.
That's even more ambitious. From DeepMind's post on AlphaProof:
First, the problems were manually translated into formal mathematical language for our systems to understand.
https://deepmind.google/discover/blog/ai-solves-imo-problems...
DeepMind had to resort to this manual translation because LLMs are not reliable enough, and natural language is not precise enough, to declare a complete specification of a formal statement, like a program or a mathematical problem (as in AlphaProof) at least not easily.
I think you point that out in the rest of your comment but you say "the LLM won't be able to generate a valid proof" where I think you meant to say "a valid specification". Did I misunderstand?
>> Program synthesis begins after you can coherently express an idea of what you want to do.
That's not exactly right. There are two kinds of program synthesis. Deductive program synthesis is when you have a complete specification in a formal language and you basically translate it to another language, just like with a compiler. That's when you "coherently express an idea of what to do". Inductive program synthesis is when you have an incomplete specification, consisting of examples of program behaviour, usually in the form of example pairs of the inputs and outputs of the target program, but sometimes program traces (like debug logs), abstract syntax trees, program schemas (a kind of rich program template) etc.
Input-output examples are the simplest case. Today, if you can express your problem in terms of input-output examples there are approaches that can synthesize a program that is consistent with the examples. You don't even need to know how to write that program yourself.
What do you mean when you say "a valid specification"? There are known algorithms to check validity of a proof. How do you check that specification is valid? People are inspecting it and agree that "yes, it seems to be expressing what was intended to be expressed in the natural language", or, "no, this turn of phrase needs to be understood in a different way" or some such. Today there's no other system that can handle this kind of a task besides humans (who are fallible) and LLMs (that are much more fallible).
That is deciding that specification is valid cannot be done without human involvement. I left that part out and focused on what we can mechanistically check (that is validity of a proof).
So, no, I didn't mean "a valid specification". And, yes, I don't think that today's LLMs would be good at producing specifications that would be deemed valid by a consensus of experts.
> Today, if you can express your problem in terms of input-output examples there are approaches that can synthesize a program that is consistent with the examples
In a limited domain with agreed-upon rules of generalization? Sure. In general? No way. The problem of generalizing from a limited number of examples with no additional restrictions is ill-defined.
And the problem "generalize as an expert would do" is in the domain of AI.
This why you should write good tests, review the code, and don’t approve anything you don’t understand.
It’s not a reason to reject pretty good autocomplete, though.
Who said that? Everyone I've talked to warns about their shortcomings (including their creators) and even the platform where I use them has a warning plastered right under the input box saying "ChatGPT can make mistakes. Check important info."
Compilers are complex programs fraught with bugs. Modern microprocessors are hideously complex devices fraught with bugs. But at least we understand them in principle and practice.
LLMs are nonsense generators, you need a second device that can recognize correct programs to use them effectively. Only humans can do that all-important second part.
> Programming has always been about finding the next black box that is both powerful and flexible.
That's the opposite of programming. Programming is the art and science of developing reliable algorithms. You can treat programs as black boxes only after you're sure that they work correctly. Otherwise you're just engaged in a kind of cargo cult.
That the scary thing about the LLM fad: so many people seem so willing to abdicate their responsibility to actually think.
>> Scratch that, I guess you're not using a modern microprocessor to generate microcode from a higher-level instruction set either?
Hell, I design gate level logic -> map it to instructions -> use them in C for the very LLMs and can fully understand[0] every aspect of it (if it doesn't behave as expected, that is a bug) but I cannot fathom or predict how the LLMs behave when i use them even though I know their architecture and implementation.
[0] Admittedly I treat the tools I use during the process, like cad tools, compiler, as black boxes, however I know that if I want to or the need arises, I can debug/understand them.
Actually the opacity of the abstraction layer is the core of the issue. First we note that opacity is a measure in both the inner-workings of the 'box' and orthogonally (in context of LLMs) a measure of deterministic outcome.
Programming, it is asserted by some of us, is exactly the act of instructing a deterministic 'black' box.
Peer-coding with an LLM is the act of cajoling a mechanism to hopefully consistently produce the input to a sensible "blackbox". It is not programming, it is getting help. Now if the help was 100% reliable, we could discuss programming the helper.
The other day I had a vision of the future AI whisperer in the corporate setting. They wear capes of varied colors and possibly sport a wand. "It's an art you see".
Unlike art where you can tell if the end product is AI generated even, software is not. The end users do not care if you used AI to generate your react front-end or back-end, your employers dont care who wrote the code as long as its bug free and works like in the scope document.
Ironically, I don't see software developers writing manifestos or complaining that AI generated software isn't software or that they are stealing from them.
Likewise, software that was produced artistically may or may not be better, or more valuable, or distinguishable in any way from other software products, but if the author feels like it's art it may be art. That may be because there's no tangible reason for it to exist other than a creative endeavor, in which case maybe it's not actually software.
are you talking about games? thats not really what article author was speaking to.
Artistically produced software though could certainly be a CRUD app just as much as it could be a photo that you pulled off the shelf at Target, I'm arguing that the type of product it is doesn't necessarily have a bearing on whether it can be art or have elements of art in it.
Many people use the term in different ways to describe a kind of abstract ambiguously valuable process of creation, or the product of it. I think it's possible to interpret software as art, but it's not a necessary quality for the software to exist.
For purely artistic works of software, my opinion is that they'd pretty much serve no explicit purpose at all other than as a creative exploration of some sort, completely open to interpretation with no success or failure case. Like the webfont Candy which was made with CSS shapes, creative coding, or various kinds of digital illustrations. Exploring what you can do with ai generated imagery is surely one of them, but not necessarily so if it's just the solution to a problem.
Most things people would describe as "having an art to it" or "art" have nuanced colloquial interpretations, but it's usually just an aspect to the creation process that embodies some of these qualities. Someone could say "there's an art to sucking as bad as you do at X" and although it's meant to be figuratively derisive rather than literally artistic, it could also refer to the abstract nebulous means by which someone fails to be good. Likewise, it could mean the abstract nebulous process by which someone makes CRUD apps good/bad.
These high level abstractions are where I find the most joy from programming. Perhaps for some there is still some modicum of enjoyment from writing a for loop but for most people twenty years into a career there's nothing but the feeling of grinding out the minutia.
There's still a lot of room for better abstractions when it comes to interfacing with computing devices. I'd love to write my own operating system, CLI interface, terminal, and scripting language, etc from scratch and to my own personal preferences. I don't imagine I could ever have the time to handcraft such a vast undertaking. I do imagine that within a few decades I will be able to guide a computing assistant through the entire process and with great joy!
[0] https://github.com/williamcotton/guish
People that refuse to program with AI or intellisense or any other assistance are like carpenters who refuse to build furniture with power saws and power drills. Which is perfectly fine, but IMO that choice doesn't really affect the artistry of the final product
Fun analogy because if you're especially negligent you can injure yourself so badly you'll make programming forever more difficult than it needs to be or end your career altogether - like with a tablesaw cutting off fingers.
Author needs to get into Bret Victor. Has no idea how much more fun he could be having.
Programming is a step on the way to access to the state space of information. When we get to that stage, programming will seem like a maze of syntax, that has its own idiosyncrasies that force you into corners or regions in the state space, just like any DAW plugin or 3D tool, or any tool at all that exists.
But to the point of the post, yeah, I agree. The people using LLMs are just using it to compensate for a horrible toolkit. It doesn't help them think or break down problems that nobody has seen before. If it makes them quicker it's because they were using caveman-level tools before. I haven't typed every character in the code I produce for at least a decade at this point. Most of my time and effort is in thinking. But for braindead code monkeys, sure, anything will make you quicker.
We had an overdemand for devs during late ZIRP early COVID leading to bootcamps and self taught pulling a lot of untrained into the industry. Many of them have left the industry.
Add to that the whole data science bubble and it’s bursting where we had tons of degrees and job openings for sort-of-devs. Lot of those jobs are gone now too.
Don’t forget the pull of “product management” and its demise outside big tech.
Now we have hiring freezes and juniors leaning on LLMs instead of actually spending an hour trying to solve problems.
Interesting times.
I would never judge anyone who finds your non-capitalized, grammatical-error ridden essays seductive.
Do we have grammar/spelling checkers for markdown editors or whatever most are using to write their blogs?
Maybe the author should have had the LLM write the article while they wrote software instead. Play each to their strengths.
Not helped by the choice of using a monospaced font. I get that it is often an aesthetic choice, but given that a blog post is written with the idea to be read, one I don't think is a particularly good one. Although the last time I made a remark about that on HN it became clear to me that a lot of people don't see the issue. Even if there are decades worth (at this point) of research that makes it clear that a sans serif font (or even a serif font on modern displays) works better for readability. ¯\_(ツ)_/¯
In this case though, the combination of the monospaced font, everything being in lowercase and the run-on sentences I really am scratching my head here.
Are you trying to get a message out there? Or are you mostly going for aesthetics?
---
As LLMs get better and better at writing code, more and more people, at least on twt, have started to incorporate LLMs into their workflow. Most people seem to agree that LLMs have been a game changer for coding. They praise them for how much they have improved their productivity. They also mention how much easier it is to write code. Some claim that programmers who refuse to use them are "not using them correctly" and will eventually get left behind.
In my opinion, the effectiveness of LLMs in coding at their current state is vastly overblown. Even if LLMs were as good as what avid users of them claim, I still won't see myself using it in any meaningful capacity.
## The art of programming
Programming can be broken down into two parts. The first is solving problems algorithmically, breaking problems into steps that computers can follow within some constraints, thus forming a solution to the original problem. The second is expressing the solution in a way that the computer can understand.
Both parts provide the programmers with an infinite canvas on which they can express their creativity. There are practically limitless ways to approach and solve a problem, and a practically infinite way to express a solution to the problem. Hence, programming is a form of self-expression - it is an art form. What is produced through programming is a kind of art - an art few appreciate.
## I am a programming artist
In that sense, I see myself as an artist, one that expresses his creative self through programming. I enjoy creating programming art, because only through it do I find my true self, one who has a burning passion to create and build things.
## LLM is not for me
Using LLM to write code is like asking an artist to paint for you. If you only want the end result, by all means! If you are like me who enjoy the process of painting, then why would you bother automating the fun part away? One may say, "But I am only using LLM to write code. I am still doing the problem solving myself!". To me, programming isn't complete if I don't get to express the solution in code myself. It isn't my art if I don't create it myself.
## A sad reality
It is sad to me just how much people are trying to automate away programming and delegating it to a black box that can't even count letters in a word sometimes. They are even going as far as trying to emulate a software engineer on top of the black box. Does no one not find programming fun anymore? Does no one care enough about programming to go further beyond getting things working "well enough"? Is this just another case of availability bias?
Please don't take this as a judgmental piece to anyone that I am alluding to. It's fine to not find programming enjoyable. It's fine to just want things to work. I am just disappointed at how the ones who care appear to be an ever dying breed.
That annoying trend is infecting HN too. There’s at least a half-dozen comments like that on this thread alone.
Ironically, while looking into it, I found that one of their projects seems to use generative AI to make it work.
It might be because of what you said, though the same way I asked an LLM to insert proper uppercasing I can also ask it to remove it. So it would be more for the "vibes" more than anything.