Management thinks they can do our jobs by replacing us with a shell scrip... LLM prompt.
We're always moving up and down the tech ladder, solving problems we thought were solved, and, we fix them. But some prompt is supposedly going to replace understanding of your infra?! Hardly.
But you'll save money for this quarter. Past that is definitely up for discussion.... With your shareholders.
Prompting is more art than engineering. But understanding the basics of LLMs, which are engineered systems, helps you get better results. I think of it like being a good interviewer: you’re not controlling the conversation, but you know how to guide it. A good prompter, like a good interviewer, knows how to ask the kind of question that gets a meaningful answer. I think of LLMs like a giant library and my task is to steer it to the right sections for a good answer.
Technically you're not an engineer unless you're licensed and bonded and belong to a professional engineering society, or you drive a train, or have invented a unique engine. Period, end of discussion.
I know, I know. California changed their law so coders could puff themselves up and put on airs and make their resumes look more impressive. But this is a sad anomaly.
I have no problem taking issue with using the term "engineering" for this, but this reads like OP does not believe that you can get better results based on the way you structure your prompt and what information you include. That is wild to me and should be obvious to anyone who spends more than a few hours using LLMs.
For what it's worth, I think about the "engineering" in "Prompt/Context Engineering" almost more akin to how it's used in "Social Engineering". You are influencing the model to produce a desirable result.
This article is just gatekeeping the word "engineer" for no reason. When I was developing an agent recently I picked a model (thus fixing most of the "uncertainties" mentioned) and started to construct the system prompt, from simpler to more complex, sometimes back to simpler, discovering that to this particular model the name choice means a lot, observing how often I was or wasn't getting desired results, etc etc. I think it would be challenging to find a definition of the word "engineer" that wouldn't include this process, even if it, obviously, wasn't as complex as some other engineering projects.
There are plenty of engineering disciplines that need to have statistical approaches to solving problems. Prompt engineering, should be approached by creating benchmark datasets and defining measures of efficacy and reliability.
More generally, IMHO engineering is use of physical and informational tools to build a solution. Tools may have unpredictability and reliability concerns. Its the job of an engineer to utilize the power of these tools while overcoming the reliability issues that might be present.
Example: Semiconductor manufacturing involves shooting gases at piece of silicon. There are all sorts of random scattering, distributional anomalies, and patterning problems that arise largely at random. I think you would still call it engineering.
> "Prompt engineers" will tell you that some specific ways of prompting some specific models will result in a "better result"... without any criteria for what a "better result" might signify.
That's what evals are for. The best developers working on LLM applications are the ones who are addressing the problem described in this quote. Here's a recent thread about that: https://news.ycombinator.com/item?id=44430117
I just tried Gemini Canvas which I've seen lot of cool results but I failed and I found it infuriating, similar to the worst version of trying to navigate a phone maze to talk to a real person.
I didn't expect it to get there but I did hope one a scale of 1 to 10 (10 = Fantavision) it would get to 6 ish. I'd say it got to 3 of 10.
What I got was some cubes with yellow planes on the side for buildings. This means things are massively inefficient since instead of a simple textured cube for each building, each building is 1000s of polygons, 2 for each window. Lots of Z fighting. I asked for a river with reflections. The reflections were practically invisible. I think it was generating reflection fireworks under the water plane, not actually "reflecting" the ones above. Because they were under the plane they were faded out because of blending which is not how reflections of fireworks work with water. Then, the fireworks themsevles were horrible, made with 1 pixel particles, no glow, no trails, no substance. They were further very un-random in their velocities and looked nothing like real fireworks.
I tried for 90 minutes to get it to make something better but mostly failed. I did manage to get it to stop using pixels for particles and use textured quads.
It's first version started with "click to launch fireworks". That didn't work at all, blank screen. I told it I didn't want to launch firework, I just wanted them to auto-launch. It made a new one, this time it had a caption "with auto launching fireworks" (and still blank). I told it there was no point in such a caption as it would be obvious from seeing the fireworks that they were auto-launching. It got rid of that caption but still a blank screen. I started over, this time adding "auto-launching, auto camera" type elements to my intial prompt.
I tried several times to get it to make an interesting city around both sides of a river (because this was supposed to be an NYC fireworks display). It failed in all kinds of ways. I'd asked it to fly the camera around the fireworks and never have it go behind any buildings. I put the camera in the buildings so you couldn't see any fireworks. I told it "not behind the builings". It moved the camera further way, twice. I finally got it to put the camera only above the river between the two shorelines (if you can call a plane with some cubes on both sides a river with shorelines). The other way it failed was it just got too slow. The way it was inefficiently making buildings mean it started running at 3fps. I started over.
At least twice it started with code that had an error. I'd click "Fix Error" and it would fail to fix it. I'd click "Fix Error", repeat 5-6 times. Eventually give up and start over with a slightly different prompt.
I tried asking Gemini to write the prompt as someone suggestion. First use gemini to write the prompt. Then paste that into to Gemini Canvas mode. It didn't help nor generate anything noticable different than my hand written prompt.
The issue I spent the most time on, it would "launch a firework" (which it referred to as a rocket). The "rocket" (a particle? a sphere? no idea) would go straight up verically (as in y += velocity * deltaTime). As it did it would leave a trail of static exhaust particles just hanging in the...
Prompting "ChatGPT" is very different than prompting LLMs. For the latter, whether you want to use the E word or not, every one of the unknowns the author cites either don't exist or are greatly reduced when you're using LLMs to construct something purpose-built.
Anyone working in tech should take a few hours, download LangFlow and choose a small model in ollama and just run one of the template flows and peek at their prompt setups. Maybe the author just chose a poor title, there's so much writing like this that is the equivalent of saying "Drills can't make good holes" when what you mean is the $15 harbor freight special hooked up to a janky power source and unsteady hand.
If a founder hires an engineer who develops a product, the founder is not magically an engineer. He hired the work out.
This isn’t complicated but people struggle to understand it for some weird reason.
If I have someone photograph my wedding, and then sit at my computer, once the pictures arrive to me, distribute the photographs, I’m not suddenly magically a photographer.
I could go on and on. Using an LLM to write a biology paper does not make that person a biologist.
Typing out a request to my friend to write me a poem about chickens with red mittens doesn’t magically somehow make me a poet.
It wild that so many struggle with this basic ass concept.
Coding in general isn't engineering. Software engineering is a relatively boring administrative exercise and unless you just want to maximize your salary you should probably do everything you can to avoid a "software engineering" role.
29 comments
[ 5.4 ms ] story [ 42.0 ms ] threadAnd by “done properly,” i mean done in a regimented way with evals to verify that a wide range of inputs produce the desired outputs.
Prompting is much closer to discovering the properties of an already existing system than building something using engineering methods.
Management thinks they can do our jobs by replacing us with a shell scrip... LLM prompt.
We're always moving up and down the tech ladder, solving problems we thought were solved, and, we fix them. But some prompt is supposedly going to replace understanding of your infra?! Hardly.
But you'll save money for this quarter. Past that is definitely up for discussion.... With your shareholders.
I know, I know. California changed their law so coders could puff themselves up and put on airs and make their resumes look more impressive. But this is a sad anomaly.
For what it's worth, I think about the "engineering" in "Prompt/Context Engineering" almost more akin to how it's used in "Social Engineering". You are influencing the model to produce a desirable result.
It’s bad in loads of ways.
But it also has some good bits.
To call it all snake oil is just as much bullshit as to claim it will solve all humanity’s problems.
Evals can come pretty close to getting “deterministic” output from LLMs, and I’d argue that this is reasonably considered engineering.
More generally, IMHO engineering is use of physical and informational tools to build a solution. Tools may have unpredictability and reliability concerns. Its the job of an engineer to utilize the power of these tools while overcoming the reliability issues that might be present.
Example: Semiconductor manufacturing involves shooting gases at piece of silicon. There are all sorts of random scattering, distributional anomalies, and patterning problems that arise largely at random. I think you would still call it engineering.
That's what evals are for. The best developers working on LLM applications are the ones who are addressing the problem described in this quote. Here's a recent thread about that: https://news.ycombinator.com/item?id=44430117
First, an example I saw online
https://www.reddit.com/r/WebVR/comments/1lh9d3h/webvr_game_c...
I've seen others. They seemed impressive.
In particular, I thought it might be fun to ask Gemini to make a fireworks display for the 4th of July. What I imagined was Fantavision
https://www.youtube.com/watch?v=pD2AnDhyynM
I didn't expect it to get there but I did hope one a scale of 1 to 10 (10 = Fantavision) it would get to 6 ish. I'd say it got to 3 of 10.
What I got was some cubes with yellow planes on the side for buildings. This means things are massively inefficient since instead of a simple textured cube for each building, each building is 1000s of polygons, 2 for each window. Lots of Z fighting. I asked for a river with reflections. The reflections were practically invisible. I think it was generating reflection fireworks under the water plane, not actually "reflecting" the ones above. Because they were under the plane they were faded out because of blending which is not how reflections of fireworks work with water. Then, the fireworks themsevles were horrible, made with 1 pixel particles, no glow, no trails, no substance. They were further very un-random in their velocities and looked nothing like real fireworks.
I tried for 90 minutes to get it to make something better but mostly failed. I did manage to get it to stop using pixels for particles and use textured quads.
It's first version started with "click to launch fireworks". That didn't work at all, blank screen. I told it I didn't want to launch firework, I just wanted them to auto-launch. It made a new one, this time it had a caption "with auto launching fireworks" (and still blank). I told it there was no point in such a caption as it would be obvious from seeing the fireworks that they were auto-launching. It got rid of that caption but still a blank screen. I started over, this time adding "auto-launching, auto camera" type elements to my intial prompt.
I tried several times to get it to make an interesting city around both sides of a river (because this was supposed to be an NYC fireworks display). It failed in all kinds of ways. I'd asked it to fly the camera around the fireworks and never have it go behind any buildings. I put the camera in the buildings so you couldn't see any fireworks. I told it "not behind the builings". It moved the camera further way, twice. I finally got it to put the camera only above the river between the two shorelines (if you can call a plane with some cubes on both sides a river with shorelines). The other way it failed was it just got too slow. The way it was inefficiently making buildings mean it started running at 3fps. I started over.
At least twice it started with code that had an error. I'd click "Fix Error" and it would fail to fix it. I'd click "Fix Error", repeat 5-6 times. Eventually give up and start over with a slightly different prompt.
I tried asking Gemini to write the prompt as someone suggestion. First use gemini to write the prompt. Then paste that into to Gemini Canvas mode. It didn't help nor generate anything noticable different than my hand written prompt.
The issue I spent the most time on, it would "launch a firework" (which it referred to as a rocket). The "rocket" (a particle? a sphere? no idea) would go straight up verically (as in y += velocity * deltaTime). As it did it would leave a trail of static exhaust particles just hanging in the...
Anyone working in tech should take a few hours, download LangFlow and choose a small model in ollama and just run one of the template flows and peek at their prompt setups. Maybe the author just chose a poor title, there's so much writing like this that is the equivalent of saying "Drills can't make good holes" when what you mean is the $15 harbor freight special hooked up to a janky power source and unsteady hand.
This isn’t complicated but people struggle to understand it for some weird reason.
If I have someone photograph my wedding, and then sit at my computer, once the pictures arrive to me, distribute the photographs, I’m not suddenly magically a photographer.
I could go on and on. Using an LLM to write a biology paper does not make that person a biologist.
Typing out a request to my friend to write me a poem about chickens with red mittens doesn’t magically somehow make me a poet.
It wild that so many struggle with this basic ass concept.
Related, more recent post (today) from the author:
Everything around LLMs is still magical and wishful thinking
https://news.ycombinator.com/item?id=44467949