Ask HN: Is prompt engineering just snake oil?
Is prompt engineering a snake oil? The engineering, as the word stands so far in history is about observing clear limitations and capabilities of something and then "engineering" things around it such as "engineering a compiler" is all about knowing the underlying processor, memory and other characteristics and then engineering a solution that converts a text notation to a stream of instructions that try to strike a balanced tradeoff.
But with LLMs, no one knows their workings once they are trained. The other day, Bark model[0] for text to speech, the team itself has following to say on details:
> Below is a list of some known non-speech sounds, but we are finding more every day. Please let us know if you find patterns that work particularly well on Discord! [laughter], [laughs], [sighs]....
So that's the team themselves not knowing what their model is capable of then how come prompt engineering is any engineering at all?
205 comments
[ 2.3 ms ] story [ 228 ms ] threadJust because we don't know all the workings of something doesn't mean we can't effectively use it, or measure how well that approach of use is valuable or repeatable for similar uses.
For example, while we have a good understanding of the basic principles of magnetism, there is still much that we do not fully understand about how magnets work, and ongoing research is focused on unraveling these mysteries and expanding our understanding of this fascinating natural phenomenon.
That's not to say we don't know how to use magnets to make sound, however.
Prompt engineering, in my opinion, is a process of optimized querying of a frozen model. The approach can be augmented by using "hot" data from vector databases or other types of text storage engines. The approach to picking the right content for prompt building is not "snake oil", but based on well known processes including autocomplete, synonyms and related terms, personalization, and query expansion.
And it would work in both common meanings in current usage.
This is also one of the reasons why the problem of transparency ("but why and how does it work?") is so important in Machine Learning: allowing control.
It seems like people think that one day they will be able to give a vague prompt and generate exactly what they were looking for, which I don't believe will ever happen. It feels like there will always be some skill in navigating the model.
There is a world of difference between the needs in governance in just having to write "Those decorations where I marked the area, make them a bit more Hans Holbein as opposed to Klimt", compared to knowing patching tricks so that if you asked the engine to draw a 'T' you were not being served a 'B'.
This latter example - ask 'B', get 'T' - I have seen from Midjourney only a few hours ago, in the public showcase. I have examined several prompts and results, and very often they do not overlap but very partially - the engine is like a wild horse.
That this is not intrinsic, but just a current stage, is only part of the story: the user will want to have a tool that allows the production of what the artist had exactly in mind (not just "anything nice as /inspired/ by our suggestions"). It is the normal form of the product in its "ready for shipment" state.
I believe there will always be some amount of navigation of the latent space required, and people who don’t understand how to navigate it will struggle.
To use your analogy, even if the horse is tame you still need to know how to ride it.
Can you code? Can you illustrate? Can you build? Piece by piece, you sketch, implement, hone, finetune, improve, correct, retry, go into detail, expand... The ease of the process is relative to the tools adopted.
Of course there has to be knowledge behind it. Of course you have to know your tools. But "wild horse" and "diligent consultant", axe and scalpel require different degrees areas and modalities of competence.
A well built tool will build on the competences already natural for the professional (e.g. a graphic designer will be able to sketch and to instruct the machine in a reasonably standard language). Of course the better you know your tool, the greater the effectiveness. But currently, the systems can be strongly unreliable - they do not respect the request.
The user will have to learn to deal with a machine taking natural language as an input just like any manager has to know how to speak to personnel to obtain what is wanted. On the other hand, the implementation of "virtual personnel" which is reliable is a responsibility of the tool developers.
It seems like you agree based on parts of your comment like "A well built tool will build on the competences already natural for the professional".
Communicating to another entity (even a human) “exactly what you had in mind” (or, more to the point, so as to get them to behave so as to produce “exactly what you had in mind”) is a skill which involves understanding of the audience, understanding of the mode of communication, and understanding of the field about which you are communicating. “Prompt engineering” is just a fancy name for the particular application of that set of skills in the domain of getting desired results out of an LLM. Whether or not the terminology ends up being durable, the skill isn’t going away, as long as we are using language to communicate with models.
(Once we get the ability to just plug our brain into an AI, will their be a skill of focussing properly so that the mind-reading tech reads the right thoughts? Probably, and that will be the new “prompt engineering”.)
If anything the recent generative art advancements coming out (Firefly, MJ v5) would seem to refute your point. There was a time when diffusion models require some level of knowledge and skill to use. To get high quality output, it was in your best interest to learn about the different types of samplers, upscalers, hyper networks, textual inversions, understanding denoising etc. etc.
Now? I'd say the millions of active users on MJ would seem to prove you wrong.
As far as language models like GPT, we've already seen the average level of skill needed come down with each successive release, ChatGPT is easier to use than GPT-3 which is easier to use than GPT-2.
Furthermore, devs in the space (Stability, OpenAI, etc) could refine/train pre-processing LLM models that transform more accessible amateur prompts to more professional prompts at real-time.
Dangerous! It looks like instead of driving directly, you are tapping on the shoulders of the one at the steering wheel.
Which is useful for children, and frustrating for drivers.
In many ways, it's the difference between using a library like lodash and writing the methods yourself. You still need to understand what's being done, a library just handles more things automatically for you.
Decades later, and it looks like everyone is still working on the DWIM command...
Not that it is engineering, but there is lot of uses where it doesn't match either...
The idea of engineering being backed by solid mathematical understanding is a relatively post-calculus idea. For the longest time, we built bridges a certain way, because those are the bridges that stood. We were flying planes long before we understood flight.
At the end of the day, engineering is the cycle of "identify a pain point -> launch experiments -> observe the delta -> apply the solution". Prompt engineering meets all of those requirements and therefore, is engineering.
It is funny to hear this, because 20 years ago : "is software engineering really engineering" was a rather common phrase around STEM circles.
This isn't true. Sir George Cayley elucidated several aerodynamic principles significantly in advance of the Wright Brothers first flights. The Wrights took advantage of this knowledge in designing their propellers and planes. We have since learned a lot more about how flight works, but the guiding principles of lift, drag, and thrust predate the mechanical engineering to realize a heavier than air aircraft.
(In many countries, the word "engineer" is regulated -- you can't call yourself an engineer without professional qualifications and oversight.)
https://en.m.wikipedia.org/wiki/Regulation_and_licensure_in_...
https://en.m.wikipedia.org/wiki/Iron_Ring
But in software you can simulate and test all you want before connecting it to the real world
Software on the other hand can technically be simulated 100% accurately, barring hardware or interpreter bugs.
The engineers working on aviation software and other low-level, real-time, performance critical systems probably need to use quite a bit of maths and are closest to doing what we traditionally consider engineering.
All those teams working on web apps and advertising...yeah not really. This isn't to say what they do isn't complicated or difficult, but it's not really engineering in the classical sense insofar as it doesn't (typically) require the use of continuous mathematics and deep systems theory. If you're primarily writing systems design docs and performing calculations, you're probably doing some kind of engineering. If you're mostly writing code to spec you're just programming. It's sort of akin to a carpenter calling themselves an engineer.
But hey, this is all symptomatic of a general trend here in the states of the trivialization and devaluing of expertise; people think it's fair to identify as whatever their heart desires regardless of qualification, knowledge, or experience. A world in which people regularly accept advice from total randos on social media is not one that has any strong sense of value around knowledge or critical capacity.
What's next, Software Physician?
With all due respect, if your "engineering title" is preceded by the word salesforce or prompt, you're not an engineer.
No seriously that kind of gate keeping is just ridiculous.
Genuine question, I am not an engineer nor in any sort of role that could plausibly be considered engineering.
Building a house takes engineering (e.g. structural). It also takes carpenters and plumbers and so on. What’s fun about software is you get to do all of it on one project.
Words mean things, and if we want to consider "software engineer" as anything more than title inflation, it should mean more than just a programmer who's full of themselves.
If I shit out a webapp that's crashes if you look at it funny, that shouldn't be considered software engineering because it's a pile of shit that barely works.
If I make a webapp that actually gives useful error messages to users and is robust in the face of failure and is maintainable and someone else would want to touch my code, then we're starting to get somewhere.
Let's be honest, the main reason so many people glom on to the term "engineer" is because (as others have mentioned) the rigorous processes around licensing for other engineering disciplines keeps supply low, which attaches an inherent market value to the term. People then adopt the term without justification in the hope if nets them a pay raise. We ought to accept that it's OK not to be an engineer. Other professions are also valuable. It's also important to "gatekeep" meanings of words to a certain extent lest we think it wise to accept a language in which any word can mean anything.
This doesn't seem to apply in the realm I've been working in. We've had to do the exact opposite thing.
In small, B2B software companies you may find that accepting an uncomfortable amount of liability is usually the only way to get your foot in the door. Certain promises have to be made (very carefully!) or no business would occur at all.
For example, if we told our banking clients that "if your front-line app crashes we might have it fixed in 5-7 days" just to be safe, they likely would have little-to-no interest in utilizing our solution. We have to say things like "We will respond within 1 hour and send a polite apology letter to your CTO every time the server makes a weird noise". And, we have to "engineer" our solution so that we have a minuscule chance of hitting that promised target.
Consider that in a <10 person company that there is going to effectively be some person who has to own all of the pain that comes along with these decision. Is that person not an "engineer" by some arbitrary definition? Sure. But, for the purposes of "a professional weighing all of the pros/cons and making a decision that they intend to accept all consequences for" - I think it's the same effective practice.
Ultimately, "software engineering" is a very broad space with varying degrees of professionalism. At some ends, you will find things that approximate actual engineering. At others, you will find things that appear more academic or artistic. The context is what defines the title of "engineer" for me.
The company would have to accept it. No legal trouble for the developper.
Those are the important parts, being real or not is irrelevant. Snake-oil selling is also a very real profession.
If all the licensing to architects, doctors, engineers, lawyers, and so on, really had a big impact on quality, the world we live in would be very different and better. But that doesn't happen because licensing a profession only works if quality is upheld on all steps of the professional education chain.
A lot of software companies circumvent all of these downsides of bad quality in education by raising the bar on the hiring pipeline with a bunch of tests and so on. Does it always work? No, just like licensing doesn't. But at least it keep la the profession accessible to those who are willing to study algorithms, data structures and so on.
There's no perfect system in our world, but I am glad we don't have license requirements to develop software. I am a statistician that became a dev through self studying, training and learning on the job, and I also know many others (even real engineers) that turned to this profession without graduating in the field and contribute a ton with their specific background knowledge.
Should everyone be called engineers? Probably not as it will elude to licensing the profession, but not calling yourself one decreases chances of finding another job dramatically.
A true tragedy tbh, I'd rather if everyone called themselves devs, scientists or something less loaded than engineer.
Depends on the country and the educational system. In some countries no such sham degree is possible to be accepted for an engineering title.
Sometimes licensing is so strict that things get done anyways ignoring license requirements because there is way more demand for some services than the supply of licensed professionals. In such cases, there are so many people violating regulation that policing becomes ineffective. It's the case with some Latin American countries.
And then you get to the problem that the license is ineffective and doesn't add to anything.
There's no perfect system to uphold quality of labor across a profession, we pick our poison and deal with it.
Setting a minimum threshold for competence makes sense, but when it becomes encoded into a legal system the result frequently becomes a way to enrich incumbents.
That is, it is a whole system. And largely comes to costs and known building techniques. After all, certified engineers built "galloping gerdy."
(And I didn't even get into maintenance costs and application. )
This doesn't devalue the effort of the work crew, but it does recognize you need accredited professionals at certain parts of the process to ensure public safety.
To be a bit more pointed: causality isn't difficult here. This legislation and professional practice was deliberately designed as a result of the numerous failures observed throughout the 1900s. The cause: multiple fatalities and unreliable infrastructure let to the regulation of the engineering profession. The effect: we are safer.
Whenever this question of engineering comes up it's always bridges, airplanes, and so on, but that's such a tiny percentage of engineering. Things don't have to be high-stakes to be engineering.
But prompt engineering? Based on what knowledge exactly?
But most developers do none of these things, and many developers don't even know they exist.
But prompt engineering is just search++. You might save some time if you know a little about the underlying technology, and there's an element of creativity which is unusual for search.
But it's still not rocket surgery.
When they should, failure cost time/money
That’s engineering.
Software Engineering = Applied Math.
Both are about modelling, so you have a good idea if a solution works before you build it.
If you're gluing stuff together and hoping it's not going to fall apart too often you're not doing either.
That's precisely what the majority of "software engineers" are doing nowadays. That's what web development is. Those that are creating tools other developers use and build things using algorithms/data structures in novel ways (essentially applied math as you said), they are the real software engineers in my eyes.
Applied Physics = Applied Math.
So... different tradeoff? Sounds like engineering to me :).
(If not, then your example is one where software engineers are misusing the protected term, but as they usually do not set up shop and mislead the general public into thinking they are certified, which would be the activity the protection is designed to thwart, the protection is rarely enforced.)
That's my understanding. They don't have a PE exam like here in the US. To be P. Eng licensed, they have to graduate from an accredited engineering program and register with some board. And they do have Software Engineering programs (I know; I hired folks from these).
https://engineerscanada.ca/sites/default/files/2022-08/2022-...
There are many folks in certain provinces arguing against regulation as, they claim, it makes it difficult for workers to compete in a global market where the term is common.
You write about people who sling PHP or Javascript spaghetti?
I have engineering degree in computer science (I think it is mostly thing in eastern Europe, you have bunch of electronics, math, computer architecture to learn), I worked on automotive software which is I have to say fairly regulated. Where you have to care for things like ISO or MISRA standards in your code.
I don't mind people calling themselves "software engineer" even if they whip up front-end - but I do mind if someone calls "software engineering isn't real".
;)
I have a CS degree too and people without it are sometimes much better than me at slinging code
Actually forget people, you should realize how far this is from engineering when ChatGPT can do it better than me :) Ask ChatGPT to build a car though and we will see how far it gets.
I ain't sure what century you live in, but it's not the current one...
If I program some bits on my hard drive and now a stepper motor punches thought the side of the mount its bolted too. Or if I say "set the flaps to this pitch in this condition" and that doesn't happen, then something very real, and very bad, happens in the reality we both exist in.
Almost everything complicated that exists today uses electronic control in one way or another. Screwing up the software in that ECU is just as much as of an error as using the wrong metal in its manufacture.
If youre doing anything that physically moves something like a stepper motor i have a lot of respect for that.
> I have engineering degree in computer science
For the record, I also have bad news about our shared degree's use of the word "science".
I say all of this as a pro regulation person.
Here in France, the engineer diploma is a regulated thing, and there absolutely are software-centered engineering school who are accredited to deliver it, which is how I got mine.
It's in the "Grandes écoles" cursus, which usually follows a more classical STEM-centric cursus called "classes prépa".
As to whether "Software engineering" is a real thing, well on one hand most of it isn't very process-oriented, on the other hand there are such thing as ISO certification for software security, so it's less the individual which is regulated than it is the project in itself when it matters (like critical data-center, or handling of medical data), so it's not like there aren't any regulatory framework in the software world.
Part of the reason for that is the cost of iteration is much lower. So software engineers can afford having less rigorous process. And the tooling is getting better, with formal specification languages, type systems and static analyzer.
Rapid prototyping, microcontrollers and simulation tools brought some of the same possibilities in mechanical & electrical engineering. Even the traditionally very conservative space industry is embracing faster iteration cycles (at least, SpaceX is).
"Software on a microcontroller isn't engineering, it's just coding, but replace said software by TTL chips and PROMs and suddenly it's EE".
But how most of the sotfware done, is not really engineered.
And I think we might do better if we were more rigorous with what is being build, probably slower, but better off and lot of it could be replicated for close enough use cases.
> (In many countries, the word "engineer" is regulated -- you can't call yourself an engineer without professional qualifications and oversight.)
Those countries (at least France, Switzerland and Canada at least) also have official accredited Software Engineering degrees.
I know because I hired people who graduated from these programs!
Furthermore, I don't understand why some people have the need to regulate everything. That's not how the real world works.
But that it’s a bit further down the line. Today, like previous hype cycles, opportunistic types will cash out on the money train. And tomorrow you will do whiteboard interviews for them (just like the last cycles..)
So the relevant question for the thoughtful geek isn’t “Is crafting prompts engineering?”. No. The Q is “should I sit out this hype cycle and then end up doing leet code monkey dance for the opportunistic types who made it big by riding the hype cycle, yet again?”
Anyway, if you're trying things iteratively it becomes more a science. Engineering is more like designing stuff and building them without trying lots of things first.
Engineering takes a first-principled approach, where upon knowing those principles you can then construct abstractly without physically building first.
Is prompt engineering 'real' engineering? It seems easy enough to test whether knowledgeable, self-proclaimed prompt engineers can outperform a random person with only moderate experience requesting information from an AI in a reproducible way.
If they can't, then there is probably no engineering involved in prompt engineering, at least at present.
If they can, then it seems like they're probably not doing it with magic, so there is some set of reproducible techniques involved. At that point, would it be fair to call it engineering?
But it certainly is based on knowing the inner workings of all or as many and as much of the abstraction layers.
But who's a prompt engineer? Is anyone who understands how Tensors and Transformers work is a prompt engineer? Are prompt engineers model specific? I'm GPT-3.5 prompt engineer and we have a job openings for Alpca or StableML prompt engineers?
Lastly, if these LLMs are so intelligent and have so much deeper understanding as their emergent capability - why prompt engineering is needed in the first place?
You generally don't need prompt engineering even with average IQ humans unless one is trying to swindle someone.
Most people who call themselves software engineers have not studied engineering at all, let alone passed an engineering course.
I think there are two broad classes of benefit a real SE can provide.
The first is an understanding of how software can be proven correct, and the limitations of that vis-a-vis the real world - power failures, etc. They can help understand what portion of the software needs what level of performance, understandability, correctness, etc. Your core trade-pricing engine's needs are different than the webserver showing the status pages - both are required but the standards for both are not the same.
Second is an ability (and this is what apprenticing and is for in a real engineering career) to judge and design entire systems. We "devs" often get treated as fungible work units and tasked with various little subsystems. A professional engineer entering such a space would be required to obtain a pretty good understanding of the entirety of the company's systems and very good of what they integrate with, such that they can characterize the cost, risk, etc, of the solution space before even beginning to plan the technical solution. Engineers often tell clients that they've envisioned the wrong solution. They're trained to break down silos and integrate solutions where possible. (Not that they all succeed, but that's in the training.)
> if these LLMs are so intelligent and have so much deeper understanding as their emergent capability - why is prompt engineering needed in the first place?
Because they're language models and people are trying to solve things that aren't pure language problems with them. So you need to map the problem to something it can represent and where a solution can be formed, even if that solution may not use an LLM call. A simple example is taking a mathematical word problem and solving it. Currently the LLMs do not model math well so you'd use the LLM to separate out the clauses of the problem and turn it into variables and write an equation from it and then you'd evaluate that snippet of code for the real answer.
And then, depending on what part of what system this was, you may need to verify the parsing and other parts of the pipeline so you would need to build a chain where the results from one call are handed to other systems, maybe just back to the same LLM, with a bunch of examples, for multiple instances of more detailed checking, and then if those answers aren't the same, to another round where it tries to explain the difference and feeds that explanation into a round where you try to reprompt the initial layer and try again. Token limits often constrain how good the instructions and examples can be so you often have a few levels of prompts, nearly bulletproof ones, and shorter ones which are cheaper and allow other use of the token budget but may have more failure cases.
Building with non-deterministic tools isn't impossible, all ropes are different and yet we have rope bridges, but you have to know how they work and how they fail.
> But who's a prompt engineer? [If] I'm a GPT-3.5 prompt engineer do we need to have a job openings for Alpca or StableML prompt engineers?
No, just like architects learning different materials during their career. But all those LLMs are different and you need to experiment (scientifically) to find their characteristics in your area before building on them.
My question is - for social or political engineering, you can gain definite knowledge (spying, observing, researching, information extraction) and then devise something.
How to do the prompt engineering exactly based on what knowledge? How to formulize and document that knowledge so that it isn't just about intuition and gut feeling but rather a learnable and transferable skill?
Good question, even if I've never heard of "political engineering"
Is social engineering snake oil, too?
Most Western democracies are not that.
it's been ignored twice despite being kind of the key point:
> is social engineering snake oil, too?
>For social or political engineering, you can gain definite knowledge (spying, observing, researching, information extraction) and then devise something.
straight answers to straight questions tend to be better
In other words, you also have to understand the text that you're getting back from your prompts, and then "engineer" the prompt to fish for better results. AI itself can do this to an extent, but over-engineering is also a thing, as is the fact that AI itself does not actually know what you want.
That's all there is to it. Refine, revise, engineer - all just buzzwords that lead to the same end result: a game of ping-pong between you and the LLM.
What does matter is having stable, predictable abstractions that you can rely on, so in software engineering it would be a mistake to rely on system-dependent undefined behavior when using a language like C++. You also don't necessarily need to know what kind of low-level optimizations the compiler and linker are using to generate your executable binary, though in some cases it could be important.
With LLMs, however, it's not really clear if the same prompt always gives the same kind of output, and you can do 'regenerate response' with something like ChatGPT to see this in action.
In some sense, 'prompt engineering' can be more like 'interviewing an engineer' in that LLMs seem to provide better outputs if you start with a broad, general question and then narrow down to your specific interest over a series of questions. Helping the LLM out by defining context, asking it to expand on a specific output it generated, pointing out where it may be hallucinating etc. all appears to improve the quality of output but I don't know if this kind of iterative process is really 'prompt engineering', maybe 'prompt optimization' is a better word for it?
In other words, writing a series of prompts to an LLM that gets you the answer you need feels quite unlike writing a Python script to automate some task. You can plan the latter out from start to end, but the former is this back-and-forth process.
I'd say even in this example, there's more a sense of searching through a range of different options to achieve specific properties. This is the sense of "prompt engineering": you use skills to create prompts -- in part by searching through various alternatives -- that have specific properties.
[1] https://dictionary.cambridge.org/dictionary/english/engineer
https://mitchellh.com/writing/prompt-engineering-vs-blind-pr...
Most of what we see on Twitter or YouTube is Blind Prompting. However, it is possible to apply an engineering mindset to prompting and that is what we should call prompt engineering. Check out the article for a much more detailed framing.
Dair AI also has some nice info and resources ( with academic papers) about prompt engineering.
The article is reasonable, but also shows a big gap in tooling, as the techniques there feel closer to linting & typing then testing once you do more interesting prompts. They don't check the interesting parts..
can you elaborate a bit more on what those interesting parts are?
It could just be a limitation of computation.
It's easy to make a labeled training set for grading our homework (catching regressions, ...) in the case of classifiers, and that's basically what the blog post showed.
What about for the above qa tasks? We can ask GPT4 whether a generated A was a good answer for a Q, but that's asking it to grade itself. Likewise, in the code case, we can write unit tests for the answers. (Trick: we use the former to more quickly do the latter.) But I feel like there has to be better ways
Another: OpenAI always updates models based on use, so we have to be sure our tests are real holdout sets that never get back to them...
For example determinism in code, its required for computation and its a system's property, but generalizing a test for it is really hard. Its a property, and by knowing its true or false you can make inferences on whether a system maintains those properties, but most of this is abstracted away at lower levels and since the context can't ever be fully shared with an LLM for evaluation, nor can it automatically switch contexts when evaluation fails, this most likely will never be solveable by computers when there exists one single input that produces two separate (different) outputs, at least from what I know about automata theory and computability.
Its generally considered a class of problems that can't be solved by turing machines.
https://en.wikipedia.org/wiki/Theory_of_computation
https://medium.com/@tarcisioma/limits-of-computation-231bf28... (overview)
https://en.wikipedia.org/wiki/Undecidable_problem (crux of the problem)
$$ and demand don't change the skills?
I mean, Anakin _WAS_ good with machines.
A little tongue in cheek but it did get the noodle working
Yes, software can be engineered, but that's not what the vast majority of so-called software engineers are actually doing at their jobs. The title mostly exists to inflate the importance of a programmer with years of experience under their belt. In reality, most of them couldn't explain to you what engineering itself is, and their job primarily consists of duct-taping and building features expediently. It's like taking a carpenter whose job is to crank out barely adequate sheds for a shed company and calling them an architect.
Don't even get me started on "computer science."
Prompt engineering is a legitimate area of study, and is obviously a practice demanded by LLMs, but you gotta just ignore the "engineering" part. It's the same skill as being a good communicator. Take a room full of "software engineers", tell them "build me an app that will let me sell gadgets", and they'll do their best to build one, but chances are it won't do what you want unless you communicate with greater specificity. It's hilarious how many people think LLMs suck just because they don't do the right thing given a single shitty sentence.
React..? Golang? GUIs?! We stepped too far from The Truth (vim, ghc) and now we are all Oilers.