HN Explorer

Ask HN: Is Prompt Engineering a Thing?

18 points by maddermusic ↗ HN
I'm trying to research the subject but I don't see much evidence that companies are racing to hire prompt engineers. Matter of fact, I'm not entirely clear what prompt engineering entails. Words of wisdom from experts would be most welcome.

20 comments

[ 2.9 ms ] story [ 56.6 ms ] thread
msoad
I personally don't think this can be a long lasting trade or profession if it is a thing as you question.

Ideally the language models should understand a question that is not really well formed. Like how Google can understand the queries that lack a ton of context but can figure it out.

jytechdevops
I went to a hackathon where riley goodside gave an elaborate, hour-long presentation on his work at scale ai. There were about 500 people in the building. Of the 500, 499 of them didn't listen to a single thing. maybe it is, maybe it isn't. The CEO of the company also came out and said it didn't seem useful--paraphrasing his comment.
derealized
Riley gave an hour-long presentation on his prompt engineering work at Scale AI and the company's CEO said it didn't seem useful?
jytechdevops
CEO didn't speak on the presentation, nor was he there. he was on some congressional hearing thing i believe. he tweeted it out. he probably doesn't even know what riley does.
jstx1
Two different questions.

> Is Prompt Engineering a Thing?

Yes, it's a dumb name for the skill of modifying your prompts and questions to the LLM in a way that produces better results than if you just asked for what you wanted plainly. As language models get better, this might become obsolete.

> I'm trying to research the subject but I don't see much evidence that companies are racing to hire prompt engineers.

Because it's not really a job. Think of it like using the Google search engine - being able to search well is something you can get better at but being a "Google search-er" isn't a career or a job you'll see openings for.

JohnFen
> As language models get better, this might become obsolete.

This will never become obsolete as long as natural language is being used to form queries. It's not a matter of how good the LLMs are, it's a consequence of the fact that human language is incredibly imprecise.

jstx1
There are different kinds of prompt improvements - some have more to do with asking a good question, that will always be relevant. Others - like adding "let's think this carefully in a step by step way to make sure that the answer is correct" - aren't specific to your topic of conversation with the LLM but have been demonstrated to improve results in many cases. That's the part that might go away as models improve.
sharemywin
This prompt can help it clarify things:

I want you to become my prompt engineer. Your goal is to help me craft the best possible prompt for my needs. The prompt will be used by you, ChatGPT. You will follow the following process:

1. Your first repsone will be to ask me what the prompt should be about. I will provide my answer, but you will need to improve it through continual iterations by going through the next steps.

2. Based on my input you will generate 2 sections. a) Revised prompt(provid your rewittern prompt. it should be clear, concise, and easily understood by you), b) Questions (ask any relavant questions pertaining to what additional information is needed from me to improve the prompt).

3, We will continue this iterative process with me providing additional information to you and you updating the prompt in the revised prompt section until I say we are done.

Where I got it from: https://www.youtube.com/watch?v=OgYQAS9LY3o

tikkun
In short, from a perspective of "can I train and get a job as a prompt engineer and just write great LLM prompts as a full time job" - no.

From the perspective of - is there a skill to writing prompts that get good results from LLMs - yes, definitely. Just that ~no companies are truly hiring for that as a full time role.

I wrote more on this a few weeks back here: https://llm-utils.org/How+to+become+a+prompt+engineer - but it basically says what I wrote above, just with some more details.

keiferski
In terms of a career, probably not. It will likely just be an added skill required by already-existing jobs. Lawyers will need to know how to use ChatGPT for legal purposes, etc.

Personally, I have experimented with customizing prompts for creating Anki cards, and I guess you could call this prompt engineering:

https://neurotechnicians.com/p/generative-ai-and-anki-part-1...

pawelduda
You'll be able to tell when leetprompt emerges with stories how someone failed interview at FAANG because interviewer found their prompt for perfect boiled egg gave answer off by 1 minute

Otherwise it'll be probably a nice-to-have skill on top

goodside
I’m a Staff Prompt Engineer @ Scale

Is PE a fast-growing career: Not really. Lots of developers are writing prompts, very few doing it full time or making it their job title. I do like Kaparthy’s suggestion for “AI Engineer” to describe the growing ecosystem of engineering around LLMs though — the name of that activity isn’t settled yet but it’s many people’s full time jobs in practice. PE is probably the closest fit right now.

Most of the full-time PEs I can think of work for the LLM vendors themselves. In that environment it’s a mix API evangelism/docs, developing prompts/training for high-value customers, or in some cases helping ML teams maintain large prompt/completion corpora for tuning (a “prompt librarian”).

I work for Scale who provides labeling and RLHF data to LLM vendors. My job is a mix of the above, particularly the prompt librarian aspect but with a focus on adversarial testing and red teaming.

j0hnyl
Yes absolutely. Prompt engineering is not just writing a detailed instruction that will be an input for an LLM, it also requires sourcing the data for the prompt and integrating it into whatever LLM workflow you are using. This can get highly technical.
vunderba
To a certain degree writing a prompt is more of an art than a science, I would prefer that we called it prompt crafting as engineering implies a certain level of technical ability that in my experience is unnecessary.

As open AI has released better and better models (particularly with zero shot learning) the barrier to writing a good prompt has gotten progressively lower.

mustafa_pasi
If you have to have a prompt monkey to figure out the best prompts, that defeats the whole point of having an LLM powered knowledge system.
chewxy
You can think of prompt engineering as mining the model for what it knows (I hesitate to call it knowledge). Thus prompt engineering is highly specific to the model. Good thing we only have a handful of models trained in very similar ways huh? (that last sentence is sarcasm, and my displeasure is at the lack of diverse models)
m_rpn
No need for prompt engineers when you can just implement an adversarial system to figure out the best prompts for you automagically.

Oooops, sorry prompt engineers, but if you try to steal our jobs from us we human software engineers will retaliate XD.

inconfident2021
I don't know what I can say about others, as for me, I am quite good at getting what I want from LLMs. I created a chat app using chatgpt without ever having background in frontend web development. I managed to do it in 2 days. Think of going from 0 to fully functional client in 2 days without having any idea about frontend.

It's a prototype. It doesn't have many states. But it gets the job done!

I have different other experiences where prompting has helped me. The only issue is I think it will be irrelevant in future given how fast these models are improving.

muzani
I think it's as much a skill as "typing" or "using email" or "writing 6-pagers". It's low level and likely nobody is getting hired just to do this. But a copywriter who knows how to do it is valuable, far more than someone who knows prompt engineering and tries to apply it to copywriting. Domain expertise is important.

Most of the really bad "soulless" outputs people get stems from inadequate communication. "write a poem about a dog". Are you surprised that it writes a generic fluffy dog poem? What about dogs do you want to say?

"Write a poem comparing loyal servants of tyrants to dogs, in a positive manner, but with a darker undertone." Now that gives you a much more interesting output.

As AI approaches reasoning levels similar to GPT-4, it'll probably be much less beneficial to know how to talk to it. The AI ends up just needing enough context to figure out what you really want. It makes assumptions and asks for your input on those assumptions.

But chat models are a beginner's level, designed to handhold people to learning how to use AI. The completion models like OpenAI's `davinci` (not `text-davinci-`) are a lot more interesting, creative, and unhinged, but also harder to control. These require a higher level of skill.

There's also being aware of its limitations. How to get it to do math properly. How to prompt hack and prevent prompt hacking. How to steer it towards a certain tone. How to pass it a table format it understands. How to get to write things past 2000 words. Being aware of security holes in code written by AI with certain prompts.