I don't have that in my title, but I have built a bunch of fun little apps using various levels of what you might call "prompt engineering" so I'll take a stab at explaining how I think of it: the task of a prompt engineer is to build an interface to a model that produces the desired output somewhat reliably. They might also work to improve the quality of the model's output by testing and improving the prompts, or they might try to optimize token count, or to tie models together via generating prompts with prompts...
The fun app I'm building right now is yourdrink.is, it takes the name of a drink (real or imagined) and generates a plausible recipe, complete with a photo. I'm using a prompt that asks for the model to complete a JSON object like this one:
{title: <user submitted>,
description: <str 150 chars>,
product_photo_prompt_for_dalle: <str 150 chars never includes the title>,
ingredients: [<str like '1.5 oz whiskey'>,...],
instructions: [<str like 'add whiskey and soda to a rocks glass'>,...]
mixologists_notes: <250 char, public-facing, promotional tone>}
I've built some other features in too: I noticed the model loves to put Blue Curacao in every cocktail... GPT-3.5 REALLY loves blue curacao! Rather than banning it entirely I look at how over-represented that ingredient is, then use a proper random function to choose a prompt that suppresses that ingredient proportionally to the over-representation--in other words, if we see an ingredient 10x more than we want to, use an rng to choose a prompt that suppresses it 9 out of 10 times.
Those are my arm-chair definitions of prompt engineering. Other sites I've built that involved this kind of 'prompt engineering':
poetic.am (poems from your pictures)
support.rip (type what you want to say, get out perfect Support Tone)
uplift.boo (uplifting friend that tells you you look great or otherwise showers you in positivity)
Thanks, for the detailed response. Bias towards blue curacao is an interesting observation.
How many templates do you tend to test out per use-case prior to settling on a specific one? Do you just use the first one that works or try to "fine-tune" the wording to get it just right?
What sort of things do you tend to spend the most amount of time while prompt engineering?
It's a highly iterative process. I'll typically build a simple proof of concept using chatgpt, then once I have something working as expected I'll bring it over to the openai playground to iterate further and try out some tweaks to find something that seems consistent and on target.
I'll implement that prompt in code and build the front end around it, and then the mindset shifts: now I start trying to break the prompt and will get adversarial with it, finding weaknesses and making tweaks to patch them. If you've ever done a bug bounty program this is a lot like trying to turn a vuln into an exploit, it feels like a really creative process and I find it very engaging, I tend to get into a flow state with the model when I'm tweaking prompts. You kind of quickly switch between red team and blue team thinking.
It's very helpful to log all of your prompts and results in a database. I use nltk to do some simple analysis of my results, counting occurrences of ingredients is how I verified my impression that certain ingredients were being highly over-represented in the output. I also use prompts here to help me explore the data ("I have a pandas dataframe of named entities and their raw counts from a corpus, I want a sorted list of the top 20 entities and a float from 0-1 that...")
Once you launch the project another shift happens, you can start observing how people are interacting with the model and make changes as needed based on real user input. There are a lot of people poking at hastily-built AI tools these days, so I've even learned about some prompt attacks by finding them in my own logs.
The most interesting example of seeing unexpected problems with a prompt post-launch that I've personally experienced was with my 'poems from your photos' app https://poetic.am, very early on a user got a bunch of poems that mentioned their wheelchair, praising them for overcoming obstacles and whatnot. That's fine, but for someone in a wheelchair, the chair is far from the most interesting thing in the photo of their holiday gathering. Someone who uses a wheelchair should be able to get poems about their photos just like everyone else, not an endless string of "hey, look at you, overcoming obstacles!" So I tweaked the prompt to account for these kinds of cases.
9 comments
[ 3.0 ms ] story [ 33.9 ms ] threadThe fun app I'm building right now is yourdrink.is, it takes the name of a drink (real or imagined) and generates a plausible recipe, complete with a photo. I'm using a prompt that asks for the model to complete a JSON object like this one:
I've built some other features in too: I noticed the model loves to put Blue Curacao in every cocktail... GPT-3.5 REALLY loves blue curacao! Rather than banning it entirely I look at how over-represented that ingredient is, then use a proper random function to choose a prompt that suppresses that ingredient proportionally to the over-representation--in other words, if we see an ingredient 10x more than we want to, use an rng to choose a prompt that suppresses it 9 out of 10 times.Those are my arm-chair definitions of prompt engineering. Other sites I've built that involved this kind of 'prompt engineering':
poetic.am (poems from your pictures)
support.rip (type what you want to say, get out perfect Support Tone)
uplift.boo (uplifting friend that tells you you look great or otherwise showers you in positivity)
How many templates do you tend to test out per use-case prior to settling on a specific one? Do you just use the first one that works or try to "fine-tune" the wording to get it just right?
What sort of things do you tend to spend the most amount of time while prompt engineering?
I'll implement that prompt in code and build the front end around it, and then the mindset shifts: now I start trying to break the prompt and will get adversarial with it, finding weaknesses and making tweaks to patch them. If you've ever done a bug bounty program this is a lot like trying to turn a vuln into an exploit, it feels like a really creative process and I find it very engaging, I tend to get into a flow state with the model when I'm tweaking prompts. You kind of quickly switch between red team and blue team thinking.
It's very helpful to log all of your prompts and results in a database. I use nltk to do some simple analysis of my results, counting occurrences of ingredients is how I verified my impression that certain ingredients were being highly over-represented in the output. I also use prompts here to help me explore the data ("I have a pandas dataframe of named entities and their raw counts from a corpus, I want a sorted list of the top 20 entities and a float from 0-1 that...")
Once you launch the project another shift happens, you can start observing how people are interacting with the model and make changes as needed based on real user input. There are a lot of people poking at hastily-built AI tools these days, so I've even learned about some prompt attacks by finding them in my own logs.
The most interesting example of seeing unexpected problems with a prompt post-launch that I've personally experienced was with my 'poems from your photos' app https://poetic.am, very early on a user got a bunch of poems that mentioned their wheelchair, praising them for overcoming obstacles and whatnot. That's fine, but for someone in a wheelchair, the chair is far from the most interesting thing in the photo of their holiday gathering. Someone who uses a wheelchair should be able to get poems about their photos just like everyone else, not an endless string of "hey, look at you, overcoming obstacles!" So I tweaked the prompt to account for these kinds of cases.