from the article: "we devise the
following straightforward procedure:
1. Obtain a small set of manually created
prompts for the task.
2. Expand the set of prompts with automatic
paraphrasing using a LM (e.g., GPT3) and
backtranslation (see Section 3).
3. Rank the list of prompts by perplexity (aver-
aged on a representative sample of task inputs,
e.g. 1,000).
4. Choose the k (e.g., 3) lowest perplexity
prompts.
Using this algorithm, we show empirically that it
is best to prioritize experimenting with the lowest perplexity prompts, as they perform better than manual prompts on average, and are more stable"
1 comment
[ 3.8 ms ] story [ 14.6 ms ] thread1. Obtain a small set of manually created prompts for the task.
2. Expand the set of prompts with automatic paraphrasing using a LM (e.g., GPT3) and backtranslation (see Section 3).
3. Rank the list of prompts by perplexity (aver- aged on a representative sample of task inputs, e.g. 1,000).
4. Choose the k (e.g., 3) lowest perplexity prompts.
Using this algorithm, we show empirically that it is best to prioritize experimenting with the lowest perplexity prompts, as they perform better than manual prompts on average, and are more stable"