Better prompt caching for GPT‑6 (openai.com)

2 comments

[ 0.27 ms ] story [ 6.8 ms ] thread

Guide: https://developers.openai.com/api/docs/guides/prompt-caching

The biggest continuing limitation I see is that the cached input has to be at least 1024 tokens. This is terrible. It means a lot of good prefixes that are smaller will go uncached. The threshold should have been 128.