Does Opus 4.6 find the needle in the haystack? (georggrab.net)

A recall rate of 0% for information outside the original training data in this experiment.

„It's not magic just yet, models aren't purely performing context-based retrieval. They're leveraging their pre-training knowledge extensively. Always be careful when running retrieval experiments on items that might be well represented in the training data (read: the entire public Internet). Results for never before seen data might be much less impressive.“

This is a really important message regarding the power of LLMs.