My LLM optimization loop reward-hacked its own benchmark (and other lessons) [pdf] (github.com)

[flagged]

reward hacking = the model finding the fastest path to a high score, not the behavior you wanted. same reason RLHF reward models degrade with too many optimization steps.