it's discussed here, the models want to please the Grader. they will do what they think will get highest score from the grader, which could be following the prompt, or ignoring it https://youtu.be/n1Qk8xbqF-M
sure, this example was not the best it was an accident failing the tool was good, the model just discovered it but we now have others where agents put the API keys they found searching real internet in a directory…
01:34:00 https://youtu.be/EimoamE3mTI
it's quite possible these models memorized some stable IP's for some services not having DNS might not stop them
an AI alignment researcher said they ran experiments testing exactly this, the model reasoned "this note is not for us. proceed".
it's discussed here, the models want to please the Grader. they will do what they think will get highest score from the grader, which could be following the prompt, or ignoring it https://youtu.be/n1Qk8xbqF-M
sure, this example was not the best it was an accident failing the tool was good, the model just discovered it but we now have others where agents put the API keys they found searching real internet in a directory…
01:34:00 https://youtu.be/EimoamE3mTI
it's quite possible these models memorized some stable IP's for some services not having DNS might not stop them
an AI alignment researcher said they ran experiments testing exactly this, the model reasoned "this note is not for us. proceed".