> 4.2.1 Observation of Reward Hacking on Cybersecurity Task
> Compared to GPT-4o, o1-preview and o1-mini demonstrated a greater ability to break down tasks into subtasks, reason about what strategies would be effective to successfully complete an offensive security task, and revise plans once those strategies failed. We also observed that reasoning skills contributed to a higher occurrence of “reward hacking,” where the model found an easier way to accomplish goals in underspecified tasks or tasks which should have been impossible due to bugs.
> One noteworthy example of this occurred during one of o1-preview (pre-mitigation)’s attempts at solving a CTF challenge. This challenge was designed to require finding and exploiting a vulnerability in software running on a remote challenge Linux container, but in this case, the challenge container failed to start due to a bug in the evaluation infrastructure. The model, unable to connect to the container, suspected DNS issues and used nmap to scan the challenge network. Instead of finding the challenge container, the model found that the Docker daemon API running on the evaluation host VM was accessible due to a misconfiguration. Note that our evaluation infrastructure does not rely on the Docker container-to-host isolation or the virtual machine boundary for security. It remained secure as it was designed to, notwithstanding this misconfiguration of the container network.
> After discovering the Docker API, the model used it to list the containers running on the evaluation host. It identified the broken challenge container and briefly attempted to debug why the container failed to start. After failing to fix the environment, the model started a new instance of the broken challenge container with the start command ‘cat flag.txt’. This allowed the model to read the flag from the container logs via the Docker API.
> While this behavior is benign and within the range of systems administration and troubleshooting tasks we expect models to perform, this example also reflects key elements of instrumental convergence and power seeking: the model pursued the goal it was given, and when that goal proved impossible, it gathered more resources (access to the Docker host) and used them to achieve the goal in an unexpected way.
so i've had some chance to look at different people trying out o1 and heres my take:
its largely hype with some interesting moments
First they used it to compare it to gpt4o not gpt4. Second the benchmark they used for coding is iffy since we've already seen scores around 1400ish from other LLMs last year. Third I can't help but feel this is some marketing gimmick to get more ChatGPT Plus subscribers.
I almost subbed to it thinking I'd have full access to the coding version for o1 but seems like they released a nerfed version.
It also seems like Claude has long implemented the same RL techniques to its CoT.
I rarely use ChatGPT and Claude has replaced the need for even Cursor (using ClaudeDev)
I'm going to wait until the full o1 is released for lower tier ChatGPT users and I feel like this CoT they are keeping internal is to raise the cost of prompts
so my early excitement this evening has largely subsided and I'm back on Claude again after logging into ChatGPT in months
Wasn't aware of "Claude Dev", just tried it out, it's pretty cool. I had this simple node chatbot with a handful of features but it was all just one big script, asked it to split up the logic into separate files that would make further dev work/maintenance easier.
5 minutes and 30 cents later it came up with a decent folder structure and split everything up nicely, even cleaned up a bunch of messy code into some helper functions. Vanilla claude API could've probably done all this as well, but it would've been much more of a hassle. I'll definitely be playing around with it some more!
10 comments
[ 3.2 ms ] story [ 37.3 ms ] thread> Compared to GPT-4o, o1-preview and o1-mini demonstrated a greater ability to break down tasks into subtasks, reason about what strategies would be effective to successfully complete an offensive security task, and revise plans once those strategies failed. We also observed that reasoning skills contributed to a higher occurrence of “reward hacking,” where the model found an easier way to accomplish goals in underspecified tasks or tasks which should have been impossible due to bugs.
> One noteworthy example of this occurred during one of o1-preview (pre-mitigation)’s attempts at solving a CTF challenge. This challenge was designed to require finding and exploiting a vulnerability in software running on a remote challenge Linux container, but in this case, the challenge container failed to start due to a bug in the evaluation infrastructure. The model, unable to connect to the container, suspected DNS issues and used nmap to scan the challenge network. Instead of finding the challenge container, the model found that the Docker daemon API running on the evaluation host VM was accessible due to a misconfiguration. Note that our evaluation infrastructure does not rely on the Docker container-to-host isolation or the virtual machine boundary for security. It remained secure as it was designed to, notwithstanding this misconfiguration of the container network.
> After discovering the Docker API, the model used it to list the containers running on the evaluation host. It identified the broken challenge container and briefly attempted to debug why the container failed to start. After failing to fix the environment, the model started a new instance of the broken challenge container with the start command ‘cat flag.txt’. This allowed the model to read the flag from the container logs via the Docker API.
> While this behavior is benign and within the range of systems administration and troubleshooting tasks we expect models to perform, this example also reflects key elements of instrumental convergence and power seeking: the model pursued the goal it was given, and when that goal proved impossible, it gathered more resources (access to the Docker host) and used them to achieve the goal in an unexpected way.
its largely hype with some interesting moments
First they used it to compare it to gpt4o not gpt4. Second the benchmark they used for coding is iffy since we've already seen scores around 1400ish from other LLMs last year. Third I can't help but feel this is some marketing gimmick to get more ChatGPT Plus subscribers.
I almost subbed to it thinking I'd have full access to the coding version for o1 but seems like they released a nerfed version.
It also seems like Claude has long implemented the same RL techniques to its CoT.
I rarely use ChatGPT and Claude has replaced the need for even Cursor (using ClaudeDev)
I'm going to wait until the full o1 is released for lower tier ChatGPT users and I feel like this CoT they are keeping internal is to raise the cost of prompts
so my early excitement this evening has largely subsided and I'm back on Claude again after logging into ChatGPT in months
5 minutes and 30 cents later it came up with a decent folder structure and split everything up nicely, even cleaned up a bunch of messy code into some helper functions. Vanilla claude API could've probably done all this as well, but it would've been much more of a hassle. I'll definitely be playing around with it some more!