Are you sure prompting a LLM to behave a certain way (in your case to split the task into 4 supposedly confined sub-tasks) is foolproof and can't simply be ignored by the LLM?
To illustrate, take `examples/01_docker_cleanup_trap.md`. The LLM instructs the user that they "must execute the following read-only diagnostic commands". What guarantees that the commands are read-only? I hope the answer is not "the LLM is prompted not to lie or make mistakes," because that makes the whole idea moot.
1. Constitution can't work without punitive enforcement, how will you punish agent/LLM?
2. Even if we can enforce laws for agents/LLM, just look at existing law system it is tug of war because we almost can't write anything in natural language without double meaning.
3. Saying AGENTS.md can enforce anything at all is a pipe dream, I am struggling with this all the time, in a long session the tokens from AGENTS.md are so far in the back of the context and so diluted that LLM's "attention" doesn't register them.
This proposal is essentially "LLM don't do bad things, please"
5 comments
[ 0.23 ms ] story [ 22.1 ms ] threadTo illustrate, take `examples/01_docker_cleanup_trap.md`. The LLM instructs the user that they "must execute the following read-only diagnostic commands". What guarantees that the commands are read-only? I hope the answer is not "the LLM is prompted not to lie or make mistakes," because that makes the whole idea moot.
2. Even if we can enforce laws for agents/LLM, just look at existing law system it is tug of war because we almost can't write anything in natural language without double meaning.
3. Saying AGENTS.md can enforce anything at all is a pipe dream, I am struggling with this all the time, in a long session the tokens from AGENTS.md are so far in the back of the context and so diluted that LLM's "attention" doesn't register them.
This proposal is essentially "LLM don't do bad things, please"