How do you test your agents when the software that your agent will be working on in production is not available to developers? How do you test your agents in such cases?
I ran into a similar problem when I was working to build an agent that manages cloud infra setups.
What helped me is to look at this problem as two parts:
1. Whether the agent is making the right decision.
2. Does the real system respond the way it should be?
To test the first part, we don't need the real system. We can build a fake system with the tool interfaces, give it to the agent, and see if the agent is making the right decisions or not.
Also, we need to make sure that the fake system is not that perfect. We should inject issues like timeouts, wrong data format, not returning, or returning a half answer, to make sure that the agent works in all these different situations, not only the positive flow.
And even after this testing to make sure that it works correctly in production, I recommend you enforce a human in the loop before making any state changes. Keep a watch on these changes for some time and see if the agent is consistently doing the right thing before relaxing the controls.
Also, I would like to know whether this system that you can't reach is a third-party product or a customers environment. That answer changes a lot.
2 comments
[ 102 ms ] story [ 136 ms ] thread2. ???
3. Profit!!1
What helped me is to look at this problem as two parts: 1. Whether the agent is making the right decision. 2. Does the real system respond the way it should be?
To test the first part, we don't need the real system. We can build a fake system with the tool interfaces, give it to the agent, and see if the agent is making the right decisions or not.
Also, we need to make sure that the fake system is not that perfect. We should inject issues like timeouts, wrong data format, not returning, or returning a half answer, to make sure that the agent works in all these different situations, not only the positive flow.
And even after this testing to make sure that it works correctly in production, I recommend you enforce a human in the loop before making any state changes. Keep a watch on these changes for some time and see if the agent is consistently doing the right thing before relaxing the controls.
Also, I would like to know whether this system that you can't reach is a third-party product or a customers environment. That answer changes a lot.