Having built my company's MCP server, I have a deeper appreciation for how important well-defined tools can be. It is clear to me that tool availability and coordination is going to be a top (or THE top) focus in the near future, more important than the model itself.
If you ever look at how an LLM greps through a codebase, pulling small slices at a time to gather what it needs, you have a sense of how precise tools can broaden a model's capabilities. If I had today's Sonnet with tooling from 5 years in the future, I expect it would beat Fable 5.2 hands down - that's how dramatic the difference is.
Anthropic/OpenAI have really buried the lede about tooling, baking it into their harnesses and trying to force lock-in. We are still in the AOL days of the internet right now.
Benchmark for the end 2 end browser tasks…. The other options operate at a different level of abstraction where the actions space also reduce token usage and latency…
Does any of these handle the case where the site needs you to pass a bot detection that fails if the browser is remote controlled and so need a human to access the same profile via a non-remote controlled browser first?
That's what's stopping me from using most of these things instead of just having the agent drive a browser directly with lower level integrations.
In addition to regular unit tests, we use Playwright to test our Electron app and also allow agents to drive our app using Playwright via MCP to validate their work or investigate bugs.
Based on past experience of the extreme fragility of testing against the DOM, I had Claude build an "automation API" for generally interacting with our app. Instead of LLMs needing to have Playwright click buttons or manually fill in forms or things like that, it instead executes scripts against a well defined API which we have our Angular "controllers" attach themselves to on "load" and then detach from on "unload".
So LLMs use pure JavaScript via the API and don't really need to use the DOM to interact with the app, but will do snapshots of the DOM as needed when investigating any bugs or visual issues.
This has worked very well for us, the API (like the rest of our app) is in TypeScript which means that we have a whole bunch static type safety automatically preventing regressions.
A gap in this approach is that the views themselves aren't really tested, we're e.g. testing that the logic called on pressing a button does all the right things like transition the system to new screens, etc, but not that the view is correctly bound to the button. In practice though, we've very rarely found situations where we wished we had automated testing the HTML of a view.
16 comments
[ 0.20 ms ] story [ 10.3 ms ] threadHaving built my company's MCP server, I have a deeper appreciation for how important well-defined tools can be. It is clear to me that tool availability and coordination is going to be a top (or THE top) focus in the near future, more important than the model itself.
If you ever look at how an LLM greps through a codebase, pulling small slices at a time to gather what it needs, you have a sense of how precise tools can broaden a model's capabilities. If I had today's Sonnet with tooling from 5 years in the future, I expect it would beat Fable 5.2 hands down - that's how dramatic the difference is.
Anthropic/OpenAI have really buried the lede about tooling, baking it into their harnesses and trying to force lock-in. We are still in the AOL days of the internet right now.
https://github.com/browser-use/browser-harness https://github.com/anomalyco/browse https://github.com/vercel-labs/agent-browser
The answer is very likely not a MCP server. browse (anomalyco) is relatively new; but the other 2 are battle tested in production….
That's what's stopping me from using most of these things instead of just having the agent drive a browser directly with lower level integrations.
Based on past experience of the extreme fragility of testing against the DOM, I had Claude build an "automation API" for generally interacting with our app. Instead of LLMs needing to have Playwright click buttons or manually fill in forms or things like that, it instead executes scripts against a well defined API which we have our Angular "controllers" attach themselves to on "load" and then detach from on "unload".
So LLMs use pure JavaScript via the API and don't really need to use the DOM to interact with the app, but will do snapshots of the DOM as needed when investigating any bugs or visual issues.
This has worked very well for us, the API (like the rest of our app) is in TypeScript which means that we have a whole bunch static type safety automatically preventing regressions.
A gap in this approach is that the views themselves aren't really tested, we're e.g. testing that the logic called on pressing a button does all the right things like transition the system to new screens, etc, but not that the view is correctly bound to the button. In practice though, we've very rarely found situations where we wished we had automated testing the HTML of a view.