Would be interesting to see alternative scoring besides “tests pass”, e.g. diff size, abstraction depth added/removed, or whether the solution introduces new modules/dependencies. That would allow to see if…
It’d be interesting to see this compared against a human baseline — e.g., a competent engineer with a fixed time budget on the same tasks.
If both sides refactor the same function into multiple smaller ones (extract method) or rename it, can Weave detect that as a structural refactor, or does it become “delete + add”? Any heuristics beyond name matching?
One thing that seems under-discussed in this space is the shift from verifying programs to verifying generation processes. If a piece of code is produced by an agent loop (prompt -> tool calls -> edits -> tests), the…
Would be interesting to see alternative scoring besides “tests pass”, e.g. diff size, abstraction depth added/removed, or whether the solution introduces new modules/dependencies. That would allow to see if…
It’d be interesting to see this compared against a human baseline — e.g., a competent engineer with a fixed time budget on the same tasks.
If both sides refactor the same function into multiple smaller ones (extract method) or rename it, can Weave detect that as a structural refactor, or does it become “delete + add”? Any heuristics beyond name matching?
One thing that seems under-discussed in this space is the shift from verifying programs to verifying generation processes. If a piece of code is produced by an agent loop (prompt -> tool calls -> edits -> tests), the…