This is super interesting, and I like the idea of verifiable artifacts that an agent can produce, i.e. notebooks for analysis, links to the source for some claims. Building for scale, it would be interesting to know how…
I mean, they basically did that to make sure that Composer 2.5 sits on the right side, and to a quick reader looks good
This is an interesting finding, but very specialised. It would also be great to get some more information about the benchmark. Is it just a collection of files with vulnerabilities, or are they hidden in a real…
It's an interesting post, but i'm a bit skeptical on their decision to report the best run for each agent, and not just the mean over the 5 runs. We have seen this as well in running benchmarks, that variance within one…
Just a question on the benchmark. It states that it is on real world code, but all the repos in the dataset are intentionally vulnerable repos right, not real world codebases that have reported vulnerabilities?
It goes a bit up and down, compared to two years ago, in my feeling, both anthropic and open ai coding models have made massive jumps. In between big releases I do feel the quality of the models varies over time. 2…
There definitely is the danger of a lot of garbage being shipped, but with both the models getting better and better, and more tools and ways of working being discovered, I believe the quality of what is being outputted…
[dead]
[flagged]
This is super interesting, and I like the idea of verifiable artifacts that an agent can produce, i.e. notebooks for analysis, links to the source for some claims. Building for scale, it would be interesting to know how…
I mean, they basically did that to make sure that Composer 2.5 sits on the right side, and to a quick reader looks good
This is an interesting finding, but very specialised. It would also be great to get some more information about the benchmark. Is it just a collection of files with vulnerabilities, or are they hidden in a real…
It's an interesting post, but i'm a bit skeptical on their decision to report the best run for each agent, and not just the mean over the 5 runs. We have seen this as well in running benchmarks, that variance within one…
Just a question on the benchmark. It states that it is on real world code, but all the repos in the dataset are intentionally vulnerable repos right, not real world codebases that have reported vulnerabilities?
It goes a bit up and down, compared to two years ago, in my feeling, both anthropic and open ai coding models have made massive jumps. In between big releases I do feel the quality of the models varies over time. 2…
There definitely is the danger of a lot of garbage being shipped, but with both the models getting better and better, and more tools and ways of working being discovered, I believe the quality of what is being outputted…
[dead]
[flagged]