I feel more and more we need to move to a world where it’s easy to build and maintain our own benchmarks based off our own code base. I know it’s feasible, and I nor where I work have done it, but it’s funny that we…
This is interesting but also think about how we as humans memorize things in code I don’t think “oh it’s lines 10-50 in file A that has this code I’m looking for”. It’s much looser, something like: - maybe I can’t fully…
Idk, I just ask my agent what the status of the PR is in terms of failing checks, outstanding comments, etc. But this seems fine If anything I think the place to invest a better UX is on reviewing code, even “my”…
The most glaring downside with practically using decision models is if you need the model to not only decide something but give a reason, for every case, you end up not saving money or time, but just adding an extra…
> Calibration matters I agree. What I’ve encountered though is even building enough cases for a calibration data set is hard. And so we farm out to smarter expensive models like Astra to label cases and use that as a…
The ROI on this is massive if the bounties would actually be accepted (they won’t, right?) It begs the interesting question on the future of bug bounty programs: when it’s so easy to find and solve issues there’s less…
Very cool, great work and awesome story of how you all preserved to get to this point. Are you thinking at all about security certifications to more easily be approved with, or sell subscriptions to, enterprise…
I feel more and more we need to move to a world where it’s easy to build and maintain our own benchmarks based off our own code base. I know it’s feasible, and I nor where I work have done it, but it’s funny that we…
This is interesting but also think about how we as humans memorize things in code I don’t think “oh it’s lines 10-50 in file A that has this code I’m looking for”. It’s much looser, something like: - maybe I can’t fully…
Idk, I just ask my agent what the status of the PR is in terms of failing checks, outstanding comments, etc. But this seems fine If anything I think the place to invest a better UX is on reviewing code, even “my”…
The most glaring downside with practically using decision models is if you need the model to not only decide something but give a reason, for every case, you end up not saving money or time, but just adding an extra…
> Calibration matters I agree. What I’ve encountered though is even building enough cases for a calibration data set is hard. And so we farm out to smarter expensive models like Astra to label cases and use that as a…
The ROI on this is massive if the bounties would actually be accepted (they won’t, right?) It begs the interesting question on the future of bug bounty programs: when it’s so easy to find and solve issues there’s less…
Very cool, great work and awesome story of how you all preserved to get to this point. Are you thinking at all about security certifications to more easily be approved with, or sell subscriptions to, enterprise…