PullRequestBenchmark introduces a new standard for Large Language Models (LLMs)—reviewing pull requests with human-like discernment. Approaching full developer job automation, this benchmark mimics professional code evaluations as LLMs approve or reject PRs. If an LLM passes, it could signal the automation of developer roles.
This makes no sense whatsoever. If AI is completely replacing devs, then why would it open PRs at all? And if devs are still around, even if LLMs can review PRs, no sane dev would run a workflow where that happens because that is the safety net against AI making bone-headed mistakes -- a human checking its work.
You've brought up good point. But let's consider this: Current LLMs are capable of generating code that rivals human output. However, two major challenges persist. First, there's no infallible method to evaluate the "quality" of code produced by LLMs. Second, even if LLMs were to submit pull requests, their generated code might not always meet our trust threshold for being "good enough."
Perhaps we should reframe the issue by focusing on developing LLMs that can precisely assess whether a PR should be approved. This approach simplifies the process to:
1. Instruct the LLM to modify a codebase.
2. Have the LLM generate a PR.
3. Leverage the LLM's ability to accurately determine PR quality for approval. If it's not good enough, the process returns to step 2.
So, to put it simply, if there comes a time when LLMs can match or even exceed human judgment in deciding whether to approve or reject a PR, it probably means we've reached a stage where LLMs can (or have) replace programmers.
3 comments
[ 3.1 ms ] story [ 15.8 ms ] threadPerhaps we should reframe the issue by focusing on developing LLMs that can precisely assess whether a PR should be approved. This approach simplifies the process to:
1. Instruct the LLM to modify a codebase.
2. Have the LLM generate a PR.
3. Leverage the LLM's ability to accurately determine PR quality for approval. If it's not good enough, the process returns to step 2.
So, to put it simply, if there comes a time when LLMs can match or even exceed human judgment in deciding whether to approve or reject a PR, it probably means we've reached a stage where LLMs can (or have) replace programmers.