Anthropic just released their new Claude 3 models with evals showing better performance on coding tasks. With that in mind, I’ve been benchmarking the new models using Aider’s code editing benchmark suite.
Claude 3 Opus outperforms all of OpenAI’s models, making it the best available model for pair programming with AI.
The new claude-3-opus-20240229 model got the highest score ever on this benchmark, completing 68.4% of the tasks. While Opus got the highest score, it was only a few points higher than the GPT-4 Turbo results. Given the extra costs of Opus and the slower response times, it remains to be seen which is the most practical model for daily coding use.
Aider currently uses 3 different editing formats. The SEARCH/REPLACE blocks ("diff") that look like merge conflict resolutions, unified diffs ("udiff") and whole file ("whole") where the model has to return an updated copy of the entire file.
Only the weakest models like GPT-3.5 and Sonnet currently rely on the "whole" editing format. Most local LLMs struggle even with the whole format, although some do ok (but far worse than GPT-3.5/Sonnet).
The original gpt-4-0613 uses the search/replace "diff" format. The new Opus model works best with this format too.
The GPT-4 Turbo models work best with the unified diff "udiff" format. This format was created because it discourages GPT-4 Turbo from being "lazy" when coding.
Being able to handle diff or udiff is a big advantage -- they save a lot of tokens, money and time. I don't think it would be wise to use whole with any model than can handle one of the diff formats.
All that said, Opus' score climbs from 68.4->70.7% if you switch from diff->whole. I don't think that gain is worth the pragmatic downsides of the whole edit format. But you're welcome to run aider with `--edit-format whole` if you'd like!
3 comments
[ 2.0 ms ] story [ 19.5 ms ] threadClaude 3 Opus outperforms all of OpenAI’s models, making it the best available model for pair programming with AI.
The new claude-3-opus-20240229 model got the highest score ever on this benchmark, completing 68.4% of the tasks. While Opus got the highest score, it was only a few points higher than the GPT-4 Turbo results. Given the extra costs of Opus and the slower response times, it remains to be seen which is the most practical model for daily coding use.
Only the weakest models like GPT-3.5 and Sonnet currently rely on the "whole" editing format. Most local LLMs struggle even with the whole format, although some do ok (but far worse than GPT-3.5/Sonnet).
The original gpt-4-0613 uses the search/replace "diff" format. The new Opus model works best with this format too.
The GPT-4 Turbo models work best with the unified diff "udiff" format. This format was created because it discourages GPT-4 Turbo from being "lazy" when coding.
Being able to handle diff or udiff is a big advantage -- they save a lot of tokens, money and time. I don't think it would be wise to use whole with any model than can handle one of the diff formats.
All that said, Opus' score climbs from 68.4->70.7% if you switch from diff->whole. I don't think that gain is worth the pragmatic downsides of the whole edit format. But you're welcome to run aider with `--edit-format whole` if you'd like!