LLMs produce lower quality code if they’re asked to return it as part of a structured JSON response. This seems to be true for many top models, including those with specialized support for JSON. Benchmarks show that models struggle with syntactic issues related to quoting and escaping. The benchmark results also imply a decreased capacity for solving coding problems due to the burden of JSON formatting.
doesn't asking llms to do diffs result in a similar problem? Any additional complexity due to formatting will degrade reasoning
why not have more granular versions of whole file editing, eg whole function? I find function at a time coding yields best results for me when I write code in chatcraft.org
2 comments
[ 2.7 ms ] story [ 12.9 ms ] threadwhy not have more granular versions of whole file editing, eg whole function? I find function at a time coding yields best results for me when I write code in chatcraft.org