I can’t help but think the effective outcome of using this is gonna be a ton of language quirk usage, mass functional chaining, and single letter variables.
Optimizing for fewest LOC is probably slightly more bad than more LOC, and both are bad for the same reason - it makes it harder for humans to interpret and understand wtf terrible decisions and tradeoffs the LLM made
> Optimizing for fewest LOC is probably slightly more bad than more LOC, and both are bad for the same reason - it makes it harder for humans to interpret and understand ...
I know agents have lowered the cost to make a website to be a few cents for tokens and a couple dollars for a domain name, but isn’t all this a little excessive for a few line long markdown file!? Why not just display the ‘skill’ itself with a few lines commenting on what it does/how it helps?
I've been using this for a couple of months and it is hit or miss. It'll make actual high quality suggestions at times. But the thing it's missing is the *actual* experience that makes its namesake persona. There's no sense of nuance of context applied.
has anyone found a good way to improve code quality? just wondering -- LOC does seem like the wrong metric, but the code LLMs write is just too verbose
So far, the best way I’ve found to improve code quality from LLMs that you’ve already produced is to read the output, throw it away, and hand-write it from scratch, usually with a totally different structure.
This scales about as well as it sounds like it would.
Multi-model review does a decent job identifying things they’re outright wrong. The resulting code still doesn’t feel elegant writ large.
If you want good output, it seems that iterating on the output is inferior to providing better input inclusive of code examples. And by the time you’ve made all the decisions that go into that, something like ponytail is superfluous.
(All that said, I have ponytail installed in most harnesses.)
I'm working on a LLM code review tool that focuses on reducing concepts instead of reducing lines. I think that's more along the lines of explaining what you want out of clean code than ponytail's approach. But I'm still playing with it so, not sure. And just because it produces code I like doesn't mean it's good code for everyone
> has anyone found a good way to improve code quality? just wondering ...
- keep prompts focused on atomic tasks.
- use expert prompting[0] when possible.
- require coding agents to verify changes.
- require coding agents to create/update unit tests with 100% coverage.
- use git to commit/revert atomic tasks manually.
- leverage planning capabilities to review instead of recover.
- consider using something like Karpathy guidelines[1].
- leverage Constraint Programming[2] concepts when
formulating prompts.
So we are building a text version of a bias aid for a random number generator? None of the output of what you are expecting from this "skills.md" is even guaranteed.
Not only the GitHub stars are clearly manipulated with bots and fake accounts, this whole "skills.md" paradigm is close to being a pseudoscientific exercise in attempting to steer LLMs but throwing huge markdown files at it.
I’m struggling to build my own evaluation bench for local models against my own (scientific coding) use cases, and realising it’s quite hard. Good coding has many dimensions and it varies depending on the need. Nothing seems to collapse cleanly to a few numbers.
Anyone played with this and have examples I can steal from?
17 comments
[ 0.17 ms ] story [ 6.2 ms ] threadOptimizing for fewest LOC is probably slightly more bad than more LOC, and both are bad for the same reason - it makes it harder for humans to interpret and understand wtf terrible decisions and tradeoffs the LLM made
Concision begets perplexity.
;-)
A genuinely simple codebase is table stakes for a project like this.
---
That's enough Claude speak for one day
This scales about as well as it sounds like it would.
Multi-model review does a decent job identifying things they’re outright wrong. The resulting code still doesn’t feel elegant writ large.
If you want good output, it seems that iterating on the output is inferior to providing better input inclusive of code examples. And by the time you’ve made all the decisions that go into that, something like ponytail is superfluous.
(All that said, I have ponytail installed in most harnesses.)
1 - https://github.com/multica-ai/andrej-karpathy-skills
2 - https://en.wikipedia.org/wiki/Constraint_programming
Not only the GitHub stars are clearly manipulated with bots and fake accounts, this whole "skills.md" paradigm is close to being a pseudoscientific exercise in attempting to steer LLMs but throwing huge markdown files at it.
I guess it does happen to everyone cause most (not all) seniors I know have had long hair at some point
Anyone played with this and have examples I can steal from?