Show HN: AI Code Rating – a 3-character disclosure of how AI was used in a repo (aicoderating.com)
Have you ever wanted to quickly determine an open source project's AI use? Me too, so I've come up with a spec that I think may help solve the issue. Maybe it'll be useful, or maybe it'll be crap, you decide!
AI Code Rating - Three simple characters that tell a reader who maintains a project, how much of its code is written by AI, and how closely a human checks the AI's work.
That's it, that's the whole idea. Repo maintainers assign a rating and save it, along with a quick blurb about how AI was used in the project, in an ACR.md file in the root of their repo.
The rating doesn't judge, it simply answers some questions at a glance.
If this sounds at all interesting to you, I'd love to hear your feedback!
Example: DomainMOD, rated "B1b" (a project I wrote and have maintained since 2010)
B = "Experienced" - Works professionally in this stack. Could write the whole project unaided.
1 = "A little" - 1–25% of the code. AI wrote pieces such as tests, boilerplate, or single functions.
b = "Reviewed" - Every AI change read by a person before merge. Tests where practical.
Full Disclosure: The ACR website itself is rated B4b. AI wrote nearly all the code and text, directed by me, and I reviewed its work.
4 comments
[ 0.29 ms ] story [ 12.7 ms ] threadBut I think that at first this might create even more confusion, since people won't be able to read it "at a glance" when they meet it first time, they would need to understand "what that even is", and that creates friction and stops adoption.
So I think for such an idea to take off, it must be as immediately obvious as possible. "A2b" is not immediately obvious. Maybe the answer is something like "use only text descriptions that are easily understandable, but also code their values with font colors, so when a person familiar with the rating system opens yet another repository, they can quickly understand the content without reading, just by seeing three font colors".
Either graded from A-E (A being excellent, E being poor) and you can break down that score from your three character disclosure.
I actually started toying with single letter ratings to start with, but there are some issues with it. Giving something like a single letter grade to a repo has to make judgement calls about AI, which is something I'm trying to avoid with ACR. Some people won't touch any AI code, and others are ok with it as long as someone knowledgeable reviews it, and a single score would have to pick one of those views for everyone. The idea is to give a few key points and let the reader decide what that means to them.
A single score also kind of hides what people may be looking for. For example, an A3a (an expert wrote most of the code with AI and checked every change) and C1e (very little AI code but nobody checked it) could end up with the same grade. A multi-character rating seemed to offer up more information at a glance.
If you want a single pass/fail, you can still get one by setting your own minimum for each part, like "Oversight b or better", and any rating either meets it or doesn't.