The emulation space is particularly bad about this because there are a lot of semi-technical and "well meaning" users who will do anything to get their games to play better and AI gives them a way to make it seem like they are doing something useful, without being able to judge the quality of the output they are producing.
One of the projects I work on recently had a guy drop by and explain that he wanted to use Claude to clean up our backlog and he absolutely could not fathom why I kept bringing up that we would only accept PRs that reduced our work instead of increasing it. "Do you know what Opus 4.7 is?" "Why are you so close-minded?". Unfortunately it is very hard for these users to understand that the thing they are using has a bar for quality and the bugs that still slip through cannot be solved by waving a magic wand at it.
I recently just started using Claude/ChatGPT/China models for some PS3 homebrew work.
Every model seemingly falls flat in this scope of programming. The PS3 is very complex and the tooling is fairly undocumented in a lot of instances. It doesn't surprise me most of these AI PR's are nonsense.
If anyone else has attempted writing PS3 homebrew apps using AI and has refined their tooling/systems/automation please let me know how you got the agents to work for you (:
Similar for me, but regarding the Classic Macintosh APIs. The difference is that there are plenty of books, and some source code available… just not enough to stop Codex from writing subtly wrong gibberish.
I get the impression that the “10x velocity!!!!” claims still only reflect which areas have a sufficient corpus to learn from, rather than any inductive reasoning.
I have had the same result when trying similar things for graphics on modern consoles. I hear so much great stuff about AI coding but in my niche it just seems to fall flat. Even just rubber ducking around graphics and performance they sound like a beginner who has read a lot of good blogs but with no practical experience.
Why would you expect the LLMs to work for PS3 development? How much PS3 code do you think there is in the training set?
You do realize that’s actually how they work, right? They don’t understand or reason about anything, your prompt and other input is just about trying to guide where the pachinko balls fall in the output.
I don't know about homebrew(not done it since PSP times), but I work in games development and we use Claude extensively. The trick is just to feed it all the console docs and then it's pretty amazing. If you have access to PS3 docs still, just give them to Claude as part of the session, I'm sure it will improve tenfold.
It’s starting to feel like we may need to go back to the model where you need to be invited to be able to submit code or PRs. The barrier is just too low now for popular projects.
I'm curious what percentage of PRs are just the AI blindly writing code and submitting a PR without testing, and which have at least been locally tested to some degree. Any OS maintainers have any insights on this?
We've seen a few takes on this kind of issue, but the solution I liked the best was the linux "developers take full responsibility" approach. The "Assisted-by:" tag was a pretty nice touch too.
The article unfortunately feels more like a rant than a good exploration of the problem space.
If the submitter of a PR needs to take full responsibility for the code within, then the code within cannot be LLM-generated because—depending on whether you consider it an original work by the LLM or a resurrected copy of its training data—it’s either not subject to copyright or under someone else’s copyright.
(At least for any coding LLM that isn’t trained entirely on one company’s own code and also offered by that company. That sort of LLM might be able to make the regurgitation argument work for them.)
Thus any project requiring “full responsibility” by submitters may as well just ban submitters from using LLM-based tooling. That’s the tack I’ve taken for my projects, and a number of large projects have taken that stance too.
(Before someone trots out “Technical enforcement of this is impossible!” be assured that such rules are not negated by a lack of technical enforcement; after all, there’s also no way to technically enforce that you didn’t copy someone else’s code and paste it in. But by thinking a lack of technical enforcement matters, you’re outing yourself as someone who will happily violate rules if they think they won’t get caught.)
I’ve read so many stories like this that I’ve actually gotten scared of making PRs open source projects.
There’s one in particular where a feature I really wanted didn’t exist, so I forked and had Codex 5.5 assist with building the feature on my local version. It works perfectly. My life has been improved in being able to have this feature now.
Normally I’d want to share it back with the community so others can benefit as well (presumably if I wanted this feature, others probably want it too.) But…I am not pretending this is perfect, great, or even good code. I spent about an hour total on it - it works, I haven’t had any issues with it, but it’s probably slop by any hard-core engineering account. And I neither want to get attacked for submitting slop nor do I have the time to properly engineer it to be hand-coded, so the net result is that it lives on my machine alone.
Is this the right outcome? I feel guilty that I’m getting a better version of this software and others aren’t. I want to help makes others lives easier too, but I don’t want to burden the project maintainers or get yelled at for submitting slop.
It’s a reasonable when you frame it like this: the consequences of one AI-assisted addition are small… but maintainers are responsible for the codebase’s long-term quality after years of additions. The bar's higher. (Similarly, my friends like it when I host an occasional dinner party, but things would really suck at a restaurant run by nothing but my clones.)
There is nothing wrong with forking, and one man's "better" version is another's bloat. Also, making a fork rather than a PR avoids burdening maintainers.
If you look into the arcane architecture of the Playstation 3 console you quickly gain an appreciation for just how impressive the RPCS3 emulator is. PS3 is definitely one of hardest emulation targets, so it's wild they have the majority of the library working (with enhancements like upscaling and higher frame rates on many of the titles).
I guess it's nice people want to help and AI assisted coding can be fine but I can't imagine submitting a PR to a high-profile, much-revered project like that without reviewing and thoroughly testing it myself.
When I saw it treats SPE code in the same manner as shader code on GPUs that was an enlightening moment. It made so much sense to treat it like that. That move while complicated removes a lot of potential performance issues.
This is like when everyone started opening code of conduct addition PRs against every opensource repos except now a lot of these AI ones take actual effort to know it's in bad faith.
Or maybe it's worse because a lot of them aren't in bad faith they are well meaning people who just don't know or understand enough to realize they aren't being helpful.
I’m hopeful that in a year or so the models will be good enough to help productively with emulator development and that you will see a similar shift to these PRs that you did with security this spring.
Aww, PRs no longer open/welcome? Whatever will the usual suspects parrot now?
My personal schadenfreude aside, I wonder if this will follow a similar trajectory as security bug reports did recently. I'd be surprised, for a number of reasons, but the overall shape is looking awfully similar.
I just took a look at the RPCS3 PR history, and it doesn't look that bad. (Certainly worse than "no slop", but not what I'd call a flood.)
I went 10 pages back on GitHub, and the overwhelming number of PRs look like good PRs that have been merged. There's really only a single handful of rejected slop-looking PRs. (And another handful from a single user who seemingly didn't know how to use Git/GitHub and was turning local non-compiling commits into PRs somehow.)
The problem is really behavioural, not the tooling. People that do not understand, test and document their decision making in their PRs should not be submitting them, regardless of what tooling (AI or otherwise) they used to create them.
This problem existed before AI, but it is now just worse due to the spamming nature of these "contributors". It's another form of endless September where people unfamiliar with the norms of team software development are overwhelming existing project maintainers faster than maintainers can teach them the norms of behaviour.
In the end, some sort of gatekeeping mechanism is needed to avoid overwhelming maintainers, whether it's a reputation system, membership in an in-group, or something else.
what is the appeal of blindly blasting open source projects with high-volume PRs? If you're trying to help the project to accomplish something, it doesn't follow that a firehose approach is tenable, if only for the fact that reviewing the code takes time.
Isn't it simpler, in general, to just move the development off GitHub, while leaving existing repo in sync, so it can harmlessly drown in PRs from people who didn't bother to even read project contribution guidelines?
Their secret? A very rigorous contribution policy. Essentially, issues and PRs are autoclosed, and reviewed daily by the team. If its not slop, they whitelist either the issue/PR or the contributor (so their stuff isn't autoclosed next time).
We learned from the past litellm security incident that there are tons of compromised/fake GitHub stars (which was known from the past due to the star farms)
I don't see vibe coders trying to push PRs in good faith in a old emulator.
Could it be that the sheer amount of PRs and eventually a bad PR in the middle is being used to compromise repositories?
If someone doesn't understand every bit of the PR they are submitting it should not be submittable - yes it takes time, but you are expecting devs on the other end to take more time than that.
Though one plus point: a dev can ask the LLM to:
- Split a PR into logical patches
- Explain each one
From there as questions and edit and rebase each until it makes sense, because it's guaranteed that not all of it will until you do that.
38 comments
[ 141 ms ] story [ 1975 ms ] threadOne of the projects I work on recently had a guy drop by and explain that he wanted to use Claude to clean up our backlog and he absolutely could not fathom why I kept bringing up that we would only accept PRs that reduced our work instead of increasing it. "Do you know what Opus 4.7 is?" "Why are you so close-minded?". Unfortunately it is very hard for these users to understand that the thing they are using has a bar for quality and the bugs that still slip through cannot be solved by waving a magic wand at it.
Every model seemingly falls flat in this scope of programming. The PS3 is very complex and the tooling is fairly undocumented in a lot of instances. It doesn't surprise me most of these AI PR's are nonsense.
If anyone else has attempted writing PS3 homebrew apps using AI and has refined their tooling/systems/automation please let me know how you got the agents to work for you (:
I get the impression that the “10x velocity!!!!” claims still only reflect which areas have a sufficient corpus to learn from, rather than any inductive reasoning.
You do realize that’s actually how they work, right? They don’t understand or reason about anything, your prompt and other input is just about trying to guide where the pachinko balls fall in the output.
The article unfortunately feels more like a rant than a good exploration of the problem space.
(At least for any coding LLM that isn’t trained entirely on one company’s own code and also offered by that company. That sort of LLM might be able to make the regurgitation argument work for them.)
Thus any project requiring “full responsibility” by submitters may as well just ban submitters from using LLM-based tooling. That’s the tack I’ve taken for my projects, and a number of large projects have taken that stance too.
(Before someone trots out “Technical enforcement of this is impossible!” be assured that such rules are not negated by a lack of technical enforcement; after all, there’s also no way to technically enforce that you didn’t copy someone else’s code and paste it in. But by thinking a lack of technical enforcement matters, you’re outing yourself as someone who will happily violate rules if they think they won’t get caught.)
There’s one in particular where a feature I really wanted didn’t exist, so I forked and had Codex 5.5 assist with building the feature on my local version. It works perfectly. My life has been improved in being able to have this feature now.
Normally I’d want to share it back with the community so others can benefit as well (presumably if I wanted this feature, others probably want it too.) But…I am not pretending this is perfect, great, or even good code. I spent about an hour total on it - it works, I haven’t had any issues with it, but it’s probably slop by any hard-core engineering account. And I neither want to get attacked for submitting slop nor do I have the time to properly engineer it to be hand-coded, so the net result is that it lives on my machine alone.
Is this the right outcome? I feel guilty that I’m getting a better version of this software and others aren’t. I want to help makes others lives easier too, but I don’t want to burden the project maintainers or get yelled at for submitting slop.
What’s the future look like here?
I guess it's nice people want to help and AI assisted coding can be fine but I can't imagine submitting a PR to a high-profile, much-revered project like that without reviewing and thoroughly testing it myself.
Or maybe it's worse because a lot of them aren't in bad faith they are well meaning people who just don't know or understand enough to realize they aren't being helpful.
My personal schadenfreude aside, I wonder if this will follow a similar trajectory as security bug reports did recently. I'd be surprised, for a number of reasons, but the overall shape is looking awfully similar.
I went 10 pages back on GitHub, and the overwhelming number of PRs look like good PRs that have been merged. There's really only a single handful of rejected slop-looking PRs. (And another handful from a single user who seemingly didn't know how to use Git/GitHub and was turning local non-compiling commits into PRs somehow.)
This problem existed before AI, but it is now just worse due to the spamming nature of these "contributors". It's another form of endless September where people unfamiliar with the norms of team software development are overwhelming existing project maintainers faster than maintainers can teach them the norms of behaviour.
In the end, some sort of gatekeeping mechanism is needed to avoid overwhelming maintainers, whether it's a reputation system, membership in an in-group, or something else.
The tooling is telling laymen that they built wonderful things that definitely work and perfectly fix and add features.
The tooling gasses them up and is simply wrong in these cases.
If your tool regularly lies, gaslights and produces wrong results, that's a tooling issue.
-----
OpenCode
4.9k issues 1.7k PRs 158k stars
https://github.com/anomalyco/opencode
-----
Pi
31 issues 4 PRs 47k stars
https://github.com/badlogic/pi-mono
Their secret? A very rigorous contribution policy. Essentially, issues and PRs are autoclosed, and reviewed daily by the team. If its not slop, they whitelist either the issue/PR or the contributor (so their stuff isn't autoclosed next time).
https://github.com/badlogic/pi-mono/blob/main/CONTRIBUTING.m...
GitHub needs an issue / PR approval flow.
I don't see vibe coders trying to push PRs in good faith in a old emulator. Could it be that the sheer amount of PRs and eventually a bad PR in the middle is being used to compromise repositories?
Though one plus point: a dev can ask the LLM to:
- Split a PR into logical patches - Explain each one
From there as questions and edit and rebase each until it makes sense, because it's guaranteed that not all of it will until you do that.