Nice. Check out https://ghuntley.com/ralph to learn more about Ralph. It's currently building a Gen-Z esoteric programming language and porting the standard library from Go to the Cursed programming language. The compiler is working, I'm just finishing up the touches of the standard library before launching.
I’ve done a few ports like this with Claude Code (but not with a while loop) and it did work amazingly well. The original codebase had a good test suite, so I had it port the test suite first, and gave it some code style guidance up front. Then the agent did remarkably well at doing a straight port from one imperative language to another. Then there’s some purely human work to get it really done — 80-90% done sounds about right.
If we actually want stuff that works, we need to come up with a new process. If we get "almost" good code from a single invocation, you just going to get a lot of almost good code from a loop. What we likely need is a Cucumberesque format with example tables for requirements that we can distill an AI to use. It will build the tests and then build the code to to pass the tests.
There will be a a new kind of job for software engineers, sort of like a cross between working with legacy code and toxic site cleanup.
Like back in the day being brought in to “just fix” a amalgam of FoxPro-, Excel-, and Access-based ERP that “mostly works” and only “occasionally corrupts all our data” that ambitious sales people put together over last 5 years.
But worse - because “ambitious sales people” will no longer be constrained by sandboxes of Excel or Access - they will ship multi-cloud edge-deployed kubernetes micro-services wired with Kafka, and it will be harder to find someone to talk to understand what they were trying to do at the time.
Im really curious about what other jobs will pop up. As long as there is an element of probability associated with AI, there will need to be manual supervision for certain tasks/jobs.
> and it will be harder to find someone to talk to understand what they were trying to do at the time.
This will be the big counter to AI generated tools; at one point they become black boxes and the only thing people can do is to try and fix them or replace them altogether.
Of course, in theory, AI tooling will only improve; today's vibe coded software that in some cases generate revenue can be fed into the models of the future and improved upon. In theory.
Personally, I hate it; I don't like magic or black boxes.
> it will be harder to find someone to talk to understand what they were trying to do at the time.
These are my favorite types of code bases to work on. The source of truth is the code. You have to read it and debug it to figure it out, and reconcile the actual behaviors with the desired or expected behaviors through your own product oriented thinking
A big part of the reason that people develop solutions in Excel is that they don’t have to ask anyone’s permission. No business case, no scope, no plan, and most importantly no budget.
Unless a business allows any old employee to spin up cloud services on a whim we’re not going to see sales people spinning up containers and pipelines, AI or not.
Unless they have a linux with some libre office, I fail to see where there is no budget for Excel. Initially you have to keep up with windows licenses then office.
>it will be harder to find someone to talk to understand what they were trying to do at the time.
IMHO, there's a strong case for the opposite. My vibe coding prompts are along the lines of "Please implement the plan described in `phase1-epic.md` using `specification.prd` as a guide." The specification and epics are version controlled and a part of the project. My vibe coded software has better design documentation than most software projects I've been involved in.
Does anyone remember the websites that front page and dreamweaver used to generate from its wysiwyg editor? It was a nightmare to modify manually and convinced me to never rely on generated code.
I met a guy on the airplane the other day whose job is to vibe code for people who can't vibe code. He showed me his discord server (he paid for plane wifi), where he charges people 50$/month to be in the server and he helps them unfuck their vibe coded projects. He had around 1000 people in the server.
> After finishing the port, most of the agents settled for writing extra tests or continuously updating agent/TODO.md to clarify how "done" they were. In one instance, the agent actually used pkill to terminate itself after realizing it was stuck in an infinite loop.
Ok, now that is funny! On so many levels.
Now, for the project itself, a few thoughts:
- this was tried before, about 1.5 years ago there was a project setup to spam github with lots of "paper implementations", but it was based on gpt3.5 or 4 or something, and almost nothing worked. Their results are much better.
- surprised it worked as well as it did with simple prompts. "Probably we're overcomplicating stuff". Yeah, probably.
- weird copyright / IP questions all around. This will be a minefield.
- Lots of SaaS products are screwed. Not from this, but from this + 10 engineers in every midsized company. NIH is now justified.
Literally just read a blogpost[1] about this. Gist: The two ebb and flow in waves. "It kinda works" produces innovation, OCD hones the artifacts until it runs out of material and the cycle continues.
I would love to fix my docs with this.
I have them in the main browser-use repo.
What do you recommend that the agent does never push to main browser-use, but only to its own branch?
There are always two major results from any software development process: a change in the code and a change in cognition for the people who wrote the code (whether they did so directly or with an LLM).
Python and Typescript are elaborate formal languages that emerged from a lengthy process of development involving thousands of people around the world over many years. They are non-trivially different, and it's neat that we can port a library from one to the other quasi-automatically.
The difficulty, from an economic perspective, is that the "agent" workflow dramatically alters the cognitive demands during the initial development process. It is plain to see that the developers who prompted an LLM to generate this library will not have the same familiarity with the resulting code that they would have had they written it directly.
For some economic purposes, this altering of cognitive effort, and the dramatic diminution of its duration, probably doesn't matter.
But my hunch is that most of the economic value of code is contingent on there being a set of human beings familiar with the code in a manner that requires writing having written it directly.
Denial of this basic reality was an economic problem even before LLMs: how often did churn in a development team result in a codebase that no one could maintain, undermining the long-term prospects of a firm?
It's so much easier to build a mental model of a code base with LLMs. You just ask specific questions of a subsystem and they show files, code snippets, point out the idea, etc.
I just recently took the time to understood how the GIL works exactly in CPython, because I just asked a couple of questions about it, Claude showed me the relevant API and examples where can I find it. I looked it up in the CPython codebase and all of a sudden it clicked.
The huge difference was that it cost me MINUTES. I didn't even bother to dig in before, because I can't perfectly read C, the CPython codebase is huge and it would have taken me a really long time to understand everything.
I used the "map is not a territory" to describe this context in the article about visual programming [0]. Code is a map, territory is the mental model of the problem domain the code is supposed to be solving.
But, as other commentators mentioned, LLMs are so much better on reading large codebases, that it even invalidates the whole idea of this post (visualizing codebase in 3D in a fashion similar how I would do it in my head). Which kinda changes the game – if "comprehending" complex codebase becomes an easy task, maybe we won't need to keep developers' mental models and the code in constant sync. (it's an open question)
> It is plain to see that the developers who prompted an LLM to generate this library will not have the same familiarity with the resulting code that they would have had they written it directly
I think that's a bit too simplified. Yes, a person just blindly accepting whatever the LLM generates from their unclear prompts probably won't have much understanding or familiarity with it.
But that's not how I personally use LLMs, and I'm sure a lot of others too. Instead, I'm the designer/architect, with a strict control of exactly what I want. I may not actually have written the lines, but all the interfaces/APIs are human designed, the overall design/architecture is human designed, and since I designed it, I know enough to say I'd be familiar with it.
And if I come back to the project in 1-2 years, even if there is no document, it's trivial to spend 10-20 minutes together with an LLM to understand the codebase from every angle, just ask pointed questions, and you can rebuild your mental image quickly.
TLDR: Not everyone is a using LLMs for "vibe-coding" (blind-coding), but as an assistant sitting next to you. So my guess is that the ones who know what you need to know in order to effectively build software, will be a lot more productive. The ones who don't know that (yet?), will drown in spaghetti faster than before.
"At one point we tried “improving” the prompt with Claude’s help. It ballooned to 1,500 words. The agent immediately got slower and dumber. We went back to 103 words and it was back on track."
Isn't this the exact opposite of every other piece of advice we have gotten in a year?
Another general feedback just recently, someone said we need to generate 10 times, because one out of those will be "worth reviewing"
How can anyone be doing real engineering in such a: pick the exact needle out of the constantly churning chaos-simulation-engine that (crashes least, closest to desire, human readable, random guess)
Its not the exact opposite of what ive been reading. Basically every person claiming to have success with LLM coding that ive read have said that too long of a prompt leads to too much context which leads to the LLM diverging from working on the problem as desired.
Now I want to put one of these in a loop, give it access to some bitcoin, and tell it to come up with a viable strategy to become a billionaire within the next month.
Try actually doing it, realise how very far the outcome is from what the blog posts describe the vast majority of the time, and get dread from the state of (social) media instead.
It's not crypto. It will 100% be around for the foreseeable future. Maybe not in the form it currently exists and maybe not even at the scale it currently exists, but it's here to stay.
As developers, we're just as biased as the CEO at the top trying to hawk this stuff but in the opposite manner.
Embracing it means the software we all rely on becomes progressively worse, and our ability to understand and fix that software will decrease as well.
Embracing this also likely means we accept that our salaries will decrease, while others will lose their jobs outright.
Finally, it means we accept a world where people are now all reliant on AI trained and deployed by a select few companies to do our thinking. This is especially irksome when these companies are ran by the same people who previously ruined public discourse through social media apps, and gave a generation of children mental health issues and insecurities.
Apparently one of the lucky few who learned this special technique from Geoff just completed a $50k contract for $297. But that's not all! Geoff is generous to share the special secret prompt that unlocked this unbelievable success, if only we subscribe to his newsletter! "This free-for-life offer won't last forever!"
In one instance, the agent actually used pkill to terminate itself after realizing it was stuck in an infinite loop.
That is pretty awesome and not something I would have expected from an agent; it hints (but does not prove) that it has some awareness of its own workings.
I honestly think that partially-OSS SaaS is in for a rocky road; many popular paid or freemium tools are likely to be rewritten by AI and published as OSS with permissive licenses over the next year or two.
I also think that the same capability will largely invalidate the GPL, as people point agents at GPL software and write new software that performs the same function as OSS with more permissive licenses.
My reasoning is this: the reason that people use OSS versions of software that has restrictive licensing terms, is because it’s not worth the effort to them to rewrite.
Corporations certainly, but also individuals, will be able to use similar approaches to what these people used, and in a day or two come back to a mostly-functional (but buggy) new software package that does most of what the original did, but now you have a brand new software that you control completely and you are not beholden to or restricted by anyone.
Next time someone tries to pull an ElasticSearch license trick on AWS, AWS will just point one or a thousand agents at the source and get a brand new workalike in a week written in their language du jour, and have it fully functional in a couple of months.
Doesn’t circumvent patent or trademark issues but it’ll be hard to assert that it’s not a new work, esp. if it’s in an entirely different language.
Just something I’ve been thinking about recently, that LLM agents change the game when it comes to software licensing.
Agent-in-a-loop gets you remarkably far today already. It's not straightforward to "rip" capability even when you have the code, but we're getting closer by the week to being able to go "Project X has capability Y. Use [$approach] and port this into our project". This HAS to put a fat question mark over the viability of any SaaS that makes their code visible.
And I hired a cleaning lady, paid her £200, and when I came back, the house was clean.
The difference is that I did not write a blog post about it, nor did I got overly excited about it as if I had just discovered sliced bread, nor did I harbor any illusions that it was me who did anything of value.
Next, I will write a while loop filling my disk with files of random sizes and with random byte content inside. I will update you on the progress when I am back tomorrow. I do expect great results and a nicely filled disk!
76 comments
[ 6.6 ms ] story [ 94.2 ms ] threadThe language is called Cursed.
If we actually want stuff that works, we need to come up with a new process. If we get "almost" good code from a single invocation, you just going to get a lot of almost good code from a loop. What we likely need is a Cucumberesque format with example tables for requirements that we can distill an AI to use. It will build the tests and then build the code to to pass the tests.
Like back in the day being brought in to “just fix” a amalgam of FoxPro-, Excel-, and Access-based ERP that “mostly works” and only “occasionally corrupts all our data” that ambitious sales people put together over last 5 years.
But worse - because “ambitious sales people” will no longer be constrained by sandboxes of Excel or Access - they will ship multi-cloud edge-deployed kubernetes micro-services wired with Kafka, and it will be harder to find someone to talk to understand what they were trying to do at the time.
[0] https://x.com/PovilasKorop/status/1959590015018652141
Im really curious about what other jobs will pop up. As long as there is an element of probability associated with AI, there will need to be manual supervision for certain tasks/jobs.
This will be the big counter to AI generated tools; at one point they become black boxes and the only thing people can do is to try and fix them or replace them altogether.
Of course, in theory, AI tooling will only improve; today's vibe coded software that in some cases generate revenue can be fed into the models of the future and improved upon. In theory.
Personally, I hate it; I don't like magic or black boxes.
These are my favorite types of code bases to work on. The source of truth is the code. You have to read it and debug it to figure it out, and reconcile the actual behaviors with the desired or expected behaviors through your own product oriented thinking
Unless a business allows any old employee to spin up cloud services on a whim we’re not going to see sales people spinning up containers and pipelines, AI or not.
IMHO, there's a strong case for the opposite. My vibe coding prompts are along the lines of "Please implement the plan described in `phase1-epic.md` using `specification.prd` as a guide." The specification and epics are version controlled and a part of the project. My vibe coded software has better design documentation than most software projects I've been involved in.
Ok, now that is funny! On so many levels.
Now, for the project itself, a few thoughts:
- this was tried before, about 1.5 years ago there was a project setup to spam github with lots of "paper implementations", but it was based on gpt3.5 or 4 or something, and almost nothing worked. Their results are much better.
- surprised it worked as well as it did with simple prompts. "Probably we're overcomplicating stuff". Yeah, probably.
- weird copyright / IP questions all around. This will be a minefield.
- Lots of SaaS products are screwed. Not from this, but from this + 10 engineers in every midsized company. NIH is now justified.
"This business will get out of control. It will get out of control and we'll be lucky to live through it."
https://www.youtube.com/watch?v=YZuMe5RvxPQ&t=22s
The alexandrian solution to the halting problem.
[1]https://worksonmymachine.ai/p/safe-is-what-we-call-things-la...
Python and Typescript are elaborate formal languages that emerged from a lengthy process of development involving thousands of people around the world over many years. They are non-trivially different, and it's neat that we can port a library from one to the other quasi-automatically.
The difficulty, from an economic perspective, is that the "agent" workflow dramatically alters the cognitive demands during the initial development process. It is plain to see that the developers who prompted an LLM to generate this library will not have the same familiarity with the resulting code that they would have had they written it directly.
For some economic purposes, this altering of cognitive effort, and the dramatic diminution of its duration, probably doesn't matter.
But my hunch is that most of the economic value of code is contingent on there being a set of human beings familiar with the code in a manner that requires writing having written it directly.
Denial of this basic reality was an economic problem even before LLMs: how often did churn in a development team result in a codebase that no one could maintain, undermining the long-term prospects of a firm?
I just recently took the time to understood how the GIL works exactly in CPython, because I just asked a couple of questions about it, Claude showed me the relevant API and examples where can I find it. I looked it up in the CPython codebase and all of a sudden it clicked.
The huge difference was that it cost me MINUTES. I didn't even bother to dig in before, because I can't perfectly read C, the CPython codebase is huge and it would have taken me a really long time to understand everything.
But, as other commentators mentioned, LLMs are so much better on reading large codebases, that it even invalidates the whole idea of this post (visualizing codebase in 3D in a fashion similar how I would do it in my head). Which kinda changes the game – if "comprehending" complex codebase becomes an easy task, maybe we won't need to keep developers' mental models and the code in constant sync. (it's an open question)
[0] https://divan.dev/posts/visual_programming_go/
I think that's a bit too simplified. Yes, a person just blindly accepting whatever the LLM generates from their unclear prompts probably won't have much understanding or familiarity with it.
But that's not how I personally use LLMs, and I'm sure a lot of others too. Instead, I'm the designer/architect, with a strict control of exactly what I want. I may not actually have written the lines, but all the interfaces/APIs are human designed, the overall design/architecture is human designed, and since I designed it, I know enough to say I'd be familiar with it.
And if I come back to the project in 1-2 years, even if there is no document, it's trivial to spend 10-20 minutes together with an LLM to understand the codebase from every angle, just ask pointed questions, and you can rebuild your mental image quickly.
TLDR: Not everyone is a using LLMs for "vibe-coding" (blind-coding), but as an assistant sitting next to you. So my guess is that the ones who know what you need to know in order to effectively build software, will be a lot more productive. The ones who don't know that (yet?), will drown in spaghetti faster than before.
Isn't this the exact opposite of every other piece of advice we have gotten in a year?
Another general feedback just recently, someone said we need to generate 10 times, because one out of those will be "worth reviewing"
How can anyone be doing real engineering in such a: pick the exact needle out of the constantly churning chaos-simulation-engine that (crashes least, closest to desire, human readable, random guess)
I would be scared to run this without knowing the exact cost.
Its not a good idea to do it without a payment cap for sure, its a new way to wake up with a huge bill the next day.
It's not crypto. It will 100% be around for the foreseeable future. Maybe not in the form it currently exists and maybe not even at the scale it currently exists, but it's here to stay.
As developers, we're just as biased as the CEO at the top trying to hawk this stuff but in the opposite manner.
Embracing it means the software we all rely on becomes progressively worse, and our ability to understand and fix that software will decrease as well.
Embracing this also likely means we accept that our salaries will decrease, while others will lose their jobs outright.
Finally, it means we accept a world where people are now all reliant on AI trained and deployed by a select few companies to do our thinking. This is especially irksome when these companies are ran by the same people who previously ruined public discourse through social media apps, and gave a generation of children mental health issues and insecurities.
https://ghuntley.com/ralph/
Apparently one of the lucky few who learned this special technique from Geoff just completed a $50k contract for $297. But that's not all! Geoff is generous to share the special secret prompt that unlocked this unbelievable success, if only we subscribe to his newsletter! "This free-for-life offer won't last forever!"
I am sceptical.
That is pretty awesome and not something I would have expected from an agent; it hints (but does not prove) that it has some awareness of its own workings.
I also think that the same capability will largely invalidate the GPL, as people point agents at GPL software and write new software that performs the same function as OSS with more permissive licenses.
My reasoning is this: the reason that people use OSS versions of software that has restrictive licensing terms, is because it’s not worth the effort to them to rewrite.
Corporations certainly, but also individuals, will be able to use similar approaches to what these people used, and in a day or two come back to a mostly-functional (but buggy) new software package that does most of what the original did, but now you have a brand new software that you control completely and you are not beholden to or restricted by anyone.
Next time someone tries to pull an ElasticSearch license trick on AWS, AWS will just point one or a thousand agents at the source and get a brand new workalike in a week written in their language du jour, and have it fully functional in a couple of months.
Doesn’t circumvent patent or trademark issues but it’ll be hard to assert that it’s not a new work, esp. if it’s in an entirely different language.
Just something I’ve been thinking about recently, that LLM agents change the game when it comes to software licensing.
Agent-in-a-loop gets you remarkably far today already. It's not straightforward to "rip" capability even when you have the code, but we're getting closer by the week to being able to go "Project X has capability Y. Use [$approach] and port this into our project". This HAS to put a fat question mark over the viability of any SaaS that makes their code visible.
The difference is that I did not write a blog post about it, nor did I got overly excited about it as if I had just discovered sliced bread, nor did I harbor any illusions that it was me who did anything of value.
Next, I will write a while loop filling my disk with files of random sizes and with random byte content inside. I will update you on the progress when I am back tomorrow. I do expect great results and a nicely filled disk!