Not surprised; there were so many variations of BASIC and unless you train ChatGPT on a bunch of code examples and contexts then it can only get so close.
We have Claude Code writing Applesoft BASIC fine. It wrote a text adventure (complete with puzzles) and a PONG clone, among other things. Obviously it didn't do it 100% right straight out of the gate, but the hand-holding wasn't extensive.
I've been using Grok 4 to write 6502 assembly language and it's been a bit of a slog but honestly the issues I've encountered are due mostly my to naivety. If I'm disciplined and make sure it has all of the relevant information and I'm (very) incremental, I've had some success writing game logic. You can't just tell it to build an entire game in a prompt, but if you're gradual about it you can go places with it.
Like any tool, if you understand its idiosyncrasies you can cater for them, and be productive with it. If you're not then yeah, it's not going to go well.
I work on niche platforms where the amount of example code on Github is minimal, and this definitely aligns with my observations. The error rate is way too high to make "vibe coding" possible.
I think it's a good reality check for the claims of impending AGI. The models still depend heavily on being able to transform other people's work.
Maybe other folks’ vibe coding experiences are a lot richer than mine have been, but I read the article and reached the opposite conclusion of the author.
I was actually pretty impressed that it did as well as it did in a largely forgotten language and outdated platform. Looks like a vibe coding win to me.
This is great. I am currently vibecoding a replacement connector for some old EDI software that is written in a business basic kind of language called ProvideX, and a fork of that has undocumented behaviour.
It uses some built inet ftp tooling thats terrible and barely works, even internally anymore.
We are replacing it with a winscp implementation since winscp can talk over a COM object.
unsuprisingly the COM object in basic works great - the problem is that I have no idea what I am doing. I spent hours doing something like
WINSCP_SESSION'OPEN(WINSCP_SESSION_OPTIONS)
when i needed
WINSCP_SESSION'OPEN(*WINSCP_SESSION_OPTIONS)
It was obvious after because it was a pointer type of setup, but i didnt find it until pages and pages deep into old PDF manuals.
However the vibecode of all the agents did not understand the syntax of the system, it did help me analyse the old code, format it, and at least throw some stuff at the wall.
I finished it up friday, hopefully i deploy monday.
It's believable that we might either see an increase in the number of new programming languages since making new languages is becoming more accessible, or we could see fewer new languages as the problems of the existing ones are worked around more reliably with LLMs.
Yet, what happens to adoption? Perhaps getting people to adopt new languages will be harder as generations come to expect LLM support. Would you almost need to use LLMs to synthesize tons of code examples that convert into the new language to prime the inputs?
Once conversational intelligence machines reach a sort of godlike generality, then maybe they could very quickly adapt languages from much fewer examples. That still might not help much with the gotchas of any tooling or other quirks.
So maybe we'll all snap to a new LLM super-language in 20 years, or we could be concreting ourselves into the most popular languages of today for the next 50 years.
However I will just mention a few things. When you make an article like this please take note of the particular language model used and acknowledge that they aren't all the same.
Also realize that the context window is pretty large and you can help it by giving it information from manuals etc. so you don't need to rely on the intrinsic knowledge entirely.
If they used o3 or o3 Pro and gave it a few sections of the manual it might have gotten farther. Also if someone finds a way to connect an agent to a retro computer, like an Atari BASIC MCP that can enter text and take screenshots, "vibe coding" can work better as an agent that can see errors and self-correct.
I literally had the same experience when I asked the top code LLMs (Claude Code, GPT-4o) to rewrite the code from Erlang/Elixir codebase to Java. It got some things right, but most things wrong and it required a lot of debugging to figure out what went wrong.
It's the absolute proof that they are still dumb prediction machines, fully relying on the type of content they've been trained on. They can't generalize (yet) and if you want to use them for novel things, they'll fail miserably.
I had way better results. I'd assume the same would have happened to the author if he provided the LLM with a full documentation on what ATARI BASIC is and some example programs.
Especially when asking the LLM to create a drawing program and a game the author would have probably received working code if he supplied the ai with documentation to the graphics function and sprite rendering using ATARI BASIC.
I had more luck with a little experiment a few days ago: I took phone pics of one of the shorter BASIC listings from Tim Hartnell's "Giant Book of Computer Games" (I learned to program out of those back in the early 80s, so I treasure my copy) and asked Gemini to translate it to plain C. It compiled and played just fine on the first go.
Vibe Coding seems to work best when you are already an experienced programmer.
For example "Prompt: Write me an Atari BASIC program that draws a blue circle in graphics mode 7."
You need to know that there are various graphics modes and that mode 7 is the best for your use-case. Without that preexisting knowledge, you get stuck very quickly.
For those that don't know. x87 was the FPU for 32-bit x86 architectures. It's not terribly complicated, but it uses stack-based register addressing with a fixed size (eight entry) stack.
All operations work on the top-of-stack register and one other register operand, and push the result onto the top of the stack (optionally popping the previous top of stack before the push).
It's hard but not horribly so for humans to write.. more a case of annoyingly slow and having to be methodical, because you have to reason about the state of the stack at every step.
I'd be very curious as to whether a token-prediction machine can get anywhere with this kind of task, as it requires a strong mental model of what's actually happening, or at least the ability to consistently simulate one as intermediate tokens/words.
All these stories about vibe coding going well or wrong remind me of an old joke.
A man visits his friend's house. There is a dog in the house. The friend says that the dog can play poker. The man is incredulous, but they sit at a table and have a game of poker; the dog actually can play!
The man says: "Wow! Your dog is incredibly, fantastically smart!"
The friend answers: "Oh, well, no, he's a naïve fool. Every time he gets a good hand, he starts wagging his tail."
Whether you see LLMs impressively smart or annoyingly foolish depends on your expectations. Currently they are very smart talking dogs.
When I start getting nonsense back and forth prompting, I've found it best to just start a new chat/context with the latest working version and then try again with a slightly more detailed prompt that tries to avoid the issues encountered in the previous chat. It usually helps. AI generally quickly gets itself lost, which can be annoying.
I don't know if anyone notices that, but LLMs are very much like what you see in dreams when you reflect on them when awake. When you asleep a dream feels very coherent; but when you're awake, you see wild gaps and jumps and the overall impression of many dreams' plot is that it is pure nonsense.
I once heard advice on trying to re-read the text you see in dreams. And I did that once. It was a phrase where one of words referred to a city. The first time I read that city was "London". I remembered the advice and re-read the phrase and the word changed to the name of an old Russian city "Rostov". Yet the phrase was "same", that is it felt same in the dream, even though the city was different.
LLMs are like that dream mechanics. It is how something else is reflected in what we know (e.g. an image of a city is rendered as a name of a city, just any city). So, on one hand, we do have a similar mechanism in our minds. But on another hand our normal reasoning is very much unlike that. It would be a very wild stretch to believe that reasoning somehow stems from dreaming. I'd say reasoning is opposite to dreaming. If we amplify our dreaming mechanics we won't get a genius; more likely we'll get a schizophrenic.
33 comments
[ 4.2 ms ] story [ 45.7 ms ] threadTry a local LLM then train it
I've been using Grok 4 to write 6502 assembly language and it's been a bit of a slog but honestly the issues I've encountered are due mostly my to naivety. If I'm disciplined and make sure it has all of the relevant information and I'm (very) incremental, I've had some success writing game logic. You can't just tell it to build an entire game in a prompt, but if you're gradual about it you can go places with it.
Like any tool, if you understand its idiosyncrasies you can cater for them, and be productive with it. If you're not then yeah, it's not going to go well.
I think it's a good reality check for the claims of impending AGI. The models still depend heavily on being able to transform other people's work.
I was actually pretty impressed that it did as well as it did in a largely forgotten language and outdated platform. Looks like a vibe coding win to me.
There is a lot of nuance in how X is said.
It uses some built inet ftp tooling thats terrible and barely works, even internally anymore.
We are replacing it with a winscp implementation since winscp can talk over a COM object.
unsuprisingly the COM object in basic works great - the problem is that I have no idea what I am doing. I spent hours doing something like
WINSCP_SESSION'OPEN(WINSCP_SESSION_OPTIONS)
when i needed
WINSCP_SESSION'OPEN(*WINSCP_SESSION_OPTIONS)
It was obvious after because it was a pointer type of setup, but i didnt find it until pages and pages deep into old PDF manuals.
However the vibecode of all the agents did not understand the syntax of the system, it did help me analyse the old code, format it, and at least throw some stuff at the wall.
I finished it up friday, hopefully i deploy monday.
It's believable that we might either see an increase in the number of new programming languages since making new languages is becoming more accessible, or we could see fewer new languages as the problems of the existing ones are worked around more reliably with LLMs.
Yet, what happens to adoption? Perhaps getting people to adopt new languages will be harder as generations come to expect LLM support. Would you almost need to use LLMs to synthesize tons of code examples that convert into the new language to prime the inputs?
Once conversational intelligence machines reach a sort of godlike generality, then maybe they could very quickly adapt languages from much fewer examples. That still might not help much with the gotchas of any tooling or other quirks.
So maybe we'll all snap to a new LLM super-language in 20 years, or we could be concreting ourselves into the most popular languages of today for the next 50 years.
However I will just mention a few things. When you make an article like this please take note of the particular language model used and acknowledge that they aren't all the same.
Also realize that the context window is pretty large and you can help it by giving it information from manuals etc. so you don't need to rely on the intrinsic knowledge entirely.
If they used o3 or o3 Pro and gave it a few sections of the manual it might have gotten farther. Also if someone finds a way to connect an agent to a retro computer, like an Atari BASIC MCP that can enter text and take screenshots, "vibe coding" can work better as an agent that can see errors and self-correct.
It's the absolute proof that they are still dumb prediction machines, fully relying on the type of content they've been trained on. They can't generalize (yet) and if you want to use them for novel things, they'll fail miserably.
Especially when asking the LLM to create a drawing program and a game the author would have probably received working code if he supplied the ai with documentation to the graphics function and sprite rendering using ATARI BASIC.
For example "Prompt: Write me an Atari BASIC program that draws a blue circle in graphics mode 7."
You need to know that there are various graphics modes and that mode 7 is the best for your use-case. Without that preexisting knowledge, you get stuck very quickly.
For those that don't know. x87 was the FPU for 32-bit x86 architectures. It's not terribly complicated, but it uses stack-based register addressing with a fixed size (eight entry) stack.
All operations work on the top-of-stack register and one other register operand, and push the result onto the top of the stack (optionally popping the previous top of stack before the push).
It's hard but not horribly so for humans to write.. more a case of annoyingly slow and having to be methodical, because you have to reason about the state of the stack at every step.
I'd be very curious as to whether a token-prediction machine can get anywhere with this kind of task, as it requires a strong mental model of what's actually happening, or at least the ability to consistently simulate one as intermediate tokens/words.
A man visits his friend's house. There is a dog in the house. The friend says that the dog can play poker. The man is incredulous, but they sit at a table and have a game of poker; the dog actually can play!
The man says: "Wow! Your dog is incredibly, fantastically smart!"
The friend answers: "Oh, well, no, he's a naïve fool. Every time he gets a good hand, he starts wagging his tail."
Whether you see LLMs impressively smart or annoyingly foolish depends on your expectations. Currently they are very smart talking dogs.
For example, if the llm had a compile tool it would likely have been able to correct syntax errors.
Similarly, visual errors may also have been caught if it were able to run the program and capture screens.
I once heard advice on trying to re-read the text you see in dreams. And I did that once. It was a phrase where one of words referred to a city. The first time I read that city was "London". I remembered the advice and re-read the phrase and the word changed to the name of an old Russian city "Rostov". Yet the phrase was "same", that is it felt same in the dream, even though the city was different.
LLMs are like that dream mechanics. It is how something else is reflected in what we know (e.g. an image of a city is rendered as a name of a city, just any city). So, on one hand, we do have a similar mechanism in our minds. But on another hand our normal reasoning is very much unlike that. It would be a very wild stretch to believe that reasoning somehow stems from dreaming. I'd say reasoning is opposite to dreaming. If we amplify our dreaming mechanics we won't get a genius; more likely we'll get a schizophrenic.