HN Explorer

Show HN: NESFab – Programming language for making NES games (pubby.games)

298 points by pubby ↗ HN
This is a long-running personal project I've had to write an optimizing compiler from scratch. Everything was done by me, including the lexer/parser, SSA-based IR, high-performance data structures, and code generator.

Originally I wasn't targeting the NES. It started as a scripting language, then it morphed into a C++ replacement, and then finally I turned it into what it is today. The large scope of the project and colorful history means it's still a little rough around the edges, but it's now working well enough to post.

97 comments

[ 2.7 ms ] story [ 206 ms ] thread
[deleted]
(comment deleted)
tibbon
Not to increase your scope, but have you considered making it work across other 6502 systems?
pubby
That was the original plan, but I veered off to focus on doing one system really well. I figured if some company wanted to sponsor a port, I could do it, but otherwise I'd direct people to other compilers like LLVM-MOS. I do think it's possible to port, but it's not trivial. Some areas - like the linker - are very specifically designed for the NES.
TedDoesntTalk
> if some company wanted to sponsor a port

I don’t think any company in 2023 is going to sponsor port of a 6502 compiler, unfortunately.

detrites
You may be mistaken. Retro hw/sw and gaming is a booming business at the moment. Plenty of crowdfunded projects that have been overfunded and then delivered, and even consumer goods being shipped worldwide in major department stores.
agiacalone
This is a really neat project. It reminds me a lot of Inform, in the sense that a full “modern” language was designed around creating an older-style of game.
benj111
Why spaces and not tabs for indents?

I know this conversation has probably been had many times before. But if I prefer a 2 space tab, and you prefer a 4 space tab, we both get them how we want them with tabs.

With spaces, I have to put up with your stupidly wide indents, and try not to scratch my eyes out from the pain.

incrudible
Getting over these pet peeves is one of the best skills you can acquire as a programmer.
stcroixx
Agreed. The next challenge is getting over the annoyance every time someone wants to debate this AGAIN.
postalrat
Time to turn off linters. Seriously your linting rules suck.
incrudible
If you're unironically linting whitespace you need to get your priorities straight.
eru
Why? Sticking your whitespace rules in the linter is a great way to keep them from taking up valuable time and attention in (human!) code review.
[deleted]
(comment deleted)
incrudible
You will pay for it in wasted time and nerves on fixing whitespace warnings just to make the linter shut up.
eru
Use an auto-formatter.
incrudible
If you use an auto-formatter then just run that before you check in your code instead of setting up redundant linter rules that run every time you compile.
eru
You set up the linter to run as a check for pull-requests.

What people do locally to pre-emptively comply with the pull-requests rules is up to them, like running the auto-formatter before they commit their code, or running the test suite before making a pull-request.

incrudible
Fair enough. The thing is, if you do not fuss about whitespace, none of that even comes up. Nobody needs to set this up, make the decisions, integrate it, deploy it, maintain it and then have everyone annoyed by it until they get used to it. The people who think we need such a system are the ones I do not want on the team, because odds are that they are nitpickers whose peculiarities permeate their entire work. Brilliant people perhaps, but they better work on their own.
eru
To start: I agree that reasonable people can work together on a common code base without a strict common ruleset and only a loose shared understanding.

However: some reasonable people also like to use auto-formatters.

If you want to avoid a lot of noise when looking at diffs eg for your code review, you need to deal with those.

The most obvious problem is two people with differently set-up auto-formatters.

But you also get a problem when you mix people hand-formatting vs the auto-formatter. The first time the auto-formatting developer touches a file, you have a massive diff.

(With lots of discipline, they can stick the formatting into a separate commit or even PR, but people seldom do that.)

All that being said, I'm a very selective stickler for whitespace formatting. I like my source to be neat, and my diffs to be easy to review. I'd like people to avoid mixing tabs and spaces, avoid trailing spaces at the end of lines, and end their last line in a file with a newline.

> Brilliant people perhaps, but they better work on their own.

I have the opposite experience: people who don't cross their Ts nor dot their Is are also those who forget other niceties like freeing their malloced memory.

I approach coding from the point of view as trying to make the reviewer's life as easy as possible. So not just 'how do I get the computer to do X', but 'how do I make this so easy to understand that a reviewer has a chance of spotting any mistakes I make' (also so that reviewers and other readers of the code can take over, so I can take a vacation or move on to other things).

incrudible
I have no problem with auto formatters, because I do not fuss about white space. I turn off whitespace changes when viewing diffs. The end.
MoSattler
While I share the preference for using 2 spaces, it seems unnecessary to focus solely on criticizing the formatting choices when someone has generously shared a wonderful project they created during their free time, available for all to enjoy at no cost.
benj111
I think I may not have come across as intended.

I personally would go for tabs. That seems 'obvious' to me. The author went a different route and I'm interested in their reasoning.

As you say, it's a personal project they're giving away. My question was one of why they went in a particular direction, rather than saying that direction is wrong.

pubby
For languages with significant whitespace, mixing tabs and spaces can be very error prone. It can look like two lines of code have the same indentation, but if the compiler's tab width doesn't match yours it would not parse correctly. See Haskell for an example of what not to do.
[deleted]
(comment deleted)
mwcremer
set your tab stops to 1 space. problem solved.
Taikonerd
This is a really interesting approach. One thing that surprised me from the code snippet:

    fn play_sound()
        {$4015}(%100)
        {$4008}($FF)
Are these hard-coded memory addresses meaningful to NES devs? I'm surprised that they don't have more readable aliases, like $SOUND_ROM_BASE_ADDR or whatever.
nubinetwork
Yes.

https://www.nesdev.org/wiki/APU#Registers https://github.com/cc65/cc65/blob/master/asminc/nes.inc#L44

kristiandupont
Still seems to me like something there should be "standard library" symbols for? Or is there some cultural reason for this?
nubinetwork
You would think so, but between NIH, old unmaintainable code (like nesasm/magickit), or not knowing other code exists, a lot of times you'll see people reinventing the wheel.
pubby
There's standardized names for the graphics registers. The convention for the sound registers is to use the address itself, as there are lots of sound registers and each does multiple things.
postalrat
Would you rather have a many different set of constants for the same addresses or just the addresses themselves?

Or do you think all NES developers can agree on the same names?

Drakim
Some people prefer localhost, others like 127.0.0.1
noobermin
When you program in ASM you can choose to use an alias but sometimes it's easier just to use the addresses. If you do NES stuff for a while you just memorize them or at the very least recognize them on sight.
wk_end
I can't speak for the NES, but I've done lots of Game Boy asm coding, and personally - although others feel differently - I prefer using the register addresses rather than trying to memorize a bunch of unofficial names. Usually context/the sort of abundant comments asm necessitates indicates well enough what the read/write is doing, and if I want to look up the register to get a better idea of it, well...the names are unofficial, there isn't clear consensus about what these registers should be called, so it's easier to lookup the address in docs, too. Eventually the common ones you memorize, anyway; there's only a handful on these systems.
karmakaze
Same. I started on an Atari 8-bit and had most of the i/o memory map memorized in both hex and decimal as well as the ROM i/o routine address/parameters.
nubinetwork
Its nice that people are still writing code for the NES, but I have to ask what makes this better than using ca65... assembly isn't hard to learn.
pubby
I like 6502 assembly a whole lot, but there's definitely limitations to it at scale. Multi-byte arithmetic is a chore, bank switching is a pointless time sink, and managing local variables makes me want to pull my hair out. Those were the deciders in making this.
karmakaze
Use an assembler with macros. I spent so much time 'in the zone' of Atari Macro Assembler[0] editor MEDIT, that when someone interrupted me I'd simply respond "I'm in medit" as if in meditation.

[0] https://atariwiki.org/wiki/attach/Atari%20Macro%20Assembler/...

noobermin
I agree with you, I was worried about doing asm the first time but after you learn it you realize it's not that terrible at all. That said, multiple approaches at different levels benefits everyone.
algem
this is really cool, thanks for sharing.
bmitc
As someone trying to undertake some personal projects like this, I have a question. How do you keep motivated enough to work on this for so long and dedicatedly? As a solo developer, how did you manage things like getting stuck on technical issues that were either bugs out of your control or something you didn't have the knowledge to know how to fix?
MysteryChessBox
Honestly keeping yourself motivated through the end is a monumental task and I can only do it on smaller games (which coincidentally fit the NES hence why I actually have a finished game on it). It's all too easy to just quit when all the fun stuff gets programmed, leaving only the boring parts left. If there's a magic trick for bigger project motivation I'd like to know, too.
cloogshicer
Think of something you really enjoy doing. As a random example I'll pick "playing video games".

Now, imagine you've been playing the same game for a couple days in a row. And you're getting kinda sick of it. Not that it's bad, it's got its good moments. You've just had enough of it, for now. Would you continue playing it?

The answer is (hopefully): Of course not! You'd move on to something else you enjoy more. If the game was truly engaging, you'll come back to it anyways, maybe a few days/weeks/months later, when it captures your interest again.

Since a few years this is my attitude towards side projects. The flipside is that you have to accept the fact that 99% of the things you start will never get finished. But that's ok, as long as you enjoy the work itself. Of course that doesn't mean there aren't difficult moments. But there has to be a genuine, intrinsic motivation, which is independent of shipping something.

Keeping a journal/todo-list for every project (I use an infinite bulleted list like Dynalist) really helps, since you can just come back to a project anytime by looking at the end of the list.

pubby
Motivation was sporadic. I'd have a few weeks of intense focus, followed by months of inactivity. The nice thing about compilers it that 90% of their code is interesting, so there was always something fun worth coming back to. Besides, I put other projects on hold so I wouldn't get distracted.

I don't always know the best way to do something, but so long as I know some way to do it, I can make progress. Tough bugs are just a matter of perseverance. And hey, if it's still not working, just drop the feature. In a personal project, you can do that and nobody can stop you.

One more thing: don't talk about your projects until they're 95% done. Seeking early validation is a poison pill.

bmitc
Thanks for the response! My projects aren't that far along, but this echoes a lot of my experience and helps me remember these are common feelings.

> I don't always know the best way to do something, but so long as I know some way to do it, I can make progress.

This helps a lot. Thanks! One of my issues is being able to be ok with feature gaps or bugs. I can solve them if they were created by me, but there are some that need outside help as they are likely bugs in external software or even drivers. But maybe it makes sense to press on and revisit later.

> One more thing: don't talk about your projects until they're 95% done. Seeking early validation is a poison pill.

Is this because you draw attention to an unfinished project, get distracted by feature requests or praise, or something else?

pubby
Yeah, external bugs suck, but usually there's workarounds.

> Is this because you draw attention to an unfinished project, get distracted by feature requests or praise, or something else?

If you get a bad reaction, you'll be demotivated. If you get a good reaction, you'll feel satisfied and not have the drive to keep working. Either that, or you'll devote more and more time to posting on social media, getting less work done in the process.

detrites
> If you get a bad reaction, you'll be demotivated. If you get a good reaction, you'll feel satisfied and not have the drive to keep working.

This is so important, and not only for sw dev but anything.

Anyone who is struggling with completing personal projects, try starting the next one with the explicit goal to not speak about it - with anyone - until it's either complete, or very near. It may be hard to do at first but ultimately worth it.

proper_elb
If you don't mind: What would you working on it look like, on a typical work-day look like, hour wise? Raise up at 5am and hack away for 2h, or come back from day job and straight into the fun thing, or wait a bit and than hack away... What worked for you?

Lately, I seem to manage 1-3 days of coding on my hobby project after work, provided I have some features where I can see meaningful success within 90 minutes of coding (+ the same time for debugging or researching new libraries, if needed)

pubby
I think it helps that I'm not working as a software developer right now, so I have more energy for coding projects. (I'd like to write software professionally, but no company responds to my applications.)

Regarding time, I have thinking periods and coding periods. On thinking periods I'll take a 30-60 minute walks and figure things out in my head. On coding periods, I'll spend 1-2 hours in the text editor per day, sometimes more, at whatever time suits me. It's pretty laid back and I don't beat myself up if I stop.

proper_elb
Sorry for the assumption and thank you for your answer! Regarding time, I work similar. In work-life, it gets a bit more muddy (basically the time talking walks expands and turns into time spent in meeting, talks, support etc.)

I would encourage you to ask here on HN and maybe also over at reddit (i.e. https://old.reddit.com/r/cscareerquestions) for feedback on landing a software dev job. Getting the foot in the door can be tough, after that it get easier. I wish you best of success in landing a job! :)

Edit: It might be a good idea to mention your current job search in your most visible comments here in this thread. Also, you could mention it on your website! ;)

iancmceachern
I expect now that some company will see this awesome work here and respond to your application. Excellent work
marketdev
Do you have a specific set of criteria that jobs must meet? Your project is awesome, so I assumed you must have worked at many places already.
and0
I strongly agree on the last point about early validation, albeit with two exceptions: when you're marketing a product you want to sell and want to generate hype, or your product requires early feedback. I've kept things hidden to keep myself motivated only to find I was left with the horribly unmotivating task of shudders marketing. Also, my friends who I thought would love it had fundamental changes they'd want to find it useful.
keyle
I'm in my 40s and I have a long string of complete side projects.

The key is to have a project you know is worthwhile and will tickle your curiosity for long periods of time... So even if you burn out, you come back to it.

I go through intense weeks of productive work followed, sometimes, by months of dust covers.

Also it's important to understand that creative energy is hormonal. Some weeks you can't look at a youtube video because you have to do X; some weeks you can't open a code editor if you're not paid for it and just want to netflix and chill. It's normal.

But if the problem at hand is worth your while, if it's interesting to you personally, you'll go back to it and resume intense focus work; maybe with a different approach.

If you totally gave up and you don't want to look at it anymore, it's probably because your inner-senses tell you it's not worthwhile. So you might need some re-convincing or maybe it's just a dead fish, move on. Money is a great motivator, so when the work is unpaid, the value has to come from somewhere else, and that's usually the importance of the project to you personally.

bmitc
> I go through intense weeks of productive work followed, sometimes, by months of dust covers.

It seems this is a theme in the replies, and it's refreshing to hear because it matches my experience. But it's nice to hear since I was worried this was abnormal.

AdmiralAsshat
Does this language have support for being able to save the game's state into a simulated battery a la Legend of Zelda/Metroid?
pubby
There's no built-in support for that, but you can still do it. Nowadays flash memory is used rather than battery stuff, but the idea remains the same.
noobermin
Sorry for the nitpick but Metroid did not have a battery pack and used a password system.
eru
On NES. On Famicom it saved to disk.
a_e_k
Nice work! Pubby, it looks like the markdown for the documentation goes off the rails in 12.3.
dom96
Really cool but shouldn't this actually be a library rather than a programming language?
detaro
Given the authors top comment here the answer to that is a very clear "no".
dom96
What top comment? The one in this submission? it doesn't explain at all why this needs to be a programming language
detaro
How would you learn to write an optimizing compiler by writing a library?
dom96
Just because the author wanted to write an optimising compiler doesn't change the fact that this might be better served as a library. The author could have written an optimising compiler for a language with a purpose that couldn't be served by a library.
detaro
If the goal of the project is implementing a compiler, writing a library is not better serving the project.
wtetzner
A library for what language?
eru
You could maybe turn this into an embedded domain specific language, if your host language is powerful enough.

But seeing that you want to be spitting out optimized 6502 machine code, it's not really useful to do this as a library for most host languages.

warpspin
Pretty cool project. While I never did NES games, I did assembler on the C64 for a while. The language alone minus the NES parts would already have been super useful back then at least for the high level parts of the programs I wrote.
agentultra
This is very cool. I had wanted to do a project like this once. Nice work!
foldor
Really cool stuff! Is there an syntax highlighter available on any IDE's like VSCode available? I'm having a hard time following the documentation without it as it's hard for me to know what is a keyword and what is a variable for example. Single letter types can be hard for me to follow, but I can definitely appreciate their style.

Nice work.

leroy-is-here
I didn’t take a deep dive, but I’m interested in how you keep the binary size small. Is it simply that LLVM, etc., all have too much function overhead and don’t take advantage of routines over subroutines (function calls)? Or is your back-end just specialized enough to take advantage of architecture-specific features the larger compiler suites don’t have the time to care about?

Anyway, this looks dope. Truthfully, we could use more hyper-specific languages like this in all sorts of areas.

pubby
LLVM and GCC are great at optimizing until the last leg of the race where they convert IR to assembly code. Their back-ends were designed for modern systems, not the ancient 6502, and so getting them to work involves kludges and clever hacks. I recall there being a video on how LLVM approaches this - search LLVM MOS on youtube.

I plan on writing an article on how my own code generator works. If you check my HN profile in a week or two you'll probably see it submitted.

leroy-is-here
Good work. I’m looking forward to the article.
mysterymath
In LLVM-MOS, we mainly struggle with its register allocator; the rest of the backend is really quite reasonable.

The "Greedy register allocator" in LLVM is a just finely tuned priority allocator with nice live-range splitting. It works great for zero page cache locations, but it's just not tuned very well for tight register classes like those involving the processor's three architectural registers. I've half a mind to implement Hack's SSA-based register allocator in LLVM to use on A, X, and Y; this would clean up the oodles of spurious copies LLVM-MOS spits out in the worst cases.

mungoman2
I have long dreamed of a language for 8-bit computers/consoles with a compiler that helps scheduling routines.

For example, let's say I have a short snippet of code that needs to run every H blanking period. I should just declare something like

@every_hblank Blabla_code()

Or scheduling some code for X pixels into a line.

Actual scheduling can be done however the compiler wants to, using interrupts or inserting dummy instructions for delays.

I guess in the end it's a constraint problem to solve where all these snippets fit in.

I imagine this makes it easier to write games/special effects as you remove the tricky cycle counting. Easier, but maybe less fun...

JohnBooty
I know that the SGDK (libraries and toolchain for Sega Genesis/Megadrive) lets you do what you're asking for, within plain old C.

You just do:

    SYS_setHIntCallback(foo);
    SYS_setVIntCallback(bar);
Where "foo" and "bar" are the functions you want to be called during hblank and vblank. Here's somebody showing how to pull off some scaling effects using them: https://under-prog.ru/en/sgdk-image-resize/

A lot of recent commercial Genesis/MD releases have been made with SGDK; performance seems quite nice. After all, theoretically, if your libraries go hard enough and lean on macros and/or inline assembler C doesn't need to carry a big speed penalty vs. assembler.

I think this is also made possible by the Genesis/Megadrive hardware throwing interrupts during hblank/vblank but don't quote me on that.

Of course, the really insane effects still require cycle counting. My understanding is that the Overdrive 2 demo uses heaps of it. It's strictly 50hz only because it's that tightly coupled to various specific timings (also obviously 50hz gives you room for a few more instructions during blanking intervals vs. 60hz)

https://www.youtube.com/watch?v=OeGdJk5zb6c

hunta2097
I got a Windows Defender alert when downloading the Windows binary:

Detected: Trojan:Script/Wacatac.H!ml

Status: Removed

Details: This program is dangerous and executes commands from an attacker.

easrng
I've never seen a Wacatac detection that wasn't a false positive, but who knows.
jonny_eh
> Handling banks is normally a tedious affair for programmers, but NESFab handles it for you. The compiler smartly allocates code and data into banks, with the gritty details abstracted away.

Killer feature, but is there a cost to bank switching? Could abstracting it result in a lot of performance cost?

pubby
There is a cost: subroutine calls get wrapped in something called a trampoline, and some data is duplicated across multiple banks. You would do both of these things in assembly language too, and could even do it more optimally, but it's an NP-hard problem and takes a lot of thinking.
eru
Most practical instances of most NP-hard problems are quite tractable with something like an Integer Programming solver or an SMT solver.
pkaye
You should make a link to the examples directory prominent on the webpage so others get an quick idea of the language features.
lastdong
Congrats on the release! Website and code look super clean, and looks like super useful for anyone getting into NES game development.

- What languages did you use as inspiration?

- Is it possible to write a game to a physical cartridge and play it on a real snes?

Thanks for sharing your amazing work!

[deleted]
(comment deleted)
nunobrito
It is looking fantastic. Great work!
Timwi
Out of curiosity, why did you disallow tabs for indentation, and /.../-style comments in the middle of a line?
[deleted]
(comment deleted)
stehtisch123
Awesome! Is it hard/possible to put the finished game on a cartridge or do you play it with an emulator?
[deleted]
(comment deleted)
terrycody
Hey I think a lot of nostalgic people sometimes want to hack the game ROMs instead of making a new one. For example, they may want to add more characters, use enemy roles, add more skills and fun things, does your programming language can also provide such purposes?