Old C code – how to upgrade it?
I've got this code I'm interested in [1], and asked about 2 years ago, actually. My efforts at patching it didn't go well, in retrospect. [2]
It's a Forth variant that does typing, and lots of things that normal forth can't do... likely as powerful as Lisp, I think. It's become an obsession to get this thing running.
I do a wget, unzip, untar, ./configure, make, su make install and it works perfectly... under Debian 3
Anything newer, even with -std=c99, and it dies a horrible death.
Taking the executable from Debian 3 to Debian 11 (32 bit) and it works perfectly.
How do I, a non-C programmer, who is willing to dump tons of time into it, migrate this code into the year 2022?
Here's an example of something that seems opaque to me, from src/words.c:2012
*(--(char*)hash_ptr.parm.v.p) = fpop(sst);
I have a hint as to what's going on here, but when you throw * in there twice, I think you can see how it's confusing.
144 comments
[ 3.6 ms ] story [ 206 ms ] thread[0] https://lwn.net/Articles/885941/
Example:
I know that pointer math in C is weird, and increment/decrement aren't necessarily +/- 1, but rather the size of the object pointed to??? THis is one of the first things that blows up when you compile it with a modern gcc. src/words.c - line 2012Maybe the problem is that fpop() returns something else than a char?
As others have mentioned, take some time to learn C on its own -- it will make the porting effort go faster, and serve you well generally.
Have you read "The C Programming Language" by Brian Kernighan and Dennis Ritchie? (but unless that code base is really old, I suggest later-than-first edition)
Need to check what address alignment for target environment is and change the cast to appropriate alignment. aka sparc addresses are not 8 bit aligned!
hash_ptr.parm.v.p points to top of sst stack containing an 8 bit aligned pointer address. p == ---(.....) gets address to pop off stack. missing a step/statement. sst needs to be in struc address given by p. aka hash_ptr.parm.v.sst p should then point to the result of fpop(hash_ptr.parm.v.sst
So in the code fragment above, it seems that the fpop() number is saved into a character value (apparently with some certitude that it should fit into char type) at some pointer destination in what seems to be an array of chars.
EDIT: This might be use of generalized lvalues: https://gcc.gnu.org/onlinedocs/gcc-3.2.3/gcc/Lvalues.html - latest gcc manual doesn't mention it though, so perhaps it's been removed?
I'd look at the types of hash_ptr, hash_ptr.parm, hash_ptr.parm.v, and hash_ptr.parm.v.p; I'd try building without optimizations (-O0) and see if that changes anything; and as others have said I'd try pulling it up in a debugger so I can poke things.
Usually when I get to the stage where whatever I’m working on is ready to build I pipe stderr to a file and just go down the list fixing all the little things until it works (or usually segfaults).
I’ve successfully updated quite a few super old codebases to play around with by just fixing each compiler error until it works or I have to go hunting real bugs.
I'd be worried if the code was old enough you were seeing K&R C notation or if it was a huge codebase, but it doesn't look like either case has occurred.
Of course if you're very much not a C programmer, learn some C. It's a comparatively small language overall (IMO) even if it's a low level one. Newer versions of the K&R book should get you up to speed pretty quick.
-Werror=implicit-function-declaration -Werror=implicit-fallthrough=3 -Werror=maybe-uninitialized -Werror=missing-field-initializers -Werror=incompatible-pointer-types -Werror=int-conversion -Werror=redundant-decls -Werror=parentheses -Wformat-nonliteral -Wformat-security -Wformat -Winit-self -Wmaybe-uninitialized -Wold-style-definition -Wredundant-decls -Wstrict-prototypes
[Edit] I tried it, and it didn't make anything worse.
What I was hoping to find (and spend a few days looking) was for some set of flags I could give to a modern copy of gcc to hold it's nose and compile this code as-is, as a starting point.
Git under Debian 3 seems to be a no-go. I'm tempted to just have two virtual machines that are never on at the same time mount a separate virtual disk that holds the Stoical source code and git repository.
Make changes / test in Debian 3... when happy shut it down, fire up Debian 11 and do a commit and push to github. Shut it down, fire up Debian 3, repeat.
And those flags should be used first on the Debian 3 machine, at least those which are supported there.
In my experience, jumping straight to the new version and making it "hold its nose" only ever works if you already know what you're doing.
So I downloaded the latest version of git source, using wget from
https://mirrors.edge.kernel.org/pub/software/scm/git/git-2.9...
Then I found I needed zlib, from
https://www.zlib.net/zlib-1.2.12.tar.xz
And I also needed Tcl/Tk
https://prdownloads.sourceforge.net/tcl/tcl8.6.12-src.tar.gz
And I also needed autoconf
http://ftp.gnu.org/gnu/autoconf/autoconf-2.71.tar.gz
Autoconf needed a newer version of m4
http://ftp.gnu.org/gnu/m4/m4-1.4.19.tar.gz
m4 couldn't install... I'm not sure why... so I gave up at that point. 8(
First, we addressed the compiler issue, getting it to build with GCC instead of VC++ on Windows. This was the most time-consuming step.
Second, we built on Linux x86 with GCC, because we were quite familiar with it, and this enabled..
Third, we built on Linux PowerPC with gcc, which was the little -> big endian step.
Finally, we got it running on OS X.
For your issue in particular, I'd try to stick with 32-bit x86 over x86-64 to start, and inch my way through newer Debian and gcc versions. You could install gcc-4.2 as late as Debian 5, I think.
congratulations ! like a pop song you cannot get out your head, you have found and bonded with something rare, special and worthy.. but like an old, old automobile from your Uncle's backyard, it might take endless efforts for modest (or worse) rewards
> make install and it works perfectly... under Debian 3
you have your first target.. make a VirtualBox VM with Debian 3, install your target software, and guard it well !! backups, small environment improvements.. connection to the Internet even?! small steps are called for..
> Anything newer, even with -std=c99, and it dies a horrible death
standard C99 is not a terrible thing.. the libraries that are linked are the big weakness there, but despite lots of complaining from modern coders, the libraries actually do work for you know, your own use.. thats what you are building, something for one person to use, you!
> I, a non-C programmer, who is willing to dump tons of time into it, migrate this code into the year 2022
you already know the answer to this.. there is no way to port C code to standard C99 without knowing C. So, you have to cooperate with others somehow. This post is not a bad start, but if you are looking for volunteers, there has to be benefit for them. A random other person who also finds this obsessively compelling.. probably not a realistic outcome, even with the massive audience at YNews. There are legions of middle-aged guys who know C at the C99 level, but for whatever reason are not professionally programming right now. This might be a local search, in "meat space" or, alternatively worded invites, in other forums or mailing lists. Good luck!
The program has some kind of portability bug or bugs that make it sensitive to the compiler. That the old executable works probably means that it's not a library or environment issue, though that can't be ruled out 100% because an executable from Debian 3 may be using compatibility functions in glibc that a Debian 11 executable wouldn't use.
A C programmer would just debug this on that platform where the problem reproduces.
There are tools which can help pinpoint the origins of behavior that leads to a crash.
- compiling the code with the "undefined behavior sanitizer a.k.a. ubsan" using -fsanitize=undefined.
- compiling the code also with the "address sanitizer (asan)" using -fsanitize=address.
- using valgrind on it
Of course you have the debugger gdb, and the printf function.
Interestingly, the project isn't using optimization (-O0 is specified in the CFLAGS of the root Makefile) so the failure is likely not coming from that class of issue whereby clever optimization defeats the programmer's incorrectly expressed intent.
Yep, I figured as much. I want this thing to exist, and be multi-platform, and supported... so I'm willing to do that.
Other things that may help find issues:
* the -pedantic -Wall -Werror flags
* building with -ggdb3, run the program via gdb. With any luck it will lead you to a stack trace
> Anything newer, even with -std=c99, and it dies a horrible death.
Yes because the compiler is catching loads of undefined behavior that the code employs. I wouldn't suggest using this library and depending on what it does I'd just suggest finding a newer, more up to date library. Using it as-is probably worked for gcc at the time but it's anyone's guess as to how it'll run today.
If you don't care about any of this, just keep turning off warnings (`-Wno-unsequenced`) until it compiles correctly.
The answer depends on the exact meaning of that quote.
Likely there's a version of GCC where it works, and one where it doesn't, or the equivalent for a library.
Getting the latest GCC to build on Debian 3 might be an adventure in itself, of course.
I don't know if my best bet is to start from another project that works/compiles/run and then port my old code to that or if I should first try to compile it again.
I was "wise" enough to backup all my code and assets (gfx, music, sound fx) and executables, but foolish enough to not backup the tooling. I never thought that 30 years+ later I'd like to compile it again.
If you have the .exe, we can figure out which compiler was used.
I'll drop you an email (but I'm in the middle of moving to another country so things are a bit hectic atm).
- Learn well a modern lang (Rust, Zig, Modern C?, ... lets call it the porting LANG).
If you are not a OLD HAT C developer and was "raised" in more modern things, going straight to some modern alternative to C (Rust, Zig, ...) could pay off much more if you are really for the long term.
This not replace the fact you need to know enough C to learn this, but is for your sanity long term (You can disregard this if plan to be a C developer for long).
Is important because:
- Collect a list of requirements and dependencies. C requiere to recreate a lot of basic stuff (like hash, strings, etc).
You can certainly throw all of that. Even if that implementation has some useful property, you don't need that burden (re-implement that) right know.
Modern langs also have better idioms to use memory (no manual mallocs!) and that is another burden that is not worth long term.
p.d: For what I see, this project has TOO BIG of a "re-creating basic stuff that any sane lang has in their std library" than actual code for make the idea work. So spend time there is lost time, IMHO). If stay on C, look for already-implemented alternatives that let you focus in the main tasks.
- Eliminate/Simplify your tooling. If CMake sucks, it sucks. Replace it with the most sane alternative for your LANG.
- Prepare yourself for quickly and merciless refactoring all, all the time. The faster you get in the cycle of "attempt understand, implement, test, rewrite, restart" the better.
All the above is to improve your velocity at the start or in the long term.
In no particular order:
- Put a debugger. Step in the code.
- If you manage to compile the old code, and knowing this is a lang, you have a good chance to make this great for testing: Input STDIN and output STDOUT and use that to confirm things!
- Collect a "call tree" using any means (notes, a tree editor, etc).
Implement things in a new blank project, and write the functions. Be sure you understand the small steps. When stuck, create another mini project and try to pull the idea alone.
This is harder than "just move forward" but the point is that you need to understand the project, not just make it compile, right?
- DO NOT punish yourself if the porting is not 100% exact.
Is more important to match "outcomes" than "exact outputs". Is very likely that you hit a bug or a hack and recreating that stuff is not worth in this case, IMHO.
- Ask things to the community! Go into a good place where discuss stuff for your LANG and ask for help. Also, you can consider go into places like https://www.reddit.com/r/ProgrammingLanguages/ for META questions about language development & paradigms.
P.D: Consider this ideas as not replacements for the others in this thread, just extra things to consider!
I’m no professional coder but I like to play around with stuff and have learned enough C/C++ over the years to code my way out of a paper bag specifically from not trying to reimplement whatever I’m playing with in another language — well, unless that language is Ruby[0] then I fully agree.
[0] there was this project that took xml schema and output something (dunno cause ruby) that I ported over to python so I could actually make sense of it because the technique they used was kind of brilliant.
That depends. I make in my career A LOT of ports from OLD -> NEW. In the very short term, is not good.
But in the mid/long term is certainly right if the new lang provide significant advantages (and in the case of C: Almost any other lang provide significant advantages!).
The case of C make some no-brainer stuff go out of the window. Making your own hash-table? String? Memory management? All that is a mini-major project and besides the core task.
If you don't know C very well, you will screw up your strings, your hash, your memory: How that helps when ALSO you screw up your core tasks?.
First is make some simple test case scripts that work on Debian 3, so you can cheaply get a feel for if it still works once it's building again. Run these after evety build.
You might want to run it under valgrind on Debian 3 and fix any bugs coming up there before trying to clean it on a modern toolchain. Because it will be much easier to find and fix those when it runs well and you don't have to guess if the problem is the newer toolchain.
You might want to crank up the compiler warnings step by step on Debian 3 first, and fix those first.
Then bring the cleaned and still working code to a new toolchain, under valgrind.
Don't worry if there are pages of compile errors... these are usually a handful of things wrong appearing in multiple places.
https://m.youtube.com/watch?v=_cIVa-RctcA&t=325s
PS. some of the first things to do will be to:
- if there's a way to get hold of previous versions, make a fresh Git history that includes them, so that the changes can be correlated with release notes
- use the sanitizers offered by today's compilers, or valgrind if having to compile it with old compilers.
- read the code about how it encodes pointers, and adapt that to 64 bit if needed
- maybe instrument the garbage collector if issues show up there, maybe move to allocating memory via malloc so that the memory sanitizer will catch violations
- write documentation about the workings that we study (that helps getting a clear mind and staying focused, and may help people in the future)
[1] https://github.com/pflanze/lilyvm [2] https://github.com/pflanze/copycat
Copycat looks interesting. I can see how Stoical and Copycat are mirrors of each other.
As for LilyVM, trying to cram anything into a 6502 is an interesting exercise.
I noticed some people working on an LLVM backend for the 6502; I may revisit the C-64 target once that matures (it will generate better code than cc65, and I'm now relying on modern C features that cc65 doesn't support). I currently concentrate on getting LilyVM to do anything useful for me on modern systems. (I've got some more work that I haven't pushed yet; but I mainly need to finally start working on the compiler to bytecode, which is a larger can of worms than the VM. I might first re-implement Copycat on it.)
[1] Viola, one of the first graphical web browsers that assumes sizeof(int) == sizeof(long) == sizeof(every pointer).
It would be hard for me to not ask questions, though. :)
Public live streaming is new to me and so far I've evaded any public records of my video and voice, maybe that's pointless since this will become increasingly impossible into the future, but I'm undecided and so far am planning for conference calls that, while open for anyone to join (see [1]), will not make recordings public. I might be swayed. I'm also still not omniscient in the C world (the most glaring is probably that I've never used the GNU autotools!), OTOH I'm yearning for somewhat higher levels of abstraction so my C coding is often untypical, so YMMV if you're after learning standard ways of working in C.
[1] https://github.com/pflanze/stoical-mentoring
- Write lots of tests with high test coverage before migration. Then you can rely on them on a continuous basis to make sure any updates do not break the existing functionality
Just in case you're interested in joining the sessions that I initiated, see https://news.ycombinator.com/item?id=30973570
The line of code is basically two lines:
(If I were doing this, I'd probably change the p field to a char *. That's probably what it wants to be anyway. Then see what comes of that.)
(*char) --> compiler will detect how to align char address for given hardware.
aka, 32bit machine,there are 4 places that a byte can evenly fit in 32 bit address/pointer.
May not be a gcc issue.
Directly manipulating a memory address (aka the numeric location where value is stored) usually prohibited by OS (or not allowed by hardware used) for security reasons.
May also be reason why works under Debian <X> and not Debian <Y>
if addresses are 64bits, and only 48 bits are used for addressing, there are 16 bits available for other use!
In standard C, -- operates on what in C terms is known as a modifiable lvalue, a value that may be used on the left hand side of an assignment expression (https://en.cppreference.com/w/c/language/value_category). And a cast returns a value that is specifically not an lvalue of any kind (https://en.cppreference.com/w/c/language/cast), meaning a cast expression is not a valid operand for the -- operator.
http://port70.net/~nsz/c/c89/c89-draft.html#3.3.4
Footnote 36.
edit: but then the code supports alpha, so it should be 64bit clean...
Will also need to understand the hardware assumptions implied in C code being ported.
Unfortunately, I suspect the answer is going to strongly involve dumping time into becoming a C programmer. (I say this as someone in a somewhat similar situation; I had no intention of learning C, but I wanted to get into dev work on existing unix-family OSs, which are pretty much all written in C, so guess what I'm learning?)
Backstory - About a week ago, I got the original compiled, learning about autoconf and make a bit along the way. I turned on ALL of the debugging options to try to figure it out.
Stoical is a variant of Forth from BEFORE Fig-Forth, so I didn't even realize it was working at first. It uses = to print, not . which took a while to figure out. Once the penny dropped, and I had it working, I was able to turn debugging back off and find that it works pretty much out of the box. I replicated it on a new Debian 3 VM and it does indeed do so.
Unfortunately, GIT doesn't come with Debian 3, and my attempts to compile it ran into far too many prerequisites, including zlib, Tcl/TK, autoconf (which requires m4, but a newer m4), eventually I quit that approach.
How it works:
There's a small set of built-in words, which it then uses to bootstrap an interpreter (14 words long) which then loads in the rest of the language. All of this works right out of the box in Debian 3. The executable compiled under Debian 11 starts to run, but can't bootstrap itself. Interestingly, the image compiled under Debian 3 does work in it's place.
Writing interpreters in C is a solid path to really learning the language from my experience, but jumping head first into a code base like that without knowing C is a recipe for frustration and suffering.
There are a million more or less advanced, custom Forths out there; here are some of my own implemented in various languages:
https://github.com/codr7/ampl
https://github.com/codr7/snabel
https://github.com/codr7/alang
https://github.com/codr7/gfoo
https://github.com/codr7/forthy2
Are you trying to port it to some other host or architecture? If so, you'll need to learn C to port it.
Are you trying to port it to a 64-bit target or something like that? If so, you'll need to learn some C.
Does the existing code compile and run but has show-stopper bugs for you? If so, you'll need to learn C and fix them.
Agree that they'll definitely have to learn C. Sounds like this will definitely teach them some of that!
(*char) is used to force pointer address inside of a struct to align on byte boundary which is not true of all hardware! aka adresses allignment may need to be 16, 32, or 64.
(For context: When Debian 3 was released, the current browser versions were Netscape 6.2.3 and Internet Explorer 6.0)
See also https://www.debian.org/News/2002/20020719