Feedback appreciated, be it on the article, project or website ! Be wary that english is not my mother tongue, so there might be some grammatical weirdities and such.
Wow, this is seriously great! This is exactly what I needed for my project, Viola[1], that provides an abstraction on top of libCello[2] to allow you to write really high-level C. This really helps make Viola a little more high-level. I am really grateful for your blog post!
I've read with great interest the source of libCello when I first saw it a few days ago on reddit, it seems like a fun experiment. Good luck with your project ! :)
I first thought about wrapping malloc to register all allocated data, and then free everything before exiting, but it wasn’t a satisfying solution.
Quick question, Isn't this the same with just leak all the memory that you've allocated. Because, after a process died, operating system will just take it back the memory it used.
Yep, but if I just let the system get all the memory back, valgrind would have yelled at us for leaking memory, and it would have been detected by the testing system, slapping us in the face with a malus on our final grade.
GCC and Clang specific, as said in the readme page.
I know, apply.h is ugly, but I did not want to lose too much time on that (and it was easy to dump these lines from a shell command), as it is only used for an optional macro.
When the recursion limit is hit, every parameter after that is ignored, and I doubt someone will ever use a struct with 64 members needing destruction, considering of course they would use this, and use the completely optional "DESTRUCTOR" macro helper.
But yes, I get it, I'll probably find the time to implement some kind of recursion based off OBSTRUCT/DEFER that we often see on stackoverflow.
You can make smart pointers which are not specific to a GNU language. Here is how: doh, use C++.
Let's see, ISO C++ portable smart pointers? Or GNU/LLVM C smart pointers?
If all you want is a C dialect, with smart pointers (and nothing else from C++), you can easily program that way.
So here is an idea: in a similar vein, you can make your scheme work with C or C++. You then have code which can be compiled as standard C++, and has smart pointers. Or, it can be compiled using the LLVM/GNU dialect of C and it can have smart pointers. (Okay, so what is the benefit, then? Just that someone can build such a program without a full toolchain that includes a C++ compiler: the program is buildable with a more bare-bones toolchain. There are some added benefits in that you can validate the code on more compilers!)
From the point of view of the implementation of this, it could be great. Add a few #ifdef __cplusplus in there and see if you can target C++. You then have more ways to test things.
Anyway, this smart pointer interface doesn't have to dictate any such a dialect to its users; it can just have C++ as a target. It was just a thought.
When I work in a restricted dialect of C that is portable to C++ compilers, I have better type safety: safer treatment of void *, string literals that are const-qualified, type-safe enumerations, and safer casting operators (reinterpret_cast, const_cast, static_cast) which I can hide behind macros that work in C or C++. On the other hand, I don't have C99 designated initializers. It would be great to have both!
If I had some static data that would so greatly benefit from these initializers that I cannot do without them, I might put that into a separate module that is compiled as C99. Not terribly convenient if you just want to throw a handful of static functions into an operations structure, though or whatever.
There are solutions for better initialization available when you're working in C90. For each structure that is instantiated somewhere with an initializer, you can write an initializer macro instead. The macro is close to the structure definition and is maintained together. If you need a few different ways of initializing the same aggregate, then several macros can be provided that have different argument lists and different defaulting.
When a new member is added (that isn't defaulted), the macros get a new argument. Either way, their expansions are carefully edited to initialize that member. The macro bodies of such initializer macros provide a complete initializer, covering every member explicitly.
If the addition of a new member results in a new macro argument, the existing macro calls then do not compile; they have to be maintained. (Whereas naked initializers just keep compiling and become incorrect when structure members shuffle around: the nasty problem we want to avoid.) Ultimately this functional approach is more disciplined and safer than designated initializers.
From a safety point of view, designated initializers only solve the problem of the wrong initial value going to the wrong aggregate member (that by chance has a suitable type to accept that initializer without a diagnostic). Functional initializers take a more complete approach of specifying the initialization in terms of constructor-like parameters, which are required, with the possibility of defaulting other elements to values other than zero.
The one thing designated initializers do which cannot be beaten, is that you can declare a million element array and initialize element [999999] to 42, statically. This may be a bad idea; some linkage models may actually implement that by adding one million words of initialized data to your executable image. Whereas if you leave it uninitialized, and stick in the 42 at run-time, the whole object is in the "BSS" section.
Of course you can do things like macros, but it's ugly and becomes unreadable the more you add to the code. Structure initializers grow and expand more gracefully than anything c++ has.
I like that a lot but I definitely use it less than the structure initializers.
I hear people tout default values as a real important feature, but I've worked on large projects where backward compatibility was important, and it was almost always possible to encode the value in such a way that zero meant "default". This allows you to add new members to structures easily and anyone that doesn't set them gets the default value. It may sound hokey but it works extremely well in practice. I never once wished for default values when I was doing large C99 projects.
Although the designated initializer has the value that it shows the name of each member, so that it resembles a keyword parameter list (like what is available ins ome languages: Common Lisp, Python, ...) it is terribly weak in that there is no logic for defaulting anything other than to zero. You have no way to express that, for instance, .handler is to be mandatory. Or if it is not mandatory, that it should default to "dfl_button_callback" or whatever.
Another thing you can't do is distinguish whether x was initialized to zero by default, or whether it was an explicit .x = 0. With the macro we know that since x isn't defaulted, it came from the macro. Or if it is defaulted, it couldn't have come from the macro. In the keyword arguments of the Lisp language, we know whether or not a keyword argument which has a default value was explicitly passed with that same value, or whether we are getting that value by default.
(In C++, which is somewhat offtopic because the subject is what is possible in C) we can do something clearer, if a bit verbose, by defining some framework for combining traits. I have done this more than once. The result looks like:
The various traits classes have an overloaded operator | which combines traits (Traits | Traits -> Traits). The Button constructor is the Button(const Traits &) one.
> …it is terribly weak in that there is no logic for defaulting anything other than to zero.
That is the weakness I addressed. IE, it may seem terribly weak up front, but in practice it's really not a weakness at all.
And, if zero is the default, then it doesn't matter if someone explicitly requested the default, or got the default by omission… It's like using 'exists' in Perl. It seems like it may be important to tell the difference between 'existing as undef' vs. not existing at all, but in practice it's just not needed (I can count the times I've legitimately used 'exists' during my almost 20 years of Perl programs on just a couple fingers).
This is really neat, but I actually wonder if we aren't trying to solve a mostly solved problem.
As a C programmer, yes, sometimes you forget to free memory that you've allocated, but then you run valgrind or some other leak checking tool and the problem is solved. When you add all these compiler specific attributes and introduce a lot of preprocessor macros I think you are creating a situation where you've created code that is easier to write, but potentially harder to read as there is now a lot more abstraction that a programmer has to wade through (assuming they need to debug that abstraction and not just trust that it just works). Now, if you're using a leak checker you can save that abstraction and switch back to using the standard free(3) library call and then 100% of C programmers will be able to follow the code.
Not really, otherwise even the standard library would not exist, and we would manually ask pages to the kernel every time we want memory.
C, as a language, is indeed low level, but that does not mean that programs and libraries shouldn't build abstractions on top of the low level interface.
Not at all. The point of C is to have as much abstraction as possible while still allowing maximal control. And 'as much abstraction as possible' is a matter of historical context, since we learn all the time that there are better ways of doing abstraction than C does.
I guess it's more about writing less & better than using known functions -- in the same reasoning, I /could/ use new/delete in C++, but I always end up using unique_ptr<T>(...) because it's just better, and we usually don't really know/care about the implementation details.
Now I realize that the comparison I made isn't fair since unique_ptr<T> is standard C++ (and hence, polished), and mine isn't and will never be standard C, but abstraction here is not that much of a threat to understanding -- in the end, it's still syntactic sugar on top of smalloc/sfree.
As a last word, I'll say that this is mostly a toy project,
and I probably won't have the occasion to fully use it. This is just a proof of concept to say that it is possible to have modern idioms in C.
Side story: I witnessed a situation in which an embedded board had to be manufactured in a variant supporting a bigger DRAM chip, just so there would be room to run Valgrind on it to solve leak issues!
There is also the problem that valgrind lets you confirm that there are leaks, and pinpoint their source. What it doesn't give you is the root cause of the leak. The cause is distributed! If N places in the program have a pointer to an object, and all those N places obliterate the pointer without freeing the object, then all N places are equally responsible.
Just because you know what is leaking and where it came from doesn't mean that it's obvious how to fix it. Introducing code which tries to free those objects is extremely risky, because if you err on the opposite side of a leak, you have a premature deallocation which destabilizes the program in a different way.
If N places lose a pointer, then it's not necessarily always the same one of those places which is the last one to do so. Yet the last one to lose the pointer must be the one which triggers any manual freeing.
This problem is difficult, which is why we have garbage collection and why the problem is sometimes solved in C programs by sliding in a garbage collector like Boehm.
Good abstractions isolate complexity from the application developer at the expense of the library author. This is true of every abstraction. I find your comment strange. You essentially wrote: just write code that works or is trivially debuggable- true and entirely unhelpful.
No it isn't a solved problem, as long C or any other language with manual memory management is used.
It isn't possible to control all the code, specially if a lot of third party libraries or contractors are involved in the project.
As for leak checkers, not all compilers support them (there is more out there than just gcc, clang, msvc, icc), or even when they do, memory constraints might prevent their use.
smart pointers are always dumb imo. GC or do it yourself - everything in between is always more of a headache in my experience. I've spent more time debugging reference counts in few code bases than debugging new/delete in many.
Haha, I guess, but I think it's not about using them all over your code base -- there are always pros and cons in using RAII, smart pointers, GCs, and manual memory management, it's the programmer's job to balance it correctly.
smart pointers are better than GC or do it yourself imo.
smart pointers and RAII allow me to explicitly model ownership and lifetime in a non imperative way. In addition, I am not subject to arbitrary GC runs with non deterministic amounts of work to do. Instead, objects simply go away as they are no longer needed in a completely deterministic way.
Naïve reference counting is deterministic, but it may also do an arbitrary amount of work on a deallocation because it reclaims objects eagerly. Deferred/incremental reference counting is much more suitable for, say, games, which is what it sounds like you’re thinking of.
This also provides unique_ptr, which doesn't use reference counting, and enforces rules very similar to those of Rust's borrow checker. They allow you to avoid memory leaks by ensuring that you either free any memory you allocate or pass it to another function (who must either free it or pass it to somebody else, ad infinitum).
Forgive my ignorance, I'm still a bit of a beginner. Why is this particularly impressive? Has no one done smart pointers well for C? Smart pointers have been around a while, I thought there would be a ton of libraries for this by now.
I've only been actively programming in C for the past three years, so I'm not that much informed either, but from my researches on the subject, I haven't seen any.
The probable reason behind this is that most people programming in C wanting automatic memory management will use the Boehm-Demers-Weiser garbage collector (?)
> The probable reason behind this is that most people programming in C wanting automatic memory management will use the Boehm-Demers-Weiser garbage collector
It is pretty impressive, actually. C, in general, does not have nice things. And it seems like this is on purpose. I think you're supposed to use C++ if you want nice things on top of C. That said, I've tried making green threads for C and it was a nightmare. I got something working though, so it's possible, but still a nightmare.
> That said, I've tried making green threads for C and it was a nightmare. I got something working though, so it's possible, but still a nightmare.
By green threads, do you mean coroutines, or something more complicated? Try the libco library, it implements them in under a hundred lines of C + a small bit of assembly per architecture: https://gitorious.org/bsnes/bsnes/source/1a7bc6bb8767d6464e3...
This relies on support in the GCC infrastructure. That support is there thanks to the back-end support for C++. Calling it "for the C language" is misleading, insinuating portability at the language level.
The title should be: "Smart pointers for the GNU C programming language".
If you could have smart pointers in the C language via a couple of macros, it would have been done several decades ago. Of course, it can be done by a code transformation: make a C-like language which translates to C. Wait, that was done by someone named Bjarne Stroustrup in a project called "C with Classes". People didn't accept that was C though, even when he shortened the name to C++.
It's also a case study on how to make a tidy project using the autotools. The only thing I'd change is to commit a 0-byte m4/.keep file and then remove autogen.sh in favour of autoreconf.
But why would you host autotooled projects on github, where they don't let you upload the release tarballs generated by make distcheck?
Release tarballs can be uploaded on a third party, I usually don't like to commit binary blobs, especially releases, since you could just clone the release tag and run make distcheck.
I'll commit a m4/.keep.
Running code when exiting scope is the key to many a things that would be useful, cool, or both. You can simulate some of the cases with a for loop. Other than that, you're out of luck unless you're willing to bite the bullet of compiler-specific features.
If we forget about destructors and think about memory management only then (except for objects that are returned back) allocations could be done on the stack if only the stack was big enough. I think Linux could actually grow stack dynamically but it's usually limited to some minor size.
Thus, I've sometimes emulated that with a chunk of heap-allocated memory that I use for carving new allocations one after another. Then, in some reasonable point in the execution I just reset the cursor back to the start of the big chunk, effectively "freeing" all allocations done by the functions called downwards from that point. It wastes memory but it's convenient as you basically have alloca() that uses heap. But you still need to manage any malloc()'d memory by yourself.
Writing a simple garbage collector working on raw allocations is feasible, too, especially if you use the above heap allocated stack to limit the amount of longer-living allocations you do on the gc heap, and if you have a spot in the program that can stand a slight delay when the gc runs.
I think it is a nice idea, but I do not agree about the way to implement it.
I would like to see more C extensions that work not only following C++'s ideas (dirty syntactic tricks) but are implemented using a proper C parser (like Cil) and principled program transformations (i.e. compilation).
Maybe we need a classier framework than Gnu C and macros to toy around with C extensions...
If a function may or may not return, and it is not marked for inlining, even gcc cannot infer whenever the variable exits the scope, so no.
You have to be extra careful anyways when using context switches.
59 comments
[ 4.9 ms ] story [ 128 ms ] threadFeedback appreciated, be it on the article, project or website ! Be wary that english is not my mother tongue, so there might be some grammatical weirdities and such.
[1] - https://github.com/eatonphil/Viola
[2] - https://github.com/orangeduck/libCello
Quick question, Isn't this the same with just leak all the memory that you've allocated. Because, after a process died, operating system will just take it back the memory it used.
Also, my succinct and unprofessional response to apply.h[1]: <Screams and runs away>
A slightly more professional response: What happens when you hit the implicit macro-based recursion limit?
[1]https://github.com/Snaipe/c-smart-pointers/blob/master/src/a...
I know, apply.h is ugly, but I did not want to lose too much time on that (and it was easy to dump these lines from a shell command), as it is only used for an optional macro. When the recursion limit is hit, every parameter after that is ignored, and I doubt someone will ever use a struct with 64 members needing destruction, considering of course they would use this, and use the completely optional "DESTRUCTOR" macro helper.
But yes, I get it, I'll probably find the time to implement some kind of recursion based off OBSTRUCT/DEFER that we often see on stackoverflow.
From the examples, it seems that it is fully compatible with standard pointers with no need of casting. Is it true ?
If so, it could work well in simplifying development with embedded development libraries, say like the mbed.
(This could be a problem if you want to call library code that frees a supplied pointer)
Let's see, ISO C++ portable smart pointers? Or GNU/LLVM C smart pointers?
If all you want is a C dialect, with smart pointers (and nothing else from C++), you can easily program that way.
So here is an idea: in a similar vein, you can make your scheme work with C or C++. You then have code which can be compiled as standard C++, and has smart pointers. Or, it can be compiled using the LLVM/GNU dialect of C and it can have smart pointers. (Okay, so what is the benefit, then? Just that someone can build such a program without a full toolchain that includes a C++ compiler: the program is buildable with a more bare-bones toolchain. There are some added benefits in that you can validate the code on more compilers!)
From the point of view of the implementation of this, it could be great. Add a few #ifdef __cplusplus in there and see if you can target C++. You then have more ways to test things.
Really? Which dialect of c++ lets me use c99 structure initializers?
C++ is no longer a superset of C.
When I work in a restricted dialect of C that is portable to C++ compilers, I have better type safety: safer treatment of void *, string literals that are const-qualified, type-safe enumerations, and safer casting operators (reinterpret_cast, const_cast, static_cast) which I can hide behind macros that work in C or C++. On the other hand, I don't have C99 designated initializers. It would be great to have both!
If I had some static data that would so greatly benefit from these initializers that I cannot do without them, I might put that into a separate module that is compiled as C99. Not terribly convenient if you just want to throw a handful of static functions into an operations structure, though or whatever.
There are solutions for better initialization available when you're working in C90. For each structure that is instantiated somewhere with an initializer, you can write an initializer macro instead. The macro is close to the structure definition and is maintained together. If you need a few different ways of initializing the same aggregate, then several macros can be provided that have different argument lists and different defaulting.
When a new member is added (that isn't defaulted), the macros get a new argument. Either way, their expansions are carefully edited to initialize that member. The macro bodies of such initializer macros provide a complete initializer, covering every member explicitly.
If the addition of a new member results in a new macro argument, the existing macro calls then do not compile; they have to be maintained. (Whereas naked initializers just keep compiling and become incorrect when structure members shuffle around: the nasty problem we want to avoid.) Ultimately this functional approach is more disciplined and safer than designated initializers.
From a safety point of view, designated initializers only solve the problem of the wrong initial value going to the wrong aggregate member (that by chance has a suitable type to accept that initializer without a diagnostic). Functional initializers take a more complete approach of specifying the initialization in terms of constructor-like parameters, which are required, with the possibility of defaulting other elements to values other than zero.
The one thing designated initializers do which cannot be beaten, is that you can declare a million element array and initialize element [999999] to 42, statically. This may be a bad idea; some linkage models may actually implement that by adding one million words of initialized data to your executable image. Whereas if you leave it uninitialized, and stick in the 42 at run-time, the whole object is in the "BSS" section.
What does this line do?
Yes, you can fix that by defining special true/false enums for each of the options, or you can do something reasonable: I love the array ones, but they're most useful for static look-up tables, and those are usually size constrained: I like that a lot but I definitely use it less than the structure initializers.I hear people tout default values as a real important feature, but I've worked on large projects where backward compatibility was important, and it was almost always possible to encode the value in such a way that zero meant "default". This allows you to add new members to structures easily and anyone that doesn't set them gets the default value. It may sound hokey but it works extremely well in practice. I never once wished for default values when I was doing large C99 projects.
Another thing you can't do is distinguish whether x was initialized to zero by default, or whether it was an explicit .x = 0. With the macro we know that since x isn't defaulted, it came from the macro. Or if it is defaulted, it couldn't have come from the macro. In the keyword arguments of the Lisp language, we know whether or not a keyword argument which has a default value was explicitly passed with that same value, or whether we are getting that value by default.
(In C++, which is somewhat offtopic because the subject is what is possible in C) we can do something clearer, if a bit verbose, by defining some framework for combining traits. I have done this more than once. The result looks like:
The various traits classes have an overloaded operator | which combines traits (Traits | Traits -> Traits). The Button constructor is the Button(const Traits &) one.That is the weakness I addressed. IE, it may seem terribly weak up front, but in practice it's really not a weakness at all.
And, if zero is the default, then it doesn't matter if someone explicitly requested the default, or got the default by omission… It's like using 'exists' in Perl. It seems like it may be important to tell the difference between 'existing as undef' vs. not existing at all, but in practice it's just not needed (I can count the times I've legitimately used 'exists' during my almost 20 years of Perl programs on just a couple fingers).
As a C programmer, yes, sometimes you forget to free memory that you've allocated, but then you run valgrind or some other leak checking tool and the problem is solved. When you add all these compiler specific attributes and introduce a lot of preprocessor macros I think you are creating a situation where you've created code that is easier to write, but potentially harder to read as there is now a lot more abstraction that a programmer has to wade through (assuming they need to debug that abstraction and not just trust that it just works). Now, if you're using a leak checker you can save that abstraction and switch back to using the standard free(3) library call and then 100% of C programmers will be able to follow the code.
C, as a language, is indeed low level, but that does not mean that programs and libraries shouldn't build abstractions on top of the low level interface.
Now I realize that the comparison I made isn't fair since unique_ptr<T> is standard C++ (and hence, polished), and mine isn't and will never be standard C, but abstraction here is not that much of a threat to understanding -- in the end, it's still syntactic sugar on top of smalloc/sfree.
As a last word, I'll say that this is mostly a toy project, and I probably won't have the occasion to fully use it. This is just a proof of concept to say that it is possible to have modern idioms in C.
There is also the problem that valgrind lets you confirm that there are leaks, and pinpoint their source. What it doesn't give you is the root cause of the leak. The cause is distributed! If N places in the program have a pointer to an object, and all those N places obliterate the pointer without freeing the object, then all N places are equally responsible.
Just because you know what is leaking and where it came from doesn't mean that it's obvious how to fix it. Introducing code which tries to free those objects is extremely risky, because if you err on the opposite side of a leak, you have a premature deallocation which destabilizes the program in a different way.
If N places lose a pointer, then it's not necessarily always the same one of those places which is the last one to do so. Yet the last one to lose the pointer must be the one which triggers any manual freeing.
This problem is difficult, which is why we have garbage collection and why the problem is sometimes solved in C programs by sliding in a garbage collector like Boehm.
It isn't possible to control all the code, specially if a lot of third party libraries or contractors are involved in the project.
As for leak checkers, not all compilers support them (there is more out there than just gcc, clang, msvc, icc), or even when they do, memory constraints might prevent their use.
smart pointers and RAII allow me to explicitly model ownership and lifetime in a non imperative way. In addition, I am not subject to arbitrary GC runs with non deterministic amounts of work to do. Instead, objects simply go away as they are no longer needed in a completely deterministic way.
The main reason behind it is this is not C.
By green threads, do you mean coroutines, or something more complicated? Try the libco library, it implements them in under a hundred lines of C + a small bit of assembly per architecture: https://gitorious.org/bsnes/bsnes/source/1a7bc6bb8767d6464e3...
The title should be: "Smart pointers for the GNU C programming language".
If you could have smart pointers in the C language via a couple of macros, it would have been done several decades ago. Of course, it can be done by a code transformation: make a C-like language which translates to C. Wait, that was done by someone named Bjarne Stroustrup in a project called "C with Classes". People didn't accept that was C though, even when he shortened the name to C++.
https://www.gnu.org/licenses/license-list.html#X11License
It's up to the Linux kernel team.
But why would you host autotooled projects on github, where they don't let you upload the release tarballs generated by make distcheck?
If we forget about destructors and think about memory management only then (except for objects that are returned back) allocations could be done on the stack if only the stack was big enough. I think Linux could actually grow stack dynamically but it's usually limited to some minor size.
Thus, I've sometimes emulated that with a chunk of heap-allocated memory that I use for carving new allocations one after another. Then, in some reasonable point in the execution I just reset the cursor back to the start of the big chunk, effectively "freeing" all allocations done by the functions called downwards from that point. It wastes memory but it's convenient as you basically have alloca() that uses heap. But you still need to manage any malloc()'d memory by yourself.
Writing a simple garbage collector working on raw allocations is feasible, too, especially if you use the above heap allocated stack to limit the amount of longer-living allocations you do on the gc heap, and if you have a spot in the program that can stand a slight delay when the gc runs.
I would like to see more C extensions that work not only following C++'s ideas (dirty syntactic tricks) but are implemented using a proper C parser (like Cil) and principled program transformations (i.e. compilation).
Maybe we need a classier framework than Gnu C and macros to toy around with C extensions...
* I implemented smalloc and sfree to respectively allocate and deallocate a memory block with prepended metadata
* I use a GNU variable attribute called cleanup to run sfree when the variable goes out of scope
* I made some macros to have some syntactic sugar on top of that
[1] - http://snaipe.me/c/c-smart-pointers/
[edit] What about:
fn A calls setjmp, calls...
...fn B allocates, calls...
...fn C which may or may not longjmp (so not marked noreturn).