The problem here is you’re not writing C, you’re writing a language that kind of resembles C but is nonportable. So you might as well just use a different, memory safe language at that point.
Yes. This is why people put so much effort into projects like Valgrind [1] and american fuzzy lop [2]. Preserving portability by keeping the same language and trying to find as many bugs as possible using clever tooling.
It's about the control you can have. You can gradually add in annotations without having to rewrite the whole program as well as being able to selectively check buffer-overflows (i.e. checking everything vs checking code regions that you don't have security guarantee but would like to have). Additionally they also have features that allows you check overflow statically on limited cases.
Checked C’s design is distinguished by its focus on backward-compatibility, incremental conversion, developer control, and enabling highly performant code
The problem is that there is a huge body of decades-old C code out there that we all depend on.
A typical Docker container with a web app contains maybe 10K lines of your own code in Python/Ruby/JS, 100K lines of framework code, and 1M or 10M lines of C code (i.e. the interpreter itself, the web server, SSL, the base image, etc.)
So I welcome new languages focused on incremental conversion.
I applaud projects like Corrode, a C to unsafe-Rust translator, although undoubtedly they would have an easier time if Rust was designed in the first place for conversion.
I've noticed that most programmers seem to wildly overestimate the rate at which code gets rewritten. I think it's more accurate to say that code piles up over time. And unfortunately when the foundations are unstable, what you build on top is also unstable.
There is also a new tool, c2rust, https://c2rust.com that I believe was inspired by Corrode.
In addition, we also have the bindgen tool in Rust that will auto generate the Rust FFI from C header files. It works really well. At that point you wrap the FFI in more Rust idiomatic interfaces, and expose those as the preferred library for others to use.
I mention this, because it is a path toward allowing for the carrying forward of a C codebase, but using a safe language like Rust for all future work, perhaps going back and rewriting the C if it’s deemed important.
There are many examples of this pattern being applied to projects.
In addition to that, there is a cbindgen tool, that does the inverse of the bindgen tool, which would allow you to write Rust code, and quickly generate C header files for the FFI, allowing for critical areas of C to be replaced with Rust, without converting the entire project.
Putting all of that together might be a reasonable thing for large projects to convert iteratively, should they decide it’s an important thing for them to do.
No one is going to use Rust on any of these projects, because of political reasons and portability concerns (because LLVM doesn't target all platforms). But those two issues also apply to Checked C and I doubt any project you listed will ever use it either.
Since BSD's system compiler is now LLVM, I think it's only a matter of time some enlightened BSD variant (OpenBSD maybe?) tries something like Checked C.
I don't know what programming language or extensions they will ever use, but this year's Linux Security Summit was pretty clear that are lots of work to do regarding Kernel Self-Protection Project still need to be done.
No, the problem here is that no one is going to re-write UNIX derived OSes into something else, nor change their culture around C.
So there are only two options left, make C safe in some way no matter what, or just throw everything away.
The Linux Security Summit 2018 set of talks regarding kernel mitigation exploits is pretty clear how bad security exploits due to C are turning out to be.
The thing is, C can not be made "safe" without breaking old code and specified semantics. Even if you for example follow all the strict aliasing rules, you can still 1) make a mistake 2) access/modify arbitrary memory. Then you have overflows, threading races ... It is ok, C is meant to be like that. You can make C safer, make things better (arguably), that is a different thing. And it is beeing done, slowly and without breaking (in theory) stuff.
I agree the solution is not perfect and ideally I would nuke the language out of orbit.
But that approach will never win the hearts of UNIXy kernel devs, and I guess until Quantum computers take over we are past the point of having alternative OSes to UNIX based ones, because no one is going to spend the required money to do that.
I think, eventually, the whole stack will get rewritten from the ground up. Someone is going to come up with a minimal OS that can be run on virtual machines that implements LLVM in a provably correct way, the OS hooks will get reimplemented as libraries for all the languages, and over a few years or so the entire cloud infrastructure will simply switch over.
I think we're already half the way there, Amazon Linux is how widely deployed already? The surface area between application logic and OS code is shrinking by the year, once serverless goes mainstream then even application code will start to get rare, pushed to only those specific projects that require human interaction. Everything else is 'mere' computation, something that can be specified purely algorithmically and executed by anything.
The OS will only be relevant in the context of personal computing, the human interface to the machine kingdom which shed those trappings decades ago.
20-30 years down the road I see a translation layer unifying all three major, now solely personal, operating systems, and a fundamental, free software core being driven out, eventually supplanting proprietary development. Apple will do its best to retain tight coupling between their hardware and software, but they'll slowly fade just like Microsoft did as programming itself is revealed to consist of two main activities, intricate UX and comparatively dumb data plumbing, with a small priesthood of hardware driver maintainers / database developers for those who still want to do 'real' programming.
Everybody will be able to code at some level and it'll be amazing.
At some point, backwards compatibility loses its grip on necessity. At this point, reinvention becomes possible. You can look at various parts of the tech ecosystem and sort it all into categories based on just how much the human users are depending on old behavior.
Something like vim has archaisms that are only mildly depended on, allowing a project like neovim to credibly call itself the future. But Python 2 still has a large install base of software that's still running it. Python 3 didn't offer a smooth enough upgrade path from Python 2, so we're seeing the slow bifurcation of the community into two language. This also happened with Rails 2, there is still large enough install base of Rails 2 software that there's a dedicated effort to support it, Rails LTS.
These bifurcations will last until gradual enough upgrade paths are made and failure modes are isolated enough so that moving a project off of old tech and onto new tech can be accomplished without business investment. It's merely another user story to be put in the pipeline. Old applications will slowly get either abandoned or migrated or reinvented over time.
This process fails when you consider underlying infrastructure, which by its nature largely has a much deeper base of software that depends on it. Applications can have many dependencies, but those dependencies tend to be well-enumerated, as not enumerating them leads to a maintainability nightmare for people unfamiliar with the system. And nothing but other business software depends on the application.
But infrastructure code's dependency structure is inverted. Many more applications depend on infrastructure and infrastructure tends to have comparatively fewer dependencies of its own. When infrastructure has lots of dependencies it makes it harder to scale, and scale is the whole point of infrastructure. This need for infrastructure to all 'hang together' so as to provide for scale is what's behind the modern concept of an operating system. Applications tend to rely first on the software stack, second on the operating system. It's much easier take a Rails application built to run on Ubuntu and migrate it so it can run on Red Hat or Docker than it is to get it to run on Sinatra, god forbid another general purpose programming language. In fact, it's so much easier that it's considered a major red flag for this to not be the case. Operating systems are more fungible than software stacks.
This process of bifurcation followed by slow abandonment of harder-to-maintain systems in favor of easier-to-maintain ones is just going to slowly reach its apotheosis when line of business software that is much of a career programmer's bread and butter is going to slowly lose all attachment to, not just the operating system, but also the software stack as well. FaaS, Function as a service, is something you can add to your application right now, and once you do that work once, it just gets easier and easier to offload portions of the application that simply glue things together or compute things, to a magical land where teams of elves do literally everything else for you for a small fee, is going to become mainstream sooner rather than later.
This very much matches the work I've been doing recently, which aims to remove the architecture-specificity of compiled executables from operating systems by running webassembly only.
From where I am standing, with exception of UCSD Pascal, all those platforms support at least 4 languages and Web Assembly still has a bit to catch up with them regarding its architecture design.
So how many languages do they need to be to be considered large?
Don't take me wrong, I love those bytecode environments, including Web Assembly.
I just don't think Web Assembly is that much different from other ones, other than politics.
Pretty ok for their workloads, bytecode is only used as portable executable format.
The kernel has a JIT and they are converted into native code at installation time, or when an operator decides to retrigger the operation.
In a way, Android (5 - 7), iOS bitcode and WP 8.x MDIL follow similar ideas.
And something that I forgot, MSIL also has support for C and C++, hence why all C# 7.x performance improvements don't require any runtime change, just exposing features that C and C++ already make use of on MSIL.
The mainframe ones, are just like any other OS, code gets generated at installation time, then they are OS processes just like any other.
Java and MSIL depend on how they get used.
Both have bytecode verifiers, and multiple ways to deploy.
Either as single runtime, where strong typing plays the role of sandboxing, or as multiple processes/containers.
Just to use .NET as example, not going to make an exhaustive list.
.NET Micro Framework deployed on Netduino, the runtime is the OS, only strong typing takes place.
.NET Framework on Windows, and .NET Core on Windows/Mac/Linux, each CLR instance gets their own OS process, which might be additionally wrapped into an OS container.
.NET Core on UWP, applications run as separate sandboxed process using the UWP sandbox model.
Xamarin on iOS runs with the iOS sandbox model, after AOT compiling for deployment.
Xamarin on Android, runs with the Android sandbox model where binary code gets dynamically loaded into ART instance, sandboxed via SELinux and Linux container mechanisms.
And regarding security, Web Assembly still fails short of preventing memory corruption inside of the loaded module.
I was hoping pointers would be tagged, but that doesn't seem to be part of the design goals.
> I agree the solution is not perfect and ideally I would nuke the language out of orbit.
> But that approach will never win the hearts of UNIXy kernel devs
I don't think this is fair. The Linux kernel contains (as of writing) 17913052 lines of C. If you have a plan to rewrite 18 million lines of code and patch every single dependency, fine. But until then such concerns will dominate.
I don't have such plan, but Linux kernel devs have some mitigation plans.
Not to rewrite C into something else, but to get out of the current escalation of kernel exploits, now that Linux is a commodity OS, it gets targeted as such
Hence Kernel Self Preservation Project, removing VLAs from kernel code, adding all possible ways of mitigation, and hopping that eventually tagged pointers like SPARC and ARM have become common across all supported hardware.
The charts shown by Google employees regarding exploits are quite telling.
Yes, I do. But the unfortunate reality is that your "small change" makes your code much harder to work with for other people, since you are now writing a C derivative instead of standard C.
This is the same argument that was used when Microsoft launched Typescript. As it turns out, their strategy of extending an existing language to make it safer worked great, while brand new languages that are safer than Typescript are now miles behind it in terms of adoption. So it seems to be a pretty solid strategy, and if it brings people closer to a safe language, even if not 100% safe, it's still an improvement on the status quo.
> As it turns out, their strategy of extending an existing language to make it safer worked great, while brand new languages that are safer than Typescript are now miles behind it in terms of adoption
Did it? Adoption seems to me to be a pretty silly metric given all the politics of language choice.
I wonder how this will work out. GCC had checked fat pointers for years, used by almost nobody. Microsoft can make it stick on their platform, because they own the libraries and can make them use fat pointers.
I hope this works out better than Microsoft Managed C++.
Managed C++ has been replaced by C++/CLI in 2005 and it pretty much alive.
Its purpose was to make .NET interop easier with C++ libraries and has been quite successful at it, instead of having to play around with P/Invoke until everything works properly.
Regarding Windows, Microsoft is pretty much on the ongoing transition to C++, where these kind of issues are easier to tackle.
I bet their goals regarding Checked C have more to do with the adoption of UNIX (Linux/BSD) based technologies for Azure deployments and IoT devices like Azure Sphere.
There has been dozens of "safe" additions, variants and extensions to C along the years (from commercial companies, from individuals, from university labs, ...). Not one of them ever took off. This one did not either.
We always just say "hmmm... right... this one may look a bit better than the previous one (or doesn't)" and we never bother to use them. After 20 years like this, we can conclude that we like C as it is and are not interested in extra layers.
> After 20 years like this, we can conclude that we like C as it is and are not interested in extra layers.
This says a lot about how little the industry cares about software quality. You'd think a bunch of people who make a living out of automating tasks would happily use tools that automate the checking of an importnt source of bugs. Instead, many prefer to use tools that look simpler on the surface and then end up manually testing for the same kind of bugs over and over. So you have people writing in dynamically typed languages who rely on (manually written!) unit tests to check that their program is not mixing up types, and others using unsafe languages and then relying on tests to catch memory access violations. Inevitably, tests never catch all issues and you end up with type errors and segfaults in production.
> This says a lot about how little the industry cares about software quality. You'd think a bunch of people who make a living out of automating tasks would happily use tools that automate the checking of an importnt source of bugs. Instead, many prefer to use tools that look simpler on the surface and then end up manually testing for the same kind of bugs over and over. So you have people writing in dynamically typed languages who rely on (manually written!) unit tests to check that their program is not mixing up types, and others using unsafe languages and then relying on tests to catch memory access violations.
In the case of a "safer C" I suspect it's largely because it never materialized. C still has its deficiencies relative to Rust, but Rust is the first viable competitor.
For starters people have to realize that safe C is not a research problem. It's an engineering problem of compilers, package management and C software. And all it takes to take off is some engineering work.
However funding that work is a big problem, since it mostly benefits others.
This does nothing for use-after-free or integer under/overflow, so “Making C SafeR by Extension” would be a better title.
It’s good to see people attempting this, though. I would think chances are good Microsoft will use it internally, and that already would be a win. Also, it’s a clang/LLVM extension. That makes it easier for third parties to start using it.
On the other hand, their choice to use Ptr<T> for familiarity with C++ requires users who want to keep their code compatible with C compilers that don’t support this a bit harder (it will require wrapping that in some macro, say _ptrTo(T)). That isn’t ideal for adoption in open source programs.
There is some precedent for a new language that gradually improves safety of an existing language; TypeScript did this to JavaScript and is arguably successful.
I don’t know if Checked C will have the same success story but if devs can easily add this to their existing projects and it catches bugs, that seems like a win to me.
56 comments
[ 3.7 ms ] story [ 63.0 ms ] thread[1] http://valgrind.org
[2] http://lcamtuf.coredump.cx/afl/
Checked C’s design is distinguished by its focus on backward-compatibility, incremental conversion, developer control, and enabling highly performant code
The problem is that there is a huge body of decades-old C code out there that we all depend on.
A typical Docker container with a web app contains maybe 10K lines of your own code in Python/Ruby/JS, 100K lines of framework code, and 1M or 10M lines of C code (i.e. the interpreter itself, the web server, SSL, the base image, etc.)
So I welcome new languages focused on incremental conversion.
That's what I'm doing with Unix shell: http://www.oilshell.org/blog/2018/01/28.html
I applaud projects like Corrode, a C to unsafe-Rust translator, although undoubtedly they would have an easier time if Rust was designed in the first place for conversion.
https://github.com/jameysharp/corrode
So Checked C seems like a great idea to me.
I've noticed that most programmers seem to wildly overestimate the rate at which code gets rewritten. I think it's more accurate to say that code piles up over time. And unfortunately when the foundations are unstable, what you build on top is also unstable.
In addition, we also have the bindgen tool in Rust that will auto generate the Rust FFI from C header files. It works really well. At that point you wrap the FFI in more Rust idiomatic interfaces, and expose those as the preferred library for others to use.
I mention this, because it is a path toward allowing for the carrying forward of a C codebase, but using a safe language like Rust for all future work, perhaps going back and rewriting the C if it’s deemed important.
There are many examples of this pattern being applied to projects.
In addition to that, there is a cbindgen tool, that does the inverse of the bindgen tool, which would allow you to write Rust code, and quickly generate C header files for the FFI, allowing for critical areas of C to be replaced with Rust, without converting the entire project.
Putting all of that together might be a reasonable thing for large projects to convert iteratively, should they decide it’s an important thing for them to do.
I think something like Checked C is the only way to actually make it work on those use cases.
And that is just one UNIX variant.
So there are only two options left, make C safe in some way no matter what, or just throw everything away.
The Linux Security Summit 2018 set of talks regarding kernel mitigation exploits is pretty clear how bad security exploits due to C are turning out to be.
But that approach will never win the hearts of UNIXy kernel devs, and I guess until Quantum computers take over we are past the point of having alternative OSes to UNIX based ones, because no one is going to spend the required money to do that.
So a migration path towards safety is required.
I think we're already half the way there, Amazon Linux is how widely deployed already? The surface area between application logic and OS code is shrinking by the year, once serverless goes mainstream then even application code will start to get rare, pushed to only those specific projects that require human interaction. Everything else is 'mere' computation, something that can be specified purely algorithmically and executed by anything.
The OS will only be relevant in the context of personal computing, the human interface to the machine kingdom which shed those trappings decades ago.
20-30 years down the road I see a translation layer unifying all three major, now solely personal, operating systems, and a fundamental, free software core being driven out, eventually supplanting proprietary development. Apple will do its best to retain tight coupling between their hardware and software, but they'll slowly fade just like Microsoft did as programming itself is revealed to consist of two main activities, intricate UX and comparatively dumb data plumbing, with a small priesthood of hardware driver maintainers / database developers for those who still want to do 'real' programming.
Everybody will be able to code at some level and it'll be amazing.
Something like vim has archaisms that are only mildly depended on, allowing a project like neovim to credibly call itself the future. But Python 2 still has a large install base of software that's still running it. Python 3 didn't offer a smooth enough upgrade path from Python 2, so we're seeing the slow bifurcation of the community into two language. This also happened with Rails 2, there is still large enough install base of Rails 2 software that there's a dedicated effort to support it, Rails LTS.
These bifurcations will last until gradual enough upgrade paths are made and failure modes are isolated enough so that moving a project off of old tech and onto new tech can be accomplished without business investment. It's merely another user story to be put in the pipeline. Old applications will slowly get either abandoned or migrated or reinvented over time.
This process fails when you consider underlying infrastructure, which by its nature largely has a much deeper base of software that depends on it. Applications can have many dependencies, but those dependencies tend to be well-enumerated, as not enumerating them leads to a maintainability nightmare for people unfamiliar with the system. And nothing but other business software depends on the application.
But infrastructure code's dependency structure is inverted. Many more applications depend on infrastructure and infrastructure tends to have comparatively fewer dependencies of its own. When infrastructure has lots of dependencies it makes it harder to scale, and scale is the whole point of infrastructure. This need for infrastructure to all 'hang together' so as to provide for scale is what's behind the modern concept of an operating system. Applications tend to rely first on the software stack, second on the operating system. It's much easier take a Rails application built to run on Ubuntu and migrate it so it can run on Red Hat or Docker than it is to get it to run on Sinatra, god forbid another general purpose programming language. In fact, it's so much easier that it's considered a major red flag for this to not be the case. Operating systems are more fungible than software stacks.
This process of bifurcation followed by slow abandonment of harder-to-maintain systems in favor of easier-to-maintain ones is just going to slowly reach its apotheosis when line of business software that is much of a career programmer's bread and butter is going to slowly lose all attachment to, not just the operating system, but also the software stack as well. FaaS, Function as a service, is something you can add to your application right now, and once you do that work once, it just gets easier and easier to offload portions of the application that simply glue things together or compute things, to a magical land where teams of elves do literally everything else for you for a small fee, is going to become mainstream sooner rather than later.
I see wasm as a step towards this inevitability.
Then we have JVM, MSIL and DEX.
What would wasm bring that isn't already done?
What we haven't had before are architecture-independent executables that can be compiled into from a large number of programming languages.
So how many languages do they need to be to be considered large?
Don't take me wrong, I love those bytecode environments, including Web Assembly.
I just don't think Web Assembly is that much different from other ones, other than politics.
Look for IBM redbooks on IBM i and IBM z language environments.
How was the performance of the old mainframe formats?
The kernel has a JIT and they are converted into native code at installation time, or when an operator decides to retrigger the operation.
In a way, Android (5 - 7), iOS bitcode and WP 8.x MDIL follow similar ideas.
And something that I forgot, MSIL also has support for C and C++, hence why all C# 7.x performance improvements don't require any runtime change, just exposing features that C and C++ already make use of on MSIL.
The mainframe ones, are just like any other OS, code gets generated at installation time, then they are OS processes just like any other.
Java and MSIL depend on how they get used.
Both have bytecode verifiers, and multiple ways to deploy.
Either as single runtime, where strong typing plays the role of sandboxing, or as multiple processes/containers.
Just to use .NET as example, not going to make an exhaustive list.
.NET Micro Framework deployed on Netduino, the runtime is the OS, only strong typing takes place.
.NET Framework on Windows, and .NET Core on Windows/Mac/Linux, each CLR instance gets their own OS process, which might be additionally wrapped into an OS container.
.NET Core on UWP, applications run as separate sandboxed process using the UWP sandbox model.
Xamarin on iOS runs with the iOS sandbox model, after AOT compiling for deployment.
Xamarin on Android, runs with the Android sandbox model where binary code gets dynamically loaded into ART instance, sandboxed via SELinux and Linux container mechanisms.
And regarding security, Web Assembly still fails short of preventing memory corruption inside of the loaded module.
I was hoping pointers would be tagged, but that doesn't seem to be part of the design goals.
> But that approach will never win the hearts of UNIXy kernel devs
I don't think this is fair. The Linux kernel contains (as of writing) 17913052 lines of C. If you have a plan to rewrite 18 million lines of code and patch every single dependency, fine. But until then such concerns will dominate.
It's also worth noting that the Rust many people write is not the low-level stuff that C programmers and systems programmers write. See https://www.cl.cam.ac.uk/~srk31/research/papers/kell17some-p... for a better explanation of that.
Not to rewrite C into something else, but to get out of the current escalation of kernel exploits, now that Linux is a commodity OS, it gets targeted as such
https://events.linuxfoundation.org/events/linux-security-sum...
Hence Kernel Self Preservation Project, removing VLAs from kernel code, adding all possible ways of mitigation, and hopping that eventually tagged pointers like SPARC and ARM have become common across all supported hardware.
The charts shown by Google employees regarding exploits are quite telling.
Did it? Adoption seems to me to be a pretty silly metric given all the politics of language choice.
I hope this works out better than Microsoft Managed C++.
Since P/Invoke can't marshal C++ classes, Managed C++ gives you a way to expose such things. It's a very special-purpose chimera.
Its purpose was to make .NET interop easier with C++ libraries and has been quite successful at it, instead of having to play around with P/Invoke until everything works properly.
Regarding Windows, Microsoft is pretty much on the ongoing transition to C++, where these kind of issues are easier to tackle.
I bet their goals regarding Checked C have more to do with the adoption of UNIX (Linux/BSD) based technologies for Azure deployments and IoT devices like Azure Sphere.
We always just say "hmmm... right... this one may look a bit better than the previous one (or doesn't)" and we never bother to use them. After 20 years like this, we can conclude that we like C as it is and are not interested in extra layers.
This says a lot about how little the industry cares about software quality. You'd think a bunch of people who make a living out of automating tasks would happily use tools that automate the checking of an importnt source of bugs. Instead, many prefer to use tools that look simpler on the surface and then end up manually testing for the same kind of bugs over and over. So you have people writing in dynamically typed languages who rely on (manually written!) unit tests to check that their program is not mixing up types, and others using unsafe languages and then relying on tests to catch memory access violations. Inevitably, tests never catch all issues and you end up with type errors and segfaults in production.
In the case of a "safer C" I suspect it's largely because it never materialized. C still has its deficiencies relative to Rust, but Rust is the first viable competitor.
But the OSes they were tied to were killed by UNIX's adoption, just like the browser has helped JavaScript to take over areas of other languages.
However funding that work is a big problem, since it mostly benefits others.
Mentalities are very hard to change and we all know that only lousy programmers make errors in C.
It’s good to see people attempting this, though. I would think chances are good Microsoft will use it internally, and that already would be a win. Also, it’s a clang/LLVM extension. That makes it easier for third parties to start using it.
On the other hand, their choice to use Ptr<T> for familiarity with C++ requires users who want to keep their code compatible with C compilers that don’t support this a bit harder (it will require wrapping that in some macro, say _ptrTo(T)). That isn’t ideal for adoption in open source programs.
very few languages actually solve this completely. you'd need refinement types to do so.
I don’t know if Checked C will have the same success story but if devs can easily add this to their existing projects and it catches bugs, that seems like a win to me.