The fact that a C library can easily be wrapped by just about any language is really useful. We're considering writing a library for generating a UUID (that contains a key and value) for reasons that make sense to us and I proposed writing this in C so we could simply wrap it as a library for all of the languages we use internally rather than having to re-implement it several times. Not sure if we'll actually build this library but if we do it will be in C (I did managed to get the "wrap it for each language" proposal pre-approved).
Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy.
Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring.
Rust needs to demonstrate that it can do the kinds of work that C does in SQLite without a significant speed penalty.
The latter two of these points seem pretty subjective. I feel like they're written in a way that anyone could easily make an argument for or against Rust having fulfilled them today or literally at any point in the future. There's not really any way to refute an argument that something is "changing too fast", "not boring enough", or "hasn't demonstrated it can do the kind of work C does". I'd find them more compelling if they were expressed in a way it was actually possible to verify whether Rust reaches them or not at some arbitrary point in the future. Otherwise, people will inevitably just see them as a reflection of their existing opinion on whether it's wise or foolish for SQLite to use C rather than Rust.
> Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy.
Huh it's not everyday that I hear a genuinely new argument. Thanks for sharing.
One thing I found especially interesting is the section at the end about why Rust isn’t used. It leaves open the door and at least is constructive feedback to the Rust community
> All that said, it is possible that SQLite might one day be recoded in Rust. Recoding SQLite in Go is unlikely since Go hates assert(). But Rust is a possibility. Some preconditions that must occur before SQLite is recoded in Rust include:
- Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring.
- Rust needs to demonstrate that it can be used to create general-purpose libraries that are callable from all other programming languages.
- Rust needs to demonstrate that it can produce object code that works on obscure embedded devices, including devices that lack an operating system.
- Rust needs to pick up the necessary tooling that enables one to do 100% branch coverage testing of the compiled binaries.
- Rust needs a mechanism to recover gracefully from OOM errors.
- Rust needs to demonstrate that it can do the kinds of work that C does in SQLite without a significant speed penalty.
> Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring.
Talking about C99, or C++11, and then “oh you need the nightly build of rust” were juxtaposed in such a way that I never felt comfortable banging out “yum install rust” and giving it a go.
“None of the safe programming languages existed for the first 10 years of SQLite's existence. SQLite could be recoded in Go or Rust, but doing so would probably introduce far more bugs than would be fixed, and it may also result in slower code.”
Modern languages might do more than C to prevent programmers from writing buggy code, but if you already have bug-free code due to massive time, attention, and testing, and the rate of change is low (or zero), it doesn’t really matter what the language is. SQLIte could be assembly language for all it would matter.
This begs the question of why Rust evangelists keep targeting existing projects instead of focusing writing new, better software. In theory these languages should allow software developers to write programs that they would not, or could not, attempt using languages without automatic memory management
Instead what I see _mostly_ is re-writes and proposed re-writes of existing software, often software that has no networking functions, and/or relatively small, easily audited software that IMHO poses little risk of memory-related bugs
This is concerning to me as an end-user who builds their software from source because the effects on compilation, e.g., increased resource requirements, increased interdependencies, increased program size, decreased compilation speed, are significant
Being written in a small, fast, "old and boring" language may be part of what makes SQLite apealing. Another (related) part may be the thoughtfulness and carefulness of its author, e.g., "time, attention and testing"
The former, i.e., the author's "time, attention and testing", may matter more than the later, i.e., the author's language choice
As suggested by the top comment, in effect the author's language choice, by itself, may not matter with respect to the issue of "safety". If true, then even an "unsafe language" may not reduce the "safety" of SQLite^1
djb's software is also public domain and written in an "unsafe language", mostly the same "old and boring" one as used to write SQLite. Like SQLite it is appealing to many people and is found in many places^2
1. But the thoughtlessness and carelessness of an author, no matter what language they choose, is still relevant to "safety". In sum, "safety" is partly a function of human effort, e.g., "time, attention and testing", not simply the result of a language choice. Perhaps "safe" and "unsafe" are adjectives that can apply to authors as well as languages
Seems we live in a "cool" or "flashy" era or rewriting everything in Rust.
Dont get me wrong, despite not taking time to learn Rust at this time, I am aware that memory safety is the thing at this time. Yes... some software might do better being rewritten from C to Rust. However, there are other projects that have been worked on for years and years. Sqlite is an example of this. That quote above is 100%
Various GNU tools are being replaced. Dont just expect them to be 100% despite being "memory safe" -- they will have to fo through various tweaks to boost performance. With Rust likely (still) to go through some further changes, I am sure Linus will get frustrated at some points within the Linux Kernel. We shall see.
Point is - some things are just better left with their mature state. Though.. on the flip side, we are have to think about the younger generation. Will SQlite survive if it continues to use C? It's likely to be a language that each new generation will not bother, and focus on Rust, Zig.. or whatever comes out in the future.
I am just waiting for rewrite of Doom or Quake (I am sure they already exist if I can be bothered to search.. in Rust)
So, the argument for keeping SQLite written in C is that it gives the user the choice to either:
- Build SQLite with Yolo-C, in which case you get excellent performance and lots of tooling. And it's boring in the way that SQLite devs like. But it's not "safe" in the sense of memory safe languages.
- Build SQLite with Fil-C, in which case you get worse (but still quite good) performance and memory safety that exceeds what you'd get with a Rust/Go/Java/whatever rewrite.
Recompiling with Fil-C is safer than a rewrite into other memory safe languages because Fil-C is safe through all dependencies, including the syscall layer. Like, making a syscall in Rust means writing some unsafe code where you could screw up buffer sizes or whatnot, while making a syscall in Fil-C means going through the Fil-C runtime.
The point about bounds checking in `safe' languages is well taken, it does prevent 100% test coverage. As we all agree, SQLite has been exhaustively tested, and arguments for bounds checking in it are therefore weakened. Still, that's not an argument for replicating this practice elsewhere, not unless you are Dr Hipp and willing to work very hard at testing. C.A.R. Hoare's comment on eliminating runtime checks in release builds is well-taken here: “What would we think of a sailing enthusiast who wears his life-jacket when training on dry land but takes it off as soon as he goes to sea?”
I am not Dr Hipp, and therefore I like run-time checks.
It has async I/O support on Linux with io_uring, vector support, BEGIN CONCURRENT for improved write throughput using multi-version concurrency control (MVCC),
Encryption at rest, incremental computation using DBSP for incremental view maintenance and query subscriptions.
Time will tell, but this may well be the future of SQLite.
I think beyond the historical reasons why C was the best choice when SQLite was being developed, or the advantages it has today, there's also just no reason to rewrite SQLite in another language.
We don't have to have one implementation of a lightweight SQL database. You can go out right now and start your own implementation in Rust or C++ or Go or Lisp or whatever you like! You can even make compatible APIs for it so that it can be a drop-in replacement for SQLite! No one can stop you! You don't need permission!
But why would we want to throw away the perfectly good C implementation, and why would we expect the C experts who have been carefully maintaining SQLite for a quarter century to be the ones to learn a new language and start over?
Also, Rust needs a better stdlib. A crate for every little thing is kinda nuts.
One reason I enjoy Go is because of the pragmatic stdlib. On most cases, I can get away without pulling in any 3p deps.
Now of course Go doesn’t work where you can’t tolerate GC pauses and need some sort of FFI. But because of the stdlib and faster compilation, Go somehow feels lighter than Rust.
I don't want to sound cynical but a lot of it has to deal with the simplicity of the language. It's much harder to find a good Rust engineer than a C one. When all you have is pointers and structs it's much easier to meet the requirements for the role.
59 comments
[ 1.4 ms ] story [ 45.3 ms ] threadZig gives the programmer more control than Rust. I think this is one of the reasons why TigerBeetle is written in Zig.
https://news.ycombinator.com/item?id=28278859 - August 2021
https://news.ycombinator.com/item?id=16585120 - March 2018
Huh it's not everyday that I hear a genuinely new argument. Thanks for sharing.
- Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring.
- Rust needs to demonstrate that it can be used to create general-purpose libraries that are callable from all other programming languages.
- Rust needs to demonstrate that it can produce object code that works on obscure embedded devices, including devices that lack an operating system.
- Rust needs to pick up the necessary tooling that enables one to do 100% branch coverage testing of the compiled binaries.
- Rust needs a mechanism to recover gracefully from OOM errors.
- Rust needs to demonstrate that it can do the kinds of work that C does in SQLite without a significant speed penalty.
Talking about C99, or C++11, and then “oh you need the nightly build of rust” were juxtaposed in such a way that I never felt comfortable banging out “yum install rust” and giving it a go.
Modern languages might do more than C to prevent programmers from writing buggy code, but if you already have bug-free code due to massive time, attention, and testing, and the rate of change is low (or zero), it doesn’t really matter what the language is. SQLIte could be assembly language for all it would matter.
Instead what I see _mostly_ is re-writes and proposed re-writes of existing software, often software that has no networking functions, and/or relatively small, easily audited software that IMHO poses little risk of memory-related bugs
This is concerning to me as an end-user who builds their software from source because the effects on compilation, e.g., increased resource requirements, increased interdependencies, increased program size, decreased compilation speed, are significant
Public domain. No "copyleft" license needed
Being written in a small, fast, "old and boring" language may be part of what makes SQLite apealing. Another (related) part may be the thoughtfulness and carefulness of its author, e.g., "time, attention and testing"
The former, i.e., the author's "time, attention and testing", may matter more than the later, i.e., the author's language choice
As suggested by the top comment, in effect the author's language choice, by itself, may not matter with respect to the issue of "safety". If true, then even an "unsafe language" may not reduce the "safety" of SQLite^1
djb's software is also public domain and written in an "unsafe language", mostly the same "old and boring" one as used to write SQLite. Like SQLite it is appealing to many people and is found in many places^2
1. But the thoughtlessness and carelessness of an author, no matter what language they choose, is still relevant to "safety". In sum, "safety" is partly a function of human effort, e.g., "time, attention and testing", not simply the result of a language choice. Perhaps "safe" and "unsafe" are adjectives that can apply to authors as well as languages
2. https://ianix.com
This is obviously not analogous to Rust evangelism that targets projects written in C
The author claims the program is a clone of ack; ack is written in a "safe" language
https://geoff.greer.fm/ag/
https://beyondgrep.com/
[Cries in Ada]
Dont get me wrong, despite not taking time to learn Rust at this time, I am aware that memory safety is the thing at this time. Yes... some software might do better being rewritten from C to Rust. However, there are other projects that have been worked on for years and years. Sqlite is an example of this. That quote above is 100%
Various GNU tools are being replaced. Dont just expect them to be 100% despite being "memory safe" -- they will have to fo through various tweaks to boost performance. With Rust likely (still) to go through some further changes, I am sure Linus will get frustrated at some points within the Linux Kernel. We shall see.
Point is - some things are just better left with their mature state. Though.. on the flip side, we are have to think about the younger generation. Will SQlite survive if it continues to use C? It's likely to be a language that each new generation will not bother, and focus on Rust, Zig.. or whatever comes out in the future.
I am just waiting for rewrite of Doom or Quake (I am sure they already exist if I can be bothered to search.. in Rust)
So, the argument for keeping SQLite written in C is that it gives the user the choice to either:
- Build SQLite with Yolo-C, in which case you get excellent performance and lots of tooling. And it's boring in the way that SQLite devs like. But it's not "safe" in the sense of memory safe languages.
- Build SQLite with Fil-C, in which case you get worse (but still quite good) performance and memory safety that exceeds what you'd get with a Rust/Go/Java/whatever rewrite.
Recompiling with Fil-C is safer than a rewrite into other memory safe languages because Fil-C is safe through all dependencies, including the syscall layer. Like, making a syscall in Rust means writing some unsafe code where you could screw up buffer sizes or whatnot, while making a syscall in Fil-C means going through the Fil-C runtime.
SQLite is old, huge and known for its gigantic test coverage. There’s just so much to rewrite.
DuckDB is from 2019, so new enough to jump on the “rust is safe and fast”
I am not Dr Hipp, and therefore I like run-time checks.
It has async I/O support on Linux with io_uring, vector support, BEGIN CONCURRENT for improved write throughput using multi-version concurrency control (MVCC), Encryption at rest, incremental computation using DBSP for incremental view maintenance and query subscriptions.
Time will tell, but this may well be the future of SQLite.
At this point I wish the creators of the language could talk about what rust is bad at.
We don't have to have one implementation of a lightweight SQL database. You can go out right now and start your own implementation in Rust or C++ or Go or Lisp or whatever you like! You can even make compatible APIs for it so that it can be a drop-in replacement for SQLite! No one can stop you! You don't need permission!
But why would we want to throw away the perfectly good C implementation, and why would we expect the C experts who have been carefully maintaining SQLite for a quarter century to be the ones to learn a new language and start over?
But think about all those karma points here and on Reddit, or GitHub stars!
"Why is SQLite coded in C and not Rust?" is a question, which immediately makes me want to ask "Why do you need SQLite coded in Rust?".
One reason I enjoy Go is because of the pragmatic stdlib. On most cases, I can get away without pulling in any 3p deps.
Now of course Go doesn’t work where you can’t tolerate GC pauses and need some sort of FFI. But because of the stdlib and faster compilation, Go somehow feels lighter than Rust.