Related - I was looking around for tutorials on how to write a database and stumbled across this tutorial on writing a SQLite clone, which looks very promising: https://cstack.github.io/db_tutorial/
These days when everything can live in RAM, just focus on writing a good, durable transaction log and use an in-memory data structure. That way you won't waste all your time re-implementing a B-tree and can focus instead on the query engine and data model. At least if you're doing it for your own gratification (and even if you're not, it may make sense to build the initial prototype that way.)
Just my 2¢
—signed, someone who went down that road a decade ago
As someone who works on an in-memory DB, good advice. You can go far, but it’s worth designing your system with the notion that data could be paged in/out behind the scenes.
Afaik it should be fully transparent; can you even reliably that it occurred?
I’m strictly asking, to be clear; I’m vaguely remembering an article about redis(?), describing db page-management as a poor design, as it ends up being unreliable to detect, and competes with the OS to neither’s benefit (though a similar argument is often made as to why you shouldn’t micro-optimize C — the compiler will win [except where it doesn’t])
i think the key issue here is managing the asynchrony. which is sadly still a bit of a pain.
you're absolutely right that you shouldn't have two page caches. but its entirely likely that the database has explicit lifetime and policy information that would make it a more effective cache owner. but sure, if you're just going to do a blind LRU, by all means leave it to the kernel.
Andy Pavlo from the CMU Database Group has excellent courses on Database design on YouTube. Large time commitment, but he goes into great detail about internals and design tradeoffs, and is very knowledgeable about which database systems use which approaches. The intro course is about traditional databases, while the advanced course is about modern in-memory databases.
12 comments
[ 2.9 ms ] story [ 29.1 ms ] threadJust my 2¢
—signed, someone who went down that road a decade ago
Afaik it should be fully transparent; can you even reliably that it occurred?
I’m strictly asking, to be clear; I’m vaguely remembering an article about redis(?), describing db page-management as a poor design, as it ends up being unreliable to detect, and competes with the OS to neither’s benefit (though a similar argument is often made as to why you shouldn’t micro-optimize C — the compiler will win [except where it doesn’t])
you're absolutely right that you shouldn't have two page caches. but its entirely likely that the database has explicit lifetime and policy information that would make it a more effective cache owner. but sure, if you're just going to do a blind LRU, by all means leave it to the kernel.