Show HN: A Database Written in Golang (github.com)
Recently created a minimal persistent relational database in Go. Main focus was on implementing & understanding working the of database, storage management & transaction handling. Use of B+ Tree for storage engine(support for indexing), managing a Free List (for reusing nodes), Support for transactions, Concurrent Reads.
Still have many things to add & fix like query processing being one of the main & fixing some bugs
Repo link - https://github.com/Sahilb315/AtomixDB
Would love to hear your thoughts
79 comments
[ 2.9 ms ] story [ 135 ms ] threadCool accomplishment in and of itself but hard for anyone here to really give you any criticism or feedback without understanding where it excels and how to work with it.
I think this clearly describes what was the goal.
Less than a minute and I know how to use it. It's not complicated. The source code is available and fairly easy. If you can't figure out how to use it in a trivial amount of time, you aren't going to be able to offer anything of value. When did HN go from being about interesting stuff to making bold, ignorant statements like "There is no way to look at this codebase and give any kind of feedback one way or the other without that understanding."
You should know better.
Currently there's no way to connect to it like you would a normal SQL db, but you could embed it like you would sqlite.
Don't let your employer know...
We're not at work here. Low effort posts are allowed though not encouraged. Implying that making a low effort post somehow makes you not a developer ("you're a 'dev' but don't seem out tests?") is it's own low effort post, while also being hurtful.
Please, let's keep things kind, focused, and clear.
I do find the source to be well organized and quite readable. Especially if you runs the commands in the cli and then trace how they are each implemented.
Obviously OP wrote it to understand database theory & implementation.
From a learning perspective, nice project for OP, for 'advertising' it, i'd prefer the "what's better than the alternatives" instead of focusing on the language.
Specially relevant, because it used to be that writing database engines used to be considered systems programming, and we all know managed languages cannot be possible used for such tasks. /s
I share the same goal and am working through the material after working through Codecrafters' "Build your own SQLite" [1]. Good luck!
I apologize in advance for mistakes (formatting, et cetera). I just registered this account to point you toward resources I found helpful.
[0] https://build-your-own.org/database/
[1] https://app.codecrafters.io/courses/sqlite/overview
Hope this gives you some options.
https://github.com/Arminas42/build_your_own_database
https://github.com/search?q=codecrafters+sqlite-java+languag...
One thing I’d add to the readme is an example of how you’d use the database in an example application. From the docs, it’s clear that this isn’t a sql database (yet?), so it would be good to have an example to see how to use the database.
It might also be nice to have a description of what happens when you insert or get a record, so others can learn from the code too. Or can you comment here about what your favorite part of the code is? What did you figure out that you didn’t know before? If you’re using the project to learn about databases, what have you learned so far?
Learnt about B+ Trees, transactions, managing concurrent reads & data presistence
https://github.com/MIT-DB-Class/go-db-2024 https://dsg.csail.mit.edu/6.5830/
Some recent ones:
https://www.reddit.com/r/databasedevelopment/comments/1hyig8...
https://www.reddit.com/r/databasedevelopment/comments/1ha5cc...
https://www.reddit.com/r/databasedevelopment/comments/1dqgms...
https://www.reddit.com/r/databasedevelopment/comments/1ix5dz...
https://www.reddit.com/r/databasedevelopment/comments/18knod...
https://www.reddit.com/r/databasedevelopment/comments/1h3w70...
https://www.reddit.com/r/databasedevelopment/comments/1gk18n...
https://www.reddit.com/r/databasedevelopment/comments/1bemf9...
https://www.reddit.com/r/databasedevelopment/comments/1iw6cx...
https://github.com/FireScroll/FireScroll/
https://github.com/danthegoodman1/DurableStreams (brand new)
https://github.com/danthegoodman1/ObjectKV
https://github.com/danthegoodman1/WriteAhead
https://github.com/danthegoodman1/icedb (most popular)
https://github.com/danthegoodman1/Percolators (kind of a DB on top of a DB, technically just transactions tho)
there are others that might not fit exactly what you're looking for
https://github.com/thomasjungblut/go-sstables
congratz !
At this point i think its important to mention that its nativly an in-memory graph database/storage.
At some point (4-5 years ago) i decided to change the way i handle it. Since im using the storage in many private projects, and there was no real traction (while also maintaining a database is quite some work) i decided i gonne change the way i handle it away from a standalone database to a library that works as an in-memory graph storage.
Also i added a custom query language including builder (the queries are 100% json compatible so you can in theory build queries from whatever language and transmit it via json.
The library is released under : https://github.com/voodooEntity/gits
Currently im completly reworking the query parser but since this is a privat project and im the only maintainer it may take some to to finish it. The update will takle some smaller bugs in the query parser that can occur on very complex nested queries and also will optimize query speeds. I cant provide a date when the update will occure i only can say it will (because i need it myself).
Finally said: The strength of gits is to be very fast (hence in memory handling) and easy to use from golang. Its threadsafe by default.
https://www.sqlite.org/lemon.html
You could probably use something like participle, but you'd have to translate the grammar.
https://github.com/alecthomas/participle
You watched CMU's intro to Database sytems, right? It's really good and thorough. It will save you of some common pitfalls and can help you navigate the trade-offs.
https://www.youtube.com/watch?v=otE2WvX3XdQ&list=PLSE8ODhjZX...
I hope some company that heavily relies on Postgres sponsors a project to replace the optimizer with a more modern implementation. I feel like for any large applications with olap style queries (even if on the OLTP database) will benefit incredibly from that work. SQL Servers optimizer was light-years ahead of Postgres when I worked with it last over a decade ago.
It takes me back to my school years. I never got as far as you (not by any long stretch actually), but I did enjoy creating the storage layer of a database from scratch. To actually have to deal with, instead of just think of, all the edge cases, is quite the transformative experience.
As a humble suggestion, since it seems your goal is to understand how relational databases work and not necessarily to write a new database that will compete with others, maybe don't make it an SQL one? We've got enough of those, and not enough of the others. Would be nice to have a new relational DB using Tutorial D as its language for example.
Keep hacking!
btw, README says Go 1.17+ is required, but go.mod says 1.23+.
I created an email verifier project in Go to check MX records, SMTP server status, and existence in the disposable list.
Project - https://github.com/hsnice16/email-verifier
- TiDB by PingCAP
- Vitess by PlanetScale
Both are basically only the SQL part as TiDB uses TiKV (written in Rust) and Vitess uses MySQL.
For those who want to implement a database in Go but without having to implement a network protocol there is go-mysql, which allows you to do this: https://github.com/go-mysql-org/go-mysql/blob/master/cmd/go-... As demonstration I created a networked SQLite: https://github.com/dveeden/go-mysql-examples/blob/main/go-my...
Both TiDB and Vitess have parsers that can be used outside standalone. So if you only want to implement your own on disk format, this can help.
Note that I'm working for PingCAP on TiDB and I'm also a co-maintainer for go-mysql.
Great lectures on relational databases and more.