For new products I always recommend the same thing: name your competitors and use cases. Compare them in a table; don't make the user think. Who should and should not use your product? Being in-memory and distributed, I imagine your competitors include Ignite and Hazelcast?
I read the title as CursorDB, which would have made a nice DB name.
I don't like MongoDB or Couchbase. I dislike their query languages and designs in general. I built the DB for my own liking and use cases. If others like it that's great, if not it doesn't hurt me. I made it open source to follow the process and I'm not looking for any profit or gain for doing any of it. I'd like honest opinions obviously but that's all. Eventually if people really like the DB and it requires ALL my time then I'll try to expand on the cloud offering because I am mighty passionate about the project and want to see it being used as much as possible and easily as possible!
Hi esafak. I don't really care for competitors. In regards to use cases there are so many! Currently I am working on few tutorials on how to build different applications like a Hackernews clone with a real-time frontend. These tutorials are to bring light to the functionalities of the database and give users ideas. Yeah I don't know those companies, nor do I care to. I built and designed the database the way I wanted to use a database, simple as that. I have my own use cases for a transactional email service relaying what comes through an Observer to client webhooks for example.
You have competitors whether you like it or not -- unless you don't care for users, either. A sibling commenter asked "Why not just use Couchbase?" If your answer is "figure it out yourself; I already wrote documentation" you are shooting yourself in the foot. Documentation is not marketing. Marketing means helping potential users understand why they should use your product rather than something similar.
I agree. I very much care for the users of the database because their experience if everyone elses in a way. My reply was pretty simple lol. You build something YOU like and have a need for there's a chance others will as well. If you just build something that you don't like and have no need for, others probably wont as well.
CouchDB's design to me personally is not that great. I don't like the replicated design or the query language. To boot I don't like complexities. I try to avoid them when designing something complex. CursusDB core code is under 6k lines because of that reason. I also wanted to create something I want to use. Simple. Setting up a secure, reliable and powerful database shouldn't be complex and I made certain of that when designing CursusDB. CursusDB's query language as well is very simplified yet powerful because of my complexities ideology.
I get that you need to differentiate your project from others if you want to find adoption, and one way of doing that is to compare it to who you feel are competitors. I do feel that the word “competitor” should rather be “alternative” since I don’t think this or many other projects are trying to compete, but instead just trying to exist in their own way and do their own thing. If someone likes it enough but finds a pain point they might contribute that change, too.
Also, part of the beauty of open source is not trying to conform or constantly compare your project to others. It’s awesome that such diversity exists in project communities and the cross-pollination of ideas resulting from it.
people are simple LLMs. triggering comparisons does a lot of magic that paragraphs of flavor text doesn't do, especially if you're trying to get momentum.
it doesn't really matter what you label those comparisons.
Sure. I don’t feel that every larger project released publicly should seek momentum or recognition to be considered a contribution to software, or that this has to be a goal of any software aiming to grow. Often, folks that are familiar enough with specific domains are able to draw their own comparisons from reading basic docs.
If a project was aiming to get wider adoption and traction/momentum then yes, I agree. It doesn’t seem that this is an immediate goal of this project though.
No it's not an immediate goal. It's just a good feeling building something useful to you and it be useful to others that's why I strive to perfect it in every way whether it be documentation, code, design.. doesn't matter! I'll also obviously try to share it with the world in every way I can! :)
Hey corobo, thank you for that! I will surely work on this and post either on the Github Repo Wiki or on the website, or both! Really it's an alternative to a combination of MongoDB, MySQL and Redis.
No not using Jepsen. I've written custom tests in GO running thousands of concurrent client connections on local and remote clusters with multiple nodes and node replicas I test consistency, crisis, recovery, etc. The database hasn't shown me personally any flaws whatsoever. Anyone can go ahead and write their own Jepsen tests if they please.
Maybe down the line if people REALLY want it I'll do it but I think I'll just post my tests instead. The thing is I'd rather make a video as you can't test power outages and so forth with just a script. I've done this using hardware locally and I can't find flaws. I'm not trying to make it seem any which way. I'd rather have more people write their own tests as well, why not!
Well, Jepsen has a reputation for finding problems in pretty much every database they test and has become a sort of a benchmark by now. Many people (including myself) won't consider a distributed database if it hasn't been put through Jepsen.
The only exception I know of is FoundationDB, which was written pretty much test-first, and about which Kyle Kingsbury said that their tests are likely better :-)
Thank you for the information @jwr! I've looked into it. I'm not trying to put money down on anything currently. I've tested, and I mean tested hundreds of hours possibly even thousands from the start to the now, every feature, multiple ways. I wouldn't deploy releases and mention features as an actual feature if I didn't test every single thing beforehand. I am thorough, I do crazy amounts of sanity checks on everything, I don't just claim things. I myself cannot do everything in the span of not even 60 days except test everything as I went on and continue to. I post about it on the Github. CursusDB must be manually and driven tested. Very easy to automate but very easy to miss steps.
To add, I'm not saying I wont do it. I just don't have funds for it currently. I'll save up and make sure to get it completed for the sanity of the many like you :D As stated previously I truly on every aspect test every method before implementation, I test features, I test REAL failures and scenarios like actually having a box running and power shutting down during a large transaction on nodes,I'm saying not virtually actually shutting power down and seeing if the system can persist. I did a lot and continue to for my own confidence. If it's not enough for you and others I do completely understand your reasoning hence mine to save for the Jepsen.
I think this has a place. I’ve found myself wanting something similar in the past though found other solutions. It seems you have similar reservations toward Mongo and similar software so it’s unlikely it will follow their path. I remember so many folks jumping on Mongo and hailing it as the holy grail of storage, dumping everything into it without much thought. I also remember many of those same people paying dearly for doing that. The older Mongo-based projects I’ve worked on were unmaintainable swamps of hacks and pain. Some took the time to care and used Mongo in a way that was sustainable.
I've seen this project, it's very cool! I'm all for it if you want to use it. I just didn't want to go the route of SQLite + a layer. SQLite as well is not encrypted by default. There's a lot to the comparison but it's not the same.
rqlite creator here. Wishing you success with your new project.
Before we jump into what is encrypted, and what is not, what are the threat vectors we are actually concerned about?
With rqlite all traffic between the nodes can be encrypted, as can be the HTTP API. Yes, the data at rest is not, but there are many ways to protect that[1]. One way would be to use an encrypted file system, perhaps.
Encrypted data-at-rest is a valid request, but what use cases would it address?
Hello, an absolute pleasure sir! Great work on rqlite.
Thank you very much.
I never liked the idea of having any database storing my data in a way which can be seen in plain sight regardless of permissions and so forth. Servers get broken into very often and protecting the data within the files is important to me.
I've created another tool called qenc that may help the at rest problem if someone has a problem with it. qenc.org
With a CursusDB node on signal or automatic backup the system will encrypt and compress block by block. Using ChaCha and DEFLATE just to put that out there, it's pretty cool stuff!
(Random thought: Someone should create an online list of all known databases, all known versions of those databases, Lines-Of-Code (LOC) counts, the language that the database was written in, and the OS'es that it is available on. Also, same thing but for database connectors/drivers/APIs written to interface to different languages -- show me those languages for each connector... Or, maybe combine these these ideas into a website which allows people to search for databases by OS support, LOC, language written in, languages connector/APIs support, etc., etc.
That is, search for a database based on any and all attributes relative to a database).
I am working on a video tutorial on how to build a FULL production grade Hackernews clone with real-time front end updates, authentication, automatic backups, replicas, TLS, and yes we will be using CursusDB as the database system using an Observer and Websockets to relay database activity in realtime as shown here https://github.com/cursusdb/cursusdb-observer-node-ws-exampl...
In the video I will deploy the final application to Cloud Run and keep it online for you all to play with. I will respond to this with the video!
Just want to add this bit so I've recently finished node's transaction persistence. Mind you I've actually tested power outages at my own house. Don't wanna get into too much details about that lol!!
But what I've found is that the .qqueue works as expected. If I make thousands of update requests to one document and the power goes out the CursusDB node system persists all updates on startup. Pretty cool stuff.
Ah buzzwords. I saw a database the other day with hundreds of buzzwords on their website and posts all over and nothing clicked at all actually playing with it. False marketing. I'm not into that. Could lead to disaster.
The query language is interesting. It has like grammar guide lines? No upper-case testing upper-case the cluster does not accept the queries. I like it though pretty strict.
I’d say it leaves you cursed with joy but I may be biased as I created it. I really like the name Cursus and I found it one day just translating English to Latin, just randomly and really liked the meaning of it and it’s super easy to remember so I kept it in my back pocket until I had something to use it for and when designing CursusDB it sprung up whilst drawing out the initial algorithms.
Interesting product. I'm reading through the source and... Woah... it is all in one single file with zero unit tests.
Of course there are many ways to skin a cat, this seems like a very odd choice for organizing a database codebase. Some of the methods like cluster.HandleClientConnection() are crazy long for loop + if/else + switch + goto!
I know that you've said this is just kind of in your best interests codebase, but if I was planning on making any sort of dependencies on this project, I'd probably shy away from it because it seems like unmanageable spaghetti.
It's not even 3,000 lines. That's not a lot of spaghetti.
cluster.HandleClientConnection looks like a standard network service event loop, except the code is inline instead of spread across cluster.HandleSyncData, .HandleDelete, etc.
There is a lot of them so I will create a new repository for them, possibly; Or commit them to the core repo. I’ll see. I did have tests within the main repo at one point but I took them out before v2.0.0.
I mean you feel pretty good about removing them after hundreds of automated tests runs and hundreds of hours of manual testing. I could be biased in that I will put back tests for the sanity of you all :). I do understand for contribution sake to have them there.
I don't necessarily look at tests as something for the past, I look at them for the future.
If I want to add a feature or change some code around to fix something (like, say, migrate from one giant file to a bunch of individual files), then I want some sort of assurance that I haven't broken anything. That's what tests give you.
If you have someone contributing to your project, who doesn't necessarily have the whole 3,000 lines in their head, then tests give them the ability to make those contributions safely, without breaking anything.
It is all about doing things as standard as possible. Lowest common denominator for every one looking at the code base. As soon as you start to do things strangely in code (like a big single file with no tests), then people start to curse you. "What were they thinking when they wrote this?!!#@#!@#". I try to avoid those situations.
Tests also help document "the why". They become a history for changes over time. I could start to fix something that seems like a bug or strange design decision and look at the tests and realize that things were done intentionally... and give me the "why" they were done that way.
I agree with you! I'm not going to lie to you the tests are rather different than most. I don't test things say like setting up an initial cluster or node. Those are such basic functionalities that can be manually tested. The system is tested mainly and I say mainly on:
- handling the protocol in which the client speaks to the cluster (no need)
- handling the protocol in which the cluster speaks to the node (no need)
- the query language (when testing the query language we are testing the above 2)
- finding any gaps in mistyping a query that can cause a cluster exception of any kind (this has been insanely gone through, I can't even lol)
- concurrency with consistency. One test for example is updating multiple documents multiple times through 100 client connections.
I don't see the structure of any main method changing. I just want to add tests must be driven meaning.. With a client. You don't test this system like you do a module in GO. You don't just write one test for CountLog(). If I absolutely had to sure.
When you test the client>cluster>node(s). So you always do driven testing as a client or multiple clients.
Just because you don’t see tests doesn’t mean that there weren’t any before or now. Really not unmanageable at the size of the code base for me. I do agree that method is long but I didn’t want to spread that and a few others across many others it was a design choice. I agree it could be improved but to not want to implement it because of that is your choice. The code is commented and continuously gets more comments and it’s very easy to follow if you read procedurally.
These keywords are the crux of the issue. What you're expecting is that third parties can read your mind. You have the whole codebase floating in your head so you understand where things are at and how it is laid out. If I'm working with others, or expecting people to pick up my project and either use it or contribute to it, I generally try to make it as easily understood as possible. Common sense is to break the file into manageable sections.
Again, tons of ways to skin a cat... but this is a huge red flag, for me.
Interesting, well understood. I’ll try my best to rethink that logic and get it even shorter. I have no problems with that!! Thank you for taking a look.
I really tried to make everything as understandable as I could truly and I will continuously strive to do so. Mind you I only built it in around 60 days so everything will come :)
Please don't get me wrong, this is just hopefully constructive feedback from my personal opinion having written A LOT of mission critical golang code.
Again, so many ways to skin a cat... if you can convince other people to contribute to your project as it is, then I'm definitely worthy of ignoring. =)
What do you want me to say here? It is 3000 lines, in a single file and no tests. I just said "seems like".
In order to comprehend it, I'm going to have to parse the whole thing into memory and figure out what it does. Not impossible, but if I'm looking to just fix a specific bug or contribute a new feature, I'd much rather go to the specific file that feature/bug is in, parse a page or two of code on my screen and work from there. It is just how I work.
I'm sorry that you have to deal with something that sounds worse. If it was me, I'd spend a lot of extra-curricular time trying to clean up the mess. In return, that would help me get an even better sense of the code.
OP, would you mind sharing your background (sorry if I missed this somewhere?). Like what kind of experiencer you had prior to this which got you to write a whole database from scratch?
69 comments
[ 142 ms ] story [ 1275 ms ] threadhttps://www.youtube.com/watch?v=n1t5Iexx4UY <-- Automatic Recovery, Backups, Replica Sync, Lost Reconnect Demonstration
I've put lot's of effort into the documentation and continuously try to perfect it! https://cursusdb.com/documentation
I read the title as CursorDB, which would have made a nice DB name.
Also, part of the beauty of open source is not trying to conform or constantly compare your project to others. It’s awesome that such diversity exists in project communities and the cross-pollination of ideas resulting from it.
it doesn't really matter what you label those comparisons.
think of it like LLM optimization.
If a project was aiming to get wider adoption and traction/momentum then yes, I agree. It doesn’t seem that this is an immediate goal of this project though.
Document-oriented databases might be reluctant to accept the premise that tables are better for users
'cause "alt1 vs alt2 vs alt3" is what I'm searching
The only exception I know of is FoundationDB, which was written pretty much test-first, and about which Kyle Kingsbury said that their tests are likely better :-)
Thank you for your response.
Here is a PR to fix it. Just a simple parseInt. https://github.com/cursusdb/cursusdb-web/pull/1
[1]: https://rqlite.io
Before we jump into what is encrypted, and what is not, what are the threat vectors we are actually concerned about?
With rqlite all traffic between the nodes can be encrypted, as can be the HTTP API. Yes, the data at rest is not, but there are many ways to protect that[1]. One way would be to use an encrypted file system, perhaps.
Encrypted data-at-rest is a valid request, but what use cases would it address?
[1] https://rqlite.io/docs/guides/security/
I never liked the idea of having any database storing my data in a way which can be seen in plain sight regardless of permissions and so forth. Servers get broken into very often and protecting the data within the files is important to me.
I've created another tool called qenc that may help the at rest problem if someone has a problem with it. qenc.org
With a CursusDB node on signal or automatic backup the system will encrypt and compress block by block. Using ChaCha and DEFLATE just to put that out there, it's pretty cool stuff!
Welcome CursusDB!
(Random thought: Someone should create an online list of all known databases, all known versions of those databases, Lines-Of-Code (LOC) counts, the language that the database was written in, and the OS'es that it is available on. Also, same thing but for database connectors/drivers/APIs written to interface to different languages -- show me those languages for each connector... Or, maybe combine these these ideas into a website which allows people to search for databases by OS support, LOC, language written in, languages connector/APIs support, etc., etc.
That is, search for a database based on any and all attributes relative to a database).
Anyway, welcome CursusDB!
Thank you by the way! Great to see enthusiasm for new database systems!
Also, excellent link!
In the video I will deploy the final application to Cloud Run and keep it online for you all to play with. I will respond to this with the video!
Cheers
But what I've found is that the .qqueue works as expected. If I make thousands of update requests to one document and the power goes out the CursusDB node system persists all updates on startup. Pretty cool stuff.
Of course there are many ways to skin a cat, this seems like a very odd choice for organizing a database codebase. Some of the methods like cluster.HandleClientConnection() are crazy long for loop + if/else + switch + goto!
I know that you've said this is just kind of in your best interests codebase, but if I was planning on making any sort of dependencies on this project, I'd probably shy away from it because it seems like unmanageable spaghetti.
It's not even 3,000 lines. That's not a lot of spaghetti.
cluster.HandleClientConnection looks like a standard network service event loop, except the code is inline instead of spread across cluster.HandleSyncData, .HandleDelete, etc.
The lack of tests though...
If I want to add a feature or change some code around to fix something (like, say, migrate from one giant file to a bunch of individual files), then I want some sort of assurance that I haven't broken anything. That's what tests give you.
If you have someone contributing to your project, who doesn't necessarily have the whole 3,000 lines in their head, then tests give them the ability to make those contributions safely, without breaking anything.
It is all about doing things as standard as possible. Lowest common denominator for every one looking at the code base. As soon as you start to do things strangely in code (like a big single file with no tests), then people start to curse you. "What were they thinking when they wrote this?!!#@#!@#". I try to avoid those situations.
Tests also help document "the why". They become a history for changes over time. I could start to fix something that seems like a bug or strange design decision and look at the tests and realize that things were done intentionally... and give me the "why" they were done that way.
I don't see the structure of any main method changing. I just want to add tests must be driven meaning.. With a client. You don't test this system like you do a module in GO. You don't just write one test for CountLog(). If I absolutely had to sure.
When you test the client>cluster>node(s). So you always do driven testing as a client or multiple clients.
These keywords are the crux of the issue. What you're expecting is that third parties can read your mind. You have the whole codebase floating in your head so you understand where things are at and how it is laid out. If I'm working with others, or expecting people to pick up my project and either use it or contribute to it, I generally try to make it as easily understood as possible. Common sense is to break the file into manageable sections.
Again, tons of ways to skin a cat... but this is a huge red flag, for me.
Again, so many ways to skin a cat... if you can convince other people to contribute to your project as it is, then I'm definitely worthy of ignoring. =)
In order to comprehend it, I'm going to have to parse the whole thing into memory and figure out what it does. Not impossible, but if I'm looking to just fix a specific bug or contribute a new feature, I'd much rather go to the specific file that feature/bug is in, parse a page or two of code on my screen and work from there. It is just how I work.
I'm sorry that you have to deal with something that sounds worse. If it was me, I'd spend a lot of extra-curricular time trying to clean up the mess. In return, that would help me get an even better sense of the code.