Unfortunately I don't understand what you just linked.
Often I see people resolving complexity with inscrutability but I'm afraid it doesn't help resolving the problem. Do you have a something written that explains what is this think you're sharing?
http://fuse.rupy.se (multiplayer system prooven with 350.000 customers and 5 years uptime)
TLDR: It's my own database I made after using MySQL, Postgres, Oracle for 10 years (and building my own ORM with GUI design tool: http://move.rupy.se/file/logic.html).
It out-scales everything on the planet but it has disk-space issues (that can be fixed with compression 100x but that adds complexity so I'll do it when my SSDs run out of space).
> Nope, the source code is linked and it's compact enough that you should be able to learn everything in about an hour.
I don't doubt that somebody who wants to learn this can do that in an hour as you say. But why would I invest one hour of my time to learn something I have no idea what it is !?
I skimmed through the readme and wiki and all I see is something vague about how async is good and how stellar the performances are.
The world is vast; the flood of information is a serious problem.
My piece of advice to you fellow HN member is that the burden is on you to explain why something is interesting. Otherwise people will flock to crappy complex stuff, because unfortunately those are better explained.
I'm not doing this for fame or money, I'm building a 3D MMO platform for eternity, so it's all or nothing:
Either all I did in life will disappear or I'm making the next open Minecraft in which case everyone will use it anyway. I'm fine with both outcomes.
Notch spent zero time documenting and he obfuscated the binaries, still more people know that reverse engineered source today than any other source to ever exist, maybe forever (unless I succeed)!
He's not saying you have an obligation to document your code, he's saying if you're going to add it to a discussion, you need to give us a one-sentence description of what it is so we can see how it is relevant and decide if we want to spend an hour getting to know the code. Basically, tell us what the value proposition of your comment is, otherwise it's just a link in a vacuum.
Most people have problems with persistence, it's a though subject until you realize most of the problem is fear caused by responsibility.
Most databases (SQL) where designed before the internet, that is a major flaw in them and this instead uses HTTP from the start and JSON in regular files. So it's simpler.
But it also has authentication, relational values, indexing and it's async. joint parallel distributed. So it's versatile and powerful without becoming complex and heavy!
It allows you to have coherent data at many locations around the world without a big complex monster like described in the article.
These "complex monsters" are there to give you guarantees: Transaction isolation, referential integrity, constraints, durability, atomicity. A RDBMS is complex so your application can be simple. It takes care of things so you don't have to. That's the whole point of it.
If you don't trust abstractions to handle the inherent complexity of data processing, you won't get anywhere. If you ignore decades of database research, you'll just end up reinventing the wheel badly.
What if you have a new requirement that goes against the grain of your data structures? Do you then have to refactor your entire application? How do you know if your algorithms are efficient, if they are using the right indexes? What happens in your platform if you have a software error during a write? Will you end up with incomplete data? What if two users try to change the same data at the same time, who will win, who will see what? What about hardware failures, power outages, or bit rot?
Essentially, you need to ask yourself these questions every time you make a change to your application, keeping in mind how all the parts of your program interact and what failure modes these interactions can cause. If you ignore these questions and don't have abstractions to take care of them, you will drown in complexity.
It's very simple, on every insert/update/delete the node that receives the command calls all the other nodes async. and waits until they have made the call before it makes it and then responds.
The id's are secure random longs, so collisions are rare.
You can see the code in Root.send(...) or see the live trace by going here and pressing the Make button: http://root.rupy.se/node?make&info
So if one node fails (or the network fails), some nodes will have the new value, while other nodes won't have it. I'm confused about why you call this "coherence".
The system only works for writes when every node is online.
It's a tradeoff that is well worth the simplicity.
To have fault tolerance for writes never really works and it's very wasteful. Specially when the nodes can be very reliable and report downtime for you to fix the issue manually if a node goes down.
In 5 years of operation my home hosted cluster has better uptime than all cloud providers: AWS, GCP etc. for writes and 100% uptime for reads.
This is a legit solution if your requirements allow it.
I'd advise you to not be too bullish on the universality of this principle of yours. Unfortunately real life is often more complicated than the scenario you describe and I wouldn't describe the solutions smart people came up with (various consensus algorithms or convergent replicated data types) as a waste of time.
1. The developer experience. This is crucial, developers want programmable things, everywhere. You can reap the benefits of automation.
2. Vendor lock in and platform power. There are three large players today and they add bit and pieces all the time; leading to vendor lock in. The cloud is even for mid-size projects much more expensive than raw hosting or owning the hardware.
Today, I would treat cloud as an expensive, but useful starter - to get you going, to try things out quickly. I would not bet my business on it, especially not, if I were a large company.
Ergo: Building everything in a "cloud native" mode restricts you where you can go from day one. Not good.
On the other hand, building everything "cloud native" can also enable some things that are very difficult to achieve otherwise. One thing that comes to mind is decoupling storage and compute, where you can use auto-scaling for the compute parts: great for OLAP / data-warehousing types of problems.
Regardless, I don't consider it merely a "useful starter" -- there are many mature businesses for which IT is not their core competency but still need to deal with complex architectures, and the cloud is very attractive to them.
I have to yet have people tell their most pressing need is auto-scaling. It's not, it's a straw man - and true, if you are Google or Facebook - but 90% of companies won't need that at all.
Of course it’s never the most pressing need, but it’s definitely a need in some circumstances. To completely disregard these use cases is not very useful.
I would disagree the cloud is more expensive. Yes you can purchase a server from Dell with a 5 year life expectancy for less than renting from AWS. But that is not the true cost.
Enterprises require high availability and disaster recovery. Start thinking multiple data centers to meet a RTO requirement after a regional failure with high speed network connectivity between the datacenters to meet your RPO requirements. Add off site backups and personnel at multiple locations and the cloud becomes very compelling.
And that is before you start optimizing your application for the cloud.
This is a really great collection of modern DB papers.
Folks here may also be interested in Andy Pavlo's series hosting developers from the team of many of these databases: https://youtube.com/c/CMUDatabaseGroup.
29 comments
[ 0.21 ms ] story [ 75.4 ms ] threadIt has 100% read uptime without complexity.
Often I see people resolving complexity with inscrutability but I'm afraid it doesn't help resolving the problem. Do you have a something written that explains what is this think you're sharing?
But here is the only documentation I have written so far:
https://github.com/tinspin/rupy/wiki/Storage
To see it in action with source:
http://fuse.rupy.se (multiplayer system prooven with 350.000 customers and 5 years uptime)
TLDR: It's my own database I made after using MySQL, Postgres, Oracle for 10 years (and building my own ORM with GUI design tool: http://move.rupy.se/file/logic.html).
It out-scales everything on the planet but it has disk-space issues (that can be fixed with compression 100x but that adds complexity so I'll do it when my SSDs run out of space).
It uses this old goodie for authentication: https://datatracker.ietf.org/doc/html/rfc2289
I don't doubt that somebody who wants to learn this can do that in an hour as you say. But why would I invest one hour of my time to learn something I have no idea what it is !?
I skimmed through the readme and wiki and all I see is something vague about how async is good and how stellar the performances are.
The world is vast; the flood of information is a serious problem.
My piece of advice to you fellow HN member is that the burden is on you to explain why something is interesting. Otherwise people will flock to crappy complex stuff, because unfortunately those are better explained.
Either all I did in life will disappear or I'm making the next open Minecraft in which case everyone will use it anyway. I'm fine with both outcomes.
Notch spent zero time documenting and he obfuscated the binaries, still more people know that reverse engineered source today than any other source to ever exist, maybe forever (unless I succeed)!
Most databases (SQL) where designed before the internet, that is a major flaw in them and this instead uses HTTP from the start and JSON in regular files. So it's simpler.
But it also has authentication, relational values, indexing and it's async. joint parallel distributed. So it's versatile and powerful without becoming complex and heavy!
It allows you to have coherent data at many locations around the world without a big complex monster like described in the article.
The only downside is disk space!
What you don't understand; always hurts in the end.
Also there is no perfect, those guarantees are lies.
What if you have a new requirement that goes against the grain of your data structures? Do you then have to refactor your entire application? How do you know if your algorithms are efficient, if they are using the right indexes? What happens in your platform if you have a software error during a write? Will you end up with incomplete data? What if two users try to change the same data at the same time, who will win, who will see what? What about hardware failures, power outages, or bit rot?
Essentially, you need to ask yourself these questions every time you make a change to your application, keeping in mind how all the parts of your program interact and what failure modes these interactions can cause. If you ignore these questions and don't have abstractions to take care of them, you will drown in complexity.
Still my conclusion is to write everything from scratch and slowly but surely build something good.
You need to own your future, that requires responsibility and respect, without that your life is just slavery.
http://move.rupy.se/file/wheel.jpg
The id's are secure random longs, so collisions are rare.
You can see the code in Root.send(...) or see the live trace by going here and pressing the Make button: http://root.rupy.se/node?make&info
It's a tradeoff that is well worth the simplicity.
To have fault tolerance for writes never really works and it's very wasteful. Specially when the nodes can be very reliable and report downtime for you to fix the issue manually if a node goes down.
In 5 years of operation my home hosted cluster has better uptime than all cloud providers: AWS, GCP etc. for writes and 100% uptime for reads.
I have blogged about how to make a 100% write uptime database here: http://talk.binarytask.com/task?id=4662372011677848884
But my conclusion is that it's a waste of time.
I'd advise you to not be too bullish on the universality of this principle of yours. Unfortunately real life is often more complicated than the scenario you describe and I wouldn't describe the solutions smart people came up with (various consensus algorithms or convergent replicated data types) as a waste of time.
Naming is not my best skill.
Yup, I learned everything I need to know about this project in 5 minutes:
https://github.com/tinspin/rupy/blob/7d7972932101b0bbe473a4f...
I hope you're successful in your Minecraft replacement, as I've enjoyed playing with some of the hobby projects it inspired
Sometimes I just run out of patience and don't care if something fails, in this case it was copy pasta though.
1. The developer experience. This is crucial, developers want programmable things, everywhere. You can reap the benefits of automation.
2. Vendor lock in and platform power. There are three large players today and they add bit and pieces all the time; leading to vendor lock in. The cloud is even for mid-size projects much more expensive than raw hosting or owning the hardware.
Today, I would treat cloud as an expensive, but useful starter - to get you going, to try things out quickly. I would not bet my business on it, especially not, if I were a large company.
Ergo: Building everything in a "cloud native" mode restricts you where you can go from day one. Not good.
Regardless, I don't consider it merely a "useful starter" -- there are many mature businesses for which IT is not their core competency but still need to deal with complex architectures, and the cloud is very attractive to them.
Enterprises require high availability and disaster recovery. Start thinking multiple data centers to meet a RTO requirement after a regional failure with high speed network connectivity between the datacenters to meet your RPO requirements. Add off site backups and personnel at multiple locations and the cloud becomes very compelling.
And that is before you start optimizing your application for the cloud.
Folks here may also be interested in Andy Pavlo's series hosting developers from the team of many of these databases: https://youtube.com/c/CMUDatabaseGroup.