It's not a simple sharing. It's actually share-nothing architecture and doing multi-key operations atomically is pretty complicated thing. I used paper from 2014 to solve this problem
In Redis cluster the client needs to be connected to all shards and manage those connections. Multi-key operation on different slots are not supported etc... Maintaining a cluster is not a fun responsibility. DF saves you in most cases from the need to grow horizontly which should be much simpler to maintain and work with.
it is architecturally different. DF has the orchestrator underneath. The general. The guy that handles all the incoming requests and makes sure they execute atomically. Redis cluster is an orchestra without the orchestrator. Everyone to himself. It's share-nothing that is exposed outside. To actually hide the complexity in a *transparent way* is a difficult problem. Researches write articles on how to solve this problem efficiently.
Unfortunately unlikely. Shared nothing architecture works with messaging. So threads send each other messages like "give me the contents of that key" or "update this key". Operations like SET/GET will require a single message hop. Operations like RENAME require 2 hops. Transactions and Lua scripts will require number of hops as number of operations in the transactions. When it's in the same process - the latency penalty in negligible. But between different machines - you will feel it. But who knows, AWS already have very cool UDP-like protocol with very low latency... if this will become the standard inside cloud environments maybe we can create a truly distributed memory store that spans multiple machines transparently.
Yes, VLL algorithm kinda solves the throughput problem. You will still have some issues with fault-taulerance - i.e. what you do if a machine does not answer you.
For intra-process framework it's not an issue (as long as we do not have deadlocks).
I think the Background section[0] is pretty helpful. One paper they cite is from 2014, another from 2020. The use of io_uring as well is also somewhat novel.
Redis and memcache are in memory key value storage. Writing to disk is not a primary function of those systems, it’s only used for taking snapshots or backup of the data. io_uring isn’t used as the core functionality and thus that alone wouldn’t make DF “modern”.
Yes, we use io_uring for networking and for disk. io_uring provides a unified interface on linux to poll for all I/O events. Re disk - we use it for storing snapshots. We will use it for writing WALs.
And we have more plans for using io_uring in DF in the future.
I had the same thought, but then I realized IBM DB2 was 16 years old when Redis was released, close to Redis' age now. There is a whole generation of programmers that may consider MongoDB a "legacy database".
Yes, I can confirm that Memcached can reach similar performance as DF. However, one of the goals of DF was to combine the performance of Memcached with versatility of Redis. I implemented an engine that provides atomicity guarantees for all its operations plus transparent snapshotting under write-heavy traffic and all this without reducing the performance compared to memcached.
Having said that, DF also has a novel caching algorithm that should provide better hit rate with less memory consumption.
You are correct - GET/SET do not require any locking as long as they do not contend and they do not in those benchmarks. You are right that for MSET/MGET you will see lower numbers. But still it will be much higher than with REDIS.
This is our initial release and we just did not have resources to showcase everything under different scenarios. Having said that, if you open an issue with a suggestion of a benchmark that you would like to see I will try to run soon...
The best tool out there is probably jepsen.io
but it requires domain knowledge to properly operate it. The guy behind it is a beast!
Anyway, I wrote lots of unit tests to cover those atomicity issues. You can also write a custom python/nodejs/golang/... scripts that simultenusly write and read from the same multiple keys in such way that some invariant is preserved. For example,
"mset x {i}, y {i}" for random `i` and in parallel do "mget x y" and to check that the response returns same values. You can also test this for other families using transactions like "MULTI; lpush x ${foo}; lpush y ${foo}; EXEC" .. and then similarly test that x and y have exactly the same lists.
Ah thanks I had heard of and then forgotten about jepsen. I had considered what you described but I thought there might be a more targeted approach where you can specify your invariants (like "x can only be in one of two states" or "pushing alway increases the length of a list and popping always reduces it if it's non-empty") and get the tool to try to break them.
I wonder if there could be a tool that instruments in the same way that afl does to try to detect races or inconsistent states.
I’ve seen the VLL paper before and I’ve wondered how well it would work in practice (and for what use cases). Does anyone know how they handle blocked transactions across threads? Is the locking done per-thread? If so, how do you detect/resolve deadlocks?
It also be good to see a benchmark comparing single-thread performance between DragonflyDB and Redis. How much of the performance increase is due to being able of using all threads? And how does it handle contention? In Redis it’s easy to reason about because everything is done sequentially. How does DragonflyDB handle cases where (1) 95% of the traffic is GET/SET a single key or (2) 90% of the traffic involves all shards (multi-key transaction)?
It's really good questions. I invite you to try it yourself using memtier_benchmark :)
if you pass `--key-maximum=0` you will get a single key contention when doing the loadtest. Spoiler alert - it's still much faster than Redis.
On the picture Redis tops at 200k/seconds on an instance with 64 cores (r6g), Dragonfly 1400k/seconds, Redis is single threaded DF is not but it only got 7.7x faster how come?
If you run let say 32 instances of Redis ( not using HT ) with CPU pining will be much faster than DF assuming the data is sharding/clustered.
Each CPU core doesn't have it's own independent channel to memory, there are usually 2-3 channels to DDR memory shared by all cores via multiple intermediate caches (usually shared hierarchically).
The reason for this is the networking limitations of each instance type.
DF consistently reaches the networking limits for each instance type in the cloud.
On r6g it's 1.4M qps and then it's saturated on interrupts due to ping pong nature of the protocol. This is why pipelining mode can reach several times higher throughput - your messages are big. c6gn is network-enhanced instance with 32 network queues! it's the most capable instance in AWS network-wise. This is why DF can reach there > 3.8M qps.
This is really cool. Love a section on how things are designed with links to papers, always makes me feel way better about a project - especially one that has benchmarks.
Agreed. Really great piece of technical writing. Probably written by one of the rare developers who write better than I do. I want to clone him/her/they and work with them.
KeyDB does network I/O and parsing on separate threads, but still has one big lock around the data structures themselves. DragonflyDB does full processing of the transaction on separate threads.
Aside nit-pick: I think is dangerous call anything "db" if is not safely stored with Acid.
People not read docs neither know the consequences of words like "eventual" or "in memory" and star using this kind of software as primary data stores, instead of caches/ephemeral ones...
> If you run a database without reading the documentation then you're the dangerous part
I think that covers almost all the whole dev population, for what I see in relation with RDBMS. Lucky us most RDBMs shield the mistakes in their usage, a lot.
That is why I see is "dangerous" to call ephemeral/eventual stores as "db". Marketing/positioning have impacts...
All databases are ephemeral if the person running it don't read the docs. Your comment is hence fully redundant, as opposed to the default single-node install of any DBMS.
All of A, C, and I only make sense defined relative to a particular transaction vocabulary. Redis is perfectly ACID, as long as your transactions are those supported by Redis's commands.
Conversely, plenty of DBs with programmable transactions (e.g. SQL) are considered work-a-day "ACID" enough, despite some massive gaps in their transactional model (no DDL in transactions, no nested transactions, atomic only when below a certain size, etc.)
I honestly think it's impossible to reuse Redis OSS to make it multi-threaded besides what KeyDB did. Btw, I did reuse some of the code - but it's around single value data-structures like zset, hset etc.
With multi-threading you need to think about all things holistically. How you handle backpressure, how you do snapshotting. How you implement multi-key operations or blocking transactions. So you need special algorithms to provide atomicity, you need fibers/coroutines to be able to block your calling context yet unblock the cpu for other tasks etc. All this was designed bottom up from scratch. Seastar could work theoretically but I am not a fan of coding style with futures and continuations - they are pretty confusing, especially in C++.
My choice was using fibers - which provide more natural way of writing code.
I have not designed the distributed long thingy. Will do it in the next 2 months.
I've been working on DF for the last 6 months and I implemented ~130 commands. It's still missing about 40% of the commands to reach parity with v5/v6.
I will continue working on DF. Primary/Secondary replication is my next milestone.
The license is different. Redis core is BSD 3 clause, which is a permissive open source license. Dragonflydb uses the Business Source License, with some additional use grants. Depending on your situation, that could be an impediment.
Not really but I read pelikan posts by twitter team.
One thing in common - we both thought that cache-based heuristics can be largely improved compared to memcached/redis implementations. We did it differently though. I think our cache design has academic novelty - I will write a separate post about it.
Several years ago there was memcachedb, which could flush stuff to disk. While this operation was expensive, it was also useful, because you could restart instances without being overwhelmed by missing keys (data).
For the latter: your application quickly grinds to a halt if you need to build your cache from ground up after some kind of crash. This is a deal-breaker for many.
I find the Redis benchmark suspect. Did you disable write to disk and live snapshotting? In production Redis shouldn’t be configured with write to disk.
I remember that I read about Anna. Very interesting paper.
From what I remember that require that the operations will be conflict free.
I think came to conclusion that it could be interesting as an independent (novel) store but not something that can implement Redis with its complicated multi-model API, transactions and blocking commands. I do not remember all the details though...
I do not know about aerospike, have never run their software but Scylla are the champions in what they are doing. Having said that, Scylla provide full durability guarantees, that means they need to touch a disk upon each write or update. Disk I/O is relativle expensive in the cloud and requires either large instances or special storage optimized family types like i3 etc.
Long story short, I do think Dragonfly is the fastest in-memory database in terms of throughput and latency today. We will see if we manage to stay this way when we extend our capabilities with SSD tiering.
they need at least perform fdatasync before replying to a client, imho. so I do not think it's every x seconds. I am not deeply familiar with scylla so maybe I am mistaken.
I think you're wrong. It does that every x seconds. The disk writing is async and it's meant that you protect data-loss by keeping replicas in different availability zones.
interesting. i still need to learn a lot about how other systems provide durability guarantees. Based on what you are saying, Scylla guarantees H/A instead of full durability. I was thinking about WAL (AOF) for dragonfly and I kinda thought to start with "fdatasync once in every X seconds", but dragonfly is not a database. I would expect stronger guarantees from databases. Maybe it's inpractical to expect this in cloud environments...
They probably don't want to end up being replaced by an AWS/GCP/Azure service. In my opinion, the BSL is a fair license model, especially if it is limited in duration (let's say 2-3 years BSL then automatically changing to Apache/BSD).
A duration limit in the license, after which it becomes a permissive license, seems the critical point.
Accomplishes the goal of preventing a cloud provider from stealing customers, but also ensures customers don't get caught in an "always tomorrow" trap when the deadline comes and the company realizes it only hurts them to fully share it.
Seems to align all interests pretty nicely.
(I'm as big of an OSS supporter as anyone, but we can't pretend we still live in a time where Google / Amazon / modern-Microsoft don't exist)
Exactly. Finding the right balance is the right thing to do. I am not angry at cloud providers , but i am angry at OSI and their blind idealism that leaves us out there alone. I think that not solving the problem by them, hurt first of all the oss community.
> I think that not solving the problem by them, hurt first of all the oss community.
F/OSS means a very specific thing. If one can't possibly build a rocketship business, that isn't F/OSS fault. Of course, you've got an alt-movement in response to OSI and FSF's rigid adherence to its principles, speared on by tech companies who (think they) got burnt by other tech companies.
There's a lot of room between {completely F/OSS, that a cloud provider can implement and bankrupt the company supporting project development} and {completely evil closed source company like Oracle}.
In the end, we all want good software, with the maximum amount of permissions and source, free. It's just a question of tweaking the support model to get there.
IMHO, the "Don't call yourself OSS if you're not 100% OSI OSS" is counter-productive. But I understand why they do it, and I understand the history and abuses that caused them to do it. I'd just say if we no-true-Scotsman our approach in a way that precludes profitable, sustainable OSS companies... we're going to have less quality OSS. :(
But my entire stack is already in AWS. One service provider deciding they don’t want AWS making a service out of it just means I have to make the effort of self-hosting it, not that I’ll suddenly end up using their service (which is outside my VPC).
can you clarify why you feel this way, plenty of companies use confluente Kafka or mongodb with VPC peering? Would AWS private link make you feel better about network security?
This looks quite cool from a technical perspective, but the unusual license definitely gives me pause. Largely because I'm not familiar with BSL. Others say it's increasingly popular, but with a little googling the acronym is still somewhat ambiguous - opensource.org lists BSL as the "Boost Software License" which looks more like BSD. This kind of confusion doesn't support the idea that this is a solid trustworthy OSS license.
Still, I really appreciate that you didn't choose a copy-left license.
On the license front, what is the "change license" clause listed? It says something about changing in 5 years. Does this mean it will become Apache licensed in 2027? Why would you put that in there?
It's exactly that. It gives us a little chance to fight against the Giants.
In 5 years the initial version becomes Apache 2.0 then the next version and so on and so forth. CockroachDB uses similar license. MariaDB uses that, Redpanda Data and others. You are right that acronym is confusing - it's not Boost license, it's Business License. Every major technological startup turned away from BSD/Apache 2.0 licenses due to inability to compete with cloud providers without technological edge.
I certainly don't think you should pick a permissive license, and find yourself under competition from companies running your own software.
However, I'm sad that instead of going with an Open Source license that protects against that, you're using a proprietary license. That alone is a nonstarter for many users, not because they want to compete with you but because they want to protect themselves and make sure they have a firm foundation to build on.
Much of the software you're citing as examples moved from Open Source to proprietary, harming their users in the process, and causing many users to seek alternatives.
Do you think AGPL offers such protection? I think even AGPL is fine for Amazon et al, they are happy to dump the code somewhere while selling a managed service with enough wrapper layers. But AGPL is toxic also for self-hosters who aren’t trying to build a managed hosting service, since it applies equally against all parties.
Redis is doing just fine, PostgreSQL is doing just fine. They are both real open source.
The performance improvements are not worth the legal/compliance overhead of adding non-FOSS to my stack. Much less so if the performance improvements are due just to some optimization choice in the underlying system. In the next 5 years, it will be easier to have Redis adding the io_ring optimizations than for this new project to become uniquely better.
You misunderstood. What I am saying is that is precisely because I am working on projects of different sizes and potentially with different customers, I do not want to have the burden of having to check if the projects have specific compliance requirements.
I have a SaaS [0], my own (AGPL, by the way) open source project [1] and also work have the occasional contract job. It's much easier to say "just use Redis because it is FOSS and it gets the job done" across the board then to try to special-case the tools based on licensing requirements.
And? AGPL is not a non-commercial license. If Amazon wanted to provide such a service, they would certainly be permitted to. As long as they provided any changes they make to the the AGPL source code. The AGPL would not "infect" their hosting infrastructure.
> AGPL is toxic also for self-hosters who aren’t trying to build a managed hosting service
Absolutely not. First of all, you can always use any software under AGPL as released without any limitation.
Secondly, if you were to make changes and release them you can still run it any way you want.
Thirdly, if you are making changes for internal use, and for some reasons your really want to keep them secret, you can STILL run the database to your heart's contents as long as you don't provide it as a service to the public.
AGPL is much more permissive than people think. It's just providing a degree of protection to developers and and users from patent trolls and other uncooperative entities.
Exactly - the grandparent is asking “can a true copy left / Free Software license protect a young startup from Amazon reselling their software”, and my assertion is no, it cant.
MongoDB is the most famous example to transition from AGPL to a closed license. MongoDB was AGPL until October 16, 2018. Since October 16, 2018, it uses Server Side Public License (SSPL), which they authored.
From their FAQ:
> The market is quickly moving to consume most software as a service. This is a time of incredible opportunity for open source projects, with the potential to foster a new wave of great open source server side software. The reality, however, is that once an open source project becomes interesting, it is too easy for large cloud vendors to capture all the value but contribute nothing back to the community.
The most recent notable example of open-source software going closed because of Amazon specifically is Elasticsearch, which switched from Apache to SSPL. The announcement blog post was titled "Amazon: NOT OK - why we had to change Elastic licensing".
MongoDB is definitely an example. Did switching licenses actually "fix" their "problem" though? And ElasticSearch doesn't count since it was a pushover license before and not the AGPL.
We do not choose AGPL so we would not harm our users. Lets not confuse the means and the goal here. AGPL is copyleft and restricting. Fair users refuse using it.
OSI does not help technological companies to find a fair solution and force us to go outside of "OSS".
I don't want to enter in the merit of your company's choice, it's definitely up to you what license to use, but I'm confused about that statement around AGPL. How would AGPL harm users?
More likely they'll ignore the s/w. Huge amount of talent is working in such companies. Using AGPL will practically exclude them from interacting with the s/w.
Also, smaller companies like to play safe. If FAANG with their million lawyers aren't touching it, why should I take the risk.
Because unlike FAANG you don’t have millions of engineers to create the software from scratch. Companies pay for databases (Oracle, Gemstone, MsSQL, anything at Amazon). Almost all the risks (price changes, maintenance costs, etc.) can be managed adequately via a long term enterprise contract.
> We do not choose AGPL so we would not harm our users.
This is a first. If anything, it seems clear that a FLOSS license such as AGPL achieves exactly the opposite: is highly protective of the users' best interests, both in the short term and long term.
Could you elaborate on why do you think that users are harmed by standard, run-of-the-mill FLOSS licenses?
Depends on who you consider 'the users'. If you are a 'user' in the sense of an individual working on their own stuff, then yes, AGPL protects you a lot.
However, if you are a 'user' in the sense of a company wanting to make software to sell as a service, it doesn't protect you very much, and can make you vulnerable.
Most legal departments don't allow for copyleft licenses. This is because most legal departments don't actually make their own legal determinations but instead rely on industry norms. And google set the stage here ages ago with their no-GPL (and especially, no-AGPL) policy.
However this is rather uniquely due to Google's monorepo infrastructure. Google quite literally has all its products in the same source code repository, being built and statically linked together to create a single system binary (or so I am told--I've never worked there). In this case AGPL would virally infect the rest of your code and you could find yourself in trouble. Even if they don't, they might get hit with a discovery request for the source code of some project to prove compliance, and in the process have to provide their entire monorepo and all its business secrets to the court. Etc. Etc.
But these are concerns which stem directly from Google's monorepo architecture. It's not generally true of the whole industry. And yet Google's fear of copyleft--rational though it may be in their particular instance--has been copied throughout the entire industry. 99% of the companies out there have nothing to fear from the AGPL. But hey, if you're some random in-house lawyer at company XYZ, who are you to question Google's legal precedent?
> However, if you are a 'user' in the sense of a company wanting to make software to sell as a service (...)
That does not meet the definition of "user", does it?
In addition, if that was the motivation behind that absurd claim, why not be honest and just say "we don't release our software under a FLOSS license because what we actually want is to keep it proprietary"?
It means ‘user’ in the sense of the person who is choosing what software to include in their work stack, and most software of this type is going to be used in a professional setting. If you use a license that most legal departments won’t let their developers use, your software won’t gain much adoption… which is especially important if you are planning on making money from your software. People aren’t going to be paying for support for their personal projects.
Using a non-OSI approved license hurts your users considerably more than AGPL (and this is coming from someone who generally doesn't like the AGPL). Legal teams are far less likely to approve licenses like the one you've chosen, than AGPL.
I could convince a legal team to allow an AGPL service, depending on how we were using it. It would be difficult to get them to allow us to use this license, and I definitely couldn't convince them to allow us to contribute to software using this license.
Effectively, you've ensured you won't have a community, because in reality the project is closed source, and dead if your company dies, which means it's not an option for me to consider.
I do not have any practical, first hand experience with either license. Re -agpl, i know for a fact that it is banned in corporations. We choose BSL to avoid this. Say if a team in amazon would want to use DF internally they could. I know for a fact that AGPL is a big no there. If we are mistaken and BSL is also banned then we will need to see what to do.
I hope people will understand the reason behind bsl. I hope people will see apache there and will help us to grow. Based on HN comments some already do.
As someone building software for a living I fully agree with your decision to not go full OSS. It’s always a hefty discussion but I feel most of the people only exist on the receiving end of software and do not have to worry how a license impacts real life distribution. Big corp (I’m from Germany) DOES hesitate to go with any form of non-Apache-2 licensed OSS software. In Germany often times OSS is even discouraged if there’s no commercial support for it.
I understand the emotions people have regarding licenses that do allow them to work with software as they please - but I also think it’s useless to argue around licensing of a software instead of focusing on the good it can do. Something being proprietary has NO impact on your choice to buy it when outside of the software world. Why does it here?
The problem you are pointing out is not the open source license, it's the lack of support. If they offered AGPL with a dual-license version and made a strong message about commercial support, they could please both crowds without compromises.
> I do not have any practical, first hand experience with either license.
I do have practical first hand experience with both. I can get a legal team to approve the use of AGPL, but I cannot get them to approve BSL, unless we've signed a contract with the company.
Having an understanding of the license, and some first hand experience seems like it would be a pretty important thing, when choosing it, especially for a business.
This is not representative of my experience. AGPL is unconditionally banned at most companies where I know the answer to this question. They don't care if the license is OSI-approved, they care if it aligns with their business requirements and objectives. Most companies are not ideological about open source.
BSL is at least making an attempt to address the legitimate concerns of businesses -- both sides -- in a balanced and pragmatic way. I have no dog in this fight, being neither a BSL licensor nor licensee (nor AGPL for that matter), but having been party to the legal conversations at large companies I understand why BSL is generally considered to be more acceptable than AGPL. BSL may or may not be the right license for this software, I have no opinion, but businesses don't care about arguments from ideological purity. If BSL satisfies their requirements better than AGPL, and anecdotally all the evidence seems to support this notion, then they will go with BSL software.
To put it another way, if OSI is serious about being in the conversation about the future of software licensing, they must address the reasons so many companies refuse to adopt AGPL software.
I wonder how involved you've been in these kinds of discussions. I've run open source programs at companies, and worked together with the legal teams in terms of what was and was not allowed.
Transferring copyright to another company, without a contract with that company, for the most part is something a legal team isn't going to agree to. BSL only addresses one side's concerns, and it's not in your favor.
AGPL, when used for internal services that customers don't interact with, is generally not that difficult to get approved. How many companies are using mongodb, for instance?
We may do it in the future. We will see if the community will accept Dragonfly with BSL. MariaDb were pretty famous and i do not remember any loud objections to their license.
There were a lot of loud objections to their license, and remember that MariaDB the database isn't licensed that way (it's GPL), just the much less used MaxScale and CMAPI.
I was quite excited to see Dragonfly. However now that the license has been pointed out, I won't be touching it with a 10 foot pole. I wish you the best of luck, but I'll never use it.
If it was AGPL I'd have been a happy camper though.
> Every major technological startup turned away from BSD/Apache 2.0 licenses due to inability to compete with cloud providers without technological edge.
No, there are plenty that still use permissive licenses.
Finally Redis has a BSD licensed core, a source available license for additional modules, and a closed source license for enterprise. https://redis.com/legal/licenses/
Redis has a BSD license because it was written by antirez who did not intend to make it a commercial enterprise. Today its too late to change and wont help Redis Inc to protect their ip. We looked at elastic, mongodb, cocroach labs, mariadb, redpanda, atlassian, grafana labs , scylladb. The last two are agpl but the motivation is the same: to protect themselves.
Bravo for picking the BSL and writing your license the way you have. It shows maturity in that you and your company know what they are doing and intend to be there for the long haul.
I'll be interested to see where the community lands on this years from now. It's frustrating to see so many databases relicense under licenses that are almost OSS but not quite. Yet it's totally understandable - it's hard to pay database engineers when you give your product away and then AWS neuters your hosting and support business.
I just wish more developers could follow Grafana's path and switch to the AGPL when that becomes a problem, instead of switching to fauxpen source licenses.
Why? FOSS is, at least initially, a charitable project. You work on a database for months then give it in its entirety away for free to all your potential customers and even your competitors with better established sales teams (e.g. Amazon).
They are explicitly not a charity. Isn’t it okay to be a for profit company that sells software with a closed copyrighted source?
Red Hat makes a Linux distribution. It was not their choice whether to be charitable or not, they are the recipients of the initial charity of the Linux founders and are bound by contract to continuing making their code open as a price for building on open code.
If Red Hat could change the license, would they? Maybe. (Not now but you can’t say the leadership won’t make that decision during financial crisis).
But there's more to a Linux distribution than just the kernel, and a lot of the other pieces aren't copyleft, so Red Hat can change the license, but they choose to keep them open source anyway.
No, it's a collaborative project initially. You work on a tool for months with other developers that each have their own employer, but all benefit from the sum of the work.
That there's freeloaders that benefit from the work without having to expend labour is a side-effect, not the primary motivation.
> I think its ok that we have innovation at legal front as well. And innovation requires exploration, choosing road not taken...
Chosing to release software under a non-FLOSS, proprietary license is hardly innovative, and I'm afraid trying to frame it that way sounds like you want to have your FLOSS cake and eat it too.
Apologies if this was already asked -- I saw you guys already showed benchmarks in AWS itself, but would it make sense to have some benchmarks outside of AWS (or run benchmarks in a VM with Redis/DF/Keydb installed within)?
I ask this because I'm unsure if AWS Redis has any modifications on top of the Redis software itself, which would affect the speed, or even make it a bit slower. For example I know MS Azure's version of Redis restricted certain commands, and from a quick search AWS does something similar: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/...
We compared DF with Redis OSS and KeyDB OSS, we have not used any of managed services for the comparison. AWS was used as a platform for using EC2 and cloud environment.
From my experience, Redis performance seems to be all over the place depending on the circumstances. In what cases does this solution perform well and where does it fail? Most caches I worked with really love to have everything close by at low latency and works best when the consumers have about as much memory as the cache in the first place.
Benchmarks done on only one metric are often misleading (and in commercial circumstances usually intentionally!). Would love to see visually what trade offs Dragonfly is making to achieve the numbers from the chart at the top of the README. If excellent technical work means there really is no trade off, that would also be a great reason to chart it visually.
Also as a Redis replacement, it's not clear what durability is offered, and for most Redis use cases this is close to the first question
Probably not many in production, but the "recent kernel" is going to be the more important part.
I'm not hip to how much new stuff is backport-able, so this may preclude Ubuntu 20.04, for instance. You lose the "LTS" part if you compile your own kernel, if you manage to make it functional at all.
Note: I never use kernel modules due to the issues rhel/debian and I have had with such things in the distant past.
The kernel has its own LTS releases, and running Linux in the cloud is a well tested use case. Any library not available in your distro of choice can be built from source and tracked manually, which usually works well enough.
> I'm not hip to how much new stuff is backport-able, so this may preclude Ubuntu 20.04, for instance. You lose the "LTS" part if you compile your own kernel, if you manage to make it functional at all.
I'm not sure that concern is justified. It seems io_uring was pushed as part of the 5.1 linux kernel release, and Ubuntu 20.04 LTS seems to have been shipped with 5.4.
How you would suggest to demonstrate visually that there are no trade-offs?
The tradeoff the way I see it - one needs to implement 200 Redis commands from scratch. Besides, I think DF has a marginally higher 50th percentile latency. Say, if Redis has 0.3ms for 50th percentile, DF can have 0.4ms because it uses message passing for inter-thread communication. 99th percentiles are better in DF for the same throughput because DF uses more cpu power which reduces variance under load.
Re-durability - what durability is offerred by Redis? AOF ? We will provide similar durability guarantees with better performance than AOF. We already provide snapshotting that can be 30-50 faster than of Redis.
There is usually a tradeoff between latency and throughput, although I'm not so sure this would be true for your innovation, since you've eliminated a whole chunk of fixed overhead from the system (syscall batching). However, batching in general often implies added latency
If I recall ScyllaDB has some excellent examples of demonstrating this particular tradeoff visually. A simple option would be a scatter plot where X = latency, Y = load or similar, with points coloured according to the system under test. Probably there is a better option, but this would likely be enough to sell me at least
Well in that case, I hope to have an good answer for you. Right below the benchmark graph I deliberately put a table that shows 99th percentile lantencies at *peak* throughput as reported by memtier_benchmark.
99th latency percentile of Dragonfly at its peak throughput. I put it here as well
op r6g c6gn c7g
set 0.8ms 1ms 1ms
get 0.9ms 0.9ms 0.8ms
setex 0.9ms 1.1ms 1.3ms
I've wrote in in the background section of Readme.
Basically, I worked in a cloud company in a team that provided a managed service for Redis and Memcached. I witnessed lots of problems that our customers experienced due to scale problems of Redis. I knew that these problems are solveable but only if the whole system would be redesigned from scratch.
At some point I decided to challenge the status quo, so I left the company and..and here we are.
Sorry, missed your second question.
Redis is using fork-based save for snapshotting and full replica sync. This means that memory pages are duplicated on write. That consumes more memory than DF implementation that implements algorithmic point-in-time snapshotting. In addition DF was designed to use less overhead memory in its keys table. As for the why, you can read about it here: https://github.com/dragonflydb/dragonfly#background
Indeed :)
The moment i learned about io_uring I got hooked up immediately.
People like Jens, Pavel are the type of innovators that make sure Linux stays on top of its game for the next generations.
hey, great work. I couldn't find the specifics of the benchmark. Is there, by any chance you compare 1 instance of single threaded redis running on 64vcore to a multithreaded key-value store?
Can we see such disparity in benchmark even if we run Ncore instances of redis in parallel?
Thats what i did. I did not try to cheat. Running N redises is a completely different product. You will have n listening ports and your loadtest program will need to connect to each one of them. The power of dragonfly is that it hides all this complexity. You are saying that comparing single threaded redis to df is unfair, and i say - comparing n redises to df is unfair because you compare different products.
This looks like awesome work! I appreciate you operationalizing of some of the best things to come from computer science in the past decade or so.
Out of curiosity, are you discovering any new bottlenecks to performance outside of the software, given Dragonfly is able to process far more qps than most systems? I imagine the network and disk I/O could become stressed, but also I wonder if it breaks any assumptions of cross-core performance, hypervisors, etc. I know that cloud offerings typically mean that you can attach ginormous disk IOPS and NICs, but surely there are limits.
I was mostly running on AWS. In terms of hardware, for small-packets loadtests, most systems are constrained on throughput, i.e. number of packets per second. Some instances saturate on interrupts reaching 100% CPU on all cores and some can not even saturate the CPU and you will see that CPU is at 60% but you can not go beyond in throughput. The best systems network-wise are c6gn family types. They are also better than instances that other cloud provide. btw, you mentioned hypervisors... About 8 months ago I opened a bug on AWS Graviton team https://github.com/amzn/amzn-drivers/issues/195 - about performance issue they had on their instances at high throughput. Recently they issued the fix. I suspect it was in their hypervisor.
In terms of my software I found many performance bugs at those speeds. For example, using a default allocator is a big no. I use mimalloc for uncontended allocations. In general, you can not use mutexes and spinlocks at those speeds. Those will just cripple the system. Sometimes it can be very annoying since you can not rely on a 3rd party library without carefully analyzing its design. For example, I could not use openmetrics c++ library because it was not performant enough. Even to implement a simple counter, say to gather statistics for INFO command becomes an interesting engineering problem:
With share nothing architecture, I use a lot of thread-local counters that I aggregate only when stats are pulled.
As a general note, I expect that Dragonfly will stay very performant with the tailwinds from recent hardware advancements. For example, c7g (Graviton 3) is much better than c6g and DF shows it.
I'm curious: since you're targeting Graviton, and AFAICT Graviton 2 and up support ARM LSE, have you tried directly using LSE for metrics? ARM LSE offers STADD to add to a 64-bit memory location without reading the contents or ordering the access. (I think LDADD with XZR as the destination is identical, but I could be missing some subtlety. I'm not an ARM expert.) You might be able to get good performance by using a global counter and just STADDing to it. x86 will perform terribly if you use XADD because it has no corresponding optimized forms.
Wow, its beyond my knowledge:)
I use my software skills to write a performant software but rarely i go to the assembly level. And i have zero knowledge about specifics if each command on every architecture. The rare exception is a code in dashtable. the authors of the paper designed it to allow vectorizing of the find() operation.
Interesting. Have you considered/tried to use fdio/user space networking? In my experience, it greatly improves throughout (simple ip forwarding can be more than 10mpps per cortex a53 on some platforms). Fdio also has a so that you can preload in order to use its ip stack in your app (instead of Linux's).
Have you thought about also offering dragonflydb as a library, like sqlite? This would avoid context switching when running everything on the same machine.
Quite a few :) I wish i was 20 again - to have time to learn the language and to implement df in rust :) i am an old with old habits. Btw, I spent a short time coding in rust last year and i understand why the fuzz.
curious about the consistency guarantees. redis is single threaded which seems old fashioned but guarantees that all transactions are serial, so there is no chance of conflict between reads, no issue with race conditions , a complicated problem to solve once concurrency is introduced . if the redis protocol is used how is this problem addressed?
Yes this problem is complex and i guess this is why it have not been solved before.
We do provide atomicity guarantees for all operations like Redis! We use an algorithm from a 2014 paper - see our readme, we provide the link to the paper.
FoundationDB should have been included in their perf comparison. It’s ACID compliant and a distributed Value/Key store.
For SSD based storage, it’s getting 50k reads/sec PER core and scales linearly with # of cores you have in your cluster. (They achieved 8MM reads/sec with 384 cores)
FoundationDB does have impressive performance, but implementing compound operations like INCR, APPEND, etc. would require at least an additional network round trip between the client and the server.
For example, INCR would require one read followed by one write of the new value, and of course this will result in very inefficient mutation range conflicts (which must be retried for another couple of round trips) if you have frequent updates of the same keys in multiple concurrent transactions.
I had completely forgotten about those built-in ones. Right, so some operations can be made fully atomic server-side, but there are still a bunch of ops (append, pubsub, set commands such as SADD, etc.) that would need round trips.
SADD wouldn't need a round trip since it is idempotent. You would just do a blind set which doesn't even require a transaction in FDB. Likewise, most of the other ones can be reframed / designed to not require round trips. I would still not use it as the backend unless I already had FDB in my solution and this was just an additional database type that I needed without ultimate performance as the constraint.
SADD needs to return a value indicating whether the member was added or not, so I don't think you could avoid a round trip, since FDB doesn't have a "set if not exists".
FDB scales really well with reads, but it will bottleneck on writes a lot faster than you think. For write heavy workloads, Redis and Dragonfly would both leave FDB in the dust (and I say this as a big FDB user/enthusiast).
No there is much more to it than that. FoundationDB has a completely different architecture that is optimized for completely different things than Redis is. It’s not just about writing to disk vs. not.
Redis is basically a very performant, single-threaded (mostly) single-node in-memory datastructures system with an efficient and readable server protocol strapped to it.
FoundationDB is a completely different beast that has like 6+ distinct roles, and is optimized almost exclusively for interactive serializable transactions, range reads, and correctness.
They’re just completely different things, I recommend reading the FoundationDB paper to get a sense for its architecture. The amount of “steps” involved in processing an FDB write is much higher than in Redis.
A lot of projects say "faster" without giving some hint of the things they did to achieve this. "A novel fork-less snapshotting algorithm", "each thread would manage its own slice of dictionary data", and "core hashtable structure" are all important information that other projects often leave out.
To me the focus on speed is a wash now. They’re all fast. I’d like to hear about easy cross-region replication and failover as well as effortless snapshot and restoring of backups.
I like the redis protocol compatibility and the HTTP compatibility, but from the initial skim through I guess you are using abseil-cpp and the home-grown helio (https://github.com/romange/helio) library.
Could you get me a one liner on the helio library is it used as a fiber wrapper around the io_uring facility in the kernel? Can it be used as a standalone library for implementing fibers in application code?
Also it seems that spinlock has become a defacto standard in the DB world today, thanks for not falling into the trap (because 90% of the users of any DB do not need spinlocks).
Another curious question would be - why not implement with seastar (since you're not speaking to disk often enough)?
Yes, helio is the library that allows you to build c++ backends easily similar to Seastar. Unlike Seastar that is designed as futures and continuations library, helio uses fibers which I think simpler to use and reason about. I've wrote a few blog posts a while ago about fibers and Seastar: https://www.romange.com/2018/07/12/seastar-asynchronous-c-fr... one of them. You will see there a typical Seastar flow with continuations. I just do not like this style and I think C++ is not a good fit for it. Having said that, I do think Seastar is 5-star framework and the team behind it are all superstars. I learned about shared-nothing architecture from Seastar.
Re helio: You will find examples folder inside the projects with sample backends: echo_server and pingpong_server. Both are similar but the latter speaks RESP. I also implemented a toy midi-redis project https://github.com/romange/midi-redis which is also based on helio.
In fact dragonfly evolved from it.
Another interesting moment about Seastarr - I decided to adopt io_uring as my only polling API and Seastar did not use io_uring at that time.
Thanks for taking the time to reply - yes in fact seastar does not use io_uring but it's rust equivalent glommio does use it (IIRC it is based on io_uring). Any reasons for using c++ instead of Rust (are u more familiar with it? or just the learning curve hinders the time to market? or is it the Rc/Arc fatigue with rust async? I guess Rust should be a fairly easy language to pick up for good c++ programmers like you)
If I would choose another language it would be Rust. Why I did not choose Rust?
1. I speak fluently C++ and learning Rust would take me years.
2. Foodchain of libraries that I am intimately fimiliar with in C++ and I am not familiar with in Rust. Take Rust Tokyo, for example. This is the de facto standard for how to build I/O backends. However if you benchmark Tokyo's min-redis with memtier_benchmark you will see it has much lower throughput than helio and much higher latency. (At least this is what I observed a year ago). Tokyo is a combination of myriad design decisions that authors of the framework had to do to serve tha mainstream of use-cases. helio is opinionated. DF is opinionated. Shared-nothing architecture is not for everyone. But if you master it - it's invincible.
Have Redis and Memcached aged so much we need a modern replacement? Or is this a webdevy 'modern' which just means the first commit is newer than redis' first commit?
memcached was born in 2003.
Redis was born in 2008. Both have strength and weaknesses. DF tries to unite them into a single backend and keep the strengths of each one of them.
> Have Redis and Memcached aged so much we need a modern replacement? Or is this a webdevy 'modern' which just means the first commit is newer than redis' first commit?
The docs make some of the differences clear. Worth reading the GitHub repo readme.
Interesting project. Very similar to KeyDB [1] which also developed a multi-threaded scale-up approach to Redis. It's since been acquired by Snapchat. There's also Aerospike [2] which has developed a lot around low-latency performance.
True. Keydb tackled the same problems as us. But we chose differrent paths. We decided to go for a complete redesign, feeling that there is a critical mass of innovation out there that can be applied for inmemory store. KeyDb wanted to stay close with the source and be able to integrate quickly with recent developments in Redis. Both paths have their own pros and cons.
I see from the blog posts that you looked at KeyDB and Scylla/Seastar for background. I agree with both approaches - fewer but bigger instances and shared-nothing thread-per-core architecture - and it was a major reason for switching to ScyllaDB in my previous startup.
Will definitely follow this to see how it develops. Good luck.
If I would choose another language it would be Rust. Why I did not choose Rust?
1. I speak fluently C++ and learning Rust would take me years.
2. Foodchain of libraries that I am intimately fimiliar with in C++ and I am not familiar with in Rust. Take Rust Tokyo, for example. This is the de facto the standard for how to build I/O backends. However if you benchmark Tokyo's min-redis with memtier_benchmark you will see it has much lower throughput than helio and much higher latency. (At least this is what I observed a year ago). Tokyo is a combination of myriad design decisions that authors of the framework had to do to serve the mainstream of use-cases. helio is opinionated. DF is opinionated. Shared-nothing architecture is not for everyone. But if you master it - it's invincible. (and yeah - there is zero chance I could write something like helio in Rust)...
> Probably, the fastest in-memory store in the universe!
Redis is fast enough. Read/write speed isn't usually the bottleneck, it's limiting your data set to RAM. I've long ago switched to a disk-backed Redis clone (called SSDB) that solved all my scaling problems.
Haha, I thought the same thing until it wasn't! It turns out there are a lot of humans in the world and if you are unfortunate enough to get a large portion of them to start utilizing the software you write you'll find some bottlenecks in almost every system you thought was fast enough.
What about it makes it a "modern" replacement rather than just a replacement? Is there something about Redis and Memcached that is "outdated" in the (relatively) short time span they've existed (compared to something like C)?
I'd say using io-uring is an example of modern since it has only been available since 5.10 of the kernel. They also sit on some more recent research on data structures that perform well in multithreaded systems.
It is "modern" in the sense that the design and architecture is idiomatic for high-performance software on recent computing hardware. The main advantage of modern (in this sense) implementations is that they use the hardware much more efficiently and completely for current workloads, enabling much higher throughput on the same hardware.
Yes, exactly. When people talk about about "modern" in the context of software performance, it usually denotes software architecture choices. If someone did a forklift upgrade to the software architecture of Redis etc, then they would be modern too.
In practice, software architecture is forever. It is nearly impossible to change the fundamental architecture of software people use because applications become adapted to the specific behavioral quirks of the architecture i.e. Hyrum's Law, which would not be preserved if the architecture was re-designed wholesale.
Mainline memcached is very well tuned, so it's sort of the odd one out here.
But yes, Redis is very much designed against the grain of modern hardware. It also is a very minimalist design, that works well within its limits, but falls down hard when you push those limits, particularly with snapshotting and replication.
Not really, it just implies that the competition is not modern, without qualification. I think asking for qualification in this case is fair if we are to conclude Redis and Memcache have aged to the point of needing a replacement.
The word is often tagged on anything new somebody tries to sell. Better to be specific. The problem is that most “modern” things are very old things sold as new ideas. Cause biz. So, nothing specific against this proj.
we use io_uring for everything: network and disk. Each thread maintains its own polling loop that dispatches completions for I/O events, schedules fibers etc. Everything is done via io_uring API. All socket writes are done via ring buffer etc. If you run strace on DF you won't see almolst any system calls besides io_uring_enter
And no, it's not just because of io_uring it is faster. It's also because it's multi-threaded, has absolutely different hashtable design, uses a different memory allocator and many other reasons (i.e design decisions we took on the way).
382 comments
[ 1.7 ms ] story [ 440 ms ] threadHomepage: https://dragonflydb.io/
Benchmark: https://raw.githubusercontent.com/dragonflydb/dragonfly/main...
you can dm me at roman at dragonflydb.io
Could this architecture be extended to scale across multiple machines? What would be the benefits and costs of this?
For intra-process framework it's not an issue (as long as we do not have deadlocks).
[0] https://github.com/dragonflydb/dragonfly#background
And we have more plans for using io_uring in DF in the future.
Having said that, DF also has a novel caching algorithm that should provide better hit rate with less memory consumption.
Get/set operations look like they don't need it.
This is our initial release and we just did not have resources to showcase everything under different scenarios. Having said that, if you open an issue with a suggestion of a benchmark that you would like to see I will try to run soon...
Anyway, I wrote lots of unit tests to cover those atomicity issues. You can also write a custom python/nodejs/golang/... scripts that simultenusly write and read from the same multiple keys in such way that some invariant is preserved. For example, "mset x {i}, y {i}" for random `i` and in parallel do "mget x y" and to check that the response returns same values. You can also test this for other families using transactions like "MULTI; lpush x ${foo}; lpush y ${foo}; EXEC" .. and then similarly test that x and y have exactly the same lists.
I wonder if there could be a tool that instruments in the same way that afl does to try to detect races or inconsistent states.
I’ve seen the VLL paper before and I’ve wondered how well it would work in practice (and for what use cases). Does anyone know how they handle blocked transactions across threads? Is the locking done per-thread? If so, how do you detect/resolve deadlocks?
It also be good to see a benchmark comparing single-thread performance between DragonflyDB and Redis. How much of the performance increase is due to being able of using all threads? And how does it handle contention? In Redis it’s easy to reason about because everything is done sequentially. How does DragonflyDB handle cases where (1) 95% of the traffic is GET/SET a single key or (2) 90% of the traffic involves all shards (multi-key transaction)?
If you run let say 32 instances of Redis ( not using HT ) with CPU pining will be much faster than DF assuming the data is sharding/clustered.
On r6g it's 1.4M qps and then it's saturated on interrupts due to ping pong nature of the protocol. This is why pipelining mode can reach several times higher throughput - your messages are big. c6gn is network-enhanced instance with 32 network queues! it's the most capable instance in AWS network-wise. This is why DF can reach there > 3.8M qps.
Might try this out.
[1] https://docs.keydb.dev/blog/2019/10/07/blog-post/
I am spoiled.
People not read docs neither know the consequences of words like "eventual" or "in memory" and star using this kind of software as primary data stores, instead of caches/ephemeral ones...
For _any_ database there will be important information only available in the documentation.
I think that covers almost all the whole dev population, for what I see in relation with RDBMS. Lucky us most RDBMs shield the mistakes in their usage, a lot.
That is why I see is "dangerous" to call ephemeral/eventual stores as "db". Marketing/positioning have impacts...
Conversely, plenty of DBs with programmable transactions (e.g. SQL) are considered work-a-day "ACID" enough, despite some massive gaps in their transactional model (no DDL in transactions, no nested transactions, atomic only when below a certain size, etc.)
Having said that we carefully choose to write everywhere in the docs thay we are in-memory store (and not the database).
Btw, I reserve full rights to provide full durability guarantees for DF and to claim the database title in the future.
Why not reuse seastar framework?
Can you describe your distributed log thing? Is it like facebook-logdevice or apache-bookeeper?
With multi-threading you need to think about all things holistically. How you handle backpressure, how you do snapshotting. How you implement multi-key operations or blocking transactions. So you need special algorithms to provide atomicity, you need fibers/coroutines to be able to block your calling context yet unblock the cpu for other tasks etc. All this was designed bottom up from scratch. Seastar could work theoretically but I am not a fan of coding style with futures and continuations - they are pretty confusing, especially in C++. My choice was using fibers - which provide more natural way of writing code.
I have not designed the distributed long thingy. Will do it in the next 2 months.
I will continue working on DF. Primary/Secondary replication is my next milestone.
Looks awesome so far, though!
One thing in common - we both thought that cache-based heuristics can be largely improved compared to memcached/redis implementations. We did it differently though. I think our cache design has academic novelty - I will write a separate post about it.
Where's the benchmark compared to memcached?
Several years ago there was memcachedb, which could flush stuff to disk. While this operation was expensive, it was also useful, because you could restart instances without being overwhelmed by missing keys (data).
For the latter: your application quickly grinds to a halt if you need to build your cache from ground up after some kind of crash. This is a deal-breaker for many.
I think came to conclusion that it could be interesting as an independent (novel) store but not something that can implement Redis with its complicated multi-model API, transactions and blocking commands. I do not remember all the details though...
Long story short, I do think Dragonfly is the fastest in-memory database in terms of throughput and latency today. We will see if we manage to stay this way when we extend our capabilities with SSD tiering.
They don't fsync the wal on every write, it's probably done as group commit every x seconds or every x MB.
> I would expect stronger guarantees from databases.
You can configure it and get lower performance.
> Maybe it's inpractical to expect this in cloud environments...
For benchmarks they use raid0 on local disks(not EBS) in aws vps.
Curious: Why BSL? Why not open core [0] (or xGPLv3) like what most other commercial OSS projects seem to be doing?
[0] https://archive.is/nT9DF#selection-410.0-410.1
Accomplishes the goal of preventing a cloud provider from stealing customers, but also ensures customers don't get caught in an "always tomorrow" trap when the deadline comes and the company realizes it only hurts them to fully share it.
Seems to align all interests pretty nicely.
(I'm as big of an OSS supporter as anyone, but we can't pretend we still live in a time where Google / Amazon / modern-Microsoft don't exist)
F/OSS means a very specific thing. If one can't possibly build a rocketship business, that isn't F/OSS fault. Of course, you've got an alt-movement in response to OSI and FSF's rigid adherence to its principles, speared on by tech companies who (think they) got burnt by other tech companies.
See also: https://news.ycombinator.com/item?id=25896162
There's a lot of room between {completely F/OSS, that a cloud provider can implement and bankrupt the company supporting project development} and {completely evil closed source company like Oracle}.
In the end, we all want good software, with the maximum amount of permissions and source, free. It's just a question of tweaking the support model to get there.
IMHO, the "Don't call yourself OSS if you're not 100% OSI OSS" is counter-productive. But I understand why they do it, and I understand the history and abuses that caused them to do it. I'd just say if we no-true-Scotsman our approach in a way that precludes profitable, sustainable OSS companies... we're going to have less quality OSS. :(
Still, I really appreciate that you didn't choose a copy-left license.
On the license front, what is the "change license" clause listed? It says something about changing in 5 years. Does this mean it will become Apache licensed in 2027? Why would you put that in there?
In 5 years the initial version becomes Apache 2.0 then the next version and so on and so forth. CockroachDB uses similar license. MariaDB uses that, Redpanda Data and others. You are right that acronym is confusing - it's not Boost license, it's Business License. Every major technological startup turned away from BSD/Apache 2.0 licenses due to inability to compete with cloud providers without technological edge.
However, I'm sad that instead of going with an Open Source license that protects against that, you're using a proprietary license. That alone is a nonstarter for many users, not because they want to compete with you but because they want to protect themselves and make sure they have a firm foundation to build on.
Much of the software you're citing as examples moved from Open Source to proprietary, harming their users in the process, and causing many users to seek alternatives.
So, if this were AGPL and I have an closed webservice that uses this, it is in the clear.
The performance improvements are not worth the legal/compliance overhead of adding non-FOSS to my stack. Much less so if the performance improvements are due just to some optimization choice in the underlying system. In the next 5 years, it will be easier to have Redis adding the io_ring optimizations than for this new project to become uniquely better.
I have a SaaS [0], my own (AGPL, by the way) open source project [1] and also work have the occasional contract job. It's much easier to say "just use Redis because it is FOSS and it gets the job done" across the board then to try to special-case the tools based on licensing requirements.
[0]: https://communick.com
[1]: https://hub20.io
Absolutely not. First of all, you can always use any software under AGPL as released without any limitation.
Secondly, if you were to make changes and release them you can still run it any way you want.
Thirdly, if you are making changes for internal use, and for some reasons your really want to keep them secret, you can STILL run the database to your heart's contents as long as you don't provide it as a service to the public.
AGPL is much more permissive than people think. It's just providing a degree of protection to developers and and users from patent trolls and other uncooperative entities.
There really aren't that many popular AGPL server software to start with though, so it's an unfairly biased question.
From their FAQ:
> The market is quickly moving to consume most software as a service. This is a time of incredible opportunity for open source projects, with the potential to foster a new wave of great open source server side software. The reality, however, is that once an open source project becomes interesting, it is too easy for large cloud vendors to capture all the value but contribute nothing back to the community.
HN discussion from 2018: https://news.ycombinator.com/item?id=18229452
FAQ: https://www.mongodb.com/licensing/server-side-public-license...
Today, MongoDB is an AWS partner: https://aws.amazon.com/quickstart/architecture/mongodb/
The most recent notable example of open-source software going closed because of Amazon specifically is Elasticsearch, which switched from Apache to SSPL. The announcement blog post was titled "Amazon: NOT OK - why we had to change Elastic licensing".
HN discussion from 2021: https://news.ycombinator.com/item?id=25833781
FAQ: https://www.elastic.co/pricing/faq/licensing
Here [A]GPL is Schrödinger's license. At the same time people complain that it's too restrictive and not restrictive enough.
If you want to completely prohibit any SaaS usage closed source is the only option, like BSL. And I'll stay away from your software.
Also, smaller companies like to play safe. If FAANG with their million lawyers aren't touching it, why should I take the risk.
This is a first. If anything, it seems clear that a FLOSS license such as AGPL achieves exactly the opposite: is highly protective of the users' best interests, both in the short term and long term.
Could you elaborate on why do you think that users are harmed by standard, run-of-the-mill FLOSS licenses?
However, if you are a 'user' in the sense of a company wanting to make software to sell as a service, it doesn't protect you very much, and can make you vulnerable.
*Assuming it's not a memory-store as a service, obviously.
However this is rather uniquely due to Google's monorepo infrastructure. Google quite literally has all its products in the same source code repository, being built and statically linked together to create a single system binary (or so I am told--I've never worked there). In this case AGPL would virally infect the rest of your code and you could find yourself in trouble. Even if they don't, they might get hit with a discovery request for the source code of some project to prove compliance, and in the process have to provide their entire monorepo and all its business secrets to the court. Etc. Etc.
But these are concerns which stem directly from Google's monorepo architecture. It's not generally true of the whole industry. And yet Google's fear of copyleft--rational though it may be in their particular instance--has been copied throughout the entire industry. 99% of the companies out there have nothing to fear from the AGPL. But hey, if you're some random in-house lawyer at company XYZ, who are you to question Google's legal precedent?
That does not meet the definition of "user", does it?
In addition, if that was the motivation behind that absurd claim, why not be honest and just say "we don't release our software under a FLOSS license because what we actually want is to keep it proprietary"?
I could convince a legal team to allow an AGPL service, depending on how we were using it. It would be difficult to get them to allow us to use this license, and I definitely couldn't convince them to allow us to contribute to software using this license.
Effectively, you've ensured you won't have a community, because in reality the project is closed source, and dead if your company dies, which means it's not an option for me to consider.
I hope people will understand the reason behind bsl. I hope people will see apache there and will help us to grow. Based on HN comments some already do.
I understand the emotions people have regarding licenses that do allow them to work with software as they please - but I also think it’s useless to argue around licensing of a software instead of focusing on the good it can do. Something being proprietary has NO impact on your choice to buy it when outside of the software world. Why does it here?
I do have practical first hand experience with both. I can get a legal team to approve the use of AGPL, but I cannot get them to approve BSL, unless we've signed a contract with the company.
Having an understanding of the license, and some first hand experience seems like it would be a pretty important thing, when choosing it, especially for a business.
BSL is at least making an attempt to address the legitimate concerns of businesses -- both sides -- in a balanced and pragmatic way. I have no dog in this fight, being neither a BSL licensor nor licensee (nor AGPL for that matter), but having been party to the legal conversations at large companies I understand why BSL is generally considered to be more acceptable than AGPL. BSL may or may not be the right license for this software, I have no opinion, but businesses don't care about arguments from ideological purity. If BSL satisfies their requirements better than AGPL, and anecdotally all the evidence seems to support this notion, then they will go with BSL software.
To put it another way, if OSI is serious about being in the conversation about the future of software licensing, they must address the reasons so many companies refuse to adopt AGPL software.
Transferring copyright to another company, without a contract with that company, for the most part is something a legal team isn't going to agree to. BSL only addresses one side's concerns, and it's not in your favor.
AGPL, when used for internal services that customers don't interact with, is generally not that difficult to get approved. How many companies are using mongodb, for instance?
If it was AGPL I'd have been a happy camper though.
No, there are plenty that still use permissive licenses.
GitLab uses MIT and a custom license for EE: https://docs.gitlab.com/ee/development/licensing.html
Deno uses an MIT license and has some secret sauce that is currently just in hosted services AFAIK: https://github.com/denoland/deno/blob/main/LICENSE.md
PlanetScale has hosted services and an open source tool called Vitess which is Apache licensed: https://planetscale.com/ https://github.com/vitessio/vitess
Finally Redis has a BSD licensed core, a source available license for additional modules, and a closed source license for enterprise. https://redis.com/legal/licenses/
Was excited to see the project but now seeing it is not Open Source it means 1/10th of value
https://www.gnu.org/licenses/license-list.en.html
https://opensource.org/licenses/alphabetical
That's a big red flag.
They are explicitly not a charity. Isn’t it okay to be a for profit company that sells software with a closed copyrighted source?
If Red Hat could change the license, would they? Maybe. (Not now but you can’t say the leadership won’t make that decision during financial crisis).
That there's freeloaders that benefit from the work without having to expend labour is a side-effect, not the primary motivation.
Chosing to release software under a non-FLOSS, proprietary license is hardly innovative, and I'm afraid trying to frame it that way sounds like you want to have your FLOSS cake and eat it too.
I ask this because I'm unsure if AWS Redis has any modifications on top of the Redis software itself, which would affect the speed, or even make it a bit slower. For example I know MS Azure's version of Redis restricted certain commands, and from a quick search AWS does something similar: https://docs.aws.amazon.com/AmazonElastiCache/latest/red-ug/...
(edit: added "affect the speed" for clarity)
Also as a Redis replacement, it's not clear what durability is offered, and for most Redis use cases this is close to the first question
It does sound like extendible hashing might have downsides in some scenarios also.
I'm not hip to how much new stuff is backport-able, so this may preclude Ubuntu 20.04, for instance. You lose the "LTS" part if you compile your own kernel, if you manage to make it functional at all.
Note: I never use kernel modules due to the issues rhel/debian and I have had with such things in the distant past.
I'm not sure that concern is justified. It seems io_uring was pushed as part of the 5.1 linux kernel release, and Ubuntu 20.04 LTS seems to have been shipped with 5.4.
https://packages.ubuntu.com/search?keywords=linux-image-gene...
Also, a quick Google search pointed to io_uring patches for Ubuntu 18.04 LTS.
Of course, it is also possible there are situations where it doesn't perform as well.
The tradeoff the way I see it - one needs to implement 200 Redis commands from scratch. Besides, I think DF has a marginally higher 50th percentile latency. Say, if Redis has 0.3ms for 50th percentile, DF can have 0.4ms because it uses message passing for inter-thread communication. 99th percentiles are better in DF for the same throughput because DF uses more cpu power which reduces variance under load.
Re-durability - what durability is offerred by Redis? AOF ? We will provide similar durability guarantees with better performance than AOF. We already provide snapshotting that can be 30-50 faster than of Redis.
If I recall ScyllaDB has some excellent examples of demonstrating this particular tradeoff visually. A simple option would be a scatter plot where X = latency, Y = load or similar, with points coloured according to the system under test. Probably there is a better option, but this would likely be enough to sell me at least
We plan to implement everything but your votes can affect the priority of the tasks.
Basically, I worked in a cloud company in a team that provided a managed service for Redis and Memcached. I witnessed lots of problems that our customers experienced due to scale problems of Redis. I knew that these problems are solveable but only if the whole system would be redesigned from scratch. At some point I decided to challenge the status quo, so I left the company and..and here we are.
[0] https://twitter.com/axboe/status/1531371740403617792
Can we see such disparity in benchmark even if we run Ncore instances of redis in parallel?
Out of curiosity, are you discovering any new bottlenecks to performance outside of the software, given Dragonfly is able to process far more qps than most systems? I imagine the network and disk I/O could become stressed, but also I wonder if it breaks any assumptions of cross-core performance, hypervisors, etc. I know that cloud offerings typically mean that you can attach ginormous disk IOPS and NICs, but surely there are limits.
I was mostly running on AWS. In terms of hardware, for small-packets loadtests, most systems are constrained on throughput, i.e. number of packets per second. Some instances saturate on interrupts reaching 100% CPU on all cores and some can not even saturate the CPU and you will see that CPU is at 60% but you can not go beyond in throughput. The best systems network-wise are c6gn family types. They are also better than instances that other cloud provide. btw, you mentioned hypervisors... About 8 months ago I opened a bug on AWS Graviton team https://github.com/amzn/amzn-drivers/issues/195 - about performance issue they had on their instances at high throughput. Recently they issued the fix. I suspect it was in their hypervisor.
In terms of my software I found many performance bugs at those speeds. For example, using a default allocator is a big no. I use mimalloc for uncontended allocations. In general, you can not use mutexes and spinlocks at those speeds. Those will just cripple the system. Sometimes it can be very annoying since you can not rely on a 3rd party library without carefully analyzing its design. For example, I could not use openmetrics c++ library because it was not performant enough. Even to implement a simple counter, say to gather statistics for INFO command becomes an interesting engineering problem: With share nothing architecture, I use a lot of thread-local counters that I aggregate only when stats are pulled.
As a general note, I expect that Dragonfly will stay very performant with the tailwinds from recent hardware advancements. For example, c7g (Graviton 3) is much better than c6g and DF shows it.
See https://s3-docs.fd.io/vpp/22.06/developer/extras/vcl_ldprelo...
dragonfly is a linking of a library and a main file dfly_main.cc so without this file you will have the lib.
Redis is "Remote Dictionary Server". You gonna loose the remote part :)
We do provide atomicity guarantees for all operations like Redis! We use an algorithm from a 2014 paper - see our readme, we provide the link to the paper.
For SSD based storage, it’s getting 50k reads/sec PER core and scales linearly with # of cores you have in your cluster. (They achieved 8MM reads/sec with 384 cores)
https://apple.github.io/foundationdb/performance.html
So if I have 1 machine and increase from 2 to 256 core the throughout will scale linearly without the SSD ever being a bottleneck?
For example, INCR would require one read followed by one write of the new value, and of course this will result in very inefficient mutation range conflicts (which must be retried for another couple of round trips) if you have frequent updates of the same keys in multiple concurrent transactions.
https://apple.github.io/foundationdb/api-python.html#api-pyt...
That said, I still don't think that it is necessarily the perfect match for implementing some of the Redis data structures.
Redis is basically a very performant, single-threaded (mostly) single-node in-memory datastructures system with an efficient and readable server protocol strapped to it.
FoundationDB is a completely different beast that has like 6+ distinct roles, and is optimized almost exclusively for interactive serializable transactions, range reads, and correctness.
They’re just completely different things, I recommend reading the FoundationDB paper to get a sense for its architecture. The amount of “steps” involved in processing an FDB write is much higher than in Redis.
A lot of projects say "faster" without giving some hint of the things they did to achieve this. "A novel fork-less snapshotting algorithm", "each thread would manage its own slice of dictionary data", and "core hashtable structure" are all important information that other projects often leave out.
Could you get me a one liner on the helio library is it used as a fiber wrapper around the io_uring facility in the kernel? Can it be used as a standalone library for implementing fibers in application code?
Also it seems that spinlock has become a defacto standard in the DB world today, thanks for not falling into the trap (because 90% of the users of any DB do not need spinlocks).
Another curious question would be - why not implement with seastar (since you're not speaking to disk often enough)?
Re helio: You will find examples folder inside the projects with sample backends: echo_server and pingpong_server. Both are similar but the latter speaks RESP. I also implemented a toy midi-redis project https://github.com/romange/midi-redis which is also based on helio.
In fact dragonfly evolved from it. Another interesting moment about Seastarr - I decided to adopt io_uring as my only polling API and Seastar did not use io_uring at that time.
1. I speak fluently C++ and learning Rust would take me years. 2. Foodchain of libraries that I am intimately fimiliar with in C++ and I am not familiar with in Rust. Take Rust Tokyo, for example. This is the de facto standard for how to build I/O backends. However if you benchmark Tokyo's min-redis with memtier_benchmark you will see it has much lower throughput than helio and much higher latency. (At least this is what I observed a year ago). Tokyo is a combination of myriad design decisions that authors of the framework had to do to serve tha mainstream of use-cases. helio is opinionated. DF is opinionated. Shared-nothing architecture is not for everyone. But if you master it - it's invincible.
http://docs.seastar.io/master/tutorial.html#coroutines
The docs make some of the differences clear. Worth reading the GitHub repo readme.
1. https://docs.keydb.dev/
2. https://aerospike.com/
Will definitely follow this to see how it develops. Good luck.
I see only throughput benchmarks. Redis is single threaded, beating it at latency would have been far more impressive.
Do you have latency benchmarks at peak throughput?
Now, please take into account that DF maintains 99th percentile of 1ms at 3M! qps and not at 200K.
https://github.com/dragonflydb/dragonfly/blob/main/LICENSE.m...
I understand wanting to protect your work from someone else turning into a service, but I will need to get our org's legal team to review it first.
[1]: https://github.com/rust-lang/rust/issues/29594
[2]: https://github.com/rust-lang/rust/issues/77071
1. I speak fluently C++ and learning Rust would take me years. 2. Foodchain of libraries that I am intimately fimiliar with in C++ and I am not familiar with in Rust. Take Rust Tokyo, for example. This is the de facto the standard for how to build I/O backends. However if you benchmark Tokyo's min-redis with memtier_benchmark you will see it has much lower throughput than helio and much higher latency. (At least this is what I observed a year ago). Tokyo is a combination of myriad design decisions that authors of the framework had to do to serve the mainstream of use-cases. helio is opinionated. DF is opinionated. Shared-nothing architecture is not for everyone. But if you master it - it's invincible. (and yeah - there is zero chance I could write something like helio in Rust)...
[1]: https://github.com/DataDog/glommio
Redis is fast enough. Read/write speed isn't usually the bottleneck, it's limiting your data set to RAM. I've long ago switched to a disk-backed Redis clone (called SSDB) that solved all my scaling problems.
https://github.com/dragonflydb/dragonfly#background
In practice, software architecture is forever. It is nearly impossible to change the fundamental architecture of software people use because applications become adapted to the specific behavioral quirks of the architecture i.e. Hyrum's Law, which would not be preserved if the architecture was re-designed wholesale.
But yes, Redis is very much designed against the grain of modern hardware. It also is a very minimalist design, that works well within its limits, but falls down hard when you push those limits, particularly with snapshotting and replication.
Modern is used here as a selling point