Streams (https://redis.io/topics/streams-intro) are going to be awesome. Blocking sorted set pops will also be quite useful. Looking forward to this being widely deployed enough to start using them.
I don't see TLS support in the release notes though I remember antirez mentioning reviewing a PR for it. Is that being handled separately and possibly back patched?
Unfortunately there was no way to put TLS so late in the game. We tried, but after I and my colleague Fabio reviewed the code, it was clear that in order to be a complete implementation the TLS support hooked and modified many things in the Redis internals, including sensitive things like replication. We'll try during Redis 6, now that we have time to do the things in a proper way.
Thanks! Btw I really hope we'll be able to go forward with the TLS patch ASAP. For the same reasons as above it will be great if we could merge/adapt it in the early stage of Redis 6, so that there will be a lot of time to test it.
AFAIK Microsoft abandoned the Win port, and AFAIK they run the Linux version on Azure. However with Windows 10 Linux subsystem you can run Redis for development without issues, just "make" and it's going to work out of the box.
They started the port after seeing I had no interest in writing a port for Windows. They stopped it IMHO when they realized, basically, the same things that prevented me from starting the project at all, that is: it runs very well on Linux and is very hard to port to Windows in a serious way, because of Windows lack of fork(). If you ask me Windows should absolutely get a serios POSIX layer ASAP and just let the Windows users use all the OSS that is out there.
Which is what the Windows Subsystem for Linux does.
If I recall correctly, Azure uses some combination of Windows Docker container and WSL support now to run Redis on Windows.
Though from what I heard (as scuttlebutt/rumor/lay person reading the tea leaves), that isn't why they abandoned the port (as native ports are still handy for performance, and encouraged by Microsoft as much as they can), I heard it had more to do with the usual OSS reason of abandoning a port: not enough upstream support/enthusiasm for merging necessary, underlying patches, and not enough time/energy to devote to keeping a fork up-to-date with a wildly diverging upstream in such a situation.
(Compare Git for Windows or Node for Windows, which have been officially recognized upstream, and directly merge a lot of patches upstream from Microsoft's contributions.)
I don't have a dog in the fight and think you (antirez) are doing a good job, and Microsoft independently are doing a good job with OSS the last few years, and while I'm disappointed there wasn't more Reese's Cup peanut-butter-and-chocolate collaboration here between y'all, I'm okay eating my peanut butter and chocolate separately. That's OSS, in all its bazaar beauty.
The Windows Subsystem AFAIK is not meant to run things for production, but I'm not a windows expert indeed.
About the abandoned port, yep we are saying the same thing in different terms probably: the reason why merging from upstream was so hard was probably because the windows port, for lack of POSIX interface, had to reinvent the persistence of Redis in user-space terms. That was why it was so hard to keep the patch in sync AFAIK.
IIRC, WSL hit "general availability" status in a recent release, including on servers, but I believe that yes the guidance is still that WSL is primarily "for developers" and that running production services on it is, in one of the greatest OSS traditions, "at one's own risk", but "allowed" now in a "at least it won't break your warranty" sort of way.
>Yes I totally appreciated your Tweet indeed, don’t get me wrong. I was just trying to communicate that in general I try to ignore stupid rules even if they are pretty damn real in the industry. Thanks!
I would not say Redis has an history of being unstable, actually it is regarded as not failing and not crashing often. However at the same time, some people believe that it is operationally challenging, which is a different problem. That Redis operations are harder than the ones of other DBs is in part true because the fact of having the data in memory, with different mechanisms to backup the data on disk, means that in case of an operation error (not necessarily a bug, just also an admin error), the outcomes can be worse. However in the filed of operations Redis improved year after year.
However this does not answer your question about critical bugs. Basically most of the critical bugs you'll see in the Redis 4.0 changelog is about the new version of replication that was put in Redis in Redis 4.0 itself. The new protocol, called PSYNC2, was harder to implement and has many corner cases because it is able to restore the replication link with the master after it was broken. To do so requires to take the state of the replication intact, and given that slaves can do that (partial resynchronization) after an RDB restart, we found many issues in this respect: after an RDB restart there were cases where the state was not fully restored.
Most of the time it was very hard to trigger such bugs, and in most cases those issues were found analytically and never reported. Yet I feel like I should mark every of such bug as CRITICAL in the release, regardless of the fact nobody was affected for a log time.
I've a non-competitive style when the matter is software :-) I simply think that Redis streams could be very useful, in some way I borrowed Kafka ideas, in other ways I totally refused the Kafka implementation / approach, and in general Redis streams have a lot of cons and pro at the same time in a mixed way, due to the total difference between Kafka end Redis. So I think it's the users that should apply the right tools for the right jobs, I'll avoid any comparison. I just gave Kafka credits in the Redis doc because certain ideas are clearly from Kafka or popularized by Kafka.
Hello Salvatore, in which cases you would not use Redis streams and used Kafka/other solutions ? And in which cases you think Redis streams shine compared to other solutions ?
Hello lossolo, I hope that the community will figure this out. I'll try to make videos, blog posts and programming examples about Streams in Redis, but I don't like to make direct comparisons and I don't want to become a Kafka/other solution expert as well in order to make sure to make fair comparison :-) If I think at Redis Streams, I think they work very well in a set of domains. The king use case IMHO is time series. Having fast range queries is pretty interesting for this use case. Then the second one is events and event sourcing of all the kinds when the number of events-per-second is very sustained. However I've some feeling that also thinks like delayed jobs that are currently implemented on top of blocking lists may be rewritten to use streams and offer better overall features/operations.
Just wanted to affirm this - I saw the streams announcement last year on the same day that I was having issues with Kafka and I implemented it into production for processing IoT time series data right away.
It was probably a bit too early for production use, but it worked perfectly!
One nice advantage of it vs. Kafka is that given that it's so lightweight, a Redis instance with streams can be dedicated to each customer with its own port and firewall setup and give you a really solid security story for clients who are paranoid about such things. (Kafka security is possible too of course...but harder to explain for a layman.)
Thanks antirez - it's great. I've written a python Redis iterator that I'm looking forward to PR'ing to the python client to encapsulate the XREAD timestamp tracking bit and manage timeouts.
I made a similar comment when streams were first announced, but didn't get feedback.
My question is about using streams for event-sourcing, in similar cases to how Kafka is frequently used. In such cases, the expectation for persistence is paramount, as the event log is ultimately the primary source of truth for the entire application. The actual utility of having the event log in memory is only really important while the consumer groups are reading it, after which it mostly just needs to be archived.
So my question is, are the common strategies used by Redis to persist to disk going to try to keep the whole event log in memory, or will they be able to intelligently offload to disk? Otherwise, is the expectation that clients will instead write to the stream, and consumers will be responsible for consuming, writing to some persistent layer like s3, and then removing from the stream?
Hello ralusek, the idea is to provide more the tools for the application layer to move data from/to stream. I don't think we'll have direct support for persisting streams on disk. Honestly streams as implemented in Redis are very adapt to this use case, because of the macro-nodes that are swappable as a single entity, however some time ago I decided that I wanted to focus on the on-memory aspect.
Congrats & thanks Salvatore! Redis Streams is an awesome feature, and bringing Redis Cluster management tools into redis-cli is a world of improvement!
Keep up the great work, hope to see you at next RedisConf!
I have a bad time with redis hosting from time to time. It is surely my own fault. What are your favorite resources/recipes to configure/host redis on your own?
#1 - Congratulations!
#2 - I am interested more in where clustering is going and running Redis inside K8S. Is there something someone can point me to to read on best practices for Redis 5?
35 comments
[ 2.8 ms ] story [ 52.6 ms ] threadI don't see TLS support in the release notes though I remember antirez mentioning reviewing a PR for it. Is that being handled separately and possibly back patched?
I've said it before and I'll say it again, thanks for Redis!
If I recall correctly, Azure uses some combination of Windows Docker container and WSL support now to run Redis on Windows.
Though from what I heard (as scuttlebutt/rumor/lay person reading the tea leaves), that isn't why they abandoned the port (as native ports are still handy for performance, and encouraged by Microsoft as much as they can), I heard it had more to do with the usual OSS reason of abandoning a port: not enough upstream support/enthusiasm for merging necessary, underlying patches, and not enough time/energy to devote to keeping a fork up-to-date with a wildly diverging upstream in such a situation.
(Compare Git for Windows or Node for Windows, which have been officially recognized upstream, and directly merge a lot of patches upstream from Microsoft's contributions.)
I don't have a dog in the fight and think you (antirez) are doing a good job, and Microsoft independently are doing a good job with OSS the last few years, and while I'm disappointed there wasn't more Reese's Cup peanut-butter-and-chocolate collaboration here between y'all, I'm okay eating my peanut butter and chocolate separately. That's OSS, in all its bazaar beauty.
About the abandoned port, yep we are saying the same thing in different terms probably: the reason why merging from upstream was so hard was probably because the windows port, for lack of POSIX interface, had to reinvent the persistence of Redis in user-space terms. That was why it was so hard to keep the patch in sync AFAIK.
Right on!
I mean that with no disrespect but the 4.x branch seemed to have a number of CRITICAL bug
https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEA...
Again, many thanks to Antirez who’s mostly a one-man-show.
I would not say Redis has an history of being unstable, actually it is regarded as not failing and not crashing often. However at the same time, some people believe that it is operationally challenging, which is a different problem. That Redis operations are harder than the ones of other DBs is in part true because the fact of having the data in memory, with different mechanisms to backup the data on disk, means that in case of an operation error (not necessarily a bug, just also an admin error), the outcomes can be worse. However in the filed of operations Redis improved year after year.
However this does not answer your question about critical bugs. Basically most of the critical bugs you'll see in the Redis 4.0 changelog is about the new version of replication that was put in Redis in Redis 4.0 itself. The new protocol, called PSYNC2, was harder to implement and has many corner cases because it is able to restore the replication link with the master after it was broken. To do so requires to take the state of the replication intact, and given that slaves can do that (partial resynchronization) after an RDB restart, we found many issues in this respect: after an RDB restart there were cases where the state was not fully restored.
Most of the time it was very hard to trigger such bugs, and in most cases those issues were found analytically and never reported. Yet I feel like I should mark every of such bug as CRITICAL in the release, regardless of the fact nobody was affected for a log time.
Stream data type looks very interesting as well. Is the goal to be competitive to Kafka-like use-case?
I've a non-competitive style when the matter is software :-) I simply think that Redis streams could be very useful, in some way I borrowed Kafka ideas, in other ways I totally refused the Kafka implementation / approach, and in general Redis streams have a lot of cons and pro at the same time in a mixed way, due to the total difference between Kafka end Redis. So I think it's the users that should apply the right tools for the right jobs, I'll avoid any comparison. I just gave Kafka credits in the Redis doc because certain ideas are clearly from Kafka or popularized by Kafka.
It was probably a bit too early for production use, but it worked perfectly!
One nice advantage of it vs. Kafka is that given that it's so lightweight, a Redis instance with streams can be dedicated to each customer with its own port and firewall setup and give you a really solid security story for clients who are paranoid about such things. (Kafka security is possible too of course...but harder to explain for a layman.)
Thanks antirez - it's great. I've written a python Redis iterator that I'm looking forward to PR'ing to the python client to encapsulate the XREAD timestamp tracking bit and manage timeouts.
My question is about using streams for event-sourcing, in similar cases to how Kafka is frequently used. In such cases, the expectation for persistence is paramount, as the event log is ultimately the primary source of truth for the entire application. The actual utility of having the event log in memory is only really important while the consumer groups are reading it, after which it mostly just needs to be archived.
So my question is, are the common strategies used by Redis to persist to disk going to try to keep the whole event log in memory, or will they be able to intelligently offload to disk? Otherwise, is the expectation that clients will instead write to the stream, and consumers will be responsible for consuming, writing to some persistent layer like s3, and then removing from the stream?
Keep up the great work, hope to see you at next RedisConf!
https://hub.docker.com/_/redis/
On Ubuntu this guide works well https://www.digitalocean.com/community/tutorials/how-to-inst...
Only issue I've had is remembering to add the port to iptables
https://redis.io/topics/admin
https://redis.io/topics/latency
In general our operations documentation does not still reflect all the good things put into Redis 4, we'll update everything shortly.