The simple clients just wouldn't enable client tracking, it's off by default.
It also wouldn't work very well as a proxy because then if any clients don't go through the proxy but change some of the data, there's no way for all the proxy-using clients to know.
What's the effect of this system on consistency guarantees? Seems like clients could end up in a state where they serve stale data due to (undetectably) not receiving the push updates.
It also looks like client / protocol implementations are about to get a lot more complex, even if the feature is being used by the majority of users?
My bad for not being clear enough. I was thinking of, for example, a networking issue where both server and client are still “connected” but delivery never happens. Can be solved with tuning timeouts / client-side TTL as mentioned elsewhere, but I was wondering specifically if this would change Redis’ known guarantees for data consistency, what happens during partition etc. More of a theoretical question.
On the second point, the post talks about having the feature disabled by default; I was curious regarding the protocol itself - it is currently quite simple, and by adding extra cache-control metadata it sounds like correct client implementation might become a lot more complex, but it’s hard to tell without having seen what the new protocol looks like.
Yep your concern about consistency is definitely correct, but the good thing is that the invalidation connection can be pinged continuously by the client, as often the client wants (for instance 10 times every second). As soon as the connection is timing out, the client may decide to no longer serve the cached data. Of course every time the invalidation connection is closed at all, the client should flush the cached data completely or apply a very short TTL to the set of cached data.
About your second point: today I implemented even the feature (in the unstable branch) to support RESP2, that is, the current Redis protocol. But even using RESP3 the complexity is very low. On the data connection side you have just to use the CLIENT command to activate it. In the data connection, what you could receive, if you associate a callback in your client, is push data that notifies you the invalidation connection is no longer active. Push data is normally discarded by the client implementation of RESP3 if no handler is associated. On the invalidation connection, in RESP3 mode, you get again push data with a callback, or if you are in RESP2 mode, you just subscribe to the special channel __redis__:invalidate, and you get the messages about the caching slots that need to be invalidated.
Client cache should be bound to active connection. Strong consistency also require server to receive syncronious confirmation from each client with cache bucket before completion of operation, but it's a lot of overhead and not needed in most of the cases.
Totally agree, moreover here the window of inconsistency can be tuned by pinging the invalidation connection as often as needed. However I'm sure we better have excellent documentation about best practice, like: if the invalidation connection is gone, the cache should be flushed to avoid stale reads, and things like that. Notably the reverse condition is ok: if the data connection is gone, one can reconnect and enable again redirection to the old client ID :-D
Obviously no consistency guarantees. To be fair, strong consistency models don't work for client side caching and strong eventual consistency while works, almost no one in the world implements it yet.
Redis continue to wonder me, it's yet another feature I thought to implement, already implemented by redis itself. And I had same reasoning about hash slots.
I always appreciate antirez' very candid writing style. Comes across as such an earnest and reasonable person. Very low ego, very excited about the technology, but with an eye for how people will be able to adopt it. Wish we saw more tech leaders like that.
Redis is already super useful, but this would be an absolutely killer feature. Server assisted client side invalidation! Something people have always wanted but has never been built.
Implement it quick so I can use it on AWS in a few years!
Funny enough, at my company we did just this. Having local state is a requirement for our uptime and reliability goals, and we have a dumb service that pushes this back to the services via DynamoDB streams, and also dumps the data into S3 in case that service is down.
Isn't DAX only a network-based in-memory cache, and not a in-process in-memory cache? Or is it both?
It's definitely at least network-based, b/c you have to spin up instances of it in your VPC.
However, they also require custom clients, which is a little odd afaiu (b/c in theory DAX is API-compatible with DynamoDB, so why would regular dynamodb sdk clients not work?) so maybe the custom clients also add some in-process-based caching?
A client side cache for an in memory cache. Which, out of the box you can’t use securely on a remote machine so it’s all just local fucking memory.
How many years until Redis supports TLS? They have a cluster product, for those who have too data much to fit in memory on a single machine, but no secure way to actually communicate between nodes or server/client.
How is this considered ok?
And yes, I know the traffic can be tunneled via stunnel/similar. Do you realise how much tunneling and futzing with ports you have to do, for even a basic setup with HA?
When people have been asking nicely for years, it’s been debated back and forth for literally half a decade, what purpose exactly is there to another polite “oh excuse me could we have some basic security like existed 20 years ago?”.
Adding features without any apparent care for security is negligent. How many people are using Redis over an unencrypted channel? How much potential is there for that to leak sensitive data?
Hi, Redis 6 will have SSL, however there is no obligation by an OSS developer to put any feature inside a software. If you like it with its shortcomings you use it otherwise not. If the feature is about security, nothing changes that. I provide a tool and it's up to the users to apply it in a way that makes sense, using or not an SSL proxy and so forth.
If you market a product that is intended to be used in a multi-server environment, but the software has no way to encrypt that traffic, you’re doing active harm.
That there is also a commercial/enterprise offering by your employer which “bundles” TLS support doesn’t help the situation.
You’ve had an issue open for 5 years on this. You’ve had multiple people submit PRs to add this, and even forks made just to add TLS support - any of which you could have used to make this happen sooner.
Saying “I have no obligation to provide secure software” is a cop out when you actively develop new features that are almost guaranteed to need the security you’ve refused to add thus far.
AWS which is basically the main competitor of Redis Labs has SSL since forever and they are even the contributors of the SSL PR that will be merged. So sorry, conspiracy theory not correct. About obligation I disagree, if we are in doubt let's check what the law says. I've no obligation whatever your opinion is.
Would you please not be so rude on HN? You're breaking the site guidelines. I'm sure you can make your substantive points without doing that, even if you have reason to be frustrated about something.
Attacking someone else for their work is particularly unwelcome here. Imagine if everyone did that—what reason would people have to discuss their work here?
I agree 100% that a developer isn’t required to implement any feature. One thing to think about is that it’s not necessarily a single user or a single team weighing costs and benefits. In an organization with separate dev and ops teams, a dev team may pick a solution for the benefits but the ops team bears the costs of operations and security.
If you re-read you'll find that I suggest an incremental epoch to avoid any issue :-D however starting with the idea of time to later hint about a better solution is a lot more understandable IMHO.
This might be a bit of a strange question, but it's interesting to think about and I know antirez will probably read it here, so I want to bring it up:
Could there be some potential privacy or data-leakage concerns with this, since the server is now storing and acting on info about what data has been accessed, by which clients, and even when (to a limited degree)?
For example, say that some kind of sensitive data is stored in redis (e.g. user private messages). By watching for relevant invalidation messages (or if there's any way for clients to read the data in the Invalidation Table directly), it could be possible to learn some info about how that data was accessed.
Without knowing more about the implementation, I don't know if this is even a concern at all, and I haven't come up with a specific method of abusing it even if it is. I think it's probably something to keep in mind though.
MONITOR can only be used through redis-cli or telnet, you already need a very high level of access to the server for that. I'm thinking about more subtle methods of using this info, where a client might be able to get info about the activity of other clients (or other users of the same client through something like a web application).
If someone already has full access to the redis server, then yeah, I can't imagine how this would expose anything more than what's already available.
What? MONITOR is a command like all other commands - you can run it from a client, or really, anything that can establish a TCP a connection to the Redis server. (Eg telnet.).
Right, but if someone has the ability to run arbitrary commands on your redis server, they can already look at or do whatever they want with your data. We're agreeing: if someone can already run MONITOR, this won't make anything worse.
Yes, and my point is, if the client is connecting to a redis server, it can just run MONITOR to see all the "sensitive" data you are talking about, and thus, this new feature poses no additional privacy risk. If you cannot trust your client to talk to your database, this isn't a problem with the database, but a problem with the client.
In your specific example, your client is also probably the one issuing the commands to store and retrieve the private messages in Redis - meaning, it already has access to all the data required for a breach. If you have an unrelated workload that is accessing a shared redis, but it does not need access to the sensitive data, perhaps, use an ACL to limit clients from the alternate workload to a different redis DB (which provides some level of isolation), or simply use a different redis entirely, and don't allow clients running the alternate workload to connect to the database that contains sensitive data.
Everything could be a risk, but in the specific case of a software device holding data, I believe that the main problem is the exposure of the data itself, so in the specific case of Redis the fact that the database stores informations about which key was retrieved by which client should not be an issue AFAIK. A few random arguments:
1. The database anyway sees such connections and there is a continue stream of activity about clients, so there is also an instantaneous state that discloses who does what, at any time.
2. Companies trace everything in order to have data to do debugging when there are problems, so this information is stored at a much better level of details in other places as well.
3. There is no obvious way this could be exploited.
4. In general it is supposed that the clients can trust the server.
5. If you already violated Redis in order to read such info (not accessible via APIs), you can collect a lot more.
So based on initial thoughts I don't think this is going to be ever a security problem.
It doesn’t seem to be a major vulnerability unless you have the ability to guess keys you don’t have access to via hash and get their invalidation notice. You still would have to force a hash collision, won’t know which key caused the invalidation (unless you can guarantee that there is just your key and their key on the hash), and won’t get new data. All in all it seems pretty minimal, especially since it’s an insider attack where micro services don’t trust each other.
This question doesn't make any sense. If your users can access the ip/port of your redis instance, you're already hosed regardless of whether the redis clients cache requests or not. The only hypothetical surface area opened up is if an attacker manages to install a virus on your server to snoop memory via sidechannels. But in this case, your data is already vulnerable using normal queries (the data must reside in memory eventually), again, regardless of whether data is being cached client side or not. Most applications most likely already cache data, at least in a variable or something for the duration of a particular function.
Shameless plug for in-language storage. I've found that using a fast language/framework with a local cache gives you most of the advantages of a shared cache like Redis without all the complexity. This feature feels like a tacit admission of that.
In "fast" languages (C# Java Go Rust Cxx) allocation tends to be fairly efficient so you're not wasting tons of space keeping objects in RAM.
These fast languages can max out the network interface most of the time using the right frameworks. In them, memory storage is also over 100x faster than remote like Redis.
Now if you've got something in PHP or Rails that can only manage 100 requests/second remote caches make tons of sense. The network overhead is nothing compared to how slow the language is. This advantage erodes when you're closer to native speed and local memory access is 1000 times faster than retrieving cached data over network.
TLDR: Redis and alike make sense when your language or framework is slow and the bottleneck is CPU use. They don't give you much when networking and memory bandwidth are your bottleneck because the language runs near native speed.
Yeah but this doesn't work with something like microservices, or even Kubernetes based deployments, right?
It might take you 3, 40, or a 100 tries for the same request till you hit a node that has your data cached. It could work with extremely high req/seconds, though in an unpredictable way.
A lot more knowledge in the relevant space that I have. I could spend a week or two learning necessary information, analysing our use cases and then implementing it finally (which I would never get the time to, from a business perspective).
Or I could use an off the shelf product that is probably better than whatever I could come up with, given, even, unrealistic amounts of time to perfect it. I could come up with something more suited to our use case only, but the moment the use case changes, I'd wonder why I didn't just go with Redis.
I haven't actually looked into a DIY method though. I just know I want something like Redis for our user table, because it's hit a lot, and the traffic tends to be repetitive.
>> A lot more knowledge in the relevant space that I have.
Relevant to what? You've already got a microservice mesh, which means you've also got a microservice architecture that's supposed to handle all of the RPCs.
Adding a layer of Redis on top is probably going to introduce more entities into the system and make it less homogenous.
Interesting. I built something somewhat similar about 2 years ago (not big on blogging so it’s not published - probably end up hurting my career :( )
The gist is using pubsub to send invalidation messages consisting of item ID and time stamp, but also store the latest invalidation message into a set that is polled on every longer time period (10 seconds in my case). Listening to pubsub and polling are done per server that hosts the in memory cache for all items in the cache.
One interesting complication was how to make sure I don’t end up with the wrong (stale) cached item due to race condition between the read and the message passing. Imagine 2 thread, thread A reading and populating the cache, thread B listening to invalidation events. If thread A get held up (context switching, slow network, etc) it can actually populate into the cache after thread B has received and processed the invalidation message. To avoid that I ended up holding the invalidation messages and their receive time in the receiving server for a short duration (20 seconds - twice poll time). I required and in memory cache population to tell me the time from before the read command was sent to the cache. At the time of cache population, if I saw in had any invalidation message marked after the time the read command started I discarded that item instead of caching it. In highly concurrent, highly distributed application this turned to be a must-have.
Also, mainly to placate my fears, I had “panic mode”. If I didn’t see any message (I sent periodic maintenance messages) or if I couldn’t execute multiple polling attempts I dumped the whole cache and wouldn’t let any new item be ca he’d until communication was restored.
Forgot to mention - writes also invalidated the cache locally so maintain “read your own writes” for multi step algorithms. That was crucial for us and I suspect for a lot of other use cases.
It reminds me of OVSDB + IDL (C client): https://tools.ietf.org/html/rfc7047 . The C client keeps a partial replica of the database, and the database notifies all the clients with the changes.
It makes super fast to work with data (it is already in process memory), and the centralized server handles consistency and transaction serialization.
Whether or not you like this feature or think it's a good idea, the brunt of this will be born by library writers. Yeah you can use a dedicated thread to listen to these push messages, but really this is tacitly blessing the asyncio's and whatever else.
As you can probably tell, I'm not at all a fan of the way Redis is moving. It hit a sweet spot as a "better memcached" but apparently aspires to be a real database, so now this nice simple tool is all crazy. The new protocol just kinda sucks for synchronous use.
I'd go with a hand-written in-memory caching scheme before I ever considered a Redis sandwich with this feature. Seems like a complexity nightmare and very difficult to reason about. Yeah maybe I'm being crusty, but I'm just thinking about maintaining and debugging something like this and having cold sweats. The linked redisconf talk clearly bears this out. Sounds like they had a hell of a time. But then that also provides some measure of job security?
Your hand tuned client cache won’t have active invalidation like this new proposal does, so you are stuck waiting for ttl to expire and ttl windows where data is inconsistent between nodes.
How will client libs that do connection pooling handle this? If a client opens a pool of 50 connections, will they be identified as 50 different clients, and receive 50 different pushes for invalidation? I guess the client libs will have to build a layer to balance caching on top of the pool.
55 comments
[ 2.9 ms ] story [ 105 ms ] threadI think you mean slot 5 in this sentence.
It also wouldn't work very well as a proxy because then if any clients don't go through the proxy but change some of the data, there's no way for all the proxy-using clients to know.
It also looks like client / protocol implementations are about to get a lot more complex, even if the feature is being used by the majority of users?
On the second point, the post talks about having the feature disabled by default; I was curious regarding the protocol itself - it is currently quite simple, and by adding extra cache-control metadata it sounds like correct client implementation might become a lot more complex, but it’s hard to tell without having seen what the new protocol looks like.
About your second point: today I implemented even the feature (in the unstable branch) to support RESP2, that is, the current Redis protocol. But even using RESP3 the complexity is very low. On the data connection side you have just to use the CLIENT command to activate it. In the data connection, what you could receive, if you associate a callback in your client, is push data that notifies you the invalidation connection is no longer active. Push data is normally discarded by the client implementation of RESP3 if no handler is associated. On the invalidation connection, in RESP3 mode, you get again push data with a callback, or if you are in RESP2 mode, you just subscribe to the special channel __redis__:invalidate, and you get the messages about the caching slots that need to be invalidated.
Implement it quick so I can use it on AWS in a few years!
It's definitely at least network-based, b/c you have to spin up instances of it in your VPC.
However, they also require custom clients, which is a little odd afaiu (b/c in theory DAX is API-compatible with DynamoDB, so why would regular dynamodb sdk clients not work?) so maybe the custom clients also add some in-process-based caching?
A client side cache for an in memory cache. Which, out of the box you can’t use securely on a remote machine so it’s all just local fucking memory.
How many years until Redis supports TLS? They have a cluster product, for those who have too data much to fit in memory on a single machine, but no secure way to actually communicate between nodes or server/client.
How is this considered ok?
And yes, I know the traffic can be tunneled via stunnel/similar. Do you realise how much tunneling and futzing with ports you have to do, for even a basic setup with HA?
Adding features without any apparent care for security is negligent. How many people are using Redis over an unencrypted channel? How much potential is there for that to leak sensitive data?
If you market a product that is intended to be used in a multi-server environment, but the software has no way to encrypt that traffic, you’re doing active harm.
That there is also a commercial/enterprise offering by your employer which “bundles” TLS support doesn’t help the situation.
You’ve had an issue open for 5 years on this. You’ve had multiple people submit PRs to add this, and even forks made just to add TLS support - any of which you could have used to make this happen sooner.
Saying “I have no obligation to provide secure software” is a cop out when you actively develop new features that are almost guaranteed to need the security you’ve refused to add thus far.
Attacking someone else for their work is particularly unwelcome here. Imagine if everyone did that—what reason would people have to discuss their work here?
https://news.ycombinator.com/newsguidelines.html
Disclaimer: happy user of a redis cluster :)
Could there be some potential privacy or data-leakage concerns with this, since the server is now storing and acting on info about what data has been accessed, by which clients, and even when (to a limited degree)?
For example, say that some kind of sensitive data is stored in redis (e.g. user private messages). By watching for relevant invalidation messages (or if there's any way for clients to read the data in the Invalidation Table directly), it could be possible to learn some info about how that data was accessed.
Without knowing more about the implementation, I don't know if this is even a concern at all, and I haven't come up with a specific method of abusing it even if it is. I think it's probably something to keep in mind though.
If someone already has full access to the redis server, then yeah, I can't imagine how this would expose anything more than what's already available.
In your specific example, your client is also probably the one issuing the commands to store and retrieve the private messages in Redis - meaning, it already has access to all the data required for a breach. If you have an unrelated workload that is accessing a shared redis, but it does not need access to the sensitive data, perhaps, use an ACL to limit clients from the alternate workload to a different redis DB (which provides some level of isolation), or simply use a different redis entirely, and don't allow clients running the alternate workload to connect to the database that contains sensitive data.
1. The database anyway sees such connections and there is a continue stream of activity about clients, so there is also an instantaneous state that discloses who does what, at any time.
2. Companies trace everything in order to have data to do debugging when there are problems, so this information is stored at a much better level of details in other places as well.
3. There is no obvious way this could be exploited.
4. In general it is supposed that the clients can trust the server.
5. If you already violated Redis in order to read such info (not accessible via APIs), you can collect a lot more.
So based on initial thoughts I don't think this is going to be ever a security problem.
In "fast" languages (C# Java Go Rust Cxx) allocation tends to be fairly efficient so you're not wasting tons of space keeping objects in RAM.
These fast languages can max out the network interface most of the time using the right frameworks. In them, memory storage is also over 100x faster than remote like Redis.
Now if you've got something in PHP or Rails that can only manage 100 requests/second remote caches make tons of sense. The network overhead is nothing compared to how slow the language is. This advantage erodes when you're closer to native speed and local memory access is 1000 times faster than retrieving cached data over network.
TLDR: Redis and alike make sense when your language or framework is slow and the bottleneck is CPU use. They don't give you much when networking and memory bandwidth are your bottleneck because the language runs near native speed.
It might take you 3, 40, or a 100 tries for the same request till you hit a node that has your data cached. It could work with extremely high req/seconds, though in an unpredictable way.
Or I could use an off the shelf product that is probably better than whatever I could come up with, given, even, unrealistic amounts of time to perfect it. I could come up with something more suited to our use case only, but the moment the use case changes, I'd wonder why I didn't just go with Redis.
I haven't actually looked into a DIY method though. I just know I want something like Redis for our user table, because it's hit a lot, and the traffic tends to be repetitive.
Relevant to what? You've already got a microservice mesh, which means you've also got a microservice architecture that's supposed to handle all of the RPCs.
Adding a layer of Redis on top is probably going to introduce more entities into the system and make it less homogenous.
The gist is using pubsub to send invalidation messages consisting of item ID and time stamp, but also store the latest invalidation message into a set that is polled on every longer time period (10 seconds in my case). Listening to pubsub and polling are done per server that hosts the in memory cache for all items in the cache.
One interesting complication was how to make sure I don’t end up with the wrong (stale) cached item due to race condition between the read and the message passing. Imagine 2 thread, thread A reading and populating the cache, thread B listening to invalidation events. If thread A get held up (context switching, slow network, etc) it can actually populate into the cache after thread B has received and processed the invalidation message. To avoid that I ended up holding the invalidation messages and their receive time in the receiving server for a short duration (20 seconds - twice poll time). I required and in memory cache population to tell me the time from before the read command was sent to the cache. At the time of cache population, if I saw in had any invalidation message marked after the time the read command started I discarded that item instead of caching it. In highly concurrent, highly distributed application this turned to be a must-have.
Also, mainly to placate my fears, I had “panic mode”. If I didn’t see any message (I sent periodic maintenance messages) or if I couldn’t execute multiple polling attempts I dumped the whole cache and wouldn’t let any new item be ca he’d until communication was restored.
It makes super fast to work with data (it is already in process memory), and the centralized server handles consistency and transaction serialization.
As you can probably tell, I'm not at all a fan of the way Redis is moving. It hit a sweet spot as a "better memcached" but apparently aspires to be a real database, so now this nice simple tool is all crazy. The new protocol just kinda sucks for synchronous use.
I'd go with a hand-written in-memory caching scheme before I ever considered a Redis sandwich with this feature. Seems like a complexity nightmare and very difficult to reason about. Yeah maybe I'm being crusty, but I'm just thinking about maintaining and debugging something like this and having cold sweats. The linked redisconf talk clearly bears this out. Sounds like they had a hell of a time. But then that also provides some measure of job security?
https://images.app.goo.gl/RgQcYt3MTU3AA3CF9