I've also thought of redis as a distributed cache for nearly as long as I've been familiar with it. This video "I've been using Redis wrong this whole time" caught my attention recently (yup the clickbait title got me) https://www.youtube.com/watch?v=WQ61RL1GpEE and I was pleased to learn a few new things about redis on this subject
I've come to appreciate simple APIs like the one exposed by Redis. Such simple APIs tend to require more careful planning to use but they help you to avoid running operations which are complex and inefficient. They often force you to think about your app architecture in a broader sense in order to solve problems using simpler primitives.
That said, I think the fact that Redis stores everything in memory can make it unsuitable for certain projects as a database where only a small fraction of the whole data set is ever exposed to users at any given time.
The biggest problem with SQL is that it was designed for humans, for writing by hand, not for programmatic access. So it suffers the same problems as shell commands do.
I also love thinking through how to store data, e.g. in Redis, and aligning it with your access patterns. The biggest challenge I’ve faced is that access patterns tend to change over time and Redis isn’t flexible (e.g. you use an HSET to pull user data but now you want to query by a different key). So you probably will end up storing data in SQL for persistence and Redis for caching, anyway.
I suspect this will be unpopular but I'm a Redis avoider these days. I have run it in production for a few years on large projects and while it does what it says on the tin, I'd rather use another solution because having a proper schema and database engine turns out to be a really important thing on anything more than trivial. Numerous scary moments over the years which involve late nights unfucking things which would not have been issues on schema'd and carefully typed databases.
My position is that memcache is still a better cache (with a suitable static type system and serializer) and postgres is still a better database.
Sounds like you were just using Redis for something you shouldn't have been.
I think Redis is great in the space of "I need a little bit more structure than memcache", but if you're rising to the level of unscrewing a data structure or needing nontrivial types, then Redis was not the tool for that job.
I think postgres handles all those scenarios absolutely fine too.
Edit: worth noting when you under-engineer something for Redis, it's far more difficult to get it out again later than it is to add more schema to a relational DB.
I'm not sure there's much of a reason to distinguish these cases from caching.
You're still designing the system the same way... your main considerations are still, "why/how is it OK if this thing is reset/cleared/rolled-back at any moment?" and "why/how is the data here up-to-date/recent enough for my cases?"
One of the things I've come to value as a "must have" for databases is the ability to stream CDC (Change Data Capture) out of the store.
With redis's persistance mechanisms (RDP + AOF), I see how one could set up their own CDC on top of those (tailing the AOF and reimplementing the redis instruction protocol to convert that to proper data changes), but does anyone know of a more structured way of piping out a change log?
I don't see anything obvious like a debezium connector.
I can attest that it is certainly possible to use a Key-Value store as a database in certain conditions. I created one myself to support adding metadata (tags) to objects in the file system replacement system I was designing. Each tag had a key (the object ID it was attached to) and a value (string, number, dateTime, location, etc.).
Once I had it working and could attach hundreds of tags to each of the millions of data objects (Didgets); the whole system started to look like a sparse relational database table. So I used the tagging system to create tables and run queries against them. It turned out to be extremely flexible and fast.
Serious question: why would you use Redis as a database instead of something persistence-focused with a lot of in-memory operations like MongoDB or something? Wouldn't one of those NoSQL stores like Mongo or Cassandra be the best of both worlds?
It depends on the shape of your data and whether or not you need extras like pub/sub. Redis (and Redis protocol compatible databases) have rich support for HashMaps, Sets etc. Previously I've used redis-persistence as the data storage for an RBAC system (Lots of sets, lists, hashmaps, mostly flat objects).
I've also heard of it as a database for online game scoring systems.
This needs 2021 in the title, but it's actually really out of date and might as well have 2011 in the title. Redis has modules which can provide a bunch of extra capabilities that do allow querying, indexing, etc. [Full disclosure: I used to work on Redis and created much of the basis of those modules, but haven't been involved in the past 5 years]
I wonder how does the compare with dynamodb with DAX for equivalent KV store and similar throughput as a caching service. Its a lot more easier to scale to dynamo to reads and writes without worrying about key ranges etc similar to redis.
Totally off topic, but as this is on the Wix engineering blog, is there any chance Wix will ever offer a way to deploy custom code? The last I checked (June) the only option was to use their in-browser editor. (not just talking about editing pages, but when building NodeJS extensions that run server-side)
Any suggestions of tools for redis or other databases to keep state in the database synced with another system?
For example we have a service that writes that continuously does circular writes to a redis database to keep 200 megs up to date for other services to read from.
A lot of time is taken recalculating the data being updated. But its the best way we've found to eventually update even when things are going wrong.
It would be awesome if container snapshotting was good enough that taking snapshots of in-memory databases was a semi-viable option.
The demoe/paper on Quake vm's migrating across servers in particular had some cool techniques for incremental snapshotting. It d send over one image of memory, let the game-server keep running, track what was changing, then send those deltas to the other server. Trying to re-snapshot an entire in-memory database feels sizable, but for slowly changing data incremental snapshots feel like they have potential.
41 comments
[ 3.1 ms ] story [ 89.0 ms ] threadKVRocks: Backed by RocksDB https://github.com/apache/kvrocks
Tendis: Backed by RocksDB https://github.com/Tencent/Tendis
Tidis: Distributed, backed by TiKV https://github.com/tidb-incubator/tidis
That said, I think the fact that Redis stores everything in memory can make it unsuitable for certain projects as a database where only a small fraction of the whole data set is ever exposed to users at any given time.
Which do you prefer, simple APIs, or SQL?
My position is that memcache is still a better cache (with a suitable static type system and serializer) and postgres is still a better database.
I think Redis is great in the space of "I need a little bit more structure than memcache", but if you're rising to the level of unscrewing a data structure or needing nontrivial types, then Redis was not the tool for that job.
Edit: worth noting when you under-engineer something for Redis, it's far more difficult to get it out again later than it is to add more schema to a relational DB.
I'd not recommend it for heavy-use in Production.
autocomplete = inproc caching (no network turnaround). We do that.
job queue = SQS / RabbitMQ. we do that (SQS)
You're still designing the system the same way... your main considerations are still, "why/how is it OK if this thing is reset/cleared/rolled-back at any moment?" and "why/how is the data here up-to-date/recent enough for my cases?"
With redis's persistance mechanisms (RDP + AOF), I see how one could set up their own CDC on top of those (tailing the AOF and reimplementing the redis instruction protocol to convert that to proper data changes), but does anyone know of a more structured way of piping out a change log?
I don't see anything obvious like a debezium connector.
But IMO, I'd be weary about data durability with Redis.
Once I had it working and could attach hundreds of tags to each of the millions of data objects (Didgets); the whole system started to look like a sparse relational database table. So I used the tagging system to create tables and run queries against them. It turned out to be extremely flexible and fast.
To my surprise, it could often outperform traditional DB systems like SQLite or PostgreSQL. https://www.youtube.com/watch?v=Va5ZqfwQXWI
Databases are cool but I don’t see how this is #10 on HN.
I've also heard of it as a database for online game scoring systems.
For example we have a service that writes that continuously does circular writes to a redis database to keep 200 megs up to date for other services to read from.
A lot of time is taken recalculating the data being updated. But its the best way we've found to eventually update even when things are going wrong.
The demoe/paper on Quake vm's migrating across servers in particular had some cool techniques for incremental snapshotting. It d send over one image of memory, let the game-server keep running, track what was changing, then send those deltas to the other server. Trying to re-snapshot an entire in-memory database feels sizable, but for slowly changing data incremental snapshots feel like they have potential.