In similar vein, I’d always thought Redka (https://github.com/nalgeon/redka) was a neat idea since it gives you access to a subset of the Redis API backed by either SQLite or Postgres
This isn't a great test setup.
It's testing RTT rather than the peak throughput of Redis.
I'd suggest using Redis pipelining -- or better: using the excellent rueidis redis client which performs auto-pipelining. Wouldn't be surprising to see a 10x performance boost.
When I last benchmarked Redis vs. PostgreSQL for a simple k/v cache it was about ~1ms for PostgreSQL to fetch a key, and ~0.5ms for Redis with a similar setup as in this post (although I used "value bytea" instead of "value string" – I don't know if it matters, probably not; 1ms was fast enough that I didn't care to test).
I didn't measure setting keys or req/sec because for my use case keys were updated infrequently.
I generally find ms to be a more useful metric than reqs/sec or latency at full load, as this is not a typical load. Or at least wasn't for my use case.
Of course all depends on your use case etc. etc. In some cases throughput does matter. I would encourage everyone to run their own benchmarks suited to their own use case to be sure – should be quick and easy.
As I rule I recommend starting with PostgreSQL and using something else only if you're heavily using the cache or you run in to problems. Redis isn't too hard to run, but still just one less service to worry about. Or alternatively, just use a in-memory DB. Not always appropriate of course, but sometimes it is.
If you use an UNLOGGED table in Postgres as a cache, and your DB restarts, you no longer have a cache. Then your main table gets a huge spike in traffic and likely grinds to a halt.
This seems to be testing how the software is optimized for low core deployments… how does Postgres perform vary as you add more cores and ram? It’s the sort of software where I’d presume more cores and ram yields better performance. Assuming as always that mature systems software sees more many core perf engineering.
Having the ability to set a TTL on the cache key is a critical feature of a cache, not something that can be tacked on later.
I always find these "don't use redis" posts kind of strange. Redis is so simple to operate at any scale, I don't quite get why it is important to remove it.
I hope you remember to add monitoring for your cache expiry cron job so you can be notified if it ever fails to run due to a code change or configuration change. For example, database credentials might be rolled and if you forget to update the job your primary database could fill up.
Perhaps you could have a second cron job that runs to verify that the first one completed. It could look for a last-ran entry. You should put it in the same database, so maybe perhaps you could a key value store like redis for that.
I think there is more performance to be taken from PostgreSQL. Not sure how pgx works internally since I have not used it.
There are async functions provided by PostgreSQL client library (libpq). I've used it to process around 2000 queries on a single connection per second on a logged table.
Why do we promote articles like this that have nice graphs and are well written, when they should get a grade 'F' as an actual benchmark study. The way it is presented, a casual reader would think Postgres is 2/3rds the performance of Redis. Good god. He even admits Postgres maxxed out its 2 cores, but Redis was bottlenecked by the HTTP server. We need more of an academic, not a hacker, culture for benchmarks.
> Both postgres and redis are used with the out of the box settings
Ugh. I know this gives the illusion of fairness, but it's not how any self-respecting software engineer should approach benchmarks. You have hardware. Perhaps you have virtualized hardware. You tune to the hardware. There simply isn't another way, if you want to be taken seriously.
Some will say that in a container-orchestrated environment, tuning goes out the window since "you never know" where the orchestrator will schedule the service but this is bogus. If you've got time to write a basic deployment config for the service on the orchestrator, you've also got time to at least size the memory usage configs for PostgreSQL and/or Redis. It's just that simple.
This is the kind of thing that is "hard and tedious" for only about five minutes of LLM query or web search time and then you don't need to revisit it again (unless you decide to change the orchestrator deployment config to give the service more/less resources). It doesn't invite controversy to right-size your persistence services, especially if you are going to publish the results.
Given there is nothing at all said about the many config options that would contribute to Postgres for this use case, we must assume no configuration has been done.
Also, no discussion of indexes or what the data looks like, so we must assume no attention has been paid to their critical factors either.
So, another case of lies, damned lies and benchmarks.
It seem strange to me people are so willing to post such definitive and poorly researched/argued things - if you're going to take a public position don't you want to be obviously right instead of so easily to discount?
Is this engagement bait? Sigh... fine, at least I bite the bait here:
1. The use-case is super specific to homelab where consistency doesn't matter. You didn't show us the Redis persistence setup. What is the persistence/durability setting? I bet you'd lose data the one day you forgot and flip the breaker of your homelab.
2. What happened when data is bigger than your 8GB of RAM on Redis?
3. You didn't show us the PG config as well, it is possible to just use all of your RAM as buffer and caching.
4. Postgres has a lot of processes and you give it only 2 CPU? Vanilla Redis is single core so this race is rigged to begin with. The UNLOGGED table even things out a bit.
In general, what are you trying to achieve with this "benchmark"? What outcome would you like to learn? Because this "benchmark" will not tell you what you need to know in a production environment.
Side note for other HN readers: The UNLOGGED table is actually very nifty trick for speeding up unit tests. Just perform ALTER to UNLOGGED tables inside the PG that's dedicated for CI/CD: ALTER TABLE my_test_table SET UNLOGGED;
I guess my question is: why bother with a benchmark if the pick is pre-ordained? Is it the case that at some point the results would be so lopsided that you would pick the faster solution? If so, what is that threshold? I.e. when does performance trump system simplicity? To me those are the interesting questions.
I'd be curious to see how Postgres behaves with numeric IDs instead of strings (or another built-in, faster-to-index/hash type).
Still leaves you with needing to perfomantly hash your strings into those IDs on top, but mostly as the Postgres plateau of performance compared to purposely built KV DBs.
I skimmed the article, but why is everyone always going for distributed cache? What is wrong with in-memory cache? Lowest latency, fast, easy to implement.
Yeah ok, you have 30 million entries? Sure.
You need to sync something over multiple nodes? Not sure I would call that a cache.
What is all the fuss about? In the ancient times, you put your entry into the database(ANY database), either with TTL or cache tags, and have a cron job that periodically flushes the table. Then in your application you check if you have the entry cached in memory, if not, you check the db entry, if it is not there you build it from scratch and cache it.
Why do people complicate things? We've solved caching ages ago.
I've got a similar setup with a k3s homelab and a bunch of small projects that need basic data storage and caching. One thing worth considering is that if someone wants to run both redis and postgres, they need to allocate enough memory for both including enough overhead that they don't suddenly OOM.
In that sense, seeing if the latency impact of postgres is tolerable is pretty reasonable. You may be able to get away with postgres putting things on disk (yes, redis can too), and only paying the overhead cost of allocating sufficient excess RAM to one pod rather than two.
But if making tradeoffs like that, for a low-traffic service in a small homelab, I do wonder if you even need a remote cache. It's always worth considering if you can just have the web server keep it's own cache in-memory or even on-disk. If using go like in the article, you'd likely only need a map and a mutex. That'd be an order of magnitude faster, and be even less to manage... Of course it's not persistent, but then neither was Redis (excl. across web server restarts).
52 comments
[ 2.5 ms ] story [ 68.2 ms ] threadI'd suggest using Redis pipelining -- or better: using the excellent rueidis redis client which performs auto-pipelining. Wouldn't be surprising to see a 10x performance boost.
https://github.com/redis/rueidis
I didn't measure setting keys or req/sec because for my use case keys were updated infrequently.
I generally find ms to be a more useful metric than reqs/sec or latency at full load, as this is not a typical load. Or at least wasn't for my use case.
Of course all depends on your use case etc. etc. In some cases throughput does matter. I would encourage everyone to run their own benchmarks suited to their own use case to be sure – should be quick and easy.
As I rule I recommend starting with PostgreSQL and using something else only if you're heavily using the cache or you run in to problems. Redis isn't too hard to run, but still just one less service to worry about. Or alternatively, just use a in-memory DB. Not always appropriate of course, but sometimes it is.
Definitely a premature optimization on my part.
I always find these "don't use redis" posts kind of strange. Redis is so simple to operate at any scale, I don't quite get why it is important to remove it.
Perhaps you could have a second cron job that runs to verify that the first one completed. It could look for a last-ran entry. You should put it in the same database, so maybe perhaps you could a key value store like redis for that.
Also does anyone like memcached anymore? When I compared with Redis in the past it appeared more simple.
There are async functions provided by PostgreSQL client library (libpq). I've used it to process around 2000 queries on a single connection per second on a logged table.
Ugh. I know this gives the illusion of fairness, but it's not how any self-respecting software engineer should approach benchmarks. You have hardware. Perhaps you have virtualized hardware. You tune to the hardware. There simply isn't another way, if you want to be taken seriously.
Some will say that in a container-orchestrated environment, tuning goes out the window since "you never know" where the orchestrator will schedule the service but this is bogus. If you've got time to write a basic deployment config for the service on the orchestrator, you've also got time to at least size the memory usage configs for PostgreSQL and/or Redis. It's just that simple.
This is the kind of thing that is "hard and tedious" for only about five minutes of LLM query or web search time and then you don't need to revisit it again (unless you decide to change the orchestrator deployment config to give the service more/less resources). It doesn't invite controversy to right-size your persistence services, especially if you are going to publish the results.
Also, no discussion of indexes or what the data looks like, so we must assume no attention has been paid to their critical factors either.
So, another case of lies, damned lies and benchmarks.
It seem strange to me people are so willing to post such definitive and poorly researched/argued things - if you're going to take a public position don't you want to be obviously right instead of so easily to discount?
1. The use-case is super specific to homelab where consistency doesn't matter. You didn't show us the Redis persistence setup. What is the persistence/durability setting? I bet you'd lose data the one day you forgot and flip the breaker of your homelab.
2. What happened when data is bigger than your 8GB of RAM on Redis?
3. You didn't show us the PG config as well, it is possible to just use all of your RAM as buffer and caching.
4. Postgres has a lot of processes and you give it only 2 CPU? Vanilla Redis is single core so this race is rigged to begin with. The UNLOGGED table even things out a bit.
In general, what are you trying to achieve with this "benchmark"? What outcome would you like to learn? Because this "benchmark" will not tell you what you need to know in a production environment.
Side note for other HN readers: The UNLOGGED table is actually very nifty trick for speeding up unit tests. Just perform ALTER to UNLOGGED tables inside the PG that's dedicated for CI/CD: ALTER TABLE my_test_table SET UNLOGGED;
"i do not care. i am not adding another dependency"
Still leaves you with needing to perfomantly hash your strings into those IDs on top, but mostly as the Postgres plateau of performance compared to purposely built KV DBs.
Yeah ok, you have 30 million entries? Sure.
You need to sync something over multiple nodes? Not sure I would call that a cache.
Why do people complicate things? We've solved caching ages ago.
In that sense, seeing if the latency impact of postgres is tolerable is pretty reasonable. You may be able to get away with postgres putting things on disk (yes, redis can too), and only paying the overhead cost of allocating sufficient excess RAM to one pod rather than two.
But if making tradeoffs like that, for a low-traffic service in a small homelab, I do wonder if you even need a remote cache. It's always worth considering if you can just have the web server keep it's own cache in-memory or even on-disk. If using go like in the article, you'd likely only need a map and a mutex. That'd be an order of magnitude faster, and be even less to manage... Of course it's not persistent, but then neither was Redis (excl. across web server restarts).