Probably the roving mobs on Twitter who occasionally brigade project issue trackers and mailing lists. Check out the master/slave post he wrote a while back.
Or just the toxic nature of some open source communities, especially on Reddīt for whatever reason.
The underground in underground programming refers to maintaining low visibility, as to present low target silhouette. Additional benefit is not attracting too many participants that are in it for the clout, rather than for solving problems & good engineering.
Open Source is no longer sufficient for software freedom; the current 'battlefield' is maintaining security from activist pressure or gradual take-over.
This implies that a lot of the politicization of open source projects has to do with outsiders making power plays (for in group status or just because narcissism). I think this is part of it, for sure. I also think it can also manifest when parts of the community get trapped in a purity spiral, which is somewhat related but comes from within rather than without.
Advice? I don't know. Stay underground and ignore the mobs, they will get bored and move on if you ignore them.
Ironically, I think lowering the barriers for involvement in open source has let in a mass of people who have no fucking business being there.
Counting contributors & contributions is about as sensible as counting kLOC and pages of manual.
Conway's law about software architecture implies existence of management-space CAP theorem equivalent.
Communication suffers combinatorial explosion[1], and graph problems are pretty expensive to solve in meatspace.
Attention is an exhaustible resource, and easily DDoSed.
[edit]
dilandau - please note I've carefully avoided certain spicier keywords & expressions in the previous post as to not off put a casual reader and avoid attracting quick downvotes.
The couple spicy keywords in your response end up needlessly detracting a casual reader from the correct point you are making. And I'm not talking about the "fucking" adjective :)
I created the Occupy Wall Street website nine years ago. Believe me when I say the lengths people will go, to try and control community projects, is downright traumatizing. I never could have imagined that same kind of nastiness would impact open source.
If you don't feel comfortable engaging with the new toxic culture, you can use my underground liferaft. My liferaft isn't an operating system, but rather an attempt to help us not depend on them as much. I have no idea who's controlling GNU/Linux these days and Occupy was enough drama for one lifetime. My code has the same focus on clarity that Antirez put into his Redis codebase. My liferaft also empowers you to build and distribute tiny native portable programs (like Kilo!) using hermetically sealed tools. See https://github.com/jart/cosmopolitan
This is not first time Antirez quit. 10 or 8 years ago he deleted his Twitter account with 10k followers. I think today it reached critical mass, and he does not want to be public figure anymore.
Replying to myself to highlight antirez's own comment upthread:
>Yep it's the open source, and in general the "spontaneous" development world, that happens without big money, just for hacking. This "place" once was kinda free and not observed much. Now you can't say anything, if you don't respect a good practice (LOL) people yell at you on Twitter. Even saying that commenting is a good idea is a problem. Not cool.
Underground as in programming for fun or as art, because you enjoy it as a hobby/subculture. Opposed to the "above ground" programming world of programming for other people (possibly also for money, but OSS is often just for recognition) where people have demands about unimportant features, terminology, coding styles, codes of conduct, etc. Sometimes they get quite aggressive about their demands, too.
@antirez - Thank you for Redis! It's been a joy to use across so many projects.
> However I never wanted to be a software maintainer.
And nothing say that you have to be. There's this perverted view that anytime someone creates a popular FOSS project, they need to dedicate every waking minute to maintaining it. That's neither economically feasible nor psychologically reasonable.
> Redis was the most stressful thing I did in my career, and probably also the most important. I don’t like much what the underground programming world became in recent years, but even if it was not an easy journey, I had the privilege to work and interact with many great individuals.
Agree. I’d like to thank conjectures, brodouevencode, Bjartr,
kungtotte, m0xte, fidelramos, chii and pokot0 for their contributions, but gently suggest none of the jokes were really that funny.
Said well: https://news.ycombinator.com/item?id=7609289
Since we have a norm that those who post unfunny jokes shouldn't complain they are downvoted, we should also have a norm that those who don't like jokes should just downvote without comment. Of course, I'm violating the obvious meta-norm...
That's fine, and there are probably some would-be HN comedians who haven't heard the news about this yet, but meta-discussion is really boring. This is the last thing I'll post in this thread, and I won't be surprised if it gets downvoted to oblivion.
Yep it's the open source, and in general the "spontaneous" development world, that happens without big money, just for hacking. This "place" once was kinda free and not observed much. Now you can't say anything, if you don't respect a good practice (LOL) people yell at you on Twitter. Even saying that commenting is a good idea is a problem. Not cool.
> Yep it's the open source, and in general the "spontaneous" development world, that happens without big money, just for hacking. This "place" once was kinda free and not observed much.
Thanks for explaining.
> Now you can't say anything, if you don't respect a good practice (LOL) people yell at you on Twitter. Even saying that commenting is a good idea is a problem. Not cool.
Just because they're loud does not mean they're correct or that they're the majority. I can understand not wanting to deal with any of it though.
The larger than life you become in your space, they more your words, actions, and non-actions get parsed and twisted for political intent. It's either step away entirely or hide behind the mask of a pseudonymous alt.
> Just because they're loud does not mean they're correct or that they're majority.
They're not. They're the mediocre developers with nothing substantive to contribute. Think of your developer heroes. Almost none of them will ever engage in such nonsense. They're too busy building wonderful things to bother with petty nonsense that gets nothing done.
> Even saying that commenting is a good idea is a problem.
I'm glad I'm not alone experiencing this.
One of the core open source projects I use has taken a no comments stance because "the code speaks for itself". It does if you're a core contributor, but if you dive in less frequently it takes a while to grok what's going on and re-build the mental model. Comments would reduce that.
Thanks for all you've done Salvatore, and for your blogging. Your enthusiasm puts the fun back into my programming!
I think some of these agile thought leaders have a bit to answer for here (but of course it's mainly up to the individual to be responsible for their own judgement - just because somebody else makes their money on marketing doesn't mean you are forced to buy the product)
Lowering the barriers to open-source contribution has allowed in people who have absolutely no business being there. If you have standards, you get accused of gate-keeping.
Thanks for mentioning the T- website specifically, it takes a lot of courage to call it out directly these days, because all it takes is one of them with a lot of followers and then you get swamped with hate. It's fucking unreal man.
There's a ton of us down here programming and having fun, free of politics and bullshit. Look forward to seeing you (again).
Dude, the problem is just Twitter. Everybody shouts on Twitter, it's a system that encourages mob reactions. "Back in the day" people had to subscribe to a ML to yell at you, so fewer people did; and we all knew which lists were cesspools of trolls and flamers, making it easier to avoid the bulk of it. Twitter instead gives you the whole firehose, unfiltered and amplified. It's not a problem restricted to opensource or programmers, we see it everywhere at the moment.
Everybody uses shiny FOSS tools, and some even make big $ using them, but nobody wonders where they come from and how much pain went into building them.
I think everybody wants it on their resume, but the practical day to day work tends to be more filtering poorly written GitHub issues than writing any interesting code. For a job that doesn't pay much (often at all), that's not something I would want to spend my time doing.
The problem is that people don't recognize the efforts of the ones who do it. Antirez did it for years. Do people expect someone to have the same very high-profile job with constant overtime and the same project for their entire lives? His efforts to stay on that project were heroic.
There are lots of other people who want to maintain it. Let them, and if they screw it up that is not his fault. He finished his project. Give him a break.
Personally I think it's mainly the result of having people constantly being hypercritical of your code.
And I think he's proven himself to be a superior programmer. In that in almost every category of engineering skill and knowledge, he is better than 90% of the people criticizing him. If he didn't start out that way, over the years his knowledge and skills grew to make it so.
But also maybe it reflects a bit that there is always some new "best practice" or something that some months ago no one heard about, but now many people are unfortunately using the adoption of as a proxy for programming skill because they are unwilling or unable to actually judge something or someone on actual merit.
I think he's done a great job with Redis. It goes to show sometimes that you need to let go of the thing you created for the good of the community. If I were in his position, I would have made the same move
Thanks for Redis! I never built anything that successful but can relate that being a software maintainer is really hard and many great software developers who build great software, have a hard time with it and not finding it very enjoyable.
Hey! I'm not part of how the new setup will be. I don't liked to maintain, and selecting a new development setup is the most mantainer-ing thing ever :D
Guess we'll all sleep with the fishes now. Good that you're going to guide them somewhat, I hope they carry the project forwards; good luck Salvatore! <3 We love you man
Actually the web site uses Redis as the only store. And Redis is using 0.1% of CPU. The problem is that Ruby sucks at doing anything scalable. It's just a Ruby/Sinatra app. If you do that in PHP, it will work out of the box with many concurrent accesses. With Ruby not the case. There are ways to deploy it better, but it should be fast as default, which is not the case.
Those are the ones that got through all the way, or at least the ones that got picked up by the view counter. Not the people that got error or timed-out responses.
My "Show HN" post was in the bottom half of the HN home page for less than an hour. Traffic to the site was roughly 100 times greater than its current levels (by bandwidth):
I just want to say thank you to antirez for Redis, for how simple, fast, and rock solid it remains to this day. Redis is one of those things that allows you to tilt your head a little on a problem--thinking of solutions in terms of set operations and lookups. A lot of people use it understandably for a cache but it is so much more than that...for me it's introduced an entirely new way of composing solutions to problems. Cheers for all you've done--you've shown at least one programmer that there is still a place for small, elegant design that will stand the test of time.
I say this frequently both online and when discussing system design with newer devs, but will repeat here: of all the production issues I've debugged, the culprit has has never been redis. In fact, redis has been a critical piece of achieving cost-effective scaling. It is one of only two pieces of software (along with postgres) that I blindly recommend without any caveats. From following along here and on your blog about how you approach things and think about the software, I think its clear that you and your vision for the project are a large factor of why it has been so reliable.
Redis is simple. Good. Has a nice api. Has good libraries. Single threadsed. Extremely hard to scale. Impossibly difficult to cluster in containers because it uses hard coded ips to address nodes. Performs poorly with large payloads. Doesn't run on windows properly. Is extremely expensive as a hosted service (orders of magnitude in some cases, eg. azure).
You'll love it until you don't.
The scaling and clustering story is not nearly as nice as the quick start.
It's definitely worth recommending... with caveats.
The point being that clustering redis is actually very difficult to do at all, never mind in a way that scales.
This would be a significant down side to using redis at all, except you can get away with not caring if you out source the problem with your credit card.
Redis works well until the point it doesn't - like a lot of other tools.
A half-ton pickup truck works well until you need to haul bigger loads. At some point, you either need to haul smaller loads or change to a different, bigger truck.
Redis is extremely simple to scale. Treat each instance in a cluster as wholly independent from other instances, and build a thin layer on top to implement whatever sharding and availability requirements your use case requires. I've done it many times and it's always been simple and worked brilliantly. Redis's predictability makes it a joy to operate.
We've done just that. Some folks at work wanted sentinel in front as a cluster management layer, but our particular use case did not work well there. Instead, we have some "logical clusters" of 3 nodes that we replicate reads and writes to, sharded to (currently) six clusters. Some logic around quorum for ensuring writes make it to at least 2/3 nodes in a given cluster, with some optimizations for reads sometimes only needing 1 node to respond. We had to do a bunch of tweaking around state and memory management, and all the details were in the docs, which was great. It does exactly what we need it to do, but it is more expensive to run, and we've not figured out a way to do this right in k8s. We don't care for the cost model going forward for when we want this model serving 10x traffic (which is not pie in the sky, it is known actual volume this solution would need to support). For 10x traffic, we'd be looking at 10x node count. At nearly 200 redis nodes, that can get expensive, esp. if we want to move to a managed solution. Anyway, not sure where I was going with this. Yes, redis can scale. Yes it can stay performant. At some point, it just becomes a lot to manage though and costs can add up. We are going to be designing a new solution to keep costs low and performance high.
@sethammons "... sharded to (currently) six clusters ..."
I wonder, when did you start sharding? How many GB memory did your previously single-machine-Redis-instance use, before it was time to shard? How much memory does each "sharded node" have?
(It wasn't the CPU (Redis single threaded) that forced you to shard? But because you needed more memory? (Or sth else?))
> When did you start sharding? How many GB memory did your previously single-machine-Redis-instance use . . .
Sharding is always something you do on day 1 for every project, because a single-machine-Redis-instance is a SPOF and won't pass even the most basic operational readiness checklist.
> (capacity planning questions)
The answers to these questions don't generalize, they depend on your workload. You should figure out some approximation of your data types and request profiles, and use those to get a rough understanding of what one Redis server on a given machine class is capable of delivering. This usually means applying a few different classes of read/write loads against a small cluster at steadily increasing RPS, and documenting how CPU, memory, and latency characteristics change. From these numbers it's possible to derive a capacity plan.
The last time I did this, for my workload and machine class, one Redis instance (one core) delivered 250-500k RPS, invariant to memory used. We'd use the conservative end of that range, combined with the RPS and data set growth rates we predicted, to provision for ~12 months of growth. Operationally, we would deploy (cores-1) or (cores-2) instances per host (I forget exactly) on 32- and 64-core machines. I think they had like 64-128G of RAM, and we made sure to leave enough memory overhead so the AOF or whatever persistence option wouldn't lock up the box. But even the choices of what class of machines to use is a function of your use case, if you have really large dataset with relatively non-costly (CPU) operations, you want a totally different machine profile than a relatively small dataset with complex operations. Availability SLOs also factor in.
All of this is basic operational stuff, and with Redis the answers are pretty well-understood and highly predictable. Completely opposite to systems like Elasticsearch, which was a total nightmare to predict, provision for, and operate.
> "Sharding is always something you do on day 1 for every project ... SPOF ..."
Cannot agree with that. If one does that or not, would depend on things like Service Level Agreements (SLA) — and one can have a pretty high SLA uptime %, without sharding, e.g. if there's an underlying pretty stable hosting provider that live migrates if there's a hardware failure.
Thanks for writing about the last time you used Redis. Interesting to hear that, in that case, the machines had sth like 64 – 128 RAM, and 250k – 500k RPS. Yes I agree that I'd need to benchmark and think about what type of machine(s) to use (some time later — a bit too early for that now). Sounds as if you are / were a pretty large company / project, needing that much memory and machines :- )
> Cannot agree with that. If one does that or not, would depend on things like Service Level Agreements (SLA) — and one can have a pretty high SLA uptime %, without sharding, e.g. if there's an underlying pretty stable hosting provider that live migrates if there's a hardware failure.
Sorry, no. A single instance is never acceptable from any risk perspective, no matter what guarantees the hosting provider claims.
better memory model over Memcache. If one of our memcached servers dies, that data is gone. Redis can write to disk, optionally. We mostly use redis and memcache as a cache to save load from the db. Even with dozens and dozens of dedicated read hosts, we can knock over our dbs if we are not caching data.
I think you are using it wrong. It is not meant to be run in a K8S cluster woth dynamic IPs in multiple instances. You are supposed to deploy VM mesh of fixed IPs as database store. Moreover, it is design-dependent. You can live with just multiple master-slave pairs and have your data implicitly sharded by something, e.g. by user country or continent. It's sad that some people still think "perfect db" exists: partition-tolerant, high-available, durable, horizontal-scalable, low-latency, acid-compiant, open-source database dosn't exist and never will (see the CAP theorem, see the broken guarantees found by Jepsen[1]). Live with it and design your architecture appropriately.
I don’t think most people need a fully partition tolerant solution; even at very high scale and with high SLAs quorum based solutions are used successfully. If you remove strict partition tolerance from your above list there do exist such databases. And I’d argue it’s much easier to develop against such a database and you’re more likely to actually preserve its guarantees in your application than running an ad-hoc cluster of Redis instances with a home-spun clustering scheme where your application logic is heavily implicated in maintaining consistency.
Extremely hard to scale in what sense? A single redis server running on fairly commodity hardware can serve some pretty intense loads (10s of thousands of ops/sec). Once "scale" bigger than that is an issue, companies should already have in place the development staff that will ensure scaling up is done in a sane way, i.e., that have enough sense and/or experience to choose the correct tools for the job.
Redis is one such tool--its clustering "story" may not be ideal but even a small, minimal cluster will be more than sufficient for any use-case that a distributed memory cache is fit for at a scale that will cover 99% of business' needs.
> Extremely hard to scale in what sense? A single redis server running on fairly commodity hardware can serve some pretty intense loads (10s of thousands of ops/sec).
~90k/sec on a single core of L5520. Scaling Redis is a premature optimization for 99.99% of the use cases. While it may be sexy to talk about millions of ops per second that one's project does in reality the number of projects that have that as a requirement is probably barely in triple digits globally.
It's been a word for a long time ... we used that when we were kids (at least in earshot of our parents). Searching for "dang" in Algolia results in (by a quick count) more than 80% of the results being for @dang (the person/handle) rather than the perjorative use.
As a redis user for more than 8 years, i just want to say thank you. I had embedded redis 2.8 on small IoT system with 200mhz/64Mb of ram to huge cluster as a message bus (redis-as-kafka pattern).
Want a shared memory with a type system? Use redis. Want a queue? Use redis. Want something easy to mock, with an API in all major ecosystems? Use redis.
Antirez is an inspiration, for me at least.
Not only because of what he achieved, Redis is a very nice piece of tech, but also because of his programming style.
I can think of few other programmers who have contributed so much, and ever fewer who have signed off in a way that was so filled with dignity. What a guy.
Thank you for such a great piece of software, Antirez. Redis has been an integral piece of my stack for years now.
I, too, am leaving the company I founded soon and have mixed emotions about it. I certainly don’t enjoy the monotonous nature of maintaining software and yearn for those early creative days.
Best of luck to you in the future! Looking forward to seeing what you create next.
Redis has given me more than the occasional headache when it comes to its lax security defaults and how since it's going to be local then why bother changing them? But there's no denying that when it works, it works damnably well. So for that, I give you my thanks and good luck in whatever endeavors you find yourself involved in!
@antirez - thank you so much for your work, and for your patience with dealing with us all over the years. I have benefited greatly from your willingness to share your engineering skills, and you are one of the most respected programmers in my list of people to follow online.
So, onwards and upwards to new things. Perhaps now you have time for my favourite of your projects, LOAD81? :)
307 comments
[ 1.2 ms ] story [ 224 ms ] thread"I don’t like much what the underground programming world became in recent years"
What could he mean? What underground programming world is he talking about?
Or just the toxic nature of some open source communities, especially on Reddīt for whatever reason.
The underground in underground programming refers to maintaining low visibility, as to present low target silhouette. Additional benefit is not attracting too many participants that are in it for the clout, rather than for solving problems & good engineering.
Open Source is no longer sufficient for software freedom; the current 'battlefield' is maintaining security from activist pressure or gradual take-over.
Advice? I don't know. Stay underground and ignore the mobs, they will get bored and move on if you ignore them.
Ironically, I think lowering the barriers for involvement in open source has let in a mass of people who have no fucking business being there.
Conway's law about software architecture implies existence of management-space CAP theorem equivalent.
Communication suffers combinatorial explosion[1], and graph problems are pretty expensive to solve in meatspace.
Attention is an exhaustible resource, and easily DDoSed.
[edit]
dilandau - please note I've carefully avoided certain spicier keywords & expressions in the previous post as to not off put a casual reader and avoid attracting quick downvotes.
The couple spicy keywords in your response end up needlessly detracting a casual reader from the correct point you are making. And I'm not talking about the "fucking" adjective :)
--
[1] l = (n-1)*n/2
If you don't feel comfortable engaging with the new toxic culture, you can use my underground liferaft. My liferaft isn't an operating system, but rather an attempt to help us not depend on them as much. I have no idea who's controlling GNU/Linux these days and Occupy was enough drama for one lifetime. My code has the same focus on clarity that Antirez put into his Redis codebase. My liferaft also empowers you to build and distribute tiny native portable programs (like Kilo!) using hermetically sealed tools. See https://github.com/jart/cosmopolitan
>the lengths people will go, to try and control community projects
I've heard that echoed a couple times in Tim Pool's discussions of the (american) OWS. Scary stuff indeed.
>Yep it's the open source, and in general the "spontaneous" development world, that happens without big money, just for hacking. This "place" once was kinda free and not observed much. Now you can't say anything, if you don't respect a good practice (LOL) people yell at you on Twitter. Even saying that commenting is a good idea is a problem. Not cool.
So yeah, pretty much twitter.
> However I never wanted to be a software maintainer.
And nothing say that you have to be. There's this perverted view that anytime someone creates a popular FOSS project, they need to dedicate every waking minute to maintaining it. That's neither economically feasible nor psychologically reasonable.
> Redis was the most stressful thing I did in my career, and probably also the most important. I don’t like much what the underground programming world became in recent years, but even if it was not an easy journey, I had the privilege to work and interact with many great individuals.
What is "underground programming world"?
If you have to ask... ;)
HN commenters are encouraged to post feedback in the threads about what's good and bad for HN comments.
See: https://news.ycombinator.com/item?id=7605973
it's where you program in a basement.
But I'm unsure.
Thanks for explaining.
> Now you can't say anything, if you don't respect a good practice (LOL) people yell at you on Twitter. Even saying that commenting is a good idea is a problem. Not cool.
Just because they're loud does not mean they're correct or that they're the majority. I can understand not wanting to deal with any of it though.
The larger than life you become in your space, they more your words, actions, and non-actions get parsed and twisted for political intent. It's either step away entirely or hide behind the mask of a pseudonymous alt.
They're not. They're the mediocre developers with nothing substantive to contribute. Think of your developer heroes. Almost none of them will ever engage in such nonsense. They're too busy building wonderful things to bother with petty nonsense that gets nothing done.
I'm glad I'm not alone experiencing this.
One of the core open source projects I use has taken a no comments stance because "the code speaks for itself". It does if you're a core contributor, but if you dive in less frequently it takes a while to grok what's going on and re-build the mental model. Comments would reduce that.
Thanks for all you've done Salvatore, and for your blogging. Your enthusiasm puts the fun back into my programming!
Thanks for mentioning the T- website specifically, it takes a lot of courage to call it out directly these days, because all it takes is one of them with a lot of followers and then you get swamped with hate. It's fucking unreal man.
There's a ton of us down here programming and having fun, free of politics and bullshit. Look forward to seeing you (again).
Dude, the problem is just Twitter. Everybody shouts on Twitter, it's a system that encourages mob reactions. "Back in the day" people had to subscribe to a ML to yell at you, so fewer people did; and we all knew which lists were cesspools of trolls and flamers, making it easier to avoid the bulk of it. Twitter instead gives you the whole firehose, unfiltered and amplified. It's not a problem restricted to opensource or programmers, we see it everywhere at the moment.
The programming world that nobody else sees.
Everybody uses shiny FOSS tools, and some even make big $ using them, but nobody wonders where they come from and how much pain went into building them.
But there is DEFINITELY something wrong with the fact that, seemingly, NOBODY wants to be a software maintainer.
I think everybody wants it on their resume, but the practical day to day work tends to be more filtering poorly written GitHub issues than writing any interesting code. For a job that doesn't pay much (often at all), that's not something I would want to spend my time doing.
There are lots of other people who want to maintain it. Let them, and if they screw it up that is not his fault. He finished his project. Give him a break.
Takes a rare person to say this, made me smile :)
But don't worry Antirez, Redis is far from a bad work of art, quite the contrary.
And I think he's proven himself to be a superior programmer. In that in almost every category of engineering skill and knowledge, he is better than 90% of the people criticizing him. If he didn't start out that way, over the years his knowledge and skills grew to make it so.
But also maybe it reflects a bit that there is always some new "best practice" or something that some months ago no one heard about, but now many people are unfortunately using the adoption of as a proxy for programming skill because they are unwilling or unable to actually judge something or someone on actual merit.
I think he's done a great job with Redis. It goes to show sometimes that you need to let go of the thing you created for the good of the community. If I were in his position, I would have made the same move
On a more selfish note, how is this going to affect Redis development?
a previous post of mine got 500 points and stay #1 for a few hours.
https://news.ycombinator.com/item?id=22410448
traffic from it looks like this
https://tuananh.net/2020/03/02/traffic-from-top-post-on-hack...
First screenshot looks like Google Analytics to me; I'd say it's more likely a lot of us have blocked it and Cloudflare's number is more accurate.
https://www.linkedin.com/posts/rik-roots-50349611_in-case-an...
I say this frequently both online and when discussing system design with newer devs, but will repeat here: of all the production issues I've debugged, the culprit has has never been redis. In fact, redis has been a critical piece of achieving cost-effective scaling. It is one of only two pieces of software (along with postgres) that I blindly recommend without any caveats. From following along here and on your blog about how you approach things and think about the software, I think its clear that you and your vision for the project are a large factor of why it has been so reliable.
Thank you antirez!
You'll love it until you don't.
The scaling and clustering story is not nearly as nice as the quick start.
It's definitely worth recommending... with caveats.
That seems to me an argument that should be pointed at cloud providers rather than Redis itself ;)
This would be a significant down side to using redis at all, except you can get away with not caring if you out source the problem with your credit card.
A half-ton pickup truck works well until you need to haul bigger loads. At some point, you either need to haul smaller loads or change to a different, bigger truck.
@sethammons "... sharded to (currently) six clusters ..."
I wonder, when did you start sharding? How many GB memory did your previously single-machine-Redis-instance use, before it was time to shard? How much memory does each "sharded node" have?
(It wasn't the CPU (Redis single threaded) that forced you to shard? But because you needed more memory? (Or sth else?))
> "nearly 200 redis nodes"
How much memory in each node, if I may ask?
Sharding is always something you do on day 1 for every project, because a single-machine-Redis-instance is a SPOF and won't pass even the most basic operational readiness checklist.
> (capacity planning questions)
The answers to these questions don't generalize, they depend on your workload. You should figure out some approximation of your data types and request profiles, and use those to get a rough understanding of what one Redis server on a given machine class is capable of delivering. This usually means applying a few different classes of read/write loads against a small cluster at steadily increasing RPS, and documenting how CPU, memory, and latency characteristics change. From these numbers it's possible to derive a capacity plan.
The last time I did this, for my workload and machine class, one Redis instance (one core) delivered 250-500k RPS, invariant to memory used. We'd use the conservative end of that range, combined with the RPS and data set growth rates we predicted, to provision for ~12 months of growth. Operationally, we would deploy (cores-1) or (cores-2) instances per host (I forget exactly) on 32- and 64-core machines. I think they had like 64-128G of RAM, and we made sure to leave enough memory overhead so the AOF or whatever persistence option wouldn't lock up the box. But even the choices of what class of machines to use is a function of your use case, if you have really large dataset with relatively non-costly (CPU) operations, you want a totally different machine profile than a relatively small dataset with complex operations. Availability SLOs also factor in.
All of this is basic operational stuff, and with Redis the answers are pretty well-understood and highly predictable. Completely opposite to systems like Elasticsearch, which was a total nightmare to predict, provision for, and operate.
Cannot agree with that. If one does that or not, would depend on things like Service Level Agreements (SLA) — and one can have a pretty high SLA uptime %, without sharding, e.g. if there's an underlying pretty stable hosting provider that live migrates if there's a hardware failure.
Thanks for writing about the last time you used Redis. Interesting to hear that, in that case, the machines had sth like 64 – 128 RAM, and 250k – 500k RPS. Yes I agree that I'd need to benchmark and think about what type of machine(s) to use (some time later — a bit too early for that now). Sounds as if you are / were a pretty large company / project, needing that much memory and machines :- )
Sorry, no. A single instance is never acceptable from any risk perspective, no matter what guarantees the hosting provider claims.
If you have one instance you better be damn sure that downtime doesn’t cause an an outage. IE. no redis means services still run.
The “it’s fine, the SLA covers outages” is just a) laziness and b) negligence.
You don’t do that with web servers, you don’t do it with databases. You don’t do it with your redis instance either.
...well, I suppose some people will do the it anyway; but you get what you get if you do.
If a major outage gets you fired, you have no o w to blame but yourself.
[1] https://jepsen.io/
Redis is one such tool--its clustering "story" may not be ideal but even a small, minimal cluster will be more than sufficient for any use-case that a distributed memory cache is fit for at a scale that will cover 99% of business' needs.
~90k/sec on a single core of L5520. Scaling Redis is a premature optimization for 99.99% of the use cases. While it may be sexy to talk about millions of ops per second that one's project does in reality the number of projects that have that as a requirement is probably barely in triple digits globally.
In any case, I've had the same experience with only two programs - CouchDB and Redis.
https://www.urbandictionary.com/define.php?term=Dang
I love beautiful code too, however if it's not useful, then the beauty fades quickly for me. What's the point?
Thanks for your work :)
I, too, am leaving the company I founded soon and have mixed emotions about it. I certainly don’t enjoy the monotonous nature of maintaining software and yearn for those early creative days.
Best of luck to you in the future! Looking forward to seeing what you create next.
So, onwards and upwards to new things. Perhaps now you have time for my favourite of your projects, LOAD81? :)
Don't need to do much more. Just be You man. Live on. You are beautiful. We thank you for Redis