Ask HN: Questions about MongoDB and Redis for my use case
(Warning this sentence may contain buzzwords that insight rage in some people) I'm building a social curation application. For the purpose of this question, let's just assume it works similar to pinterst.
I plan on using MongoDB as the main datastore of the application, and storing files on S3. I was thinking I'd like to add Redis into the mix so my first thought was to use Redis for caching.
What other features/aspects of (pinterest) would you move away from storing in MongoDB and move to Redis? Would it advantageous to store user follows/followings in Redis over MongoDB?
Would using Redis for more than a cache store be overkill in my application?
4 comments
[ 4.9 ms ] story [ 16.1 ms ] thread> Would using Redis for more than a cache store be overkill in my application?
It depends what scale you are at. I work for a startup that does something similar (not Pinterest) and we make heavy use of Redis for all sorts of things.
But IMHO you are looking at this the wrong way around - instead of "I need a caching solution... I'll try Redis" you seem to be saying "I want to use Redis... what can I use it for". Which is all well and good from a hacker perspective, but looks a bit like premature optimisation to me.
Why not use MongoDB, and move things to Redis if/when they become a bottleneck?
As for "I want to use Redis... what can I use it for" that's exactly what I'm asking. I think this project can gain some traction in it's targeted demographic, but more than anything it's gaining experience with technologies that I've been dying to try, so whether it fails or not I got some useful experience from the project (Trying new technology, especially something like Redis, just run on a local dev machine won't do much for getting the real world experience).
I agree that it may be a bit of premature optimisation, so if you want we can speak in hypotheticals. In an application like (pinterest) what are some use cases for moving parts of the data from MongoDB to Redis? Would it be wise to use something like Redis for follow/following and facebook style updates (Bill, John, Mary, and 4 others loved this photo; Bobby, Rick, and Jane are now following you; etc)?
One area where MongoDB (and almost everything else) suffers is with a large skip/offset value for paging. Getting the first page of results from MongoDB or from Redis will yield similar results. However, the greater the offset the worse the performance. In Redis, whether you get the 1st of the 1000000 page, the performance is constant. So, if you are doing paging with large offsets, at scale, a Redis sorted set is almost a must. It's a pretty specific use case, but I think that's what you asked for :). YMMV on the exact #...you should test it..but if, like 95% of all apps,only the first 5 pages are important, it's really a non-issue.
Redis sets are great for follows..and you can do some nice set operations on them (which you can't in MongoDB)...so something like which of my friends, and which of your friends, are following the same people. However, if you want to go more than 1 level deep (a graph), Redis isn't great (nor is Mongo for that matter, but it's probably better).
Low hanging fruit for Redis would be analytics...both can handle basic analytics, but Redis will handle it faster. For more complex analytics, the two differ substantially...MongoDB will invovled writing Map Reduce...Redis will require that you do the transformation for reports in code. Still, for simple counters...Redis. In fact, maybe this will inspire you: http://blog.getspool.com/2011/11/29/fast-easy-realtime-metri...
These examples are some of the best use cases for Redis. You can really reduce load on your DB by storing 'temporary' data in Redis, then using the DB to aggregate it. So in this example, Redis would keep track on how many 'likes' each photo has, making increments/decrements very quick. Once per day you could aggregate this info to permanent storage - then you have the best of both worlds.
IMO, one of the best uses for Redis is temporary storage of frequently changing, frequently accessed information.
> it's gaining experience with technologies that I've been dying to try, so whether it fails or not I got some useful experience from the project
One of my favourite philosophies, and why I love working in this industry... good luck on your journey!