I'll experiment with this a bit to see for myself but I would really appreciate a list of downsides of using this over mongo; if anything else than there may be less people supporting it.
1. Is this a drop in replacement if I'm on Mongo 2.4?
2. If I want to migrate back to Mongodb from TokuMX, what needs to be done?
3. How quickly does TokuMX integrate improvements from MongoDB?
4. Is anyone using this in their production deployment?
3. We support 2.2 but we've back ported some things from 2.4 (like hash-based sharding) pretty easily, I think we'll be able to keep up 4. Nope, it just came out today.
a) There is an astronomical difference between a neat technology and a complete, usable, supported product (especially with really complex software like databases). I can't tell yet how committed Toku folks are to this project. Is this a research project that may or may not go somewhere, or are they all in on the product? I think it's very important (for the customers and the industry) to get a clarification on this point.
b) I love seeing engineering projects like these. Experimentation like this (using a superb storage engine to power a popular db) is really exciting. I'd love to see where this goes.
c) RethinkDB has its own state of the art storage engine (with a very different architecture from Toku) that's tightly integrated into the full system. That lets us do very interesting things (fast path code paths, btree-aware caching system, etc.) The advantages and disadvantages of pluggable storage engines are really interesting.
d) If TokuMX does turn into a complete product, it's really exciting. It's nice to see the industry maturing.
I work for Tokutek. Slava, thanks for the kind words. Just to clear up any possible confusion, this is the product. We release and support TokuMX. The research started roughly a year ago (the blog posts we've done are out there), and it has led to this.
We consider this to be full featured in the sense that we have a feature set that we feel users can deploy in production. As with any product, as users give feedback on what more they would like to see, be it existing MongoDB features or something else, we will use that feedback to enhance the product.
TokuMX is indeed a complete product, not an experiment. We're proud of it and ready to support it yesterday :)
I completely agree that pluggable storage engines are an interesting topic. But we went the integrated route (ie: no storage API) probably for the same reason: things get simpler and easier to implement when the stack is shorter.
Most users should expect a big reduction in disk space. Depending on your data and level of fragmentation with regular MongoDB, you could see ~4x-20x reduced disk usage.
Can I put a TokuMX node into a Mongo 2.4 replica set cluster? It would be nice to let it sync to the cluster and then compare it against live Mongo nodes. Any concerns with sharding?
This really reminds me a lot of the progression of MySQL.
Originally it was used with ISAM/MyISAM and it was pretty popular. Then InnoDB came around and it quickly revolutionized the MySQL world, allowing MySQL to grow to the next level. Now InnoDB is by far the most commonly used storage engine and the default on several distributions.
Staying up to date is not an all-or-none proposition. We use feedback to drive direction. For example, even though we used 2.2 as a base, user feedback compelled us to include hash-based sharding, a 2.4 feature, in this release.
There have been very informal talks with the folks at 10gen. The topic of getting merged or not will ultimately come down to what users want. Commercialization will start with support, much like what 10gen does for regular MongoDB and what we do for TokuDB.
As for licensing, TokuMX embraces the spirit of open-source and we're confident our open-source licence plays well with the AGPL.
I'll ask the same question that I had about RethinkDB last week [1]: how does TokuMX handle secondary indexes on a sharded database?
Since it doesn't support cross-shard atomic operations, I'm assuming it takes the same approach as RethinkDB, i.e. co-locating data with its index entries, which means index lookups require querying every shard in the cluster. But I don't see this explicitly stated in the user's guide, so it would be nice to get some confirmation.
The sharing architecture is the same as mongodb: data is partitioned based according to a shard key and the secondary indexes are built separately on each shard. So queries that use indexes other than the shard key must be broadcast.
For 1.0 we wanted to match mongodb feature-wise as broadly as possible.
However, fractal trees can handle multiple clustered indexes very well, and this extends to a few ideas I have for adding things to do sharding in what I think will be a much better way. It's a bit up in the air right now but I can tell you we plan to make some significant innovations here, and if you have experience you can share, you should email us and we can talk it over and maybe change what we end up doing.
Have you evaluated alternative compressors to QuickLZ? LZ4 and Snappy both offer comparable compression speed and much faster decompression, which would seem to be a big win for read-heavy workloads.
I've been on the fence about trying this on Instrumental; we've been looking at moving to DynamoDB, but I wanted to at least see if it would give drop-in magic performance benefits, so I spun off a new SQS queue of our incoming data this evening and did a write throughput test.
These are only initial impressions, but:
* It's definitely faster. Our write behavior is largely upserts against integers and doubles, and I'm seeing roughly 100% improvement against stock Mongo 2.4. The machine in question is an m2.2xlarge with a 1000 piops EBS volume attached, and it's doing about 7000 update operations a second. ( safe mode )
* I'm seeing consistently lower IO util than stock Mongo. Stock tends to vary wildly between 200-750 write ops, while under sustained write traffic, I see about 250 write ops.
* CPU usage is pretty well balanced against all cores, as opposed to stock's behavior.
* It's too early to say whether or not the storage savings will be as good as claimed, but at this point it seems that the TokuMX reprs are about 40% of the stock reprs. Like I said, most of our data is ints and doubles tho.
VERY LARGE CAVEAT I'm not running the database in a replica set because I'm lazy. So, the write throughput numbers are likely the best case scenario of what you'd actually be running in production.
(edit: line spacing)
If you've got to use MongoDB, it seems pretty nice. If it came with a tiny person that maintained the database for you as well, it'd be a no brainer.
33 comments
[ 4.6 ms ] story [ 36.4 ms ] thread1. Is this a drop in replacement if I'm on Mongo 2.4? 2. If I want to migrate back to Mongodb from TokuMX, what needs to be done? 3. How quickly does TokuMX integrate improvements from MongoDB? 4. Is anyone using this in their production deployment?
I have a few thoughts on this:
a) There is an astronomical difference between a neat technology and a complete, usable, supported product (especially with really complex software like databases). I can't tell yet how committed Toku folks are to this project. Is this a research project that may or may not go somewhere, or are they all in on the product? I think it's very important (for the customers and the industry) to get a clarification on this point.
b) I love seeing engineering projects like these. Experimentation like this (using a superb storage engine to power a popular db) is really exciting. I'd love to see where this goes.
c) RethinkDB has its own state of the art storage engine (with a very different architecture from Toku) that's tightly integrated into the full system. That lets us do very interesting things (fast path code paths, btree-aware caching system, etc.) The advantages and disadvantages of pluggable storage engines are really interesting.
d) If TokuMX does turn into a complete product, it's really exciting. It's nice to see the industry maturing.
We consider this to be full featured in the sense that we have a feature set that we feel users can deploy in production. As with any product, as users give feedback on what more they would like to see, be it existing MongoDB features or something else, we will use that feedback to enhance the product.
I completely agree that pluggable storage engines are an interesting topic. But we went the integrated route (ie: no storage API) probably for the same reason: things get simpler and easier to implement when the stack is shorter.
Originally it was used with ISAM/MyISAM and it was pretty popular. Then InnoDB came around and it quickly revolutionized the MySQL world, allowing MySQL to grow to the next level. Now InnoDB is by far the most commonly used storage engine and the default on several distributions.
How are you (Tokutek) planning to keep up to date with the MongoDB tree?
Are you planning on talking to the MongoDB folks about upstreaming this? Or will it be a pure fork with no sharing either way?
As for licensing, TokuMX embraces the spirit of open-source and we're confident our open-source licence plays well with the AGPL.
Since it doesn't support cross-shard atomic operations, I'm assuming it takes the same approach as RethinkDB, i.e. co-locating data with its index entries, which means index lookups require querying every shard in the cluster. But I don't see this explicitly stated in the user's guide, so it would be nice to get some confirmation.
[1] https://news.ycombinator.com/item?id=5876168
For 1.0 we wanted to match mongodb feature-wise as broadly as possible.
However, fractal trees can handle multiple clustered indexes very well, and this extends to a few ideas I have for adding things to do sharding in what I think will be a much better way. It's a bit up in the air right now but I can tell you we plan to make some significant innovations here, and if you have experience you can share, you should email us and we can talk it over and maybe change what we end up doing.
These are only initial impressions, but:
* It's definitely faster. Our write behavior is largely upserts against integers and doubles, and I'm seeing roughly 100% improvement against stock Mongo 2.4. The machine in question is an m2.2xlarge with a 1000 piops EBS volume attached, and it's doing about 7000 update operations a second. ( safe mode )
* I'm seeing consistently lower IO util than stock Mongo. Stock tends to vary wildly between 200-750 write ops, while under sustained write traffic, I see about 250 write ops.
* CPU usage is pretty well balanced against all cores, as opposed to stock's behavior.
* It's too early to say whether or not the storage savings will be as good as claimed, but at this point it seems that the TokuMX reprs are about 40% of the stock reprs. Like I said, most of our data is ints and doubles tho.
VERY LARGE CAVEAT I'm not running the database in a replica set because I'm lazy. So, the write throughput numbers are likely the best case scenario of what you'd actually be running in production.
(edit: line spacing)
If you've got to use MongoDB, it seems pretty nice. If it came with a tiny person that maintained the database for you as well, it'd be a no brainer.
Have you tried ObjectRocket (if you're in US-East or US-West)? http://objectrocket.com/ High performance Mongo with replica sets.