I like the tone of RethinkDB blog posts and announcements, it's usually more like "hey we just implemented this cool feature, check it out!"
(Not affiliated with them in any way, I'm just testing rethinkdb since I have to use Mongo at work and hate it. I especially like their "stability" page: http://rethinkdb.com/stability/ )
I had a truly magical experience talking to an engineer at Mongo during our sales experience with them. This guy was some sort of super SE. We had a short list of things we simply could not make it do very quickly. He proceeded to tell us how no database could do those things quickly and they were fundamentally broken things to do anyways... (query where property x != y or x doesn't exist was one I can think of) PostgreSQL and Vertica did them all, radically faster than Mongo did.
As much as I agree with your sentiment, it still remains one of the most popular databases out there. All us techies can learn something from the hype they managed to create.
I really believe that the root of MongoDB's success is how easy they made getting started: A trivial installer, no configuration, and a trivial tutorial -- boom, suddenly a million developers were on their side.
It seems ridiculous, really -- developers commit to something because it saves them hours at the start, versus considering the months of work following -- but many technologies followed this curve. MySQL was a lot easier to start with than pgsql, so it won the early battle. PHP was a no-brainer to start with, etc.
The lesson is that your tool should default to essentially no configuration, no security, and simple beginnings, and it tends to pay off.
Just like node.js. Simple to setup. No frameworks to learn. No need to integrate with web server. Few lines of code, run, and you have a web application responding to the requests.
It's kind of funny, but MongoDB and node.js are a very natural pairing... very little disconnect.
Though I really like RethinkDB, the lack of automated failover handling is about the only concern left keeping me from them. Because of the tooling around it, I'm using ElasticSearch om my next project over both of them. I've also used Cassandra in the past, which was great in some cases, harder in others.
For the past couple years, I've tended to pair an SQL database with a key-value/document store depending on the needs. SQL is the canonical source, and saves are mirrored to the distributed database for wider reads. Most no-sql databases sacrifice parts of CAP/ACID in different ways, and can fit certain growth needs.
When you are faced with paying a over $30-50k/month (hardware, licenses, bandwidth, redundancy) just to host your sql database on hardware big enough to handle the load, or distribute... distribution becomes an option. For a company with under a million a year in gross revenue, spending $600/year for the database alone is not an option. Even for larger companies it's probably not the best idea.
There's something to be said for ease of configuration and administration in that mix.
The bigger lesson is that we techies tend to disregard the severity of certain real pains (setup, configuration, up front commitment to schema, etc). And let me tell you, all of us know it, but many of us accept it unquestionably.
I could define market share so SQLite is the most popular database. I could define it so it's MySQL. I could probably even make it Oracle, or DB2. But you do you count to make MongoDB one of the most popular databases out there?
So grandparent claims it's one of the most popular databases out there; your first links claims it's in 4th place of all open-source databases; the second link says it's 4th place overall. Sounds fine to me (of course, who knows how those sites came up with those statistics). Maybe he ninja-edited and originally said the most popular, I guess that'd explain the rage.
And 40% more hip. Gotta love those numbers. Also, don't forget about the new WiredTiger storage engine - sounds way cooler than, say, InnoDB! Hype is strong with this one.
As a funny quirk of history, Michael Cahill, the founder of WiredTiger (and now a MongoDB employee), is the primary author of "Serializable Isolation for Snapshot Databases" [0] the paper that introduced Serializable Snapshot Isolation (SSI). SSI is the concurrency control mechanism used in Postgres [1][2].
...and their most important improvement isn't even their own.
I'm not a fan of bashing any product, but after using MongoDB in several projects, in legitimate document database use cases, I just can't find any bright side.
I agree this announcement is dripping with hype, but all the same I'm excited to get my hands on this. Seems like they're trying to address (some) common pain points.
Transactions are a pain point, but since WiredTiger supports them, I hope that MongoDB will eventually implement them - better sooner than later.
That said, I still fail to see, why I should not use MongoDB. For me as a not to DB savvy developer it was easy to start developing and administrating, and I haven't had any major issues on a small sized production (replicated and sharded) cluster for 3 years now.
It is easy to use. I've not reviewed v3 yet, but based on my experience with v2, my question remains, why would I select MongoDB over a relational database for (as they seem to be suggesting) any and all database applications?
For some, sure, okay, maybe, but nearly everything I've done with databases falls quite nicely into a relational model, which doesn't seem to be MongoDB's strongest point.
Relational databases are indeed excellent, and you are better off using a relational database if a single server can handle the load. But if you have more load than a single server can handle that's when you need things like MongoDB that support automatic sharding.
Realistically most websites could use a combination of caching with a single database server behind them. If the schema isn't too horrible. You can scale up a lot, and depending on your use case and revenue model, that's a valid plan.
I'm pretty sure that Facebook is using a wide database (similar to Cassandra) with reverse indexes generated at the application layer. Yahoo, it depends on the developer group, but something like mongo in use wouldn't completely surprise me. Google has published a lot of white papers on how they use data (see Cassandra and Hadoop from some of those lessons).
MongoDB is itself a bit of a bridge... replication setup is far easier than with a typical RDBMS, and you can break data into documents for simpler queries. Mongo as a persistent caching layer is also an option depending on your load and needs with SQL as authority, mongo for public search/display.
I've worked for a reasonable subset of top Internet sites and have friends who work at the remainder I haven't been at and I can emphatically say that none of them run "a single database server" for any main portion of the site. To think that is simply ludicrous!
I didn't say "top Internet sites" ... I said "most" sites... that being anything not in the top 10000 sites, and probably over half of those within the top 10000 could be the case.
I also said, with a caching system in place. (data and/or output). It really depends on how interactive and static your needs are, which will vary... but most sites are mostly static and very cachable content...
For an example, look at StackOverflow... they were really huge before they needed to start breaking their data up more... Of course their SQL servers are enormous by anyone's standards.. the point is you can scale vertically a lot, and if you can handle a certain amount of downtime, or reduced functionality (if db is down), you can go a long way on a single database server... add caching, add replication... all of this before needing to break data up. (In mostly read scenarios)
Check out the book "High Performance MySQL" at http://shop.oreilly.com/product/0636920022343.do from chapter 9 and beyond it covers replication and high availability. All this for MySQL which is not event the shinning crown of RDBMS.
If you think that automatic sharding is the answer to handle high loads then your are in for a very exciting surprise specially when things go awry and all that missing ACID compliance comes knocking...
Replication works for managing read load, but it doesn't help manage write load. This is when you need automatic sharding.
As far as ACID, many applications don't need it, for example in a game application users will forgive you if you lose their latest scores, but in a banking application users will not forgive you if you lose their last deposit. So it depends on the application.
>Replication works for managing read load, but it doesn't help manage write load. This is when you need automatic sharding.
You've said 'need automatic sharding' multiple times now, but you've still yet to explain why you need it.
Yes, sharding is important for write scaling. But it doesn't "need" to be automatic.
Plenty of setups have completely capable manually sharded workloads running on MySQL. Some would consider the manual approach a plus, vs. relying on the database to automagically do it.
It can be nice to have automatic sharding, assuming it works properly. But nice and needed are two different things, an I think it's an important distinction. You're basically saying if you shard you have to go with mongo and you can't go with anything that requires you manually handle it.
>For me as a not to DB savvy developer it was easy to start developing and administrating
This point is repeatedly made about mongo, php and so on: to me it sounds like "This gun is so easy to use, it allows me to start spraying bullets around with no training and no knowledge of how it works!" or "This car is so simple I can go 150mph on the freeway the first time I got behind the wheel!"
Databases are massively powerful systems and they are used almost exclusively for business applications and matters of import. Why is "I can use it to make long-term consequential actions without any idea wtf I'm doing!" a plus point?
It's just a piece of software, a basic tool - not a war machine that requires 10 years of training, really no need to patronize. I'm really glad not only almighty Oracle admins can handle a database today
Right, because we all know there's just a huge void between "put it in in some kind of form, we'll get it out somehow" an "let's tweak the undocumented redo log retention policy to eke out the necessary performance for our massively joined manifested view that mostly consist of analytical functions".
It's "basic" as in your data being your foundation, not in rudimentary. And to be fair, most of the things that aim to replace SQL tend to be about as complicated. TANSTAAFL.
It allows you to get started amazingly fast and has great client support. You don't have to think much about the DB side of things - just throw data into it.
It's also where people get into trouble. After they've done this and built their MVP they don't go back and rethink their data decisions. Then scale issues occur, among other things.
But it's also very easy to change your schema while iterating to serve your use cases better. But it's up to the users to change things and to learn which types of patterns scale better and to actually learn how to use the database to their advantage after gaining some experience.
Learning how to model documents effectively is rather important.
I am old enough to remember MySQL in the 90s, those guys sneered at anyone who said they needed transactions, just do it in your application. Same with enforcing relational integrity, just do it in your application. We all know how that turned out.
MySQL is one of the top 3 databases, it is used as main database in Facebook, Wikipedia and many other high traffic we sites. There are forks like MariaDB and alternatives with more features like Postgres. MySQL with its several different db engines is unique and with the default InnoDB engine it is on par with its competitors.
Yes, the InnoDB engine adds transactions, foreign keys, and all the other traditional RDBMS features that the original authors of MySQL didn't think anyone needed.
I think it turned out fairly well for them. MySQL is pretty much the go to DBMS for many successful companies, and it and its little sister Maria command something like 75% of the OpenSource Database market.
When Google, Facebook, Twitter, and almost every other big player picks you to underpin their platforms I think you might have made some good choices.
( Yes I know all those companies also have their own DB's, but remember that they almost all started on MySQL and only switched when they out grew it )
It has always had atomic operation per-document. So, if you're able to model transactional things in a single document, you've been able to use those semantics. And as documents can be quite complex there are many use cases where it is possible and even desirable.
However, MongoDB can not guarantee any operation that affects multiple documents will affect them all - if there was a failure after some documents were updated there is no rollback.
However, between atomic single-document updates and the ease of implementing optimistic locking patterns, you still have options.
Multi-document locking and transactions over a cluster of numerous shards is difficult. This is one reason why distributing documents is easier than rows of data. I'm not even sure they will ever have distributed transactions (using a 2-phase commit) because of how slow they are. However, single shard transactions should be easy to implement. But it will be use-case specific I imagine - you'd have to lock by shard-key to guarantee related documents are on the same shard I gather.
We've been using TokuMX for a while and I highly recommend others take a serious look there. It already contains the benefits cited for MongoDB 3 and more. Toku also has a release candidate for the new Mongo storage engine API named TokuMXse, though it's interesting to see that TokuMX proper still outperforms it: http://www.tokutek.com/2015/01/first-tokumxse-performance-nu...
Problem is where it is branched. I don't think they will be able to keep up well. Plus, the new MongoDB storage engine (wire tiger) takes care of its biggest issue which Toku addresses.
Plus, I think you'll be able to just buy Toku's storage engine if you want it when they make it pluggable with the new MonogDB storage engine API.
Somebody at MongoDB must have read the early Oracle story.
Start with one consumer niche (in mongo's case I think it's the common building-cms-based-websites web developer), improve the product, market ruthlessly.
Granted, their product is not excellent and it has major flaws and lacks some common DB features, but they have the advantage of developer-ease-of-use and low barrier to use.
The differences between mongodb.com and mongodb.org are striking.
I was confused about why there was no links to documentation on the .com, until I stumbled onto the .org, which lists actual features right on the home page.
Also striking: no mention of 3.0 on the .org home page at all, and on the downloads page, you have to scroll down to "Development Releases (unstable)" to find any reference to the 3.0 builds.
The .com is the corporation behind MongoDB. .org is the open source project. Yes, they are intertwined heavily at this point (but it wasn't always this case back in the day) but I can understand why they try and keep their separation of concerns here.
I'm excited to see 3.0, but from what I've seen on JIRA, there might still be some issues with WiredTiger (fair considering how they moved from RocksDB to WT in a short period). Will be interesting to see if they'll close out those issues before they drop 3.0 in March.
A few months ago they closed up some of JIRA (comments and a bit epics/sprints), so some issues would be created and have no comments exposed. I thought they were working on transactions in this release (while they were still around 2.7.3 or so), but I guess that's not the case.
Lastly, I wonder how this will impact TokuMX, both positively and negatively? Really whether they'll be making their crown improvements into a pluggable storage type/engine, and if users would be able to take that and plug it into Mongo without patent issues on their fractal tree indexing.
Nonetheless, exciting news for some of us who are building on web-scale, MVCCABCD databases which operate at the speed of hype and web-scale :)
EDIT: saw that Tokutek has created TokuMXse, only after my post
You got your chance, Mongo, and you screwed it up.
I tried getting the most out of you. I treated you like a princess, I indulged you, your wishes became my wishes and your thoughts were my thoughts. I stopped listening to all that criticism around you and thought of you as of a misunderstood child, even as you would refuse to do even the easiest tasks one could imagine.
I gave you everything there is to give, but you broke my heart. You left me in the most critical moments. I trusted you, but you would go your own way. Tears were shed and countless sleepless nights were to follow.
Remember that night you disappeared without leaving a sign? I sent you messages which got a response only after many hours. You didn't give a damn about my needs. One you told me: "it's not me, it's you". And I believed you, I truly did.
It's over now. It's been some time without you, and I'm getting better. I have discovered, that not everyone is like you. Some DBs care, they really do. You can trust them, they give you their everything.
I'm still struggling falling in love again, but it's getting better. Don't write me back. Goodbye.
One of Mongo's fatal problems is that the database has been known to say it has fulfilled its durability promises, but that's only sending a packet to the kernel, not confirmed received by others. Same with writing to disk. So a crash can lose more data than originally thought.
Other things like continually pointing to each other, not figuring out a master for hours has made some people tear their hair out, despite the documentation saying it should behave in a sane way.
MongoDB is a database with some properties that match a specific niche, but to gain popularity they marketed it as a general purpose database that anyone could use.
The result is that many developers who bought into the hype train are now salty because they encountered the limitations at unexpected and unfortunate moments that might have cost them uptime or data.
Or in other words: Developers bought into the hype without checking the facts or reading the manual. Same attitude causes people to change frameworks every 2 days, refactor code bases three time a week and generally be flippant about technology, depending on what is cool this hour.
I agree about MongoDB being for specific niches. I have used it for offline analytics (writes to a master, with slaves on different servers performing different analytics) and also on my laptop to collect training text and for storing intermediate results (small amount of data, the JSON document format was handy for my application). I was very happy with it for those applications.
Thanks for the helpful answer. What was the specific niche it was filling, and why is it not practical for other cases? Or if you don't feel like repeating this info for the zillionth time, is it explained in an essay or blog post somewhere? My googling turned up little.
It is also the case that it was advertised to developers with the promise that you don't need Ops or DBAs or anyone like that. These developers found it did indeed work well for 1 developer on 1 machine working on 1 app with 1 very simple database accessed on 1 connection.
Then they chucked it over the fence to Ops who were expected to make it work in Production, and everything about it was found to be woefully inadequate, and the Ops guys were forced to carry the can. Those guys feel - quite rightly - that they were shafted by MongoDB, both the company and its fans.
MongoDB is like MySQL: it promises you the world, and you believe it can deliver, and it often does, till that day when it committed a transaction which it shouldn't have, tripped over its own clogged up mess and spilled your carefully collected data all over the floor.
There's "one lonely dude complaining on the internet", which you can safely ignore. And there's "half the audience has horror stories to tell", which should probably be a red flag.
- mmap was the extent of their io and memory management
- called rand() in in client to decide if it should report errors
- said a write was committed before even sending it out
- released benchmarks that measured how fast they could put data into local RAM, but compared it to real dB's
- lost data/corrupted on 32 bit builds and dismissed that as being by design
- is generally better off replaced by postgres but hyped by people that don't know any better
That's off the top of my head. I'm sure there are competent users and it makes sense in some cases but the general feeling is that it's over hyped for the little it is. Their early mistakes make people very cautious about trusting them with important data.
And by over hyped, I'm also taking about people that go on about how amazing it is to have a document store, but ignore the obvious idea of using pg and making a table consisting of (key, document) and using XML or JSON operators on top.
To answer this ironically irate question as honestly as I can, here is my perspective. I voted this up.
It speaks to me. Substitute MongoDB for PHP or vb.net or w3schools.com in that post, and it is my frustration. With these tools, I feel bad. I feel sad. I feel they are inferior. But I can never find a real, convincing, proper argument. I have to go through great lengths to stay objective, and it's never quite satisfactory.
Why? Because I'm not good at exploring my thoughts purely objectively. I'm human. When I use a tool, I feel passion. Sometimes positive, sometimes negative. When I use Lisp, it burns like fire. But when I try to explain it, I have to admit: there are only downsides the eye can see.
When I use PHP, vb.net, or MongoDB, I'm on Antartica. Without a jacket. It's cold. It's sad. There are no penguins. I feel bad. But I cannot tell you why. I have to admit: they get the job done. Millions use them. The greatest of the greatest sites and applications are built with them. Software that you use every day. Around the world.
So perhaps it's just me. As painful as it is to admit, I simply cannot find the words to match my feelings about this. And I feel alone.
Then I read this. It warms my heart. He knows what I mean. He speaks my language. Had he had a good, objective argument, he would have used it. But he didn't, because he only has bad feelings. It is precisely the fact that he doesn't have anything objectively negative to say, yet feels so badly about it, that speaks to me so much.
MongoDB might be objectively good, but I find it noteworthy that so many people consider it subjectively bad. This is useful information to me.
I'm sorry my post isn't to your satisfaction. I tried to avoid doing an extensive rant by something a bit more creative. Now I see, that HN can be a tough place to express one's ideas.
Yep, no problems here. 80GB dataset, indexes are around 10GB, one master, two replicas. Lots of reads, all writes are done in bulk during the night (low read throughput). We also use Cassandra and postgresql a lot - there's no silver bullet when it comes to storage systems.
I have been developing with MongoDB for about the past year. (Briefly, I have developed, and continue to support, an ASP.NET MVC C# Web Application that's used by a few thousand people daily. For my personal projects that use MongoDB, I develop with PHP and Python.)
Overall, I enjoy working with MongoDB, because it (generally) maps directly to your object - there is no need for an additional layer (such as an Object Relational Mapper (ORM)).
However, you have to be more careful with your data structure. For example, having sub-arrays in sub-arrays is probably not a good idea.
I will be happy to share more, so feel free to look up my profile.
I ran a website and operations for a company that did over 1mm a year in revenue. Everything was done off of a single machine that was the webserver, database server and media server. Had 10k customers all of whom were active daily. Postgres and Python/Django and I never saw load averages much over 10%. Unless you have a substantial fraction of a million daily users you probably don't need mongo.
This is objectively false. I used MongoDB for >3 years with 100% production uptime and a very large data set.
I get that some people have had bad experiences with Mongo. Sounds like you're one of them. Why not share your experience rather than just spread FUD?
It's worth noting that a lot of the complaints people had about Mongo were with things that were clearly documented (e.g. no confirmations on writes by default back in the day).
Did you use RAID on your HD's? How much RAM did you have? What types of CPU's and how many? Were you updating documents often or writing new ones? Did you have compound indexes built properly? What was your latency requirements? Was MongoDB able to saturate your resources? Did you read off of secondaries at all? Did you shard your DB and if so what was your shard key? Was it random? Could you use it for reads too? How much of your data set did you need to possibly use at any given time?
If you had to "upgrade to SSD's" at 100GB you either had amazingly poor provisions, were doing most everything wrong or both. Messages like yours indicate to me a poor user - not a poor database.
Our config was pretty standard for a few years ago (~2011-2013). RAID10 7200RPM disks, I believe they were Sandybridge, 32gb RAM. We did a decent number of updates/s.
10gen couldn't find fault with our setup other than our need to do updates. Which is weird, updating a database? Inconceivable. Anwyays, spinning disks apparently couldn't deal with the seeking. Our lock percentage was incredibly high, which makes sense given our "high" write load. It was during this evaluation that we found many of their "critical statistics" were non-deterministic (like the padding factor, rerun and you get all values 0.0-2.0. Probably fixed by now).
We had 1 index: primary. We did no other lookups. MongoDB managed to saturate only the IO subsystem. We did read from secondaries, but that didn't help overly much given that the write percent was high on them too.
On their recommendation we upgraded to SSDs. That dropped our lock percentages and it stayed low. We never made it to sharding because we simply stopped caring about MongoDB. I inherited it and was responsible for it, but ultimately decided to get rid of it in favor of Cassandra.
It's not a "poor user" when you follow their best practices of the time and have a poor experience. It (is|was) a poor database. The storage engine was braindead, their acquisition of WiredTiger admits that. It's at least improving, ever so slowly.
Fair enough and you're right, the storage system is awful. Switching to Wiredtiger was smart by them and something like that should have been done far in advance.
The problem with updates in MonogDB is that they are inline and flushed to the filesystem that way. So if you have documents which are growing then the old ones have to be 0'd out and a new one written, which results in a ton of wasted space and an unbelievable amount of thrashing. As you saw, ridiculous amounts of IO time and even with SSD's you get pretty bad SSD wear on your disks.
There's ways around this by pre-padding documents or using buckets if you are pushing into arrays but I can see why you didn't bother. Also, the write lock back then was system level - yikes.
Around 2011 MonogDB was incredibly awful. I will say it has improved manifold since then.
I had ~3TB and it was all on hard disks (although we had quite a bit of RAM). Ultimately I switched to TokuMX which substantially reduced my disk foot print, but I was a very happy MongoDB user.
My use cases may have helped (they were very appropriate for MongoDB, but I suppose I would have chosen something else if they weren't).
Agree with @Xorlev, what's "very large" to you? If it's anything less than 1TB, then your definition of "very large" has scale issues (which is fitting if you are a MongoDB advocate I guess).
Further, what's your read / write pattern? Do you do replication? Do you have requirements around writes being visible to reads on slaves near immediately? What is your data integrity requirements? All of these play into whether MongoDB is the right fit for someone's needs.
The point remains, MongoDB is good for some use cases, but if you buy into the marketing hype of "the new DBMS standard for any team, in any industry", then you deserve the inevitable pain you will endure (that said, I do feel bad for your likely eventual replacements who have to clean up the mess from your poor choices).
I had ~3TB before I migrated to TokuMX (same semantics, but at the time much faster indices and lower disk space requirements). It was replicated twice, although I could let reads on slaves fall behind a bit.
Of course it's not good for everyone's use cases though. Nothing is. Did anyone honestly believe that? I mean, even with the hype, did anyone truly decide to not evaluate their _database_, and believe the ad copy blindly?
Ultimately, I wouldn't have chosen MongoDB if it was bad for my use case. I tried it and other things -- read the documentation, made benchmarks and test cases -- and chose it after a period of evaluation. To do anything less for your database seems irresponsible (I'm looking at you, people who were surprised about the original default write semantics because you didn't read the docs).
Tangentially, the personal attacks ("which is fitting if you are a MongoDB advocate I guess", "that said, I do feel bad for your likely eventual replacements who have to clean up the mess from your poor choices") add nothing to your argument. They make you look foolish. Also, my replacements are still using my system (quite happily), over a year after my departure.
Neither of those comments were directed at you personally, but at a generic person who would fit the criteria immediately preceding them. "those who think <1TB is a 'very large dataset'" == not understanding modern scale. "if you buy into marketing hype" == you are making poor choices. Neither of those selection criteria apply to you, so I apologize that you took them as personal attacks out of context.
Same here, we've been using MongoDB in production since 2011 (v1.8) and our experience has been outstanding. Looking forward to the improvements in 3.0.
I'm not really happy with it, but it works well for me in two projects.
In first one, Mongo acts as a huge distributed (multisharded), replicated file storage — we're using GridFS to store tens of millions files with tons of searchable metadata.
In the second one it is just a backend for geo tracking software.
"MongoDB 3.0 will be generally available in March, when we finish putting it through its paces. Stay tuned for our latest release candidate, we would love it if you would try it out and give us feedback."
So... no 3.0 final yet. That's disappointing. However - I'm really excited about 3.0. Initial tests show way better performance than 2.6. Plus data takes just 1/4th of disk space. That's amazing. I hope this is just beginning and each next release will add more and more features based on what WT can deliver.
OK, I'll provide a user's perspective as we are invested in Mongo as a storage system for statistical data and have been using it for a good few years now. MongoDb has been, so far, very good to us. I did not experience any of the problems people sometimes cry so very vocally about.
My main concern about Mongo is actually one that is addressed by this release so I'm quite excited to try it out - data compression. Mongo is generally quite irresponsible about disk space usage. Yes disks are cheap but I rather not have a 2TB dataset if I can have half the size, thank you very much.
As a side note, the hype from MongoDb the company equals the anti-hype you get on places like HN, so I guess they even out.
You may also really like TokuMX. It has the same API as Mongo (even works with Mongo drivers) but way lower disk use, faster indices, and better query semantics (e.g. updates don't affect the cursor you're iterating right now).
After using MongoDB for years (very happily too, I'm not one of the anti-Mongo crowd), I ended up switching and it's been great.
I'll add to what @harel says below. If it wasn't for MongoDB, my last company would have probably not survived. Sure, our usecase was very specific (very write-heavy app) but MongoDB delivered and still delivers without any problems. I think there can be usecases where MongoDB is not the best fit, but for wide number of applications, MongoDB is the perfect fit.
I'm actually surprised that MongoDB saved your write-heavy app. In my experience, it's Mongo's poor write throughput that causes the most headaches. This 3.0 release finally begins to address that with collection and document level concurrency control (locking).
Can some people who have used Mongo in production elaborate on their experiences where Mongo worked well or didn't work well? I see a lot of more hate than I do love for Mongo on HN and I'm curious why.
I've used Mongo in two large projects and have mixed feelings about it.
Pros: It's great for building your MVP. Super easy to get up and running, super easy to work with at a superficial level (I'm talking about storing data and querying here more than ops and administration), dataset can grow to a fairly large size without you having to think about your database at all, freeing you to think about your business. Mongo was my first brush with "schemaless" and that's a big advantage particularly for early-stage projects where things are in flux. I also enjoy using JS as the query language, because I like JS, but YMMV.
Also for certain use cases, particularly the single document case, it's probably one of the best solutions available. However, in my experience, this ends up being a limitation if your data doesn't fit nicely into a single document, and the types of data that do fit this use case are rare.
The cons you've probably heard before: no transactions, no joins, not ACID compliant, until very recently writes locked the entire database, scaling/sharding just works until it doesn't.
I find Mongodb works well (OLTP) when you use it as a powerful Key-Value store with secondary indices.
You'll get easy replication/failover (rolling updates of the DB server are easy), ease of use (with node.js for example) and very decent performance...
Good for an application where you need frequent access to a "row" except that row is complex, with subobjects, sets and arrays (in this use case, a relational model performs much worse because of joins)
Not so good for an application where you need frequent access to lists of tabular data and list/details access (where relational models shine)
Your mileage may vary. Some use it for logging, for analytics, for which I have no need or experience.
GridFS also works very well (but I use nginx as a cache in front of it).
MongoDB seems to be a love/hate relation. I haven't used it myself, but I found this article[1] very useful. My conclusion is that MongoDB works, but you need to understand it and take some time to plan the deployment.
I feel like the vast majority of people that bash on MongoDB didn't fully understand it and what it was good for when they tried it. We've been using it in production for nearly 3 years in online games and have had nothing but good experiences with it. This 3.0 release will be yet another big step forward and we are excited to reap the continued benefits.
That's mainly correct. Or they used for it something it simply isn't suitable for.
You can't just throw random data into it and expect it to work. Proper document modeling and understanding write concern solve almost all the problems people have with MongoDB.
It still has a ways to go but so many of the comments here are issues from way back in v2.0.
Regardless, it's still very popular and only gaining popularity. And improving.
Agreed. I've also been using it in production for years and the hate is often misguided. My issues are almost never with lack of features, but almost always with the cavalier attitude towards bugs, even catastrophic ones.
> didn't fully understand it and what it was good for
This is why I'm not a fan, its "sweet spot" , in my experience, is not very big and not very interesting. It requires a small data set where at least the indexes (but ideally the entire "working set") fits in memory on a single server, in which case it can get pretty good read throughput. But there are a lot of options that perform great in this use case, including just mysql or postgres. Maybe the flexibility of being schema-less is a win for some people, in which case great, I'm glad it's available as an option.
But when the company claims things like it's the best database for every use case, people are bound to use it wrong, get burned, and then hate it. Where I used it it had been the exact wrong choice to make, a data set bigger than ~2x a single server's memory (and growing) and with relatively high write throughput, and it was an absolute nightmare.
That's the point so many here fail to address. Lots of businesses are using MongoDB at large scale (2PB clusters) and willing to talk about it too.
MongoDB gets a bad rep, some of it their own fault for sure, but it's not as lousy as some people here make it out to be. Careful object modeling goes a long ways. Wired Tiger will be a massive improvement on their current storage engine, which is pretty awful for write-heavy loads to be sure.
You've always been able to control this with write concern on a per-write basis. If you can handle an instance where and acknowledged write may not get replicated and possibly lost (logging use case - fast writes) then use that level of write concern. If you require extreme durability (slower writes but more durable) then use that level. In many applications you have both requirements and MongoDB lets you choose at write time the level of durability you want.
Does version 3 fix the issue where the database lies to you and says the write was complete once it hits the network socket and doesn't wait for the ack?
He was using v2.0 in that article. A version that is years old and pretty awful. The default then was that a write was fired from the client and that's it - fire and forget. A stupid default.
The default since is to get an ack from the server. You can control this on each write and make it more durable (make sure it is replicated to 1, majority or all replicas) or faster (just get to the server). The default is to be written to the primary server and put into the transaction log so if it doesn't get committed to disk (possibly 60 seconds by default) it can recover. The transaction log is flushed (fsync) every 100ms by default and is configurable. You can also specify that the write is only acknowledged after the transaction log is synced. Anyways the default is it is put into the transaction log and then acknowledged.
I'd make sure you use the new storage engine, wiredtiger too if you give them another shot. The standard one works well on read heavy use cases but saturates I/O pretty quickly if you're doing a lot of updates that make documents grow if you don't take some precautions.
I also agree, having a setting of "fire and forget", although useful in some (limited!) use cases, is not a sane default.
mongo scales just fine if you know how to use it. I maintained over a 40 node cluster with 1 billion profiles in it. I believe it scales just fine. Issues is with the developers that don't understand how to use it.
I'm actually fairly interested in the WiredTiger integration. I switched from MongoDB to TokuMX about a year ago because of disk space and atomicity concerns, and Toku's been really good to me. Mongo 3.0 promises to catch up in many respects; if it does, then it might actually solve the vast majority of the complaints that people have historically had with it.
The marketing copy still pretends that TokuMX doesn't exist, though - it's had these features and more (including transactions) for quite some time now.
Slightly off topic: I prepared this MongoDB Quick Reference Card last year; never put it online before. Hope it'll be useful to someone looking for a quick peek into how mongo db commands work - https://dl.dropboxusercontent.com/u/58502821/mongodb_qrc.pdf
150 comments
[ 3.8 ms ] story [ 212 ms ] threadIt's good to see they haven't sacrificed the hype while working on other features :-/
(Not affiliated with them in any way, I'm just testing rethinkdb since I have to use Mongo at work and hate it. I especially like their "stability" page: http://rethinkdb.com/stability/ )
It seems ridiculous, really -- developers commit to something because it saves them hours at the start, versus considering the months of work following -- but many technologies followed this curve. MySQL was a lot easier to start with than pgsql, so it won the early battle. PHP was a no-brainer to start with, etc.
The lesson is that your tool should default to essentially no configuration, no security, and simple beginnings, and it tends to pay off.
Though I really like RethinkDB, the lack of automated failover handling is about the only concern left keeping me from them. Because of the tooling around it, I'm using ElasticSearch om my next project over both of them. I've also used Cassandra in the past, which was great in some cases, harder in others.
For the past couple years, I've tended to pair an SQL database with a key-value/document store depending on the needs. SQL is the canonical source, and saves are mirrored to the distributed database for wider reads. Most no-sql databases sacrifice parts of CAP/ACID in different ways, and can fit certain growth needs.
When you are faced with paying a over $30-50k/month (hardware, licenses, bandwidth, redundancy) just to host your sql database on hardware big enough to handle the load, or distribute... distribution becomes an option. For a company with under a million a year in gross revenue, spending $600/year for the database alone is not an option. Even for larger companies it's probably not the best idea.
There's something to be said for ease of configuration and administration in that mix.
That's a lot. Could you elaborate what kind of hardware and software one gets at that price level?
Here's a couple disproving your claim:
https://www.scalebase.com/the-state-of-the-open-source-datab...
http://db-engines.com/en/ranking
And 40% more hip. Gotta love those numbers. Also, don't forget about the new WiredTiger storage engine - sounds way cooler than, say, InnoDB! Hype is strong with this one.
[0] - https://courses.cs.washington.edu/courses/cse444/08au/544M/R...
[1] - https://wiki.postgresql.org/wiki/SSI
[2] - http://drkp.net/papers/ssi-vldb12.pdf
I'm not a fan of bashing any product, but after using MongoDB in several projects, in legitimate document database use cases, I just can't find any bright side.
So much claim without any real information to back it up.
So...Mongo "can be the new DBMS standard for any team, in any industry", and they don't even support transactions yet?
http://source.wiredtiger.com/2.4.1/transactions.html
For some, sure, okay, maybe, but nearly everything I've done with databases falls quite nicely into a relational model, which doesn't seem to be MongoDB's strongest point.
I'm pretty sure that Facebook is using a wide database (similar to Cassandra) with reverse indexes generated at the application layer. Yahoo, it depends on the developer group, but something like mongo in use wouldn't completely surprise me. Google has published a lot of white papers on how they use data (see Cassandra and Hadoop from some of those lessons).
MongoDB is itself a bit of a bridge... replication setup is far easier than with a typical RDBMS, and you can break data into documents for simpler queries. Mongo as a persistent caching layer is also an option depending on your load and needs with SQL as authority, mongo for public search/display.
I also said, with a caching system in place. (data and/or output). It really depends on how interactive and static your needs are, which will vary... but most sites are mostly static and very cachable content...
For an example, look at StackOverflow... they were really huge before they needed to start breaking their data up more... Of course their SQL servers are enormous by anyone's standards.. the point is you can scale vertically a lot, and if you can handle a certain amount of downtime, or reduced functionality (if db is down), you can go a long way on a single database server... add caching, add replication... all of this before needing to break data up. (In mostly read scenarios)
If you think that automatic sharding is the answer to handle high loads then your are in for a very exciting surprise specially when things go awry and all that missing ACID compliance comes knocking...
As far as ACID, many applications don't need it, for example in a game application users will forgive you if you lose their latest scores, but in a banking application users will not forgive you if you lose their last deposit. So it depends on the application.
You've said 'need automatic sharding' multiple times now, but you've still yet to explain why you need it.
Yes, sharding is important for write scaling. But it doesn't "need" to be automatic.
Plenty of setups have completely capable manually sharded workloads running on MySQL. Some would consider the manual approach a plus, vs. relying on the database to automagically do it.
It can be nice to have automatic sharding, assuming it works properly. But nice and needed are two different things, an I think it's an important distinction. You're basically saying if you shard you have to go with mongo and you can't go with anything that requires you manually handle it.
This point is repeatedly made about mongo, php and so on: to me it sounds like "This gun is so easy to use, it allows me to start spraying bullets around with no training and no knowledge of how it works!" or "This car is so simple I can go 150mph on the freeway the first time I got behind the wheel!"
Databases are massively powerful systems and they are used almost exclusively for business applications and matters of import. Why is "I can use it to make long-term consequential actions without any idea wtf I'm doing!" a plus point?
It's "basic" as in your data being your foundation, not in rudimentary. And to be fair, most of the things that aim to replace SQL tend to be about as complicated. TANSTAAFL.
It's also where people get into trouble. After they've done this and built their MVP they don't go back and rethink their data decisions. Then scale issues occur, among other things.
But it's also very easy to change your schema while iterating to serve your use cases better. But it's up to the users to change things and to learn which types of patterns scale better and to actually learn how to use the database to their advantage after gaining some experience.
Learning how to model documents effectively is rather important.
InnoDB by the way is an Oracle product.
When Google, Facebook, Twitter, and almost every other big player picks you to underpin their platforms I think you might have made some good choices.
( Yes I know all those companies also have their own DB's, but remember that they almost all started on MySQL and only switched when they out grew it )
However, MongoDB can not guarantee any operation that affects multiple documents will affect them all - if there was a failure after some documents were updated there is no rollback.
However, between atomic single-document updates and the ease of implementing optimistic locking patterns, you still have options.
Multi-document locking and transactions over a cluster of numerous shards is difficult. This is one reason why distributing documents is easier than rows of data. I'm not even sure they will ever have distributed transactions (using a 2-phase commit) because of how slow they are. However, single shard transactions should be easy to implement. But it will be use-case specific I imagine - you'd have to lock by shard-key to guarantee related documents are on the same shard I gather.
Plus, I think you'll be able to just buy Toku's storage engine if you want it when they make it pluggable with the new MonogDB storage engine API.
Granted, their product is not excellent and it has major flaws and lacks some common DB features, but they have the advantage of developer-ease-of-use and low barrier to use.
I think this product is simply not yet "DONE".
I was confused about why there was no links to documentation on the .com, until I stumbled onto the .org, which lists actual features right on the home page.
Also striking: no mention of 3.0 on the .org home page at all, and on the downloads page, you have to scroll down to "Development Releases (unstable)" to find any reference to the 3.0 builds.
https://www.mongodb.org/downloads
Quite the split personality between the two sites.
A few months ago they closed up some of JIRA (comments and a bit epics/sprints), so some issues would be created and have no comments exposed. I thought they were working on transactions in this release (while they were still around 2.7.3 or so), but I guess that's not the case.
Lastly, I wonder how this will impact TokuMX, both positively and negatively? Really whether they'll be making their crown improvements into a pluggable storage type/engine, and if users would be able to take that and plug it into Mongo without patent issues on their fractal tree indexing.
Nonetheless, exciting news for some of us who are building on web-scale, MVCCABCD databases which operate at the speed of hype and web-scale :)
EDIT: saw that Tokutek has created TokuMXse, only after my post
I tried getting the most out of you. I treated you like a princess, I indulged you, your wishes became my wishes and your thoughts were my thoughts. I stopped listening to all that criticism around you and thought of you as of a misunderstood child, even as you would refuse to do even the easiest tasks one could imagine.
I gave you everything there is to give, but you broke my heart. You left me in the most critical moments. I trusted you, but you would go your own way. Tears were shed and countless sleepless nights were to follow.
Remember that night you disappeared without leaving a sign? I sent you messages which got a response only after many hours. You didn't give a damn about my needs. One you told me: "it's not me, it's you". And I believed you, I truly did.
It's over now. It's been some time without you, and I'm getting better. I have discovered, that not everyone is like you. Some DBs care, they really do. You can trust them, they give you their everything.
I'm still struggling falling in love again, but it's getting better. Don't write me back. Goodbye.
Other things like continually pointing to each other, not figuring out a master for hours has made some people tear their hair out, despite the documentation saying it should behave in a sane way.
The result is that many developers who bought into the hype train are now salty because they encountered the limitations at unexpected and unfortunate moments that might have cost them uptime or data.
For most applications I usually use Postgres.
Then they chucked it over the fence to Ops who were expected to make it work in Production, and everything about it was found to be woefully inadequate, and the Ops guys were forced to carry the can. Those guys feel - quite rightly - that they were shafted by MongoDB, both the company and its fans.
Without telling you.
There's "one lonely dude complaining on the internet", which you can safely ignore. And there's "half the audience has horror stories to tell", which should probably be a red flag.
- had a global write lock.
- mmap was the extent of their io and memory management
- called rand() in in client to decide if it should report errors
- said a write was committed before even sending it out
- released benchmarks that measured how fast they could put data into local RAM, but compared it to real dB's
- lost data/corrupted on 32 bit builds and dismissed that as being by design
- is generally better off replaced by postgres but hyped by people that don't know any better
That's off the top of my head. I'm sure there are competent users and it makes sense in some cases but the general feeling is that it's over hyped for the little it is. Their early mistakes make people very cautious about trusting them with important data.
And by over hyped, I'm also taking about people that go on about how amazing it is to have a document store, but ignore the obvious idea of using pg and making a table consisting of (key, document) and using XML or JSON operators on top.
I might be totally wrong.
Edit: I guess for the same reasons I am being down voted without comment.
It's sympathy karma because the poor guy got dumped. We've all been there.
looks away in the the distance CVS. I had such high hopes. You're dead to me now. sigh
I wouldn't have minded it as much if it wasn't so long and including some detailed issues OP faced with Mongo.
It speaks to me. Substitute MongoDB for PHP or vb.net or w3schools.com in that post, and it is my frustration. With these tools, I feel bad. I feel sad. I feel they are inferior. But I can never find a real, convincing, proper argument. I have to go through great lengths to stay objective, and it's never quite satisfactory.
Why? Because I'm not good at exploring my thoughts purely objectively. I'm human. When I use a tool, I feel passion. Sometimes positive, sometimes negative. When I use Lisp, it burns like fire. But when I try to explain it, I have to admit: there are only downsides the eye can see.
When I use PHP, vb.net, or MongoDB, I'm on Antartica. Without a jacket. It's cold. It's sad. There are no penguins. I feel bad. But I cannot tell you why. I have to admit: they get the job done. Millions use them. The greatest of the greatest sites and applications are built with them. Software that you use every day. Around the world.
So perhaps it's just me. As painful as it is to admit, I simply cannot find the words to match my feelings about this. And I feel alone.
Then I read this. It warms my heart. He knows what I mean. He speaks my language. Had he had a good, objective argument, he would have used it. But he didn't, because he only has bad feelings. It is precisely the fact that he doesn't have anything objectively negative to say, yet feels so badly about it, that speaks to me so much.
MongoDB might be objectively good, but I find it noteworthy that so many people consider it subjectively bad. This is useful information to me.
That's why I voted it up.
Overall, I enjoy working with MongoDB, because it (generally) maps directly to your object - there is no need for an additional layer (such as an Object Relational Mapper (ORM)).
However, you have to be more careful with your data structure. For example, having sub-arrays in sub-arrays is probably not a good idea.
I will be happy to share more, so feel free to look up my profile.
I get that some people have had bad experiences with Mongo. Sounds like you're one of them. Why not share your experience rather than just spread FUD?
It's worth noting that a lot of the complaints people had about Mongo were with things that were clearly documented (e.g. no confirmations on writes by default back in the day).
We started having issues around 100GB that forced an upgrade to SSDs. That lasted us til 2TB, at which point we switched.
Did you use RAID on your HD's? How much RAM did you have? What types of CPU's and how many? Were you updating documents often or writing new ones? Did you have compound indexes built properly? What was your latency requirements? Was MongoDB able to saturate your resources? Did you read off of secondaries at all? Did you shard your DB and if so what was your shard key? Was it random? Could you use it for reads too? How much of your data set did you need to possibly use at any given time?
If you had to "upgrade to SSD's" at 100GB you either had amazingly poor provisions, were doing most everything wrong or both. Messages like yours indicate to me a poor user - not a poor database.
10gen couldn't find fault with our setup other than our need to do updates. Which is weird, updating a database? Inconceivable. Anwyays, spinning disks apparently couldn't deal with the seeking. Our lock percentage was incredibly high, which makes sense given our "high" write load. It was during this evaluation that we found many of their "critical statistics" were non-deterministic (like the padding factor, rerun and you get all values 0.0-2.0. Probably fixed by now).
We had 1 index: primary. We did no other lookups. MongoDB managed to saturate only the IO subsystem. We did read from secondaries, but that didn't help overly much given that the write percent was high on them too.
On their recommendation we upgraded to SSDs. That dropped our lock percentages and it stayed low. We never made it to sharding because we simply stopped caring about MongoDB. I inherited it and was responsible for it, but ultimately decided to get rid of it in favor of Cassandra.
It's not a "poor user" when you follow their best practices of the time and have a poor experience. It (is|was) a poor database. The storage engine was braindead, their acquisition of WiredTiger admits that. It's at least improving, ever so slowly.
The problem with updates in MonogDB is that they are inline and flushed to the filesystem that way. So if you have documents which are growing then the old ones have to be 0'd out and a new one written, which results in a ton of wasted space and an unbelievable amount of thrashing. As you saw, ridiculous amounts of IO time and even with SSD's you get pretty bad SSD wear on your disks.
There's ways around this by pre-padding documents or using buckets if you are pushing into arrays but I can see why you didn't bother. Also, the write lock back then was system level - yikes.
Around 2011 MonogDB was incredibly awful. I will say it has improved manifold since then.
My use cases may have helped (they were very appropriate for MongoDB, but I suppose I would have chosen something else if they weren't).
Further, what's your read / write pattern? Do you do replication? Do you have requirements around writes being visible to reads on slaves near immediately? What is your data integrity requirements? All of these play into whether MongoDB is the right fit for someone's needs.
The point remains, MongoDB is good for some use cases, but if you buy into the marketing hype of "the new DBMS standard for any team, in any industry", then you deserve the inevitable pain you will endure (that said, I do feel bad for your likely eventual replacements who have to clean up the mess from your poor choices).
Of course it's not good for everyone's use cases though. Nothing is. Did anyone honestly believe that? I mean, even with the hype, did anyone truly decide to not evaluate their _database_, and believe the ad copy blindly?
Ultimately, I wouldn't have chosen MongoDB if it was bad for my use case. I tried it and other things -- read the documentation, made benchmarks and test cases -- and chose it after a period of evaluation. To do anything less for your database seems irresponsible (I'm looking at you, people who were surprised about the original default write semantics because you didn't read the docs).
Tangentially, the personal attacks ("which is fitting if you are a MongoDB advocate I guess", "that said, I do feel bad for your likely eventual replacements who have to clean up the mess from your poor choices") add nothing to your argument. They make you look foolish. Also, my replacements are still using my system (quite happily), over a year after my departure.
As a side note, the hype from MongoDb the company equals the anti-hype you get on places like HN, so I guess they even out.
After using MongoDB for years (very happily too, I'm not one of the anti-Mongo crowd), I ended up switching and it's been great.
Pros: It's great for building your MVP. Super easy to get up and running, super easy to work with at a superficial level (I'm talking about storing data and querying here more than ops and administration), dataset can grow to a fairly large size without you having to think about your database at all, freeing you to think about your business. Mongo was my first brush with "schemaless" and that's a big advantage particularly for early-stage projects where things are in flux. I also enjoy using JS as the query language, because I like JS, but YMMV.
Also for certain use cases, particularly the single document case, it's probably one of the best solutions available. However, in my experience, this ends up being a limitation if your data doesn't fit nicely into a single document, and the types of data that do fit this use case are rare.
The cons you've probably heard before: no transactions, no joins, not ACID compliant, until very recently writes locked the entire database, scaling/sharding just works until it doesn't.
You'll get easy replication/failover (rolling updates of the DB server are easy), ease of use (with node.js for example) and very decent performance...
Good for an application where you need frequent access to a "row" except that row is complex, with subobjects, sets and arrays (in this use case, a relational model performs much worse because of joins)
Not so good for an application where you need frequent access to lists of tabular data and list/details access (where relational models shine)
Your mileage may vary. Some use it for logging, for analytics, for which I have no need or experience. GridFS also works very well (but I use nginx as a cache in front of it).
[1] https://blog.serverdensity.com/does-everyone-hate-mongodb/
You can't just throw random data into it and expect it to work. Proper document modeling and understanding write concern solve almost all the problems people have with MongoDB.
It still has a ways to go but so many of the comments here are issues from way back in v2.0.
Regardless, it's still very popular and only gaining popularity. And improving.
This is why I'm not a fan, its "sweet spot" , in my experience, is not very big and not very interesting. It requires a small data set where at least the indexes (but ideally the entire "working set") fits in memory on a single server, in which case it can get pretty good read throughput. But there are a lot of options that perform great in this use case, including just mysql or postgres. Maybe the flexibility of being schema-less is a win for some people, in which case great, I'm glad it's available as an option.
But when the company claims things like it's the best database for every use case, people are bound to use it wrong, get burned, and then hate it. Where I used it it had been the exact wrong choice to make, a data set bigger than ~2x a single server's memory (and growing) and with relatively high write throughput, and it was an absolute nightmare.
In reality, MongoDB works well enough to be where they are, armed with the money they have.
MongoDB gets a bad rep, some of it their own fault for sure, but it's not as lousy as some people here make it out to be. Careful object modeling goes a long ways. Wired Tiger will be a massive improvement on their current storage engine, which is pretty awful for write-heavy loads to be sure.
http://hackingdistributed.com/2013/01/29/mongo-ft/
The default since is to get an ack from the server. You can control this on each write and make it more durable (make sure it is replicated to 1, majority or all replicas) or faster (just get to the server). The default is to be written to the primary server and put into the transaction log so if it doesn't get committed to disk (possibly 60 seconds by default) it can recover. The transaction log is flushed (fsync) every 100ms by default and is configurable. You can also specify that the write is only acknowledged after the transaction log is synced. Anyways the default is it is put into the transaction log and then acknowledged.
I guess I'll give it another look then.
But I'm still a little wary of a database built by people who ever thought that such behavior was acceptable for a database...
I also agree, having a setting of "fire and forget", although useful in some (limited!) use cases, is not a sane default.
http://www.datastax.com/wp-content/uploads/2013/02/WP-Benchm...
Those with different data access patterns may not be as fortunate.
The marketing copy still pretends that TokuMX doesn't exist, though - it's had these features and more (including transactions) for quite some time now.