Funny that just two days ago he posted: "It's easy to be negative on Mongo. It's far harder to scale the alternatives. I've been through many ups and downs w/ Mongo. Sticking w/ it." https://twitter.com/NewsBlur/status/314120501356265472
Though the error in that bug was due to a regular replication event ("Fatal Assertion 16360") and this one was happening during an initial sync ("Fatal Assertion 16361"). Maybe the bug was fixed in one and not the other.
I dont understand why you are posting this on here? Is it to point at him and laugh? Its seems pretty childish to point out someone's hardship on here.
NewsBlur is one of the top contenders to replace Google Reader, which is closing in 3 months. The guy behind it posted some of his scaling troubles before https://news.ycombinator.com/item?id=5391713 and his twitter @NewsBlur has been keeping users up-to-date with performance issues and planned downtime for migration etc. This story is interesting to watch.
Many of us here just don't have time to waste on technology that purports to be suitable for high-end work, but then fails us time and time again. As this incident shows, MongoDB can, and often does, end up in this position of failure.
Anyone faced with a MongoDB failure of some sort shouldn't put up with it, and should be very vocal about it. It's important to let others know the problems that can be experienced when using MongoDB.
First, they are not attempting to shard, they are just adding redundancy.
Second, it is not surprising that someone encountered a bug in a piece of server software released literally two days ago.
I guess people tend to expect server _database_ software to not have critical bugs like this. I understand, mistakes happen. But this doesn't help Mongo's less-than-stellar reputation.
Or he could have, I don't know, tested it first? Nah, no one tests first before making such a large change on a production server.
And don't give me crap about it's a one man operation either and we should feel sorry for him, if he's hoping to take over a good portion of Google Reader's subscribers (which he seems to hoping to do) then he should be planning better. If he can't plan to cope with a big influx of new users, that he's asked for by advertising his service, then perhaps migrating to his service isn't the right thing to do. I'd rather have Google Reader notify me that they're shutting down in 3 months than a one man operation shutdown without notice because they can't handle the load and the hassle and costs are way over what they were expecting.
Hope this issue can be resolved in manner that isn't toxic. I actually ran into this issue like this on a fairly large collection 2 months ago (almost 300GB, happened on a single replica set). It looked like an index got irreversibly corrupted. Luckily for us this collection got rebuilt everyday, so we could just delete the whole thing, but the pain was we weren't aware of the issue until a user reported he couldn't access data. When ever the corrupted data was accessed, it would lead to an assertion failure. Seemed like one of those dreadful moments when mongo decides to corrupt your data.
If you actually can't shard a 2.4 db, I feel like we'd have heard about it earlier. There's (almost guaranteed) something else going on here, but nice sensationalist title.
I didn't sensationalize the title. That's just what the bug report says. I can't make any new copies of the data! And I need to start sharding because my service is exploding in growth. What am I supposed to do if I can't replicate?
NewsBlur's developer here. Just want to point out that this bug has bitten me back in 2.2, a few months ago. I need to be able to replicate my db in order to shard. But the biggest beef I have with this bug is that it doesn't become apparent until AFTER you spend 6 hours replicating (which results in poor load times for the primary db).
I would love it if I could choose which server to sync from. This option used to exist, but they removed it. But that would only solve the performance penalty of replication.
To their credit, the MongoDB folks have been stellar. I had a hardware failure a year ago and the CTO ssh'ed into my machines to figure out what was wrong. This time I'm having a bit more difficulty getting the problem fixed, but it's understandable as I have 100GB of highly variable data.
Because that server isn't in the cloud. It's on leased hardware, so no ability to snapshot. Plus, I like it when I can replicate my data without worry. I don't want to have to rely on block-level replication.
The transfer speed is between data centers. West coast to east coast.
As for the 100GB, I have 12 task servers with 6-8 processes each reading 25-50 stories each every 4 seconds. I also have 18 app servers with 6-8 processes each reading 12 stories every second. Each story is on average 4KB. That's several MB a sec.
Armchair quarterbacking here of course but do you think it would be feasible/a good idea to try to switch to a less..troubled product at this point? I don't hear many horror stories about Cassandra for example and AIUI that's serving most of reddit.
Every last tool I have ever worked with has trade-offs. I don't have any problem with that; I've sometimes even gone as far as phrasing it as, "If you don't hate your tech stack, you aren't really using it." I can tell you all kinds of things that C#, the .NET CLR, IIS, Apache, Mercurial, Git, elasticsearch, Redis, Gunicorn, Python, Celery, and SQL Server do that make me livid, because I've used them heavily. But that's because I use them heavily. I've never had any of these tools bite me in the butt early in the process, and definitely not had any of them bite me in the butt in unexpected ways down the road. They bite me when I push them incredibly hard, right to their limits, and, due to well-understood design constraints that I'm frequently anticipating hitting ahead of time, they fall down. That's normal and fine, and handling those situations is just good software engineering. Your tools will have limits, and that stinks, but handling those limits is part of what your job entails, and you need to deal with it.
I do not use MongoDB. But here's what I see: about once a month, I come across an article where something incredibly fundamental to Mongo does not work properly. Not only does it not work properly: the way it doesn't work properly is exceedingly bad. In this case, Newsblur can't shard, which removes one of Mongo's best benefits, and the way it fails isn't to tell you early on that you will not go to space today, but rather to segfault and die after six hours of replication.
That's not predictable. That's not documented. And that's not something you can anticipate. As a developer, that concerns me, and it should concern you, too.
I understand that 10gen is an awesome, responsive company, and they have always been there to help. I don't want to malign that. When the Trello team had Mongo-related issues the other week, they were trying to help them out, too. But I genuinely do not view as paranoia my belief that the frequency and severity of stories like this mean that MongoDB is still not a good technology choice.
not to argue mongodb being good or not, but how many of these issues would be caught if the sites built and used proper dev and test environments? seems like as the cloud has caught on I have read about horrible things happening with all kinds of software that may not have happened with proper procedures. not to say this is that bad or not normal. I think it's a growing curve we are going to see as more people get involved with running and maintaining major systems.
Remind me again why that you think MongoDB is "still not a good technology choice" ?
Given that you're (a) not using it, (b) making judgements based on a series of blog posts, (c) extrapolating bugs that only manifest under specific scenarios, (d) are clearly new to technologies like Apache, ElasticSearch, Git, Redis etc so haven't seen some equally disturbing bugs.
I have to agree - if I had the choice to make again, I wouldn't have chosen MongoDB. It is a fundamentally broken product by design.
For example, for their sharding setup they require three separate servers to host shard config data. The idea is to have high availability of that data should one of those config servers go away.
However, by design, losing a single config server can cause multiple shard masters to die, requiring a manual restart. How they die and when is random, determined by where in a block migration a server is.
How is this high availability? All three config servers must be running at all times without interruption for your cluster to be stable. Their roadmap to deal with this issue is sometime next year.
One month not too long ago, my company found 90% of the bugs listed in their bug tracker for a specific release of MongoDB - many of which would have been found with a basic unit testing suite and some minor load testing. We were effectively performing QA functions for 10gen in our production environment.
I've gone through nine releases of their PHP driver to deal with broken Data Center Awareness and none of them have worked - DCA still eludes them two years later. We have to do an OS level hack to make this work, that breaks other HA functions.
Finally, their mmap design means that memory use is extremely inefficient - on a box with 256GB of memory, with a database that is only 100GB in size, it still hits disk on db reads because they offload memory management to the OS. Any other enterprise-level DB would preload the entire dataset in memory if there's room, but not MongoDB.
Just going to get on the hater train here. (not any poster in particular but Mongo in general).
The day Postgres releases an upgrade, I expect, and get 100% functionality. So the fact that this is "Just Released" is FAR from an excuse, in fact it shows EXACTLY why you shouldn't use Mongo.
Also a 100GB DB, with 300 connections and several MB per second.. Seriously. WTF is wrong with you people? Multiply by 10, and you can still do that on 1 machine.
39 comments
[ 3.1 ms ] story [ 85.8 ms ] thread(Speaking as a user of Postgres and MongoDB, I'm way more worried about the latter screwing me than the former.)
Though the error in that bug was due to a regular replication event ("Fatal Assertion 16360") and this one was happening during an initial sync ("Fatal Assertion 16361"). Maybe the bug was fixed in one and not the other.
This is a JIRA. It became a story once you posted it and commented on it in Hacker News.
Just waiting for the inevitable "this would never happen with PostreSQL" comments to appear.
Many of us here just don't have time to waste on technology that purports to be suitable for high-end work, but then fails us time and time again. As this incident shows, MongoDB can, and often does, end up in this position of failure.
Anyone faced with a MongoDB failure of some sort shouldn't put up with it, and should be very vocal about it. It's important to let others know the problems that can be experienced when using MongoDB.
True, and thanks to whoever edited the title.
I would love it if I could choose which server to sync from. This option used to exist, but they removed it. But that would only solve the performance penalty of replication.
To their credit, the MongoDB folks have been stellar. I had a hardware failure a year ago and the CTO ssh'ed into my machines to figure out what was wrong. This time I'm having a bit more difficulty getting the problem fixed, but it's understandable as I have 100GB of highly variable data.
Not to sound rude, but what is so difficult about managing 100GB of data? You can fit that in RAM without much work.
As for the 100GB, I have 12 task servers with 6-8 processes each reading 25-50 stories each every 4 seconds. I also have 18 app servers with 6-8 processes each reading 12 stories every second. Each story is on average 4KB. That's several MB a sec.
Every last tool I have ever worked with has trade-offs. I don't have any problem with that; I've sometimes even gone as far as phrasing it as, "If you don't hate your tech stack, you aren't really using it." I can tell you all kinds of things that C#, the .NET CLR, IIS, Apache, Mercurial, Git, elasticsearch, Redis, Gunicorn, Python, Celery, and SQL Server do that make me livid, because I've used them heavily. But that's because I use them heavily. I've never had any of these tools bite me in the butt early in the process, and definitely not had any of them bite me in the butt in unexpected ways down the road. They bite me when I push them incredibly hard, right to their limits, and, due to well-understood design constraints that I'm frequently anticipating hitting ahead of time, they fall down. That's normal and fine, and handling those situations is just good software engineering. Your tools will have limits, and that stinks, but handling those limits is part of what your job entails, and you need to deal with it.
I do not use MongoDB. But here's what I see: about once a month, I come across an article where something incredibly fundamental to Mongo does not work properly. Not only does it not work properly: the way it doesn't work properly is exceedingly bad. In this case, Newsblur can't shard, which removes one of Mongo's best benefits, and the way it fails isn't to tell you early on that you will not go to space today, but rather to segfault and die after six hours of replication.
That's not predictable. That's not documented. And that's not something you can anticipate. As a developer, that concerns me, and it should concern you, too.
I understand that 10gen is an awesome, responsive company, and they have always been there to help. I don't want to malign that. When the Trello team had Mongo-related issues the other week, they were trying to help them out, too. But I genuinely do not view as paranoia my belief that the frequency and severity of stories like this mean that MongoDB is still not a good technology choice.
Given that you're (a) not using it, (b) making judgements based on a series of blog posts, (c) extrapolating bugs that only manifest under specific scenarios, (d) are clearly new to technologies like Apache, ElasticSearch, Git, Redis etc so haven't seen some equally disturbing bugs.
There are two things it's good at: 1) indexing arbitrary json and 2) searching for geo points offline.
For example, for their sharding setup they require three separate servers to host shard config data. The idea is to have high availability of that data should one of those config servers go away.
However, by design, losing a single config server can cause multiple shard masters to die, requiring a manual restart. How they die and when is random, determined by where in a block migration a server is.
How is this high availability? All three config servers must be running at all times without interruption for your cluster to be stable. Their roadmap to deal with this issue is sometime next year.
One month not too long ago, my company found 90% of the bugs listed in their bug tracker for a specific release of MongoDB - many of which would have been found with a basic unit testing suite and some minor load testing. We were effectively performing QA functions for 10gen in our production environment.
I've gone through nine releases of their PHP driver to deal with broken Data Center Awareness and none of them have worked - DCA still eludes them two years later. We have to do an OS level hack to make this work, that breaks other HA functions.
Finally, their mmap design means that memory use is extremely inefficient - on a box with 256GB of memory, with a database that is only 100GB in size, it still hits disk on db reads because they offload memory management to the OS. Any other enterprise-level DB would preload the entire dataset in memory if there's room, but not MongoDB.
It really is terrible.
Or rather: It hits the OS's disk cache. I'm not saying that this isn't a problem, but it's far from as bad you make it sound.
(http://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Serve... actually recommends limiting `shared_buffers` (PostgreSQL's in-memory cache) to let the OS disk cache do its magic.)a
http://www.alexa.com/siteinfo/newsblur.com#
There appears to be zero CPU or memory optimization in place. Mostly I blame the mmapped files.