The full title of this post is "The Impact of SSDs on Database Performance and the Performance Paradox of Data Explodification" but I thought the first heading summed it up better.
I too am a huge fan of SQL, what mysql and postgres has given to the world and how it has inspired and set seeds in many of us to use open source, contribute to open source.
I am dipping my toes into couchdb, just to see what all the noise is about.
Still getting my head around map,reduce and the fact im writing in javascript. All that aside the biggest exciting factor for me.
It is so much easier to write custom functions for it than SQL. ( Mysql, and yes I only tried via phpmyadmin )
Uh, he said "fan of SQL Server", which is Microsoft's SQL product. Fans of SQL Server tend not to be fans of mySQL at all, since SQL Server makes mySQL look like a "bottom-feeder" in the original poster's words. Not that I disagree.
I'm getting more and more annoyed by these kind of articles. Why are people talking about NoSQL and relational databases as if they're solving the same problems? There is no "NoSQL" mindset to fight against, and there is no "RDBMS" mindset to fight against: the only mindset to fight against, is a mindset that says a silver bullet exists.
I do not have the vocabulary to explain how much I agree with you. Haven't we gotten to the point where people understand what "use the tool that fits your need" means?
Your criticism of this article is off the mark (though it may apply to other articles). The entire point of this article is that he is trying to replicate the exact same problem that Digg was reporting using a different RDBMS and discussing what that means.
Maybe his numbers are off and his technique is wrong, but this article isn't making a one-sized-fits-all mistake.
Uh, using language like "bottom-feeder RDBMS" and saying that an engineering team is using it "horribly incorrectly on clearly comically deficient hardware" makes this article highly biased, and trollish.
It's pretty clear that this guy things Digg is a bunch of idiot engineers - I mean, why else allude to "rudimentary comp. sci. knowledge"?
These articles might have a central point, but when it's surrounded by a bunch of opinion, it like watching TV news (e.g., Fox News). The real information gets drowned out.
MySQL is a good product, but it does have some significant failings when it comes to getting top-notch performance out of indexes.
Digg's description of their entire setup seems a bit unusual -- from how they've defined their tables to their query methods. It seems at least somewhat likely that they were not making optimal use of their technology.
I'd be curious what the traffic/content difference between Digg and Stackoverflow is. Stackoverflow uses an architecture very similar to what Forbes proposes here and they have plenty of capacity on rather unremarkable hardware.
So while SO runs on a minimal amount of closet-room hardware efficiently and without incident, Digg has heroics and clusters and scale-outs and all sorts of drama to handle just 10x the load? I'm not sure what angle you were coming from, but that makes Digg look like clowns.
Not saying that Digg aren't clowns but the traffic balance makes a huge difference. It's not implausible that Digg has an enormously bigger fraction of writes to reads than SO does which changes the scaling dynamics considerably.
Edit: To expand I'd imagine SO serves a lot of static "how do I" google hits for logged-out users which is about as easy to serve as it gets. Digg puts a higher emphasis on the logged in experience and commenting which reduces the ability to cache.
I agree with you! And on that note I would like to see more articles that describe problems that NoSQL stuff is good at. This is probably the main reason why there is so much confusion. Everybody is talking about how great cassandra/mongodb/couchdb/... is, but nobody tells you where and how to actually use it.
What I'd like to see are articles that discuss design techniques. Like, how to incrementally develop a key store db without inconsistencies? There's no schema so .. I'm worried about partial records after adding values to the non-schema.
First these addresses didn't have a zipcode field, but now they do. What about relationships between records, isn't there going to be redundant data? What if the user changes their email in the account record, do I have to manually update their contact record? This is what foreign keys are for. OMG, Where am I ... Where's the exit? We're all gonna die!
That's basically what I go through every time I think about it. It's a big scary fully loaded gnarly nested hash table aimed directly at my foot.
That's exactly how I feel, and so far I haven't read a single thing that would clear this things up for me. And I read pretty much everything about this topic that I get my hands on. I'm really egger to learn this stuff, to know where to use it, so I could use it in my projects where appropriate.
Somebody who truly gets NoSQL and has use it, should really really write something on this subject.
Yes! That's the same stuff I've been smelling. I look at stuff like documents Google has published and Hadoop and I do think to myself that map-reduce and some of the massively distributed stuff could potentially teach RDBMs some new tricks. Then I see these articles and they're all about someone converting a blog from MySQL to MongoDB.
I'd love to see the technical details of how Digg used MySQL, why it was slow and why NoSQL is better, how they use it. The more of this I see and the less real information the more I think they didn't know what they were doing in the first place.
Just because there's overlap in the problem domains the two classes of tools can be used for does not make them fully interchangeable. If you use a NoSQL tool for transaction processing, you're probably doing it wrong; if you use an RDBMS to house vast amounts of schema-less data, you're probably also doing it wrong. Yes, exceptions exist for every rule, but in the grand Venn diagram of applicability of NoSQL and RDBMSen, there are huge areas that don't overlap. You ignore those areas at your own peril.
In this case, is the data schema-less? They started with a defined schema and maybe now they don't. Your data defines the tool, but you define the data.
I can store JSON in a MySQL column and you'd say I'm using the wrong tool. But I decided to define a schema instead, then I'm using the right tool? It's pretty arbitrary. Some requirements do lean one way or the other but most of the time there's not much of a difference.
The areas where clearly an RDBMS is the correct solution or where NoSQL is clearly the correct solution aren't an issue. I think there are huge areas that do overlap. NoSQL is being advocated in far more situations than just where it's clearly the correct solution.
My point wvenable, is that we are absolutely not dealing with two different kinds of screw drivers here.
Yes they both store "data", but there are types of data that one is well suited to deal with, and alternative types of data that the other one is better for. On my team we've already split out data into two sets, one that lives in ACID RDBMS and one that lives in NoSQL.
If you are dead set in using one solution for everything, my pity for you.
used to solve a problem != meant to solve a problem
Let's bring up the age-old vehicle analogy. Any vehicle fundamentally solves the the same problem but that's naïve outlook that doesn't acknowledge any specific use-cases. I can drive a truck down the street to get a Coke or ride my bicycle across Canada but that doesn't mean bicycles and trucks are meant to solve the same problem. The differences are obvious when you need to move a pallet of Coke (or 50).
What the stuff is "meant for" is often not important in real life situations.
If you have pallet of Coke to move, a bike and no driving licence that you don't care what the bike was meant for.
If you have to hammer a nail but you only have pilers in your close vicinity you hammer it with pilers and don't care what inventors of pilers meant them for.
I'm not trying to draw any parallels to RDBMS vs NoSQL just pointing out that what often matters in real life is what problems are solved with which tools and why. What the tools were meant for is pie in the sky.
For legacy stuff that's true. For new projects we can and should make better decisions (if politics aren't in the way). Many DB systems are free and open source. If you have Unix servers you can choose from PostgreSQL, MySQL, Cassandra, Riak, CouchDB, MongoDB, Voldemort, etc. There is no reason not to choose the right tool for the job. You can assess them all if you have the time and resources.
It's not the physical world and we basically always have whichever tool we need available to us. I mainly use open source software so that may not be true for everyone I suppose. That's their choice though.
I have a feeling that "I made up a random database as big as Digg's, and look, I'm getting 1000x the speed!!!" would kind of tend to imply that you're not accurately replicating their problem.
Agreed. Reading the digg article I'm interpreting their perf claim to mean "on a loaded server in our cluster, if we try to do this query on a cold cache it takes 1.5 seconds to complete", because this is exactly how I would measure the performance.
When I test the performance of a query, I query a live mysql node (or at least a live replicated standby) of some data that's not actively being used (to give a realistic "cold-cache" scenario, even though the caches aren't necessarily cold).
If digg used this method, it would completely account for performance discrepancy. Digg did not release a benchmark, and trying to treat their findings as a repeatable benchmark is wrong.
>Digg did not release a benchmark, and trying to treat their findings as a repeatable benchmark is wrong.
Yet they released record counts, schema, and then performance numbers, and then used their results to demonstrate the failure of the RDBMS (which they led into by saying that it, as some given philosophy, optimizes writes at the cost of reads, hence their poor read performance).
Many of the comments in here are baffling. Digg specifically used the hammer of NoSQL to pound the nail of their database needs, replacing MySQL. They've made a big deal about this. So why the noise about "they're different, man?" And now the petty whines about benchmark methodology when Digg made concrete claims about RDBMS systems?
I think the parent author's main point is they used a defective RDBMS as a representative example of all.
MySQL does not do certain things well. However many mature RDBMS's have solved these problems and its worth pointing that out before you drop the entire class of tools for the next shiny tech.
Because many of us have had the experience of going into a new project where we thought all the existing engineers were idiots, miraculously being given enough latitude to do things our way, and then finding that there were good reasons for those idiotic decisions, and our proposed replacement completely falls down under real-world conditions.
Eventually, you learn not to criticize projects that you aren't actively in the trenches with. There's almost always some subtlety you're missing, and the existing team is too busy fixing it to correct your misconception. I bet the Digg team is looking at these comments (well, if they have time) and thinking to themselves, "We tried that a year ago, and it didn't work. If only they knew..."
You'd have a pretty low opinion of digg engineers to think they'd optimize that poorly. Which the article author does.
It's actually pretty depressing to see content like this from an alleged Database expert. I've never been able to get 1:1 results between "lab" tests and actual deployments, and I've gone to much greater lengths than the author to simulate workloads.
Digg couldn't do a rudimentary join. I'm sorry but if you can read their original entry and not picture an almost comical scene, that you are out of your league.
Now the Joe Plump or whatever guy is telling people that you should sort in PHP. That is the level of expertise of Digg.
I didn't read their original entry. I read an article that was basically "Hurf durf, Digg is retarded, I can do better at home. NoSQL is stupid."
I actually flagged the parent article. It's a troll, and not worth anyone's time. RDBMS is good sometimes. NoSQL is good sometimes. Do we really need _another_ holy war?
And you think this article started the holy war? I believe I just saw another article on his about how you should do SQL sorting in PHP until you graduate to MySQL, or so claims Digg's former architect.
From what I read of the Digg article, it seemed they were hitting situations where doing some work in PHP rather than the DB was the most efficient solution. I don't find anything unreasonable about that; I occasionally run into situations where it's just not possible to get the performance needed while only using the DB, as does probably anyone who's worked enough with databases.
The specific case -- IIRC -- Digg mentioned was a query which required MySQL to generate a temporary table too large to sit in memory, so it ended up being done on disk. Moving some of the processing out of the DB query and into PHP avoided that, understandably resulting in a huge performance difference (in-memory versus on-disk has a way of doing that...).
It certainly didn't start it, but I don't want to see more vapid articles from DBAs about how NoSQL sucks. I want to see actual articles about the real strengths and weaknesses of both technologies. This article just fuels the flames.
You seriously flagged the parent article? That's out of line. While it may be wrong (I personally don't think it is) it's chock full of interesting analysis. It's exactly the kind of article that belongs on this site. Please think before you flag.
Uh, using language like "bottom-feeder RDBMS" and saying that an engineering team is using it "horribly incorrectly on clearly comically deficient hardware" makes this article highly biased, and trollish.
It's pretty clear that this guy things Digg is a bunch of idiot engineers - I mean, why else allude to "rudimentary comp. sci. knowledge"?
These articles might have a central point, but when it's surrounded by a bunch of opinion, it like watching TV news (e.g., Fox News). The real information gets drowned out.
That's a pretty good reason to flag a story. The entire argument is based on the premise that others are idiots: hubris of which we are often guilty. After you've been in the trenches for a while, you should realize that there is usually a pretty reasonable explanation for seemingly stupid problems.
I _very_ rarely flag articles. But look at this thread. Has it generated good discussion? There's a lot of hot air with no backing up on both sides of yet another holy war that I'm already sick of hearing about.
And a few choice statements from the article:
> I would say Digg's case is an example of a bottom-feeder RDBMS product (apologies for being incendiary, but why does the problem always come down to MySQL? These examples always end up being "we moved from MySQL to NoSQL" rather than "We moved from Sybase ASE to NoSQL"), used arguably suboptimally on unpowered hardware,
> went contrary to the demonstration that even a mediocre machine can beat their results.
> Nonetheless, it is a warning sign of a foundational product issue.
> Decent database products like SQL Server even allow you to include
> So either MySQL is an atrociously bad product at the larger limits, which ample evidence seems to point as a truism,
> Please get away from the compiler and save the world from your monstrosities until you have some knowledge of these basic concepts.
> Alternately you can just clutch onto NoSQL and bleat about how it changes all of the rules anyways, which is the route quite a few have decided to pursue
Okay, I'm done. Point is, dude is straight up trolling about how MySQL sucks. This article does nothing but fuel the fire of yet another flamewar, and so it gets flagged. I'd like discussions to remain sane around here.
They all are. Trolling isn't about what one says, it's about the intention behind the things that are said. Completely factual statements are still trolling if said with the intention to cause disruption.
Perhaps you missed the point of the whole exercise: To respond to a high-profile yet poor representation of RDBMS performance.
The OP repeated this reasoning more than once.
He showed that the facts cited by Digg do not make sense unless we take into account poor database technology, poor database configuration, or poor database skills, or all three.
Perhaps you disagree with that. However, that's not what you said above. The OP also discussed the nature of this micro-benchmark, and it's relevance despite his own poor knowledge of the actual data characteristics.
So in other words, he has already directly addressed your concern in advance, more than once on that too as a matter of fact. Considering that fact, you haven't actually responded to his article, you just wrote a "tends to" point about micro-benchmarks, I think it's pretty clear that Dennis Forbes knows a thing or two about benchmarks.
Blah blah blah to empty air, this comment page is pretty much a fact-free and nuance-free flame war anyway, so what's the point, sorta embarrassing for the esteemed HN crowd.
It would have helped the author's case to try (at least approximately) replicating Digg's performance problem on a local MySQL install before switching technologies - that would have given some indication that he'd made the right assumptions about the problem and wasn't testing something unrelated. No matter how good the author is (and I've got no reason to suspect that he's anything other than a top-rate engineer), it's awfully easy to be led down a blind alley on this sort of problem.
> you just wrote a "tends to" point about micro-benchmarks
I am not a database guy. I also don't know enough about Digg's set up to say with authority if these comments make sense. So I specifically wrote "I feel," "tends to," and "imply" because I'm not comfortable making an absolute statement about the issue.
However... I don't see how this test is in any way relevant. a 30GB database? Running on totally different hardware?
In any case, re-reading the article again, I see that relevance paragraph now. I guess I missed it the first time around between all of the flaming, trollish comments about both NoSQL and MySQL. But I still don't see how we can extrapolate this test in any way to imply anything about Digg's practices at all. Then again, it's 8:30am.
Because it was 8:30am when I read this, and I usually wake up at 10.
It still doesn't change my original point, however. Just because he acknowledges that the benchmark is unrelated to what he's talking about doesn't excuse him from the fact that it's unrelated to what he's talking about.
It also doesn't change the fact that the article is still a troll, regardless of the correctness of his benchmark.
After experiencing how easy it is to get started on and develop against MongoDB, I feel like RDBMS are a premature optimization. Its so much easier to evolve your data model, and write arbitrary queries, and its plenty fast enough for 90% of the web apps out there. Save the RDBMS for when you have relational data that needs to be faster, or whatever other feature you happen to need for that part of your app.
Great point. The thing I'm interested in on these new data storage technologies is at the low level, rapid development, early stage (low traffic) portion of a project since thats where I'm at.
I'd rather be able to remove barriers like having to design a schema, and get some early efficiencies to develop my app fast and iterate.
Though, my experience in scaling every site that needed to be scaled has concluded with sharding. So MongoDB sort of fits there as well with its autosharding capabilities.
nosql databases do not free you from having to design your schema! they do free you from having to run migrations, but those are trivial while you are small, migrations are only a problem when you have huge tables. With nosql dbs you can skip migrations and do 'repair on read' schema changes, but you still need to design your schema. and coming from sql world, designing nosql schemas can be a lot of pain since you have to change your perspective completely. and crazy naming (of cassandra and other big-table derivatives) doesn't help either: wtf is column family, etc :).
Exactly. If Cassandra fits my needs, I've got a fast, trivially-scalable database that basically needs no administration, for free. Why would I spend money on a commercial sql database?
That said, Cassandra won't always fit my needs. Sometimes I really do need sophisticated queries and arbitrary transactions. I've been enjoying this guy's articles, because he's bringing up a lot of good performance tips for those times when a relational database is the right tool for the job.
Exactly what I'm asking. Anyone can make read benchmarks fast. He's adding index optimizations such as clustering that kill write performance. Balance please.
The original Digg entry specifically makes the entire point that Digg is willing to sacrifice write performance to improve read performance (which they did to a massive degree with Cassandra), and then demonstrated how absolutely horrendous their read performance was.
Cassandra is hugely biased towards making writes fast [1] and there's no indication in either Digg article that they (somehow) changed that fact in their deployment by tuning. Digg loads the complexity during the write by denormalizing but that load is on the application server not the data store.
> Alternately you can just clutch onto NoSQL and bleat about how it changes all of the rules anyways, which is the route quite a few have decided to pursue.
Bath water and baby gone without a second thought.
Not to mention that later the author says:
> SSDs change everything.
To which I say: "Or you can clutch onto SSDs and bleat about how they change all of the rules anyway, which is the route this author has decided to pursue."
Silly us. The engineers at Google and Amazon also don't know what they're doing either. They should go back to computer science class and let the big boys and SQL Server run the shop.
Wow, did the article say somewhere that Google and Amazon don't know what they're doing? I don't believe it did. In fact I think it quite specifically questioned some performance claims that aren't valid.
It's a different problem. I can get a database that holds 30 GB, although I suspect the actual DB was much larger. I can't get a database that holds >PB.
The problem with this article is that he is testing as ONE ACTIVE USER, as if only one person was ever using digg at any given time. Of course the queries are going to return thousands of times faster. Try replicating digg's actual environment, which I know nothing of, but I know the site gets a ton of traffic and probably has between 10-50,000 users online at any given time.
Now run the same queries you were doing again on your test machine, but simulating 50K users online at once. Oh, and don't forget about thousands of writes per second, which was conveniently not part of this test. What's that you say? The performance is suddenly complete shit? Color me shocked.
Oh, and don't forget about thousands of writes per second, which was conveniently not part of this test. What's that you say? The performance is suddenly complete shit?
I'm not convinced that read performance has to suffer for writes. Reads don't have to block writes or block for writes - use the NOLOCK hint.
These examples always end up being "we moved from MySQL to NoSQL" rather than "We moved from Sybase ASE to NoSQL"
That's because there's a zillion MySQL installs out there, with users that talk about them. On the contrary, there are a lot less Sybase installs out there and their (corporate) users don't talk about them. Go figure that you only hear about MySQL. But please, keep on spreading the FUD; that just gives us the edge of using a free, OSS, system.
DISCLAIMER: This is not a high-fidelity reproduction of Digg's situation
And it's probably not even a low-fidelity reproduction. The article gives us no reason to suppose he actually knew or understood the problem Digg had. He just shows that it was not a trivial one, as that would've been easy to solve.
MySQL is slow on writes because you'll get a random write for every index you maintain (+the table it's self). MySQL's replication will help scale reads but does nothing for writes. Cassandra is actually said to be slower on reads and it's thought people will already be using memcache so it's not a problem.
The article doesn't seem to have any writes going on while he is reading.
There is lots of stuff in SQL server for backups, hot spares, clustering. It can be fault-tolerant if you set it up that way. I wonder if you knew about that functionality, and if not, why make claims in ignorance.
77 comments
[ 3.1 ms ] story [ 137 ms ] threadI am dipping my toes into couchdb, just to see what all the noise is about.
Still getting my head around map,reduce and the fact im writing in javascript. All that aside the biggest exciting factor for me.
It is so much easier to write custom functions for it than SQL. ( Mysql, and yes I only tried via phpmyadmin )
Uh, he said "fan of SQL Server", which is Microsoft's SQL product. Fans of SQL Server tend not to be fans of mySQL at all, since SQL Server makes mySQL look like a "bottom-feeder" in the original poster's words. Not that I disagree.
Maybe his numbers are off and his technique is wrong, but this article isn't making a one-sized-fits-all mistake.
It's pretty clear that this guy things Digg is a bunch of idiot engineers - I mean, why else allude to "rudimentary comp. sci. knowledge"?
These articles might have a central point, but when it's surrounded by a bunch of opinion, it like watching TV news (e.g., Fox News). The real information gets drowned out.
Digg's description of their entire setup seems a bit unusual -- from how they've defined their tables to their query methods. It seems at least somewhat likely that they were not making optimal use of their technology.
I'd be curious what the traffic/content difference between Digg and Stackoverflow is. Stackoverflow uses an architecture very similar to what Forbes proposes here and they have plenty of capacity on rather unremarkable hardware.
http://www.alexa.com/siteinfo/stackoverflow.com#trafficstats
Edit: To expand I'd imagine SO serves a lot of static "how do I" google hits for logged-out users which is about as easy to serve as it gets. Digg puts a higher emphasis on the logged in experience and commenting which reduces the ability to cache.
First these addresses didn't have a zipcode field, but now they do. What about relationships between records, isn't there going to be redundant data? What if the user changes their email in the account record, do I have to manually update their contact record? This is what foreign keys are for. OMG, Where am I ... Where's the exit? We're all gonna die!
That's basically what I go through every time I think about it. It's a big scary fully loaded gnarly nested hash table aimed directly at my foot.
I'd love to see the technical details of how Digg used MySQL, why it was slow and why NoSQL is better, how they use it. The more of this I see and the less real information the more I think they didn't know what they were doing in the first place.
If you can switch from one to the other and back again, how are they not solving the same problems?
I can store JSON in a MySQL column and you'd say I'm using the wrong tool. But I decided to define a schema instead, then I'm using the right tool? It's pretty arbitrary. Some requirements do lean one way or the other but most of the time there's not much of a difference.
The areas where clearly an RDBMS is the correct solution or where NoSQL is clearly the correct solution aren't an issue. I think there are huge areas that do overlap. NoSQL is being advocated in far more situations than just where it's clearly the correct solution.
Yes they both store "data", but there are types of data that one is well suited to deal with, and alternative types of data that the other one is better for. On my team we've already split out data into two sets, one that lives in ACID RDBMS and one that lives in NoSQL.
If you are dead set in using one solution for everything, my pity for you.
Let's bring up the age-old vehicle analogy. Any vehicle fundamentally solves the the same problem but that's naïve outlook that doesn't acknowledge any specific use-cases. I can drive a truck down the street to get a Coke or ride my bicycle across Canada but that doesn't mean bicycles and trucks are meant to solve the same problem. The differences are obvious when you need to move a pallet of Coke (or 50).
If you have pallet of Coke to move, a bike and no driving licence that you don't care what the bike was meant for.
If you have to hammer a nail but you only have pilers in your close vicinity you hammer it with pilers and don't care what inventors of pilers meant them for.
I'm not trying to draw any parallels to RDBMS vs NoSQL just pointing out that what often matters in real life is what problems are solved with which tools and why. What the tools were meant for is pie in the sky.
It's not the physical world and we basically always have whichever tool we need available to us. I mainly use open source software so that may not be true for everyone I suppose. That's their choice though.
When I test the performance of a query, I query a live mysql node (or at least a live replicated standby) of some data that's not actively being used (to give a realistic "cold-cache" scenario, even though the caches aren't necessarily cold).
If digg used this method, it would completely account for performance discrepancy. Digg did not release a benchmark, and trying to treat their findings as a repeatable benchmark is wrong.
Yet they released record counts, schema, and then performance numbers, and then used their results to demonstrate the failure of the RDBMS (which they led into by saying that it, as some given philosophy, optimizes writes at the cost of reads, hence their poor read performance).
Many of the comments in here are baffling. Digg specifically used the hammer of NoSQL to pound the nail of their database needs, replacing MySQL. They've made a big deal about this. So why the noise about "they're different, man?" And now the petty whines about benchmark methodology when Digg made concrete claims about RDBMS systems?
MySQL does not do certain things well. However many mature RDBMS's have solved these problems and its worth pointing that out before you drop the entire class of tools for the next shiny tech.
Eventually, you learn not to criticize projects that you aren't actively in the trenches with. There's almost always some subtlety you're missing, and the existing team is too busy fixing it to correct your misconception. I bet the Digg team is looking at these comments (well, if they have time) and thinking to themselves, "We tried that a year ago, and it didn't work. If only they knew..."
It's actually pretty depressing to see content like this from an alleged Database expert. I've never been able to get 1:1 results between "lab" tests and actual deployments, and I've gone to much greater lengths than the author to simulate workloads.
That's all in the article.
Now the Joe Plump or whatever guy is telling people that you should sort in PHP. That is the level of expertise of Digg.
I actually flagged the parent article. It's a troll, and not worth anyone's time. RDBMS is good sometimes. NoSQL is good sometimes. Do we really need _another_ holy war?
The specific case -- IIRC -- Digg mentioned was a query which required MySQL to generate a temporary table too large to sit in memory, so it ended up being done on disk. Moving some of the processing out of the DB query and into PHP avoided that, understandably resulting in a huge performance difference (in-memory versus on-disk has a way of doing that...).
Uh, using language like "bottom-feeder RDBMS" and saying that an engineering team is using it "horribly incorrectly on clearly comically deficient hardware" makes this article highly biased, and trollish. It's pretty clear that this guy things Digg is a bunch of idiot engineers - I mean, why else allude to "rudimentary comp. sci. knowledge"? These articles might have a central point, but when it's surrounded by a bunch of opinion, it like watching TV news (e.g., Fox News). The real information gets drowned out.
That's a pretty good reason to flag a story. The entire argument is based on the premise that others are idiots: hubris of which we are often guilty. After you've been in the trenches for a while, you should realize that there is usually a pretty reasonable explanation for seemingly stupid problems.
And a few choice statements from the article:
> I would say Digg's case is an example of a bottom-feeder RDBMS product (apologies for being incendiary, but why does the problem always come down to MySQL? These examples always end up being "we moved from MySQL to NoSQL" rather than "We moved from Sybase ASE to NoSQL"), used arguably suboptimally on unpowered hardware,
> went contrary to the demonstration that even a mediocre machine can beat their results.
> Nonetheless, it is a warning sign of a foundational product issue.
> Decent database products like SQL Server even allow you to include
> So either MySQL is an atrociously bad product at the larger limits, which ample evidence seems to point as a truism,
> Please get away from the compiler and save the world from your monstrosities until you have some knowledge of these basic concepts.
> Alternately you can just clutch onto NoSQL and bleat about how it changes all of the rules anyways, which is the route quite a few have decided to pursue
Okay, I'm done. Point is, dude is straight up trolling about how MySQL sucks. This article does nothing but fuel the fire of yet another flamewar, and so it gets flagged. I'd like discussions to remain sane around here.
Maybe MySQL really isn't a decent database product, or not for high performance needs. How many of his statements are troll-ish in that light?
The OP repeated this reasoning more than once.
He showed that the facts cited by Digg do not make sense unless we take into account poor database technology, poor database configuration, or poor database skills, or all three.
Perhaps you disagree with that. However, that's not what you said above. The OP also discussed the nature of this micro-benchmark, and it's relevance despite his own poor knowledge of the actual data characteristics.
So in other words, he has already directly addressed your concern in advance, more than once on that too as a matter of fact. Considering that fact, you haven't actually responded to his article, you just wrote a "tends to" point about micro-benchmarks, I think it's pretty clear that Dennis Forbes knows a thing or two about benchmarks.
Blah blah blah to empty air, this comment page is pretty much a fact-free and nuance-free flame war anyway, so what's the point, sorta embarrassing for the esteemed HN crowd.
I am not a database guy. I also don't know enough about Digg's set up to say with authority if these comments make sense. So I specifically wrote "I feel," "tends to," and "imply" because I'm not comfortable making an absolute statement about the issue.
However... I don't see how this test is in any way relevant. a 30GB database? Running on totally different hardware?
In any case, re-reading the article again, I see that relevance paragraph now. I guess I missed it the first time around between all of the flaming, trollish comments about both NoSQL and MySQL. But I still don't see how we can extrapolate this test in any way to imply anything about Digg's practices at all. Then again, it's 8:30am.
What a unclever excuse for missing the point.
It still doesn't change my original point, however. Just because he acknowledges that the benchmark is unrelated to what he's talking about doesn't excuse him from the fact that it's unrelated to what he's talking about.
It also doesn't change the fact that the article is still a troll, regardless of the correctness of his benchmark.
I'd rather be able to remove barriers like having to design a schema, and get some early efficiencies to develop my app fast and iterate.
Though, my experience in scaling every site that needed to be scaled has concluded with sharding. So MongoDB sort of fits there as well with its autosharding capabilities.
That said, Cassandra won't always fit my needs. Sometimes I really do need sophisticated queries and arbitrary transactions. I've been enjoying this guy's articles, because he's bringing up a lot of good performance tips for those times when a relational database is the right tool for the job.
Need more be said? Seriously?
[1] http://spyced.blogspot.com/2010/01/cassandra-05.html
Bath water and baby gone without a second thought.
Not to mention that later the author says:
> SSDs change everything.
To which I say: "Or you can clutch onto SSDs and bleat about how they change all of the rules anyway, which is the route this author has decided to pursue."
Now run the same queries you were doing again on your test machine, but simulating 50K users online at once. Oh, and don't forget about thousands of writes per second, which was conveniently not part of this test. What's that you say? The performance is suddenly complete shit? Color me shocked.
I'm not convinced that read performance has to suffer for writes. Reads don't have to block writes or block for writes - use the NOLOCK hint.
That's because there's a zillion MySQL installs out there, with users that talk about them. On the contrary, there are a lot less Sybase installs out there and their (corporate) users don't talk about them. Go figure that you only hear about MySQL. But please, keep on spreading the FUD; that just gives us the edge of using a free, OSS, system.
DISCLAIMER: This is not a high-fidelity reproduction of Digg's situation
And it's probably not even a low-fidelity reproduction. The article gives us no reason to suppose he actually knew or understood the problem Digg had. He just shows that it was not a trivial one, as that would've been easy to solve.
sed s/NoSQL/NoMySQL/
and avoid the confusion.
MySQL is slow on writes because you'll get a random write for every index you maintain (+the table it's self). MySQL's replication will help scale reads but does nothing for writes. Cassandra is actually said to be slower on reads and it's thought people will already be using memcache so it's not a problem.
The article doesn't seem to have any writes going on while he is reading.