"The paper concludes by advising analysts to not go through the Hadoop hoops until your data size passes standard hard drive limits (currently around 1 Terabyte) or at least reasonable memory limits (512 GB)."
Its hard to put a handle on what is 'Big' and what isn't. I think a decent place to draw the line is on data that can't fit on one disk, and 1 TB seems to be a pretty standard disk size right now.
I would say big enough to start thinking about alternate solutions than a monolithic RDBMS. Of course the solutions may include sharding and not necessarily a non-relational store.
Unless you've got a ridiculous RAID array, there's plenty of times where processing anything over 100GB without a cluster is going to suck. Sequentially reading 1TB of data is going to take a few hours even with a RAID array, or be very expensive with SSDs.
I maintain a database that's over 1TB and Oracle handles it very well. The trick there is understanding that, under absolutely no circumstance should you ever need to do a full table scan, because the table isn't designed for that.
So, I'd argue that a monolithic database is okay even up to 10TB even with slower disks as long as you never need to touch more than 10% of it. If you need to touch 100% of the data 100% of the time, I'd say anything over 100GB is too big for one machine.
The reality is that it just depends. There's times where you're going to want a hadoop cluster even for only 16GB of data, and there's going to be times where a database is going to be fine with 10TB of data.
The wild confusion around "big data" stems in part from the fact that many people use it to mean something unrelated to data size.
I was recently in a back-and-forth on twitter about this. Some people argued that "big data" refers to the complexity of the analysis or the value of the insight, rather than the size of the data.
Kaggle CEO Anthony Goldbloom advocated for a definition "too big to fit in an excel spreadsheet."
I advocated for for a definition "large enough that the storage and manipulation becomes part of the challenge (in addition to the analysis).
The phrase has taken on so many definitions as to become meaningless.
Yes I think sheer size is actually just a small part of "big data". The number of dimensions/facets to the data is more important...don't know if that's the correct term, but I mean the ability to, given the schema of your vistor records, analyze which browser has the highest change in bounce-rate given a certain kind of design change in your site. The old-fashioned visitor record type...the number of database pings per day...would not have this much dimension for analysis.
Another part of "big data": the ability to gather, clean, and organize it in a normalized way.
And here I thought it had to do with the size of the companies involved. "Big Pharma", "Big Tobacco", "Big Oil", "Big Blue" and so on, and now there's "Big Data".
The presumption is that big companies have a lot of data, but not necessarily. They may just have unique data needs.
For me, and from an enterprise poit of view, Big Data has always been data external to the enterprise, i.e.: not stored in the warehouse but rather harvested from the web or other sources.
There's this – potentially apocryphal – story that the etymology of the term "monkey patch" comes from someone mistaking the then-current term "guerilla patch" for the similar-sounding "gorilla patch".
Big data seems like a very similar story to me. It has nothing to do with the size of the data, not even with the complexity of the analysis, it simply means, "a movement that wants to do more with more kinds of data." Nowhere near what big data used to mean, but outside of technical circles, we're beyond the point where that even matters anymore.
A former Yahoo big data engineer here. At some point I was bombarded with enquires from different startups who were looking for a "Big Data Guy". The conversation would go as:
- (Me) So, how much data are we talking about?
- (them) 50GB
- Per hour?
- No thats our dataset so far.
- (pause)
- (pause)
(FIN)
The only definition of big data I've thought was remotely insightful is "too large to process with one machine."
Note (as the example in the Wired article indicates) that the converse isn't true: just because you are using multiple machines to process data doesn't mean it's big.
we are company, main products Wigs,False Eyelashes,Remy Hair Extensions,Indian Remy Hair Extensions,Stock Hair http://www.allbeautybox.com Piece,Print Hair Extensions,Fashion Jewelry..... they are 100% original and brand new with cheap price.
the our company have different kinds, there have males, females also girls needing, products are suit for different groups of people, maybe there is something you are just waiting for.
To some people love jewelrys, sure this is the good news for you, our company because our company Remy Hair Extensions is a manufactures, so our company can be lace front wigs sure you are the only one to be focus on the world, the most False Eyelashes important, these are the jewelrys and fashion and design.
our company also have the Do you feeling amazing, then go and action, our company will take best services waiting for Fashion Jewelrys you !
Important!!: our company accept safe payment, like Paypal.... do fast shipping , sure we protect our clients privacy, make order to our company, you get the safe business and privacy !!!!!
At Yammer we used to have a phrase that analytics was "domain knowledge + counting". Lately we've moved to "analytics is workflow" which better captures the deep integration of data, analysis and business decisions that we have now. I still think the first thing matters a lot. You don't necessarily need fancy technology. If you can count and know everything about your business and what makes it valuable then you can start building real insights.
I have my own definition of big data or maybe this is the next "after big data" and that is "streaming data".
Data being generated non stop at a high enough rate that it doesn't make sense to store it. You can only analyse, extra relevant statistics or some features and move on.
Storing it is just putting in a huge buffer and as new data comes in the old data falls of the end.
In some situations where products are up 24/7 in multiple time zones, there is no time for offline batch processing. By the time the batch has finished there is newer possibly bigger batch and so on.
22 comments
[ 3.1 ms ] story [ 47.2 ms ] threadIs a TB really considered Big currently?
I maintain a database that's over 1TB and Oracle handles it very well. The trick there is understanding that, under absolutely no circumstance should you ever need to do a full table scan, because the table isn't designed for that.
So, I'd argue that a monolithic database is okay even up to 10TB even with slower disks as long as you never need to touch more than 10% of it. If you need to touch 100% of the data 100% of the time, I'd say anything over 100GB is too big for one machine.
The reality is that it just depends. There's times where you're going to want a hadoop cluster even for only 16GB of data, and there's going to be times where a database is going to be fine with 10TB of data.
I was recently in a back-and-forth on twitter about this. Some people argued that "big data" refers to the complexity of the analysis or the value of the insight, rather than the size of the data.
Kaggle CEO Anthony Goldbloom advocated for a definition "too big to fit in an excel spreadsheet."
I advocated for for a definition "large enough that the storage and manipulation becomes part of the challenge (in addition to the analysis).
The phrase has taken on so many definitions as to become meaningless.
"Big data is when the size of the data becomes part of the problem."
Mike Loukides (O'Reilly) mentioned to me that Roger Magoulas (O'Reilly) was the first he heard using that definition.
According to this definition, physicists in the 80's were doing big data.
O'Reilly's just pimpin' the term "Big Data" to sell books. They did the same with "Web 2.0", Java and the initial internet boom.
Another part of "big data": the ability to gather, clean, and organize it in a normalized way.
-- Also: [acquisition] (not/just storage). the genius of Google and FB. they [create] massive, usable data sets.
The presumption is that big companies have a lot of data, but not necessarily. They may just have unique data needs.
Big data seems like a very similar story to me. It has nothing to do with the size of the data, not even with the complexity of the analysis, it simply means, "a movement that wants to do more with more kinds of data." Nowhere near what big data used to mean, but outside of technical circles, we're beyond the point where that even matters anymore.
(Originally from this Wavii Engineering blog post http://blog.wavii.com/2011/12/29/your-mileage-may-vary/ )
P.S. If I could down vote twice, I'd give it a second one for turning a sentence into an article.
Note (as the example in the Wired article indicates) that the converse isn't true: just because you are using multiple machines to process data doesn't mean it's big.
the our company have different kinds, there have males, females also girls needing, products are suit for different groups of people, maybe there is something you are just waiting for.
To some people love jewelrys, sure this is the good news for you, our company because our company Remy Hair Extensions is a manufactures, so our company can be lace front wigs sure you are the only one to be focus on the world, the most False Eyelashes important, these are the jewelrys and fashion and design.
our company also have the Do you feeling amazing, then go and action, our company will take best services waiting for Fashion Jewelrys you !
Important!!: our company accept safe payment, like Paypal.... do fast shipping , sure we protect our clients privacy, make order to our company, you get the safe business and privacy !!!!!
Data being generated non stop at a high enough rate that it doesn't make sense to store it. You can only analyse, extra relevant statistics or some features and move on.
Storing it is just putting in a huge buffer and as new data comes in the old data falls of the end.
In some situations where products are up 24/7 in multiple time zones, there is no time for offline batch processing. By the time the batch has finished there is newer possibly bigger batch and so on.