8:9 But there was a certain man, called Simon, which beforetime in the
same city used sorcery, and bewitched the people of Samaria, giving
out that himself was some great one: 8:10 To whom they all gave heed,
from the least to the greatest, saying, This man is the great power of
God.
8:11 And to him they had regard, because that of long time he had
bewitched them with sorceries.
8:12 But when they believed Philip preaching the things concerning the
kingdom of God, and the name of Jesus Christ, they were baptized, both
men and women.
8:13 Then Simon himself believed also: and when he was baptized, he
continued with Philip, and wondered, beholding the miracles and signs
which were done.
Said this before, still want to see it fixed, as I can't stand to read the page with the huge grey box on the right side that disables scrolling without being moused over content. Last time though, I didn't have my environment info.
Windows 7 Ultimate SP1
Chrome Version 26.0.1410.64 m
The fact that so many people are calling things "big data" when the data is not high volume (the most popular definition I've seen is the 5 V's definition--big seems to be a misnomer in this case, as only volume could really be called a measure of "big") lends credence to your statement.
There's an important distinction to be made between the storage layer and the analysis layer. Something like HDFS can make sense as a storage layer once you hit the > 10TB range even if your average dataset for analysis is reasonably small (and it should be; 99% of the time you can get by with sampling down to single-machine size). That doesn't mean you need to be setting up all your analysis jobs to run via map-reduce; you can usually dump the dataset to a dedicated machine and do it all in one go with sequential algorithms. As a side benefit, you have access to algorithms that are really difficult to express efficiently as map-reduce (eg, computations over ordered time series).
I am grateful to finally see this in an article. The "big data" craze is being pushed in areas where it really doesn't make sense. We've been bit by the Big Data bug where I'm at, but it's not coming from the statisticians. It's usually the executives proposing a shift to big data.
People underestimate how much work it would be to shift an old server onto modern technologies and tell the statisticians to use MapReduce and NoSQL instead of SAS and SQL. If the Fortune 500 world has taken this long to catch on to R, imagine how long it'll take to completely change the DBMS and analysis software!
Sure if you're dealing with 1GB of data it probably isn't worth spinning up a Hadoop cluster to run your analysis. However, if you already have Hadoop up an running for something that genuinely requires it, that 1GB job might make sense there. The data may already be in HDFS, and you already have the infrastructure there to manage and monitor jobs.
The references to Facebook & Yahoo running small jobs on huge clusters may be a little misleading. It may be simply the easiest place for them to deploy those jobs consistently.
But yeah... "Big Data" is a total meaningless buzzard.
Like that huge firetruck used to put out small fires. Cities only need them for big fires, but, if you gotta have one and keep it ready, it makes sense to deploy it every time.
I think that big data has made math sexy, and selling applied statistics and operations research to small and medium-sized businesses under the guise of "big data" with the intention of providing applied mathematical tools is what is happening in the market.
It's still amazing what businesses are able to accomplish with summing, counting, percentage of total, % change period over period, average, median, min, max.
Add in bonuses based on those numbers and its amazing any consistency exists in their calculation. Basically in practice you're only allowed a consistent and analytically defensible system if no ones bonus depends on the process being obfuscated. This is why a lot of "big data" is oriented around generating new ideas and new numbers, rather than fixing existing systems and data...
Statistics involves checking modeling assumptions. A lot of what I've seen with the big data people is the repetition of algorithms to the exclusion of understanding and checking modeling assumptions.
While it's nice that the big data craze is making statistics more popular in the mainstream press, it is important that statistics does not become just an application of numerical methods without consideration of underlying assumptions. I stress this because this has been largely underappreciated in my experience.
I meant checking assumptions not just to see whether the use of the big data moved a business metric, but also that the model makes sense from a statistical perspective.
A lot of statistics in business does not bother to check modeling assumptions. Models are chosen based on whether they've been used in the past and what the team is familiar with.
I don't doubt that big data (as we call it now) will one day rule. Ronald Fisher would keel over if he saw the size of datasets we work with nonchalantly on a daily basis. 50 data points (the size of the Iris data) is laughable these days.
My reservation with big data is that the technologies are often unnecessary for the size of the tasks being done. Other than a few data scientists working on truly large projects, most of the big data talk I hear comes from people who aren't fighting in the trenches (execs, marketing, journalists).
This is why I am unconvinced about the prefab products that are currently available. No matter how much you "automate" things, the fact is that you need a human brain, and a decent and careful one at that, to do anything worthwhile. I don't think the majority of companies understand this.
There are a huge number of useful machine learning techniques that don't have checkable "modelling assumptions" per se, just good performance on given tasks (decision trees for instance are really difficult to think about in terms of underlying statistical properties). Heck, even most statistical models are demonstrably false for any given application, yet simultaneously very useful.
Reminds me of a quote I read somewhere about simulation models (specifically referring to Agent Based Modelling) saying that (heavily paraphrasing): "A lot of models are great random number generators"... or "garbage in, garbage out".
I suspect a lot of these people doing "big data models" are as you say, ignoring the importance of having solid assumptions.
Oh well, that's exactly in part what brought down the financial collapse: A bunch of kids get a formula (Blach-Scholes) and believe blindly in its magic powers so they apply it to everything. Fast forward several years and we've got what everybody knows.
This is exactly right. I'm a member of INFORMS (the operations research professional society), and I can report that a staggering amount of ink has been spilled over the last few years about how to capitalize on the recent "Analytics" and "Big Data" trends.
On the one hand, people are starting to realize that quantitative analysis can help their businesses (mind blowing, right?) -- on the other hand, so much of what you see about "analytics" and "big data" is nonsensical jargon. You have two camps within the OR world: people who want to ride this bandwagon all the way to the bank, and people who want to refocus on getting the message out about what OR really is.
The bandwagon-riders have succeeded to some extent. INFORMS created a monthly "Analytics" magazine[1], created an Analytics Certification[2] (their first professional certification), and so on.
The other camp has a legitimate concern that OR already has an "identity crisis" (operations research vs. management science vs. systems engineering vs. industrial engineering vs. applied math vs. applied statistics etc etc). INFORMS has spent millions trying to get business people to just be aware that it exists. The fear is that hitching our wagon to these trends will just be another blow to our profile when these fad words are replaced by the next big thing.
[1] http://analytics-magazine.org/ (you can get a good feel for the type of content in this publication just by reading the article titles...)
The reason the "big data" pimps can get away with this is that most of the people that should know, (that aren't DB programmers, DBAs, true scientists or engineers in the domain), don't know shit about data and generally too fscking lazy to learn. So they buy into the latest wave of buzz words and hype.
Clarifying the scope of a project or data collection & analysis effort is paramount. You never want to attempt to boil the ocean. The key is to figure out the data the matters most per your company or organization's strategy.
Most companies today are already using scaled up servers to host their medium size warehouses (think Teradata or Exadata). That approach is very expensive (> millions of dollars), only works well with well-defined data, and does not scale well beyond a few TBs.
Hadoop is not just about running large jobs on very large data. Hadoop also makes sense when trying to scale on commodity hardware or running ad hoc queries (which can target a small amount of data) on medium to large data sets.
Expensive - yes, comparatively.
Only works well with well defined data - yes, poorly defined data is hard to use in any statistical caclulation too.
Does not scale well beyond a few TB - bullshit. It does scale really well.
If I ever want to get rich, I'll set up shop convincing small businesses they need to do things the way Google does, if only they want to remain competitive.
Oracle has used exactly this business model to great success, and obscene profit, for over 30 years.
I know that's not what you mean, but I find it quite amusing that you describe Oracle (a 30 year old company)'s business as convincing people they need to do things the same way Google (a 15 year old company) does it.
dewitt said "Oracle has used exactly this business model to great success, and obscene profit, for over 30 years". You claim that business models are abstract. Perhaps, except in the case where the words "exact" are used, and a specific company is named. I, like SeoxyS find the paradox of Oracle being accused of helping businesses to be "me-too" copies of Google - a company half of Oracle's age - amusing.
The model in question is "convince small and medium businesses that they need to buy my software in order to do things the same way as large companies X and Y and have a hope of remaining competitive". For Oracle X and Y were banks, retail and logistics companies, for the new generation of "big data" vendors it is Google and Facebook.
But the poster in question didn't say "X" and "Y" did they...this feels like an exercise in pedantry now, but he really did say "exactly" and "google".
But if google has only existed for 15 years how can the business model of "get money out of people trying to help them do a google me-too" have existed for 30?
Well, there is SAP, whose business model is "look at what the big companies are doing. You small time fella, you need the same thing to grow big as well" as well
They need to do it if the cost is affordable. And the fact the cost of Big Data processing became cheap enough to do it en masse is a driver.
Take a simple example. Every small offline shop will not refuse to count every customers head turn (with direction, angle and frequency), calculate averages, and get sales floor attention heatmap divided by day, hour, age and sex.
If that would cost them $500 one-time fee for 10 cameras and $5/day for the cloud service, and produced by pressing one big green key.
That's the Big Data driver, businesses are opening their eyes to the ability to analyze (cheap!) a huge number of small (and even smaller) factors and make better decisions.
Believe it or not, there's a world outside silicon valley where not every company can hire a large team of engineers to create and maintain their data infrastructure to process sales, do crm, keep track of manufacturing, etc. That's why companies like Oracle exist (and are very successful)
Edit: I noticed you meant small businesses. However, Oracle does this mainly for the large companies that don't excel at technology.
I've maintained for awhile now that the distinction isn't between "big" and "small" data, but between coarse and fine data. Now that everything is done through the web, previously common data sources (surveys, sales summaries, etc) are being supplanted by microdata (web logs, click logs, etc). It does take a different skill set to analyze noisy, machine-generated data than to analyze clean, survey-like data; it's a skill set that is more biased towards computational knowledge than classical experimental design, hence the shift in emphasis.
But I also don't really mind if Big Data is truly big, because it's clearly different data than what businesses are used to collecting and interpreting today.
I like this distinction. I walked into a world of hurt when I was brought on to look at application user data after years of working with international trade data and national statistics. Even when it comes to formulating a hypothesis and subsequent experiment, the approach is entirely different.
I will say that the article's distinction between small and big data is also important, but that just comes down to processing power. I think the distinction you make is far more important and knowing whether you need coarse or fine data can help keep you out of the issues that are introduced moving from small to big data.
I also like that distinction. To me "big" data isn't big until there is a lot of it...and there is a definite distinction between "How many bananas were sold Tuesday?" and "Was user's LED email indicator on when xyz happened?"
I agree completely. I run a company that handles high levels of compute load for financial applications. I often describe what we do as "big compute," not big data, because the data is actually very small in size. OTOH, this tiny bit of data (real-time prices on some 1,000 assets) causes an ENORMOUS amount of computation. Often this distinction doesn't get picked up either, and people might mistakenly classify us as "big data."
Extrapolating relatively few truly random data points from massive datasets, for analysis and modeling, is what "Big Data" is all about. This article would have you think that working with clusters or snippets of impossibly ginormous datasets is somehow less "Big", but that's sorta the point. Perhaps somehow should inform the author that the more data available doesn't translate into working with more data.
Be wary about drawing conclusions from "most of the jobs were small." Most of my jobs are small -- because I'm running experiments so I won't have to redo the big one.
That said, I'm a huge proponent of running stuff simply at first. Few businesses will ever grow to the point that they need more than a single large database server and one or two backups. Don't waste your time prepping for something you'll probably never need, especially when fixing the problem when the time comes is only marginally more painful than doing it right in the first place.
"We don't have big data. Our data is small, and could be easily stored in a MySQL or even a flat file" said no dev team ever. Everyone is "just like Google" so they need NoSQL, scaling, clouds and so on.
I've seen a fair number of startups that throw around how they are going to make big money by utilizing the data they gather (called "big data" regardless of size) - it's all a bit of magical underpants thinking: we'll gather a bunch of people/users, we can't figure out how to make money off of advertising or charging them, so then we'll talk about how the "big data" they produce will be worth a fortune and people will pay to have access to it. Know some folks in the HR SaaS space that think this is how they'll hit $100m. It's just comedy.
You can do crazy things with the current hardware specs. Like loading all the data the world bank offers you to download, index it and use it for regressions (I do). In 2013 you only need a laptop for that.
Most data is not big. Big data is "big" like in a gold rush, where the ones selling the tools are making the biggest profits.
EDIT: Thanks for the postgresapp.com link! It is a little bit diffent- here I wanted to use the very same sources as Apple, without adding too much cruft (like a UI to start/stop the daemon as I had seen in other packages). I also wanted to see by myself how hard it was to 'make it work' with OSX (quite easy besides the missing AEP.make and the logfile error). It was basically an experiment in recompiling from the sources given by apple opensource website, while staying as close to the OSX spirit as possible (ex: keeping the same user group, using dscl, using launchdaemon to start the daemon automatically during the boot sequence like for Apache)
That being said, you're right, for most people postgresapp.com will be a simpler and faster way to run a postgresql server :-)
I think this is probably even quicker than the steps you provided (though you have to remember to start it manually, rely on them updating the build, etc) http://postgresapp.com/
SQLite is also an excellent option for a datastore on OSX.Its not nearly as full featured as postgres but no application is required and as you have a OS independent file per db which is extremely portable.SQLite Professional is a relatively decent free gui you can use also.
SQLite has limitations on the data types it supports. Most of this can be worked around by application code, but it can be a pain when you have data that needs to be accessable by more than one application.
In an effort to Store All the Things a lot of companies have talked themselves into a rhetorical corner of poorly fitting shoes. At this point, we've stopped wasting time when asked about Big Table and NoSQL and instead demo their storage stack on a different framework until their eyes widen.
Then, when questions about how much engineering went into this "thing" that does such a good job of keeping data secure, and so much of it, we say it's built on Postgres.
For me, "big" data is increasing the linkage between your data. It's not simply more data, but much richer, less formal data relationships. It's taking your sales data and linking it to your website clicks, linking that to the weather (or whatever). Or you take something traditionally static and add a temporal dimension.
This kind of deep linking you can't measure with straight megabytes. A few gig doesn't seem that large, but if it's a complex graph with a complex hypothesis - then, sure - that's big.
I think that's the entire point behind this article. What you're talking about and what "big data" actually exists as, are two different things. What you're talking about is data refinement. As you said, linking things together and trying to look at your data in different contexts and dimensions than what you do normally. This can be done on a nearly any data set small or large. Big data gets into the realm of literally having so much data to process that data refinement becomes nearly impossible without a significant combination of thinkers, doers, machinery and money.
Well, that's the whole point - "A few gig doesn't seem that large, but if it's a complex graph with a complex hypothesis"... then it's still not 'big data'.
It's maybe smart data, maybe detailed data, but definitely not big data - that problem will have completely opposite needs and techniques than big data analysis, and should not be mischaracterised as such.
1. The accumulation, integration and analysis of a larger number of data sources.
2. A volume of data that presents challenges running analysis functions across them... Due to the limits of the tools available.
1 is fraught with the kind of statistical pitfalls that are mentioned in the posted article. 2 describes a set of problems and boundaries that are time sensitive. What was BigData in 2006 (to, say LiveJournal or Digg) may not longer hold. As a data engineer, its important to keep a skeptical eye on marketing and make sure we're delivering valuable solutions that increase the bottom line for our business, not just produce "ain't it cool" type correlations.
What is the author of this article trying to say here?
>it appears that for both Facebook and Yahoo, those same clusters are unnecessary for many of the tasks which they’re handed. In the case of Facebook, most of the jobs engineers ask their clusters to perform are in the “megabyte to gigabyte” range (pdf), which means they could easily be handled on a single computer—even a laptop.
That facebook or yahoo could be run from a laptop?
159 comments
[ 4.1 ms ] story [ 263 ms ] thread8:8 And there was great joy in that city.
8:9 But there was a certain man, called Simon, which beforetime in the same city used sorcery, and bewitched the people of Samaria, giving out that himself was some great one: 8:10 To whom they all gave heed, from the least to the greatest, saying, This man is the great power of God.
8:11 And to him they had regard, because that of long time he had bewitched them with sorceries.
8:12 But when they believed Philip preaching the things concerning the kingdom of God, and the name of Jesus Christ, they were baptized, both men and women.
8:13 Then Simon himself believed also: and when he was baptized, he continued with Philip, and wondered, beholding the miracles and signs which were done.
God says... son's obeyed interrupt fruit fervently treasure-house unsought overflowed deeper cold student fruit-trees dissolvest readily exceedingly wear assuring re-collected reform pierced
Windows 7 Ultimate SP1 Chrome Version 26.0.1410.64 m
People underestimate how much work it would be to shift an old server onto modern technologies and tell the statisticians to use MapReduce and NoSQL instead of SAS and SQL. If the Fortune 500 world has taken this long to catch on to R, imagine how long it'll take to completely change the DBMS and analysis software!
The references to Facebook & Yahoo running small jobs on huge clusters may be a little misleading. It may be simply the easiest place for them to deploy those jobs consistently.
But yeah... "Big Data" is a total meaningless buzzard.
While it's nice that the big data craze is making statistics more popular in the mainstream press, it is important that statistics does not become just an application of numerical methods without consideration of underlying assumptions. I stress this because this has been largely underappreciated in my experience.
http://krugman.blogs.nytimes.com/2013/05/11/harpooning-ben-b...
Statistics involves inference over prediction, but either one when done right validates assumptions.
By the way big data will sit on your face for days.
A lot of statistics in business does not bother to check modeling assumptions. Models are chosen based on whether they've been used in the past and what the team is familiar with.
I don't doubt that big data (as we call it now) will one day rule. Ronald Fisher would keel over if he saw the size of datasets we work with nonchalantly on a daily basis. 50 data points (the size of the Iris data) is laughable these days.
My reservation with big data is that the technologies are often unnecessary for the size of the tasks being done. Other than a few data scientists working on truly large projects, most of the big data talk I hear comes from people who aren't fighting in the trenches (execs, marketing, journalists).
I suspect a lot of these people doing "big data models" are as you say, ignoring the importance of having solid assumptions.
Oh well, that's exactly in part what brought down the financial collapse: A bunch of kids get a formula (Blach-Scholes) and believe blindly in its magic powers so they apply it to everything. Fast forward several years and we've got what everybody knows.
On the one hand, people are starting to realize that quantitative analysis can help their businesses (mind blowing, right?) -- on the other hand, so much of what you see about "analytics" and "big data" is nonsensical jargon. You have two camps within the OR world: people who want to ride this bandwagon all the way to the bank, and people who want to refocus on getting the message out about what OR really is.
The bandwagon-riders have succeeded to some extent. INFORMS created a monthly "Analytics" magazine[1], created an Analytics Certification[2] (their first professional certification), and so on.
The other camp has a legitimate concern that OR already has an "identity crisis" (operations research vs. management science vs. systems engineering vs. industrial engineering vs. applied math vs. applied statistics etc etc). INFORMS has spent millions trying to get business people to just be aware that it exists. The fear is that hitching our wagon to these trends will just be another blow to our profile when these fad words are replaced by the next big thing.
[1] http://analytics-magazine.org/ (you can get a good feel for the type of content in this publication just by reading the article titles...)
[2] https://www.informs.org/Build-Your-Career/Analytics-Certific...
I think they are talking about the Sharpshooter Fallacy
http://en.wikipedia.org/wiki/Texas_sharpshooter_fallacy
http://en.wikipedia.org/wiki/Nate_Silver#Book
Hadoop is not just about running large jobs on very large data. Hadoop also makes sense when trying to scale on commodity hardware or running ad hoc queries (which can target a small amount of data) on medium to large data sets.
Oracle has used exactly this business model to great success, and obscene profit, for over 30 years.
He said: "Exactly this business model" -- that is, as it pertains to it's essence.
NOT to be read as:
"Exactly this business model as it pertains to inconsequential details, like which big company they should be imitating".
Take a simple example. Every small offline shop will not refuse to count every customers head turn (with direction, angle and frequency), calculate averages, and get sales floor attention heatmap divided by day, hour, age and sex.
If that would cost them $500 one-time fee for 10 cameras and $5/day for the cloud service, and produced by pressing one big green key.
That's the Big Data driver, businesses are opening their eyes to the ability to analyze (cheap!) a huge number of small (and even smaller) factors and make better decisions.
Edit: I noticed you meant small businesses. However, Oracle does this mainly for the large companies that don't excel at technology.
But I also don't really mind if Big Data is truly big, because it's clearly different data than what businesses are used to collecting and interpreting today.
I will say that the article's distinction between small and big data is also important, but that just comes down to processing power. I think the distinction you make is far more important and knowing whether you need coarse or fine data can help keep you out of the issues that are introduced moving from small to big data.
That said, I'm a huge proponent of running stuff simply at first. Few businesses will ever grow to the point that they need more than a single large database server and one or two backups. Don't waste your time prepping for something you'll probably never need, especially when fixing the problem when the time comes is only marginally more painful than doing it right in the first place.
Personally, I am loading the data I play with on a postgreSQL database on my laptop (if you have a mac and want to do that quickly, you may want to check out the link I just submitted http://en.blog.guylhem.net/post/50310070182/running-postgres... )
You can do crazy things with the current hardware specs. Like loading all the data the world bank offers you to download, index it and use it for regressions (I do). In 2013 you only need a laptop for that.
Most data is not big. Big data is "big" like in a gold rush, where the ones selling the tools are making the biggest profits.
EDIT: Thanks for the postgresapp.com link! It is a little bit diffent- here I wanted to use the very same sources as Apple, without adding too much cruft (like a UI to start/stop the daemon as I had seen in other packages). I also wanted to see by myself how hard it was to 'make it work' with OSX (quite easy besides the missing AEP.make and the logfile error). It was basically an experiment in recompiling from the sources given by apple opensource website, while staying as close to the OSX spirit as possible (ex: keeping the same user group, using dscl, using launchdaemon to start the daemon automatically during the boot sequence like for Apache)
That being said, you're right, for most people postgresapp.com will be a simpler and faster way to run a postgresql server :-)
I think this is probably even quicker than the steps you provided (though you have to remember to start it manually, rely on them updating the build, etc) http://postgresapp.com/
Then, when questions about how much engineering went into this "thing" that does such a good job of keeping data secure, and so much of it, we say it's built on Postgres.
As DevOps Borat says : https://twitter.com/DEVOPS_BORAT/status/313322958997295104
How well you can utilize it and how quickly is just as important as what kind of data you store in the first place.
This kind of deep linking you can't measure with straight megabytes. A few gig doesn't seem that large, but if it's a complex graph with a complex hypothesis - then, sure - that's big.
It's maybe smart data, maybe detailed data, but definitely not big data - that problem will have completely opposite needs and techniques than big data analysis, and should not be mischaracterised as such.
1. The accumulation, integration and analysis of a larger number of data sources.
2. A volume of data that presents challenges running analysis functions across them... Due to the limits of the tools available.
1 is fraught with the kind of statistical pitfalls that are mentioned in the posted article. 2 describes a set of problems and boundaries that are time sensitive. What was BigData in 2006 (to, say LiveJournal or Digg) may not longer hold. As a data engineer, its important to keep a skeptical eye on marketing and make sure we're delivering valuable solutions that increase the bottom line for our business, not just produce "ain't it cool" type correlations.
>it appears that for both Facebook and Yahoo, those same clusters are unnecessary for many of the tasks which they’re handed. In the case of Facebook, most of the jobs engineers ask their clusters to perform are in the “megabyte to gigabyte” range (pdf), which means they could easily be handled on a single computer—even a laptop.
That facebook or yahoo could be run from a laptop?