Ask HN: All of you working with Big Data, what is your Data?
Big data is a trending topic these days and I'd like to get my hands dirty both out of curiosity and to make myself more relevant on the marketplace. That being said, I'm not sure which data sets are both interesting to play with and easily accessible. My question is:
For those of you already working with big data, what kind of data do you work with?
102 comments
[ 4.3 ms ] story [ 174 ms ] threadSide project: Poll responses on http://www.correlated.org
PriceMatch is hiring in Paris!
The mariadb table storing this information currently takes a bit more than 500GB, it has about 4 billion rows (based on the statistics, I don't run SELECT count(*) on it anymore).
I usually don't use the term "big data" because the buzzword is so popular that it doesn't mean anything anymore.
I have been collecting data from the twitter API for a few days as well. I wanted to get an idea of average tweeting pattern, but without access to the firehose API, I got a feeling that the sample I have isn't very "neutral" as I have been mostly pulling from the popular and local tweets endpoints.
Any advice on how should I approach these kind of data collection?
If user streams are not releavant to you, you may use the `filter` endpoint instead of `sample`, and focus on keywords describing a relevant niche for your analysis.
In case you want to limit yourself to tweets geo-located in a certain location, you have to be aware that the bounding box filter of twitter is buggy: it will return you tweets geo-located outside of the desired area, and you are not sure to be getting all tweets.
Thanks!
NOTE: there are people in the world who would laugh at my definition and say that big data starts at 1Pb.
I'm you were trying to do anything that's O(n.log n) or O(n^2) (think graph processing) then you'll run into trouble at much smaller scales.
If it fits in RAM on your laptop, it isn't big data.
If you can't process/handle it in a reasonable time on a single machine and your methods need to explicitly worry about how to scale to handle the data volumes it probably is "Big Data".
Problems that are embarrassingly parallel need far more data before I'd consider them big (I'd be in the >10PB camp), whereas for relational data I'd say >1TB.
These days Amazon have Multi-AZ RDS, which should handle the 2nd item.
You can't practically fit 50TB on one machine and have reasonable performance, that means multiple machines with the data spread across them.
There's then two potential issues: 1) You're doing 1-to-1 joins across tables in a query, network latency may be an issue at high query rates 2) You're going 1-to-many or many-to-many joins across tables in a query, the resulting combinatorial explosion of data is too much to handle
You want to have your inner loops/joins as deep down in the stack as possible. If you can structure things so all the heavy lifting stays inside one rack/machine/NUMA node/ processor/core you'll be able to scale a good bit further further.
Designing things not to require joins, denormalising and putting it in a column store like Cassandra is also a good approach.
Another is not having pockets deep enough to solve it with intellectual property, either in the form of a parallel proprietary rdbs (spensive) or the need to implement clever stuff.
Big data as a technology is about dumb as brick, cheap as chips, brute force.
I commend them for having a larger penis ^H^H^H^H^H^H data stack than you.
I thought big data was less about the actual size of the data store and more about where it comes from (typically passive collection from user activity) and how it's accessed (through some kind of large map-reduce style framework) and used (to inform product decisions or learn more about human behavior)?
An excellent example was on HN the other day, using the NYC taxi data to determine which drivers are observant Muslims. It's not something anyone set out to record, but the data set has gotten so large that if you turn it sideways and shake, random facts like that fall out.
Big data is neither big nor particularly complex.
If it cannot, then you pay the price of all the complexity and overheads of big data processing techniques so that you can get your processing done.
It's correlated with data size, bot not so strictly - you can get, for example, NLP processing problems where you need a painful pipeline split over a huge cluster for a single gb of input data, and you can have problems where the best way to process a petabyte dataset is just to stick a single powerful machine to get the performance benefits of locality and low latency, and avoid managing splits/failed nodes/whatever.
So, in the first problem you would need to use Big Data techniques and the second problem you don't, it's not related to big data and the recommendations on how best to do that won't help people who need to do big data processing.
1PB is arguably enough data to store genetic variation across all human beings.
I was working on the principle that the effective population size of humans is 10,000.
(And your genome is oversized, no? 3 billion base-pairs is less than 1 Gigabyte)
It's not just eh size though, IMO it implies a certain dimensionality and/or lack of structure. At work were sittin on several petabytes and I don't view it as Big Data because it's actually pretty simple. We share many of the same problems as Big Data but not all
AlienVault is hiring security researchers.
edit: we have some limited data sets that we make public, incase you're interested, hence the name 'open threat exchange'.
Fire up a VM with a single-node install on it [1] and just grab any old CSVs. Load them into HDFS, query them with Hive, query them with Impala (Drill, SparkQL, etc.). Rinse and repeat for any size of syslog data, then JSON data. Write a MapReduce job to transform the files in some way. Move on to some Spark exercises [2]. Read up on Kafka, understand how it works and think about ways to get exactly-once message delivery. Hook Kafka up to HDFS, or HBase, or a complex event processing pipeline. You'll probably need to know about serialization formats too, so study up on Avro, protobuf and Parquet (or ORCfile, as long as you understand columnar storage).
If you can talk intelligently about the whole grab bag of stuff these teams use, that'll get you in the door. Understanding RDBMSes, data warehousing concepts, and ETL is a big plus for people doing infrastructure work. If you're focused on analytics you can get away with less of the above, but knowing some of it, plus stats and BI tools (or D3 if you want to roll your own visualization) is a plus.
[1] http://www.cloudera.com/content/cloudera/en/downloads/quicks... [2] http://ampcamp.berkeley.edu/5/
In my experience your job generally dictates what you specialize in. I ended up being more data engineer than scientist since my job had a lot of tricky data warehousing problems.
- getting data out of production systems and transforming it (infrastructure or ETL) - analytical querying and reporting - system administration - machine learning
There's also the wide world of NoSQL data stores, which people lump in with big data, but which require vastly different skills.
The Hadoop VM I linked to above is good for working through exercises for all of the above.
As a starting point, this book[1] walks through the motivation behind Hadoop, and then gets a little into internals and use cases. It's out of date, but you can work through it and get into the right frame of mind, understand HDFS, etc. It's a good starting point.
AMP Camp (that I linked to above) is an introduction to Spark for people with a little Hadoop experience. Spark is getting a lot of attention, you could run into it in a number of roles.
If you're going to be planning the whole pipeline, or doing any sort of infrastructure role, I recommend Hadoop Application Architecture[2] for more modern tools and design patterns. This blog post[3] is a pretty good overview of distributed logs, which are essential for horizontal scale. Understanding Kafka and ZooKeeper is really useful for infrastructure roles, maybe less so for admins.
If you're planning to be in the reporting layer, having a deep understanding of SQL and data warehousing is useful. This book[4] is old hat, but I would say it's expected knowledge for anyone planning a warehouse, and it's interesting to understand best practices. Most places will also expect knowledge of Tableau or a similar BI tool, but that's tougher to learn on your own since licenses are brutal. Visualization with D3 is nice to have in this space, especially if you're coming from a web background - Scott Murray's tutorials [5] are a good starting place.
It's harder to point to resources for sysadmins - if you weren't a sysadmin before, you need to understand a lot of other concepts before you worry about Hadoop stuff. ML is similar - you need to understand the principles and be able to work on a single node. There's lots of good resources out there about getting started in data science.
1. http://shop.oreilly.com/product/0636920021773.do
2. http://shop.oreilly.com/product/0636920033196.do
3. http://engineering.linkedin.com/distributed-systems/log-what...
4. http://ca.wiley.com/WileyCDA/WileyTitle/productCd-0471200247...
5. http://alignedleft.com/tutorials
Just for giggles I built a tool that extracts all oracle permissions, sums that up into relationship information using PIG (this schema owner reads from here, writes to there, etc) and used first R/ggplot2 and then later Gephi to plot the results.
None of the data sets could be called big data by any stretch, and I could have done the processing more quickly with perl or python, or even a mix of shell commands. But that wasn't the point. It was to expand on the one day of training I'd had and help cement the ideas, and frankly it was to have fun.
Find something that you're passionate about or just plain sounds like fun and then use the tool you want to learn to solve your problem.
1. With a big data set, there is no easy way to verify the correctness of your algorithms. The data is too big to hand inspect, and so assuming your code is syntactically well-formed and doesn't crash, you will get an answer. Is your answer correct? Well, you don't actually know, and any number of logic errors might throw it off without causing a detectable programming error.
2. Big data is messy. There will be some records in your data set that are formatted differently than you expect, or contain data that means something semantically different than you expect. Best case, your Hadoop job crashes 4 hours in. Worst case, it silently succeeds, and you have no idea that your results were polluted by spurious results that you had no idea existed.
3. Big data will expose basically every code path and combination of code paths in your analysis program, so it all better be bulletproof. Learn how to write code correctly the first time, or you're going to be spending a lot of time waiting for the script to run and then fixing crashes several hours in.
4. Big data contains outliers. Oftentimes, the outliers will dominate your results, and so if you don't have a way of filtering them out or making your algorithm less sensitive to them, you will get garbage as your final answer.
There are techniques to deal with these, but they are techniques that are built into your workflow as a data scientist, and not the tools that are available. One thing that always amazed me at Google was how much time the data scientists on staff spent not writing code. Writing your MapReduces takes perhaps 5-10% of your day; most of the rest of it is mundane stuff like staring at data and compiling golden sets.
This is sadly true, for now. I don't think folks here disagree with the "true" part. Let me explain why it is "sadly" and "for now".
The biggest issue with big data is most of it sits unused. In many organizations, HDFS ends up being an alternative to NetApp storage servers, storing terabytes of data with the hopes of them being useful one day.
In fact, if you already get to that stage of using HDFS as a storage server, you must have a decent ETL team that can put data into HDFS with a menacing combination of ad hoc scripts and a workflow that looks like a cobweb produced by a deranged spider. For now, knowing the ins and outs of various semi-functional open source components and the tenacity, patience and skill to deal with the gnarliest of ETL tasks get you a high-paying data engineering job.
But, in the long term, there will be a big change.
1. Tools are getting better: many data practitioners are realizing there are huge gaps between different data infrastructure components, and they are trying to fill these gaps. There is a lot of attention given to query execution engines (Presto, Impala, Spark, etc.) but I find data collection/workflow management tools are just as critical (if not higher leverage) right now. Tools like Fluentd (log collector) [1], Luigi (workflow engine) are OSS software in this direction.
2. Data-related cloud services are becoming really, really good: huge kudos to services like AWS, GCP, Heroku (through Addons). They are quickly building a great ecosystem of data processing/analysis/database components that frankly work better than most self-administered OSS counterparts. (Disclaimer: my perception might be colored here since I work for a data processing/collaboration SaaS myself [3])
So, back to the question. I think aspiring data engineers have two distinct career paths:
1. Becoming an expert in a particular data engineering component: this would be building a query execution engine, designing a distributed stream processing system, etc. (It would be awesome if you decide to release as open source)
2. Becoming an expert on quickly and effectively deploying cloud services to get the job done: this is the skill most desired among data engineers at startups.
What not to become is one of these OSS DIY bigots: not good enough to build truly differentiating technology, but adamant about building and running their own <up and coming OSS technology>. These folks will be wiped out in the next decade or so.
[1] https://www.fluentd.org [2] http://luigi.readthedocs.org [3] http://www.treasuredata.com
This is really variable. If you're at a place where they jumped on the bandwagon, then yes. There are also lots of companies (and not just Google/FB/LinkedIn) that build mission critical reporting and ML infratstructure on Hadoop. These companies appreciate the value of workflow coordination, and they wouldn't move ahead without (at least) Oozie/Azkaban in place to give some visibility into their workflow.
> But, in the long term, there will be a big change.
I think more types of work will become commoditized. If you just want log processing, there are lots of on-premises and cloud options. Splunk has been doing this forever. Ostensibly with good-enough BI software you could just focus on ingest, and everything else is drag and drop. On a long enough time frame, hand-rolling pipelines will become obsolete. This is like a 10+ year timeline for any player to get significant market share. In the meantime, people have to actually get stuff done, and their skills will be transferable because they understand distributed systems, ETL, warehousing, and a lot of other stuff that hasn't really changed in a decade.
> Becoming an expert in a particular data engineering component
Are you advocating that nobody writes Spark Streaming jobs, because they should rewrite Spark instead? Don't learn to work with Impala, learn to rewrite Impala? I disagree, the tools are only getting better, and it's going to take more and more work to replace the entrenched players. Working on top of solid tools will make you far more productive than engaging in NIH and making your own SQL engine.
> Becoming an expert on quickly and effectively deploying cloud services to get the job done
Like RedShift, EMR and Amazon Data Pipeline? They're hardly turn-key solutions. Amazon's Kinesis is just Kafka with paid throughput - you can absolutely re-use your skills in the cloud, without having to cave and get locked in to a single vendor serving one specific use-case.
> What not to become is one of these OSS DIY bigots: not good enough to build truly differentiating technology, but adamant about building and running their own <up and coming OSS technology>
So in your mind you either pick a vendor to handle all your data for you, or you're an "OSS DIY bigot"? Something like owning your entire user analytics pipeline isn't mission critical for a startup, it's stupid to build it yourself?
> These folks will be wiped out in the next decade or so.
Even though Oracle is amazing and great, lots of people still use Postgres, MySQL, etc. There's always going to be a continuum from "We should buy his turnkey thing" to "we started by rolling our own SQL query engine". You need to be able to identify when each is appropriate, not shoehorn in a one-size-fits-all solution.
Colleague of mine is at a company that's advertising for someone with "big data" experience. Collectively, for more than 10 years in business, they have maybe 100g of data. They just do not know how to organize the data sanely in a relational database, and actively refuse to consider normal data structures.
If you're looking for a cool dataset to play with, I think it is more productive to ask yourself what questions you want to answer and then find/curate the data VS find a dataset and then ask "what questions can I answer?". The former approach will also keep motivations high if you're driven by curiosity.
I second that. An old remark is, "We often find that a good question is more important than a good answer.", as I recall, due to Richard Bellman, say, the leading proponent of dynamic programming, i.e., usually a case of optimal control, either for the deterministic or stochastic (the system gets random exogenous inputs while we are trying to control it). Bellman was into a lot in pure and applied mathematics, engineering, medicine, etc. Bright guy. As I recall, his Ph.D. was in stability of solutions of initial value problems for ordinary differential equations, from Princeton.
I think most data sets could be handled via RDBMS and Big Data is just another choice. The more interesting thing to me is what you accomplish and if a new tech can get you there faster or cheaper, etc.
The largest open-access event stream archive I know about is from GitHub, I think it's about 100Gb: https://www.githubarchive.org/
For me it's primarily population data. It's not exactly 'big' data in the raw form, but what makes it bigger are the variations analyzing, applications of predictive models, and new metadata values extracted from it.
The data grows faster than we're collecting it exponentially because of all the analysis.
Application idea from top of my head: Generate turnstile data for different subway stations (enter/exit, time) and wrote an application to show the density of those stations with times. You can create a scenario where a certain station is more dense than others, and this could be your test. And this application could be your proof of concept
All of them are basically time series with some master data, none of them is more than a few dozen GB
So in any case, I think time series data is worth a look.
MongoDB was bad for this case because it devours memory and it lacks a primary key (so saving a tick that's already creates a second tick).
cassandra is better (it has a PK, although it's a bit weird because of its distributed-first attitude) but in the long run I think postgres would be even better (because it's space efficient).
Apart from that I use some Java libraries and clojure/incanter and program the rest around it myself.
I don't think of the options stream as big data. It is more of a "fast" data problem, where at the open you need to handle 100,000 ticks per second, doing a implied volatility calculation on each tick. It is pretty demanding, when, for example, you consider that when the price of IBM moves by one penny, you get 400 options quote right then. You end up with a billion quotes in a typical day, if memory serves correctly.
I didn't think of this as Big Data, somehow. It seems to me that big data is more complex than that. You most often just process the data, say for backtesting, in a strict series, from the start to the end, or for an interval. I think of Big Data having more structure.
Utilizing Splunk as analytics and alerting platform to correlate real time financial activity events with multiple threat intelligence feeds.
To be clear 'big data' is poorly defined, and I mostly do not work with terabyte+ data sets, but rather with highly dimensional data in moderate volume. Data is only 'big' relative to the algorithms you try to use on it.
The data, currently, is mainly from various Network Security Monitoring appliances & SIEMs.
The data, currently, is mainly from various Network Security Monitoring appliances & SIEMs.