There is something that somewhat irks me about the whole domain of data science. I still can't quite put my finger on it, but I think it's a mixture of lack of rigor, handwaving arguments.
Maybe if it were called data journalism, or data storytelling rather than science I would somewhat lower my expectations and enjoy it more fully, but in my experience doing science and doing data science are not really close.
I think you could make the same argument about computer science, especially relating it to the actual practice of professional programming, which resembles pop culture more than science.
And while both fields have mathematical underpinnings, the foundations of data science -- probability and statistics -- have deeper roots and a longer history.
I guess I'm saying you shouldn't impugn the field itself because of its popular practice.
Data science is about building testable models, which fundamentally is what science is about.
What probably makes you uncomfortable is that Data Science (unlike most scientific fields) doesn't have a requirement for the models to make intuitive sense to humans.
I agree whole heartedly. When I first started playing around with data, despite my lite statistics background, I had no idea what I was doing and no amount of education would have prepared me for noisey, real life data sets. I had to become proficient with SQL and a certain type of problem solving and hacker mentality before I was able to do anything useful.
What's interesting now though is that since returning to college and taking more advanced courses in probability and statistics, I now have a "practical" background that allow me to see real life use cases for almost everything I'm learning (oh wonderful Bayes theorem!) that is a huge advantage.
I find the don't-go-to-school-just-do-it mentality that pervades tech blogs, article, discussions a bit short-sighted. I fully appreciate the importance of getting "hands-on experience" in addition to book learning, but what I don't get is the black-and-white perspective that formal education gives you NO EXPERIENCE WHATSOEVER.
As a PhD candidate, my perspective is quite definitely skewed, but my experience in school so far has not given me the impression that all I've been doing is book learning with no transferrable, real-world skills to work with data. I've had to work with plenty of data sets during my Masters and PhD research. I firmly believe that the experience I've gained from applying what I've learned in the classroom and from surveying existing literature on data mining and statistical methods stands me in good stead to tackle "real data" (whatever that means).
Maybe I've got the wrong end of the stick, but I think that something more than the "odd stats class" can be of value to a budding data scientist ...
Don't you find it odd that education start with the abstract to have you end up in reality.
Data science is not about well structured data, it's not about learning what others do. I
It's about finding meaningful information through experimentation with data and for that all you really need is a lot of experimentation, a creative mindset and the ability to extrapolate meaning and correlation out of seemingly uncorrelated data.
First experiment, learn to play around, explore, then apply theory.
Actually, I don't think the process of going from abstract to reality is that odd. The reason I say that is because I think understanding the abstract prepares you to be flexible. In terms of data science, the way I see this is that by experimenting without having a grounding in the algorithms, technologies etc. means that you can only really ever get good at the specific implementations you're working with. The abstract understanding is precisely what allows you to be versatile in adopting new technologies and seeing the implementation for what it is: one concrete version of the underlying abstract ideas.
There's nothing that says that formal education only prepares you to work with well-structured data. In fact, I think that in grad school you are challenged to work at the cutting edge almost all the time. Granted, a lot of classroom examples at the undergrad level are perhaps trivial, but my experience is that grad school doesn't by definition only serve to prep you with pointless examples.
> It's about finding meaningful information through experimentation with data and for that all you really need is a lot of experimentation, a creative mindset and the ability to extrapolate meaning and correlation out of seemingly uncorrelated data.
I totally agree, and I believe that school is not at odds with this. I'm working in the fields of planetary science and space engineering at the moment, and I am constantly stimulated to experiment with data, be creative and understand correlations.
Maybe for some reason my experience is just not a good 3-sigma example ...
I am totally on-board with what you're saying, I guess I just don't buy into the idea that school != connected to the real world. I think the idea that everyone in academia is being philosophical and scholarly without contact with the real world is just wrong. There are plenty of practical skills that you can learn through school. Given the general anti-school sentiment I tend to come across in HN discussions, it might just be that I'm the odd one out when it comes to thinking that going to (grad) school is not at odds with gaining exposure to working with real world data.
It's not that school isn't connected to the real wold. It's that you don't need it besides a very few areas.
And if that is the case why put yourself to debt to go to school when you don't have to? Why pay a lot of money up front to acquire a knowledge you can get for free. You will have to learn the rest of your life anyway without having to go to school.
You have the worlds knowledge at your fingertips. You have any expert in any field you can ask about anything you want today for more or less free.
I would even claim advicing against school in most cases is a moral just cause.
I guess the debt argument is one that I don't have direct experience with, since I'm going to school in Europe, where the norm is not that you come out of school saddled with debt that you'll be paying off for the rest of your life. So in that sense, I guess I get your argument for the "moral cause", but I think that's more a problem specific to the US and the administration of colleges as opposed to the problem with academia as a whole.
It's true that you don't need school for a lot of things in life, but that's not the same as saying there's no value to going to school.
Again, what irked me was simply the statement in the blog that other than the odd stats class, there's no value in going to school if you're an aspiring data scientist.
I concede that there's a large difference between the added value of undergrad programs and grad school. I strongly disagree that you can substitute what grad school gives you by simply Googling it or by taking a few online classes. In particular, mastering your own research project I think gives you a lot of real-world skills and that includes working with real, unstructured data sets.
I am also from Europe and went to European school.
What exactly is the school giving you that you can't learn on your own, by experimenting, reading, learning, studying. You can take the same courses and so on.
You seem to assume that the school somehow has something unique that cant be replicated outside of it.
Why can't you master your own research projects? Why wouldn't you have access to real, unstructured data outside school?
Clearly the author managed just fine. I myself have managed doing quite well in another field without any formal training via a school.
I'm not saying you can't do it on your own. I do believe that there are multiple ways to skin a cat, but what I don't agree with is that opting to pick the route of going through school is not going to prep you to work with real-world data. You can argue that formal training is not necessary, but I fail to see how that means that there's no value in what's taught in school.
The very fact of the matter is that through my research I have gained exposure to working with real, unstructured data, and I have had the opportunity to develop skills that I believe stand me in good stead to pursue a career in industry as a data scientist.
So, basically I guess I'm reversing the argument and saying that as much as you can acquire any skills without going to school (and that includes primary and secondary school), there's nothing in my experience that points to the fact that picking the route of formal training is going to place you behind the 8-ball as an aspiring data scientist.
The author has managed fine indeed. My contention is with the following quote from the GigaOm post:
> "“I think the applied experience is a lot more important than the academic experience. It probably can’t hurt to take a stats class in college.”
I don't see applied experience and academic experience as being mutually exclusive concepts, so to make the statement that one is more important than the other is to fail to understand that as much as you can learn things on your own, school can also provide you with hands-on skils to become a data scientist.
Your first statement I don't agree with -> I don't see how you can't break out of your frame of reference. If your frame of reference is working with data and building skills that are equally applicable in a business environment, then I think you can make the transition without major hiccups. That's the belief of the people running the Insight Fellowship as well, which has had great success in providing a channel to bring people with an academic background into industry.
I absolutely agree, formal education doesn't not imply greater influence nor greater success, however, neither does it imply that you can't reach the highest heights. So there's a fallacy in the thought process that if you want to be a data scientist, school is not the a possible option. I think there are people who can be successful in the data science business who pick it up by themselves, with all the free material out there, and equally I think you can approach the field with formal training and make a name for yourself.
Anyway, by the sounds of it, we're probably reaching the point of agreeing to disagree.
I didn't get the impression that he was suggesting people interested in Data Science should forego theory and lecture altogether.
His argument for building applied skills first, was that it would motivate academic pursuit later (as well as being immediately applicable). If you're already reading for your Ph.D. we might assume you're not at a loss for motivation :-)
(okay now I went back and reread)
I also get the impression he's talking to an application-focused audience. 16 year olds looking at colleges, or 20 year olds considering a focus for a masters, are probably not the target audience of the Harvard Business Review.
Yea, I appreciate the fact that he isn't suggesting that people abandon school althogether. I think what irked me is his suggestion that school is maybe useful for the "odd stats class", suggesting to me that he doesn't believe that there is value beyond that.
Perhaps my reading of the blog post isn't entirely correct. I agree that he's not making any statement directly to dissuade people from going to school. I just think that it's important to note that my experience of being in school is that it has given me freedom to be creative, experiment, play around with new technologies, and ultimately build applied skills. Perhaps I'm in the minority that feels that way ...
EDIT: After reading the full blog post, I can see that his view is more nuanced than comes across on GigaOm.
It's worth pointing out that Nate Silver is a rare individual. He's been wildly successful with data science despite not having a deep academic background, which is not the common case. His story is a useful data point though.
Do you need multiple degrees to master data science? Hell no. The practice of statistical inference requires only a few key skills. The important part is understanding how to think in terms of probability and algorithms, which doesn't require a rigorous understanding of proofs and formulae. An advanced academic degree is useful but not required. The world is looking for people who can consistently get the right answer, regardless of their background.
The other side to this is that a degree will be required to work for wall st, google, harvard, etc. Degrees still matter and social signalling and so on and so forth. We all can't be as well known and popular as Nate Silver and at some point institutions need a way of establishing that a particular candidate is truly qualified. So if you're not a messianic ml rock star, sweating the degree might be a good idea.
The beauty of a purely empirical craft is that results are the only arbiter of success. Anyone can get the right answer. It might be awhile before society catches up to and utilizes this powerful idea, but the hiring competitions on sites like kaggle are a great start [1]. Until then the best case is being the next Nate Silver.
Like anything, real world experience complements academic background, and over time the credential means less.
But as a thought experiment... All things equal, who wouldn't hire someone with 2 years experience + a masters, over someone with just 2 years experience, or just the masters.
I like to see something that signifies intelligence and curiosity, and then something that shows someone has done something with it. Usually this is a combination of academics and work. It can be shown just with work. It's hard to show it just with academics.
“… Getting your hands dirty with the data set is, I think, far and away better than spending too much time doing reading and so forth,” Silver said in a Q&A with HBR’s Walter Frick.
I'm in a data science class (hi carlob) right now. This is exactly how we've approached the subject. There were a few readings the first couple weeks to get people up to speed on Python (or scare them away if they didn't have any coding experience), then our homework problem sets have jumped straight into real-world data scraping and modeling.
It's been a fantastic way to start learning the field, because after just 3.5-ish weeks I already feel like I have the tools I need to start exploring and fitting rudimentary models to the information I'm collecting.
It's exciting and enabling, and I think that's definitely a strength of doing while learning.
I feel like Data Science is not a real academic field, and it's not something you need a college degree in. The programs are so new and rudimentary, the universities don't really even know what to teach the students in an "Intro to Data Science" class. But it's a buzzword in the media, and there's employer demand now, so I think that universities saw the $$$ and just ran with it.
It's really more of a technical skillset and toolset for statisticians and other actual scientists who use statistics to test their hypotheses. Make Intro CS, Relational Algebra and Set Theory, and Computational Statistics electives for Bachelor of Science degrees. This really doesn't need its own degree program, and neither did "business intelligence" back when that was the buzzword for this. Universities aren't for "latest tools and fads" training.
> I feel like Data Science is not a real academic field, and it's not something you need a college degree in.
Its an emergent academic field in the way that bioinformatics was a few years ago; "data science" is a name for something that's been taught in less breadth and depth as parts of other academic disciplines (including parts of the "research methods" curriculum in the life and social sciences, parts of statistics, parts of computer science), but certainly its as legitimate an area of focus as the disciplines its that parts of it have been hosted in. There's more than one way to divide up intellectual pursuit, and a number of ways that have cut across the previously predominant divisions have been coming to the fore recently. This is, really, a good thing.
Yeah, I guess what I'm trying to say is that Data Science is really a "horizontal", not a "vertical" in academia, to borrow business jargon. It's an interdisciplinary research skill set that students and scholars across academic disciplines should be trained in. As such, I'm not convinced it needs its own degree programs, though.
I think the quality of work being done in the disciplines you call "verticals" is improved by increasing the degree of development in the "horizontals" -- and that includes recognizing that those "horizontals" are legitimate fields of focus on their own, because the further development and focus on the "horizontal" feeds back into the tool sets available to all the "verticals" the "horizontal" touches.
most people don't really think logically or quantitatively. whatever math people learned in high school and college was forgotten when trying to make adult decisions. and they didn't even get to the good stuff.
data science is successful now because they are applying quantitative methods to the social sciences -- problems most people think about. the average educated person can understand by using the magic of "machine learning" we can solve problems in politics, health, social issues, etc.
Nate Silver's lack of a math background works to his advantage. he realizes the bar for quantitative understanding in the public sphere is very low. whatever little bit he does, he has to explain to a room full of drunk people in 30 seconds or less. he is very good at this.
neither employers nor the public want to admit that at the end of the day SOMEONE has to be responsible for these calculations. i have a master's degree in math -- which nobody gave a hoot about (see item 5 in this article http://gigaom.com/2013/04/16/how-to-hire-data-scientists-and... ) we weren't taught how to communicate our ideas and results to the public -- in fact we were discouraged from it. that is difficult to unlearn.
i've heard of attempts to "democratize" machine learning.
http://blog.bigml.com/2013/03/06/democratizing-machine-learn...
articles like these are usually written by companies with a product or book to sell or distribute. what needs to be democratized is engineering based problem solving - using the powerful computers we hold in our pockets, our phones.
Nate Silver's lack of a math background? Why does this keep getting repeated. He has a BA in economics from one of the most quantitative and rigorous programs in the country, plus some work at LSE. I'm sure he has more formal math and statistics education than most self-labeled "engineers" on HN.
Looks fairly rigorous. Can't say for sure, but it does appear that while you are encouraged to take the most rigorous calc sequence (typically required of math, hard science, engineering majors), econ majors are allowed to take an easier calculus track. That's pretty typical of econ undergraduate degrees - many universities do provide a slower, easier calculus track for certain majors (typically life science majors or social science majors).
I have no idea what track Nate Silver took, and of course I can't speak to the "self-labeled" engineer. I'd agree that an econ from a reputable school has some mathematical rigor, but not quite at the level required of a typical math, hard science, or engineering major.
Either way, I would agree with you that it is silly to say Nate Silver doesn't have a math background - clearly he does.
Is there anyone around that "does data science" for a living that can talk about what it is they do exactly?
I'm under the impression that "data scientist" is as ambiguous a term as "designer". What's the minimum amount of math or statistics required to be labeled a data scientist? And what's the minimum amount of programming too?
36 comments
[ 3.5 ms ] story [ 112 ms ] threadMaybe if it were called data journalism, or data storytelling rather than science I would somewhat lower my expectations and enjoy it more fully, but in my experience doing science and doing data science are not really close.
And while both fields have mathematical underpinnings, the foundations of data science -- probability and statistics -- have deeper roots and a longer history.
I guess I'm saying you shouldn't impugn the field itself because of its popular practice.
What probably makes you uncomfortable is that Data Science (unlike most scientific fields) doesn't have a requirement for the models to make intuitive sense to humans.
What's interesting now though is that since returning to college and taking more advanced courses in probability and statistics, I now have a "practical" background that allow me to see real life use cases for almost everything I'm learning (oh wonderful Bayes theorem!) that is a huge advantage.
As a PhD candidate, my perspective is quite definitely skewed, but my experience in school so far has not given me the impression that all I've been doing is book learning with no transferrable, real-world skills to work with data. I've had to work with plenty of data sets during my Masters and PhD research. I firmly believe that the experience I've gained from applying what I've learned in the classroom and from surveying existing literature on data mining and statistical methods stands me in good stead to tackle "real data" (whatever that means).
Maybe I've got the wrong end of the stick, but I think that something more than the "odd stats class" can be of value to a budding data scientist ...
Data science is not about well structured data, it's not about learning what others do. I
It's about finding meaningful information through experimentation with data and for that all you really need is a lot of experimentation, a creative mindset and the ability to extrapolate meaning and correlation out of seemingly uncorrelated data.
First experiment, learn to play around, explore, then apply theory.
There's nothing that says that formal education only prepares you to work with well-structured data. In fact, I think that in grad school you are challenged to work at the cutting edge almost all the time. Granted, a lot of classroom examples at the undergrad level are perhaps trivial, but my experience is that grad school doesn't by definition only serve to prep you with pointless examples.
> It's about finding meaningful information through experimentation with data and for that all you really need is a lot of experimentation, a creative mindset and the ability to extrapolate meaning and correlation out of seemingly uncorrelated data.
I totally agree, and I believe that school is not at odds with this. I'm working in the fields of planetary science and space engineering at the moment, and I am constantly stimulated to experiment with data, be creative and understand correlations.
Maybe for some reason my experience is just not a good 3-sigma example ...
We start as infants and learn to crawl, gobble, walk, talk, read, write, calculate through our interaction with the world, it informs us.
The danger of starting with the abstract is that you make everyone generic and thus give them the same same frame of reference to think within.
And if that is the case why put yourself to debt to go to school when you don't have to? Why pay a lot of money up front to acquire a knowledge you can get for free. You will have to learn the rest of your life anyway without having to go to school.
You have the worlds knowledge at your fingertips. You have any expert in any field you can ask about anything you want today for more or less free.
I would even claim advicing against school in most cases is a moral just cause.
It's true that you don't need school for a lot of things in life, but that's not the same as saying there's no value to going to school.
Again, what irked me was simply the statement in the blog that other than the odd stats class, there's no value in going to school if you're an aspiring data scientist.
I concede that there's a large difference between the added value of undergrad programs and grad school. I strongly disagree that you can substitute what grad school gives you by simply Googling it or by taking a few online classes. In particular, mastering your own research project I think gives you a lot of real-world skills and that includes working with real, unstructured data sets.
What exactly is the school giving you that you can't learn on your own, by experimenting, reading, learning, studying. You can take the same courses and so on.
You seem to assume that the school somehow has something unique that cant be replicated outside of it.
Why can't you master your own research projects? Why wouldn't you have access to real, unstructured data outside school?
Clearly the author managed just fine. I myself have managed doing quite well in another field without any formal training via a school.
The very fact of the matter is that through my research I have gained exposure to working with real, unstructured data, and I have had the opportunity to develop skills that I believe stand me in good stead to pursue a career in industry as a data scientist.
So, basically I guess I'm reversing the argument and saying that as much as you can acquire any skills without going to school (and that includes primary and secondary school), there's nothing in my experience that points to the fact that picking the route of formal training is going to place you behind the 8-ball as an aspiring data scientist.
The author has managed fine indeed. My contention is with the following quote from the GigaOm post:
> "“I think the applied experience is a lot more important than the academic experience. It probably can’t hurt to take a stats class in college.”
I don't see applied experience and academic experience as being mutually exclusive concepts, so to make the statement that one is more important than the other is to fail to understand that as much as you can learn things on your own, school can also provide you with hands-on skils to become a data scientist.
If you start with academic experience you your frame of reference is already established and its much harder to break out of that.
Sure you can get a job, but you are not more likely to be influential in your field just because you have an education.
I absolutely agree, formal education doesn't not imply greater influence nor greater success, however, neither does it imply that you can't reach the highest heights. So there's a fallacy in the thought process that if you want to be a data scientist, school is not the a possible option. I think there are people who can be successful in the data science business who pick it up by themselves, with all the free material out there, and equally I think you can approach the field with formal training and make a name for yourself.
Anyway, by the sounds of it, we're probably reaching the point of agreeing to disagree.
I didn't get the impression that he was suggesting people interested in Data Science should forego theory and lecture altogether.
His argument for building applied skills first, was that it would motivate academic pursuit later (as well as being immediately applicable). If you're already reading for your Ph.D. we might assume you're not at a loss for motivation :-)
(okay now I went back and reread)
I also get the impression he's talking to an application-focused audience. 16 year olds looking at colleges, or 20 year olds considering a focus for a masters, are probably not the target audience of the Harvard Business Review.
Perhaps my reading of the blog post isn't entirely correct. I agree that he's not making any statement directly to dissuade people from going to school. I just think that it's important to note that my experience of being in school is that it has given me freedom to be creative, experiment, play around with new technologies, and ultimately build applied skills. Perhaps I'm in the minority that feels that way ...
EDIT: After reading the full blog post, I can see that his view is more nuanced than comes across on GigaOm.
Do you need multiple degrees to master data science? Hell no. The practice of statistical inference requires only a few key skills. The important part is understanding how to think in terms of probability and algorithms, which doesn't require a rigorous understanding of proofs and formulae. An advanced academic degree is useful but not required. The world is looking for people who can consistently get the right answer, regardless of their background.
The other side to this is that a degree will be required to work for wall st, google, harvard, etc. Degrees still matter and social signalling and so on and so forth. We all can't be as well known and popular as Nate Silver and at some point institutions need a way of establishing that a particular candidate is truly qualified. So if you're not a messianic ml rock star, sweating the degree might be a good idea.
The beauty of a purely empirical craft is that results are the only arbiter of success. Anyone can get the right answer. It might be awhile before society catches up to and utilizes this powerful idea, but the hiring competitions on sites like kaggle are a great start [1]. Until then the best case is being the next Nate Silver.
[1] http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extr...
But as a thought experiment... All things equal, who wouldn't hire someone with 2 years experience + a masters, over someone with just 2 years experience, or just the masters.
I like to see something that signifies intelligence and curiosity, and then something that shows someone has done something with it. Usually this is a combination of academics and work. It can be shown just with work. It's hard to show it just with academics.
Just curious, what do you think they are?
I'm in a data science class (hi carlob) right now. This is exactly how we've approached the subject. There were a few readings the first couple weeks to get people up to speed on Python (or scare them away if they didn't have any coding experience), then our homework problem sets have jumped straight into real-world data scraping and modeling.
It's been a fantastic way to start learning the field, because after just 3.5-ish weeks I already feel like I have the tools I need to start exploring and fitting rudimentary models to the information I'm collecting.
It's exciting and enabling, and I think that's definitely a strength of doing while learning.
It's really more of a technical skillset and toolset for statisticians and other actual scientists who use statistics to test their hypotheses. Make Intro CS, Relational Algebra and Set Theory, and Computational Statistics electives for Bachelor of Science degrees. This really doesn't need its own degree program, and neither did "business intelligence" back when that was the buzzword for this. Universities aren't for "latest tools and fads" training.
Its an emergent academic field in the way that bioinformatics was a few years ago; "data science" is a name for something that's been taught in less breadth and depth as parts of other academic disciplines (including parts of the "research methods" curriculum in the life and social sciences, parts of statistics, parts of computer science), but certainly its as legitimate an area of focus as the disciplines its that parts of it have been hosted in. There's more than one way to divide up intellectual pursuit, and a number of ways that have cut across the previously predominant divisions have been coming to the fore recently. This is, really, a good thing.
data science is successful now because they are applying quantitative methods to the social sciences -- problems most people think about. the average educated person can understand by using the magic of "machine learning" we can solve problems in politics, health, social issues, etc.
Nate Silver's lack of a math background works to his advantage. he realizes the bar for quantitative understanding in the public sphere is very low. whatever little bit he does, he has to explain to a room full of drunk people in 30 seconds or less. he is very good at this.
neither employers nor the public want to admit that at the end of the day SOMEONE has to be responsible for these calculations. i have a master's degree in math -- which nobody gave a hoot about (see item 5 in this article http://gigaom.com/2013/04/16/how-to-hire-data-scientists-and... ) we weren't taught how to communicate our ideas and results to the public -- in fact we were discouraged from it. that is difficult to unlearn.
i've heard of attempts to "democratize" machine learning. http://blog.bigml.com/2013/03/06/democratizing-machine-learn... articles like these are usually written by companies with a product or book to sell or distribute. what needs to be democratized is engineering based problem solving - using the powerful computers we hold in our pockets, our phones.
http://collegecatalog.uchicago.edu/thecollege/economics/#pro...
Looks fairly rigorous. Can't say for sure, but it does appear that while you are encouraged to take the most rigorous calc sequence (typically required of math, hard science, engineering majors), econ majors are allowed to take an easier calculus track. That's pretty typical of econ undergraduate degrees - many universities do provide a slower, easier calculus track for certain majors (typically life science majors or social science majors).
I have no idea what track Nate Silver took, and of course I can't speak to the "self-labeled" engineer. I'd agree that an econ from a reputable school has some mathematical rigor, but not quite at the level required of a typical math, hard science, or engineering major.
Either way, I would agree with you that it is silly to say Nate Silver doesn't have a math background - clearly he does.
I'm under the impression that "data scientist" is as ambiguous a term as "designer". What's the minimum amount of math or statistics required to be labeled a data scientist? And what's the minimum amount of programming too?