Somewhere between 40 and 60 percent of data science is dealing with stuff like "some of the people collecting the data used 1 for yes and 0 for no, and some of them used 'Y' for yes and 'N' for no, and some misread the question and put in numbers, your job is to figure out which 1s mean yes and which mean that the data is bad," and you can go your entire career in data science without ever using a SVM. But sure, let's have yet another article on methods.
When I opened this thread, this was downvoted. I can understand why, it's kinda dismissive of the article. However, I can also understand the frustration and would generally agree.
Context: Was a data scientist for a bigCo for a few years. 90% of my day to day was cleaning, organizing, building pipelines for, building golden sets for, messy data. I was in a _very fortunate_ position where I got to actually develop some models from the ground up/do some "Creative Work" as it were, but the vast majority of my workload was not stuff I had ever really been prepared for despite a degree in stats.
I don't say this to paint one or the other as "more important/better/more respectable" work, but compared to how much I read about stats vs. going elbow deep in real world data munging, there's certainly a bulk of the former that's not aligned with my use of each. My guess? More generalizable, well encapsulated, easy to talk about. I'd be hard pressed to discuss "general techniques of figuring out how to un-fuck whatever data was being collected in that legacy telemetry system that really wasn't meant for what it's being used for", but that sort of work for me at least proved to be the bulk of my learning on the job.
> When I opened this thread, this was downvoted. I can understand why, it's kinda dismissive of the article. However, I can also understand the frustration and would generally agree.
Although I didn't downvote, I often downvote comments that are negative on knowledge. You know the type. If we're talking about CS they will argue that you don't have to know algorithms and data structures to be a good developer, because you can just go on stackexchange. If we're talking about data, they will argue that knowing what a p-value is useless because the methodology is flawed anyway. If we're talking about knowledge itself they will argue that going to university is a waste of time. And of course, no matter the subject matter, you never need "all that maths".
I don't usually see many comments like this, which "downvote knowledge" as you say.
Instead, I see comments arguing that the emphasis placed on that knowledge is wrong-headed or misplaced.
For example, knowledge about data structures is a great thing. But defining an interview process with data structure trivia as an implicit standard for hiring someone is bad, and is a good teachable moment for pointing out why it's a silly way to filter candidates.
Similarly, detailed knowledge about advanced statistical methods is a great thing. But it's silly to force candidates to fill resumes with buzzwords or treat their stats knowledge like peacock feathers because those are the hoops to get hired, when then in reality the work will be 90% data engineering and devops. It's similarly a teachable moment to point out why an emphasis on these methods is silly, unless you're in a business that actually plans to give an employee projects and tasks that require these things.
I guess if I saw a comment that truly was dismissive of the knowledge, in and of itself, then I would agree that is worth downvoting.
I honestly can say I've not seen that very often, though. Only comments reacting to that way signalling effects are used to subvert the knowledge for e.g. silly hiring trivia or data science credential.
> Similarly, detailed knowledge about advanced statistical methods is a great thing.
We have to keep in mind, though, that nothing in the article is advanced statistical methods. They're the very, very beginning. Being able to run a simple linear regression or bootstrap a data set is equivalent to being able to write fizzbuzz for a programmer.
I’d be more inclined to say that understanding expected value and variance at a baseline level is like fizzbuzz of statistics.
Basic regression at least requires understanding sampling standard error of estimators, distinguishing p-values from the posterior probability a hypothesis is true, transformation of random variables (like for understanding why classical 95% confidence interval bounds are +/- 1.96 * standard error) and some linear algebra.
But I agree the methods discussed, with the exception maybe of support vector machines, are all elementary statistical methods.
What’s sad is that if these represent the fizzbuzz of statistics, it means most jobs in statistics only require you to do tasks easier than fizzbuzz (cursory data cleaning & summary stats). Because in most jobs, dimensionality reduction, SVMs, clustering and boosting would be seen as incredibly advanced, scarce work projects for people to fight over who gets to work on them.
I object to the comparison entirely. Basic statistical concepts are orders of magnitude more complex than a loop with a couple of ifs.
As a software engineer you CAN'T fail fizzbuzz. Whereas in statistics I can come up with problems that rely only on basics but are still counter-intuitive enough that some practitioners will make mistakes.
I used to work as a statistician. Now I work as a software engineer. The comparison is valid between what is considered the basics to operate as a professional.
I would like to know what you consider as knowledge.
Yes, there are lots of developers who do not know algorithms and data structures in depth, yet they get the job done because most jobs simply do not require it.
It's not a universally accepted fact that college helps everyone, it helps some kind of students while others are better off without it.
But i think you simply downvote the comments who you do not agree with.
I was a "business analyst" rather than "data scientist", but this was before the term was a thing (late 90s) - I was an industrial engineering grad student hired to analyze increasing amounts of operational data to improve processes. Pretty similar thing.
One thing I learned from the experience (like you, I believe) is that collecting, managing, and disseminating data is very complicated. I'm glad you put "creative work" in quotations when talking about building ground up models, because I do believe there's a lot of actual creativity and decision making that goes into producing this data. By the time it gets to the model, often the most important assumptions and transformations have already been made.
Here's a story I like to tell to illustrate this - we were looking for ways to improve inventory management, and we noticed a huge spike in orders toward the end of each fiscal quarter. This was a supply chain management issue, a big deal in the early 00s (still a big deal now), and wide variations in demand make it more difficult and expensive to store the right inventory. We were all about creating a model to balance expensive of carrying extra inventory vs shortening lead times.
Except... it turned out here was no spike in orders! The system for adding to an order was irritating to use, so the sales people were just deleting orders and creating new ones. So if they'd had 10 orders over the last 10 weeks, and got 2 more, they'd remove the entire 10 orders from the database and enter a new one for 12 orders at the end of the fiscal quarter! We realized this only by meeting with people on the factory floor, tapping into different databases (some of which were not on the network yet), and finding a different way to track orders.
Experiences like this really do drive home a few things - first, how little you actually do know coming out of college/grad school (it's awesome that you know the mathematics around optimizing stochastic processes, but honestly, it probably isn't worth as much as being aware of a shadow database system on the factory floor that gives a much better picture of what's going on that the sales database that you were all poised to use as input to your optimization model). Second, it shows the amount of creativity, insight, and skills needed. Hell, an investigative journalist would have been more useful to our company than me, an engineer type who preferred to sit at his desk and code up nifty, "Creative" mathematical solutions. There are a lot, and I mean a lot of skills that go into being good at this. Math is important, and engineering degrees are great. But there is an 80/20 rule here. There are companies where knowing the outer orbits of integer optimization or stochastic gradient descent are helpful (my grad program, at Berkeley, seemed mainly interested with putting me through mathematical proofs about convex sets). But in most situations, I think competence in these areas combined with some qualities of an investigative journalist will be considerably more effective.
Not only that, but most people dealing with data at non-technical companies have a hard time to build a structurally useful spreadsheet.
Nobody ever seems to simply put a pivot table on data and aggregate away gaining insight. A few simple transformations and aggregations usually open up kingdoms without even touching on correlating data sets.
So I concur adding value in many situations requires much less than the most advanced probabilistic tools available (even though I do respect these deeply).
It may not be in the job description but usually somebody has to clean the data by hand or build specialized tools before it can be used.
Something as simple as a text field for entering a date can produce interesting effects. Even if your data looks good on the surface, when you dig in, you might find interesting things. Like data entered when the system was definitely shut down for maintenance.
Data engineering cleanse data, manage workflows for ETL, build views, encapsulate rules and logic.
Data science is taking that cleaned up data and testing hypotheses.
Plenty of people can do both, and frankly it's very condescendimg when I run into a data scientist who prefers to outsource the engineering (because they're bad at it) so they can do "real work."
Source: I hire data scientists for Fortune 100 companies.
> Data science is taking that cleaned up data and testing hypotheses.
Dimensionality reduction is a big part of data science, and it tends to be far more important than data cleanup and outlier detection as it dictates which modeling methods are viable in practice given your resources.
I'm paraphrasing a bit, but I read somewhere "Programmers fail because they can't apply statistical methods. Statisticians fail because they can't get their data into a comma separated file".
I don't really share your irritation with the article, which I thought was good. But I do agree data prep is not an afterthought. It’s an important part of data science that requires real skill and expertise.
> The main take away is that Machine Learning is Statistics.
That and brute force computation. A lot of ML models use vastly more data and bigger feature sets than was feasible when these techniques were originally developed. Stepwise regression, lassos, cross validation, etc have become much more popular as raw computing power has increased.
thanks for this comment. I was reading the article thinking "do people really use stepwise / all-subsets regression these days?", but your point is fair.
This blog post's topics are remarkably similar to Stanford's Introduction to Statistical Learning's topic order.
<snip>
Topics include
* Overview of statistical learning
* Linear regression
* Classification
* Resampling methods
* Linear model selection and regularization
* Moving beyond linearity
* Tree-based methods
* Support vector machines
* Unsupervised learning
22 comments
[ 2.9 ms ] story [ 61.6 ms ] threadContext: Was a data scientist for a bigCo for a few years. 90% of my day to day was cleaning, organizing, building pipelines for, building golden sets for, messy data. I was in a _very fortunate_ position where I got to actually develop some models from the ground up/do some "Creative Work" as it were, but the vast majority of my workload was not stuff I had ever really been prepared for despite a degree in stats.
I don't say this to paint one or the other as "more important/better/more respectable" work, but compared to how much I read about stats vs. going elbow deep in real world data munging, there's certainly a bulk of the former that's not aligned with my use of each. My guess? More generalizable, well encapsulated, easy to talk about. I'd be hard pressed to discuss "general techniques of figuring out how to un-fuck whatever data was being collected in that legacy telemetry system that really wasn't meant for what it's being used for", but that sort of work for me at least proved to be the bulk of my learning on the job.
Although I didn't downvote, I often downvote comments that are negative on knowledge. You know the type. If we're talking about CS they will argue that you don't have to know algorithms and data structures to be a good developer, because you can just go on stackexchange. If we're talking about data, they will argue that knowing what a p-value is useless because the methodology is flawed anyway. If we're talking about knowledge itself they will argue that going to university is a waste of time. And of course, no matter the subject matter, you never need "all that maths".
Instead, I see comments arguing that the emphasis placed on that knowledge is wrong-headed or misplaced.
For example, knowledge about data structures is a great thing. But defining an interview process with data structure trivia as an implicit standard for hiring someone is bad, and is a good teachable moment for pointing out why it's a silly way to filter candidates.
Similarly, detailed knowledge about advanced statistical methods is a great thing. But it's silly to force candidates to fill resumes with buzzwords or treat their stats knowledge like peacock feathers because those are the hoops to get hired, when then in reality the work will be 90% data engineering and devops. It's similarly a teachable moment to point out why an emphasis on these methods is silly, unless you're in a business that actually plans to give an employee projects and tasks that require these things.
I guess if I saw a comment that truly was dismissive of the knowledge, in and of itself, then I would agree that is worth downvoting.
I honestly can say I've not seen that very often, though. Only comments reacting to that way signalling effects are used to subvert the knowledge for e.g. silly hiring trivia or data science credential.
We have to keep in mind, though, that nothing in the article is advanced statistical methods. They're the very, very beginning. Being able to run a simple linear regression or bootstrap a data set is equivalent to being able to write fizzbuzz for a programmer.
Basic regression at least requires understanding sampling standard error of estimators, distinguishing p-values from the posterior probability a hypothesis is true, transformation of random variables (like for understanding why classical 95% confidence interval bounds are +/- 1.96 * standard error) and some linear algebra.
But I agree the methods discussed, with the exception maybe of support vector machines, are all elementary statistical methods.
What’s sad is that if these represent the fizzbuzz of statistics, it means most jobs in statistics only require you to do tasks easier than fizzbuzz (cursory data cleaning & summary stats). Because in most jobs, dimensionality reduction, SVMs, clustering and boosting would be seen as incredibly advanced, scarce work projects for people to fight over who gets to work on them.
As a software engineer you CAN'T fail fizzbuzz. Whereas in statistics I can come up with problems that rely only on basics but are still counter-intuitive enough that some practitioners will make mistakes.
Yes, there are lots of developers who do not know algorithms and data structures in depth, yet they get the job done because most jobs simply do not require it.
It's not a universally accepted fact that college helps everyone, it helps some kind of students while others are better off without it.
But i think you simply downvote the comments who you do not agree with.
I was a "business analyst" rather than "data scientist", but this was before the term was a thing (late 90s) - I was an industrial engineering grad student hired to analyze increasing amounts of operational data to improve processes. Pretty similar thing.
One thing I learned from the experience (like you, I believe) is that collecting, managing, and disseminating data is very complicated. I'm glad you put "creative work" in quotations when talking about building ground up models, because I do believe there's a lot of actual creativity and decision making that goes into producing this data. By the time it gets to the model, often the most important assumptions and transformations have already been made.
Here's a story I like to tell to illustrate this - we were looking for ways to improve inventory management, and we noticed a huge spike in orders toward the end of each fiscal quarter. This was a supply chain management issue, a big deal in the early 00s (still a big deal now), and wide variations in demand make it more difficult and expensive to store the right inventory. We were all about creating a model to balance expensive of carrying extra inventory vs shortening lead times.
Except... it turned out here was no spike in orders! The system for adding to an order was irritating to use, so the sales people were just deleting orders and creating new ones. So if they'd had 10 orders over the last 10 weeks, and got 2 more, they'd remove the entire 10 orders from the database and enter a new one for 12 orders at the end of the fiscal quarter! We realized this only by meeting with people on the factory floor, tapping into different databases (some of which were not on the network yet), and finding a different way to track orders.
Experiences like this really do drive home a few things - first, how little you actually do know coming out of college/grad school (it's awesome that you know the mathematics around optimizing stochastic processes, but honestly, it probably isn't worth as much as being aware of a shadow database system on the factory floor that gives a much better picture of what's going on that the sales database that you were all poised to use as input to your optimization model). Second, it shows the amount of creativity, insight, and skills needed. Hell, an investigative journalist would have been more useful to our company than me, an engineer type who preferred to sit at his desk and code up nifty, "Creative" mathematical solutions. There are a lot, and I mean a lot of skills that go into being good at this. Math is important, and engineering degrees are great. But there is an 80/20 rule here. There are companies where knowing the outer orbits of integer optimization or stochastic gradient descent are helpful (my grad program, at Berkeley, seemed mainly interested with putting me through mathematical proofs about convex sets). But in most situations, I think competence in these areas combined with some qualities of an investigative journalist will be considerably more effective.
Nobody ever seems to simply put a pivot table on data and aggregate away gaining insight. A few simple transformations and aggregations usually open up kingdoms without even touching on correlating data sets.
So I concur adding value in many situations requires much less than the most advanced probabilistic tools available (even though I do respect these deeply).
Something as simple as a text field for entering a date can produce interesting effects. Even if your data looks good on the surface, when you dig in, you might find interesting things. Like data entered when the system was definitely shut down for maintenance.
Data engineering cleanse data, manage workflows for ETL, build views, encapsulate rules and logic.
Data science is taking that cleaned up data and testing hypotheses.
Plenty of people can do both, and frankly it's very condescendimg when I run into a data scientist who prefers to outsource the engineering (because they're bad at it) so they can do "real work."
Source: I hire data scientists for Fortune 100 companies.
Dimensionality reduction is a big part of data science, and it tends to be far more important than data cleanup and outlier detection as it dictates which modeling methods are viable in practice given your resources.
I don't really share your irritation with the article, which I thought was good. But I do agree data prep is not an afterthought. It’s an important part of data science that requires real skill and expertise.
Classification
Resampling Methods
Subset Selection
Shrinkage
Dimension Reduction
Nonlinear Models
Tree-based methods
Support Vector Machines
Unsupervised learning
The main take away is that Machine Learning is Statistics.
That and brute force computation. A lot of ML models use vastly more data and bigger feature sets than was feasible when these techniques were originally developed. Stepwise regression, lassos, cross validation, etc have become much more popular as raw computing power has increased.
<snip>
</snip>source: https://online.stanford.edu/courses/stats216v-introduction-s...
There is a free (R-lang based) course, which covers this material by Trevor Hastie: https://online.stanford.edu/courses/sohs-ystatslearning-stat... - it's mentioned in the blog post.