There are lots of things wrong with "Machine Learning" practice that are problems of the practice instead of the subject itself.
One of the most dangerous things that comes out of academia is the "project orientation", that is, the idea that something is "done" at a certain point in time.
The "technical debt" aspect comes in when you are managing a product or a program, which can be thought of as a series of projects that influence each other.
The "paradigm" is that somebody wants to learn machine learning so they download the same set of digits that Hinton used in his papers, and then they replicate the result, and then they announce they are done. When you see everybody is chasing the same problems (Hinton's digits, the Kaggle of the day) you start to realize that the setup of a problem which can be solved by machine learning is actually worth more than the learning.
> you start to realize that the setup of a problem which can be solved by machine learning is actually worth more than the learning.
This is a fantastic point, but I'd like to also caution readers that they should spend some time understanding the elements of statistics and machine learning before jumping into theano or Weka tutorials. Lately, I've seen a lot of voodoo ML in the valley, starting with billboards on Highway 101 that pretty much just say "Artificial Intelligence & Machine Learning" along with the name of some company, like its some sort of panacea.
It is surprising how often we take reproducibility for granted in the software world. With unit tests and continuous integration scenarios it becomes such a fundamental concept that it would be unheard of for developing products without them.
Then all of a sudden you have terabytes of data updating a model faster than you can keep up with it, and after a couple months, starting from scratch to get the same model doesn't seem plausible anymore. Two years of production time and you have a honking mess.
Also it is easy to fall into the trap of prototype-to-production, when you've developed a nice looking demo that suddenly has all of the real customers and none of the engineering rigor.
I agree - it's so challenging to keep up with Terabytes and Terabytes of moving data ... but reproducibility is so fundamental to science in general.
I'm not disagreeing, I'm just wondering if there's somehow a middle ground. What do you think? Can you share any experiences or tips?
I am running my own company at the moment, and my main dataset is about 2.5 TB, it's a very skinny table (it's actually a 191,000,000 x 6,000,000 matrix). It takes too long to iterate the whole dataset, so when I'm training my machine learning classifier I subsample the large dataset then compare previous model results to the new model, and inspect the differences.
If I have one piece of advice to give (that you probably already know), it is to have a sound way to measure results and ensure that you are improving. Usually that involves custom tooling that falls within the integration test layer. So if 'inspecting the differences' is a reproducible and statistically reliable way to know that you are good - then you probably are!
11 comments
[ 3.6 ms ] story [ 36.8 ms ] threadThere are lots of things wrong with "Machine Learning" practice that are problems of the practice instead of the subject itself.
One of the most dangerous things that comes out of academia is the "project orientation", that is, the idea that something is "done" at a certain point in time.
The "technical debt" aspect comes in when you are managing a product or a program, which can be thought of as a series of projects that influence each other.
The "paradigm" is that somebody wants to learn machine learning so they download the same set of digits that Hinton used in his papers, and then they replicate the result, and then they announce they are done. When you see everybody is chasing the same problems (Hinton's digits, the Kaggle of the day) you start to realize that the setup of a problem which can be solved by machine learning is actually worth more than the learning.
This is a fantastic point, but I'd like to also caution readers that they should spend some time understanding the elements of statistics and machine learning before jumping into theano or Weka tutorials. Lately, I've seen a lot of voodoo ML in the valley, starting with billboards on Highway 101 that pretty much just say "Artificial Intelligence & Machine Learning" along with the name of some company, like its some sort of panacea.
Models in production that are total black boxes. There's no way to deterministically recreate them.
But I treated it as an opportunity --- was able to teach some engineering to the data science guys and they taught me some data science :)
Then all of a sudden you have terabytes of data updating a model faster than you can keep up with it, and after a couple months, starting from scratch to get the same model doesn't seem plausible anymore. Two years of production time and you have a honking mess.
Also it is easy to fall into the trap of prototype-to-production, when you've developed a nice looking demo that suddenly has all of the real customers and none of the engineering rigor.
It is an interesting time to be in software :)
I'm not disagreeing, I'm just wondering if there's somehow a middle ground. What do you think? Can you share any experiences or tips?
I am running my own company at the moment, and my main dataset is about 2.5 TB, it's a very skinny table (it's actually a 191,000,000 x 6,000,000 matrix). It takes too long to iterate the whole dataset, so when I'm training my machine learning classifier I subsample the large dataset then compare previous model results to the new model, and inspect the differences.
"Hidden Technical Debt in Machine Learning Systems" D Sculley*, Google Research; Gary Holt, ; Daniel Golovin, Google, Inc.; Eugene Davydov, Google, Inc.; Todd Phillips, Google, Inc.; Dietmar Ebner, ; Vinay Chaudhary, Google, Inc.; Michael Young, Google, Inc.; Jean-Francois Crespo, Google, Inc.; Dan Dennison, Google, Inc.
(see https://nips.cc/Conferences/2015/AcceptedPapers )
Unfortunately, I haven't been able to find the paper yet.
https://news.ycombinator.com/item?id=10006293
https://news.ycombinator.com/item?id=8775772