I agree with the premise of your post - (top) CS departments do offer machine learning / stats classes in later year of their programmes. I, myself, am taking probabilistic modelling and ML at the graduate level (although, still an undergrad).
What I disagree is that a main thing data scientists do is "work with regular expressions (not hard!), find, grep, sed, awk, and other Unix tools". In fact, if you've done any work on real datasets and have, say, 40GB of data, the last thing you want to be doing is writing regexps or playing with awk. Instead, you want to parallelize your pre-processing steps with Hadoop, or, perhaps, some other higher-level MapReduce framework.
Now that you have data this big, you also run into another problem - Matlab is no longer capable of loading your data into RAM. So this is where various stochastic sampling techniques come into play. And once you're done building your model, then you might want to use your full dataset. At this point, we again turn to Hadoop and, perhaps, Mahout.
To sum this up, the premise of your article is correct, but the advice you're giving is not, given the context. Sure, awk, grep and other Unix tools are good to know and learn, but won't help you much beyond working with small datasets. And when the dataset is small, you might just as well write your own Python parser and not worry too much about it ;)
1 comment
[ 3.1 ms ] story [ 9.6 ms ] threadI agree with the premise of your post - (top) CS departments do offer machine learning / stats classes in later year of their programmes. I, myself, am taking probabilistic modelling and ML at the graduate level (although, still an undergrad).
What I disagree is that a main thing data scientists do is "work with regular expressions (not hard!), find, grep, sed, awk, and other Unix tools". In fact, if you've done any work on real datasets and have, say, 40GB of data, the last thing you want to be doing is writing regexps or playing with awk. Instead, you want to parallelize your pre-processing steps with Hadoop, or, perhaps, some other higher-level MapReduce framework.
Now that you have data this big, you also run into another problem - Matlab is no longer capable of loading your data into RAM. So this is where various stochastic sampling techniques come into play. And once you're done building your model, then you might want to use your full dataset. At this point, we again turn to Hadoop and, perhaps, Mahout.
To sum this up, the premise of your article is correct, but the advice you're giving is not, given the context. Sure, awk, grep and other Unix tools are good to know and learn, but won't help you much beyond working with small datasets. And when the dataset is small, you might just as well write your own Python parser and not worry too much about it ;)