I think you are overegenralizing applicability of Neural Architecture Search etc. and cherry picking individual examples. There is an enormous gap between what gets published in academia with what’s actually useful.
E.g. Compute wars have only intensified with TPUs and FPGA. sure for training you might be okay with few 1080ti but good luck building any reliable, cheap and low latency service that uses DNNs. Similarly big data for academia is few terabytes but real Big data is Petabytes of street level imagery, Videos/Audio etc.
Your last comment reminded me of this article [1] on "Google Maps's Moat", which discusses the vast resources that Google has poured into collecting data at a global scale to make Google Maps what it is.
The authors were able to outperform Google ML by a large margin for a vision task that involved recognizing numbers from car registration documents. With just 160 manually collected training samples they were able to train a neural net that could recognize characters with 99.7 % accuracy. GoogleML performed very poorly in comparison, which I found very surprising because it didn't seem to be such a hard recognition task (clean, machine-written characters on a structured, green background).
Isn't that a bit like a synthetic benchmark though? This reads to me like those mongodb vs mysql comparisons that were made 7 years ago where they compared object store efficiencies for the two.
5 comments
[ 2.9 ms ] story [ 18.6 ms ] threadE.g. Compute wars have only intensified with TPUs and FPGA. sure for training you might be okay with few 1080ti but good luck building any reliable, cheap and low latency service that uses DNNs. Similarly big data for academia is few terabytes but real Big data is Petabytes of street level imagery, Videos/Audio etc.
[1] https://www.justinobeirne.com/google-maps-moat/
https://www.slideshare.net/FlorianWilhelm2/performance-evalu...
The authors were able to outperform Google ML by a large margin for a vision task that involved recognizing numbers from car registration documents. With just 160 manually collected training samples they were able to train a neural net that could recognize characters with 99.7 % accuracy. GoogleML performed very poorly in comparison, which I found very surprising because it didn't seem to be such a hard recognition task (clean, machine-written characters on a structured, green background).
You can write a lot of papers about Penn Treebank data but I can't imagine anything you do with Penn Treebank will be commercially useful.