I'm pleasantly surprised by how well it works! No offense to Barret Zoph, Quoc Le et al but the "let's throw 10000 GPUs and TPUs at the problem" approach was frankly absurd.
And a huge +1 for the inclusion of random architectures in the results.
I'm not the original commenter, but I think it's fair to say that throwing compute at the problem is absurd from an academic point of view (where the goal is understanding), but not from a commercial point of view. Although there's still some academic value in throwing compute at a problem, just to see what's possible.
Can you link the paper of Zoph,Le that did the “1000 GPUs” thing? Thanks! I tried searching for it but apparently there was quite a few papers on parameter search.
Not GP, but you are probably looking for the "Neural Architecture Search" series [1] [2] [3]. First one uses something like 1k GPUs for a month, next one is a bit more reasonable, and the last one actually has a reasonable training time, and has comparable performance to DARTS (see ENAS in comparison tables).
Cool, was thinking about this subject earlier today.
Abstract:
>This paper addresses the scalability challenge of architecture search by formulating
the task in a differentiable manner. Unlike conventional approaches of applying evolution
or reinforcement learning over a discrete and non-differentiable search space,
our method is based on the continuous relaxation of the architecture representation,
allowing efficient search of the architecture using gradient descent. Extensive experiments
on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our
algorithm excels in discovering high-performance convolutional architectures for
image classification and recurrent architectures for language modeling, while being
orders of magnitude faster than state-of-the-art non-differentiable techniques.
Not to get existential here, but the analogy to human neural networks is compelling. The algorithm efficiently redesigns itself to fit the data, just like neural synapses reconnect one another as knowledge is learned, to optimize the storage and retrieval of the information.
8 comments
[ 6.2 ms ] story [ 71.2 ms ] threadAnd a huge +1 for the inclusion of random architectures in the results.
1: https://arxiv.org/abs/1611.01578 2: https://arxiv.org/abs/1707.07012 3: https://arxiv.org/abs/1802.03268
Abstract:
>This paper addresses the scalability challenge of architecture search by formulating the task in a differentiable manner. Unlike conventional approaches of applying evolution or reinforcement learning over a discrete and non-differentiable search space, our method is based on the continuous relaxation of the architecture representation, allowing efficient search of the architecture using gradient descent. Extensive experiments on CIFAR-10, ImageNet, Penn Treebank and WikiText-2 show that our algorithm excels in discovering high-performance convolutional architectures for image classification and recurrent architectures for language modeling, while being orders of magnitude faster than state-of-the-art non-differentiable techniques.
Zoph's paper on architecture search: https://arxiv.org/abs/1611.01578
Concept of architecture search from ages ago: https://static1.squarespace.com/static/58e2a71bf7e0ab3ba886c...