HN Explorer

6 comments

[ 3.7 ms ] story [ 20.1 ms ] thread
Hawkenfall
A more in-depth paper about this found the Swish activation often outperformed other functions: https://arxiv.org/abs/1710.05941
rickdeveloper
I wasn’t aware of that one. Definitely interesting, thanks for sharing!
excessive
That's interesting. I didn't read the paper closely, but skipping to the pictures, it looks like ReLU, but smoothed out so the derivative is continuous. Intuitively, that seems useful.
albertzeyer
These are small differences, and this is on MNIST (very small toy dataset). This is likely just noise. How big is the variance when each experiment is tried with different random seeds? And more interestingly, how about more difficult problems? E.g. try out on some real world tasks, like e.g. speech recognition (e.g. Librispeech). I don't think you can draw any conclusion from these current results.