L2 Regularization and Batch Norm (blog.janestreet.com)

Jane Street is noted for its use of OCaml, so it's interesting to see that their researchers do indeed use Python (judging from the code in that post, at least).

Excellent article! I'd be really curious to see a treatment of the same topic, but with ADAM.

David Wu wrote KatoGo which is doing new and exciting stuff in Go Ai. He's writing a ton of code and performing even more experiments.

I wasn't familiar with batch normalization before, but I've had to do something similar before in Stan to enforce that some model parameters (not data) were exactly mean of 0 and standard deviation of 1.

Thanks for taking efforts to explain this — more people should be aware of this interesting effect. I agree with all you have said except for the conclusion part. BN without L2 is not underfitting. In my experience it is overfitting due to small effective learning rate. It is easy to verify — just compare the gap between training and test losses / errors. So my conclusion from the derivation is, L2 penalty in BN acts as a regularizer in a different way — by increasing the effective learning rate of the weights. On a related note, this effect could present even without normalization. By adding just a scalar multiplier parameter in the branch, the weight’s scale could be more or less decoupled from its direction. For reference, I will make some shameless self-promotion here about our recent work on training residual networks without normalization: http://arxiv.org/abs/1901.09321

Wow I didn't even know janestreet even posts these kind of articles. i always assumed they were super secretive

Myrtle has a similar series that includes discussion about batch normalisation: https://myrtle.ai/learn/how-to-train-your-resnet/

https://twitter.com/dcpage3/status/1141700299071066112

Disclaimer: I work at Myrtle!