Dynamic Automatic Differentiation of GPU Broadcast Kernels (arxiv.org)

One of the author here. Working on this paper was fun, one of the most interesting lessons I learned was that divergence on GPUs has dramatically fallen in cost on modern architectures.