High-Low Frequency Detectors (distill.pub)

The "Universality" section is especially interesting. They found these high-low frequency neurons in a number of different network architectures, and at around the same network depth in each case. I wonder how this insight could be useful. Can you train a network layer by layer and then connect the pre-trained layers? Like maybe train a "high-low frequency detector" layer on generated inputs of high-low frequency images, and then somehow put it 1/3 of the way into a network, maybe one whose other layers have been trained similarly to recognize other features.

sidenote: the journal uses github for peer review https://distill.pub/guide/