If I understand correctly that's sort of exactly what this is. The geometry of the diffraction gratings encodes a forward propagation model trained as classifier of the number "2".
I don't quite understand the mathematics of how it was trained, but they were able to discretize the geometry of those layers somehow into little 0.4mm pixels of "trainable diffractive neurons" and they simulated light transmission through the layers to compute a loss function.
I'm really surprised that this was computationally feasible. Simulation of light through the gratings must have been cheap enough as a function evaluation to train the network.
I would imagine that you generate the desired transform function of a diffractive structure rather than the structure itself, because the structure is ultimately derived from the transform function. Since the transform function is basically a 2D fourier transform and a spatial frequency/phase plot, it's not _that_ computationally costly. Once you settle on functions you like, you then generate and or simulate a diffractive structure and see if it behaves how you expect.
Sort of, I didn't dive too far into the math, but it looks like each diffractive structure is akin a layer of a neural net, which is tuned for a set of spatial frequencies and phases, which when combined (like layers of a neural net) to form recognition of more complex objects.
There are a few gotchas in that statement though - for one, I didn't dive too far into the math, and I would assume that the convolutional algorithms as well as the underlying matrix functions may be different. But at the end of the day, you're approximating a complex function using an array of simple functions with different weights and scale factors.
The other gotcha is that diffractive structures use monochromatic light, so it's probably not too useful in most normal situations with normal light sources.
It can do the same sort of computation work any stack of analogue filters can do: it does one thing very fast and if you want something else done you must create those filters first and the frame holding the stack is of no help at all.
> a camera that can only take pictures of the number two, no other numbers.
Well, to be precise it makes a complete (deterministic) mess of any other numbers. But given the output and the filters you can probably unfold the camera "psf" and get back whatever it was it saw.
>Importantly, this diffractive camera is not based on a standard point-spread function-based pixel-to-pixel mapping between the input and output FOVs, and therefore, it does not automatically result in signals within the output FOV for the transmitting input pixels that statistically overlap with the objects from the target data class. For example, the handwritten digits ‘3’ and ‘8’ in Fig. 2c were completely erased at the output FOV, regardless of the considerable amount of common (transmitting) pixels that they statistically share with the handwritten digit ‘2’. Instead of developing a spatially-invariant point-spread function, our designed diffractive camera statistically learned the characteristic optical modes possessed by different training examples, to converge as an optical mode filter, where the main modes that represent the target class of objects can pass through with minimum distortion of their relative phase and amplitude profiles, whereas the spatial information carried by the characteristic optical modes of the other data classes were scattered out.
It seems like that may not be so possible.
Later on in the article:
>It is important to emphasize that the presented diffractive camera system does not possess a traditional, spatially-invariant point-spread function. A trained diffractive camera system performs a learned, complex-valued linear transformation between the input and output fields that statistically represents the coherent imaging of the input objects from the target data class.
Note here that the learned transformations are linear, and the Fourier Transform is linear, but you cannot invert from output to input because the sensor measures real-valued intensities of complex-valued fields. All the phase information is lost.
There are point spread functions, but they vary, in an extreme way, across the "image plane". The diffraction patterns scatter the coherent light. Since the light is coherent, the output image brightness is the result of the number of photons, and the interference from all their waves coming from the patterns. This can't be reversed because the phase is a degree of freedom that means, even though we know how any ray of light will pass through, the image that made it is undetermined?
I assume these diffraction patterns, and the "film" have to be placed with sub-wavelength accuracy?
If you exposed this with holographic film, what would you see? Maybe a bright "2" protruding from a rough, bumpy, background? Surely it's 3d, that falls into chaos right behind, right?
Thinking of the underlying physics, I think all this means that their filters do funky things to the polarisation of the light to get cancellations, and since the detector isn't made to measure polarisation the process isn't invertible.
Looks like they made a version that scatters images of everything except trousers! Unfortunately, they hid the results at the bottom of the supplementary pdf [1].
19 comments
[ 3.9 ms ] story [ 63.8 ms ] threadThe outcome, though, is mind-boggling: a camera that can only take pictures of the number two, no other numbers. Totally magical!
Our definitions of "not that weird" are very different.
It blew my mind when I did in school, yet... there was the proof that it worked!
You can use analog electronics to achieve a similar scrambler off volumetric CCD or CMOS sensors...
I don't quite understand the mathematics of how it was trained, but they were able to discretize the geometry of those layers somehow into little 0.4mm pixels of "trainable diffractive neurons" and they simulated light transmission through the layers to compute a loss function.
I'm really surprised that this was computationally feasible. Simulation of light through the gratings must have been cheap enough as a function evaluation to train the network.
There are a few gotchas in that statement though - for one, I didn't dive too far into the math, and I would assume that the convolutional algorithms as well as the underlying matrix functions may be different. But at the end of the day, you're approximating a complex function using an array of simple functions with different weights and scale factors. The other gotcha is that diffractive structures use monochromatic light, so it's probably not too useful in most normal situations with normal light sources.
Well, to be precise it makes a complete (deterministic) mess of any other numbers. But given the output and the filters you can probably unfold the camera "psf" and get back whatever it was it saw.
>Importantly, this diffractive camera is not based on a standard point-spread function-based pixel-to-pixel mapping between the input and output FOVs, and therefore, it does not automatically result in signals within the output FOV for the transmitting input pixels that statistically overlap with the objects from the target data class. For example, the handwritten digits ‘3’ and ‘8’ in Fig. 2c were completely erased at the output FOV, regardless of the considerable amount of common (transmitting) pixels that they statistically share with the handwritten digit ‘2’. Instead of developing a spatially-invariant point-spread function, our designed diffractive camera statistically learned the characteristic optical modes possessed by different training examples, to converge as an optical mode filter, where the main modes that represent the target class of objects can pass through with minimum distortion of their relative phase and amplitude profiles, whereas the spatial information carried by the characteristic optical modes of the other data classes were scattered out.
It seems like that may not be so possible.
Later on in the article:
>It is important to emphasize that the presented diffractive camera system does not possess a traditional, spatially-invariant point-spread function. A trained diffractive camera system performs a learned, complex-valued linear transformation between the input and output fields that statistically represents the coherent imaging of the input objects from the target data class.
Note here that the learned transformations are linear, and the Fourier Transform is linear, but you cannot invert from output to input because the sensor measures real-valued intensities of complex-valued fields. All the phase information is lost.
As a layperson, do I understand this correctly?
There are point spread functions, but they vary, in an extreme way, across the "image plane". The diffraction patterns scatter the coherent light. Since the light is coherent, the output image brightness is the result of the number of photons, and the interference from all their waves coming from the patterns. This can't be reversed because the phase is a degree of freedom that means, even though we know how any ray of light will pass through, the image that made it is undetermined?
I assume these diffraction patterns, and the "film" have to be placed with sub-wavelength accuracy?
If you exposed this with holographic film, what would you see? Maybe a bright "2" protruding from a rough, bumpy, background? Surely it's 3d, that falls into chaos right behind, right?
And the fact that the paper is available for free is just added gravy!
[1] https://static-content.springer.com/esm/art%3A10.1186%2Fs435...