I asked this on Twitter, but maybe folks here can answer better: how important is nonlinearity for deep neural networks? This method's output seems to be a linear function of its (complex) input. Does that put important bounds on performance?
https://mobile.twitter.com/AndrewGYork/status/10228414045888...
Nonlinearity is very important and is the only reason why neural nets can approximate arbitrary functions. You can’t do that with linear transformations alone. Though from briefly skimming the paper they do seem to achieve similar effects through phase modulation. Otherwise even MNIST would be out of the question.
Echoing the other respondents–if you don't have a nonlinearity, your whole network is just a sequence of linear transforms, which (multiplied out) is the same as a single linear transform. Meaning that removing the nonlinearities gives you (effectively) a one-layer network.
Though the other comments are correct, I want to point out that you can get some nontrivial behavior with only linear functions. For example, low-rank matrix factorization is kind of like a neural network
f(x) = U * V * x,
where U is an n by k matrix and V is a k by m matrix, where k is much smaller than n and m. Basically, we are constraining the set of allowed linear transformations, which is a form of regularization. Convolutional layers in neural networks similarly restrict the allowed linear transformations.
Nevertheless, the power of linear neural networks is far less than that off nonlinear networks.
Mind boggled that this article comes up now. Been working on similar tech recently, and the question of non-linearality arose right away. The discussed conclusion was "impossible". Yet I was able to design a crude NAND gate. So there has to be non-linearality it the quantum nature of diffraction and interference.
Can someone help me Grok how the 3D printed Neural Networks back propagate? As I am trying to go through the paper they describe it as a pure optical approach, but what adjusts the refraction elements during the learning process?
Can't access the article, but the abstract reads like this is inference only. If that's so, this still could be useful. For instance, you could train a network somewhere else, then implement the learned function optically.
"[...] learnable network parameter that is iteratively adjusted during the training process of the diffractive network, using an error back-propagation method. After this numerical training phase implemented in a computer, the D^2NN design is fixed and the transmission/reflection coefficients of the neurons of all the layers are determined. This D^2NN design, once physically fabricated using e.g., 3D-printing, 3lithography, etc., can then perform, at the speed of light propagation, the specific task that it is trained for, using only optical diffraction and passive optical components/layers, creating an efficient and fast way of implementing machine learning tasks."
This is really important if it can be scaled up. Imagine being able to replace an onsite heavy compute core pulling several hundred watts (like in the trunk of a Waymo car) with a passive unpowered optical element. Pretty amazing.
The title doesn't do the article justice. This is a physical, 3D-printed neural network, in form of plates that pass/reflect light. You shine your input at one end, get results at the other.
I'm very impressed. Not sure if this has any chance of being more efficient than traditional NNs implemented in silicon, but I can imagine some fun applications. For instance, with some optics in front, I think it could be used as a passive classifier of what's in front of the detector - you could set up an array of photodetectors in the back, that operate a low-power device only when appropriate pattern is detected.
I'd just like to point out that back in the 60s there was this physical neural network called the Perceptron. Yeah, like the ones you learn about. However, it was a patchpanel machine (the wires were the neurons) with weights that were implemented as potentiometer knowbs attached to little servos. It could do backprop (IIRC it was a single layer) and basic image recognition.
Some days I feel like neural network hardware is the new laser: at one point, nobody thought it could exist, but once one was made, new designs started to fall out of the woodwork. Like gravitational lenses, there are actually "galactic laser foundries" that generate lasers purely out of stellar physics.
25 comments
[ 3.2 ms ] story [ 51.8 ms ] threadBut yeah, in NN nonlinearities are very important, otherwise they would be simplifiable to a single transformation
f(x) = U * V * x,
where U is an n by k matrix and V is a k by m matrix, where k is much smaller than n and m. Basically, we are constraining the set of allowed linear transformations, which is a form of regularization. Convolutional layers in neural networks similarly restrict the allowed linear transformations.
Nevertheless, the power of linear neural networks is far less than that off nonlinear networks.
"[...] learnable network parameter that is iteratively adjusted during the training process of the diffractive network, using an error back-propagation method. After this numerical training phase implemented in a computer, the D^2NN design is fixed and the transmission/reflection coefficients of the neurons of all the layers are determined. This D^2NN design, once physically fabricated using e.g., 3D-printing, 3lithography, etc., can then perform, at the speed of light propagation, the specific task that it is trained for, using only optical diffraction and passive optical components/layers, creating an efficient and fast way of implementing machine learning tasks."
https://www.osa.org/en-us/about_osa/newsroom/news_releases/2...
They implemented a back propagation algorithm using just optical.
I'm very impressed. Not sure if this has any chance of being more efficient than traditional NNs implemented in silicon, but I can imagine some fun applications. For instance, with some optics in front, I think it could be used as a passive classifier of what's in front of the detector - you could set up an array of photodetectors in the back, that operate a low-power device only when appropriate pattern is detected.
Some days I feel like neural network hardware is the new laser: at one point, nobody thought it could exist, but once one was made, new designs started to fall out of the woodwork. Like gravitational lenses, there are actually "galactic laser foundries" that generate lasers purely out of stellar physics.