uGMM-NN is a novel neural architecture that embeds probabilistic reasoning directly into the computational units of deep networks. Unlike traditional neurons, which apply weighted sums followed by fixed nonlinearities, each uGMM-NN node parameterizes its activations as a univariate Gaussian mixture, with learnable means, variances, and mixing coefficients.
Upshot: Gaussian sampling along the parameters of nodes rather than a fixed number. This might offer one of the following:
* Better inference time accuracy on average
* Faster convergence during training
It probably costs additional inference and training compute.
The paper demonstrates worse results on MNIST, and shows the architecture is more than capable of dealing with the Iris test (which I hadn’t heard of; categorizing types of irises, I presume the flower, but maybe the eye?)
The paper claims to keep the number of parameters and depth the same, but it doesn’t report as to
* training time/flops (probably more I’d guess?)
* inference time/flops (almost certainly more)
Intuitively if you’ve got a mean, variance and mix coefficient, then you have triple the data space per parameter — no word as to whether the networks were normalized as to total data taken by the NN or just the number of “parameters”.
Upshot - I don’t think this paper demonstrates any sort of benefit here or elucidates the tradeoffs.
Quick reminder, negative results are good, too. I’d almost rather see the paper framed that way.
Thank you for your work! I would be interested to see what this means to a CNN architecture. Maybe it wouldn't actually be needed to have the whole architecture based on uGMM-NNs but only the last layers?
It's not clear from the formulas how x=[x1,...,xN] relates to y, μ, and σ since these are defined without x. Assuming y = Wx + b, and μ, σ, and π are learnable parameters for each output dimension. The symbol π seems to mean both weight and the constant 3.14159 in the same formula.
Overall it looks similar to radial basis activations, but the activations look to be log of weighted "stochastic" sums (weights sum to one) of a set of radial basis functions.
Thanks for the comment - uGMM neurons are not just "RBFs with log outputs". Each neuron is a mixture of Gaussians with trainable means, variances, and mixture weights, encoding uncertainty and multimodality that propagates through the network. The log-likelihood output is simply a consequence of this probabilistic formulation, not an innovation.
7 comments
[ 3.5 ms ] story [ 28.0 ms ] threadUpshot: Gaussian sampling along the parameters of nodes rather than a fixed number. This might offer one of the following:
* Better inference time accuracy on average
* Faster convergence during training
It probably costs additional inference and training compute.
The paper demonstrates worse results on MNIST, and shows the architecture is more than capable of dealing with the Iris test (which I hadn’t heard of; categorizing types of irises, I presume the flower, but maybe the eye?)
The paper claims to keep the number of parameters and depth the same, but it doesn’t report as to
* training time/flops (probably more I’d guess?)
* inference time/flops (almost certainly more)
Intuitively if you’ve got a mean, variance and mix coefficient, then you have triple the data space per parameter — no word as to whether the networks were normalized as to total data taken by the NN or just the number of “parameters”.
Upshot - I don’t think this paper demonstrates any sort of benefit here or elucidates the tradeoffs.
Quick reminder, negative results are good, too. I’d almost rather see the paper framed that way.
They state the output of a neuron j is a log density P_j(y), where y is a latent variable.
But how does the output from the previous layer, x, come into play?
I guess I was expecting some kind of conditional probabilities, ie the output is P_j given x or something.
Again, perhaps trivial. Just struggling to figure out how it works in practice.
Overall it looks similar to radial basis activations, but the activations look to be log of weighted "stochastic" sums (weights sum to one) of a set of radial basis functions.
The biggest difference is probably log outputs.