It's scary to think that while we have the Transformer and it seems extremely complex, things like this throw it all away and give us a glimpse into what the neural (or even non-neural) networks of the future will be like. Perhaps even just simulating full biological systems as well as a brain-like system.
While ANNs are rather trained for unidirectional propagation, action potential propagation in biological neurons is symmetric e.g. ”it is not uncommon for axonal propagation of action potentials to happen in both directions” ( https://journals.aps.org/pre/abstract/10.1103/PhysRevE.92.03... ).
Also, while current ANNs use guessed parametrizations, objectively available is joint distribution - biological neuron should be evolutionarily optimized to exploit, and it is relatively simple in approach from this arXiv.
Such joint distribution neurons bring additional training approaches - maybe some of them are used by biological neural networks?
However, it degenerates to ~KAN if restring to pairwise dependencies (can consciously add triplewise and higher), and gives many new possibilities, like multidirectional propagation, of values or probability distributions, with novel additional training approaches like through tensor decomposition.
(Multidirectional) biological neural networks are no longer superior in MNIST benchmark ... but e.g. consciousness, or being able to learn from single examples.
And no, recreating it is not a task a single person can complete.
Just represent joint density for each neuron as a linear combination - then you can inexpensively propagate in both directions e.g. as E[X|Y,Z] or E[Y,Z|X] by substituting and normalizing ... the formulas turn out quite simple - could be hidden in dynamics of (bidirectional) biological NN ...
And for pairwise distribution becomes ~KAN, which turned out quit successful ... so we are talking about its extension: adding more possibilities, like triplewise dependencies and multidirectional propagation.
First, this work needs work. The English needs to be improved before I would recommend wading into the contents deeply.
Second this assertion or citation is wrong:
>for biological neurons e.g. "it is not uncommon for axonal propagation of action potentials to happen in both directions" - suggesting they are optimized to continuously operate in multidirectional way.
What is true is that dendritic spikes can propagate bidirectionally in some neurons (but can also fade or be blocked).
What we often forget is that spikes are a kludge to enable faster INTRAcellular communication (not needed in retinal processing).
The classic action potential connects the axon hillock (the spike initiation zone) to a variable subset of responsive presynaptic sites that may or may not release neurotransmitters that may or may not modulate behaviors of neighboring processes and cells.
Sure biological NN are much more complicated, but basically action propagation can travel in both directions, and evolution should optimize for that.
In contrast, current ANNs are focused on unidirectional propagation, and are much worse at training from single samples - to reach abilities of biological, maybe it is worth to start thinking about multidirectional?
Neurons containing joint distribution model can do propagate conditional distributions in various direction, and it is not that difficult to represent - maybe something like that could be hidden in biological (?)
14 comments
[ 4.2 ms ] story [ 36.9 ms ] threadAlso, while current ANNs use guessed parametrizations, objectively available is joint distribution - biological neuron should be evolutionarily optimized to exploit, and it is relatively simple in approach from this arXiv.
Such joint distribution neurons bring additional training approaches - maybe some of them are used by biological neural networks?
Doesn't look like it.
However, it degenerates to ~KAN if restring to pairwise dependencies (can consciously add triplewise and higher), and gives many new possibilities, like multidirectional propagation, of values or probability distributions, with novel additional training approaches like through tensor decomposition.
There are a lot of ideas that are clever and seem promising... but fail to perform well on such benchmarks.
Is there a github repo with code available?
Multidirectional are biological neurons, but I don't know how to compare with them?
---
To be 100% clear: My question about practical application today is orthogonal to the question about whether this research is worth pursuing!
And no, recreating it is not a task a single person can complete.
And for pairwise distribution becomes ~KAN, which turned out quit successful ... so we are talking about its extension: adding more possibilities, like triplewise dependencies and multidirectional propagation.
>for biological neurons e.g. "it is not uncommon for axonal propagation of action potentials to happen in both directions" - suggesting they are optimized to continuously operate in multidirectional way.
What is true is that dendritic spikes can propagate bidirectionally in some neurons (but can also fade or be blocked).
What we often forget is that spikes are a kludge to enable faster INTRAcellular communication (not needed in retinal processing).
The classic action potential connects the axon hillock (the spike initiation zone) to a variable subset of responsive presynaptic sites that may or may not release neurotransmitters that may or may not modulate behaviors of neighboring processes and cells.
In contrast, current ANNs are focused on unidirectional propagation, and are much worse at training from single samples - to reach abilities of biological, maybe it is worth to start thinking about multidirectional?
Neurons containing joint distribution model can do propagate conditional distributions in various direction, and it is not that difficult to represent - maybe something like that could be hidden in biological (?)