Optimistic guess: This information can be found in the paywall'ed article at [1]
> Dissecting our recurrent neural network model revealed strong inhibitory-to-inhibitory connections underlying a disinhibitory microcircuit as a critical component for long neuronal timescales and WM maintenance
If the article is worth the money, i would expect it to go into detail on it.
This is explained in the Nature article (summary on page 133) and shown as a diagrams in figure 6...
> In summary, these findings imply that robust inhibition of oppositely tuned inhibitory subpopulations is critical for memory maintenance in our RNN model. For example, a positive cue stimulus activates the inhibitory and excitatory subgroups selective for that stimulus and deactivates the negative stimulus subgroups. During the delay period, the inhibition strength between these two inhibitory subgroups dictates the stability of the cue-specific activity patterns generated during the stimulus window. The positive feedback provided by the similarly tuned excitatory neurons sustains the stimulus-specific activity of the inhibitory subgroups.
My intuitive understanding of these findings is that short term memory works by maintaining a consistent firing pattern in a particular region of the prefrontal cortex. By default a given firing pattern might activate other patterns because neurons tend to be a part of multiple potential pattern groups, and so left unchecked it could result in an unstable, random walk type of sequence. To stay in the attractor of one memory though, inhibitory neurons are used to turn off everything that isn't a part of the memory, thereby allowing it to remain in a more steady state.
To further clarify, I believe the authors are advocating a WM model where an excitatory subnet (activated by a Positive cue) sustains its firing (in the absence of that cue; the Delay period) by activating inhibitors of its own inhibitors. (these inhibitory subnets, like goldenshale mentions, also seem to inhibit excitatory subnets, diffusely).
> Kim and Sejnowski found that good working memory required both that long-timescale neurons be prevalent, and that connections between inhibitory neurons–which suppress brain activity–be strong. When they altered the strength of connections between these inhibitory neurons in their model, the researchers could change how well the model performed on the working memory test as well as the timescale of the pertinent neurons.
Neuroscience is so laughably primitive. Imagine saying you were close to understanding computers because you know which capacitors you really can’t cut out. It’s just absurd how little actual knowledge of the mind we have, as opposed to anatomical facts or behavioral generalizations.
I guess neuroscience mostly compares to the condensed matter / semiconductor field, in the computer analogy. They know everything about the transistors but guess at why they're connected like they are. And some in their respective fields will also be very interested in the high-level connections although maybe with less depth..
Unfortunately, in the neuroscience area, there is not yet any analogy to a computer programmer.
Its hard to comprehend how complex of a machinery the brain is when you are "just" a computer scientist. I often see people from the AI field eager to do a short rotation in a neuroscience lab to "learn how the brain works" and then go back to their company and develop a novel algorithm to make a paradigm shift in AI field - it's enthusiastic but laughably primitive.
The technology available to understand this complex machine is primitive when you compare it to electronics or AI tech. This is not because of the lack of funding and talent being invested in the field but because of the highly complex and efficient nature of the brain that has evolved over millions of years but only started to be investigated anatomically for less than a couple of centuries. Arguably, the same problem exists in the AI field too when it comes to understanding the black-box models. We don't even entirely comprehend how the artifical neural network based models work in many use-cases; there are ideas floating around to make sense of it which use the same strategy of "plug in/out and play" to understand what parameter affects what in your model which to neuroscientists seems (yet again) laughably primitive.
If you look at almost any complex phenomena that humans didn't actually create, you will see little actual knowledge. Neuroscience, much of biology and economics are examples.
Most areas of the brain are rather homogenous. They've done experiments rewiring inputs going to the visual cortex to the auditive cortex in rodents and they could see. Further, artificial neural nets demonstrate that extremely simple repetitive structures can implement a vast function space simply by training them by stochastic gradient descent. The most complex structures in the brain may be hardwired social behaviors, including facial expression and recognition, but these may not be necessary for the intelligence part as individuals strongly impaired in hardwired social behavior (autists) are sometimes capable of complex thought.
Being a mathematician who's dabbled in neuroscience a bit, I think that's essentially right (though perhaps I wouldn't have said "laughable"). But I also think this reflects the differences between how science and engineering progress -- our ability to exploit a natural phenomenon (or class of phenomena) often outstrips our ability to explain what's going on. As examples, I'd argue that for much of aviation history (and perhaps now too, I don't know), our ability to build flying machines far outstrips our understanding of how birds and insects (or even planes) fly.
Anyone can point out that autistic, schizophrenic or very old people lack certain inhibitions. But it's downright fascinating to think it may be explained by dulled inhibitory neurons or hypersensitive "regular" ones. That could be an easy fix!
The sense I intended was that it may indeed be an easy fix, but second- third- etc- order effects may actually make us all worse off in ways we never considered.
Looks like another paper on how everything in the brain can be tied to RNN or GPT2 or some other trendy ML model. 'Cure cancer with deep learning' style.
Before something like RNN can be used in a model of biological memory it has to respect the constraints of biological memory. Taking an arbitrary model and twisting the arm of data to fit it makes for some hard sale.
Claiming the findings somehow elucidate memory impairments in schizophrenia and autism (which are not even close to being understood), makes it even more look like bad science.
I agree with this sentiment. To be fair though, the authors did attempt to generate a biologically plausible RNN, which they published in a previous article...
I'm not fully sold on this model, but the article is fairly dense and I dont fully understand it. (Sejnowski, the senior author, was on my dissertation committee; he fell asleep during my defense lol)
Oh god, what he calls constraints are the "spiking
nature of biological neurons", then he happily proceeds, just like you'd guess, to create this abomination of 'spiking RNN' and train this pretty much plain vanilla RNN via backpropagation, completely disregarding the energy spent, learning mechanism, and actual biological plausibility or relevance of this horrendous architecture as memory model.
The first author knows just enough to be dangerous.
Dropping a link below in case you're interested - I have a side-project that takes a more literal approach to modeling biologically plausible neural networks (MCMC simulations inside 3D models)...
Here's the GitHub with their code for "Working Memory Spiking RNN Model" (This repository provides the code for the model and analyses presented in the paper):
https://github.com/rkim35/wmRNN
Glad to see more work in the 'stable memories' camp win out against the 'the brain is too stochastic and therefore unstable' theories. As a complement to this work on short-term memory, here's one of my favorite papers on long-term memories!
A model based on some core principles or constraints of the natural system will often solve the problem by discovering the same structure used by nature. And so facts of the model can reveal facts of the natural system. Of course such discoveries need to be validated against the real thing.
29 comments
[ 4.2 ms ] story [ 74.7 ms ] thread> Dissecting our recurrent neural network model revealed strong inhibitory-to-inhibitory connections underlying a disinhibitory microcircuit as a critical component for long neuronal timescales and WM maintenance
If the article is worth the money, i would expect it to go into detail on it.
[1] https://www.nature.com/articles/s41593-020-00753-w
> In summary, these findings imply that robust inhibition of oppositely tuned inhibitory subpopulations is critical for memory maintenance in our RNN model. For example, a positive cue stimulus activates the inhibitory and excitatory subgroups selective for that stimulus and deactivates the negative stimulus subgroups. During the delay period, the inhibition strength between these two inhibitory subgroups dictates the stability of the cue-specific activity patterns generated during the stimulus window. The positive feedback provided by the similarly tuned excitatory neurons sustains the stimulus-specific activity of the inhibitory subgroups.
model diagram: https://i.ibb.co/nrh1pVJ/figs.png
Neuroscience is so laughably primitive. Imagine saying you were close to understanding computers because you know which capacitors you really can’t cut out. It’s just absurd how little actual knowledge of the mind we have, as opposed to anatomical facts or behavioral generalizations.
You might also be interested in "Could a Neuroscientist Understand a Microprocessor?"
https://journals.plos.org/ploscompbiol/article?id=10.1371/jo...
It's a thoughtfully executed set of experiments that IMHO should be part of every computational neuroscientist's training.
Unfortunately, in the neuroscience area, there is not yet any analogy to a computer programmer.
Its hard to comprehend how complex of a machinery the brain is when you are "just" a computer scientist. I often see people from the AI field eager to do a short rotation in a neuroscience lab to "learn how the brain works" and then go back to their company and develop a novel algorithm to make a paradigm shift in AI field - it's enthusiastic but laughably primitive.
The technology available to understand this complex machine is primitive when you compare it to electronics or AI tech. This is not because of the lack of funding and talent being invested in the field but because of the highly complex and efficient nature of the brain that has evolved over millions of years but only started to be investigated anatomically for less than a couple of centuries. Arguably, the same problem exists in the AI field too when it comes to understanding the black-box models. We don't even entirely comprehend how the artifical neural network based models work in many use-cases; there are ideas floating around to make sense of it which use the same strategy of "plug in/out and play" to understand what parameter affects what in your model which to neuroscientists seems (yet again) laughably primitive.
Maybe both fields could stop flinging shit and realize that they are both missing a key piece of the puzzle...
Famous last words?
Looks like another paper on how everything in the brain can be tied to RNN or GPT2 or some other trendy ML model. 'Cure cancer with deep learning' style.
Before something like RNN can be used in a model of biological memory it has to respect the constraints of biological memory. Taking an arbitrary model and twisting the arm of data to fit it makes for some hard sale.
Claiming the findings somehow elucidate memory impairments in schizophrenia and autism (which are not even close to being understood), makes it even more look like bad science.
https://www.pnas.org/content/pnas/116/45/22811.full.pdf
I'm not fully sold on this model, but the article is fairly dense and I dont fully understand it. (Sejnowski, the senior author, was on my dissertation committee; he fell asleep during my defense lol)
The first author knows just enough to be dangerous.
>Sejnowski fell asleep
it surely looks like it
https://github.com/bradmonk/plasticity
https://science.sciencemag.org/content/365/6455/821.abstract
No, a model models, does not (and can not) reveal anything.