I’m in no way an expert on this stuff, but I’ve long wondered if AI is really a hardware problem. That is, the way that conventional computers work is nothing like the way brains work, and if we’re trying to model brain-like intelligence, perhaps we need brain-like hardware to do it. It’s fascinating stumbling across more and more articles describing neuromorphic hardware like this recently. I guess this idea has occurred to a lot of other people.
We are trying everything I guess and the reason it’s happening in silicon is because thats whats available mainstream and it’s possible the wrong road. Regardless, at this point Im sure we’ll get there sooner or later. But there are serious questions like what will we do with it or will we, as humans, become completely irrelevant at that point? Lots of questions worth pondering about
Perhaps progress requires a move away from human-like brains. There are infinitely many kinds of potential intelligence. It's very unlikely ours is the pinnacle.
I know of almost no human-like brain inspiration in current AI models. Current ANNs are similar to biological brains, but only in the sense that they can be understood as networks of threshold units. One can also understand most modern ANN models purely in terms of linear algebra. They are super-abstract, super-generic models of computation, and biological brains happen to fit this category (at a very high level of simplification and abstraction).
Then, gradient descent is a centralized and supervised learning algorithm that has nothing to do with all the decentralized and emergent processes that take place in biological brains. For example, consider all the complex interactions of the various neurotransmitters and neuroreceptors.
Biological brains (let alone human) are actual physical realizations of networks with very complex and specific topologies. New connections grow and are formed in a very real sense, and this process interacts with the environment in a variety of ways, plus it is spatially embedded.
Human brains (neural networks) are little like networks of threshold units, as you observe. Neurons can process time-dependent integrations and differentiations with dynamic thresholds. Not the tiny static networks that AI uses. More like an analogue computer but with more types of computing elements.
I am also in no way an expert, but I would not say it is "really a hardware problem". There is already serious research going on neuromorphic computing, but you have to know what hardware to build and how to actually use / train / evolve it. We are trying to imitate our brains and each single neuron has very rich dynamics. Then you have a general graph with 80 billions of them, and this network also interacts with and is regulated by another network of chemical reactions... One could maybe give up understanding and build a fairly generalized evolutionary process, but that hardware could end up needing a degree of flexibility that would make it either closer to a biological brain grown in a lab or something much closer to a conventional computer...
Depends on the values of "enough". Organic chemistry comes to mind - we can tell what proteins are floating around your body, but we can't tell how exactly they're interacting, as it's computationally intractable. Perhaps that qualifies as "not enough knowledge", but it seems to me that this is a kind of problem where you'll only have enough knowledge once you can do a molecular-scale simulation, and there isn't much to simplify there.
It's actually the reverse. The hardware we have is way more efficient than our brains. One TPU can simulate billions of neurons a second in a tiny area. Computer hardware has a significant time advantage. Electrical impulses in the brain travel very slowly.
The problem is we don't understand how the brain is architected at anything other than a very coarse level. That architecture is the product of millions of years of evolution so we have some catching up to do.
Except that the TPU does not simulate real neurons in any way. It just performs simple arithmetic using an architecture that is suited to the ANNs we use today. If you managed to understand how the brain works and tried to run the model on a cluster of TPUs, you would need way more energy to simulate one second of brain activity than the 20J a real brain uses in a second. Only by some magical coincidence would any of the hardware we use today turn out to be efficient.
Are neurons binary in the way they operate, in that they either fire or they don't? Or are they analog in nature, where a signal can be either strong, weak, nonexistent or anywhere in between?
For starters, they use a pulse-frequency based signalling system.
And more generally, it’s pretty clear that there is a mixture of context-sensitive inhibitory and excitatory signals both deriving from the dendrites (inputs) and from the chemical stew of hormones and so forth they find themselves immersed in.
There are neurobiologists who claim (controversially, I concede) that a single neurone is a fully self-contained ‘processor’[1], and not the equivalent of single logic gate.
Those who somehow confuse our “neural network architectures” with the actual biology in our skulls are really misguided. Based on the evidence, one can at best opine that a TPU-based matrix add/multiply unit is approximating a very clunky and un-nuanced neural network. To assert that they are one and the same is... indefensible, when one goes beyond even the coarsest detail. A neural network as we implement it technologically is nothing but a weighted network of threshold units. That is all.
[1] To clarify: some kind of highly context-aware DSP with both long- and short-term storage facilities that allow it to react not only to the inputs that it is being fed, but also to inputs it was fed previously (and possibly outputs it produced previously as well).
I do not exempt those who make such statements just because they have authority. If anything, they should know better.
I started off along much the same path (I bought and read Neural Networks: A Comprehensive Foundation (1994) by Simon Haykin) when I was in high school, read it all and thought I knew everything there was to know on the topic, and then started to read about neurobiology and was stunned to discover I was living a lie. Then I read The Computational Beauty of Nature (1998) by Gary William Flake and decided I was going to dedicate myself to understanding the complexities we brush under the rugs.
I think the Hacker News/Silicon Valley/Machine Learning/Developer crowd really need to look beyond the virtuosismo of their implementations. It’s a very introverted and self-referential mindset.
To add on to the child comment, neurons seem to fire either as a one off pulse or at a fixed frequency for a duration of time known as a burst. There is no explicit 'input layer' or 'output layer' and signal propagation takes time so there is nothing stopping a self sustaining loop signal from forming. These are just two examples of effects which are not captured in the artificial neuron model used in ML.
Switching speed is very fast with silicon, but we seem to be nowhere near the computing power of the brain, which incidentally runs on an absurdly low 20W. I think this power usage aspect is quite underappreciated.
The authors inject artificial noise in cycles to their Spiking Neural Networks. They call this noise injection as Slow Wave Cycles equivalent to what brain generates during sleep. I do not know what exactly goes during SW sleep in mammalian brain so it's hard to judge if this analogy by the authors makes sense.
However, noise injection on its own has been extensively shown to decorrelate signal from noise so that the networks don't fixate on noise in the input data while training. This is widely used when using CNNs on image data and is known to increase performance of network predictions. So I am not sure how much different this article's findings are from this.
No link to arxiv publication makes it harder to make sense of these findings.
Edit: "Sleep" is not relevant for most ANN architectures used today. Synaptic pruning however is, similar to dropout.
Cool, yes we also add similar random (Gaussian) noise to image data before the networks see it. The decorrelation theory comes originally form signal processing field like your example shows. Old wisdom in new bottle.
It's fascinating to think that my dreams might just be "noise injection". And this is why we do science folks - an explanation for sleep that came from trying to teach silicon to recognise cats. Unabticipated benefits. That's what we like.
I have always felt that there should exist some analogy of the heat engines (and Carnot cycle) in the information theory. The heat engine increases the entropy between two heat reservoirs, and uses it to do work, decreasing entropy elsewhere.
If the purpose of intelligence is to "convert data to information", then perhaps it is akin to doing some work that decreases information theoretic entropy?
For example, in regression, we express input data as parameters of the model and error, so in the heat engine analogy, the input is the incoming heat, the parameters are the work output, and the error is the waste heat. (The error has higher entropy because there is less to explain, and you typically throw it away.)
So perhaps, if somebody would figure out this analogy correctly, we would see that we always need some cycles to continue the operation, just like in heat engines. So then we could conclude that any agent might require some kind of cycles of information creation and destruction in order to have intelligence.
The best definition of intelligence I have ever come across (cannot find the reference sadly, maybe I’m paraphrasing here) is “that which you do to keep your options open when you have no experience or training or prior information regarding the circumstances you are currently in”. This will manifest, paradoxically, as a ‘force’.
Thought experiment in order to clarify: how would aliens, staring at our planet for aeons through a telescope, reach the conclusion that there is or is not intelligent life here on Earth? Well... they could observe that on average, historically, about one every hundred million years or so there is a major asteroid impact. They would eventually observe that from some point onwards a larger and larger delay would set in from the last impact and the next, far longer than expected. If they zoomed in they’d notice that somehow asteroids on impact trajectories would be deflected as if by a physical force emanating from the planet.
This would be our descendants, using the amazing technology at their disposal, deflecting asteroids to ensure that the last major extinction level event will be the one that killed off the dinosaurs 65 million years ago (and counting). They’d be using their intelligence to keep their options open and avoid being wiped out. They’re exerting a physical force on their environment and clearing the sidereal neighbourhood of dangerous rocks and comets.
Conclusion: intelligence is about perception of data but for the sake of control of the environment. “Data into information”, honestly, is too close to being a truism to be of any utility.
My working definition is that intelligence is about accurate modelling of the environment, both internal and external.
This includes the ability to make accurate predictions about threats and opportunities within the environment, and the likely outcomes of planned actions.
The more intelligent an entity is, the more successfully it can generalise from past experience and anticipate and influence future events.
This is a completely different definition to plain old IQ test intelligence, which seems to be closer to raw mental agility. I suspect you can score high on mental agility - e.g. figure and number series manipulation - and still have poor applied intelligence. The latter needs the ability to synthesise a range of diverse stimuli, which is not the same as being able to manipulate abstracted symbols with no surrounding context.
Our definitions seem entirely compatible: a more accurate and flexible model will enable greater capacity to face circumstances. A deer perceives risks from predators. We have a totemic fear of predators but also things that are less immediate, some innate (e.g. heights) and some acquired culturally (e.g. radiation). Our broadened threat model protects us from demise in a wider set of circumstances: e.g. deer flock to to the exclusion zone forests near Chernobyl because there;s few predators (including few humans), humans fled from there because they knew or were told by those who knew of the risks of radiation exposure.
The late Dr. Gerald M. Edelman (Nobel Prize in Physiology or Medicine) had a fascinating theory he called "Neural Darwinism" that relied upon "reentrant mapping" he insisted was not feedback. He had a lab that formulated his ideas into software and a robot called Darwin 4. Does anyone here know what happened to that software? Is it described in detail anywhere? The following is an unflattering but useful discussion of Edelman's robot https://blogs.scientificamerican.com/cross-check/my-testy-en...
I have not heard of this one but there's a similar robotics implementation of a network based on the connectome (neural circuit wiring diagram) of C. Elegans brains(the only complete connectome we have so far). Its still in preliminary phase of doing cool stuff though but the idea is interesting.
31 comments
[ 3.1 ms ] story [ 76.0 ms ] threadThen, gradient descent is a centralized and supervised learning algorithm that has nothing to do with all the decentralized and emergent processes that take place in biological brains. For example, consider all the complex interactions of the various neurotransmitters and neuroreceptors.
Biological brains (let alone human) are actual physical realizations of networks with very complex and specific topologies. New connections grow and are formed in a very real sense, and this process interacts with the environment in a variety of ways, plus it is spatially embedded.
The problem is we don't understand how the brain is architected at anything other than a very coarse level. That architecture is the product of millions of years of evolution so we have some catching up to do.
And more generally, it’s pretty clear that there is a mixture of context-sensitive inhibitory and excitatory signals both deriving from the dendrites (inputs) and from the chemical stew of hormones and so forth they find themselves immersed in.
There are neurobiologists who claim (controversially, I concede) that a single neurone is a fully self-contained ‘processor’[1], and not the equivalent of single logic gate.
Those who somehow confuse our “neural network architectures” with the actual biology in our skulls are really misguided. Based on the evidence, one can at best opine that a TPU-based matrix add/multiply unit is approximating a very clunky and un-nuanced neural network. To assert that they are one and the same is... indefensible, when one goes beyond even the coarsest detail. A neural network as we implement it technologically is nothing but a weighted network of threshold units. That is all.
[1] To clarify: some kind of highly context-aware DSP with both long- and short-term storage facilities that allow it to react not only to the inputs that it is being fed, but also to inputs it was fed previously (and possibly outputs it produced previously as well).
To be fair, many lectures/books/articles/blog posts on ANNs start off by presenting a biological neuron, and saying that ANNs approximate this.
E.g. https://youtu.be/uXt8qF2Zzfo?t=385 (MIT 6.034 Artificial Intelligence, Fall 2010. 12a: Neural Nets)
I started off along much the same path (I bought and read Neural Networks: A Comprehensive Foundation (1994) by Simon Haykin) when I was in high school, read it all and thought I knew everything there was to know on the topic, and then started to read about neurobiology and was stunned to discover I was living a lie. Then I read The Computational Beauty of Nature (1998) by Gary William Flake and decided I was going to dedicate myself to understanding the complexities we brush under the rugs.
I think the Hacker News/Silicon Valley/Machine Learning/Developer crowd really need to look beyond the virtuosismo of their implementations. It’s a very introverted and self-referential mindset.
You might like this article: https://blog.piekniewski.info/2018/08/28/fun-numbers-about-t...
However, noise injection on its own has been extensively shown to decorrelate signal from noise so that the networks don't fixate on noise in the input data while training. This is widely used when using CNNs on image data and is known to increase performance of network predictions. So I am not sure how much different this article's findings are from this.
No link to arxiv publication makes it harder to make sense of these findings.
Edit: "Sleep" is not relevant for most ANN architectures used today. Synaptic pruning however is, similar to dropout.
https://en.wikipedia.org/wiki/Dither
If the purpose of intelligence is to "convert data to information", then perhaps it is akin to doing some work that decreases information theoretic entropy?
For example, in regression, we express input data as parameters of the model and error, so in the heat engine analogy, the input is the incoming heat, the parameters are the work output, and the error is the waste heat. (The error has higher entropy because there is less to explain, and you typically throw it away.)
So perhaps, if somebody would figure out this analogy correctly, we would see that we always need some cycles to continue the operation, just like in heat engines. So then we could conclude that any agent might require some kind of cycles of information creation and destruction in order to have intelligence.
Thought experiment in order to clarify: how would aliens, staring at our planet for aeons through a telescope, reach the conclusion that there is or is not intelligent life here on Earth? Well... they could observe that on average, historically, about one every hundred million years or so there is a major asteroid impact. They would eventually observe that from some point onwards a larger and larger delay would set in from the last impact and the next, far longer than expected. If they zoomed in they’d notice that somehow asteroids on impact trajectories would be deflected as if by a physical force emanating from the planet.
This would be our descendants, using the amazing technology at their disposal, deflecting asteroids to ensure that the last major extinction level event will be the one that killed off the dinosaurs 65 million years ago (and counting). They’d be using their intelligence to keep their options open and avoid being wiped out. They’re exerting a physical force on their environment and clearing the sidereal neighbourhood of dangerous rocks and comets.
Conclusion: intelligence is about perception of data but for the sake of control of the environment. “Data into information”, honestly, is too close to being a truism to be of any utility.
This includes the ability to make accurate predictions about threats and opportunities within the environment, and the likely outcomes of planned actions.
The more intelligent an entity is, the more successfully it can generalise from past experience and anticipate and influence future events.
This is a completely different definition to plain old IQ test intelligence, which seems to be closer to raw mental agility. I suspect you can score high on mental agility - e.g. figure and number series manipulation - and still have poor applied intelligence. The latter needs the ability to synthesise a range of diverse stimuli, which is not the same as being able to manipulate abstracted symbols with no surrounding context.
EDIT: Revised preamble.
https://en.wikipedia.org/wiki/Landauer%27s_principle
https://youtu.be/ufywxxu7jww
https://arxiv.org/pdf/1705.00612.pdf