>"Researchers have demonstrated learning at a rate that is a 1 million times improvement compared with other typical spiking neural nets as measured by total operations to achieve a given accuracy when solving MNIST digit recognition problems."
>"Further, it is up to 1,000 times more energy-efficient than general purpose computing required for typical training systems."
MNIST is a dataset, VGG is a software model (network architecture). You can train/test VGG on MNIST, train/test this spiking network model on MNIST, and compare the accuracy between the two.
If you want to compare energy efficiency, you look at the hardware these models are running on (CPU, GPU, FPGA, ASIC), and see how much energy is spent by each of these chips to reach a certain accuracy (preferably on ImageNet).
You can also compare the speed of processing (significantly affected by image size):
1. How fast can you get to a certain accuracy?
2. How many images per second can you classify with that accuracy?
Strictly speaking, true. But one discovers early enough that you can screw up a number of things when training a model on something as simple as MNIST and the model will still work, because there's so much redundancy, the correct layers will just train their way around the ones that don't really work.
VGG trained for general purpose image classification with a decent sized input is a much better indicator of whether or not some ML hardware improvement is useful.
You're contradicting yourself. If you're afraid that redundancy will hide hardware problems, then VGG is a bad choice, because it's pretty much the most redundant NN model ever :)
Results on MNIST can be highly deceiving because it is so easy. ImageNet would be a good benchmark.
Also, are they only comparing against other spiking neural nets instead of deep nets? If so, that doesn't tell much about whether these chips are useful for modern AI. I'd have to see their paper.
A million times faster and a thousand times more energy efficient means a thousand times the power usage. I find that hard to believe, so those “other typical spiking neural nets” must be quite different from “typical training systems” => as others said, this is marketing speak.
"It’s a future where first responders using image-recognition applications can analyze streetlight camera images and quickly solve missing or abducted person reports."
Erp. That's also known as "a massive surveillance network", right?
"It’s a future where [your government] using image-recognition applications can analyze streetlight camera images and quickly solve [whether you're involved in pre-crime]."
This stuff is going to happen (or is already here) so perhaps it's time to switch to managing pervasive surveillance rather than preventing it.
Looking forward to when there is more information than just a press release (the press release for Intel's 3D X-Point memory was over two years ago, and we still aren't seeing those in production). Regardless, very exciting work, and I look forward to getting my hands on the API/instruction set/manual.
This hardware is designed for running spiking neural network models? What is the current state of the art in this field? I was under the impression that training a spiking neural network was somewhat of an unsolved problem, because backprop doesn't easily apply. Anyone have information on this?
There are several frameworks for the construction of spiking neural networks given an abstract mathematical description of the desired function. The most complete (note that I'm currently a student in the lab that developed this method) is the Neural Engineering Framework [1]. There is a Python reference implementation called Nengo [2] which can target several backends, including neuromorphic hardware.
Training deep spiking neural networks is possible as well [3].
There's this common claim that spiking networks are somehow more power efficient. Yet I have never seen any evidence to support it, or even any decent explanation why would it ever be the case.
They are more power efficient when implemented as analogue circuits. Two main reasons:
First, you only need a handful of transistors to implement a single spiking neuron, whereas digital circuits require ten to hundred thousands of transistors for corresponding numerical computations.
Second, spiking neural networks are intrinsically asynchronous. All the computing elements (neurons) are independent and only communicate via rare one-bit events (spikes). Correspondingly you (if any) only need a clock for the event bus but not for the vast majority of the chip.
Edit: To clarify: the asynchronicity argument is valid for digital implementations of spiking neural networks as well, especially as you scale up to networks of a million (or more) neurons. See http://apt.cs.manchester.ac.uk/projects/SpiNNaker/ for an implementation.
He is answering the question that his parent post asked, which is how can a spiking network be more efficient than a non-spiking one. He provided the details to a specific case of spiking networks which are more efficient, and explanations as to why it is more efficient.
It should be obvious that analog hardware is more efficient than digital one. No explanation is needed there. I, on the other hand, would like to know how spiking helps when dealing with dense input signal matrices of common deep learning applications. It seems to me that analog non-spiking hardware is better suited for those tasks, especially when you're able to arrange processing elements in a large crossbar array, and colocate memory and processing (e.g. weights and multipliers) in the same element. Representing a signal value (of a certain precision) using a constant voltage level, or a current magnitude seems to be easier, more natural and more efficient than using either timings between spikes, or spike trains (spiking rate encoding the signal).
You're right that analogue non-spiking hardware in theory would be the way to go for deep neural networks.
However, when implemented in a modern semiconductor process, it is relatively hard to implement analogue computations with negligible noise; especially when connecting distant parts of the chip. Spiking neuron models have the advantage of producing digital output (either they are spiking at a given moment in time or they don't). Thus this pulse can be easily transported without information loss, and the analogue computation is confined to a relatively small region, which makes it possible to reduce noise in the internal analogue signals.
A different way to think about it: say you wanted to implement an analogue model of a classical neuron (as used in deep neural nets) and wanted to transfer its output as a digital signal to mitigate the noise problem. In that case, the required analogue/digital converter would be far more complex than the analogue neuron itself. As you continue to reduce the size and precision of the A/D converter you'd end up with a 1-bit delta-sigma A/D converter, and you're circuit would very much look like a spiking neuron.
In an ideal world, yes a spike would be a simple binary signal, easy to generate/detect. Unfortunately, in the world we live in, it's quite a bit more complicated. Have you ever looked at Spice simulations where you needed to propagate a pulse (a spike with duration of one clock cycle)? It quickly gets distorted and attenuated. I don't see how noise in the circuit is anything less of a problem for a spike than it is for a constant signal. It actually seems to be more susceptible to noise, because now some spurious noise injection could be interpreted as a spike by the receiver!
What makes you think short pulses are more noise resistant than long ones? And if you try to utilize interspike timings to encode information you expose yourself to a whole bunch of additional challenges (not sure if those timings are used in the current spiking network models though).
In analogue hardware implementations spike events are transmitted on a standard digital bus using AER (address-event representation). The signal is not attenuated, since standard digital hardware (though asynchronous, depending on the implementation) is used to transmit the spike. Spurious noise injection is thus not a significant problem as well.
Note that I'm only a layman when it comes to analogue neuromorphic hardware implementation details, I encourage you to have a look at [1] for more detailed energy computations.
Edit: Added a "significant" above. There indeed are minor problems with cross talk causing additional spikes, but mostly in the analogue neuron subthreshold regime, not in the AER bus. [2]
Wait a second, so your solution is a digital shared bus? With a need to look stuff up in a separate RAM block? Do you realize that this is pretty much the worst thing from an energy efficiency standpoint?
I've looked at the NeuroGrid link. It's still not clear to me why they want to use spikes. Some parts of their design decisions (e.g. AER bus) are forced by the need to deal with spikes, and it feels like their hardware architecture would be simpler and more efficient if they didn't. Is the only reason "biological plausibility"?
Ultimately, what we want is hardware that can do what my 250 Watts 1080Ti card does (e.g. reach high classification accuracy on ImageNet) but faster and using less less power. Spikes don't seem like the best approach to achieve that.
I'm from the lab that you're citing with those links and I'm kind of amazed you're so up to date with our research. Why do you know so much about Nengo? Are you actually one of my lab-mates, but I just can't identify you from your username?
Nah. There are legitimate reasons for doing this. Especially when spiking neurons are implemented using analogue circuits (which I could find no indication of in the press release however, but see [1,2,3,4] for examples) spiking neurons offer significant advantages, such as extremely low power consumption, intrinsic dynamics, and asynchronicity (which again, reduces power consumption by not requiring clocks).
I was looking for clarification as to whether the neurons are implemented as analog circuits as well, especially with the reference to Dr. Carver Mead in the beginning of the article, who has a long history of working on analog neuron circuits, but I did not see any explicit references to whether the neurons were digital or analog.
intel is not a charity. if they do something, they either have a customer lined up or they think there's money on the table in the market, just as it should be. bragging rights are low cash flow.
I'm pretty surprised by all the focus on spiking neural networks too; they are really still much more of an academic topic and I have never heard of any real practical applications (yet).
There are a number of methods of training these kinds of networks like Spike Timing Dependent Plasticity (STDP) which is essentially a reinforcement learning algorithm that increases the weights between neurons that spike often together and modulates the increases by a reward signal. However, all these methods are really more focused on replicating and modeling the biological phenomena and not being performant.
In theory, spiking networks should be much more efficient, real neurons are able to take advantage of many non-linear effects to create incredibly complex analog to digital (spikes) computation in every cell using incredibly little energy - our brains only use about 20 watts of power.
But, the current approach to CPUs and even GPUs means simulating all this stuff is ridiculously inefficient and ultimately looks nothing like the real thing.
How does this compare with Google's proprietary neural network optimized chip? That seems like a place that Intel could kick ass in and sell tons to Amazon and Microsoft's cloud services for NN acceleration, especially if they supported the main NN libraries.
And if Intel isn't pursuing the opportunity to compete for deep learning NN against Google's chips, why aren't they? It seems like a well defined opportunity, more so than whatever these chips are.
I don't know for sure, but google's chips are advanced matrix multiplying engines. the "neuromorphic" chips are (in a very stupid-fied explanation) kind of like imagine a configurable fpgas with more-than-one-bit lines connecting adjacent tiles.
From what I can tell, Google's TPUs are boiling GPUs down to the essence of what they do in TensorFlow, namely a "higher volume of reduced precision computation" [1].
GPUs are generally much more flexible and can run something resembling a normal C program. From the little public information available chips like Google TPU or NVIDIA's Tensor Core (a functional unit on the Volta architecture) are essentially hard-wired to do exactly low precision matrix multiplication. Less flexible but more efficient when you know that's what your workload is.
no, GPUs are vector math engines. To be fair, you can assemble a matrix multiply from vector math, but it is not quite the same. In spirit, GPUs are closer to matrix engines than neuromorphic chips.
Maybe there is a way to port regular neural nets onto spiking nets - you'd train a regular network first using your algorithm of choice, then transfer-learn to the spiking net.
> This extremely energy efficient chip, which uses the data to learn and make inferences, gets smarter over time and does not need to be trained in the traditional way.
Does not need to be trained? The writer maybe wanted to say it does not need to be programmed in the traditional way.
If this was a news site, I'd understand - the same writers cover too many topics, they can't be experts in all. But when it comes straight from Intel, it's weird to see such mistakes from someone who signs as Dr. Michael Mayberry.
> Researchers have demonstrated learning at a rate that is a 1 million times improvement compared with other typical spiking neural nets as measured by total operations to achieve a given accuracy when solving MNIST digit recognition problems.
MNIST is a joke already, why don't they show off the ImageNet numbers? And spiking neural nets are less efficient compared to normal neural nets, so the 1 million speedup might not be as impressive as it appears at first hand. They don't compare with GPU run TensorFlow or CNTK here, but with other spiking neural nets which are seldom used, mostly as a proof of concept.
> A total of 130,000 neurons and 130 million synapses.
So the network size is medium - you can do a lot with 130 mil weights, but there are models on GPU that go up to 1 billion weights. Even the "cat detector" unsupervised neural net created by Google in 2011, had 1 billion weights, and that's ages ago in AI time.
> Does not need to be trained? The writer maybe wanted to say it does not need to be programmed in the traditional way.
Nope, "trained" is correct. The current state of deep learning involves "training" and "inference" as distinct components where entirely different kinds of math are applied to each. GPUs can do training as training is kind of "dumb" and repetitive. CPUs, however, can do training and inference (and also can do allocations among resources dedicated to training and inference).
Sure, GPUs are the trendy thing being pumped by stock market day traders these days, but they are only about as efficient as the black Model T Ford in the "traditional" sense.
To be clear, when I say "training" in this context, I mean they don't require iterative rounds of stochastic gradient descent. They just need to be given the function that you want to approximate and then they're done.
Yes, maybe it uses something other than gradient descent, but why? Nothing works nearly as well by comparison. In a chip we can do things we can't do in the brain, such as sending gradients back on the same connections.
Depends on what you mean by "well". It would be foolish of me to deny the advances of deep learning. However, there are limitations and the NEF can act as a bridge in those cases. For example, if you want:
1. To approximate a dynamic system
2. To use symbolic computation
3. Use the neural networks as an initial guess at a solution to a problem and then want to use gradient descent to figure out the actual solution
I talk about using NEF as a bridge in [this blog post](https://medium.com/@seanaubin/deep-learning-is-almost-the-br...), if you're curious. Unfortunately, I don't know of any examples using the "initial guess" paradigm that I describe, but that's more of a "not enough people using Nengo" problem.
If you can do that, either by writing a deterministic algorithm, or even defining it analytically, there's no need for machine learning, neural networks or etc.
> there's no need for machine learning, neural networks or etc.
There is if:
1. You want an energy-efficient distributed representation of the function.
2. You want to use the function as an initial guess and then learn online afterwards.
51 comments
[ 3.2 ms ] story [ 107 ms ] thread>"Further, it is up to 1,000 times more energy-efficient than general purpose computing required for typical training systems."
Wow. That is tremendous.
If you want to compare energy efficiency, you look at the hardware these models are running on (CPU, GPU, FPGA, ASIC), and see how much energy is spent by each of these chips to reach a certain accuracy (preferably on ImageNet).
You can also compare the speed of processing (significantly affected by image size): 1. How fast can you get to a certain accuracy? 2. How many images per second can you classify with that accuracy?
Strictly speaking, true. But one discovers early enough that you can screw up a number of things when training a model on something as simple as MNIST and the model will still work, because there's so much redundancy, the correct layers will just train their way around the ones that don't really work.
VGG trained for general purpose image classification with a decent sized input is a much better indicator of whether or not some ML hardware improvement is useful.
Also, are they only comparing against other spiking neural nets instead of deep nets? If so, that doesn't tell much about whether these chips are useful for modern AI. I'd have to see their paper.
Erp. That's also known as "a massive surveillance network", right?
"It’s a future where [your government] using image-recognition applications can analyze streetlight camera images and quickly solve [whether you're involved in pre-crime]."
This stuff is going to happen (or is already here) so perhaps it's time to switch to managing pervasive surveillance rather than preventing it.
https://www.anandtech.com/show/11209/intel-optane-ssd-dc-p48...
Training deep spiking neural networks is possible as well [3].
[1] http://compneuro.uwaterloo.ca/research/nef.html [2] https://www.nengo.ai/ [3] https://arxiv.org/abs/1611.05141
First, you only need a handful of transistors to implement a single spiking neuron, whereas digital circuits require ten to hundred thousands of transistors for corresponding numerical computations.
Second, spiking neural networks are intrinsically asynchronous. All the computing elements (neurons) are independent and only communicate via rare one-bit events (spikes). Correspondingly you (if any) only need a clock for the event bus but not for the vast majority of the chip.
See http://web.stanford.edu/group/brainsinsilicon/documents/IEEE... for a high-level overview.
Edit: To clarify: the asynchronicity argument is valid for digital implementations of spiking neural networks as well, especially as you scale up to networks of a million (or more) neurons. See http://apt.cs.manchester.ac.uk/projects/SpiNNaker/ for an implementation.
However, when implemented in a modern semiconductor process, it is relatively hard to implement analogue computations with negligible noise; especially when connecting distant parts of the chip. Spiking neuron models have the advantage of producing digital output (either they are spiking at a given moment in time or they don't). Thus this pulse can be easily transported without information loss, and the analogue computation is confined to a relatively small region, which makes it possible to reduce noise in the internal analogue signals.
A different way to think about it: say you wanted to implement an analogue model of a classical neuron (as used in deep neural nets) and wanted to transfer its output as a digital signal to mitigate the noise problem. In that case, the required analogue/digital converter would be far more complex than the analogue neuron itself. As you continue to reduce the size and precision of the A/D converter you'd end up with a 1-bit delta-sigma A/D converter, and you're circuit would very much look like a spiking neuron.
Edit: Reformulated some parts for clarity
What makes you think short pulses are more noise resistant than long ones? And if you try to utilize interspike timings to encode information you expose yourself to a whole bunch of additional challenges (not sure if those timings are used in the current spiking network models though).
Note that I'm only a layman when it comes to analogue neuromorphic hardware implementation details, I encourage you to have a look at [1] for more detailed energy computations.
[1] http://web.stanford.edu/group/brainsinsilicon/documents/IEEE...
Edit: Added a "significant" above. There indeed are minor problems with cross talk causing additional spikes, but mostly in the analogue neuron subthreshold regime, not in the AER bus. [2]
[2] http://journal.frontiersin.org/article/10.3389/fncom.2017.00...
I've looked at the NeuroGrid link. It's still not clear to me why they want to use spikes. Some parts of their design decisions (e.g. AER bus) are forced by the need to deal with spikes, and it feels like their hardware architecture would be simpler and more efficient if they didn't. Is the only reason "biological plausibility"?
Ultimately, what we want is hardware that can do what my 250 Watts 1080Ti card does (e.g. reach high classification accuracy on ImageNet) but faster and using less less power. Spikes don't seem like the best approach to achieve that.
[1] https://brainscales.kip.uni-heidelberg.de/ [2] http://www.kip.uni-heidelberg.de/vision/spikey/ [3] https://web.stanford.edu/group/brainsinsilicon/neurogrid.htm... [4] http://ncs.ethz.ch/
Your comment supports what I wrote.
There are a number of methods of training these kinds of networks like Spike Timing Dependent Plasticity (STDP) which is essentially a reinforcement learning algorithm that increases the weights between neurons that spike often together and modulates the increases by a reward signal. However, all these methods are really more focused on replicating and modeling the biological phenomena and not being performant.
In theory, spiking networks should be much more efficient, real neurons are able to take advantage of many non-linear effects to create incredibly complex analog to digital (spikes) computation in every cell using incredibly little energy - our brains only use about 20 watts of power.
But, the current approach to CPUs and even GPUs means simulating all this stuff is ridiculously inefficient and ultimately looks nothing like the real thing.
And if Intel isn't pursuing the opportunity to compete for deep learning NN against Google's chips, why aren't they? It seems like a well defined opportunity, more so than whatever these chips are.
Excuse my ignorance, but aren't GPUs exactly that?
[1] https://en.wikipedia.org/wiki/Tensor_processing_unit
https://devblogs.nvidia.com/parallelforall/inside-volta/
https://www.nextplatform.com/2017/04/05/first-depth-look-goo...
Does not need to be trained? The writer maybe wanted to say it does not need to be programmed in the traditional way.
If this was a news site, I'd understand - the same writers cover too many topics, they can't be experts in all. But when it comes straight from Intel, it's weird to see such mistakes from someone who signs as Dr. Michael Mayberry.
> Researchers have demonstrated learning at a rate that is a 1 million times improvement compared with other typical spiking neural nets as measured by total operations to achieve a given accuracy when solving MNIST digit recognition problems.
MNIST is a joke already, why don't they show off the ImageNet numbers? And spiking neural nets are less efficient compared to normal neural nets, so the 1 million speedup might not be as impressive as it appears at first hand. They don't compare with GPU run TensorFlow or CNTK here, but with other spiking neural nets which are seldom used, mostly as a proof of concept.
> A total of 130,000 neurons and 130 million synapses.
So the network size is medium - you can do a lot with 130 mil weights, but there are models on GPU that go up to 1 billion weights. Even the "cat detector" unsupervised neural net created by Google in 2011, had 1 billion weights, and that's ages ago in AI time.
Nope, "trained" is correct. The current state of deep learning involves "training" and "inference" as distinct components where entirely different kinds of math are applied to each. GPUs can do training as training is kind of "dumb" and repetitive. CPUs, however, can do training and inference (and also can do allocations among resources dedicated to training and inference).
Sure, GPUs are the trendy thing being pumped by stock market day traders these days, but they are only about as efficient as the black Model T Ford in the "traditional" sense.
There are two ways to build spiking neural net architectures without training. The [Neural Engineering Framework](https://medium.com/@seanaubin/a-way-around-the-coming-perfor...) and [Deneve's approach which I'm more uncertain about](https://forum.nengo.ai/t/robustness-of-deneves-networks/320?...).
To be clear, when I say "training" in this context, I mean they don't require iterative rounds of stochastic gradient descent. They just need to be given the function that you want to approximate and then they're done.
Depends on what you mean by "well". It would be foolish of me to deny the advances of deep learning. However, there are limitations and the NEF can act as a bridge in those cases. For example, if you want:
1. To approximate a dynamic system 2. To use symbolic computation 3. Use the neural networks as an initial guess at a solution to a problem and then want to use gradient descent to figure out the actual solution
I talk about using NEF as a bridge in [this blog post](https://medium.com/@seanaubin/deep-learning-is-almost-the-br...), if you're curious. Unfortunately, I don't know of any examples using the "initial guess" paradigm that I describe, but that's more of a "not enough people using Nengo" problem.
If you can do that, either by writing a deterministic algorithm, or even defining it analytically, there's no need for machine learning, neural networks or etc.
There is if: 1. You want an energy-efficient distributed representation of the function. 2. You want to use the function as an initial guess and then learn online afterwards.