>But even NorthPole’s 224 megabytes of RAM are not enough for large language models, such as those used by the chatbot ChatGPT, which take up several thousand megabytes of data even in their most stripped-down versions.
These are the trained weights and biases, the training data is unknown in size but could be terabytes… I’ve no idea how to even guess at the size of the training data but that doesn’t all need to in ram at the same time.
Once the model is trained, it doesn't need to keep the training data around.
GPT-3 is about 175 billion parameters (though I have no idea how many bits per parameter OpenAI uses at inference-time), and is apparently trained on 45 TB of data[0]
Presumably if you are training a robot to use a new/different tool you'll want the ability to train on site. If you buy an iHop restaurant the pancake robot in the kitchen ought to be able to be repurposed as a hamburger robot for your cheeseburger business. Omlette scrambling robots could be trained to mix small batches of cookie dough. Etc etc. Toyota is working on developing a framework for this already.
On-site training is… not really solved yet. Not efficiently, at any rate: any task can be trained with sufficient compute and/or examples, but probably more than most companies would care to bother with, and certainly more than we'd get onto one of the chips in the article.
That's not to diss the chips: As I understand it, one of the biggest issues is the power envelope of mobile units, which means making the computations more energy efficient is going to help massively, it's just that "training" and "inference" are currently very distinct tasks with very different hardware requirements.
(Also, I'm not sure if you mean those examples as illustrations or are serious about them: if you're serious, I suspect an old-fashioned robot arm bolted to the ground and following a pre-programmed path will probably cover your needs — GOFAI is great in restricted domains, the more modern AI models are more appropriate when the environment is more chaotic and less predictable, such as collaborating in a kitchen that also has humans or being asked on the fly to do a new recipe it's never encountered before).
I would expect LLM hardware to routinely support between 32 and 512GB memory in the Very Near Future. 1-4TB by the end of the decade. Custom hardware for GPT and LLM technology only started being developed in earnest in September 2022
At a glance, the supplementary materials from IBM's paper claim even less:
> NorthPole's core array includes 192 MB of flexible memory (768KB of unified memory per core). Assigning 2/3rd of this memory to parameters, such as weights and biases, provides 128MB for network storage.
My understanding is that this is a small energy-efficient chip for edge computing, like to stuff in an IoT device. Way too little memory to expect to run any recent language models, but could maybe do some basic object detection in a doorbell camera, for example.
The article's author seems to believe 224 megabytes is a huge amount of memory, and is a few orders of magnitude too low on the ChatGPT estimate too.
I do generally agree with your point here that this isn't very revolutionary. However it's worth pointing out that Apple chips have lots of on-package memory but relatively little on-die and they are quite different from this IBM chip because of that.
I know basically zero about chip fabrication but I remember reading somewhere a while (15÷ years) ago that processor-in-memory was always a desirable design objective for obvious reasons, but that there are fundamental differences in the process for fabbing memory versus logic (different regions of Si doping not possible in same wafer, something like that? See this is where I should stay out of these discussions) that haven't been resolved, so the next best thing is on package pairing.
Roughly speaking, CPU wants smaller space and RAM wants bigger space. At a high-level take:
* CPU design is the most expensive space due to it having the greatest quality and capability requirements. RAM is mostly just a very, very large repetitive structure, so more space better.
* A CPU fault can be potentially corrected by microcode changes to route around the damage (ie part binning). RAM cannot generally take faults.
* DRAM is simpler to make, "just" a capacitor, but capacitance leaks over time; which, means generating resistive heat in an area that we want as little heat as possible. You could use SRAM (two transistors) but now you have substantially more complex part to fail.
* DRAM quality requirements are much less stringent if you make just bigger cells.
"It's worth clarifying a few things early here. First, NorthPole does nothing to help the energy demand in training a neural network; it's purely designed for execution. Second, it is not a general AI processor; it's specifically designed for inference-focused neural networks. As noted above, inferences include things like figuring out the contents of an image or audio clip so they have a large range of uses, but this chip won't do you any good if your needs include running a large language model."
37 comments
[ 4.1 ms ] story [ 89.1 ms ] threadhttps://www.science.org/doi/10.1126/science.adh1174
Is that right?
Small LLMs that have been quantized to optimize for space still sit at 4-9 gigs
GPT-3 is about 175 billion parameters (though I have no idea how many bits per parameter OpenAI uses at inference-time), and is apparently trained on 45 TB of data[0]
[0] Caution: citation was first hit on google, YMMV — https://www.springboard.com/blog/data-science/machine-learni...
On-site training is… not really solved yet. Not efficiently, at any rate: any task can be trained with sufficient compute and/or examples, but probably more than most companies would care to bother with, and certainly more than we'd get onto one of the chips in the article.
That's not to diss the chips: As I understand it, one of the biggest issues is the power envelope of mobile units, which means making the computations more energy efficient is going to help massively, it's just that "training" and "inference" are currently very distinct tasks with very different hardware requirements.
(Also, I'm not sure if you mean those examples as illustrations or are serious about them: if you're serious, I suspect an old-fashioned robot arm bolted to the ground and following a pre-programmed path will probably cover your needs — GOFAI is great in restricted domains, the more modern AI models are more appropriate when the environment is more chaotic and less predictable, such as collaborating in a kitchen that also has humans or being asked on the fly to do a new recipe it's never encountered before).
> NorthPole's core array includes 192 MB of flexible memory (768KB of unified memory per core). Assigning 2/3rd of this memory to parameters, such as weights and biases, provides 128MB for network storage.
https://www.science.org/doi/suppl/10.1126/science.adh1174/su...
My understanding is that this is a small energy-efficient chip for edge computing, like to stuff in an IoT device. Way too little memory to expect to run any recent language models, but could maybe do some basic object detection in a doorbell camera, for example.
The article's author seems to believe 224 megabytes is a huge amount of memory, and is a few orders of magnitude too low on the ChatGPT estimate too.
And Tesla’s Dojo is already running processors and memory together at wafer scale, as are a number of startups.
I love it when uninformed writers just regurgitate the press release rather than do research to give context.
It's also pretty clear this is a prototype, not a final product that can never be improved in any possible way.
"It's worth clarifying a few things early here. First, NorthPole does nothing to help the energy demand in training a neural network; it's purely designed for execution. Second, it is not a general AI processor; it's specifically designed for inference-focused neural networks. As noted above, inferences include things like figuring out the contents of an image or audio clip so they have a large range of uses, but this chip won't do you any good if your needs include running a large language model."
[0] https://arstechnica.com/science/2023/10/ibm-has-made-a-new-h...
The problem is that even that amount of memory is tiny by today's LLM standards.