There are times where diverting from normal conventions make sense. The average consumer might not know that 1Gb/s is faster than 750Mb/s. That being said, I don't think I've ever seen anything along the lines of 1G,000b/s.
My guess is the titles get auto adjusted by Hacker News, but the script that does it doesn’t have logic for a trillion and only goes up to a billion, hence the weirdness of a string match and replace
Billion is a word for a large number, and it has two distinct definitions:
1,000,000,000, i.e. one thousand million, or 10^9 (ten to the ninth power), as defined on the short scale. This is now the most common sense of the word in all varieties of English; it has long been established in American English and has since become common in Britain and other English-speaking countries as well.
1,000,000,000,000, i.e. one million million, or 10^12 (ten to the twelfth power), as defined on the long scale. This number is the historical sense of the word and remains the established sense of the word in other European languages. Though displaced by the short scale definition relatively early in US English, it remained the most common sense of the word in Britain until the 1950s and still remains in occasional use there.
Title should be either "4,000,000,000,000 Transistors" (as in the actual video title) or "4 Trillion Transistors" or maybe "4T Transistors". "4B,000" ("four billion thousand"?) looks like 48,000 (forty-eight thousand).
This is something I'm clueless about and can't really understand. They say this is 24kW hungry. How does CPU power consumption really work on electrical level, what warrants that much power, even for regular CPUs? Like, from the basics level.. is it resistance of the material with frequency of switching or what is really going on there? Where does the power go on such a relatively small surface?
Yes, the power consumption comes from the resistance of the circuit. In case of CMOS circuits there would ideally be no current flow when no signal changes but transistors are not perfect and have leakage currents. When signals change, primarily triggered by the clock rising or falling, there is a short time in which the supply rail and ground rail are essentially shorted out.
Each gate has logically two transistors of which exactly one is always conducting, either connecting the output to the supply rail making the output a one, or connecting the output to the ground rail making the output a zero. When the output of the gate changes, both transistors have to switch in order to connect the output to the other rail than before. While this happens both transistors are conducting at the same time allowing current to flow from the supply rail to the ground rail.
In addition to that the input capacitances of subsequent gates get charged from the supply rail when the output goes high and discharged into the ground rail when input goes low. So every signal change pumps some charge from the supply rail through the input capacitances to the ground rail.
This is completely wrong. I've desiged ICs from 350nm down to 3nm technology. I don't know what do you mean by transistors are biased. If you mean SOI body bissing, it's not a common technology.
Crowbar current is still a big problem. Leakage of Finfets are crazy, especially when you use ulvt devices, which you need for high performance.
The previous post is fully correct. Though the bulk of the power is switching capacitor.
It simply takes a non-zero amount of energy to turn a transistor on and off.
Add up trillions of transistors, flicking on and off billions of times a second, and you get enormous power draws.
What is actually drawing power is the gate capacitance of the transistors. If the transistor were a physical switch, the gate capacitance is the "weight" that must be put on the switch to flip it. Of course this weight gets smaller as the switches shrink and as the tech improves, but it will always be non-zero.
None of this accounts for resistive losses either, which is just the cost of doing business for a CPU.
Modern CPUs are built using CMOS MOSFET transistors[1]. The gate, which controls if the transistor conducts or not, is effectively a small capacitor. The gate capacitor has to be charged up for the transistor to conduct[2], ie you have to stuff some electrons into it to turn the transistor on.
Once you've done that, the transistor is on until the gate capacitor is discharged. This requires getting rid of the electrons you stuffed into it. The easiest is to just connect the gate to ground, essentially throwing the electrons away.
So for each time the transistor goes through an on-off cycle, you need to "spend" some electrons, which in turn need to be supplied from the power supply. Thus higher frequency means more current just from more on-off cycles per second.
There's also resistive losses and leakage currents and such.
Now in theory I suppose you could recycle some of these electrons (using a charge pump arrangement[3]), reducing the overall demand. But that would require relatively large capacitors, and on-chip capacitors take a lot of chip area which could have been used for many transistors instead.
a lot of good answers. just couple points to add:
- Charging a capacitor from 0 to Vdd volts takes E=CVdd^2 amount of energy. To find the power (energy/second) you need to multiply this with how frequently you do this, i.e. clock frequency. So, P=FCVdd^2. So most digitak circuits power scales linearly with frequency, and quadratically with power supply voltage Vdd.
- Half of themis power is resistive loses. This is kinda a fundamental thing. Charging a capacitor with a resistor is "lossy".
- Some of the charge is reused within the circuits
Just to get some sense of perspective, the Zen 4-based Ryzen 7950X3D is built on TMSC's 5nm node and is listed[1] as being two 71mm^2 dies. The 5nm node uses a 300mm wafer[2], which means roughly 900 dies or 450 7950X3D's on one wafer, for a total of 8 trillion transistors.
The peak power average of the 7950X3D is roughly 150W[3], which means if you could somehow run all 450 CPUs (900 dies) at peak, they'd consume around 68kW.
edit: I forgot about the IO die which contains the memory controller, so that will suck some power as well. So if we say 50W for that and 50W for the CPU dies, that's 45kW.
That's assuming you get a "clean wafer" with all dies working, not "just" the 80% yield or so.
Interesting. I know there's a lot of attempts to hobble China by limiting their access to cutting edge chips and semiconductor manufacturing technology, but could something like this be a workaround for them, at least for datacenter-type jobs?
Maybe it wouldn't be as powerful as one of these, due to their less capable fabs, but something that's good enough to get the job done in spite of the embargoes.
Is that 150GB/s between elements that expect to run tightly coupled processes together? Maybe the bandwidth between chips is less important.
I mean, in a cluster you might have a bunch of nodes with 8x GPUs hanging off each, if this thing replaces a whole node rather than a single GPU, which I assume is the case, it is not really a useful comparison, right?
"CSL allows for compile time execution of code blocks that take compile-time constant
objects as input, a powerful feature it inherits from Zig, on which CSL is based. CSL will be largely
familiar to anyone who is comfortable with C/C++, but there are some new capabilities on top of
the C-derived basics."
It's 215 x 215 mm so it fits in a large laptop, some 15" and definitely 17" ones. The keyboard could get a little warm and battery life doesn't look good.
According to the company, the new chip will enable training of AI models with up to 24 trillion parameters. Let me repeat that, in case you're as excited as I am: 24. Trillion. Parameters. For comparison, the largest AI models currently in use have around 0.5 trillion parameters, around 48x times smaller.
Each parameter is a connection between artificial neurons. For example, inside an AI model, a linear layer that transforms an input vector with 1024 elements to an output vector with 2048 elements has 1024×2048 = ~2M parameters in a weight matrix. Each parameter specifies by how much each element in the input vector contributes to or subtracts from each element in the output vector. Each output vector element is a weighted sum (AKA a linear combination), of each input vector element.
A human brain has an estimated 100-500 trillion synapses connecting biological neurons. Each synapse is quite a complicated biological structure[a], but if we oversimplify things and assume that every synapse can be modeled as a single parameter in a weight matrix, then the largest AI models in use today have approximately 100T to 500T ÷ 0.5T = 200x to 1000x fewer connections between neurons that the human brain. If the company's claims prove true, this new chip will enable training of AI models that have only 4x to 20x fewer connections that the human brain.
> but if we oversimplify things and assume that every synapse can be modeled as a single parameter in a weight matrix
Which, it probably can't... but offsetting those simplifications and 4-20x difference is the massive difference in how quickly those synapses can be activated.
To quote their official response "If the WSE weren't rectangular, the complexity of power delivery, I/O, mechanical integrity and cooling become much more difficult, to the point of impracticality.".
If you were to add up all transistors fabricated worldwide, up until <year>, such that total roughly matches the # on this beast, what year would you arrive? Hell, throw in discrete transistors if you want.
How many early supercomputers / workstations etc would that include? How much progress did humanity make using all those early machines (or any transistorized device!) combined?
4004 from the 1970s used 2300 transistors, so it would have needed to sell billions.
Pentium from 1990s had 3M transistors, so it could hit our target by selling a million units.
I'm betting (without much research) that the Pentium line alone sold millions, and the industry as a whole could hit those numbers about 5 years earlier.
Ex Cerebras engineer. In my opinion, this is not going to be the case. The WSE-2 was a b** to program and debug. Their compilation strategy is a dead end, and they invest very little into developer ease. My two cents.
I would be more worried about the fact that next year every CPU is going to ship with some kind of AI accelerator already integrated to the die, which means the only competitive differentiation boils down to how much SRAM and memory bandwidth your AI accelerator is going to have. TOPS or FLOPS will become an irrelevant differentiator.
One thing I don't understand about their architecture is that they have spent so much effort building this monster of a chip, but if you are going to do something crazy, why not work on processing in memory instead? At least for transformers you will primarily be bottlenecked on matrix multiplication and almost nothing else, so you only need to add a simple matrix vector unit behind your address decoder and then almost every AI accelerator will become obsolete over night. I wouldn't suggest this to a random startup though.
Previous versions have had basically 100% yield, because when you're working with the whole wafer it's pretty easy to squeeze in enough redundancy to route around defects unless you get a really unlucky cluster of defects.
Hm, let's wait and see what the gemm/W perf is, and how many programmer hours it takes to implement say an mlp. Wafer scale data flow may not be a solved problem?
92 comments
[ 2.8 ms ] story [ 156 ms ] threadhttps://www.cerebras.net/product-chip/
"4,000,000,000,000 Transistors, One Giant Chip (Cerebras WSE-3)"
So I guess they're trying to stay true to it.
Quote:
Billion is a word for a large number, and it has two distinct definitions:
1,000,000,000, i.e. one thousand million, or 10^9 (ten to the ninth power), as defined on the short scale. This is now the most common sense of the word in all varieties of English; it has long been established in American English and has since become common in Britain and other English-speaking countries as well.
1,000,000,000,000, i.e. one million million, or 10^12 (ten to the twelfth power), as defined on the long scale. This number is the historical sense of the word and remains the established sense of the word in other European languages. Though displaced by the short scale definition relatively early in US English, it remained the most common sense of the word in Britain until the 1950s and still remains in occasional use there.
https://en.wikipedia.org/wiki/Billion
https://vimeo.com/853557623
https://web.archive.org/web/20230812020202/https://www.youtu...
(Vimeo/Archive because the original video was taken down from YouTube)
200,000 electrical contacts
850,000 cores
and that's the "old" one. wow.
edit: thanks people, makes sense now!
Each gate has logically two transistors of which exactly one is always conducting, either connecting the output to the supply rail making the output a one, or connecting the output to the ground rail making the output a zero. When the output of the gate changes, both transistors have to switch in order to connect the output to the other rail than before. While this happens both transistors are conducting at the same time allowing current to flow from the supply rail to the ground rail.
In addition to that the input capacitances of subsequent gates get charged from the supply rail when the output goes high and discharged into the ground rail when input goes low. So every signal change pumps some charge from the supply rail through the input capacitances to the ground rail.
Crowbar current is still a big problem. Leakage of Finfets are crazy, especially when you use ulvt devices, which you need for high performance.
The previous post is fully correct. Though the bulk of the power is switching capacitor.
Add up trillions of transistors, flicking on and off billions of times a second, and you get enormous power draws.
What is actually drawing power is the gate capacitance of the transistors. If the transistor were a physical switch, the gate capacitance is the "weight" that must be put on the switch to flip it. Of course this weight gets smaller as the switches shrink and as the tech improves, but it will always be non-zero.
None of this accounts for resistive losses either, which is just the cost of doing business for a CPU.
Once you've done that, the transistor is on until the gate capacitor is discharged. This requires getting rid of the electrons you stuffed into it. The easiest is to just connect the gate to ground, essentially throwing the electrons away.
So for each time the transistor goes through an on-off cycle, you need to "spend" some electrons, which in turn need to be supplied from the power supply. Thus higher frequency means more current just from more on-off cycles per second.
There's also resistive losses and leakage currents and such.
Now in theory I suppose you could recycle some of these electrons (using a charge pump arrangement[3]), reducing the overall demand. But that would require relatively large capacitors, and on-chip capacitors take a lot of chip area which could have been used for many transistors instead.
[1]: https://en.wikipedia.org/wiki/CMOS
[2]: https://en.wikipedia.org/wiki/MOSFET#Modes_of_operation
[3]: https://en.wikipedia.org/wiki/Charge_pump
- Half of themis power is resistive loses. This is kinda a fundamental thing. Charging a capacitor with a resistor is "lossy".
- Some of the charge is reused within the circuits
The peak power average of the 7950X3D is roughly 150W[3], which means if you could somehow run all 450 CPUs (900 dies) at peak, they'd consume around 68kW.
edit: I forgot about the IO die which contains the memory controller, so that will suck some power as well. So if we say 50W for that and 50W for the CPU dies, that's 45kW.
That's assuming you get a "clean wafer" with all dies working, not "just" the 80% yield or so.
[1]: https://www.techpowerup.com/cpu-specs/ryzen-9-7950x3d.c3024
[2]: https://www.anandtech.com/show/15219/early-tsmc-5nm-test-chi...
[3]: https://www.tomshardware.com/reviews/amd-ryzen-9-7950x3d-cpu...
Imagine the heat sink on that thing. Would look like a cast-iron Dutch oven :)
"We call it the engine block because it somewhat resembles a three cylinder motorcycle engine"
(referring to the power distribution and cooling)
I'm sure those TSVs connect to a huge array of switching power supplies, so the 24kW doesn't travel very far at such low voltages.
Maybe it wouldn't be as powerful as one of these, due to their less capable fabs, but something that's good enough to get the job done in spite of the embargoes.
Yes.
- non-sparse fp16 in WSE-2 was 7.5 tflops (about 8 H100s, 10x worse performance per dollar)
Does anyone know the WSE-3 numbers? Datasheet seems lacking loads of details
Also, 2.5 million USD for 1 x WSE-3, why just 44GB tho???
I mean, in a cluster you might have a bunch of nodes with 8x GPUs hanging off each, if this thing replaces a whole node rather than a single GPU, which I assume is the case, it is not really a useful comparison, right?
You can order one with 1.2 Petabytes of external memory. Is that enough?
"External memory: 1.5TB, 12TB, or 1.2PB"
https://www.cerebras.net/press-release/cerebras-announces-th...
"214Pb/s Interconnect Bandwidth"
https://www.cerebras.net/product-system/
https://f.hubspotusercontent30.net/hubfs/8968533/Virtual%20B...
https://www.cerebras.net/blog/whats-new-in-r0.6-of-the-cereb...
"CSL allows for compile time execution of code blocks that take compile-time constant objects as input, a powerful feature it inherits from Zig, on which CSL is based. CSL will be largely familiar to anyone who is comfortable with C/C++, but there are some new capabilities on top of the C-derived basics."
https://github.com/Cerebras/csl-examples
And far fewer blinking lights.
Each parameter is a connection between artificial neurons. For example, inside an AI model, a linear layer that transforms an input vector with 1024 elements to an output vector with 2048 elements has 1024×2048 = ~2M parameters in a weight matrix. Each parameter specifies by how much each element in the input vector contributes to or subtracts from each element in the output vector. Each output vector element is a weighted sum (AKA a linear combination), of each input vector element.
A human brain has an estimated 100-500 trillion synapses connecting biological neurons. Each synapse is quite a complicated biological structure[a], but if we oversimplify things and assume that every synapse can be modeled as a single parameter in a weight matrix, then the largest AI models in use today have approximately 100T to 500T ÷ 0.5T = 200x to 1000x fewer connections between neurons that the human brain. If the company's claims prove true, this new chip will enable training of AI models that have only 4x to 20x fewer connections that the human brain.
We sure live in interesting times!
---
[a] https://en.wikipedia.org/wiki/Synapse
Which, it probably can't... but offsetting those simplifications and 4-20x difference is the massive difference in how quickly those synapses can be activated.
So only 4-20 of these systems are necessary to match the human brain. No?
...
It's meaningless to say something can train a model that has 24 trillion parameters without specifying the dataset size and time it takes to train.
How many early supercomputers / workstations etc would that include? How much progress did humanity make using all those early machines (or any transistorized device!) combined?
4004 from the 1970s used 2300 transistors, so it would have needed to sell billions.
Pentium from 1990s had 3M transistors, so it could hit our target by selling a million units.
I'm betting (without much research) that the Pentium line alone sold millions, and the industry as a whole could hit those numbers about 5 years earlier.
I trust that gamers will outlast every hype, be it crypto or AI.
* Power Usage
* Rack Size (last one I played with was 17u)
* Cooling requirements