Heat is mostly driven by leakage current and gate capacitance. The big issue today is leakage currents. They typical account for around 30%-50% of total chip thermal budget, and they get increasingly difficult to…
If you have two threads on different cores that write to the same cacheline, the CPU has to enforce write ordering. The way it does this is for one of the cores to acquire a write lock on the cacheline. The actual…
Is there really that big a different in TFLOPS between the GB100 and GB202 chips? The GB100 has fewer SMs than the GB202, so I'm confused about where the 10x performance would be coming from?
Land the western australia wheat belt sells for less than $1000/acre. Is that very expensive?
Yes that bugged me too. If you replace 'precisely' with 'approximately' everywhere in the article it becomes much improved ;)
There's basically a difference in philosophy. GPU chips have a bunch of cores, each of which is semi-capable, whereas TPU chips have (effectively) one enormous core. So GPUs have ~120 small systolic arrays, one per SM…
The hand-waving explanation: The slower you're going, the easier (cheaper) it is to change direction. And for eliptical orbits, the outer-most part of the orbit is where you're going slow. So to make a drastic change in…
Not quite: It's taking advantage of (1+a)(1+b) = 1 + a + b + ab. And where a and b are both small-ish, ab is really small and can just be ignored. So it turns the (1+a)(1+b) into 1+a+b. Which is definitely not the same!…
I had the same thought: Just eye-balling the graphs, the result of the subtraction looks very close to just reducing the temperature. They're effectively doing softmax with a fixed temperature, but it's unclear that…
The paper says "... optimized on next-word prediction only". Which is absolutely correct in 2023. ChatGPT (and indeed all recent LLMs) using much more complex training methods than simply 'next-word prediction'.
Like most things, it's more complex than that, and as a result it can be either faster or slower than 'median(RTT to each DC in quorum)'. It's a delicate balance based on the locations that rows are being read and…
> Gradient accumulation doesn't work with batch norms so you really need that memory. Last I looked, very few SOTA models are trained with batch normalization. Most of the LLMs use layer norms which can be accumulated?…
It's hard to convey just how ridiculously complex and expensive 5nm is compared with 90nm. 5nm is a multi-billion dollar fab, absurdly high running costs, and very expensive wafers. As a data point: 5nm lithography…
Excellent example: "reflective cistern tank with a reflection of the back of a truck transporting stop signs" People tend to forget just how hard the edge cases in vision are!
Do you know why the blood thinners helped? (I'm assuming there was some underlying condition that this treated?)
The problem is that predicting a pixel requires knowing what the pixels around it looks like. But if we start with lots of noise, then the neighboring pixels are all just noise and have no signal. You could also think…
Minimum is around 1 kWh/m^3 for sea-water levels of salt concentration. (It varies a fair bit depending on salinity, the actual salts involved, the temperature etc etc).
This is surprisingly poor production? Peak insolation varies widely, but 1000W/m^2 is a typical value. 5.8L/hr/m^2 means that it's using something like 180kWh/m^3 on raw solar insolation. For comparison, reverse osmosis…
Nuts, you are correct that it isn't optimal. It's not quite as simple as just 2 bits per digit, because the last digit can only be large if the earlier digits are small. This makes the worst case is (1,2,6) which…
For the first case, you can't pick a number lower than any existing number. So if 4 is already picked, then only 5 or 6 can be picked, but if we pick 6, then there's no third number available to pick, so there only 1…
Yes. The formulation in the blog post is a bit messy, but possibly more general way to think about it is something like: 0. set y = 1. set x = 0. 1. How many possible choices are there _given what we've picked so far_?…
It's still interesting as it reflects that despite 60,000 images, there's a very small amount of data that the network actually learns. The total entropy in 10 images (even carefully engineered ones) is very low in…
The difficultly here is that there's an implicit assumption: 'Noise' is implicitly defined to be anything that isn't learned by the network. Now in same cases that may indeed be actual process noise! But there are many…
Can't up-vote this hard enough!
'half' is ambiguous here: They lose 1 bit of expressiveness for the mantissa, and 1 bit in the exponent, giving an efficiency loss of 2 bits from 32 == ~3%. So there's very little loss in using single-precision floating…
Heat is mostly driven by leakage current and gate capacitance. The big issue today is leakage currents. They typical account for around 30%-50% of total chip thermal budget, and they get increasingly difficult to…
If you have two threads on different cores that write to the same cacheline, the CPU has to enforce write ordering. The way it does this is for one of the cores to acquire a write lock on the cacheline. The actual…
Is there really that big a different in TFLOPS between the GB100 and GB202 chips? The GB100 has fewer SMs than the GB202, so I'm confused about where the 10x performance would be coming from?
Land the western australia wheat belt sells for less than $1000/acre. Is that very expensive?
Yes that bugged me too. If you replace 'precisely' with 'approximately' everywhere in the article it becomes much improved ;)
There's basically a difference in philosophy. GPU chips have a bunch of cores, each of which is semi-capable, whereas TPU chips have (effectively) one enormous core. So GPUs have ~120 small systolic arrays, one per SM…
The hand-waving explanation: The slower you're going, the easier (cheaper) it is to change direction. And for eliptical orbits, the outer-most part of the orbit is where you're going slow. So to make a drastic change in…
Not quite: It's taking advantage of (1+a)(1+b) = 1 + a + b + ab. And where a and b are both small-ish, ab is really small and can just be ignored. So it turns the (1+a)(1+b) into 1+a+b. Which is definitely not the same!…
I had the same thought: Just eye-balling the graphs, the result of the subtraction looks very close to just reducing the temperature. They're effectively doing softmax with a fixed temperature, but it's unclear that…
The paper says "... optimized on next-word prediction only". Which is absolutely correct in 2023. ChatGPT (and indeed all recent LLMs) using much more complex training methods than simply 'next-word prediction'.
Like most things, it's more complex than that, and as a result it can be either faster or slower than 'median(RTT to each DC in quorum)'. It's a delicate balance based on the locations that rows are being read and…
> Gradient accumulation doesn't work with batch norms so you really need that memory. Last I looked, very few SOTA models are trained with batch normalization. Most of the LLMs use layer norms which can be accumulated?…
It's hard to convey just how ridiculously complex and expensive 5nm is compared with 90nm. 5nm is a multi-billion dollar fab, absurdly high running costs, and very expensive wafers. As a data point: 5nm lithography…
Excellent example: "reflective cistern tank with a reflection of the back of a truck transporting stop signs" People tend to forget just how hard the edge cases in vision are!
Do you know why the blood thinners helped? (I'm assuming there was some underlying condition that this treated?)
The problem is that predicting a pixel requires knowing what the pixels around it looks like. But if we start with lots of noise, then the neighboring pixels are all just noise and have no signal. You could also think…
Minimum is around 1 kWh/m^3 for sea-water levels of salt concentration. (It varies a fair bit depending on salinity, the actual salts involved, the temperature etc etc).
This is surprisingly poor production? Peak insolation varies widely, but 1000W/m^2 is a typical value. 5.8L/hr/m^2 means that it's using something like 180kWh/m^3 on raw solar insolation. For comparison, reverse osmosis…
Nuts, you are correct that it isn't optimal. It's not quite as simple as just 2 bits per digit, because the last digit can only be large if the earlier digits are small. This makes the worst case is (1,2,6) which…
For the first case, you can't pick a number lower than any existing number. So if 4 is already picked, then only 5 or 6 can be picked, but if we pick 6, then there's no third number available to pick, so there only 1…
Yes. The formulation in the blog post is a bit messy, but possibly more general way to think about it is something like: 0. set y = 1. set x = 0. 1. How many possible choices are there _given what we've picked so far_?…
It's still interesting as it reflects that despite 60,000 images, there's a very small amount of data that the network actually learns. The total entropy in 10 images (even carefully engineered ones) is very low in…
The difficultly here is that there's an implicit assumption: 'Noise' is implicitly defined to be anything that isn't learned by the network. Now in same cases that may indeed be actual process noise! But there are many…
Can't up-vote this hard enough!
'half' is ambiguous here: They lose 1 bit of expressiveness for the mantissa, and 1 bit in the exponent, giving an efficiency loss of 2 bits from 32 == ~3%. So there's very little loss in using single-precision floating…