> If you want low latency you have to be really careful with HBM, not only because of the delay involved, but also the non-determinacy. One of the huge benefits of our LPU architecture is that we can build systems of hundreds of chips with fast interconnect and we know the precise timing of the whole system to within a few parts per million. Once you start integrating non-deterministic components your latency guarantees disappear very quickly.
The reason HBM "isn't deterministic" is because the memory controller sometimes stalls reads when doing a refresh operation. It's possible to pause refreshes [1] or schedule refreshes ahead of time to get HBM and DRAM to act deterministic.
To be fair, that's a huge hassle, and I have no clue what level of support commercial HBM controllers have for refresh pausing or manual refresh scheduling. It could very well be impossible to do in practice -- and regardless, their performance numbers speak for themselves :)
6 comments
[ 3.1 ms ] story [ 25.8 ms ] thread> If you want low latency you have to be really careful with HBM, not only because of the delay involved, but also the non-determinacy. One of the huge benefits of our LPU architecture is that we can build systems of hundreds of chips with fast interconnect and we know the precise timing of the whole system to within a few parts per million. Once you start integrating non-deterministic components your latency guarantees disappear very quickly.
https://news.ycombinator.com/item?id=39432119
To be fair, that's a huge hassle, and I have no clue what level of support commercial HBM controllers have for refresh pausing or manual refresh scheduling. It could very well be impossible to do in practice -- and regardless, their performance numbers speak for themselves :)
[1] https://memlab.ece.gatech.edu/papers/TACO_2014_1.pdf
(I work for Groq)