HN Explorer

11 comments

[ 3.3 ms ] story [ 231 ms ] thread
sebzim4500
It's insane that FlashAttention was released 16 months ago. It feels like a decade.
[deleted]
(comment deleted)
ttul
It’s basically a way of more efficiently making use of memory transfers during the calculation of the attention blocks in a transformer. You transfer a block at a time, increasing inference throughout because less time is spent overall fetching things from slow memory.
1024core
Has anybody used FlashAttention in their model? Are there any benchmark numbers on the quality impact?
pama
The result is identical to regular attention in transformers but training can be about four times faster, so there is almost no reason to not use it.
NLPaep
Not quite. There can be non-deterministic race conditions, and strange head size and sequence length requirements.
pama
Yes. For a model within the limits of the head requirements, however, you wouldn’t be able to see a quality difference from regular attention. Non determinism is a performance price; regular transformers may also suffer from it depending on the implementation.
fzysingularity
Also, isn't the author Tri Dao at Together AI now as their chief scientist?