Ask HN: Didactic Implementation of the Attention Mechanism? 1 points by extasia 3y ago ↗ HN Does anybody have a good reference implementation of the Attention mechanism? Preferably one that's easy to understand
[–] jstx1 3y ago ↗ For completeness - Formal Algorithms for Transformers - https://arxiv.org/pdf/2207.09238.pdfFor accessibility - Let's Build GPT from Scratch, Andrej Karpathy - https://www.youtube.com/watch?v=kCc8FmEb1nYYou can also read the PyTorch source code - https://pytorch.org/docs/stable/_modules/torch/nn/modules/tr...
1 comment
[ 3.7 ms ] story [ 9.5 ms ] threadFor accessibility - Let's Build GPT from Scratch, Andrej Karpathy - https://www.youtube.com/watch?v=kCc8FmEb1nY
You can also read the PyTorch source code - https://pytorch.org/docs/stable/_modules/torch/nn/modules/tr...