A mathematician's introduction to transformers and large language models (x-dev.pages.jsc.fz-juelich.de)

This is a very good summary with great narrative, helpful diagrams, and very approachable math explained clearly. Good references. Well worth the read.