> yet somehow people married the acronym to one very particular implementation of the idea. Likely due to the rise in popularity of semantic search via LLM embeddings, which for some reason became the main selling point…
Ad-free and fast, Wikipedia is imo the best learning resource on-the-go, at home, or at work.
> The information cost of making the RNN state way bigger is high when done naively, but maybe someone can figure out a clever way to avoid storing full hidden states in memory during training or big improvements in…
> It's astounding to me (and everyone else who's being honest) that LLMs can accomplish what they do when it's only linear "factors" (i.e. weights) that are all that's required to be adjusted during training, to achieve…
> I remember one of the initial transformer people saying in an interview that they didn't think this was the "one true architecture" but a lot of the performance came from people rallying around it and pushing in the…
> yet somehow people married the acronym to one very particular implementation of the idea. Likely due to the rise in popularity of semantic search via LLM embeddings, which for some reason became the main selling point…
Ad-free and fast, Wikipedia is imo the best learning resource on-the-go, at home, or at work.
> The information cost of making the RNN state way bigger is high when done naively, but maybe someone can figure out a clever way to avoid storing full hidden states in memory during training or big improvements in…
> It's astounding to me (and everyone else who's being honest) that LLMs can accomplish what they do when it's only linear "factors" (i.e. weights) that are all that's required to be adjusted during training, to achieve…
> I remember one of the initial transformer people saying in an interview that they didn't think this was the "one true architecture" but a lot of the performance came from people rallying around it and pushing in the…