I spent the last weekend thinking about continual learning. A lot of people think that we can solve long term memory and learning in LLMs by simply extending the context length to infinity. I analyse a different perspective that challenges this assumption.
Also interesting to think about: could a single system be generally intelligent, or is a certain bias actually a power. Can we have billions of models, each with their own "experience"
7 comments
[ 4.5 ms ] story [ 33.2 ms ] threadI spent the last weekend thinking about continual learning. A lot of people think that we can solve long term memory and learning in LLMs by simply extending the context length to infinity. I analyse a different perspective that challenges this assumption.
Let me know how you think about this.
Well, I think of every Large Language Model as if it were a spectacularly faceted diamond.
More on these lines in a recent-ish "thinking in public" attempt by yours truly, lay programmer, to interpret what an LLM-machine might be.
Riff: LLMs are Software Diamonds
https://www.evalapply.org/posts/llms-are-diamonds/
It is also my very uninformed intuition: https://news.ycombinator.com/item?id=44910353
Also interesting to think about: could a single system be generally intelligent, or is a certain bias actually a power. Can we have billions of models, each with their own "experience"
Does this imply that a completely untrained model (random weights) should show intelligent behavior only by providing enough context?
www.aravindjayendran.com sent an invalid response.
ERR_SSL_PROTOCOL_ERROR