https://arxiv.org/abs/2310.02207 https://transformer-circuits.pub/2024/scaling-monosemanticit...
Of course. A trademark exists to mutually protect consumers and businesses from deceptive advertising. When a term referring to a specific product becomes a term for a product category etc, trademark protections then…
If you have an evaluation function which does this accurately and generalizes, you pretty much already have have AGI.
Gary Marcus was arguing in 2020 that scaling up GPT-2 wouldn't result in improvements in common sense or reasoning. He was wrong, and he continues to be wrong. It's called the bitter lesson for a reason. Nobody likes…
From reporting GPT-5 finished pre-training while ago and was in the process of red-teaming.
Assuming there is a development that makes GPUs obsolete, I think it's safe to assume that what will replace them at scale will still take the form dedicated AI card/rack 1. Tight integration necessary for fundamental…
Because attention is all you need. I.E. the KV cache is 'just' a time saving measure because an LLM goes back and calculates those values anyway. (Which is why per-token compute increases exponentially otherwise) You're…
>Why it's maximal is not in the model at all, nor the data >It replays the data to us and we suppose the LLM must have the property that generates this data originally. So to clarify, what you're saying is that under…
The modelling is advanced enough that you can't fundamentally distinguish it from (lossy, limited) planning in the way you're describing. If the KQV doesn't encode information about likely future token sequences then a…
Transformers are still stateless, KV cache is just a compute-saving measure (but otherwise correctly described)
In the case of 3sum, because the LLM has been fine tuned to use 'blank' token key values as a register to represent the sums of specific integer triplets.
https://arxiv.org/abs/2310.02207 https://transformer-circuits.pub/2024/scaling-monosemanticit...
Of course. A trademark exists to mutually protect consumers and businesses from deceptive advertising. When a term referring to a specific product becomes a term for a product category etc, trademark protections then…
If you have an evaluation function which does this accurately and generalizes, you pretty much already have have AGI.
Gary Marcus was arguing in 2020 that scaling up GPT-2 wouldn't result in improvements in common sense or reasoning. He was wrong, and he continues to be wrong. It's called the bitter lesson for a reason. Nobody likes…
From reporting GPT-5 finished pre-training while ago and was in the process of red-teaming.
Assuming there is a development that makes GPUs obsolete, I think it's safe to assume that what will replace them at scale will still take the form dedicated AI card/rack 1. Tight integration necessary for fundamental…
Because attention is all you need. I.E. the KV cache is 'just' a time saving measure because an LLM goes back and calculates those values anyway. (Which is why per-token compute increases exponentially otherwise) You're…
>Why it's maximal is not in the model at all, nor the data >It replays the data to us and we suppose the LLM must have the property that generates this data originally. So to clarify, what you're saying is that under…
The modelling is advanced enough that you can't fundamentally distinguish it from (lossy, limited) planning in the way you're describing. If the KQV doesn't encode information about likely future token sequences then a…
Transformers are still stateless, KV cache is just a compute-saving measure (but otherwise correctly described)
In the case of 3sum, because the LLM has been fine tuned to use 'blank' token key values as a register to represent the sums of specific integer triplets.