What the model is doing in latent space is auxilliary to anthropomorphic interpretations of the tokens, though. And if the latent reasoning matches a ground-truth procedure (A*), then we'd expect it to be projectable to…
It's more about the lack of semantic meaning in the intermediate tokens, not that they aren't effective (even when the intermediates are wrong)
I like to think of the intermediate tokens as low-dimensional hidden states. Also see the Coconut paper/citation
Exactly, the traces lack semantics and shouldn't be anthropomorphized. (I'm one of the students in the lab that wrote this, but not one of the authors)
What the model is doing in latent space is auxilliary to anthropomorphic interpretations of the tokens, though. And if the latent reasoning matches a ground-truth procedure (A*), then we'd expect it to be projectable to…
It's more about the lack of semantic meaning in the intermediate tokens, not that they aren't effective (even when the intermediates are wrong)
I like to think of the intermediate tokens as low-dimensional hidden states. Also see the Coconut paper/citation
Exactly, the traces lack semantics and shouldn't be anthropomorphized. (I'm one of the students in the lab that wrote this, but not one of the authors)