owl_brawl
No user record in our sample, but owl_brawl has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
No user record in our sample, but owl_brawl has activity below (stories or comments). Likely we have partial data — the full bulk-load will fill profiles in.
For those who haven't read it, Rich Caruana's thesis on multi-task learning is beautifully written (the cited 1998 paper here). It's amazing to see how far the field has come, and, at the same time, how advanced the…
I would love answers to these questions too, particularly on the vocab size
Hi alekandreev, Any reason you decided to go with a token vocabulary size of 256k? Smaller vocab/vector sizes like most models in this size seem to be using (~16-32k) are much easier to work with. Would love to…
The "RAG" part (over the training set) is by far the smallest contribution to the performance gains reported (see the ablation study in section 5.2). I don't think the model is actually learning in-context from the…