Reddit's caches are set up to only ever return the last 1,000 of anything. So for example - you can't scroll past 1k items on /new, and if you save more than 1k posts then you'll have to unsave some to retrieve the…
Related comment from gwern: https://news.ycombinator.com/item?id=38438859. Can't find the docs now - I think they were the old GPT 3 ones - but they suggested a low value somewhere around 0.01 and 0.1. Also - why qlora…
That's what I ended up doing (`[Author] username [Title] post title...`) > Adding new tokens needs a ton of data to train what the token means. But how much? 300M tokens is fine for a simple version of ChatML with ~4…
I tried adding special tokens for a reddit-style dataset once. The format was: `<|post_author|>username<|post_title|>title here...` The resulting model was so much worse than just formatting everything plaintext. This…
Thanks for writing up. Rather than zeroing out the loss for the prompt, did you also try using weighted loss with Axolotl? At one point, Microsoft's GPT 3 docs suggested this was beneficial when the responses are short…
There's a breakdown here for anyone interested (ctrl+f "weight flops for") https://medium.com/@dzmitrybahdanau/the-flops-calculus-of-la...
The prompt for Bing Chat was previously reproduced by the same person as here, using the same trick. The Bing lead disclaimed it as inaccurate, though: https://twitter.com/MParakhin/status/1627491603731423232
Two other recent literature reviews worth reading: "Transformer Taxonomy" - https://kipp.ly/blog/transformer-taxonomy/ "Five years of progress in GPTs" - https://finbarrtimbers.substack.com/p/five-years-of-progress...
Reddit's caches are set up to only ever return the last 1,000 of anything. So for example - you can't scroll past 1k items on /new, and if you save more than 1k posts then you'll have to unsave some to retrieve the…
Related comment from gwern: https://news.ycombinator.com/item?id=38438859. Can't find the docs now - I think they were the old GPT 3 ones - but they suggested a low value somewhere around 0.01 and 0.1. Also - why qlora…
That's what I ended up doing (`[Author] username [Title] post title...`) > Adding new tokens needs a ton of data to train what the token means. But how much? 300M tokens is fine for a simple version of ChatML with ~4…
I tried adding special tokens for a reddit-style dataset once. The format was: `<|post_author|>username<|post_title|>title here...` The resulting model was so much worse than just formatting everything plaintext. This…
Thanks for writing up. Rather than zeroing out the loss for the prompt, did you also try using weighted loss with Axolotl? At one point, Microsoft's GPT 3 docs suggested this was beneficial when the responses are short…
There's a breakdown here for anyone interested (ctrl+f "weight flops for") https://medium.com/@dzmitrybahdanau/the-flops-calculus-of-la...
The prompt for Bing Chat was previously reproduced by the same person as here, using the same trick. The Bing lead disclaimed it as inaccurate, though: https://twitter.com/MParakhin/status/1627491603731423232
Two other recent literature reviews worth reading: "Transformer Taxonomy" - https://kipp.ly/blog/transformer-taxonomy/ "Five years of progress in GPTs" - https://finbarrtimbers.substack.com/p/five-years-of-progress...