NTK-Aware Scaled RoPE allows models to have extended context without finetuning (old.reddit.com) 1 points by ianbutler 3y ago ↗ HN
[–] ianbutler 3y ago ↗ I had to edit the title a bit to fit the character limits, original:NTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.---------This is a claimed improvement over the recent Superhot scaling blog post.
1 comment
[ 0.23 ms ] story [ 12.7 ms ] threadNTK-Aware Scaled RoPE allows LLaMA models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation.
---------
This is a claimed improvement over the recent Superhot scaling blog post.