Tokenization Is the Bottleneck You're Not Measuring (ranvier.systems) 1 points by mindsaspire 3mo ago ↗ HN
[–] olaird25 3mo ago ↗ Have you tried just using a faster tokenization library?Github's BPE crate (https://crates.io/crates/bpe) advertises >10x speedup relative to HF, as do others.
1 comment
[ 0.20 ms ] story [ 13.1 ms ] threadGithub's BPE crate (https://crates.io/crates/bpe) advertises >10x speedup relative to HF, as do others.