bazlightyear

↗ HN profile [ 62.1 ms ] full profile
Karma
0
Created
()
Submissions
0
  1. A greedy phrase-based tokenizer that outperforms GPT-4 and GPT-4o tokenizers on compression, with a smaller vocabulary. Benchmark (enwik9, 1 GB): Tokenizer Vocab Size Total Tokens Ratio Throughput GreedyPhrase 65,536…