bazlightyear
- Karma
- 0
- Created
- ()
- Submissions
- 0
- AI code looks clean. That's the Trap (thinkpol.ca)
- Claude vs. Gemini vs. ChatGPT vs. Grok (boreal.social)
-
A greedy phrase-based tokenizer that outperforms GPT-4 and GPT-4o tokenizers on compression, with a smaller vocabulary. Benchmark (enwik9, 1 GB): Tokenizer Vocab Size Total Tokens Ratio Throughput GreedyPhrase 65,536…