Turning any tokenizer into a greedy one (stephantul.github.io)

2 points by stephantul 1y ago ↗ HN

1 comment

[ 4.8 ms ] story [ 16.0 ms ] thread

stephantul 1y ago ↗

I recently did some work on making tokenizers greedy.