Gigatoken: Fastest Tokenizer (twitter.com)

GitHub: https://github.com/marcelroed/gigatoken

[dead]

I kind of assumed the model would process the text 'directly', from what I understand, wouldn't this be biasing the input based on how you tokenise as it's lossy?

I assume this tradeoff is purely for speed/compression. Or am I missing what's going on here?

Doesn't really explain why its faster. "Optimizing for every kind of CPU" is not really enough info.

Cool project nonetheless, I will go through the code later tomorrow