[–] jakecyr 2y ago ↗ The first (that I found) tokenizer for open source LLMs. It retrieves config files from HuggingFace and can encode / decode text and tokens.Was created in an hour and can definitely use some work. Would love contributions and feedback!
1 comment
[ 4.5 ms ] story [ 10.7 ms ] threadWas created in an hour and can definitely use some work. Would love contributions and feedback!