[–] jakecyr 2y ago ↗ The first (that I found) tokenizer for open source LLMs. It retrieves config files from HuggingFace and can encode / decode text and tokens.Was created in an hour and can definitely use some work. Would love contributions and feedback!
1 comment
[ 3.3 ms ] story [ 11.4 ms ] threadWas created in an hour and can definitely use some work. Would love contributions and feedback!