OpenAI's Tokenizer Page for OSS Models (github.com)

I kept having to debug prompt issues with open models

So I built OpenAI's Tokenizer page for all tokenizers on HuggingFace: Llama, Mistral, GPT2, MPT, Persimmon, T5 etc

http://tokenwiz.rahul.gs