Hello (author here), I found the OpenAI tool (https://platform.openai.com/tokenizer
) for visualising tokens really useful when working on token optimisation strategies (and other token related tasks). But unfortunately that site doesn't support the tokeniser that GPT4 uses (currently).
So I threw together a stackblitz that uses the js-tiktoken to do the same thing.
1 comment
[ 4.9 ms ] story [ 15.5 ms ] threadSo I threw together a stackblitz that uses the js-tiktoken to do the same thing.
Hopefully it will be useful for someone else too!
Try it here: https://react-ts-h9yrjh.stackblitz.io/