Calculate the number of language model tokens for a string (blog.mastykarz.nl)

17 points by waldekm 1y ago ↗ HN

3 comments

[ 4.9 ms ] story [ 36.3 ms ] thread

  import tiktoken
  enc = tiktoken.encoding_for_model('o1-mini')
  len(enc.encode('some string'))

Does that work on anything other than openai?

No but most models have their own libs you can use