The Tokenizer Playground
A page that colors in how your sentence is split into tokens inside an AI model. You can pick and compare the tokenizers of 13 models including GPT-4, Claude, Llama 3, Gemma, Mistral, BERT, and T5.
How to try it
Choose a model in the select box at the top and type a sentence in the field below; the tokens appear as colored chunks with a count. Try the same sentence in Korean and in English and you will see Korean split into far more tokens, which is what drives cost and length limits when using Korean. The tokenizer files are served from our own site, so no sign-up or outside connection is needed.