Tiktokenizer (Token Splitting Explorer)
A tool for seeing how a language model splits text into tokens. Type something and it immediately shows the token count, the colour-coded token pieces and the numeric ID of each token. You can also see the special markers that wrap a conversation before it reaches the model.
How to try it
Pick a model such as gpt-4o at the top left, then type into the System and User fields — the token count, the token pieces and the list of numeric IDs on the right update instantly. Use 'Add message' to extend the conversation, and turn on 'Show whitespace' to see how spaces attach to tokens. Entering Korean text is a good way to compare how many more tokens it costs than English; the interface is English only.