Tiktoken Tokenizer Visualizer (cl100k / o200k)
Shows your text as coloured tokens using the exact OpenAI cl100k_base and o200k_base encodings, plus a table of token ids and bytes. It also reports token, character and byte counts.
Use the tool
Why this tool exists
A token count is only a number until you see where the splits fall. Why does a long number cost three tokens per group, why does an emoji take several, and why is a leading space attached to a word? Looking at the actual pieces answers those questions and helps you tighten prompts.
This visualizer implements the tiktoken byte-pair encoding for GPT-4 and GPT-3.5 (cl100k_base) and for GPT-4o and newer models (o200k_base). Each token is shaded, optionally labelled with its id, and listed in a table with its bytes. The vocabulary file is fetched from this same site on first use, and the tokenising happens in your browser. Your text is not uploaded. Only OpenAI encodings are included.
How to use it
- Paste or type your text into the input box.
- Choose an encoding: cl100k_base or o200k_base.
- Set special tokens to plain or allow, and tick the option to show ids in the coloured view.
- Read the token count and the colour-coded breakdown, then check the table of ids and bytes.
Frequently asked questions
Are these the same counts the API uses?
For the text itself, yes, the encodings are exact implementations of OpenAI's public vocabularies. Billed totals for chat requests also include message framing and other overhead, which this view does not add.
What do plain and allow mean for special tokens?
In plain mode, text like the end-of-text marker is tokenised as ordinary characters, which is how most API text is treated. In allow mode, it is turned into the single special token id.
Does my text leave the browser?
No. Only a vocabulary file is fetched from this site, once. After that the splitting runs locally, so your prompt stays on your device.
Why not Claude or Gemini?
Their tokenizer vocabularies are not available for offline use, so only the OpenAI encodings are included here. Use the token estimator for approximate counts for those models.