Multi-Tokenizer Token Counter (GPT, DeepSeek, Qwen, Mistral, Phi, ...)

Counts the tokens of your text under every included tokenizer and sorts them from fewest to most. It shows characters per token for each and flags the legacy Claude 1/2 encoding.

Use the tool

Why this tool exists

Different models cut the same text into very different numbers of tokens. A paragraph of Chinese, a block of code or a long number can cost noticeably more on one tokenizer than another, which changes both the price and how much fits in the context window.

Paste one text and compare the exact counts across GPT-2, GPT-3, GPT-4 and GPT-4o encodings and the DeepSeek, Qwen, Phi-4, GPT-NeoX, StarCoder2 and Mistral tokenizers. The vocabularies are loaded from this site on first use, which is a download of roughly 20 MB in total, then everything runs locally. The legacy Claude 1/2 tokenizer is included but is not accurate for Claude 3, 4 or 5 models. Gemini, Llama and Gemma are not included.

How to use it

  1. Paste your text into the input box.
  2. Choose whether special tokens are plain text or allowed.
  3. Run it and wait while the vocabularies load the first time.
  4. Compare the sorted table of token counts and characters per token.

Frequently asked questions

Why does it take longer the first time?

Each tokenizer needs its vocabulary file, which is downloaded from this site and cached. Together they are about 20 MB. Later runs in the same session reuse what has already loaded.

Which models are covered?

GPT-2, r50k, p50k, cl100k and o200k from OpenAI, DeepSeek-V3 and R1, Qwen2.5, Phi-4, GPT-NeoX-20B, StarCoder2, Mistral-7B-v0.3 and the legacy Claude 1/2 tokenizer. Gemini, Llama, Gemma and Claude 3 and later are not included.

Can I trust the Claude number?

No, not for current Claude models. It uses the old public tokenizer for Claude 1 and 2, which Anthropic has not replaced with a published vocabulary. It is marked as legacy in the results.

What if one tokenizer fails?

The row for that encoding shows the error message, and the others still appear. A failure is usually a network problem fetching its vocabulary.