Truncate / Chunk Text by Tokens (tiktoken)

Truncates text to a maximum number of tokens, or splits it into chunks of at most N tokens with optional overlap. Cuts fall on token boundaries that keep multi-byte characters whole.

Use the tool

Why this tool exists

Models have a context limit, and embedding endpoints reject input that is too long. Cutting by characters is a poor proxy: a thousand characters of English and a thousand of Japanese use very different token counts, so a slice that fits one document overflows the next.

This tool cuts by real tokens using the OpenAI cl100k_base or o200k_base vocabularies. In truncate mode it keeps the first N tokens. In chunk mode it produces consecutive pieces of at most N tokens, with an overlap you control so context is not lost between pieces. Each boundary is moved back if needed so that no emoji or accented letter is cut in half. The work runs in your browser and the text is not uploaded.

How to use it

  1. Paste your text into the input box.
  2. Choose the encoding and the mode: truncate or chunks.
  3. Set the maximum tokens N and, for chunks, the overlap in tokens.
  4. Copy the truncated text or the numbered chunks.

Frequently asked questions

How does overlap work?

Each new chunk starts a few tokens before the previous one ended, so the end of one chunk appears again at the start of the next. The overlap must be smaller than the chunk size or an error is shown.

Why is a chunk slightly shorter than N?

If a cut would fall inside a multi-byte UTF-8 character, the boundary moves back to the nearest safe token. So a chunk can hold fewer than N tokens, and the summary notes it for truncation.

Does it split on sentences or paragraphs?

No. It cuts purely on token counts. For structure-aware splitting, use the text chunk splitter, or the RAG chunking visualiser.

Is the token count exact?

It is exact for the OpenAI encodings listed. Other models tokenise differently, so a chunk that fits cl100k_base may be a bit over or under in another tokenizer.