robots.txt Minifier (Strip Comments and Duplicates)

Produces a compact robots.txt that behaves the same: no comments, no blank lines, no duplicate rules and merged groups.

Use the tool

Why this tool exists

If your robots.txt is approaching the size limit or you serve it at scale, comments and repeated rules are dead weight. Rewriting by hand risks changing behaviour.

The minifier removes only what is safe: comments, blank lines, exact duplicate rules, and merges groups with identical user-agent sets. It deliberately keeps rules that look redundant, because their effect can depend on Allow rules. It shows the bytes saved.

How to use it

  1. Paste the robots.txt.
  2. Copy the compact output.
  3. Check the bytes saved.
  4. Test key URLs on the new file before publishing.

Frequently asked questions

Is the result equivalent?

Yes for groups, rules and sitemaps. The tool does not remove rules that merely seem redundant.

Are comments lost?

Yes. Keep a commented source copy elsewhere.

Why is a blank Disallow added?

An empty group would otherwise merge into the next one, so an empty Disallow keeps it separate.

Is minifying needed?

Rarely. It matters near the size limit.