robots.txt Diff Viewer (Rule Changes per User-agent)

Compares an old and a new robots.txt by meaning, not by text, and lists the rules added or removed for each user-agent and each Sitemap line.

Use the tool

Why this tool exists

A normal text diff buries the real change under moved lines, new comments and different spacing. What you need to know after a deploy is which crawler can now fetch or no longer fetch which paths.

This tool merges groups the way crawlers do, then compares the rules for each user-agent. If only comments or order changed, it says there are no rule changes. Paste archived copies to see how a policy changed over time; the site itself makes no network requests.

How to use it

  1. Paste the old robots.txt in the first box.
  2. Paste the new one in the second box.
  3. Read the changes grouped by user-agent.
  4. Run the new file through the tester for your key URLs.

Frequently asked questions

Does the order of rules matter?

Not for Google or RFC 9309 crawlers: the longest matching rule wins. The diff ignores order for that reason.

Are two groups for the same bot merged?

Yes. Crawlers combine them, so the diff does too before comparing.

Can it fetch old versions?

No, that needs a network request. Copy archived versions from a web archive and paste them.

Does it show comment changes?

No. Comments do not change behaviour, so they are left out.