AI Crawler Policy Comparator (Compare robots.txt Files)

Builds a table of AI crawlers against the robots.txt files you paste.

Use the tool

Why this tool exists

To see where you stand, compare your AI crawler policy with other sites in your field. The policy is in each site's robots.txt, and archived copies work too.

Paste each file after a line === name and read the crawler-by-site table, with a count of blocked crawlers per site. The page makes no network requests, so you fetch the files yourself.

How to use it

  1. Open each robots.txt and copy it.
  2. Paste each after a === name line.
  3. Read the table.
  4. Adjust your own policy.

Frequently asked questions

Can it fetch other sites?

No. Cross-site fetching is not possible in a web page without a server.

What does via * mean?

The crawler is not named; the * rules apply.

Which crawlers are compared?

The ones in the AI crawler directory that were checked at the source.

Does a block mean the content is safe?

No. Blocking is advisory.