Sitemap vs robots.txt Cross-Check

Finds sitemap URLs that robots.txt blocks for the crawler you choose.

Use the tool

Why this tool exists

A URL in the sitemap that robots.txt blocks sends mixed signals. The crawler cannot read the page, yet you asked it to.

The tool applies the usual matching: the most specific user-agent group, the longest matching path, Allow winning a tie, and * and $ patterns. It matches the path and query only, not the host.

How to use it

  1. Paste the sitemap or URL list.
  2. Paste robots.txt.
  3. Set the crawler name, such as Googlebot.
  4. Review the blocked URLs and rules.

Frequently asked questions

Which crawler should I test?

Start with Googlebot and test others you care about.

Is the host checked?

No. robots.txt applies per host, so paste the matching one.

What if a URL is blocked on purpose?

Remove it from the sitemap.

Is this exact?

It follows the common rules but a live fetch is the final test.