Sitemap vs robots.txt Cross-Check
Finds sitemap URLs that robots.txt blocks for the crawler you choose.
Use the tool
Why this tool exists
A URL in the sitemap that robots.txt blocks sends mixed signals. The crawler cannot read the page, yet you asked it to.
The tool applies the usual matching: the most specific user-agent group, the longest matching path, Allow winning a tie, and * and $ patterns. It matches the path and query only, not the host.
How to use it
- Paste the sitemap or URL list.
- Paste robots.txt.
- Set the crawler name, such as Googlebot.
- Review the blocked URLs and rules.
Frequently asked questions
Which crawler should I test?
Start with Googlebot and test others you care about.
Is the host checked?
No. robots.txt applies per host, so paste the matching one.
What if a URL is blocked on purpose?
Remove it from the sitemap.
Is this exact?
It follows the common rules but a live fetch is the final test.