Bulk robots.txt URL Checker (CSV Export)
Tests a long list of URLs against a robots.txt for one or more crawlers and gives a table or a CSV file.
Use the tool
Why this tool exists
After a robots.txt change you want to know which of your important URLs are now blocked, for Googlebot, Bingbot and AI crawlers. Testing them one at a time does not scale.
Paste the file and a URL list, name the crawlers, and get the verdict and deciding rule for every pair. Switch to CSV to open it in a spreadsheet. The same longest-match rules as the single-URL tester apply.
How to use it
- Paste the robots.txt.
- Paste the URLs, one per line, from a sitemap or crawl.
- List the crawlers you care about.
- Choose table or CSV and copy or download.
Frequently asked questions
How many URLs can I test?
Up to 50,000 per run.
Can I see only blocked URLs?
Yes, tick the Show only blocked option.
Which rule decides?
The longest matching pattern, with Allow winning a tie.
Where do I get a URL list?
The sitemap URL extractor turns a sitemap into a list.