Bulk robots.txt URL Checker (CSV Export)

Tests a long list of URLs against a robots.txt for one or more crawlers and gives a table or a CSV file.

Use the tool

Why this tool exists

After a robots.txt change you want to know which of your important URLs are now blocked, for Googlebot, Bingbot and AI crawlers. Testing them one at a time does not scale.

Paste the file and a URL list, name the crawlers, and get the verdict and deciding rule for every pair. Switch to CSV to open it in a spreadsheet. The same longest-match rules as the single-URL tester apply.

How to use it

  1. Paste the robots.txt.
  2. Paste the URLs, one per line, from a sitemap or crawl.
  3. List the crawlers you care about.
  4. Choose table or CSV and copy or download.

Frequently asked questions

How many URLs can I test?

Up to 50,000 per run.

Can I see only blocked URLs?

Yes, tick the Show only blocked option.

Which rule decides?

The longest matching pattern, with Allow winning a tie.

Where do I get a URL list?

The sitemap URL extractor turns a sitemap into a list.