robots.txt Tester

Paste a robots.txt and a list of URLs, pick a user-agent such as Googlebot or Bingbot, and see which URLs are allowed or blocked and which rule decided it.

Use the tool

Why this tool exists

A single wrong line in robots.txt can hide a whole section of a site from search engines, or leave a private area open to crawlers. The rules are subtle: the longest matching path wins, wildcards and the end-of-URL dollar sign change matching, and different user-agent groups apply to different bots.

This tester parses the robots.txt you paste, finds the most specific group for the user-agent you choose, and checks each URL against its allow and disallow rules, showing the winning rule next to the result. It shows allowed and blocked counts at the top. It follows the common behaviour of major crawlers but is not the exact parser used by any search engine, so confirm important changes in Search Console. The test runs in your browser.

How to use it

  1. Paste your robots.txt into the first box.
  2. Paste the URLs you want to test, one per line, into the second box.
  3. Enter the user-agent in the options, for example Googlebot.
  4. Run the tool and review which URLs are blocked, and by which rule.

Frequently asked questions

How are conflicts between Allow and Disallow decided?

The rule with the longest matching path wins. If an Allow and a Disallow match with equal length, Allow wins. Wildcards using an asterisk and a trailing dollar sign are supported.

Which group does a user-agent use?

The most specific matching group applies, falling back to the asterisk group. Rules from other groups are not merged. If no group matches, everything is allowed.

Does blocking in robots.txt remove a page from search?

Not reliably. It stops crawling, but a blocked URL can still be indexed if other pages link to it. Use a noindex directive on a crawlable page to keep it out of results.

Is my robots.txt uploaded?

No. The calculation runs in your browser and nothing you paste is uploaded, stored or sent to a server, so drafts and confidential copy are safe to check.