robots.txt Compliance Log Checker (Bots Hitting Disallowed Paths)

Applies your robots.txt to the bot requests in a log and counts the ones that hit disallowed paths.

Use the tool

Why this tool exists

A rule in robots.txt only helps if crawlers follow it. Your logs show who does not: bots that request paths you disallowed.

Paste the robots.txt and a combined-format log. Bots with a named group are judged by it, the rest by the * group. Caveats are listed: the rules may have changed after the visit, crawlers cache robots.txt, and user-triggered fetchers are expected to ignore it.

How to use it

  1. Paste the robots.txt.
  2. Paste log lines.
  3. Read hits on disallowed paths per bot.
  4. Block repeat offenders at the server.

Frequently asked questions

What does served mean?

The request got a status below 400, so the content was delivered.

Why might a compliant bot show hits?

It may have cached an older robots.txt from before your change.

Are fetchers like ChatGPT-User violators?

They act for a user and may not follow robots.txt by design.

Is the user-agent trustworthy?

No. It can be forged.