robots.txt Compliance Log Checker (Bots Hitting Disallowed Paths)
Applies your robots.txt to the bot requests in a log and counts the ones that hit disallowed paths.
Use the tool
Why this tool exists
A rule in robots.txt only helps if crawlers follow it. Your logs show who does not: bots that request paths you disallowed.
Paste the robots.txt and a combined-format log. Bots with a named group are judged by it, the rest by the * group. Caveats are listed: the rules may have changed after the visit, crawlers cache robots.txt, and user-triggered fetchers are expected to ignore it.
How to use it
- Paste the robots.txt.
- Paste log lines.
- Read hits on disallowed paths per bot.
- Block repeat offenders at the server.
Frequently asked questions
What does served mean?
The request got a status below 400, so the content was delivered.
Why might a compliant bot show hits?
It may have cached an older robots.txt from before your change.
Are fetchers like ChatGPT-User violators?
They act for a user and may not follow robots.txt by design.
Is the user-agent trustworthy?
No. It can be forged.