robots.txt Fetch Monitor (Bot Requests, Status and Timing from Logs)

Extracts robots.txt requests from your log and shows when each bot fetched the file, what you answered and how fast.

Use the tool

Why this tool exists

Crawlers read robots.txt before they crawl, and how your server answers it affects what they do next. A file that returns errors or times out is something you want to know about early.

The tool shows a table of fetches per day and bot, the latest fetches with status and bytes, and the response time if your log has one. Look for non-200 answers, for sudden changes in file size and for bots that never ask for the file at all.

How to use it

  1. Paste access log lines, ideally from several days.
  2. Choose the bots, or all known bots.
  3. Pick the unit of the response time number if your log has one.
  4. Check the status codes and the latest fetches.

Frequently asked questions

Why does a robots.txt 5xx matter?

Crawlers handle server errors on this file differently from a missing file. Check each vendor's robots.txt documentation for what they do.

My log has no robots.txt lines.

The file may be served by a CDN or another host, or the period is too short.

Does a fetch mean the rules were obeyed?

No. It only shows the file was requested. Use the compliance log checker to test the rules against requests.

Can I trust the bot names in the result?

Only as claims. The tool reads the user-agent text, and anyone can send a fake one. For anything that matters, check the IP addresses against the operator's published ranges or with reverse DNS.