Crawl Frequency per URL (Hits and Days Since Last Bot Visit)

Ranks URLs by crawler hits and shows first seen, last seen, days since the last visit and the average gap between visits.

Use the tool

Why this tool exists

Important pages should be revisited often. Pages that were fetched once and never again may be low value or poorly linked, and pages you changed recently should show recent visits.

The table is sortable by most crawled, by longest time since the last visit, or by pages crawled only once. Days since the last visit are counted from the newest line in your log, not from today, so an old export still gives meaningful numbers.

How to use it

  1. Paste access log lines.
  2. Choose the bots and whether query strings count as separate URLs.
  3. Pick an order: most crawled, longest since last visit, or crawled once.
  4. Open the pages that matter most to you and check how recent their last visit is.

Frequently asked questions

What is a good crawl frequency?

There is no universal number. Compare your key pages with each other and with how often they change.

Why group query strings?

By default they are ignored so that one page is one row. Turn the option on to study parameter URLs.

Can I trust the bot names in the result?

Only as claims. The tool reads the user-agent text, and anyone can send a fake one. For anything that matters, check the IP addresses against the operator's published ranges or with reverse DNS.

Is my log uploaded?

No. The log is parsed in your browser and never sent anywhere. Logs contain IP addresses, so you may want to remove them first with the log privacy workbench.