Googlebot Crawl Budget Analyzer (Hits by Section, Template and Day)

Counts crawler requests per site section or URL template, with unique URLs, hits per URL, status classes and a daily total.

Use the tool

Why this tool exists

Search Console shows totals. Your server log shows where each request went. If one folder takes a third of all crawler hits but holds a handful of useful pages, that is where crawl effort leaks.

The tool groups paths by folder, by two folder levels, or by a URL template where numbers and slugs are folded together. A high hits-per-URL value means the same URLs are fetched again and again. The first and last day of a log are often partial, so read the daily table with that in mind.

How to use it

  1. Copy access log lines from your server, CDN or hosting panel.
  2. Pick which bots to count, for example Google crawlers.
  3. Choose how to group URLs: folder, two levels or template.
  4. Read the share, unique URLs and status columns, then check the busiest groups first.

Frequently asked questions

Which log format does it read?

Apache and nginx combined format: IP, time in brackets, the quoted request, status, bytes, then the quoted referrer and user agent. Lines in another format are skipped and counted.

Can I trust the bot names in the result?

Only as claims. The tool reads the user-agent text, and anyone can send a fake one. For anything that matters, check the IP addresses against the operator's published ranges or with reverse DNS.

Does crawl budget matter for a small site?

Rarely. It matters for large sites, sites with many parameter URLs, and sites that publish a lot of pages quickly.

Is my log uploaded?

No. The log is parsed in your browser and never sent anywhere. Logs contain IP addresses, so you may want to remove them first with the log privacy workbench.