Crawl Depth vs Bot Hits (URL Path Depth)

Shows hits, distinct URLs and hits per URL for each path depth, with a small bar chart.

Use the tool

Why this tool exists

Sites that bury content deep in folders tend to see fewer visits there. A table of hits by depth shows whether crawling falls off sharply at a certain level.

Depth here means the number of path segments in the URL. It is not the number of clicks from the home page, so a deep URL can still be linked from the home page. Use it as a first look, then compare with a click-depth report from a crawler.

How to use it

  1. Paste access log lines.
  2. Choose the bots to count.
  3. Decide whether query strings make separate URLs.
  4. Look for the depth where hits per URL drop.

Frequently asked questions

Is path depth the same as click depth?

No. Click depth is the number of links from the home page. Path depth only counts folders in the URL.

What should I do if deep URLs are rarely crawled?

Link to them from stronger pages, add them to the sitemap and check that they are worth indexing.

Can I trust the bot names in the result?

Only as claims. The tool reads the user-agent text, and anyone can send a fake one. For anything that matters, check the IP addresses against the operator's published ranges or with reverse DNS.

Is my log uploaded?

No. The log is parsed in your browser and never sent anywhere. Logs contain IP addresses, so you may want to remove them first with the log privacy workbench.