Resource Crawl Report (CSS, JS, Images, Fonts Requested by Bots)

Counts bot requests by file type, with distinct files, error counts and the resources that returned errors.

Use the tool

Why this tool exists

To render a page, a crawler needs its CSS, JavaScript and often its images. If those requests fail, the crawler may see a broken page. If they dominate the log, your pages may load many files that bots keep refetching.

The type comes from the file extension in the URL. URLs without an extension count as pages. An access log cannot show whether a resource is blocked by robots.txt, so test that separately.

How to use it

  1. Paste access log lines.
  2. Choose the bots to count.
  3. Read the table of types and the error share.
  4. Fix the resources listed under errors, then check robots.txt for blocked folders.

Frequently asked questions

Why do bots fetch CSS and JavaScript?

Rendering search engines load them to see the page as a browser does.

Is a high resource share bad?

Not by itself. Look at errors and at files that are fetched over and over.

Does the report show blocked files?

No. Use the blocked resources checker with your robots.txt.

Can I trust the bot names in the result?

Only as claims. The tool reads the user-agent text, and anyone can send a fake one. For anything that matters, check the IP addresses against the operator's published ranges or with reverse DNS.