Bot Status Code Report (2xx, 3xx, 4xx, 5xx per Crawler)
Shows status classes per crawler, the exact status codes and the URLs that received the most 4xx and 5xx answers.
Use the tool
Why this tool exists
Every redirect and error served to a crawler is a request that did not fetch a page. A small share is normal. A large share of 3xx means internal links and sitemaps point to old URLs, and a rise in 5xx means the server struggled while bots were visiting.
The table separates crawlers, because Googlebot, Bingbot and AI bots often see different mixes. The error URL list shows what to fix first.
How to use it
- Paste access log lines.
- Choose the bots, or all known bots.
- Read the status classes per crawler.
- Fix the URLs listed under 4xx and 5xx, then update links that point to redirects.
Frequently asked questions
Is a 304 an error?
No. It means not modified, which is a cheap and good answer to a conditional request.
What error share is too high?
There is no fixed limit. Anything that keeps growing, or that sits on important URLs, deserves a fix.
Can I trust the bot names in the result?
Only as claims. The tool reads the user-agent text, and anyone can send a fake one. For anything that matters, check the IP addresses against the operator's published ranges or with reverse DNS.
Which log format does it read?
Apache and nginx combined format: IP, time in brackets, the quoted request, status, bytes, then the quoted referrer and user agent. Lines in another format are skipped and counted.