Bot Response Time Report (Slow Sections and URLs for Crawlers)
Summarises response times for bot requests by folder or URL template and lists the slowest URLs.
Use the tool
Why this tool exists
Slow answers make crawlers visit less. Looking at crawler response times by section shows which templates are slow, and it often differs from what a page speed test says because bots hit different URLs than visitors do.
The tool needs a response time at the end of each log line, such as nginx $request_time in seconds or Apache %D in microseconds. Pick the unit that matches your log. If another number is last on your lines, the result will be wrong.
How to use it
- Add a response time to your log format if it is not there yet.
- Paste log lines and pick the unit of the last number.
- Choose how to group URLs and what counts as slow.
- Open the slowest sections and URLs and look for slow queries or uncached pages.
Frequently asked questions
My log has no response time.
In nginx add $request_time to the log format. In Apache use %D for microseconds or %T for seconds.
What is the 95th percentile?
The time that 95 percent of requests were faster than. It shows slow tails that averages hide.
Can I trust the bot names in the result?
Only as claims. The tool reads the user-agent text, and anyone can send a fake one. For anything that matters, check the IP addresses against the operator's published ranges or with reverse DNS.
Which log format does it read?
Apache and nginx combined format: IP, time in brackets, the quoted request, status, bytes, then the quoted referrer and user agent. Lines in another format are skipped and counted.