Wasted Crawl Detector (Parameters, Facets, Redirects, Errors in Logs)
Sorts bot requests into waste categories and useful requests, with examples, so you can see which patterns drain crawl effort.
Use the tool
Why this tool exists
Crawlers spend requests on whatever your site offers. Filter combinations, tracking parameters, internal search results, redirects and error pages are all things you rarely want fetched.
The tool assigns each request to the first matching category. You can edit the lists of facet, tracking and search parameter names to match your site. The categories are heuristics. Open the examples and check them before you change robots.txt, redirects or canonical tags.
How to use it
- Paste access log lines.
- Edit the parameter name lists to match your site.
- Optionally set a minimum page size to flag thin pages.
- Read the waste share and the top examples per category.
Frequently asked questions
How is a request assigned to a category?
In a fixed order: server errors, client errors, redirects, facet parameters, tracking parameters, internal search, other parameters, thin pages. The first match wins.
Should I block all parameter URLs in robots.txt?
Not blindly. Some parameters change the content. Review the examples first.
What is a thin page here?
A 200 page smaller than the byte limit you set. It is only a hint of a soft 404 and needs a manual check.
Can I trust the bot names in the result?
Only as claims. The tool reads the user-agent text, and anyone can send a fake one. For anything that matters, check the IP addresses against the operator's published ranges or with reverse DNS.