Canonical vs Crawled URL Mismatch (Bot Hits on Non-Canonical URLs)
Matches log requests against url,canonical pairs from a crawl export and shows how many hits went to non-canonical URLs.
Use the tool
Why this tool exists
A canonical tag is a hint. Crawlers still fetch the variants, and if most of their hits go to URLs that point elsewhere, the signals on your site may be mixed: internal links, sitemaps or redirects still lead to the variants.
Paste the address and canonical columns from a crawl export and your access log. The tool lists the non-canonical URLs that received hits and their declared canonical. It matches by path and query, so parameter variants show up as their own rows.
How to use it
- Export your crawl with the URL and its canonical URL.
- Paste the pairs as url,canonical lines in the second box.
- Paste access log lines in the first box.
- Fix internal links and sitemap entries that point to the top variants.
Frequently asked questions
Do crawlers ignore canonicals?
No, but they still request the variants to read them. Many hits on variants mean the variants are easy to find.
What about URLs missing from the export?
They are counted as unknown and not judged.
Can I trust the bot names in the result?
Only as claims. The tool reads the user-agent text, and anyone can send a fake one. For anything that matters, check the IP addresses against the operator's published ranges or with reverse DNS.
Is my log uploaded?
No. The log is parsed in your browser and never sent anywhere. Logs contain IP addresses, so you may want to remove them first with the log privacy workbench.