Orphan Page Finder: Bot-Crawled URLs Missing From Your URL List
Lists pages that bots requested but that are not in the sitemap or URL list you paste, with hits, last seen time and the bots involved.
Use the tool
Why this tool exists
Pages that bots visit but you do not list are worth a look. They may be old URLs that are still linked, pages you forgot to put in the sitemap, or true orphans that only external links reach.
The tool matches paths from your log against the URLs you paste, ignoring host, query and trailing slash. A sitemap is not a map of internal links, so treat the result as candidates. Confirm with an internal link crawl before you call a page an orphan.
How to use it
- Paste access log lines in the first box.
- Paste your sitemap XML or one URL per line in the second box.
- Choose which bots to count and whether to keep only 200 answers.
- Review the pages that are not in your list, starting with the most requested.
Frequently asked questions
Is a page missing from the sitemap an orphan page?
Not always. An orphan has no internal links. A page can be linked and still be absent from the sitemap. Use a crawl of your internal links to confirm.
Can I trust the bot names in the result?
Only as claims. The tool reads the user-agent text, and anyone can send a fake one. For anything that matters, check the IP addresses against the operator's published ranges or with reverse DNS.
Why are query strings ignored?
Parameter variants would flood the list. Use the wasted crawl detector to study parameter URLs.
Is my log uploaded?
No. The log is parsed in your browser and never sent anywhere. Logs contain IP addresses, so you may want to remove them first with the log privacy workbench.