New Content Discovery Lag (Publish Date to First Bot Visit)
Compares the publish time of new URLs with the first bot request for each URL in your log and shows the lag.
Use the tool
Why this tool exists
How fast new pages are picked up is one of the few crawl numbers you can act on. Slow discovery points to weak internal links, a stale sitemap or a site that bots visit rarely.
You paste a list of paths with publish times and your access log. The tool finds the first request from the bots you choose and reports the gap, the median and the slowest page. Times without a zone are read as UTC, and log times use their own offset, so check your server's time zone.
How to use it
- Paste access log lines in the first box.
- Paste one line per new URL in the second box: /path,2025-10-02 08:00.
- Choose the bots and whether only 200 answers count.
- Read the lag per URL and the median.
Frequently asked questions
What if a bot visited before the publish time?
The row is marked for review. The publish time may be wrong, or the page was live earlier than you recorded.
Why is the lag unreliable for old pages?
A page published before the log starts may have been crawled before the log begins. Those rows are flagged.
How can I speed up discovery?
Link new pages from the home page or a hub, update the sitemap at once and make sure the pages return 200 quickly.
Can I trust the bot names in the result?
Only as claims. The tool reads the user-agent text, and anyone can send a fake one. For anything that matters, check the IP addresses against the operator's published ranges or with reverse DNS.