Crawl Spike and Drop Alerts from Access Logs

Compares each day's crawler requests with the average of the days before and flags spikes and drops above a threshold.

Use the tool

Why this tool exists

A sudden drop in crawling can follow a server problem, a robots.txt mistake or a block. A spike can be a new crawler, a fake bot or a change in your internal links. Spotting either early helps.

You set the baseline window, the percent change and a minimum baseline so that quiet days do not raise false alarms. Days with less than 20 hours of log lines are marked as partial, since a half day always looks like a drop.

How to use it

  1. Paste log lines for as many days as you have.
  2. Choose all bots together or each bot separately.
  3. Set the window, the threshold in percent and the minimum baseline.
  4. Open the flagged days and compare them with deploys, outages and robots.txt changes.

Frequently asked questions

How is the baseline computed?

As the plain average of the previous days in the window. A longer log gives better baselines.

Why is the first day not checked?

There are no earlier days to compare it with.

A spike came from one IP.

Check it with the fake Googlebot detector. Fake bots often cause spikes.

Is my log uploaded?

No. The log is parsed in your browser and never sent anywhere. Logs contain IP addresses, so you may want to remove them first with the log privacy workbench.