Crawl-delay Calculator (Requests per Day)

Turns a Crawl-delay value in seconds into a request ceiling and the time a crawler would need to fetch your pages.

Use the tool

Why this tool exists

A line such as Crawl-delay: 10 sounds modest, but it caps a crawler at 8,640 requests a day, which is slow for a big site. Many site owners set it without knowing that Google ignores it entirely.

Enter the delay and your page count to see the ceiling and the days needed to fetch everything once. The page lists what the vendors document (Google, Apple and Amazon do not use it) and is open about what it did not check.

How to use it

  1. Enter the delay in seconds.
  2. Optionally enter how many pages you have.
  3. Read the requests per day and the time for a full pass.
  4. Decide whether Crawl-delay is the right tool at all.

Frequently asked questions

Does Google follow Crawl-delay?

No. Google's documentation lists only user-agent, allow, disallow and sitemap.

What should I use to slow a crawler?

Serve 429 or 503 responses when you are overloaded, or use the crawl settings in a search engine's webmaster tools where available.

Is the result exact?

It is the upper bound for a crawler that reads the value as one request per N seconds.

Do all crawlers read it the same way?

No. Some treat it as a pause, others as a time window.