robots.txt HTTP Status Code Explainer (200, 301, 404, 429, 503)

Explains what crawlers do when robots.txt answers with a given status code.

Use the tool

Why this tool exists

A 404 means no rules, but a 503 can stop crawling of your whole site, and a 429 is treated as a server error by Google. Redirect chains longer than five hops count as a 404. These details decide whether a hiccup is harmless or costly.

Enter the status codes you see and read the behaviour documented by Google and by RFC 9309, which differ in places such as the 30-day rule.

How to use it

  1. Type the status codes, one per line.
  2. Read Google's behaviour and the RFC behaviour.
  3. Check your server returns 200 or 404.
  4. Never serve a 5xx for robots.txt.

Frequently asked questions

What happens on a 404?

Crawlers treat it as no restrictions.

What happens on a 503?

Google stops crawling for a while and uses a cached copy if it has one.

How many redirects are followed?

At least five, per RFC 9309 and Google.

Is 403 a way to block crawlers?

No. Google treats it as no robots.txt.