robots.txt HTTP Status Code Explainer (200, 301, 404, 429, 503)
Explains what crawlers do when robots.txt answers with a given status code.
Use the tool
Why this tool exists
A 404 means no rules, but a 503 can stop crawling of your whole site, and a 429 is treated as a server error by Google. Redirect chains longer than five hops count as a 404. These details decide whether a hiccup is harmless or costly.
Enter the status codes you see and read the behaviour documented by Google and by RFC 9309, which differ in places such as the 30-day rule.
How to use it
- Type the status codes, one per line.
- Read Google's behaviour and the RFC behaviour.
- Check your server returns 200 or 404.
- Never serve a 5xx for robots.txt.
Frequently asked questions
What happens on a 404?
Crawlers treat it as no restrictions.
What happens on a 503?
Google stops crawling for a while and uses a cached copy if it has one.
How many redirects are followed?
At least five, per RFC 9309 and Google.
Is 403 a way to block crawlers?
No. Google treats it as no robots.txt.