Unsupported robots.txt Directives Detector (Noindex, Host, Crawl-delay)
Lists lines that Google does not use and suggests the working alternative for each.
Use the tool
Why this tool exists
Old advice put Noindex, Nofollow and Host lines into robots.txt. Google supports only user-agent, allow, disallow and sitemap, so these lines do nothing there. Google stopped honouring Noindex in robots.txt in 2019.
The tool shows each unsupported line with its number and the usual replacement. Other search engines may read some of them; that is not checked here.
How to use it
- Paste the robots.txt.
- Read the list of ignored lines.
- Replace them with the suggested method.
- Re-validate the file.
Frequently asked questions
Is Noindex in robots.txt dead?
For Google, yes. Use a robots meta tag or an X-Robots-Tag header.
Should I delete Crawl-delay?
Google ignores it. Keep it only if another crawler you care about reads it.
What replaces Host?
A 301 redirect to the preferred host and a canonical tag.
What does Google support?
Only user-agent, allow, disallow and sitemap, per its documentation.