Googlebot robots.txt Rule Simulator (Parser Edge Cases)
Models Google's robots.txt handling for a chosen Google crawler and explains the special cases it applied.
Use the tool
Why this tool exists
Google's parser is forgiving in ways other parsers are not: it accepts frequent misspellings such as Disalow, merges groups with the same name, drops everything past 500 KiB and ignores directives like Crawl-delay.
This simulator follows Google's published documentation and the typo list in its open-source parser. It also falls back from Googlebot-Image or Googlebot-News to the Googlebot group. It is a model, not the live crawler, so confirm important pages in Search Console.
How to use it
- Paste the robots.txt and URLs.
- Choose the Google crawler.
- Read each verdict and the notes about typos or ignored lines.
- Confirm in the Search Console robots.txt report.
Frequently asked questions
Which typos does Google accept?
Disallow variants such as dissallow, dissalow, disalow, diasllow and disallaw, plus useragent, user agent and site-map, from its open-source parser.
Does Googlebot-Image use the Googlebot group?
If no group names it, yes, the most specific matching group is used and Googlebot is the fallback.
Does Google read Crawl-delay?
No.
Is this the real Google result?
No. It is a model built from documentation. The Search Console report is authoritative.