robots.txt Validator (RFC 9309 Syntax Check)

Checks a pasted robots.txt line by line against RFC 9309 and Google's documentation, and lists errors, warnings and notes with line numbers.

Use the tool

Why this tool exists

A single wrong character in robots.txt can quietly turn a rule off. A missing colon, a misspelled directive, a rule placed before any User-agent line or a relative Sitemap URL is simply ignored by crawlers, and nothing tells you.

This validator reads the file the way a crawler does and reports what would be ignored or looks risky. Some findings depend on the crawler (Google accepts frequent typos, others may not), so those are marked as items to review, not as hard failures. Nothing is uploaded.

How to use it

  1. Paste your robots.txt into the box.
  2. Read the errors first, then the warnings, then the notes.
  3. Fix the lines named by number and paste again.
  4. Test the important URLs with the robots.txt tester.

Frequently asked questions

Does it fetch my live robots.txt?

No. This site makes no network requests. Open yoursite/robots.txt in a browser, copy it and paste it here.

Why is a misspelling only a warning?

Google accepts a short list of frequent typos such as Disalow, but other crawlers may ignore the line. The safe fix is the correct spelling.

Is Noindex in robots.txt valid?

Google stopped supporting it in 2019. The tool lists it as a note with the supported alternative.

What size limit does it check?

500 KiB, the minimum parsing limit in RFC 9309 and the limit Google documents.