robots.txt Encoding and Invisible Character Checker

Scans a pasted robots.txt for characters you cannot see that break parsing, and can print a cleaned copy.

Use the tool

Why this tool exists

A robots.txt that looks perfect can still fail. A byte order mark at the start, a zero-width space inside a path, a non-breaking space after the colon or a curly quote pasted from a document makes a line unreadable. Google treats a byte order mark as an invalid line.

The checker lists each odd character with line, column and code point, notes line endings and shows a cleaned version. Pasted text is already decoded, so a file saved in the wrong encoding shows up as replacement characters.

How to use it

  1. Paste the robots.txt, ideally copied straight from the live file.
  2. Read the list of suspicious characters.
  3. Use the cleaned copy as a starting point.
  4. Save the file as UTF-8 without BOM.

Frequently asked questions

Why does a BOM matter?

Google's documentation lists a byte order mark among invalid lines, so the first rule may be lost.

Does it detect files saved as Latin-1?

Only indirectly, through replacement characters after pasting. For a file on disk use the file command shown in the output.

Should paths contain non-ASCII letters?

No. Write them as percent-encoded UTF-8.

Which line endings are fine?

CR, LF and CRLF are all valid.