robots.txt Encoding and Invisible Character Checker
Scans a pasted robots.txt for characters you cannot see that break parsing, and can print a cleaned copy.
Use the tool
Why this tool exists
A robots.txt that looks perfect can still fail. A byte order mark at the start, a zero-width space inside a path, a non-breaking space after the colon or a curly quote pasted from a document makes a line unreadable. Google treats a byte order mark as an invalid line.
The checker lists each odd character with line, column and code point, notes line endings and shows a cleaned version. Pasted text is already decoded, so a file saved in the wrong encoding shows up as replacement characters.
How to use it
- Paste the robots.txt, ideally copied straight from the live file.
- Read the list of suspicious characters.
- Use the cleaned copy as a starting point.
- Save the file as UTF-8 without BOM.
Frequently asked questions
Why does a BOM matter?
Google's documentation lists a byte order mark among invalid lines, so the first rule may be lost.
Does it detect files saved as Latin-1?
Only indirectly, through replacement characters after pasting. For a file on disk use the file command shown in the output.
Should paths contain non-ASCII letters?
No. Write them as percent-encoded UTF-8.
Which line endings are fine?
CR, LF and CRLF are all valid.