robots.txt Comment Annotator (Plain-English Rule Notes)
Writes a short explanation above each Allow and Disallow line.
Use the tool
Why this tool exists
A robots.txt full of patterns like /*?sort= or /*.pdf$ is a mystery to the next person who edits it. Comments help, but few people write them.
The annotator reads each rule and adds a comment such as Blocks everything inside the folder /admin/ or Blocks any URL ending in .pdf. Existing comments and lines with their own comment are left alone.
How to use it
- Paste the robots.txt.
- Copy the annotated version.
- Edit comments that are not quite right.
- Keep the annotated file as your source.
Frequently asked questions
Does it change the rules?
No, it only adds comment lines.
Are descriptions always right?
They are generated from the pattern shape. Odd wildcard mixes get a generic description, so read them.
Does it touch existing comments?
No.
Do comments increase the file size?
Yes, slightly. Mind the 500 KiB limit.