robots.txt to JSON Converter
Parses a robots.txt into JSON you can use in scripts, audits or spreadsheets.
Use the tool
Why this tool exists
Scripts that audit many sites need robots.txt as data, not text. Writing a parser means handling comments, shared user-agent lines and odd spacing.
This tool outputs groups with their user-agents, rules (with type, path and line number), other directives, sitemaps and the lines that were ignored. It follows RFC 9309 grouping.
How to use it
- Paste the robots.txt.
- Pick an indent size.
- Copy the JSON.
- Use it in your own script or tool.
Frequently asked questions
What does it do with Crawl-delay?
It appears under other for the group it belongs to.
Are misspelled directives repaired?
No. This converter is strict and lists them as other directives, so you see what a strict parser sees.
Are line numbers included?
Yes, for groups and rules.
Are comments included?
No.