robots.txt to JSON Converter

Parses a robots.txt into JSON you can use in scripts, audits or spreadsheets.

Use the tool

Why this tool exists

Scripts that audit many sites need robots.txt as data, not text. Writing a parser means handling comments, shared user-agent lines and odd spacing.

This tool outputs groups with their user-agents, rules (with type, path and line number), other directives, sitemaps and the lines that were ignored. It follows RFC 9309 grouping.

How to use it

  1. Paste the robots.txt.
  2. Pick an indent size.
  3. Copy the JSON.
  4. Use it in your own script or tool.

Frequently asked questions

What does it do with Crawl-delay?

It appears under other for the group it belongs to.

Are misspelled directives repaired?

No. This converter is strict and lists them as other directives, so you see what a strict parser sees.

Are line numbers included?

Yes, for groups and rules.

Are comments included?

No.