robots.txt Viewer: Grouped Rules per Crawler
Turns a robots.txt into a readable summary: one block per group, with what it blocks, its exceptions and any other directives.
Use the tool
Why this tool exists
Real robots.txt files grow long, with several groups, shared user-agent lines and comments. It is hard to see at a glance who is blocked from what.
The viewer lists each group, the crawlers it applies to, whether it blocks the whole site, the Disallow patterns and the Allow exceptions. Lines crawlers would ignore are listed too. A web page cannot fetch other sites' files, so you paste the text.
How to use it
- Open the robots.txt in your browser and copy it.
- Paste it here.
- Read each group and its blocked patterns.
- Use the validator or tester for deeper checks.
Frequently asked questions
Can it load a live robots.txt from a domain?
No. Cross-site fetching needs a network request or a server, which this site avoids. Paste the text.
Does it show Crawl-delay?
Yes, under Other for the group where it appears. Google ignores it.
What about groups with several User-agent lines?
They are shown as one group with all names listed, as crawlers read them.
Are comments shown?
Only a count. They do not affect crawling.