robots.txt User-agent Group Merger (Effective Rules)
Merges duplicate user-agent groups like a crawler does and prints the effective rule set for each named crawler.
Use the tool
Why this tool exists
Many robots.txt files have two groups for Googlebot, or a Googlebot group plus a * group. Crawlers merge groups with the same name, and a crawler with its own group ignores the * group completely. People often expect the opposite and end up with a crawler that is blocked from less than they intended.
The tool lists the effective rules per user-agent and warns when named groups do not inherit the * rules.
How to use it
- Paste the robots.txt.
- Read the merged rule list for each user-agent.
- Check whether named groups need the * rules copied in.
- Verify with the tester.
Frequently asked questions
Are the * rules added to a named group?
No. A crawler uses the most specific group that names it and ignores the * group.
Are groups merged?
Yes, groups with the same user-agent name are combined, in any order.
Is the user-agent name case sensitive?
No.
Does it handle several User-agent lines in one group?
Yes. Each name gets the same rules.