robots.txt User-agent Group Merger (Effective Rules)

Merges duplicate user-agent groups like a crawler does and prints the effective rule set for each named crawler.

Use the tool

Why this tool exists

Many robots.txt files have two groups for Googlebot, or a Googlebot group plus a * group. Crawlers merge groups with the same name, and a crawler with its own group ignores the * group completely. People often expect the opposite and end up with a crawler that is blocked from less than they intended.

The tool lists the effective rules per user-agent and warns when named groups do not inherit the * rules.

How to use it

  1. Paste the robots.txt.
  2. Read the merged rule list for each user-agent.
  3. Check whether named groups need the * rules copied in.
  4. Verify with the tester.

Frequently asked questions

Are the * rules added to a named group?

No. A crawler uses the most specific group that names it and ignores the * group.

Are groups merged?

Yes, groups with the same user-agent name are combined, in any order.

Is the user-agent name case sensitive?

No.

Does it handle several User-agent lines in one group?

Yes. Each name gets the same rules.