Faceted Navigation robots.txt Rule Builder

Builds the wildcard rules that keep crawlers out of faceted URLs, and tests sample URLs against them.

Use the tool

Why this tool exists

Filters such as ?color=red&size=m create endless URL combinations. Blocking the parameters in robots.txt saves crawl effort, but the patterns must cover both ?param= and &param=, and a careless pattern blocks pagination.

List the parameters to block, or block everything and keep a few, and get the rules plus a test on sample URLs. Remember robots.txt stops crawling, not indexing; canonical tags are the safer consolidation tool.

How to use it

  1. List parameters to block, one per line.
  2. Choose block listed or block all except the kept ones.
  3. Optionally limit to a folder.
  4. Test your real URLs with the robots.txt tester.

Frequently asked questions

Why two rules per parameter?

A parameter can follow ? or &, so both are covered.

Will blocked URLs leave the index?

Not necessarily. Linked blocked URLs can stay indexed without content.

Is canonical better?

Often yes, because crawlers can read it. Blocked pages cannot show it.

What about pagination?

Keep it crawlable using the kept parameters option.