Content Signals Policy Builder for robots.txt

Builds the Content-Signal line for robots.txt from your yes, no or not stated choices.

Use the tool

Why this tool exists

Content signals are a convention from Cloudflare to say what you allow beyond crawling: building a search index, using content as input to AI answers, or training models. They sit in robots.txt as one line, such as Content-Signal: search=yes, ai-train=no.

Leaving a signal out states no preference, which is different from yes or no. The line only expresses a preference; nothing is blocked unless you add Disallow rules too.

How to use it

  1. Choose yes, no or not stated for each signal.
  2. Set the user-agent.
  3. Copy the lines into robots.txt.
  4. Add Disallow rules for crawlers you want to stop.

Frequently asked questions

Does it block crawlers?

No. It is a preference, not an access rule.

What do the three signals mean?

search: index and show links and short excerpts. ai-input: use in AI answers. ai-train: train models.

What does not stated mean?

No preference, which is not the same as yes.

Who reads it?

Not guaranteed. Cloudflare introduced it; support by other crawlers is not verified.