Content Signals Policy Builder for robots.txt
Builds the Content-Signal line for robots.txt from your yes, no or not stated choices.
Use the tool
Why this tool exists
Content signals are a convention from Cloudflare to say what you allow beyond crawling: building a search index, using content as input to AI answers, or training models. They sit in robots.txt as one line, such as Content-Signal: search=yes, ai-train=no.
Leaving a signal out states no preference, which is different from yes or no. The line only expresses a preference; nothing is blocked unless you add Disallow rules too.
How to use it
- Choose yes, no or not stated for each signal.
- Set the user-agent.
- Copy the lines into robots.txt.
- Add Disallow rules for crawlers you want to stop.
Frequently asked questions
Does it block crawlers?
No. It is a preference, not an access rule.
What do the three signals mean?
search: index and show links and short excerpts. ai-input: use in AI answers. ai-train: train models.
What does not stated mean?
No preference, which is not the same as yes.
Who reads it?
Not guaranteed. Cloudflare introduced it; support by other crawlers is not verified.