AI Preferences (IETF aipref) Header and robots.txt Generator
Builds Content-Usage statements for an HTTP header and robots.txt from the IETF aipref drafts.
Use the tool
Why this tool exists
The IETF AI Preferences working group is defining a shared vocabulary for how content may be used by AI and a way to attach it to files: a Content-Usage header and a Content-Usage line in robots.txt.
These are drafts and may change. The tool uses draft-ietf-aipref-vocab-08 and draft-ietf-aipref-attach-05 as read on 2026-10-23. Choose y, n or unspecified per category, and optionally path-specific lines. Whether any crawler reads these today is not verified.
How to use it
- Choose a value for each category.
- Optionally add path-specific lines.
- Copy the header or robots.txt lines.
- Re-check the drafts before relying on them.
Frequently asked questions
Is this a standard?
No. The drafts say they do not reflect working group consensus.
What do the categories mean?
train-ai: training models. ai-use: input to a model other than training. search: finding assets and linking to them.
What if I leave one unspecified?
It stays unknown, which is the default.
Do crawlers honour it?
Not verified.