AI Preferences (IETF aipref) Header and robots.txt Generator

Builds Content-Usage statements for an HTTP header and robots.txt from the IETF aipref drafts.

Use the tool

Why this tool exists

The IETF AI Preferences working group is defining a shared vocabulary for how content may be used by AI and a way to attach it to files: a Content-Usage header and a Content-Usage line in robots.txt.

These are drafts and may change. The tool uses draft-ietf-aipref-vocab-08 and draft-ietf-aipref-attach-05 as read on 2026-10-23. Choose y, n or unspecified per category, and optionally path-specific lines. Whether any crawler reads these today is not verified.

How to use it

  1. Choose a value for each category.
  2. Optionally add path-specific lines.
  3. Copy the header or robots.txt lines.
  4. Re-check the drafts before relying on them.

Frequently asked questions

Is this a standard?

No. The drafts say they do not reflect working group consensus.

What do the categories mean?

train-ai: training models. ai-use: input to a model other than training. search: finding assets and linking to them.

What if I leave one unspecified?

It stays unknown, which is the default.

Do crawlers honour it?

Not verified.