AI Crawler User-Agent Directory (GPTBot, ClaudeBot and More)

A reference list of AI bots with their robots.txt token, operator, purpose and what the operator says about robots.txt.

Use the tool

Why this tool exists

AI crawlers come in kinds: training collectors, search bots and fetchers that act for a user. Blocking the wrong one costs you search visibility; leaving the right one open may feed a model you did not agree to.

Each row says whether it was read on the operator's own page, with the date, or listed from memory and not verified. Rows from memory are marked so you can check them. Purposes change and new bots appear often.

How to use it

  1. Type a bot or operator name, or choose a purpose.
  2. Read the purpose and robots.txt behaviour.
  3. Check the Source column before relying on a row.
  4. Use the AI bot builder to write the rules.

Frequently asked questions

How current is it?

Verified rows were read on 2026-10-23. Check the vendor page for changes.

Do all of these follow robots.txt?

No. User-triggered fetchers often do not, and the table says so where the operator does.

Is Google-Extended a crawler?

No. It is a control token with no user-agent string of its own.

Why are some rows unverified?

They are from memory. No source was read for them.