AI Block Adoption Timeline from robots.txt Snapshots

Reads a series of dated robots.txt copies and shows when AI crawlers were blocked or unblocked.

Use the tool

Why this tool exists

A site's AI policy changes over time. Archived copies of its robots.txt, taken on different dates, show when it started blocking GPTBot or ClaudeBot.

Paste snapshots after === date lines and read the first block date per crawler and the list of changes. Fetching archived copies needs the network, so copy them yourself. A bot that did not exist yet cannot be blocked, so early snapshots show nothing for it.

How to use it

  1. Collect robots.txt copies from different dates.
  2. Paste each after a === date line.
  3. Read first blocked dates.
  4. Read the list of changes.

Frequently asked questions

Which date formats work?

Any that sort as text, such as 2024-05-01.

Does it fetch archives?

No.

Why is a crawler missing?

It was never blocked in the snapshots.

What counts as blocked?

The home page is disallowed for it.