AI Block Adoption Timeline from robots.txt Snapshots
Reads a series of dated robots.txt copies and shows when AI crawlers were blocked or unblocked.
Use the tool
Why this tool exists
A site's AI policy changes over time. Archived copies of its robots.txt, taken on different dates, show when it started blocking GPTBot or ClaudeBot.
Paste snapshots after === date lines and read the first block date per crawler and the list of changes. Fetching archived copies needs the network, so copy them yourself. A bot that did not exist yet cannot be blocked, so early snapshots show nothing for it.
How to use it
- Collect robots.txt copies from different dates.
- Paste each after a === date line.
- Read first blocked dates.
- Read the list of changes.
Frequently asked questions
Which date formats work?
Any that sort as text, such as 2024-05-01.
Does it fetch archives?
No.
Why is a crawler missing?
It was never blocked in the snapshots.
What counts as blocked?
The home page is disallowed for it.