AI Bot robots.txt Audit: Allowed, Blocked or Not Addressed
Reads your robots.txt and reports how each known AI crawler is treated.
Use the tool
Why this tool exists
Many sites think they block AI crawlers because they block one or two. The audit lists every crawler in the directory, the effect of your rules on the home page, and whether any rule names it.
A crawler that is not named follows the * group, which is usually open. The summary lists training crawlers you have not named. Blocking is advisory, and user-triggered fetchers may ignore it.
How to use it
- Paste your robots.txt.
- Read the effect and Named columns.
- Add groups for crawlers you want to block.
- Use the AI bot builder to generate them.
Frequently asked questions
What does via * mean?
No group names the crawler, so the * group applies.
What is partly blocked?
Some paths are disallowed but the home page is open.
Does blocking guarantee no crawling?
No. Only crawlers that obey robots.txt respect it.
Does it list every AI bot?
No. It covers the directory list, which changes over time.