AI Bot robots.txt Builder
Choose how to treat AI training crawlers, AI search crawlers and user-triggered fetchers, then generate robots.txt rules, with per-bot overrides for GPTBot, ClaudeBot, CCBot and others.
Use the tool
Why this tool exists
Site owners increasingly want a say in how AI companies use their content. You may be happy to be cited in an AI search answer but not want your pages used to train models, or the other way round. These crawlers use different user-agent names, and the list keeps changing.
This builder groups known bots into training crawlers, AI search crawlers and user-triggered fetchers, and lets you set block or allow for each group. You can override single bots, either with allow, block or a path to disallow. The output is a commented robots.txt section that ends with a rule for all other agents. Robots.txt is advisory: well-behaved bots follow it, others may not. Check each company's documentation for current names. Generation happens in your browser.
How to use it
- Set the default for AI training crawlers, AI search crawlers and user-triggered fetchers: block or allow.
- Add overrides in the box, for example GPTBot: block or ClaudeBot: /private/.
- Optionally enter your sitemap URL.
- Run the tool and add the result to your robots.txt file.
Frequently asked questions
Will blocking these bots stop AI training on my content?
Only for crawlers that honour robots.txt, and only for future crawling. Content already collected, or fetched by bots that ignore the file, is not affected.
Does blocking Google-Extended affect search rankings?
No. Google-Extended is a separate control for AI training uses, and blocking it does not change Google Search crawling or ranking. Check Google's documentation for the current scope.
Why separate training, search and user fetchers?
They serve different purposes. Training crawlers collect data for models, search crawlers fetch pages to cite in answers, and user fetchers load a page because a person asked an assistant to. Policies often differ.
Is anything sent to a server?
No. The calculation runs in your browser and nothing you paste is uploaded, stored or sent to a server, so drafts and confidential copy are safe to check.