Bot Firewall Rule Builder (Cloudflare WAF, Nginx, Apache)
Builds user-agent block or rate-limit rules for three platforms from a list of bot names.
Use the tool
Why this tool exists
robots.txt is a request; a firewall rule is enforcement. For crawlers that ignore robots.txt, you can refuse them at the server.
List the bot tokens and get a Cloudflare custom rule expression, an Nginx map with a 403 or a limit_req setup, and an Apache rewrite rule. User-agent rules only catch bots that identify themselves honestly, and they also block real visitors who carry the word in their string. Test with curl -A.
How to use it
- List bot tokens, one per line.
- Choose block or rate limit and the platform.
- Copy the rules into your config.
- Test with curl -A "GPTBot" -I.
Frequently asked questions
Will this stop a determined scraper?
No. It changes its user-agent. Use IP ranges, rate limits or a bot management service.
Does Apache support rate limiting?
Not built in; the tool says so in the output.
Is the Cloudflare syntax tested?
Not against a live account here.
Should I also edit robots.txt?
Yes, keep both.