Bot Firewall Rule Builder (Cloudflare WAF, Nginx, Apache)

Builds user-agent block or rate-limit rules for three platforms from a list of bot names.

Use the tool

Why this tool exists

robots.txt is a request; a firewall rule is enforcement. For crawlers that ignore robots.txt, you can refuse them at the server.

List the bot tokens and get a Cloudflare custom rule expression, an Nginx map with a 403 or a limit_req setup, and an Apache rewrite rule. User-agent rules only catch bots that identify themselves honestly, and they also block real visitors who carry the word in their string. Test with curl -A.

How to use it

  1. List bot tokens, one per line.
  2. Choose block or rate limit and the platform.
  3. Copy the rules into your config.
  4. Test with curl -A "GPTBot" -I.

Frequently asked questions

Will this stop a determined scraper?

No. It changes its user-agent. Use IP ranges, rate limits or a bot management service.

Does Apache support rate limiting?

Not built in; the tool says so in the output.

Is the Cloudflare syntax tested?

Not against a live account here.

Should I also edit robots.txt?

Yes, keep both.