X-Robots-Tag Header Builder for Apache, Nginx and Cloudflare

Generates X-Robots-Tag configuration for the common servers, for chosen file types or the whole site.

Use the tool

Why this tool exists

You cannot put a meta tag in a PDF or an image, so noindex for those files has to be an HTTP header. The config differs per server, and a small mistake such as a missing always flag makes the header vanish on error responses.

Enter the directives, the file extensions and an optional crawler name and get snippets for Apache, Nginx, a Cloudflare Transform Rule and a _headers file. Directives are checked against Google's list.

How to use it

  1. Enter the directives, for example noindex, nofollow.
  2. Choose file extensions or the whole site.
  3. Pick the server format or all.
  4. Test with curl -I on a file URL.

Frequently asked questions

Why not use robots.txt for PDFs?

A block stops crawling but not indexing of linked URLs. A header with noindex removes the file, if the crawler can fetch it.

Can I target one crawler?

Yes, enter its name such as googlebot and the header gets that prefix.

Does Nginx add_header need always?

Yes if you want it on error responses too.

Is the Cloudflare snippet tested?

Not on a live account here; check your dashboard wording.