X-Robots-Tag Header Builder for Apache, Nginx and Cloudflare
Generates X-Robots-Tag configuration for the common servers, for chosen file types or the whole site.
Use the tool
Why this tool exists
You cannot put a meta tag in a PDF or an image, so noindex for those files has to be an HTTP header. The config differs per server, and a small mistake such as a missing always flag makes the header vanish on error responses.
Enter the directives, the file extensions and an optional crawler name and get snippets for Apache, Nginx, a Cloudflare Transform Rule and a _headers file. Directives are checked against Google's list.
How to use it
- Enter the directives, for example noindex, nofollow.
- Choose file extensions or the whole site.
- Pick the server format or all.
- Test with curl -I on a file URL.
Frequently asked questions
Why not use robots.txt for PDFs?
A block stops crawling but not indexing of linked URLs. A header with noindex removes the file, if the crawler can fetch it.
Can I target one crawler?
Yes, enter its name such as googlebot and the header gets that prefix.
Does Nginx add_header need always?
Yes if you want it on error responses too.
Is the Cloudflare snippet tested?
Not on a live account here; check your dashboard wording.