robots.txt Templates for WordPress, Shopify, Magento and More
Pick a platform and get a sensible starting robots.txt with comments explaining why.
Use the tool
Why this tool exists
Most sites need the same few rules: keep admin, cart, internal search and parameter URLs out of the crawl, and point to the sitemap. Starting from a template is faster than from a blank file.
These templates are generic starting points, not vendor defaults (Shopify, for example, generates its own file). Edit them, then validate and test before publishing.
How to use it
- Choose a template.
- Add your sitemap URL.
- Copy the result and adapt paths to your site.
- Validate and test the file.
Frequently asked questions
Are these the platforms' official files?
No. They are common practice. WordPress and Shopify generate their own files, and the notes say so.
Will the staging template remove my pages from Google?
No. Blocking crawling does not remove indexed URLs; use noindex or a password.
Should I block AI crawlers?
That is your choice. The AI template covers common training crawlers; see the directory for details.
Do I need robots.txt at all?
No, but a missing file returns a 404 on every request, and a simple file avoids that.