robots.txt Generator

Create a robots.txt by listing allow and disallow paths. Choose the user-agent, an allow-all or block-all default, an optional crawl delay and a sitemap URL.

Use the tool

Why this tool exists

Most sites need only a short robots.txt: keep crawlers out of admin pages, carts, internal search and parameter URLs, and point them at the sitemap. The syntax is simple but unforgiving, and a block-all file left over from staging is a classic way to disappear from search.

This generator turns a few lines into a valid file. List your Allow and Disallow rules, choose the user-agent, and add a sitemap URL and, if you want, a crawl delay. A default of allow all or block all is available, with a warning for the latter. Keep in mind that robots.txt controls crawling, not indexing, and that it is public, so never list secret paths in it. The file is built in your browser, and you upload it to your site root yourself.

How to use it

  1. List your rules in the box, one per line, such as Disallow: /admin/.
  2. Choose the default (allow all or block all) and the user-agent.
  3. Add your sitemap URL and an optional crawl delay.
  4. Run the tool, copy the result into a file named robots.txt and upload it to your site root.

Frequently asked questions

Does robots.txt hide pages from search results?

Not reliably. It asks well-behaved crawlers not to fetch URLs, but blocked pages can still be listed if other sites link to them. Use noindex for pages that must stay out of search.

Is the crawl delay useful?

Google ignores Crawl-delay, though some other crawlers honour it. The tool adds a comment saying so. For Google, crawl rate is managed through other means.

Should I list private folders in robots.txt?

No. The file is public, so listing a path tells everyone it exists. Protect private areas with authentication, not with robots.txt.

Is my data uploaded?

No. The calculation runs in your browser and nothing you paste is uploaded, stored or sent to a server, so drafts and confidential copy are safe to check.