Staging robots.txt Left on Production Checker
Checks whether the robots.txt you paste blocks ordinary pages, and whether the page headers or HTML carry noindex, for production or staging.
Use the tool
Why this tool exists
One of the classic launch mistakes is shipping the staging robots.txt (Disallow: /) or a noindex tag to production. Traffic falls over days and nobody sees why. The reverse also happens: a staging site left open gets indexed.
Paste the live robots.txt, optionally the headers and page head, and say which environment it is. The tool checks the root and a deep page for * and the big crawlers and tells you what to review.
How to use it
- Paste the robots.txt your site serves.
- Optionally paste response headers and the page head.
- Select production or staging.
- Fix anything marked REVIEW.
Frequently asked questions
Does robots.txt hide a staging site?
It asks crawlers to stay out, but it is not security. Use a password.
Why check noindex too?
A noindex tag has the same effect on search as a block and is often forgotten.
Will it check my live site?
No, it only reads what you paste. Fetching a site needs a network request.
How can I avoid the mistake?
Add a robots.txt and noindex check to your release checklist.