robots.txt URL Lister (http, https, www and Subdomains)
Builds the list of robots.txt addresses for your domains and subdomains and their protocol and www variants.
Use the tool
Why this tool exists
A robots.txt applies only to the exact protocol, host and port it is served from. Your blog subdomain, the www host and the old http address each need their own check, and they are easy to forget.
Enter hosts and get every robots.txt URL to open. Discovering subdomains needs certificate or DNS lookups, which this site does not do, so paste a list from your DNS zone.
How to use it
- Enter your domains and subdomains.
- Choose whether to add http and www variants.
- Open each URL.
- Check each returns 200 or redirects to the preferred host.
Frequently asked questions
Does example.com/robots.txt cover subdomains?
No. Each subdomain needs its own file.
Should the http version exist?
It should redirect to https, or serve the same rules.
Can it find my subdomains?
No, that needs a lookup. Paste a list you already have.
What if www and non-www differ?
Crawlers read the file of the host they crawl, so make them consistent.