Common Crawl Index Result Analyzer
Summarises Common Crawl index output: captures by crawl, status, type and URL.
Use the tool
Why this tool exists
Common Crawl is the open web archive behind many AI training sets. Its index API tells you which of your URLs were captured in each crawl.
Run the query yourself (the page shows a sample command) and paste the JSON lines here. The tool counts captures per crawl, status codes, MIME types and the most captured URLs. Presence in the archive shows exposure to datasets built from it; it does not show who used them.
How to use it
- Query the index API for your domain with output=json.
- Paste the lines.
- Read captures per crawl.
- Block CCBot in robots.txt if you decide to.
Frequently asked questions
Can the page call the API?
No. It makes no network requests.
Does a capture mean a model was trained on it?
No, only that the page is in the dataset.
What does blocking CCBot do?
It asks the crawler not to capture future pages; it does not remove old captures.
Which fields are needed?
url, status, mime and filename.