What the tool checks
Your robots.txt
What your file allows CCBot, GPTBot, ClaudeBot, PerplexityBot and Google-Extended to do. A group naming a crawler overrides the catch-all rule.
The real answer to CCBot
Your server can block a crawler your robots.txt allows. So we request your site with the CCBot agent and look at what actually comes back.
Archived pages
How many pages are really in the index, crawl by crawl, separating genuine pages from the robots.txt file alone and from errors. The URL list is shown and exportable as CSV.
Archive.org captures bonus
The other large public index. It goes back to 1996 and keeps what it has already captured, even if the site blocks crawlers today: a domain with zero pages in Common Crawl can hold twenty years of archives there. We give the first and last capture, the list of pages, and the Wayback URL of each one, to open or export as CSV.
Does being archived in Common Crawl matter?
Yes, within an AI visibility strategy, but in its rightful place. Common Crawl archives genuinely do feed training corpora: FineWeb and FineWeb2, among the most widely used open datasets, are built directly from them, after cleaning, filtering and deduplication. Three things are worth separating, though, because they are routinely confused: being archived, being kept for training and being cited in an answer.
Being archived
Your pages exist in the public corpus. That is what this tool measures, and the only one of the three stages you control directly.
Being kept
Your pages survive the quality filters and deduplication applied by whoever builds the training sets. Nothing guarantees it.
Being cited
An assistant mentions you in its answer. That depends on a different mechanism, and on crawlers other than CCBot.
Being present opens a possibility, not a result
The Common Crawl index lists the captures held in the archives. Having pages genuinely present makes them available to everyone who works from that corpus, and there are many. It is a way into training datasets, not proof that any given model used them. That presence is far from pointless: it is the precondition for everything else, and its absence is a certainty of being out of the picture.
An archived page is not a kept page
Derived datasets do not carry over every capture. FineWeb2 documents its deduplication and filtering stages, and most of the collected volume is discarded along the way. You cannot conclude "my site trains the models" from a positive result alone.
For an audit, the useful question is not "am I present", but which pages were captured, on what dates, with what status and what content actually retrievable. That is exactly what this tool lists, page by page. A trace covering only the robots.txt proves nothing about your articles.
Being cited by an assistant is a different mechanism
Training and search-at-answer-time are two distinct uses, and model vendors separate them explicitly. OpenAI formalises this with two crawlers that have different jobs: GPTBot for collection intended for training, and OAI-SearchBot for search inside ChatGPT. Their documentation states that the two settings are independent: a site may allow the second, to appear in answers, while disallowing the first.
The consequence is direct: presence in Common Crawl says nothing about your presence in answers from ChatGPT, Claude or Perplexity. That is why this tool also checks what your robots.txt tells those crawlers, and not only CCBot.
What to track, and in what order
Treat archiving as a useful checkpoint, not as a performance indicator. Three measures to follow separately, because they move neither at the same pace nor for the same reasons: archived pages, citations earned in assistants, and traffic actually sent. The first is a prerequisite, the other two are the outcome.
Making sense of the results
"Seen, but nothing archived": what does that mean?
This is the result most often misread. Common Crawl captures fall into three families, and only one proves a page was kept:
- pages, the content actually archived;
- the robots.txt alone, which the crawler always reads first;
- crawl diagnostics, the redirects and errors encountered.
A domain can therefore appear in the index with its robots.txt alone and zero pages of content. The one-line command doing the rounds answers "present" in that case, even though nothing was kept. This tool counts the three families separately.
My robots.txt allows CCBot, yet the site is still blocked
This is the most common situation, and the most often misdiagnosed. Your robots.txt can be perfectly permissive while your host, your CDN or your security plugin refuses the crawler on its user-agent name alone. Many setups ship with a "block bad bots" rule enabled by default, which sweeps up AI crawlers without anyone deciding it.
When the tool detects this, it says so explicitly: the server refuses the CCBot agent while answering a browser. The fix then belongs in your firewall or security plugin. Editing the robots.txt will change nothing, it was already allowing it.
My site returns 403 to CCBot: how do I unblock it?
The table names what answered: Cloudflare, LiteSpeed, Akamai, Sucuri, or simply the server. That is where to act, and the place differs from case to case.
On Cloudflare. AI crawlers are classified there by behaviour rather than under a single label: Search for those indexing in order to answer later, Agent for those acting live on someone's behalf, Training for those collecting to train a model. CCBot falls under the last one. The setting sits in the AI crawler section of the dashboard, where each behaviour is allowed or refused. If a firewall rule blocks earlier, add an exception for that agent. Up-to-date documentation is on developers.cloudflare.com.
On shared hosting, LiteSpeed or Apache with cPanel: the block almost always comes from a generic "block bad bots" rule, enabled by default, which sweeps up AI crawlers indiscriminately. Support can lift it for your domain. That is the request to make, rather than "unblock my site", which tells them nothing about what to look for.
On WordPress, look at security plugins too: several filter by user-agent, using ready-made lists that include CCBot.
CCBot's official addresses
If your host would rather allow addresses than a user-agent name, Common Crawl publishes them and keeps them current:
3.41.188.32/29
18.97.9.168/29
18.97.14.80/29
18.97.14.88/30
2600:1f28:365:8000::/56 (IPv6)That is 28 IPv4 addresses. The source to pass on is index.commoncrawl.org/ccbot.json, always current, rather than a copied list that will age.
The argument that gets the request accepted. CCBot now runs on dedicated ranges with reverse DNS: its identity is verifiable, unlike a user-agent name anyone can spoof. The round trip must resolve back to the same result:
$ host 18.97.14.84
84.14.97.18.in-addr.arpa domain name pointer 18-97-14-84.crawl.commoncrawl.org.
$ host 18-97-14-84.crawl.commoncrawl.org
18-97-14-84.crawl.commoncrawl.org has address 18.97.14.84A support team will accept a rule on *.crawl.commoncrawl.org far more
readily than an exception on a user-agent. This check only works for IPv4: the IPv6
range has no reverse DNS yet.
Once the rule is lifted, run the analysis again: the answer to CCBot should turn 200. The pages themselves will only show up in the next crawl, roughly a month later.
On PlanetHoster: WAF rule 390509
A case encountered and verified: on PlanetHoster, CCBot is refused by the server's web application firewall. The symptom is recognisable. The robots.txt is sometimes served normally, the pages answer 403, and the robots.txt itself forbids nothing. So you go looking for a configuration mistake where there is none.
The rule responsible is numbered 390509. It is disabled from the N0C panel:
- Security then Web App Firewall (WAF), to reach the domain's rule window;
- find the rule and flip its toggle to disable it.
To confirm it is the one blocking before touching anything, Security then Firewall history lets you filter by domain and by rule ID, showing the refused requests. The full procedure is in the N0C documentation.
Three useful caveats. Disable the specific rule rather than the whole firewall, which the documentation itself presents as a last resort. The number may differ depending on the setup, but the principle holds: a generic bot-filtering rule that catches AI crawlers along the way. And unblocking has no retroactive effect: pages will only appear in the index at the next crawl, roughly a month later.
The same symptom on another host usually points to an equivalent rule, often labelled "block bad bots". That is the wording to use with support.
Why query several crawls?
Common Crawl publishes a new crawl roughly every month, and each one is a sample of the web, not a complete copy. A site present in one crawl can be missing from the next without anything having changed on its side.
Querying a single crawl therefore produces false negatives. Three are queried by default.
"JavaScript shell": why count the words?
Common Crawl does not execute JavaScript. It records the HTML exactly as the server sends it. A site that builds all its content in the browser therefore returns an almost empty page to the crawler, even with a perfectly valid 200 status.
The tool counts the words of the visible text received with the CCBot agent. Below roughly one hundred and twenty words on a home page, the content is probably rendered browser-side, and the crawler has nothing to keep. The diagnosis only shows on a genuine response, never on an error page.
How do I get my site into Common Crawl?
There is no submission form. The crawler discovers sites on its own, by following links. What you can do comes down to a few points:
- allow CCBot in your robots.txt, and check no catch-all rule blocks it;
- check that the firewall, CDN or security plugin lets it through;
- serve content readable without JavaScript;
- earn inbound links, since that is how the crawler arrives.
Then it is a matter of waiting for the next crawl, roughly a month.
Should I allow every AI crawler?
That is an editorial choice, not a technical question, and both positions hold. Allowing means accepting your content as raw material and, in exchange, being eligible for citation in answers. Blocking means protecting your content, knowing that a blocked crawler will never cite you.
The useful distinction is between crawlers that collect for training and those that fetch a page at the moment a user asks a question. Blocking the latter makes you invisible in live answers, which is rarely the original intention.
Should I publish an llms.txt file?
It achieves nothing today, but it costs nothing either. Worth saying plainly, because the file is often presented as an AI visibility lever with nothing to back that up.
The idea is appealing: a file at the root summarising the site for models, along the lines of robots.txt. The problem is that no major vendor documents reading it. Google's crawler documentation does not mention it. OpenAI and Anthropic do mention it, but for their own documentation, so that agents can navigate it more easily. That is probably what keeps the confusion alive: seeing an llms.txt at OpenAI does not mean OpenAI reads yours.
So, in practice: publish one if you like, it weighs a few kilobytes and may be read one day, by navigation agents if not by training crawlers. But do not count it as a visibility action, and be wary of any service selling it as one. Time spent on it is better invested in checking that your pages are reachable and readable without JavaScript.
This site publishes one, for consistency with its subject. It does not present it as a visibility factor for all that.
Where do the backlinks come from?
From the link graph Common Crawl publishes alongside its archives. It records the links between domains seen over three months of crawling, 2.45 billion of them in the current release.
Two things to know before comparing these figures with a paid tool.
These links are domain-level. You will know that example.com points at your site, never from which page, nor with what anchor text, nor whether it is nofollow. That is a limit of the graph itself, not of this tool.
Common Crawl is a sample. A link that exists but whose source page was never crawled will not show up. The figures are a floor, not a total. In exchange they are public, verifiable and free, where commercial tools rely on private crawls nobody can audit.
The .fr, .net, .org, .com, .de, .io, .app, .pro, .ai, .dev, .vision, .ch, .be, .it, .es, .nl, .ca, .info, .eu, .community, .paris, .bzh, .alsace, .corsica, .quebec, .shop, .online, .site, .store, .blog, .day, .co, .africa, .top, .lol, .ma and .cloud extensions are precomputed, which makes the answer immediate. For the others, the computation runs on demand and the result arrives by email.
Where does the data come from?
From the public Common Crawl index, queried live on every analysis, and from your own server, queried live as well. Nothing is stored between analyses, except the list of available crawls, kept for a day to avoid asking for it on every visit.
Crawl identifiers are never hard-coded: they are read from the source, otherwise the tool would one day query a stale crawl without saying so.