crawl-ia.com

AI crawlers list

Which AI crawler reads your site, what for, how to block it, and how to check it really is who it claims. 985 IP ranges refreshed today from 12 crawlers, straight from the files their owners publish.

A user-agent name proves nothing. Anyone can write GPTBot in a header, and server logs are full of them. Only the source address holds up, which is why this page gives the official ranges rather than a list of names.

Check an IP address

A visitor claims to be an AI crawler in your logs? Paste its address.

Behind a CDN, your logs no longer show the visitor. Cloudflare, Fastly and any other reverse proxy receive the request in your place, then pass it on. What your access.log records is therefore their address, the same handful of ranges for a crawler and for a human alike. Matching those against the ranges on this page can never produce a hit.

The visitor's real address travels in a header the proxy adds, and that is what you must compare. Two places to do it:

On your origin, tell the server to trust that header. On nginx, real_ip_header CF-Connecting-IP together with set_real_ip_from for the proxy's own ranges. That repairs logs, filtering and rate limiting in one go. ⚠️ The set_real_ip_from part is not optional: without it anyone can forge the header and pass for Googlebot.
Or at the CDN itself, in a WAF rule, where the source address is already the real one. The crawler is then refused before it ever reaches your server.

And whichever you pick, do not freeze these ranges into a config file. Publishers change them without notice. Refetch the JSON files on a schedule and regenerate the list, which is exactly what this page does every day.

What your site tells each of them

Your robots.txt is read live and checked against all 24 known crawlers.

Three uses, three different decisions

This is the distinction missing everywhere, and the only one that matters: blocking training costs you nothing, blocking indexing removes you from the answers.

Training

Your content is used to train a model. Blocking protects your content and costs you nothing in visibility.

Crawler	Owner	What it does	Addresses
GPTBot	OpenAI	Collects content to train OpenAI's GPT models. documentation	21 ranges source

ClaudeBot	Anthropic	Collects content to train Anthropic's Claude models. documentation	26 ranges source

Google-Extended	Google	Not a crawler but a directive: it says whether Google may use your pages to train Gemini. Crawling is still done by Googlebot, and refusing it does not remove you from Google Search. documentation	none

GoogleOther	Google	Internal Google uses, outside public search. documentation	272 ranges source

CCBot	Common Crawl	The crawler of the public Common Crawl archive, which trains no model but whose data feeds many. Blocking it removes you from the starting point of numerous models, and from this tool. documentation	5 ranges source

Applebot-Extended	Apple	Like Google-Extended, a directive: refuse the use of your pages to train Apple's models, without leaving Siri. documentation	none

Meta-ExternalAgent	Meta	Collects content to train Meta's Llama models. documentation	none

Bytespider	ByteDance	Collects content for ByteDance's models. Known for poor robots.txt compliance.	none

cohere-ai	Cohere	Collects content for Cohere's models.	none

Diffbot	Diffbot	Turns pages into structured data, resold notably for training. documentation	none

omgili	Webz.io	Indexes forums and discussions, resold as a corpus.	none

Indexing for answers

Your content is indexed so it can be cited in the assistant's answers. Blocking removes you from AI answers. This is the heaviest choice.

Crawler	Owner	What it does	Addresses
OAI-SearchBot	OpenAI	Indexes the web for ChatGPT's search. Blocking it removes your site from OpenAI's sourced answers, without changing anything about training. documentation	39 ranges source

Claude-SearchBot	Anthropic	Indexes the web for Claude's answers. documentation	26 ranges source

Googlebot	Google	Google Search's crawler, which also feeds AI Overviews. Blocking it removes you from Google. documentation	317 ranges source

PerplexityBot	Perplexity	Indexes the web for Perplexity's answer engine. documentation	8 ranges source

bingbot	Microsoft	Bing's crawler, which also feeds Copilot. documentation	28 ranges source

Applebot	Apple	Feeds Siri and Spotlight suggestions. documentation	none

Amazonbot	Amazon	Feeds Alexa and Amazon services. documentation	none

DuckAssistBot	DuckDuckGo	Feeds DuckDuckGo's assisted answers. documentation	none

YouBot	You.com	Indexes the web for You.com. documentation	none

On-demand visit

Triggered when someone pastes your address into the assistant. Blocking breaks the page for someone who wanted to read you.

Crawler	Owner	What it does	Addresses
ChatGPT-User	OpenAI	Fetches a specific page because a user asked for it. documentation	213 ranges source

Claude-User	Anthropic	Fetches a page at the request of a Claude user. documentation	26 ranges source

Perplexity-User	Perplexity	Fetches a page when a user clicks a source. documentation	4 ranges source

MistralAI-User	Mistral	Fetches a page at the request of a Le Chat user. documentation	none

Build your robots.txt

Tick what you want to refuse. The file updates itself.

The robots.txt file is a request, not a barrier. Serious crawlers honour it, the others do not: against those, the only effective measure is address filtering, using the ranges above.

What next?

Allowing a crawler is not enough to be read. Check what your server actually answers, and what is really archived, with the crawl-ia analysis. The site ranking shows who is most cited across the web.