AI Crawler List: GPTBot, ClaudeBot, PerplexityBot, and Every Bot Reading Your Site
A verified directory of every major AI crawler — GPTBot, ClaudeBot, PerplexityBot, and more — with user-agent strings, robots.
AI crawler list: GPTBot, ClaudeBot, PerplexityBot, and every bot reading your site
AI crawlers are automated bots operated by AI companies that visit websites to collect content for model training, AI-search indexing, or live user-triggered retrieval. Not every AI crawler does the same thing, and blocking each type carries different visibility consequences. This page is a verified, purpose-based directory of every major AI crawler, its operator, user-agent string, robots.txt token, and how to allow or block it.
Last verified: August 2026
What are AI crawlers?
AI crawlers are automated programs that request web pages on behalf of AI companies to collect, index, or retrieve content. They differ from traditional search-engine crawlers like Googlebot and Bingbot, whose primary job is building a search index for web-search results pages.
The line is blurring. OpenAI, Anthropic, Perplexity, Mistral, Amazon, and Meta now each operate separate crawlers for model training, AI-powered search indexing, and live user-triggered page fetches. Google and Apple, meanwhile, use control tokens rather than separate crawlers to govern how already-crawled data may be used by their AI products.
These bots interact with robots.txt, the file governed by RFC 9309 (the Robots Exclusion Protocol). RFC 9309 defines robots.txt as a voluntary protocol: it tells crawlers what a site owner prefers, but it does not enforce access. Understanding that distinction is critical to managing AI crawler traffic.
Training vs. search vs. user-triggered fetch: why the distinction matters
The most common mistake site owners make is treating all AI crawlers as training bots. AI crawlers fall into four distinct purpose categories, a framework drawn from Cloudflare's analysis of AI crawler traffic:
- Training: Bulk collection of web content to build or fine-tune foundation models. This is the largest category by volume. In Cloudflare's data from the first 28 days of July 2025 (measured across Cloudflare's network), training represented nearly 80% of all AI-bot crawling.
- Search/Indexing: Automated crawling to populate an AI-search index, comparable to how Googlebot builds Google's index, but for AI answer engines like ChatGPT Search, Perplexity, or Claude search.
- User-triggered retrieval: A fetch that happens because a real user asked the AI to read or summarize a specific page. In that same Cloudflare July 2025 window, user-action and undeclared traffic together accounted for less than 5% of AI-bot requests. ChatGPT-User was responsible for nearly three-quarters of the user-action cohort.
- Control token: Not a crawler at all. A robots.txt-only token that governs how data already collected by a traditional crawler may be reused for AI purposes (e.g., Google-Extended, Applebot-Extended).
Blocking a training crawler does not necessarily block an AI search crawler from the same operator. OpenAI, Anthropic, Amazon, and Mistral each provide separate robots.txt tokens for training and search. If you block only the training token, your site can still appear in that company's AI-search answers. If you block the search token too, it cannot.
User-triggered fetchers present a special challenge: because the request originates from a human user's action, operators like OpenAI, Perplexity, Meta, and Amazon state that these bots may not fully honor robots.txt.
Quick classification key
Master AI crawler lookup table
The table below covers every major AI crawler with current first-party documentation. The "Full HTTP User-Agent" column is kept separate from the "robots.txt Token" column because they are not always identical. Where a full HTTP UA string has not been officially published by the operator, the entry is marked accordingly; this article does not insert third-party guesses as official.
Footnotes:
¹ OpenAI notes the version number may change. Source: OpenAI: Overview of OpenAI Crawlers. ² Operator states user-triggered requests may not follow all robots.txt directives. ³ The official Anthropic bot page confirms bot names, purposes, robots.txt behavior, and IP-range URL, but does not publish full HTTP User-Agent strings. Do not treat third-party strings as officially verified. ⁴ Anthropic states all three documented bots honor robots.txt, including Crawl-delay. ⁵ Perplexity states that user-triggered fetches generally ignore robots.txt. ⁶ Labeled as official published example from Toutiao's Bytespider introduction; cooperation page confirms Robots Exclusion Protocol support. Not guaranteed to be the sole current format.
Related crawlers and traditional bots
These crawlers overlap with or relate to AI crawling but serve primarily traditional search, link-preview, or hybrid functions.
Bots omitted from this reference: GrokBot, DeepSeekBot, YouBot, cohere-ai, and CopilotBot are excluded because no current first-party documentation confirms their exact UA string, purpose, robots.txt behavior, and IP verification method. Third-party crawler directories alone are insufficient for a verified reference.
OpenAI crawlers in detail
OpenAI operates four distinct crawlers, each controllable independently through robots.txt. Blocking one does not block the others. Source: OpenAI: Overview of OpenAI Crawlers.
GPTBot crawls content that may be used to train OpenAI's foundation models. Its robots.txt token is GPTBot. Blocking it signals that your content should not be used for future model training, but it has no effect on ChatGPT Search. Verify requests against the live IP file at openai.com/gptbot.json.
OAI-SearchBot surfaces and links websites in ChatGPT Search. Its token is OAI-SearchBot. Sites that block this crawler are not shown in ChatGPT Search answers, though they may still appear as navigational links. Verify IPs at openai.com/searchbot.json.
ChatGPT-User fetches pages for certain user actions in ChatGPT and Custom GPTs. Because the request is user-initiated, OpenAI says robots.txt rules may not apply. Verify IPs at openai.com/chatgpt-user.json.
OAI-AdsBot validates ad landing pages. Its token is OAI-AdsBot.
OpenAI states that a robots.txt change can take about 24 hours to affect Search systems. Do not hard-code IP CIDRs; always pull from OpenAI's live JSON files, as ranges are updated periodically.
Does blocking GPTBot block ChatGPT Search?
No. GPTBot controls training data collection. OAI-SearchBot controls ChatGPT Search. They are independent tokens with separate IP ranges. To block training while staying visible in ChatGPT Search, disallow GPTBot and allow OAI-SearchBot.
Anthropic crawlers in detail
Anthropic operates three documented crawlers, each with a distinct purpose. Source: Anthropic / Claude Help Center.
ClaudeBot collects web content that could contribute to model training. Blocking its robots.txt token (ClaudeBot) signals that future material should be excluded from Anthropic training datasets.
Claude-SearchBot crawls the web to improve Claude search-result quality. Blocking it (Claude-SearchBot) can reduce visibility and accuracy in Claude search results.
Claude-User retrieves web content after a Claude user asks a question. Blocking it (Claude-User) can prevent user-directed retrieval and reduce visibility in user-directed Claude search.
Anthropic states all three bots honor robots.txt and support the non-standard Crawl-delay directive. Verify request IPs against the published IP-range file at claude.com/crawling/bots.json.
Important limitation: The official Anthropic documentation reviewed confirms bot names, purposes, robots.txt behavior, and IP-range URL, but it does not publish full HTTP User-Agent strings. Any full strings found in third-party directories should not be treated as officially verified without a further primary-source check.
Does Anthropic respect robots.txt?
Anthropic says all three documented bots honor robots.txt, including Crawl-delay. Because full HTTP UA strings are not officially published, site owners relying on string matching in WAF rules should cross-reference against Anthropic's published IP ranges for stronger verification.
Perplexity crawlers in detail
Perplexity operates two crawlers with very different behaviors. Source: Perplexity: Perplexity Crawlers.
PerplexityBot is an automated search indexer that surfaces and links websites in Perplexity search results. Perplexity explicitly states it is not used for foundation-model training. It honors robots.txt. Verify IPs at perplexitybot.json.
Perplexity-User fetches a page when a user asks Perplexity a question. Because the fetch is user-triggered, Perplexity says it generally ignores robots.txt. Perplexity recommends combining User-Agent matching with published IP ranges for enforcement. Verify IPs at perplexity-user.json.
For sites that want to appear in Perplexity search but need to control live user-triggered fetches, a WAF or CDN access-control rule matched against Perplexity-User's IP ranges is more reliable than robots.txt alone. See also the Perplexity help center robots.txt explanation (July 16, 2026) for context.
Does Perplexity-User respect robots.txt?
Generally no. Perplexity says user-triggered fetches may not follow robots.txt. For hard enforcement, use WAF rules or IP-based access controls.
Google-Extended and Applebot-Extended: control tokens, not crawlers
Neither Google-Extended nor Applebot-Extended is a crawler. Both are robots.txt control tokens that govern how content already collected by a real crawler may be reused for AI purposes. This is one of the most common misunderstandings in AI crawler management.
Google-Extended has no separate HTTP User-Agent and makes no page requests. It is a standalone robots.txt product token that controls whether content crawled by Google's existing user agents (Googlebot, etc.) may be used for Gemini and Vertex AI training and grounding. Blocking Google-Extended does not affect Google Search inclusion or ranking. Source: Google: List of Google's common crawlers.
Applebot-Extended does not crawl webpages. It controls whether data already crawled by regular Applebot may be used for Apple foundation-model training (Apple Intelligence). Blocking it does not affect Spotlight, Siri, or Safari discovery. Source: Apple: About Applebot.
Is Google-Extended a real crawler?
No. Google-Extended has no HTTP User-Agent and makes no page requests. It exists only as a robots.txt token.
Does blocking Google-Extended hurt Google rankings?
No. Google explicitly states that blocking Google-Extended does not affect Google Search inclusion or ranking. It only controls Gemini/Vertex AI data use.
Meta crawlers in detail
Meta operates five documented crawlers with distinct purposes. Source: Meta: Meta Web Crawlers (documentation updated May 21, 2026).
meta-externalagent trains foundation AI models or improves Meta products by indexing content directly. It honors robots.txt.
meta-externalfetcher fetches individual links at a user's request and supports agentic AI tasks. Meta states this crawler may bypass robots.txt because the fetch is user-requested.
meta-webindexer improves Meta AI search-result quality and helps Meta cite and link content. It honors robots.txt. Blocking it may reduce citation and linking in Meta AI answers.
Meta-ExternalAds improves advertising and business-related products. It honors robots.txt.
FacebookExternalHit generates link previews when URLs are shared on Facebook or Messenger. It is not primarily an AI crawler but commonly appears in server logs alongside the newer Meta bots.
Meta says robots.txt changes may take up to 24 hours to take effect because crawlers may cache the file.
Amazon, Common Crawl, ByteDance, and Mistral crawlers
Amazon: Amazonbot, Amzn-SearchBot, and Amzn-User
Amazon operates three crawlers with separate controls. Source: Amazon: About AmazonBot.
Amazonbot improves Amazon products and services; content may be used to train Amazon AI models. It honors robots.txt.
Amzn-SearchBot indexes content for Amazon search experiences such as Alexa. It also honors robots.txt.
Amzn-User fetches live information for user-triggered Alexa actions. Amazon states that user-triggered requests may not follow all robots.txt directives.
Amazon may cache robots.txt for up to 30 days, so changes are not immediate. Amazon explicitly does not support the Crawl-delay directive.
Common Crawl (CCBot)
CCBot builds an open web dataset for research and analysis. Its current user-agent string is CCBot/2.0 (https://commoncrawl.org/faq/). It honors robots.txt and supports Crawl-delay. Verify IPs at ccbot.json. Source: Common Crawl FAQ.
Blocking CCBot stops future crawling of your site by Common Crawl, but it does not retroactively remove content from existing archives. Third-party use of data captured before the block is a separate issue outside CCBot's control.
ByteDance (Bytespider)
Bytespider is the Toutiao search crawler operated by ByteDance. Official Toutiao documentation describes it as the crawler used to fetch webpages for the Toutiao search ecosystem and confirms support for the Robots Exclusion Protocol. Source: Toutiao: Cooperation instructions; Toutiao: Bytespider introduction.
The officially published example UA string is shown in the master table above. Describe its broader purpose cautiously: no current official IP-range verification file was found.
Mistral AI: MistralAI-User, MistralAI-Index, MistralAI-Training
Mistral AI provides a clean three-way separation comparable to OpenAI's approach. Source: Mistral: Mistral crawlers.
MistralAI-User handles user-triggered page fetches in Vibe (Mistral's chat product).
MistralAI-Index performs automated indexing for Mistral search. Mistral explicitly states it is not used for generative AI training.
MistralAI-Training builds datasets for Mistral's generative AI model training. It is not used for search indexing or live Vibe queries.
This means you can block training (MistralAI-Training) while keeping your site visible in Mistral search (MistralAI-Index) and accessible to user requests (MistralAI-User).
How to block (or allow) AI crawlers in robots.txt
All rules go in /robots.txt at the root of each relevant hostname. These copy-and-paste blocks cover the most common scenarios. Per RFC 9309, robots.txt matching on product tokens is case-insensitive.
Allow AI search while blocking training (OpenAI example)
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
Caveat: OpenAI says ChatGPT-User is user-triggered and robots.txt may not fully apply. Including the Allow directive here is a declaration of intent, not a guarantee the bot will check it.
Block Anthropic training while preserving Claude search
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
Anthropic states all three bots honor robots.txt, making this configuration more reliable than equivalents for operators whose user-triggered bots may ignore the file.
Opt out of Google Gemini use without blocking Google Search
User-agent: Google-Extended
Disallow: /
Warning: Do not add User-agent: Googlebot / Disallow: / unless you intend to remove your site from Google Search entirely. Google-Extended is a separate token that does not affect search indexing or ranking.
Block Apple foundation-model training while retaining Apple discovery
User-agent: Applebot-Extended
Disallow: /
Do not block regular Applebot unless you also want to reduce Spotlight, Siri, and Safari discovery.
Key implementation cautions
- robots.txt is a voluntary request, not a technical access wall (RFC 9309). Treat it as a signal, not enforcement.
- User-Agent strings can be forged. Any HTTP client can send any User-Agent header.
- Combine with WAF, CDN, or IP controls for hard enforcement. Match the bot's declared User-Agent and its published IP ranges where possible.
- Rules must be placed on every relevant subdomain. A robots.txt file at
example.comdoes not apply toblog.example.com. - Do not use
User-agent: * / Disallow: /casually. This blocks Googlebot, Bingbot, and all other compliant crawlers. - Crawl-delay is non-standard and support varies: Anthropic and Common Crawl support it; Amazon explicitly does not.
- robots.txt caching means changes are not instant. OpenAI and Meta report roughly 24-hour propagation. Amazon may cache for up to 30 days.
How to verify genuine AI crawler traffic
A User-Agent header is trivially spoofed. Any script can claim to be GPTBot or ClaudeBot. Verification requires a second factor beyond the header itself.
Method 1: IP range matching. Compare the request's source IP against the operator's published IP-range JSON files (linked in the master table above). Automate this check by periodically pulling the live JSON; never hard-code a static list, as ranges change.
Method 2: Forward-confirmed reverse DNS (FCrDNS). Perform a reverse DNS lookup on the request IP to get a hostname, then perform a forward DNS lookup on that hostname to confirm it resolves back to the original IP. Google documents this as its preferred verification method. Source: Google: Verify requests from Google crawlers and fetchers.
Method 3: CDN/WAF bot management. Services like Cloudflare, Akamai, and Fastly maintain verified-bot databases and can flag or challenge unverified User-Agent claims automatically. Cloudflare Radar is a useful resource for monitoring AI-bot traffic patterns across the web.
For the strongest reliability, combine User-Agent matching with IP verification. Keep your IP allowlists automated by pulling from each operator's live JSON endpoint rather than maintaining a manual spreadsheet.
Visibility and server-load tradeoffs of blocking
Blocking decisions are not binary, and consequences vary by crawler purpose.
Blocking training crawlers (GPTBot, ClaudeBot, CCBot, MistralAI-Training, meta-externalagent, Amazonbot) generally has no effect on traditional SEO or AI-search visibility. Your site remains in Google Search, Bing, and the AI operator's own search product (provided you allow the search-specific crawler).
Blocking AI-search crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot, MistralAI-Index, meta-webindexer, Amzn-SearchBot) reduces your site's citations and answers in those AI search platforms. This tradeoff is directly comparable to blocking Googlebot for traditional search.
Blocking user-triggered fetchers (ChatGPT-User, Perplexity-User, meta-externalfetcher, Amzn-User, Claude-User) may not work through robots.txt alone because operators state these requests may bypass it. WAF rules or IP-based blocking are more effective.
Blocking control tokens (Google-Extended, Applebot-Extended) does not affect traditional search at all. It only governs how crawled data may be reused for AI purposes.
Server-load concerns are legitimate. Training crawlers can be aggressive. If server resources are a concern but you do not want to block entirely, rate limiting via HTTP 429 Too Many Requests responses is an alternative to a full Disallow. Operators that support Crawl-delay (Anthropic, Common Crawl) can also be throttled through robots.txt.
FAQ
What is the difference between an AI crawler and a search engine crawler?
A traditional search-engine crawler (Googlebot, Bingbot) indexes pages for search results. An AI crawler may collect data for model training, AI-search indexing, or user-triggered retrieval. Some bots serve both purposes; many AI companies now operate separate crawlers for each job.
Can robots.txt stop AI bots completely?
No. RFC 9309 defines robots.txt as a voluntary request, not a technical barrier. User-triggered fetchers from OpenAI, Perplexity, Meta, and Amazon may not follow robots.txt. For hard enforcement, combine robots.txt with WAF rules, IP blocking, or authentication.
Will blocking AI crawlers hurt my SEO?
Blocking AI-specific crawlers (GPTBot, ClaudeBot, CCBot) does not affect Google Search or Bing rankings. Blocking Google-Extended does not affect Google Search inclusion. Only blocking Googlebot or Bingbot directly impacts traditional search visibility.
How do I find AI crawlers in my server logs?
Search server access logs or CDN analytics for known AI user-agent tokens (GPTBot, ClaudeBot, PerplexityBot, etc.). Cross-reference the source IP against the operator's published IP ranges to confirm the request is genuine and not spoofed.
How do I allow AI search crawlers but block training crawlers?
Use separate robots.txt rules for each bot. For example, disallow GPTBot (training) while allowing OAI-SearchBot (search). Disallow ClaudeBot (training) while allowing Claude-SearchBot (search). Disallow MistralAI-Training while allowing MistralAI-Index. Each operator provides distinct tokens for this purpose.