AI crawler list 2026: every AI user agent, and what allowing it does
Most AI companies now run three kinds of bot: a training crawler, a search indexer and a fetcher that visits pages when a user asks. Block the training crawlers if you want to opt out of training, but keep the search indexers and user fetchers allowed, because they decide whether AI assistants can find, read and cite you.
- 31 user agents, 11 companies: this list covers every AI-related crawler, fetcher and robots.txt control token that OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, Microsoft, Mistral, DuckDuckGo and Common Crawl document on their own sites, checked on 11 November 2026.
- Search and training are separate switches: OpenAI says its bot settings are independent, so a site can allow
OAI-SearchBotto appear in ChatGPT search while disallowingGPTBotfor training. Anthropic, Perplexity, Amazon, Mistral and Meta also run separate search and training bots. - User-triggered fetchers may skip robots.txt: OpenAI says robots.txt rules "may not apply" to
ChatGPT-User, Perplexity saysPerplexity-User"generally ignores robots.txt rules", and Google says the same of its user-triggered fetchers, including the newGoogle-Agent. - Two tokens never visit your site:
Google-ExtendedandApplebot-Extendedare control tokens. Apple says Applebot-Extended "does not crawl webpages"; it only decides how data crawled by Applebot is used. - Google-Extended also covers Gemini grounding: Google's documentation says the token controls both Gemini training and grounding in Gemini Apps, and that sites which allow it "are eligible to appear in responses and receive clicks via grounding in Gemini Apps".
- A shared "no AI training" line is coming: Google intends to support the IETF
__usage_ai-trainingtoken in early 2027, and Anthropic says it began recognising an aipref-based "no AI training" signal on 30 October 2026.
The three kinds of AI bot
Before the list, it helps to know what you are deciding. Every AI user agent falls into one of these roles:
| Role | What it does | If you block it |
|---|---|---|
| Training crawler | Collects pages that may be used to train AI models | Your future content is excluded from that company's training data. Your visibility in its search answers is usually unaffected |
| Search indexer | Builds the index an AI assistant searches when it answers | You are less likely, or unable, to be cited and linked in that assistant's answers |
| User-triggered fetcher or agent | Opens a page because a person asked the assistant to, or to complete a task for them | The assistant can't read your page when a user asks about it. Some of these ignore robots.txt anyway |
| Control token | Not a crawler. A name you use in robots.txt to set a preference about how data is used | Only the use changes. Crawling for search continues |
Most websites that want AI visibility should allow the search indexers and user fetchers. The training crawlers are a business decision: blocking them doesn't remove you from AI search, and allowing them doesn't guarantee citations.
The full AI crawler list (November 2026)
"Follows robots.txt" is what each company says in its own documentation. Where a company says nothing, we say so.
OpenAI
| User agent | Role | Follows robots.txt? | What allowing it does |
|---|---|---|---|
OAI-SearchBot |
Search indexer | Yes | Lets your pages appear in ChatGPT search answers. OpenAI says opted-out sites "will not be shown in ChatGPT search answers" except as navigational links |
GPTBot |
Training crawler | Yes | Content may be used to train OpenAI's foundation models |
ChatGPT-User |
User-triggered fetcher | "May not apply" | ChatGPT and Custom GPTs can open your page when a user asks. OpenAI says it is not used to decide what appears in search |
OAI-AdsBot |
Ads checker | Not stated | Only visits pages submitted as ads on ChatGPT, to check them against policy. OpenAI says its data is not used for training |
Anthropic
| User agent | Role | Follows robots.txt? | What allowing it does |
|---|---|---|---|
Claude-SearchBot |
Search indexer | Yes | Helps your pages show up and be cited in Claude's search results |
Claude-User |
User-triggered fetcher | Yes | Claude can fetch your page when a user asks. Anthropic says blocking it "may reduce your site's visibility for user-directed web search" |
ClaudeBot |
Training crawler | Yes | Content may be used for training. Blocking it signals that future content should be excluded |
Perplexity
| User agent | Role | Follows robots.txt? | What allowing it does |
|---|---|---|---|
PerplexityBot |
Search indexer | Yes | Lets Perplexity surface and link your site. Perplexity says it is not used to crawl content for AI foundation models |
Perplexity-User |
User-triggered fetcher | "Generally ignores" it | Perplexity can visit your page to answer a user's question and link to it |
| User agent | Role | Follows robots.txt? | What allowing it does |
|---|---|---|---|
Googlebot |
Search crawler | Yes | Google Search, including "all Google Search features", Discover, Images and News. AI Overviews and AI Mode are Search features |
Google-Extended |
Control token | n/a, no separate crawler | Allows content Google crawls to be used for Gemini training and for grounding in Gemini Apps and Vertex AI. Google says it does not affect inclusion or ranking in Google Search |
GoogleOther |
General-purpose crawler | Yes | Used by product teams for one-off crawls, such as internal research. Google says it doesn't affect any specific product |
Google-CloudVertexBot |
Site-owner-requested crawler | Yes | Crawls requested by site owners building Vertex AI Agents. No effect on Google Search |
Google-Agent |
User-triggered agent | Generally ignores it | Used by agents on Google infrastructure "to navigate the web and perform actions upon user request" |
Google-GeminiNotebook |
User-triggered fetcher | Generally ignores it | Fetches URLs that Gemini Notebook users add as sources. It replaced Google-NotebookLM, which Google supported until August 2026 |
Apple
| User agent | Role | Follows robots.txt? | What allowing it does |
|---|---|---|---|
Applebot |
Search crawler | Yes | Search in Spotlight, Siri and Safari. Apple says its data may also be used to train Apple foundation models and as context for AI answers in Siri and Search |
Applebot-Extended |
Control token | n/a, it "does not crawl webpages" | Allows Applebot data to train Apple's foundation models. Disallowing it doesn't remove you from Apple search |
Meta
| User agent | Role | Follows robots.txt? | What allowing it does |
|---|---|---|---|
Meta-WebIndexer |
Search indexer | Yes | Meta says allowing it "helps us cite and link to your content in Meta AI's responses" |
Meta-ExternalAgent |
Training and indexing crawler | Yes | Used "for use cases such as training foundation AI models or improving products by indexing content directly" |
Meta-ExternalFetcher |
User-triggered fetcher | "May bypass" it | Fetches links at a user's request, including helping AI complete tasks on websites |
Meta-ExternalAds |
Ads crawler | Yes | Improves advertising and other business products |
FacebookExternalHit |
Link previews | Yes, except security checks | Builds the title, description and image when your link is shared on Facebook, Instagram or Messenger |
Amazon
| User agent | Role | Follows robots.txt? | What allowing it does |
|---|---|---|---|
Amzn-SearchBot |
Search indexer | Yes | Makes your content eligible for search experiences such as Alexa. Amazon says it does not crawl for AI training |
Amazonbot |
Product and training crawler | Yes | Improves Amazon products and "may be used to train Amazon AI models" |
Amzn-User |
User-triggered fetcher | "May not follow all" directives | Fetches live information to answer a customer's question, for example in Alexa. Not used for training |
Microsoft, Mistral, DuckDuckGo and Common Crawl
| User agent | Company | Role | Follows robots.txt? | What allowing it does |
|---|---|---|---|---|
Bingbot |
Microsoft | Search crawler | Yes | Bing's standard crawler, which "handles most of our crawling needs". Microsoft's crawler page lists no separate AI training bot |
MistralAI-Index |
Mistral | Search indexer | Yes, via its robots.txt tag | Indexes pages for Mistral search, which answers questions in Vibe. Not used for training |
MistralAI-User |
Mistral | User-triggered fetcher | Yes, the token controls these requests | Visits a page when a Vibe user asks, and can link to it |
MistralAI-Training |
Mistral | Training crawler | Yes | Builds datasets to train Mistral's models. Not used for search |
DuckAssistBot |
DuckDuckGo | AI answer crawler | Yes, change takes 72 hours | Crawls pages in real time for DuckDuckGo's AI-assisted answers. Not used for training, and blocking it doesn't affect organic rankings |
CCBot |
Common Crawl | Open web archive | Yes | Adds your pages to Common Crawl's open dataset, which anyone can download and analyse |
That's 31 user agents. You will see others in your logs, such as Bytespider, but we only list bots whose owners publish documentation we could check.
Which ones should you allow?
For a business that wants to be found and cited by AI assistants, this is the setup we'd start from:
| Decision | User agents | Why |
|---|---|---|
| Allow (AI search) | OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Amzn-SearchBot, MistralAI-Index, DuckAssistBot, Googlebot, Bingbot, Applebot |
These decide whether you appear in AI answers |
| Allow (user requests) | ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, Meta-ExternalFetcher, Amzn-User |
A person asked about your page. Blocking them mostly frustrates potential customers |
| Your choice (training) | GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot, MistralAI-Training, CCBot |
Opting out keeps future content out of training. It won't remove you from AI search, with one caveat: blocking Google-Extended also opts you out of grounding in Gemini Apps |
If being cited inside the Gemini app matters to you, read Google's description of the token before you block it.
A robots.txt template you can copy
This version allows AI search and user fetchers and opts out of training. To allow training as well, change the last group to Allow: /.
# AI search indexers: allowed, so you can be cited
User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
User-agent: Meta-WebIndexer
User-agent: Amzn-SearchBot
User-agent: MistralAI-Index
User-agent: DuckAssistBot
Allow: /
# Fetchers acting for a user: allowed
User-agent: ChatGPT-User
User-agent: Claude-User
User-agent: Perplexity-User
User-agent: MistralAI-User
Allow: /
# AI training: opted out
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: Amazonbot
User-agent: MistralAI-Training
User-agent: CCBot
Disallow: /
# Everyone else, including Googlebot, Bingbot and Applebot
User-agent: *
Allow: /
Sitemap: https://www.example.com/sitemap.xml
Remember the "one group per bot" rule: a crawler follows only the most specific User-agent group that matches it, so a bot named in a group ignores your User-agent: * rules. Our guide on allowing AI search bots while blocking AI training walks through this step by step.
Three rules that catch people out
Some bots borrow other bots' rules. Apple says that if your robots.txt doesn't mention Applebot but does mention Googlebot, Applebot follows the Googlebot rules. Amazon says Amzn-SearchBot follows the rules you give "other search bots" when it isn't named. A strict Googlebot group can therefore affect Apple and Amazon too.
Robots.txt is a request, not a lock. It asks bots to stay out. Your firewall or CDN decides what is actually served, and user agent names can be faked. To check that the bots you allowed can really get in, see our guide on checking which AI crawlers can access your website. OpenAI, Perplexity, Amazon, Apple, DuckDuckGo and Common Crawl publish IP ranges or reverse DNS so you can verify real visits.
Blocking by IP can backfire. Anthropic warns that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out", because its bots then can't read your robots.txt. Say what you want in robots.txt first.
For the files around robots.txt, see robots.txt vs llms.txt vs sitemap.xml, or browse all our AI crawlers & robots.txt guides.
What we found on our own site
When we built this list, we checked it against our own robots.txt. It names 19 bots, but not three of the newer search indexers: Meta-WebIndexer, Amzn-SearchBot and MistralAI-Index. They are still allowed through our User-agent: * group, so nothing is blocked. But it shows how quickly the list moves, and why we recheck it every month. Google's own page, for example, says Google-NotebookLM was only supported until August 2026 before Google-GeminiNotebook replaced it.
Check your own site
Want to know which of these bots your site lets in? The free Web2Agents AI check tests 12 AI bots against your homepage and gives you a readiness score out of 30. You can build a new file with the free robots.txt and llms.txt generator, and the €19 AI-Ready Kit tests 19 search and AI bots against both your robots.txt and what your server actually returns.
FAQ
What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls content that may be used to train OpenAI's models. OAI-SearchBot indexes pages for ChatGPT search. OpenAI treats them as independent settings, so you can block GPTBot and still appear in ChatGPT search answers.
Does blocking Google-Extended remove me from AI Overviews?
No. Google says Google-Extended does not affect inclusion in Google Search, and AI Overviews are part of Search. But it does control grounding in Gemini Apps, so blocking it can keep your content out of Gemini app answers.
Which AI bots ignore robots.txt?
The user-triggered ones, according to their owners: OpenAI says robots.txt "may not apply" to ChatGPT-User, Perplexity-User "generally ignores" it, Meta-ExternalFetcher "may bypass" it, Amzn-User "may not follow all" directives, and Google's user-triggered fetchers, including Google-Agent, generally ignore it. Anthropic says Claude-User follows robots.txt.
Is there one line that blocks all AI training?
Not yet. The IETF is working on a shared token, User-agent: __usage_ai-training. Google intends to support it in early 2027, and Anthropic says it began recognising an aipref-based training signal on 30 October 2026. Until more companies confirm, keep disallowing each training bot by name.
What is Google-Agent?
A user agent Google uses for agents running on Google infrastructure that browse websites and take actions when a user asks. As a user-triggered fetcher, it generally ignores robots.txt. Google publishes its IP ranges in user-triggered-agents.json.
How often does this list change?
Often. Google, for example, supported Google-NotebookLM only until August 2026 and now uses Google-GeminiNotebook. We recheck every company's documentation each month and update this page.
Sources
- Overview of OpenAI crawlers, OpenAI Developers
- Does Anthropic crawl data from the web, and how can site owners block the crawler?, Claude Help Center
- Perplexity Crawlers, Perplexity Docs
- Google's common crawlers, Google Crawling Infrastructure
- Google user-triggered fetchers, Google Crawling Infrastructure, last updated 19 August 2026
- Supporting a shared AI training control for websites, Google Search Central Blog, 30 October 2026
- About Applebot, Apple Support
- Meta Web Crawlers, Meta for Developers, updated 21 May 2026
- Amazon Web Search crawlers, Amazon Developer, last updated 26 October 2026
- Which crawlers does Bing use?, Bing Webmaster Tools
- Mistral crawlers, Mistral Docs
- Is DuckAssistBot related to DuckDuckGo?, DuckDuckGo Help Pages
- CCBot, Common Crawl
- RFC 9309: Robots Exclusion Protocol, IETF
- Web2Agents check of www.web2agents.com/robots.txt, 11 November 2026
