The old web had a simple deal: search engines crawled websites, indexed pages, and sent traffic back. Website owners allowed bots because visibility usually meant visitors.
That deal is changing.
AI crawlers and search bots now serve different purposes. Some help your pages appear in Google, Bing, ChatGPT Search, Perplexity, Claude, Siri, and other discovery systems. Others collect public web content for model training. Some are user-triggered fetchers, meaning they visit a page only when a person asks an AI assistant to open or summarize it.
This AI crawlers and search bots list explains what each major bot does, why it matters, and how website owners can manage access without accidentally hurting search visibility.
What Are AI Crawlers and Search Bots?
AI crawlers and search bots are automated systems that visit websites to read, index, retrieve, summarize, or train on web content.
They are not all the same.
A traditional search bot, such as Googlebot or Bingbot, crawls pages so they can appear in search results. An AI training crawler may collect public content to improve future AI models. An AI search crawler may index content so it can be cited in AI-generated answers. A user-triggered fetcher may visit a page because a user asked an AI tool to open it.
That difference matters because blocking the wrong bot can reduce visibility.
For example, blocking a training bot may protect your content from future model training. Blocking an AI search bot may reduce your chance of being cited or surfaced in AI answers. Blocking Googlebot can remove your site from Google Search.
Why Website Owners Need to Understand Bots in 2026
Bot management is now a technical SEO issue, a content strategy issue, and a business decision.
A publisher may want AI tools to cite its articles but not use those articles for model training. An e-commerce site may want Googlebot and Bingbot to crawl product pages but restrict aggressive scrapers. A SaaS company may want ChatGPT, Claude, or Perplexity to access public documentation because AI citations can drive brand discovery.
The challenge is that every crawler has a different purpose.
OpenAI, for example, separates its bots by function. OAI-SearchBot is used to improve search visibility in ChatGPT’s search features, while GPTBot is associated with training OpenAI’s generative AI foundation models. OpenAI states that these settings are independent, so a web admin can allow OAI-SearchBot while disallowing GPTBot.
That is the new SEO reality. Website owners need to stop treating all bots as one category.
Main Types of Crawlers Website Owners Should Know
Before looking at the full list, it helps to understand the main bot categories.
| Bot Type | What It Does | Example Bots | Main SEO Impact |
| Traditional search bots | Crawl and index pages for search engines | Googlebot, Bingbot, DuckDuckBot | Critical for search visibility |
| AI search crawlers | Crawl or index pages for AI search answers | OAI-SearchBot, PerplexityBot, Claude-SearchBot | Important for AI visibility and citations |
| AI training crawlers | Collect content that may improve AI models | GPTBot, ClaudeBot, Amazonbot, CCBot | Usually, there is no direct referral traffic |
| User-triggered fetchers | Visit pages when a user asks an AI tool to access them | ChatGPT-User, Perplexity-User, Claude-User | Helps AI assistants answer user requests |
| AI control tokens | Do not crawl directly, but control data use | Google-Extended, Applebot-Extended | Controls AI training or AI usage permissions |
This is why a modern robots.txt file should be intentional. A blanket block may feel safe, but it can also make your content invisible in valuable discovery systems.
Complete AI Crawlers and Search Bots List
Below is a practical list of major bots website owners are likely to see in logs or need to manage in robots.txt.
1. Googlebot
Operator: Google
User-agent token: Googlebot
Category: Traditional search bot
Purpose: Crawls pages for Google Search and related search features
Googlebot is still one of the most important search engine bots on the web. It crawls pages so Google can understand, index, and rank them.
Google states that its common crawlers are used to build Google search indexes and that they automatically obey robots.txt rules. Google also notes that user-agent strings can be spoofed, so site owners should verify real Googlebot traffic through IP and reverse DNS checks.
For most websites, blocking Googlebot is a serious SEO mistake unless the content is intentionally private, duplicated, staged, or not meant for search.
Best practice: Allow Googlebot to crawl important public pages. Use noindex for pages you do not want indexed, rather than blocking them with robots.txt.
2. Google-Extended
Operator: Google
User-agent token: Google-Extended
Category: AI control token
Purpose: Controls whether site content can be used for certain Google AI products
Google-Extended is different from Googlebot. It does not appear as a separate crawler in server logs. Google says Google-Extended does not have a separate HTTP user-agent string; instead, it is a robots.txt control token used for control purposes.
This means Googlebot may still crawl your site for Google Search even if you disallow Google-Extended.
Example robots.txt rule:
User-agent: Google-Extended
Disallow: /
Best practice: Do not confuse Google-Extended with Googlebot. Blocking Googlebot can affect search visibility. Blocking Google-Extended is a separate AI data-use preference.
3. Bingbot
Operator: Microsoft
User-agent token: bingbot
Category: Traditional search bot
Purpose: Crawls pages for Bing search results and Microsoft search experiences
Bingbot powers Microsoft Bing’s web index. It is important not only for Bing Search but also for many search experiences that depend on Bing data.
For websites that care about search traffic, Bingbot should usually be allowed to crawl public pages. Bing visibility also matters more now because Microsoft has integrated AI experiences into its search and productivity ecosystem.
Best practice: Allow Bingbot for important pages, especially if your site depends on search visibility beyond Google.
4. OAI-SearchBot
Operator: OpenAI
User-agent token: OAI-SearchBot
Category: AI search crawler
Purpose: Helps surface websites in ChatGPT search features
OAI-SearchBot is one of the most important AI search crawlers for website owners to understand. OpenAI says OAI-SearchBot is used to surface websites in ChatGPT’s search features. If a site opts out of OAI-SearchBot, it will not appear in ChatGPT search results, though it may still appear as a navigational link.
This makes OAI-SearchBot different from GPTBot.
If your goal is AI visibility, citations, and discovery inside ChatGPT Search, you should think carefully before blocking OAI-SearchBot.
Example allow rule:
User-agent: OAI-SearchBot
Allow: /
Best practice: Allow OAI-SearchBot to make your content eligible for ChatGPT search visibility.
5. GPTBot
Operator: OpenAI
User-agent token: GPTBot
Category: AI training crawler
Purpose: Crawls content that may be used to improve OpenAI foundation models
GPTBot is not the same as OAI-SearchBot. OpenAI says GPTBot is used to crawl content that may be used in training its generative AI foundation models, and disallowing GPTBot signals that the site’s content should not be used for that purpose.
This gives publishers more control. You can block GPTBot while still allowing OAI-SearchBot.
Example block rule:
User-agent: GPTBot
Disallow: /
Best practice: If you want ChatGPT search visibility but do not want your content used for training, allow OAI-SearchBot and disallow GPTBot.
6. ChatGPT-User
Operator: OpenAI
User-agent token: ChatGPT-User
Category: User-triggered fetcher
Purpose: Visits pages when ChatGPT users request actions or access
ChatGPT-User is different again. OpenAI says it is used for certain user actions in ChatGPT and Custom GPTs. It is not used for automatic web crawling, and because users initiate these actions, robots.txt rules may not apply.
That makes this bot more like a browser request made on a user’s behalf.
Best practice: Do not treat ChatGPT-User the same way you treat GPTBot. If users need ChatGPT to access your public documentation, product pages, or help center, blocking this fetcher may create a poor user experience.
7. ClaudeBot
Operator: Anthropic
User-agent token: ClaudeBot
Category: AI training crawler
Purpose: Collects public web content that may contribute to Claude model training
ClaudeBot is Anthropic’s crawler for collecting web content that may help improve its generative AI models. Anthropic says restricting ClaudeBot signals that the site’s future materials should be excluded from its AI model training datasets.
Example block rule:
User-agent: ClaudeBot
Disallow: /
Best practice: Block ClaudeBot if your main concern is preventing future use of your content for training the Anthropic model. Do not assume this also controls Claude search visibility.
8. Claude-SearchBot
Operator: Anthropic
User-agent token: Claude-SearchBot
Category: AI search crawler
Purpose: Improves search result quality and relevance for Claude users
Anthropic separates ClaudeBot, Claude-User, and Claude-SearchBot. Claude-SearchBot is used to improve search results for users by analyzing online content to generate search responses. Anthropic notes that disabling Claude-SearchBot may reduce a site’s visibility and the accuracy of its results in user search results.
Best practice: Allow Claude-SearchBot if you want your content to be discoverable and accurately represented in Claude search experiences.
9. Claude-User
Operator: Anthropic
User-agent token: Claude-User
Category: User-triggered fetcher
Purpose: Retrieves content when a Claude user asks for website access
Claude-User supports user-directed requests. Anthropic says disabling Claude-User prevents its system from retrieving your content in response to a user query, which may reduce visibility for user-directed web search.
Best practice: If your public content is useful to customers, researchers, or support users, be careful before blocking Claude-User.
10. PerplexityBot
Operator: Perplexity
User-agent token: PerplexityBot
Category: AI search crawler
Purpose: Surfaces and links websites in Perplexity search results
PerplexityBot is designed for AI search visibility. Perplexity says it is used to surface and link websites in Perplexity search results and is not used to crawl content for AI foundation model training.
That makes it a valuable bot for websites that want citations from AI search engines.
Example allow rule:
User-agent: PerplexityBot
Allow: /
Best practice: Allow PerplexityBot if your site benefits from being cited in AI answers.
11. Perplexity-User
Operator: Perplexity
User-agent token: Perplexity-User
Category: User-triggered fetcher
Purpose: Fetches pages when users ask Perplexity questions
Perplexity says Perplexity-User supports user actions and may visit a page to help answer a user’s question. It is not used for web crawling or foundation model training. Perplexity also states that because the fetch is user-requested, it generally ignores robots.txt rules.
Best practice: If you see Perplexity-User in the logs, understand that it may represent real user demand rather than automated crawling.
12. Applebot
Operator: Apple
User-agent token: Applebot
Category: Search bot and AI-related crawler
Purpose: Powers Apple search experiences, including Spotlight, Siri, and Safari
Applebot is Apple’s web crawler. Apple says data crawled by Applebot powers search technology across Apple experiences such as Spotlight, Siri, and Safari. Apple also says Applebot data may be used to help train Apple foundation models, while publishers can opt out of model training through Applebot-Extended.
Best practice: Allow Applebot if you want your content discoverable across Apple search surfaces. Use Applebot-Extended if your concern is AI training.
13. Applebot-Extended
Operator: Apple
User-agent token: Applebot-Extended
Category: AI control token
Purpose: Controls whether Applebot-crawled content can be used to train Apple Foundation models
Applebot-Extended does not crawl webpages itself. Apple says it is used to determine how data crawled by Applebot may be used. Pages that disallow Applebot-Extended can still be included in Apple search results.
Example block rule:
User-agent: Applebot-Extended
Disallow: /
Best practice: Use Applebot-Extended if you want Apple search visibility but do not want your content used to train Apple’s general-purpose foundation models.
14. Meta-ExternalAgent
Operator: Meta
User-agent token: meta-externalagent
Category: AI crawler
Purpose: Used by Meta for AI-related crawling and indexing
Meta lists its common web crawlers for web admins, including Meta-ExternalAgent and Meta-ExternalFetcher. Search result snippets from Meta’s developer documentation describe Meta-ExternalFetcher as fetching individual links at a user’s request and supporting product functions such as agentic AI evaluation and improvement.
Because Meta’s crawler documentation may be rate-limited or difficult to access, website owners should verify current Meta bot details from Meta’s official crawler page and server logs.
Example block rule:
User-agent: meta-externalagent
Disallow: /
Best practice: Monitor this bot separately from Facebook’s older social sharing crawlers. Blocking it may affect access to Meta AI, not just social previews.
15. Meta-ExternalFetcher
Operator: Meta
User-agent token: Meta-ExternalFetcher
Category: User-triggered or product-triggered fetcher
Purpose: Fetches individual links for Meta product and AI functions
Meta-ExternalFetcher appears to be more user-action-oriented than broad crawling. This distinction matters because user-triggered fetchers often behave differently from traditional crawlers.
Best practice: Do not block it mindlessly if your content is often shared, referenced, or accessed through Meta products.
16. Amazonbot
Operator: Amazon
User-agent token: Amazonbot
Category: AI and product improvement crawler
Purpose: Improves Amazon products and may be used to train Amazon AI models
Amazon says Amazonbot is used to improve its products and services, provide more accurate information to customers, and train Amazon’s AI models. Amazon also publishes its user-agent string and IP address information.
Example block rule:
User-agent: Amazonbot
Disallow: /
Best practice: Review your logs before making a decision. Some sites may see little benefit from Amazonbot. Others may prefer access if their content is useful for voice assistants or product-related discovery.
17. CCBot
Operator: Common Crawl
User-agent token: CCBot
Category: Web archive and AI training data source
Purpose: Crawls public web pages for Common Crawl’s open web corpus
CCBot is Common Crawl’s crawler. Common Crawl says CCBot identifies itself as CCBot/2.0 and provides a robots.txt block example for sites that do not want to be crawled.
Many AI companies and researchers have used Common Crawl datasets in their training pipelines, so CCBot is important even though it is not owned by any single AI assistant provider.
Example block rule:
User-agent: CCBot
Disallow: /
Best practice: If you want to reduce the chance of your public content appearing in open crawl datasets, review CCBot access.
18. DuckDuckBot
Operator: DuckDuckGo
User-agent token: DuckDuckBot
Category: Traditional search bot
Purpose: Improves DuckDuckGo search results
DuckDuckGo says DuckDuckBot is its web crawler and that it helps improve search results. The crawler’s user-agent appears as DuckDuckBot/1.1; (+http://duckduckgo.com/duckduckbot.html), and it respects robot rules.
Best practice: Allow DuckDuckBot if privacy-search visibility matters to your audience.
19. Bytespider
Operator: ByteDance
User-agent token: Bytespider
Category: Crawler, often associated with search, recommendations, and AI data collection
Purpose: Collects web data for ByteDance products and systems
Bytespider is commonly associated with ByteDance, the parent company of TikTok. Third-party bot directories describe it as collecting data for search functionality and content recommendations across ByteDance platforms.
Some website owners have reported aggressive crawl behavior from Bytespider, so it is a bot worth monitoring closely.
Example block rule:
User-agent: Bytespider
Disallow: /
Best practice: Check actual server load before deciding. If the bot consumes resources without sending meaningful traffic, consider rate limiting or blocking at the firewall level.
How to Decide Which Bots to Allow or Block
Do not build your robots.txt file from fear. Build it from strategy.
Ask four questions:
- Does this bot send or support valuable traffic?
- Does this bot help my brand appear in AI answers?
- Does this bot use my content for model training?
- Does this bot create server load or security concerns?
A small business blog may want broad discovery. A paid publisher may want strict AI training controls. A documentation site may want AI assistants to read public docs so users get better product answers. A private staging site should block almost everything.
There is no universal rule.
Recommended Bot Strategy for Most Public Websites
For most public websites, a balanced approach works best.
| Goal | Recommended Action |
| Rank in Google | Allow Googlebot |
| Rank in Bing | Allow Bingbot |
| Appear in ChatGPT Search | Allow OAI-SearchBot |
| Avoid using OpenAI for training | Disallow GPTBot |
| Appear in Perplexity answers | Allow PerplexityBot |
| Avoid using Claude for training | Disallow ClaudeBot |
| Appear in Claude search | Allow Claude-SearchBot |
| Appear in Apple search | Allow Applebot |
| Avoid using Apple AI for training use | Disallow Applebot-Extended |
| Avoid Common Crawl datasets | Disallow CCBot |
This approach protects some training use while keeping the door open for AI search visibility.
Sample robots.txt for AI Search Visibility With Training Limits
Here is a practical starting point for many publishers and business websites:
# Allow traditional search engines
User-agent: Googlebot
Allow: /
User-agent: bingbot
Allow: /
User-agent: DuckDuckBot
Allow: /
# Allow AI search visibility
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Limit AI training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
# Control Google and Apple AI data use without blocking search crawling
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
# Optional sitemap
Sitemap: https://www.example.com/sitemap.xml
This is not legal advice or a universal setting. It is a practical, SEO-friendly framework for websites that want visibility in search and AI-driven answers while limiting access to some model training.
Common Mistakes Website Owners Make
Mistake 1: Blocking All Bots With User-agent: *
This can destroy organic visibility if search engine crawlers are blocked.
User-agent: *
Disallow: /
Use this only for staging sites, private websites, or content that no one should crawl.
Mistake 2: Confusing Crawling With Indexing
Robots.txt controls crawling. It does not always remove a page from search results. Google explains that robots.txt is mainly used to control crawler access and is not the right method to keep a page out of Google; for that, use noindex or password protection.
If a page is already indexed and you block crawling, search engines may not see a noindex tag.
Mistake 3: Blocking AI Search Bots and Expecting AI Visibility
If you block OAI-SearchBot, PerplexityBot, or Claude-SearchBot, you may reduce the chance of your content appearing in AI-generated search results.
That may be fine for some publishers. But it should be a deliberate choice.
Mistake 4: Trusting User-Agent Strings Alone
User-agent strings can be spoofed. Google specifically warns that HTTP user-agent strings can be spoofed and recommends verification.
Better verification includes:
- reverse DNS lookup
- published IP ranges
- server log analysis
- WAF rules
- bot management tools
Mistake 5: Forgetting Subdomains
Robots.txt applies per host. If your main site is example.com and your docs are on docs.example.com, each needs its own robots.txt file.
This is especially important for SaaS companies, ecommerce stores, publishers, and multilingual websites.
Expert Checklist for Managing AI Crawlers and Search Bots
Use this checklist before updating robots.txt.
- Identify which bots currently visit your site.
- Separate search bots from AI training crawlers.
- Decide whether AI visibility matters for your brand.
- Allow Googlebot and Bingbot for public SEO pages.
- Allow AI search crawlers if you want AI citations.
- Block training crawlers only when they match your content policy.
- Add sitemap references.
- Test robots.txt after changes.
- Monitor server logs after deployment.
- Review rules every quarter because bot names change.
Should You Block AI Crawlers?
The honest answer: sometimes.
You should consider blocking AI training crawlers if your content is paid, licensed, sensitive, original research, or part of a commercial product. You may also block bots that create server strain or do not provide business value.
You should consider allowing AI search crawlers if your website depends on discovery, citations, leads, brand visibility, or educational reach.
The best strategy is selective access.
Do not block everything. Do not allow everything. Decide based on the bot’s purpose.
Final Recommendation
The best website owners in 2026 will manage bots like they manage SEO, analytics, and security. They will not use one global rule for every crawler.
Traditional search bots still matter. AI search crawlers now matter too. Training bots requires a separate decision.
A strong AI crawlers and search bots list gives you control over how your content is discovered, cited, crawled, and potentially used by AI systems. Keep your robots.txt file clean, review your logs regularly, and treat crawler access as part of your broader search visibility strategy.
FAQ
What is the difference between AI crawlers and search bots?
Search bots crawl websites to index pages for search engines. AI crawlers may collect, retrieve, or analyze content for AI products, AI search answers, user-requested browsing, or model training.
Should I block all AI crawlers?
Not always. Blocking all AI crawlers may reduce your visibility in AI search tools. A better approach is to allow AI search crawlers and block training crawlers if training use is your concern.
Does robots.txt stop AI companies from using my content?
Robots.txt is a public instruction file that compliant crawlers may follow. It is not a security system. For sensitive content, use authentication, paywalls, server-side restrictions, or legal controls.
Which AI crawlers should website owners know first?
The most important AI-related bots to understand are OAI-SearchBot, GPTBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, and Applebot-Extended.
Can I allow ChatGPT Search but block OpenAI training?
Yes. OpenAI separates OAI-SearchBot and GPTBot. You can allow OAI-SearchBot for ChatGPT search visibility while disallowing GPTBot to signal that your content should not be used for training OpenAI foundation models.
Is Google-Extended the same as Googlebot?
No. Googlebot crawls for Google Search. Google-Extended is a robots.txt control token related to Google AI data use and does not appear as a separate crawler in logs.
How often should I update my AI crawler rules?
Review your robots.txt and server logs at least every quarter. AI bot names, user-agent strings, and crawler purposes change frequently.
What is the safest setup for SEO?
For most public websites, allow major search bots like Googlebot and Bingbot. Then decide separately which AI search crawlers, user-triggered fetchers, and training crawlers fit your business goals.
