Skip to content
Search Insider
Sign up free
← AI Search Infographics
AI crawlers robots.txt infographic: an annotated robots.txt file allowing search bots OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot and disallowing training bots GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot, plus notes on user-triggered fetchers.

Oct 5, 2026

AI Crawlers robots.txt Infographic: GPTBot, ClaudeBot and More

Which AI crawlers to allow in robots.txt: the search and answer bots that get you cited, the training bots you can block, and the fetchers that act on request.

Download the image Read the walkthrough

Embed this infographic

Every infographic here is free to republish, on any site, with a link back to the study it comes from. Paste this code and the credit is done for you.

<a href="https://searchinsider.io/infographics/ai-crawlers-robots-txt"><img src="https://searchinsider.io/research/ai-crawlers-robots-txt/ai-crawlers-robots-txt-infographic.png" alt="AI crawlers robots.txt infographic: an annotated robots.txt file allowing search bots OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot and Bingbot and disallowing training bots GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot, plus notes on user-triggered fetchers." style="max-width:680px;width:100%;height:auto;" /></a><p>Source: <a href="https://searchinsider.io/infographics/ai-crawlers-robots-txt">AI crawlers and robots.txt infographic</a> by Search Insider.</p>

Which AI crawlers should you allow in robots.txt?

The short answer: AI companies run separate bots for training models and for answering searches. Block the training bots if you like, but block the search bots and you disappear from that AI's answers.

Based on each company's own crawler documentation.

What it shows

  • Search and answer bots: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot.
  • Training bots: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot.
  • Google-Extended doesn't affect Google Search inclusion or ranking.
  • Sites that block OAI-SearchBot won't be shown in ChatGPT search answers.
  • OpenAI says robots.txt may not apply to ChatGPT-User; Perplexity says Perplexity-User generally ignores robots.txt.

What to do

  • Want AI citations: Allow the search bots.
  • Don't want training use: Block the training bots.
  • After updates: Recheck robots.txt.

The walkthrough

What is the difference between AI training bots and AI search bots?

Training bots collect content to train AI models. Search bots fetch pages so an AI can show and link them in answers. OpenAI, for example, uses GPTBot for training and OAI-SearchBot to surface websites in ChatGPT search [1].

What happens if you block GPTBot or OAI-SearchBot?

Blocking GPTBot signals your content shouldn't be used to train OpenAI's models [1]. Sites that opt out of OAI-SearchBot won't be shown in ChatGPT search answers, though they can still appear as navigational links [1].

Does blocking Google-Extended affect Google Search?

No. Google says Google-Extended does not affect a site's inclusion in Google Search and isn't used as a ranking signal [2]. It controls use of content for Gemini training and grounding [2].

Which Anthropic bots crawl the web?

ClaudeBot collects content for training, Claude-SearchBot improves search results, and Claude-User fetches pages when a user asks Claude a question [3]. Anthropic says its bots honor robots.txt [3].

What do PerplexityBot and Applebot-Extended do?

PerplexityBot surfaces and links websites in Perplexity search results and isn't used to train foundation models [4]. Applebot-Extended lets you opt out of Apple model training; pages that disallow it can still appear in Apple's search results [5].

Do AI bots always follow robots.txt?

Not the ones acting on a user's request. OpenAI says robots.txt rules may not apply to ChatGPT-User [1], and Perplexity says Perplexity-User generally ignores robots.txt [4].

What about Bing and Common Crawl?

Microsoft doesn't run a separate AI crawler; it offered page-level meta tags to limit use in Bing Chat answers in 2023 [6]. Common Crawl's CCBot builds an open web archive and can be blocked by its user-agent [7].

Sources

  1. Overview of OpenAI Crawlers, OpenAI
  2. Google's common crawlers, Google Search Central
  3. Does Anthropic crawl data from the web?, Claude Help Center
  4. Perplexity Crawlers, Perplexity
  5. About Applebot, Apple Support
  6. Announcing new options for webmasters to control usage of their content in Bing Chat, Bing Webmaster Blog
  7. CCBot, Common Crawl

More digital marketing infographics

404 errors infographic: a 404 banner with Google's view that 404s are normal, which status to return in four cases, three facts on how Google treats 4xx and soft 404s, a checklist for a good custom 404 page, and three steps on what to do.

SEO · Oct 6, 2026

Do 404 errors hurt your SEO?

Do 404 errors hurt SEO? How Google treats 404s, why soft 404s are the real problem, which status code to return, and how to build a good custom 404 page.

Download See the details
Embed code 
<a href="https://searchinsider.io/infographics/404-errors"><img src="https://searchinsider.io/research/404-errors/404-errors-infographic.png" alt="404 errors infographic: a 404 banner with Google's view that 404s are normal, which status to return in four cases, three facts on how Google treats 4xx and soft 404s, a checklist for a good custom 404 page, and three steps on what to do." style="max-width:680px;width:100%;height:auto;" /></a><p>Source: <a href="https://searchinsider.io/infographics/404-errors">404 errors infographic</a> by Search Insider.</p>
AI-generated content infographic: a quality not method banner, a split of fine uses versus uses against Google's rules, a pre-publish checklist (accuracy, metadata, who, how, why), a Merchant Center note on labeling AI images and product data, and three steps on what to do.

AI search · Oct 6, 2026

Does Google penalize AI-generated content?

Google's stance on AI-generated content: quality over production method, what breaks the scaled content abuse policy, and a checklist before publishing AI-assisted content.

Download See the details
Embed code 
<a href="https://searchinsider.io/infographics/ai-generated-content"><img src="https://searchinsider.io/research/ai-generated-content/ai-generated-content-infographic.png" alt="AI-generated content infographic: a quality not method banner, a split of fine uses versus uses against Google's rules, a pre-publish checklist (accuracy, metadata, who, how, why), a Merchant Center note on labeling AI images and product data, and three steps on what to do." style="max-width:680px;width:100%;height:auto;" /></a><p>Source: <a href="https://searchinsider.io/infographics/ai-generated-content">AI-generated content infographic</a> by Search Insider.</p>
Anchor text infographic: an example link with the anchor text underlined, four vague or stuffed anchors swapped for descriptive ones, four rules on image links, internal links, length and keyword stuffing, a note on Google's link spam examples, and three steps on what to do.

SEO · Oct 6, 2026

What is anchor text, and what should it say?

What anchor text is, what Google says it should say, good and bad examples, image link alt text, and which anchor practices Google lists as link spam.

Download See the details
Embed code 
<a href="https://searchinsider.io/infographics/anchor-text"><img src="https://searchinsider.io/research/anchor-text/anchor-text-infographic.png" alt="Anchor text infographic: an example link with the anchor text underlined, four vague or stuffed anchors swapped for descriptive ones, four rules on image links, internal links, length and keyword stuffing, a note on Google's link spam examples, and three steps on what to do." style="max-width:680px;width:100%;height:auto;" /></a><p>Source: <a href="https://searchinsider.io/infographics/anchor-text">Anchor text infographic</a> by Search Insider.</p>