What is an AI crawler?
An AI crawler is a bot run by an AI company that visits web pages and downloads them, the same way a search engine crawler does, but for AI products. GPTBot (OpenAI), ClaudeBot (Anthropic) and PerplexityBot (Perplexity) are the best-known examples. Each announces itself with a user-agent name in its requests, which is how site owners recognize it in server logs and address it in robots.txt.
The term "AI crawler" covers bots doing three different jobs, and the difference matters more than the company behind them. A site that blocks one kind can still be fully visible to the other two.
What are the three types of AI crawler?
The three types of AI crawler are training crawlers, AI search crawlers and user-triggered fetchers. The major AI companies now publish separate user agents for each job [1][2][3].
| Job | What it does | OpenAI | Anthropic | Perplexity |
|---|---|---|---|---|
| Training | Collects pages that may be used to train future models | GPTBot | ClaudeBot | None (Perplexity says its bots do not train models) |
| AI search indexing | Builds the index an AI search product cites from | OAI-SearchBot | Claude-SearchBot | PerplexityBot |
| User-triggered | Fetches a page live because a user asked about it | ChatGPT-User | Claude-User | Perplexity-User |
- Training crawlers. Disallowing GPTBot "signals that a site's content should not be used" for training, in OpenAI's words [1]. Anthropic says the same of ClaudeBot [2].
- AI search crawlers. These decide whether a site can be cited. OpenAI says sites that block OAI-SearchBot will not appear in ChatGPT search answers, apart from navigational links [1]. Anthropic says blocking Claude-SearchBot "may reduce your site's visibility and accuracy in user search results" [9].
- User-triggered fetchers. These act for one person in real time. OpenAI says that because the actions are started by a user, "robots.txt rules may not apply" to ChatGPT-User [1]. Perplexity says Perplexity-User "generally ignores robots.txt rules" [3].
Is Googlebot an AI crawler?
Googlebot is not usually listed as an AI crawler, but Google's AI features depend on it. AI Overviews and AI Mode cite pages from Google's normal search index, which Googlebot builds. Google has no separate AI search bot to block.
Google-Extended is often mistaken for one. It is a robots.txt token, not a crawler: Google says it "doesn't have a separate HTTP request user agent string," that it controls whether content may be used to train Gemini models, and that it "does not impact a site's inclusion in Google Search nor is it used as a ranking signal" [4]. Blocking Google-Extended does not remove a site from AI Overviews.
To leave Google's AI answers, a site uses the Search generative AI control in Search Console instead, which removes it from AI Overviews, AI Mode and Discover's AI features without affecting normal results [11]. Cloudflare's 2025 data shows why Googlebot sits in a category of its own: Googlebot alone made 4.5% of HTML requests across Cloudflare's network, more than all other AI bots combined at about 4.2% [6].
What is the difference between an AI crawler and a search engine crawler?
A search engine crawler collects pages to rank them as links, while an AI crawler collects pages to train a model or to quote them inside a written answer. Both fetch HTML and both read robots.txt. Three differences show up in practice:
- Return traffic. A search crawler's visits lead to a results page full of links. AI crawlers send far fewer visitors back. In July 2025, Cloudflare reported OpenAI crawling about 1,700 pages for every referral it sent, and Anthropic about 73,000 [7].
- JavaScript. Googlebot renders JavaScript. Vercel's December 2024 analysis found none of the major AI crawlers from OpenAI, Anthropic, Meta, ByteDance or Perplexity executed JavaScript, while AppleBot and Google's Gemini crawling did render it [5].
- Efficiency. In the same analysis, ChatGPT's crawler spent 34.82% of its fetches on 404 pages and Claude's 34.16%, against 8.22% for Googlebot [5].
The JavaScript point has a direct consequence. Content that only appears after client-side JavaScript rendering is invisible to most AI crawlers, so it cannot be cited from those crawls.
How much traffic do AI crawlers send to websites?
AI crawlers send a large and growing share of bot requests. Across one month on Vercel's network in late 2024, GPTBot made 569 million requests and Claude's crawler 370 million, together with AppleBot and PerplexityBot about 28% of Googlebot's 4.5 billion [5]. Cloudflare reported that user-triggered AI fetching grew more than 15 times during 2025 [6].
Request volume is not the same as visitors. The crawl-to-referral ratios above show most AI crawling returns few clicks [7], and no AI company publishes referral figures per site.
When did AI crawlers start?
Named AI crawlers became a mainstream concern in 2023, when OpenAI documented GPTBot, and the major AI companies have since split their crawling into separate training, search and user bots. The dates below are the changes site owners most need to know.
2023: GPTBot is documented
In August 2023, OpenAI published documentation for GPTBot, describing it as gathering web content to "improve future models" and explaining how to block it in robots.txt [10]. It gave site owners a named training crawler to refuse for the first time.
2024: the first measurements
Vercel published one of the first large measurements of AI crawler behavior in December 2024, covering request volumes, content types, 404 rates and the lack of JavaScript rendering [5].
2025: blocking becomes the default on Cloudflare
On July 1, 2025, Cloudflare began blocking AI crawlers by default for its customers' sites, letting owners allow specific bots case by case [8]. It also launched a managed robots.txt that asks training crawlers such as Google-Extended and Applebot-Extended not to use a site for training. At the time only 37% of the top 10,000 domains had a robots.txt file at all [7].
2026: one company, three bots
In February 2026, Anthropic's documentation moved from describing ClaudeBot alone to three bots with separate purposes, replacing older user agents named Claude-Web and anthropic-ai [9]. The three-way split now matches OpenAI's and Perplexity's published structure [1][3].
How do you check if AI crawlers visit your site?
To check if AI crawlers visit your site, search your server access logs for their user-agent names, then confirm the requesting IP addresses against the ranges each company publishes. A quick first pass:
grep -E "GPTBot|OAI-SearchBot|ChatGPT-User|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User" access.logUser-agent names are easy to fake, so verification matters. OpenAI publishes IP ranges for each crawler [1], Anthropic publishes an official list of bot IP addresses [2], and Perplexity publishes JSON endpoints it calls "the source of truth" [3]. Treat any request that claims an AI user agent from an unlisted IP as an impostor. This is the same discipline as log file analysis for Googlebot.
If you use Cloudflare, also check whether the default AI bot blocking is switched on for your zone. A site can be blocking AI crawlers without anyone having written a robots.txt rule [8].
Should you block AI crawlers?
Whether to block AI crawlers depends on which kind, because each block has a different cost. Search Insider's view is that most businesses that want to be cited should allow AI search crawlers and decide separately about training crawlers.
The robots.txt below allows AI search and citation while refusing model training:
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /- Blocking a training crawler keeps content out of future models but does not remove a site from AI search answers [1][2].
- Blocking an AI search crawler removes a site from that product's citations, which is usually the opposite of what an AEO strategy wants [1][9].
- Blocking a user-triggered fetcher in robots.txt may not work, because those fetchers may not follow robots.txt [1][3]. Only a firewall rule reliably stops them.
What is not known about AI crawlers?
No AI company publishes how often a crawl leads to a citation, how long crawled pages stay in its index, or how much a training crawl influences what a model later says about a brand. Robots.txt is also a request, not an enforcement mechanism. Search Insider does not know of any published audit showing how consistently each company's bots obey it, so the safest assumption is that a robots.txt rule states a preference and a firewall rule enforces it.