What is ClaudeBot?
ClaudeBot is the web crawler Anthropic uses to collect publicly available web content for training its Claude AI models [1]. ClaudeBot identifies itself with the token ClaudeBot in its user agent, and site owners control it through robots.txt.
A typical robots.txt group for ClaudeBot looks like this:
User-agent: ClaudeBot
Disallow: /private/ClaudeBot is one of three Anthropic bots, and the three do different jobs [1]. Most confusion about ClaudeBot comes from treating it as the bot that lets Claude read or cite a website in real time. It is not. ClaudeBot gathers training data; the other two handle search and user requests.
What does ClaudeBot do?
ClaudeBot crawls the web to collect content that Anthropic says helps improve the usefulness and safety of its generative AI models [1]. In plain terms, ClaudeBot builds training data.
Anthropic says ClaudeBot respects robots.txt directives and does not try to get around anti-circumvention technology such as CAPTCHAs [1]. Blocking ClaudeBot signals that a site's future content should be excluded from training [1].
What is the difference between ClaudeBot, Claude-SearchBot and Claude-User?
ClaudeBot collects training data, Claude-SearchBot indexes pages to improve the quality of Claude's search results, and Claude-User fetches pages when a person asks Claude a question [1]. Each bot has its own robots.txt token, so each can be allowed or blocked on its own.
| Anthropic bot | What it does | What blocking it means |
|---|---|---|
| ClaudeBot | Collects web content for training Claude models | Future content is excluded from training |
| Claude-SearchBot | Navigates the web to improve search result quality | Claude cannot index the site for search, which may reduce its visibility in Claude's search answers |
| Claude-User | Visits pages when a Claude user asks a question | Claude cannot retrieve the site's content in response to user requests |
The practical upshot: a site that wants to stay out of training but still be found and cited by Claude should block ClaudeBot and allow Claude-SearchBot and Claude-User.
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /How do you block ClaudeBot?
To block ClaudeBot from an entire site, add this group to robots.txt [1]:
User-agent: ClaudeBot
Disallow: /Anthropic says the rule must be added to the robots.txt file of each subdomain separately [1]. A block on example.com does not cover blog.example.com.
One more rule catches people out. Under the robots.txt standard as Google documents it, a crawler follows only the single most specific group that names it [4]. Once robots.txt has a User-agent: ClaudeBot group, ClaudeBot ignores the User-agent: * group entirely, so any rule meant for all bots has to be repeated inside the ClaudeBot group.
Can you slow ClaudeBot down instead of blocking it?
Yes. ClaudeBot supports the Crawl-delay directive, which asks it to wait between requests [1]:
User-agent: ClaudeBot
Crawl-delay: 1This is a real difference between crawlers that many guides miss. Google says its crawlers do not support crawl-delay at all [4], so a site owner who learned robots.txt on Googlebot may assume the directive is dead. For ClaudeBot it works, which gives a site under load an option between full access and a full block.
Do the old anthropic-ai and Claude-Web user agents still work?
No. The older Anthropic tokens anthropic-ai and Claude-Web are deprecated [2]. Robots.txt rules that only name those tokens do not control Anthropic's current crawlers, so a site that blocked "Anthropic" years ago using those names may not be blocking anything today.
To check, open the site's robots.txt and search for anthropic-ai and Claude-Web. If either appears without a matching ClaudeBot, Claude-SearchBot or Claude-User group, the rule needs updating.
How has ClaudeBot changed over time?
Anthropic has moved from a single crawler name with broad wording to three named bots with separate purposes. The biggest documented change came in February 2026.
Before 2026: ClaudeBot and the older tokens
Anthropic's documentation referred only to ClaudeBot, with broader language about collecting data for model development [2]. Earlier user agents, anthropic-ai and Claude-Web, appeared in many robots.txt files and are now deprecated [2].
February 2026: three bots, three purposes
Anthropic updated its crawler documentation on February 20, 2026 [3]. The update listed ClaudeBot for training, Claude-SearchBot for search indexing and Claude-User for user-requested fetches, and confirmed support for robots.txt and Crawl-delay [3]. The update added a warning that blocking Claude-SearchBot may reduce a site's visibility in search results [2]. The three-bot split mirrors the structure OpenAI had already adopted with GPTBot, OAI-SearchBot and ChatGPT-User [2][5].
How do you know if a visit is really ClaudeBot?
To confirm a visit is really ClaudeBot, match the request's IP address against the list Anthropic publishes at claude.com/crawling/bots.json [1]. A user agent string is just text that any scraper can copy, so a log line that says ClaudeBot proves nothing on its own.
The same approach applies across search engines and AI companies. Google, for example, tells site owners to verify its crawlers by IP range or DNS lookup rather than by user agent [7]. If a "ClaudeBot" request comes from outside Anthropic's list, it is not from Anthropic, and Anthropic asks site owners who think its crawler is misbehaving to contact it with their domain details [1].
What is the difference between ClaudeBot and GPTBot?
ClaudeBot and GPTBot do the same job for different companies: ClaudeBot collects training data for Anthropic's Claude models, and GPTBot collects training data for OpenAI's models [1][5]. Both respect robots.txt, both have separate sister bots for search and user requests, and blocking either one is a training opt-out rather than a search opt-out.
The differences are in the details. ClaudeBot honors Crawl-delay [1]. OpenAI says GPTBot and its search crawler share information to avoid duplicate crawling, and that robots.txt changes take about 24 hours to apply [5]. Google's equivalent, Google-Extended, is different again: it is a robots.txt token with no crawler of its own [6].
Should you block ClaudeBot?
Block ClaudeBot only if you do not want your content used to train Claude models; blocking it does not improve or protect search visibility. Leave Claude-SearchBot and Claude-User open if you want Claude to find and cite your pages [1].
A short checklist:
- Name each bot. Keep a crawler allowlist with separate groups for ClaudeBot, Claude-SearchBot and Claude-User.
- Remove dead tokens. Replace
anthropic-aiandClaude-Webrules with current names [2]. - Use Crawl-delay for load problems rather than a full block [1].
- Repeat the file on every subdomain [1].
- Verify by IP, not by user agent [1].