What is a crawler allowlist?
A crawler allowlist is a site's written decision about which crawlers may access it, with every allowed bot named and everything else refused or left to a default rule. A typical crawler allowlist lets Googlebot, Bingbot, OAI-SearchBot, Claude-SearchBot and PerplexityBot in, keeps training crawlers out, and blocks any bot that claims to be one of those but comes from the wrong IP address.
A crawler allowlist usually lives in two places at once:
- robots.txt, where each permitted crawler gets its own named group. This is a request that well-behaved bots honor.
- The firewall or CDN, where requests are checked against each company's published IP ranges. This is enforcement.
The term overlaps with "crawler whitelist" and "bot allowlist", which mean the same thing.
How does a crawler allowlist work in robots.txt?
A crawler allowlist in robots.txt works by giving each permitted crawler its own User-agent group and putting a default rule in the wildcard group for everything else. Google's documentation explains the mechanism that makes this possible: a crawler follows only the single group with the most specific user agent that matches it, matching is case-insensitive, and only one group ever applies [1].
A strict, default-deny allowlist looks like this:
User-agent: *
Disallow: /
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /Any crawler not named falls into the * group and is asked to stay out. A softer allowlist keeps User-agent: * open and names only the bots that need special treatment.
The most common mistake follows from the same rule. Because a named group replaces the wildcard group completely [1], a line such as Disallow: /checkout/ written under User-agent: * does not apply to any crawler that has its own group. Shared rules have to be copied into every named group.
Why do you need a crawler allowlist for AI bots?
A crawler allowlist is needed for AI bots because each major AI company now runs separate crawlers for model training, for search and for user requests, and a single "block AI" rule cannot tell them apart. Blocking the wrong one costs visibility; allowing the wrong one hands over training data you meant to keep.
| Company | Training crawler | Search crawler | User-request fetcher |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | None (Perplexity says PerplexityBot is not used for training) | PerplexityBot | Perplexity-User |
| Google-Extended (a token, not a crawler) | Googlebot | Not covered here |
Sources for the table: OpenAI [2], Anthropic [3], Perplexity [4] and Google [5].
OpenAI and Anthropic both say their bots are controlled independently [2][3]. A site can allow OAI-SearchBot and Claude-SearchBot so it can be cited in ChatGPT and Claude, while blocking GPTBot and ClaudeBot to stay out of training. Google-Extended is the odd one out: it has no user agent of its own, and Google reads the token only to decide whether crawled content may be used for Gemini training and grounding, without any effect on Google Search [5].
What is the difference between a crawler allowlist and a blocklist?
A crawler allowlist names the bots that may enter and treats every other bot as refused by default, while a blocklist names the bots to refuse and lets every other bot in. The difference shows up when a new crawler appears: an allowlist keeps it out until someone adds it, and a blocklist lets it in until someone notices.
Neither approach is right for every site. A default-deny allowlist suits sites with sensitive or licensed content, but it also blocks the next new AI search crawler until someone updates the file. A blocklist is easier to maintain but admits every new training crawler by default. Many sites end up with a hybrid: open by default, with named groups for the AI bots they have made a decision about.
Is robots.txt enough to allowlist crawlers?
No. Robots.txt is a request that compliant bots follow, not a lock, and it cannot check whether a bot is who it says it is. A real crawler allowlist also needs firewall or CDN rules that verify each bot by IP address.
Two things make the firewall layer necessary:
- User agents can be faked. Any scraper can send
GPTBotorGooglebotin its user agent. Google tells site owners to verify its crawlers by reverse DNS lookup or by matching IP addresses against its published ranges [6]. OpenAI, Anthropic and Perplexity each publish IP lists for their bots for the same reason [2][3][4]. - Some fetchers do not follow robots.txt. Perplexity says Perplexity-User generally ignores robots.txt because a user requested the fetch [4], and OpenAI says robots.txt rules may not apply to ChatGPT-User for the same reason [2]. In August 2025, Cloudflare reported that Perplexity reached blocked test sites using a generic browser user agent and IP addresses outside its published ranges [8]. Only network rules can act on traffic like that.
The firewall cuts both ways. Perplexity notes that sites behind a web application firewall may need to allowlist its IP ranges before PerplexityBot can reach them, and that the ranges change regularly [4].
How did crawler allowlists become standard practice?
Crawler allowlists moved from a niche security practice to an everyday SEO task between 2023 and 2026, as AI companies launched named crawlers and network providers began blocking them by default. These are the dated changes that shaped it.
August 2023: the first wave of AI blocks
OpenAI published GPTBot's user agent, and within weeks The New York Times, CNN, Reuters and others had blocked it in robots.txt [9]. Most of those rules were blocklist entries, written one bot at a time.
July 2025: Cloudflare blocks AI crawlers by default
On July 1, 2025, Cloudflare started blocking AI bots by default on its customers' sites, with controls to allow specific crawlers case by case [7]. For Cloudflare customers, AI access became opt-in, which turned "which bots do we allow?" into a question every site owner had to answer.
August 2025: verification beats user agents
Cloudflare's report on Perplexity's undeclared crawling, and its removal of Perplexity from its verified bot list, made the case that an allowlist must check IP addresses, not just names [8].
December 2025 to February 2026: three-bot structures
OpenAI revised its documentation so robots.txt rules apply to GPTBot and OAI-SearchBot but not necessarily to ChatGPT-User [2], and Anthropic documented ClaudeBot, Claude-SearchBot and Claude-User as three separate bots on February 20, 2026 [10]. An allowlist now has to make three decisions per company, not one.
How do you build a crawler allowlist?
To build a crawler allowlist, list every crawler that matters to the site, decide allow or block for each one, write a named robots.txt group per bot, and mirror the decision in the firewall using each company's published IP ranges.
- Audit what is visiting. Log file analysis shows which bots actually request pages and which ones are hitting errors or blocks.
- Check the CDN and firewall defaults first. A site on Cloudflare may already be blocking AI crawlers without any robots.txt rule [7].
- Decide per purpose, not per company. Search crawlers drive citations; training crawlers do not. Decide on each separately [2][3].
- Write one robots.txt group per bot, and copy shared rules into each group [1].
- Allow by verified IP in the firewall, using the JSON lists OpenAI, Anthropic and Perplexity publish, and update them automatically [2][3][4].
- Repeat on every subdomain. Anthropic notes that robots.txt rules apply per subdomain [3].
- Recheck after changes. OpenAI and Perplexity say robots.txt changes can take up to about 24 hours to apply [2][4].
What mistakes break a crawler allowlist?
The mistakes that break a crawler allowlist are usually ones of omission: a named group that drops shared rules, an allowed bot blocked at the firewall, or a rule written for a bot name that no longer exists.
- Allowing in robots.txt, blocking at the edge. An
Allowline means nothing if the CDN refuses the request first [7]. - Out-of-date bot names. Anthropic's older
anthropic-aiandClaude-Webtokens are deprecated [11], so an allowlist written around them does not describe Anthropic's current crawlers. - One rule for all of a company's bots. Blocking GPTBot does not block OAI-SearchBot, and allowing OAI-SearchBot does not allow GPTBot [2].
- Trusting user agents. A request that claims to be an allowed bot but comes from an unlisted IP address should be treated as an unknown scraper [6].
- Using
Crawl-delayeverywhere. Anthropic supports it for ClaudeBot [3], but Google says its crawlers ignore it [1].
Should every site have a crawler allowlist?
Every site that cares about AI search visibility should have at least a written crawler allowlist, even if the robots.txt stays open by default. The value is the decision itself: knowing which AI search crawlers can reach the pages you want cited, and confirming that nothing upstream is blocking them.
Small sites can keep it simple: allow the major search crawlers and AI search crawlers by name, make a conscious choice about training crawlers, and check the CDN's AI bot setting. Larger publishers and sites with licensed content usually need the full version, with default-deny rules and IP verification at the firewall.