What is PerplexityBot?
PerplexityBot is the web crawler Perplexity uses to find pages it can surface and link in its AI search results [1]. When Perplexity answers a question and lists sources, PerplexityBot is the crawler that built the index those sources come from.
PerplexityBot identifies itself with this user agent string [1]:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)Unlike GPTBot and ClaudeBot, PerplexityBot is a search crawler, not a training crawler. Perplexity says PerplexityBot does not crawl content for training AI foundation models [1]. That single fact changes the decision about whether to block it.
What does PerplexityBot do?
PerplexityBot crawls and indexes web pages so Perplexity can surface and link them in search answers [1]. Perplexity's documentation says webmasters should allow PerplexityBot if they want their site to appear in Perplexity's results [1].
In practice, Perplexity works as an answer engine: it searches its index, reads the most relevant pages and writes an answer with numbered citations. That process is a form of retrieval-augmented generation. A page PerplexityBot cannot crawl is a page Perplexity's index does not hold, so it is far less likely to be retrieved or cited.
What is the difference between PerplexityBot and Perplexity-User?
PerplexityBot crawls the web to build Perplexity's search index and respects robots.txt, while Perplexity-User visits a page when a Perplexity user's question needs it and generally ignores robots.txt [1]. Perplexity's reasoning is that a Perplexity-User fetch was requested by a person, not started by an automated crawl [1].
| Perplexity bot | What it does | Respects robots.txt? | IP list |
|---|---|---|---|
| PerplexityBot | Surfaces and links websites in search results | Yes | perplexity.com/perplexitybot.json |
| Perplexity-User | Visits pages to answer a user's question | Generally no | perplexity.com/perplexity-user.json |
This difference has practical consequences. A robots.txt block on PerplexityBot keeps a site out of Perplexity's index, but it does not stop Perplexity-User from fetching a page a user asks about. Stopping Perplexity-User takes a server or firewall rule matched to its published IP addresses [1].
How do you block or allow PerplexityBot?
To block PerplexityBot, add this group to the site's robots.txt [1]:
User-agent: PerplexityBot
Disallow: /To allow PerplexityBot explicitly, which is what Perplexity recommends for sites that want to appear in its results [1]:
User-agent: PerplexityBot
Allow: /Perplexity says changes can take up to 24 hours to take effect [1]. Remember that a crawler obeys only the most specific group that names it [8]: once there is a User-agent: PerplexityBot group, the rules under User-agent: * no longer apply to PerplexityBot.
Robots.txt is only half the job. Perplexity notes that sites behind a web application firewall may need to allowlist its IP addresses for PerplexityBot to reach the content at all, and that the IP ranges change, so firewall rules should update automatically [1]. Since July 1, 2025, Cloudflare has blocked AI crawlers by default for its customers unless the site owner allows them [5], so a site on Cloudflare can be invisible to PerplexityBot without anyone having touched robots.txt. This is the case for keeping a deliberate crawler allowlist.
Does Perplexity respect robots.txt?
PerplexityBot respects robots.txt, according to Perplexity, but Perplexity-User generally does not, and Perplexity has twice been accused of reaching blocked content by other means [1][2][3]. The honest answer depends on which bot is meant.
The disputes are about fetches that did not identify themselves as PerplexityBot:
- In 2024, Wired reported that a machine tied to Perplexity reached areas of Wired and other Condé Nast sites that robots.txt restricted, and that Perplexity returned text verbatim from a test page [2]. The accusations were widely reported by July 2, 2024 [2]. Perplexity's response was that summarizing a URL a user supplied is not crawling [2].
- In August 2025, Cloudflare reported that after PerplexityBot and Perplexity-User were blocked on test domains, Perplexity still reached the content using a browser-like user agent and IP addresses outside its published ranges [3]. Cloudflare removed Perplexity from its verified bot list [3][4]. Perplexity argued that AI assistants acting on a user's request work differently from traditional crawlers [3].
Search Insider cannot verify either side's account. What the disputes show is narrower and useful: a robots.txt rule is a request, and a user agent string can be changed, so robots.txt alone is not an access control.
When did PerplexityBot's treatment change?
PerplexityBot itself has stayed a search crawler that follows robots.txt; what changed is how the rest of the web treats it. The key dates are about publishers and network providers reacting to Perplexity's crawling.
2024: the Wired investigation
Wired and developer Robb Knight reported Perplexity fetching pages from sites that blocked it, which Perplexity said were user-requested summaries, not crawling [2].
July 2025: Cloudflare blocks AI crawlers by default
On July 1, 2025, Cloudflare began blocking AI bots by default on its customers' sites, letting owners allow specific crawlers case by case [5]. PerplexityBot became something many sites had to opt in to.
August 2025: Cloudflare delists Perplexity
Cloudflare published findings that Perplexity used undeclared crawlers to get around no-crawl rules, and removed Perplexity from its verified bot list [3][4].
What is the difference between PerplexityBot and GPTBot?
PerplexityBot crawls to power Perplexity's search answers, while GPTBot crawls to train OpenAI's models [1][6]. Blocking PerplexityBot removes a site from Perplexity's results; blocking GPTBot only opts a site out of OpenAI training, because ChatGPT search uses a separate crawler, OAI-SearchBot [6].
PerplexityBot is the closer match to OAI-SearchBot and to Anthropic's Claude-SearchBot [7]. All three are the search-facing crawlers of their companies, and blocking any of them costs visibility in that company's AI answers.
Should you block PerplexityBot?
Most businesses that want to be cited in AI answers should allow PerplexityBot, because it is a search crawler that Perplexity says is not used for model training [1]. Blocking PerplexityBot protects nothing from training and removes the site from one of the answer engines that links its sources.
Block PerplexityBot only if you do not want to appear in Perplexity at all. Either way:
- Allow it in robots.txt and in the firewall. A robots.txt
Allowdoes nothing if a firewall or CDN blocks the request first [1][5]. - Decide on Perplexity-User separately, using IP rules, since robots.txt generally does not apply to it [1].
- Verify visits by IP address against Perplexity's published lists, not by user agent [1].
- Check log files to confirm PerplexityBot is reaching the pages you want cited.