What is GPTBot?
GPTBot is the web crawler OpenAI uses to collect publicly available pages for training its generative AI foundation models [1]. GPTBot identifies itself with a user agent string that contains the token GPTBot, and site owners control it through robots.txt like any other crawler.
GPTBot's current user agent string looks like this [1]:
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbotWhen GPTBot launched in 2023, the string ended in GPTBot/1.0 [2]. The version number changes, so server rules and log filters should match the token GPTBot, not the full string.
GPTBot is one of four OpenAI bots. The others fetch pages for ChatGPT search, for actions a ChatGPT user asks for, and for checking ad landing pages [1]. Treating "GPTBot" as shorthand for all of OpenAI's crawling is the most common mistake in guides to the bot.
What does GPTBot do?
GPTBot crawls web pages so OpenAI can use the content to train its foundation models [1]. GPTBot is not the bot that finds pages for ChatGPT's search answers. That job belongs to OAI-SearchBot.
When OpenAI announced GPTBot in August 2023, it said pages crawled by GPTBot were filtered to remove paywall-restricted sources, sources that violate OpenAI's policies and sources that gather personally identifiable information [3]. Coverage at the time described GPTBot more loosely as the crawler feeding ChatGPT's knowledge [2]. OpenAI's current documentation draws a clear line: GPTBot is for model training only [1].
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot collects training data, OAI-SearchBot finds pages to show in ChatGPT search results, and ChatGPT-User fetches pages when a person using ChatGPT or a custom GPT asks it to [1]. Each one is controlled separately in robots.txt.
| OpenAI bot | What it does | Respects robots.txt? | Effect of blocking it |
|---|---|---|---|
| GPTBot | Crawls content for training foundation models | Yes | Signals your content should not be used for training |
| OAI-SearchBot | Surfaces websites in ChatGPT search features | Yes | Your site stops appearing in ChatGPT search results, though navigational links may still show |
| ChatGPT-User | Visits pages for user-initiated actions in ChatGPT and custom GPTs | Not necessarily, since a user started the request | Limited; use server or firewall rules instead |
| OAI-AdsBot | Checks the safety of ad landing pages | Not stated | Applies only to ads; data is not used for training |
OpenAI says the settings are independent [1]. A site can block GPTBot and still allow OAI-SearchBot, which keeps the site eligible to be cited in ChatGPT search while opting out of training. OpenAI also says OAI-SearchBot and GPTBot share information to avoid crawling the same page twice [6], so allowing both does not double the load on a server.
How do you block GPTBot in robots.txt?
To block GPTBot from a whole site, add this group to the robots.txt file at the root of the domain [1][2]:
User-agent: GPTBot
Disallow: /To block GPTBot from part of a site only, list the folders [2]:
User-agent: GPTBot
Allow: /blog/
Disallow: /members/To opt out of training while staying visible in ChatGPT search, block GPTBot and allow OAI-SearchBot:
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Allow: /Three details trip people up:
- Changes are not instant. OpenAI says robots.txt updates take about 24 hours to be picked up [1].
- A named group replaces the wildcard group. Under the robots.txt standard as Google documents it, a crawler obeys only the single most specific group that matches its name [8]. If a site has a
User-agent: GPTBotgroup, GPTBot ignores everything inUser-agent: *. Rules meant for every bot must be repeated inside the GPTBot group. - Every hostname needs its own file. A robots.txt file only covers the host it sits on, so
shop.example.comneeds its own rules.
How do you check if GPTBot is crawling your site?
To check whether GPTBot is crawling a site, search the server access logs for the token GPTBot in the user agent field. Log file analysis shows which pages GPTBot requested, how often, and what status codes it received.
A user agent can be faked, so confirm the request really came from OpenAI by comparing its IP address against the list OpenAI publishes at openai.com/gptbot.json [1]. A request that says GPTBot but comes from an IP address outside that list is not GPTBot, and blocking it does not affect OpenAI at all.
When was GPTBot introduced?
OpenAI introduced GPTBot on August 7, 2023, publishing its user agent and instructions for blocking it [2][3]. OpenAI has since split its crawling across separate bots and narrowed GPTBot to training only.
August 2023: GPTBot launches
OpenAI documented GPTBot with the user agent GPTBot/1.0 and robots.txt instructions for blocking it in whole or in part [2]. It said the crawled pages were filtered to remove paywalled sources, sources that broke its policies and sources gathering personal information [3].
August 2023: publishers block it within weeks
Within three weeks, The New York Times, CNN, Reuters, the Chicago Tribune and the Australian Broadcasting Corporation were among the publishers blocking GPTBot in robots.txt [4]. An analysis by Originality.ai, reported on August 25, 2023, found more than 60 of the world's 1,000 most visited websites blocking it [4].
2024: search crawling separates from training
By late 2024, OpenAI documented GPTBot, OAI-SearchBot and ChatGPT-User as separate crawlers with separate robots.txt rules [5]. This made it possible to opt out of training without leaving ChatGPT search.
December 2025: ChatGPT-User drops robots.txt language
In December 2025, OpenAI revised its documentation so that robots.txt rules were described as applying to OAI-SearchBot and GPTBot, not ChatGPT-User, because ChatGPT-User's visits are started by a user [6]. The same revision stated that OAI-SearchBot and GPTBot share information to avoid duplicate crawling [6].
Does blocking GPTBot affect SEO or AI visibility?
Blocking GPTBot does not affect Google rankings and does not remove a site from ChatGPT search results. GPTBot is OpenAI's training crawler, so Google never sees the rule, and ChatGPT search relies on OAI-SearchBot, which is controlled separately [1].
What blocking GPTBot changes is narrower than most guides suggest:
- Future training. Blocking GPTBot signals that new content should not be used to train OpenAI's models [1]. A robots.txt rule only governs future crawling, so it cannot reach content GPTBot has already collected, and OpenAI does not publish how blocking affects models already trained.
- What a model knows without searching. A brand that blocks GPTBot may be less represented in a model's built-in knowledge over time. No published study measures this effect, so any figure quoted for it is a guess.
The honest position is that the trade-off is a judgment call. Allowing GPTBot gives a site's content a chance to shape how future models describe a topic or brand. Blocking GPTBot keeps that content out of training. Neither choice changes how Google ranks the site.
What is the difference between GPTBot and Google-Extended?
GPTBot is a real crawler that visits pages, while Google-Extended is only a robots.txt token with no crawler of its own. Google says Google-Extended has no separate user agent string: Google crawls with its existing user agents and reads the Google-Extended token only to decide whether content may be used to train Gemini models [7].
Google also says blocking Google-Extended does not affect inclusion or ranking in Google Search [7]. That makes Google-Extended the closest equivalent to blocking GPTBot: both are training opt-outs that leave search visibility alone.
Should you block or allow GPTBot?
Allow GPTBot if being represented in OpenAI's models matters more to you than keeping your content out of training, and block GPTBot if it does not. Either way, decide about OAI-SearchBot separately, because that is the bot that decides whether ChatGPT search can cite you [1].
A practical approach for most businesses:
- Allow OAI-SearchBot on any page you want cited in ChatGPT answers.
- Decide on GPTBot based on how you feel about training use, not on search visibility.
- Keep a deliberate list. A crawler allowlist that names each AI bot and states what it may access is easier to maintain than a wildcard rule that changes meaning every time a new bot appears.
- Check the logs after any change, allowing about 24 hours for OpenAI to read the new file [1].