What is CCBot?
CCBot is the crawler that collects web pages for Common Crawl. It identifies itself with the user agent name CCBot, and site owners can address it in robots.txt by that name [1].
Common Crawl is a nonprofit web archive. It has "collected and stored hundreds of billions of web pages since 2008" and publishes the results as snapshots anyone can download for free [1]. Researchers use them to study the web. AI companies use them, far more heavily, to train language models.
So when people ask whether CCBot is an "AI crawler," the honest answer is: not directly. CCBot belongs to a nonprofit archive, not an AI company. But what it collects becomes the raw material for many AI training datasets, which is why it sits on most lists of AI crawlers to block.
What is Common Crawl used for?
Common Crawl is used mainly as training data for large language models, either directly or through cleaned datasets built from it. Its snapshots are the starting point for widely used datasets including C4, RefinedWeb and Dolma, each built from a different set of monthly snapshots [1].
Published training recipes show how central it is:
| Model or dataset | What came from Common Crawl |
|---|---|
| GPT-3 (OpenAI, 2020) | Filtered Common Crawl, 410 billion tokens, 60% of the training mix, from 41 monthly shards covering 2016 to 2019 [2] |
| LLaMA (Meta, 2023) | 67% English CommonCrawl (five dumps, 2017 to 2020) plus 15% C4 [3] |
| C4 (Google, 2019) | The "Colossal Clean Crawled Corpus," built from Common Crawl to train T5 [4] |
| FineWeb (Hugging Face, 2024) | 15 trillion tokens from 96 Common Crawl snapshots [6] |
The LLaMA figure understates the dependence. C4 is itself made from Common Crawl [4], so 82% of LLaMA's pretraining mix traces back to CCBot's crawling: 67% directly and 15% through C4 [3]. Most published summaries quote only the 67%.
This is the sense in which Common Crawl is "upstream" of AI. A page CCBot fetched years ago can sit in several derived datasets and the models trained on them, without any AI company crawling the page itself.
How do you block CCBot?
To block CCBot, add a rule for its user agent to your robots.txt file. Common Crawl's crawler obeys the CCBot user agent in robots.txt [1]:
User-agent: CCBot
Disallow: /To block only part of a site, disallow those folders instead:
User-agent: CCBot
Disallow: /members/
Disallow: /research/robots.txt is a request, not a lock. Compliance "is completely voluntary and not centrally monitored," so a crawler that ignores it, or a scraper pretending to be another bot, is not stopped by it [8]. For a hard block, deny the requests at the server or firewall as well.
What does blocking CCBot actually change?
Blocking CCBot stops Common Crawl from collecting your pages in future crawls. It does not reach backward. robots.txt governs crawling, so pages already captured in published snapshots stay in those snapshots, and datasets and models already built from them are unchanged.
What blocking CCBot does not do:
- It does not affect Google Search. CCBot has nothing to do with Googlebot, rankings or AI Overviews.
- It does not opt you out of Gemini training. Google uses its own
Google-Extendedtoken for that, which Google says "does not impact a site's inclusion in Google Search" [7]. - It does not block other AI crawlers. OpenAI, Anthropic, Perplexity and others use their own user agents, which need their own rules.
How many websites block CCBot?
Restrictions on AI crawlers, CCBot included, rose sharply between 2023 and 2024, but CCBot is blocked less often than OpenAI's crawler. A large audit of the sites behind AI training datasets found Common Crawl's agents restricted for 13.3% of tokens in the most actively maintained part of C4, against 25.9% for OpenAI and 9.8% for Google's AI crawler [1].
Sites that block AI crawlers tend to include CCBot in the list: in 83.4% of cases where a site disallowed any of the AI organizations studied, it also disallowed Common Crawl [1]. The authors note the side effect: blocking a nonprofit archive also restricts data used for non-commercial AI and academic research [1].
What is the difference between CCBot and GPTBot?
CCBot collects pages for a public nonprofit archive that anyone can download, while GPTBot collects pages for OpenAI's own use. The practical difference is who gets your content once it is crawled.
| CCBot | GPTBot | |
|---|---|---|
| Operator | Common Crawl Foundation (nonprofit) | OpenAI |
| Where the data goes | Public snapshots, free to download | OpenAI only |
| Who trains on it | Any company or researcher | OpenAI |
| Blocking it stops | Future Common Crawl snapshots | Future OpenAI collection by that bot |
| Restricted in audited C4 sources | 13.3% of tokens [1] | 25.9% of tokens (OpenAI) [1] |
Because Common Crawl data is public, blocking GPTBot alone does not keep a page out of every dataset OpenAI or anyone else could use. Blocking CCBot alone does not stop AI companies crawling directly.
When was Common Crawl's role in AI training established?
Common Crawl became a standard AI training source between 2019 and 2020, when Google built C4 from it and OpenAI made it the majority of GPT-3's training mix [2][4]. It has collected pages since 2008, long before that use [1].
2008: collecting begins
Common Crawl began collecting and storing web pages in 2008 [1].
2019: C4
Google researchers released the Colossal Clean Crawled Corpus, a cleaned subset of Common Crawl, to train the T5 model [4].
2020: GPT-3
OpenAI's GPT-3 paper listed filtered Common Crawl as 60% of the training mix [2].
2021: what is inside C4
An audit of C4 found text from unexpected sources such as patents and U.S. military websites, machine-generated text, and filtering that "disproportionately removes text from and about minority individuals" [5]. It was an early sign that what Common Crawl holds, and how it is filtered, shapes what models learn.
2023: LLaMA
Meta's LLaMA drew 82% of its pretraining mix from Common Crawl, directly and through C4 [3].
2023 to 2024: blocking spreads
The share of AI training sources restricting crawlers, CCBot included, rose rapidly within a single year [1].
Should you block CCBot?
Block CCBot if you do not want your content in a public archive that any AI company can train on, and accept that this applies to future crawls only. Leave it open if you want your pages in the open web record that researchers and many models learn from.
There is no published evidence either way on whether blocking CCBot changes how often AI assistants mention a brand. Training data shapes what a model knows without searching, but no study has isolated Common Crawl inclusion as a cause of AI visibility. Answer engines that cite sources rely on live search, which CCBot does not touch. For most businesses, the decision is about how their content is reused, not about rankings or AI citations.