What is crawl budget?
Crawl budget is the set of URLs on a site that Google can and wants to crawl [1]. Google sets crawl budget from two things: the crawl capacity limit (how many connections Googlebot can use without overloading the server) and crawl demand (how much Google wants the content, based on its size, freshness, quality and popularity) [1].
A simple example: an online store has 40,000 product pages, but its filter menu creates 3 million combinations of color, size and price URLs. Googlebot can only fetch so many URLs a day from that server. If most of those fetches go to filter URLs, new products wait days or weeks to be crawled. That is a crawl budget problem.
Does crawl budget matter for my site?
Crawl budget matters only for large or fast-changing sites. Google says its crawl budget guide is for sites with more than 1 million unique pages whose content changes about weekly, sites with more than 10,000 unique pages whose content changes daily, and sites with a large share of URLs stuck as "Discovered, currently not indexed" in Search Console [1].
Google is blunt about everyone else: if a site does not have a large number of pages that change rapidly, or its pages seem to be crawled the same day they are published, the site owner does not need to read the guide [1]. For a typical small business site or blog, crawl budget is not the reason a page is missing from Google.
The exception is a small site that generates a huge number of URLs by accident. A 500-page site with faceted navigation or calendar pages can expose millions of crawlable URLs, and that site has a crawl budget problem whatever its "real" size.
How does Google decide crawl budget?
Google decides crawl budget by combining crawl capacity (what the server can take) and crawl demand (what Google wants to fetch) [1].
- Crawl capacity limit. Google calculates how many parallel connections Googlebot can use and how long to wait between fetches. If the site responds quickly and consistently, including stable latency and time to first byte, the limit goes up. If the server slows down or returns errors, the limit goes down [1]. Google's crawlers also temporarily slow down when they receive 5xx or 429 responses [10].
- Crawl demand. Google weighs the site's perceived inventory (how many URLs it knows about, including duplicates), popularity, and staleness (how often content needs refreshing) [1]. Duplicate and low-value URLs inflate the inventory and waste crawl time [1].
Two consequences follow. Adding server resources only helps if capacity is the bottleneck. If demand is the bottleneck, the fix is making the content more worth crawling: popular, unique and useful [1].
What wastes crawl budget?
Crawl budget is wasted by any URL Googlebot fetches that should never be indexed. Google lists these as the main causes [1]:
| Waste | Why it costs crawl | Google's fix |
|---|---|---|
| Duplicate URLs | Each duplicate is fetched separately | Consolidate duplicates; see canonicalization |
| Faceted navigation | Filter combinations create near-infinite URL spaces | robots.txt, URL fragments, nofollow on filter links, 404 for empty combinations [7] |
| Soft 404s | They return 200, so Google keeps crawling them | Return a real 404 or 410 |
| noindex pages | Google still requests the page, then drops it | Block crawling in robots.txt instead, if the page never needs indexing |
| Long redirect chains | Each hop is an extra request | Point links straight at the final URL |
| Slow responses | Lower crawl capacity | Faster servers and pages; support 304 Not Modified |
The noindex point surprises many site owners. Google says not to use noindex to save crawl budget, because Googlebot still requests the page and only drops it after seeing the noindex tag or header [1]. Robots.txt stops the fetch; noindex does not.
What are the myths about crawl budget?
Several widely repeated crawl budget tips are wrong according to Google's own myths page [2]:
- "Crawling more helps rankings." Google says improving crawl rate will not necessarily lead to better positions in Search [2].
- "Small sites get crawled less." Google says that if a site has important content that changes often, Google crawls it often, regardless of size [2].
- "Zipping sitemaps saves crawl budget." Compressed sitemaps still have to be fetched, so little crawl time is saved [2].
- "Crawl-delay in robots.txt controls Googlebot." Google's crawlers do not process the non-standard crawl-delay rule [2].
- "Only HTML pages count." Any URL Googlebot crawls counts, including alternate URLs such as hreflang versions and embedded resources such as CSS and JavaScript [2].
- "Tweaking pages keeps them fresh." Google says content is rated by quality, regardless of age [2].
A myth Google's page does not cover directly but is common: that crawl budget explains every indexing problem. A page that was crawled and still not indexed ("Crawled, currently not indexed" in Search Console) has a quality or duplication problem, not a crawl budget problem [8].
How has Google's guidance on crawl budget changed?
Google first explained crawl budget publicly in January 2017, and has since moved the topic into a standing guide for large sites, added reporting, and removed manual controls [3][1]. The dated changes below come from Google's own blog and documentation.
2017: Google defines the term
In January 2017, Google published "What Crawl Budget Means for Googlebot" on its Search Central Blog [3], its first dedicated explanation of a term SEOs had used informally for years.
2020: the Crawl Stats report
In November 2020, Google announced new and improved crawl stats in Search Console [4], giving site owners a direct view of how many requests Googlebot makes and what responses it gets.
2023: the crawl rate limiter is retired
In November 2023, Google announced the deprecation of the Crawl Rate Limiter tool in Search Console [5]. Today, Google's documented way to slow Googlebot urgently is to return 500, 503 or 429 status codes for a short period, and it warns against doing so for longer than one to two days because it may hurt how the site appears in Google products [6].
2026: the current guide
Google's crawl budget guide was last updated on July 22, 2026, and still defines crawl budget by capacity and demand [1].
How do you check your crawl budget?
To check crawl budget, compare how many URLs Googlebot fetches with how many URLs you actually want indexed. Three places show it:
- Search Console Crawl Stats report. Shows total crawl requests, average response time, and host status over time [4][1].
- Search Console Page indexing report. A large or growing count of "Discovered, currently not indexed" URLs means Google found URLs but has not crawled them yet; Google says this typically happens when crawling was expected to overload the site [8].
- Server log files. Log file analysis is the only way to see exactly which URLs Googlebot requested, how often, and what status codes it got. Logs show waste that reports summarize away, such as thousands of hits on parameter URLs.
Google has never published a number for any site's crawl budget, and there is no universal "good" crawl rate. The useful measure is the ratio: of all Googlebot requests, what share went to URLs you want indexed?
How do you improve crawl budget?
Improve crawl budget by removing URLs Googlebot should not fetch and making the rest fast to fetch [1]:
- Consolidate duplicate content to one canonical URL.
- Block crawling of unimportant URL spaces, such as filter combinations, in robots.txt [1][7].
- Return 404 or 410 for permanently removed pages, and fix soft 404s [1].
- Keep XML sitemaps current, with accurate
lastmoddates [1]. A sitemap helps discovery but does not guarantee crawling [9]. - Avoid long redirect chains and support 304 Not Modified responses [1].
- Link to important pages internally. Googlebot finds most URLs by following links, so an orphan page depends on the sitemap alone.
AI crawlers add a new load on the same server. Bots such as GPTBot and ClaudeBot do not share Google's crawl budget, but they compete for the same server capacity, and a server that slows down under their load can lower Google's crawl capacity limit too [1].
When should you worry about crawl budget?
Worry about crawl budget when a site is large, changes fast, or generates URLs automatically, such as ecommerce filters, programmatic SEO pages, or internal search results. Signs to act on: new pages take days to appear in Google, "Discovered, currently not indexed" keeps growing, or logs show Googlebot spending most of its requests on URLs you do not want indexed. If none of these apply, spend the time on content and links instead.