What is index bloat?
Index bloat is when a search engine has indexed, or keeps crawling, far more URLs from a site than the site has pages worth finding. The extra URLs are usually near-duplicates or empty shells generated by the site's own software.
A typical example: an online store sells 2,000 products across 40 categories, but Google's Page indexing report shows 30,000 indexed URLs. The difference is made of filter combinations (?color=red&size=9&sort=price), internal search result pages, tag archives with one post each, printer versions and tracking parameters. None of those URLs was written for a searcher. All of them compete for Google's crawling and for a place in the index.
Index bloat is an SEO industry term. Google's documentation does not use it. Google talks about the same problem as duplicate content, low-value URLs and what its crawl budget guide calls a site's "perceived inventory" [2].
What causes index bloat?
Index bloat is caused by site features that create new URLs without creating new content. The common sources:
| Cause | Example URL | Why it bloats |
|---|---|---|
| Faceted navigation | /shoes?color=red&size=9 | Every filter combination is a new URL [5] |
| Tracking and session parameters | /shoes?gclid=ABCD | Same page, many addresses [4] |
| Internal search results | /search?q=red+shoes | Unlimited URLs, created by visitors and bots |
| Tag and archive pages | /tag/running/ with one post | Thin lists that repeat other pages |
| Paginated and sorted lists | /blog/page/47?sort=old | Many near-identical list pages |
| Thin generated pages | /plumbers/smalltown-ks with no real listings | Pages produced from a template with no useful data |
| Error pages that return 200 | A "no results" page with a 200 status | Soft 404s Google may keep crawling [2] |
Faceted navigation is the biggest single source on ecommerce sites. Google says crawlers "will typically access a very large number of faceted navigation URLs before the crawlers' processes determine the URLs are in fact useless" [5].
Does index bloat hurt rankings?
Index bloat does not trigger a penalty, and Google has never published a ratio of indexed pages to useful pages that causes harm. The damage it does is indirect, and it falls in three places:
- Crawling. On large sites, time Googlebot spends on junk URLs is time not spent on new or updated pages. Google says removing duplicate and unwanted URLs is the part of crawl demand a site owner can most control [2]. For small sites this rarely matters: Google says sites whose pages are crawled the same day they are published do not need its crawl budget guide [2].
- Consolidation. When several URLs show the same content, Google has to pick one canonical and consolidate signals to it [4]. If it picks a parameter URL, the wrong version can rank.
- Quality. Thousands of empty pages are not neutral if they exist to catch search traffic. Google's own self-assessment asks, "Are you producing lots of content on many different topics in hopes that some of it might perform well in search results?" [8]. At the extreme, many pages generated "for the primary purpose of manipulating search rankings and not helping users" is scaled content abuse under Google's spam policies [7].
What is not known: whether a large count of low-value indexed pages lowers rankings for a site's good pages. Google has not said so, and the claim is often repeated without evidence.
What is the difference between index bloat and crawl budget?
Index bloat is the problem of too many low-value URLs, while crawl budget is the limited amount of crawling Google gives a site. Index bloat is one of the main ways a site wastes crawl budget [2].
A second distinction most guides miss: there are two kinds of bloat, and they need different fixes.
- Indexed bloat is junk that Google has put in its index. It shows up as indexed URLs you never wanted and can appear in search results.
- Crawl bloat is junk Google keeps fetching but mostly declines to index. It shows up in server logs and as large counts of "Crawled - currently not indexed" in Search Console, meaning Google fetched the page and left it out [1]. A large "Discovered - currently not indexed" count, meaning Google knows the URL but has not crawled it yet, is a related warning sign on big sites [1][2].
Many sites diagnose crawl bloat as indexed bloat and add noindex tags to pages Google was already ignoring. That does nothing for crawling, because Google still has to fetch a page to see its noindex tag [2].
How do you check for index bloat?
To check for index bloat, compare the number of indexed pages in Search Console's Page indexing report with the number of pages you actually want indexed, usually the count of URLs in your XML sitemaps. A large gap, in either direction, is the signal to investigate.
Then look at examples:
- Open the indexed URL list in the Page indexing report and scan for parameters, search pages, tag pages and other patterns [1].
- Read the "not indexed" reasons. Large counts of "Duplicate without user-selected canonical" point to duplicates. Large counts of "Crawled - currently not indexed" point to pages Google has judged not worth indexing [1].
- Check server logs to see which URL patterns Googlebot requests most. This shows crawl bloat that Search Console does not.
Do not use a site: search to count indexed pages. Google says the operator "doesn't necessarily return all the URLs that are indexed" and that bigger sites "shouldn't expect to see all their URLs in the results" [6].
Also keep the right target in mind. Google says "Don't expect every URL on your site to be indexed" and that "Non-indexed URLs can be fine" [1]. A site with some non-indexed URLs is normal. A site whose indexed count is several times its real page count is not.
How do you fix index bloat?
To fix index bloat, match the fix to the cause: consolidate duplicates, noindex pages people need but searchers do not, block crawling of URL patterns that should never be fetched, and remove or improve thin pages.
| Situation | Fix |
|---|---|
| Duplicate versions of one page | Redirect, or a rel="canonical" to the main version [4] |
| Pages visitors need but search does not (internal search, account pages) | noindex meta tag or X-Robots-Tag header [3] |
| Filter and parameter URLs that should never be crawled | robots.txt disallow, or move filters into URL fragments [2][5] |
| Pages removed for good | Return 404 or 410 [2] |
| Thin template pages with no real data | Merge into a stronger page, add real content, or delete |
Two ordering mistakes cause most failed cleanups:
- Blocking before deindexing. For noindex to work, the page "must not be blocked by a robots.txt file" [3]. If you disallow a URL in robots.txt first, Google can no longer see its noindex tag, and an already indexed URL can stay indexed. Noindex first, wait until the URLs drop out, then block crawling if you need to.
- Using noindex to save crawling. Google says not to, because it "will still request, but then drop the page" [2]. Noindex controls the index. Robots.txt controls crawling.
Expect cleanup to take time. Google notes that pages may take months to be recrawled [3].
How has Google's treatment of low-value pages changed?
Google's guidance has moved from treating mass low-value URLs as a crawling nuisance to treating mass-produced low-value pages as possible spam. The documentation dates below mark the turning points.
March 5, 2024: scaled content abuse becomes a spam policy
Google added three spam policies on March 5, 2024, including scaled content abuse [9]. The policy covers creating large amounts of unoriginal content that provides little value "no matter how it's created" [7]. Bloat produced by a filter menu is a technical problem. Bloat produced on purpose, as thousands of template pages aimed at search queries, can now fall under this policy.
December 2025: faceted navigation guidance updated
Google's faceted navigation guidance, last updated in December 2025, recommends preventing crawling of filter URLs with robots.txt or URL fragments, and calls rel="canonical" and rel="nofollow" "generally less effective in the long term" [5].
How do you prevent index bloat when scaling a site?
To prevent index bloat when scaling a site, decide before launch which URL types should be indexed, and make every other URL type noindexed or uncrawlable by default. This matters most for programmatic SEO builds, glossaries, directories and location pages, where one template can create thousands of URLs overnight.
A short pre-launch check:
- Every indexable page has something no other page on the site has. If two pages would show the same list or the same text with a city name swapped, they should be one page.
- Empty states are not pages. A filter or location with no results should return a 404 or carry noindex, not a 200 "no results" page.
- Internal search, sort orders and session parameters are excluded from indexing and, where possible, from crawling.
- The XML sitemap lists only canonical, indexable URLs, so the gap between sitemap count and indexed count stays easy to read.
- New sections are linked from the site, not just listed in the sitemap, so their click depth stays low.
Then watch the Page indexing report for the first few weeks after launch. Bloat is cheap to stop at the template and slow to undo once Google has indexed it.