What is deindexation?
Deindexation is when a page or a whole website is dropped from a search engine's index, so it no longer appears in search results for any query, including its own name. For example, a site that publishes thousands of scraped pages and receives a manual action for major spam problems may find that searching for its exact domain returns nothing.
Deindexation is the worst-case outcome in search. A demoted page still exists in Google's index and can be found further down. A deindexed page cannot be found through Google at all. Google's spam policies name both outcomes: sites that violate the policies "may rank lower in results or not appear in results at all" [1]. Deindexation is the second half of that sentence.
The term also covers deliberate removals. A site owner who adds noindex to a thank-you page is deindexing it on purpose [3].
What causes deindexation?
Deindexation has two families of causes: things a site owner does on purpose or by mistake, and actions Google takes against spam.
Self-inflicted causes (usually by accident):
- A noindex rule. When Googlebot finds a
noindexmeta tag orX-Robots-Tag: noindexheader, Google will "drop that page entirely from Google Search results, regardless of whether other sites link to it" [3]. A staging site's sitewide noindex pushed to production is a classic accident. - A removed page. Deleting a page so it returns 404 or 410 removes it permanently over time [4][5].
- Password protection. Limiting access blocks Googlebot from the content [4].
Penalty causes:
- A manual action. When a site has a manual action, "some or all of that site will not be shown in Google search results" [2]. The "Major spam problems" action covers sites that use "aggressive spam techniques such as scaled content abuse, cloaking, and/or other repeated" violations [2].
- Automated spam systems. Google's automated systems can keep spam out of results without a human review [6], and spam updates can cause sites to drop or disappear [7].
What is the difference between deindexation and a penalty?
Deindexation is an outcome, while a penalty is a cause. A penalty such as an unnatural links penalty can lead to deindexation of some pages, all pages, or none, with only lower rankings instead [2]. Many deindexed pages were never penalized: they carry a stray noindex tag.
| Demotion | Temporary removal | Deindexation | |
|---|---|---|---|
| Page in Google's index | Yes | Yes | No |
| Shows in results | Yes, lower | No, for about six months [5] | No |
| Typical cause | Algorithm or partial manual action | Removals tool request | noindex, 404/410, spam action |
| Reverses by itself | Sometimes | Yes, after the block expires [5] | No, the cause must be fixed |
The middle column is the one most guides miss. The Search Console Removals tool hides a URL from results for "only about six months," and it "does not prevent Google from crawling your page, only from showing it in Search results" [5]. A Removals tool request is a temporary hide, not deindexation. For permanent removal, Google points to deleting the content, returning 404 or 410, password protection or noindex [5].
Does blocking a page in robots.txt deindex it?
No. Blocking a page in robots.txt does not deindex it, and it can stop deindexation from happening. Google's noindex documentation says the page "must not be blocked by a robots.txt file" for the noindex rule to work, because Googlebot has to crawl the page to see it [3]. A URL blocked in robots.txt can still appear in search results [3].
To deindex a page, allow crawling and add noindex:
<meta name="robots" content="noindex">Or send the equivalent HTTP header:
X-Robots-Tag: noindexGoogle does not support noindex inside robots.txt [3].
How do you check if a page or site is deindexed?
To check if a page or site is deindexed, look in Google Search Console first, then confirm in search.
- Manual actions report. If there is a manual action, the report names it. A green check mark means there is none [2].
- URL Inspection in Search Console. This shows whether Google has the URL in its index and, if not, the reason it gives (for example, excluded by a
noindextag). - Removals report. Check that nobody with access submitted a temporary removal [5].
- A site: search such as
site:example.comgives a quick, rough signal. Zero results for a domain that used to show thousands is a strong sign of sitewide deindexation. A low count on its own proves little, because the site: operator is not a precise index count.
Check the self-inflicted causes before assuming a penalty. A stray noindex and a manual action look identical in search results and need opposite fixes.
How has search engine deindexation changed over time?
Search engines have deindexed spam sites since the early link-based engines, and the main change is that removal moved from blunt domain bans toward automated filtering and reported manual actions. The dated examples below show the shift.
1999 onward: link farms removed from indexes
Search engines countered early link farms by filtering their pages from indexing and results, and in some cases "entire domains were removed from the search engine indexes" [9]. See link farm.
2006: Google removes BMW's German site
In 2006, Google ousted carmaker BMW's German site for using doorway pages [8].
2014: private blog network deindexing
Google targeted and penalized private blog network users on several occasions starting in 2014 [8]. Search Insider does not support or suggest the use of private blog networks.
Today: manual actions with a route back
Google now tells site owners when it takes manual action and lets them submit content for reconsideration once the problem is fixed [6]. Google says it processes all of the reconsideration requests it receives [6].
How do you get a deindexed site back into Google?
To get a deindexed site back into Google, remove the cause, then let Google recrawl or request review. The route depends on the cause:
- Accidental noindex: remove the tag or header, make sure robots.txt allows crawling [3], and request indexing with URL Inspection in Search Console.
- Manual action: fix every issue the report lists, then submit a reconsideration request. Most reviews take several days or weeks [2].
- Automated spam systems: there is no request to submit. Google says changes may help "if our automated systems learn over a period of months that the site complies with our spam policies" [7].
Google publishes no figure for how often deindexed sites return, or how long a full recovery takes after a spam-related removal. Any timeline beyond Google's "days or weeks" for reviews [2] and "a period of months" for automated systems [7] is an estimate.
What should you do to avoid deindexation?
Avoid deindexation by checking your own settings and by staying inside Google's spam policies. Search Insider's practical list:
- Audit noindex after every launch or migration. A single template or plugin setting can noindex a whole site.
- Don't confuse robots.txt with noindex. Use
noindexto remove pages; use robots.txt to manage crawling [3]. - Watch the Manual actions report. A manual action is the clearest early warning [2].
- Don't build on spam. Link farms, scaled content and similar tactics are what spam policies say can keep a site out of results entirely [1][2].