What is a holdout test?
A holdout test is an experiment where a change is applied to one group and withheld from a matched group, so the gap between the two shows what the change actually did. The withheld group is the holdout (or control) group.
In SEO, a typical holdout test looks like this: take 40 similar product category pages, randomly pick 20 to receive new backlinks, and leave the other 20 alone. If the linked pages gain traffic and the holdout pages gain the same amount, the links did nothing measurable. If the linked pages pull ahead, the gap is the effect of the links.
The holdout group is what makes the result trustworthy. It absorbs everything else that happened during the test (algorithm updates, seasonality, competitors, a sitewide technical fix), because those things hit both groups equally.
Why can't a before-and-after report prove links worked?
A before-and-after report cannot prove links worked because rankings move for many reasons that have nothing to do with the links. A page that climbs three positions in the month after a link campaign may have climbed anyway.
Google says it conducts hundreds of thousands of experiments every year [3] and that it regularly improves its ranking systems [6]. Results pages also shift on their own: when Search Insider's stop word study searched the same keyword twice, seconds apart, about 1 in 10 websites on page one changed [4]. Over weeks or months, that background movement is far larger. A before-and-after report credits all of it to whatever was done last.
| Before-and-after report | Holdout test | |
|---|---|---|
| Compares | A page against its own past | Changed pages against unchanged pages, over the same period |
| Algorithm updates | Counted as part of the result | Cancel out, because both groups are hit |
| Seasonality | Counted as part of the result | Cancels out |
| Can show the change did nothing? | Rarely | Yes |
| Evidence of cause | Weak | Strong, if groups were assigned at random |
How does a holdout test work in SEO?
A holdout test in SEO works by splitting pages, not visitors, into a test group and a holdout group, because Google sees only one version of each page. In a website A/B test, each visitor is randomly shown version A or version B [2]. A search engine cannot be split that way, so SEO tests randomize at the page level instead. This page-level design is often called an SEO split test.
The test works best on many similar pages built from the same template: product pages, category pages, location pages, blog posts of one type. The groups need to be alike before the change so that any later gap can be put down to the change.
The result is read as a difference of differences: how much the test group changed, minus how much the holdout group changed over the same dates. Google's own live traffic experiments follow the same logic. A change is enabled for a small share of people, usually starting at 0.1%, and compared with a control group that did not get it [1].
How do you run a holdout test for link building?
To run a holdout test for link building, pick a set of comparable pages, randomly assign half to receive links and half to receive none, then compare how each group's traffic or rankings change over the same period. In practice:
- Choose comparable pages. Same template, same site section, similar traffic and ranking history. More pages give a more reliable answer than fewer; see sample size.
- Record a baseline. Collect several weeks of clicks, impressions and rankings for every page before anything changes, so you can confirm the groups behaved alike.
- Assign at random. Use a random draw, not judgment. Picking the "best" pages for links guarantees the linked group differs from the start.
- Freeze everything else. Avoid content rewrites, internal linking changes or new pages in either group during the test, or apply them to both groups equally.
- Build the links to the test group only. Record the date each link went live and check it is indexed. A link Google has not found cannot have an effect.
- Compare the change, not the level. Measure how much each group moved from its own baseline, and compare the two movements.
- Decide in advance what counts as a result. Set the metric, the test length and the threshold before you look. See statistical significance in SERP testing.
What can go wrong with a link building holdout test?
The main thing that goes wrong with a link building holdout test is contamination: value from the linked pages leaks into the holdout pages, shrinking the measured gap. Link value flows along internal links, so if the test and holdout pages link to each other through navigation, related-product blocks or breadcrumbs, the holdout group is partly treated too. Choosing page groups that do not link to each other heavily, or accepting that the test will understate the effect, are the two honest options.
Other common problems:
- Too few pages. With a handful of pages, one page's random swing can decide the result.
- Unequal groups. If the holdout group happens to contain the site's strongest pages, the comparison is unfair from day one. Checking the baseline period catches this.
- Sitewide effects. Links that raise the authority of the whole domain lift holdout pages too. A page-level holdout test measures the page-level effect of a link, not its whole value.
- Unindexed links. Links that are not crawled during the test period count as "built" but cannot be tested.
- Stopping early. Checking daily and stopping the moment the test group pulls ahead overstates the effect, because random swings look like wins for a while.
What is the difference between a holdout test and an A/B test?
A holdout test withholds a change from a control group, while an A/B test compares two versions against each other. In practice the terms overlap: an A/B test where version A is "no change" is a holdout test. The useful distinction in SEO is the unit being split.
| Website A/B test | SEO holdout test | |
|---|---|---|
| What is split | Visitors | Pages |
| Who sees each version | Each visitor sees one version | Everyone, including Google, sees one version per page |
| Typical metric | Conversion rate | Organic clicks, impressions, rankings |
| Typical length | Days to weeks | Weeks to months, because Google must recrawl |
| Can test links? | No | Yes |
A website A/B test needs a large sample to produce a statistically significant result [2]. An SEO holdout test faces the same constraint with far fewer units, since even a big site has hundreds or thousands of comparable pages, not millions of visitors.
How long has holdout testing been used in search?
Holdout testing has been used in search since at least 2000, when Google ran its first A/B test on its own results [2]. SEO practitioners adopted page-level holdout tests much later, because a website cannot split Google's visitors.
2000: Google's first controlled test
Google engineers ran their first A/B test in 2000 to find the best number of results to display on a page [2].
2011: more than 7,000 tests
Google ran more than 7,000 different A/B tests in 2011 [2].
2023: 16,871 live traffic experiments
Google reported 16,871 live traffic experiments and 4,781 launches in 2023, each experiment compared against a control group that did not get the change [1].
2026: SEO split tests on title tags
Search Insider's year-in-title study, which observed 18,321 ranking pages without changing any, notes that SearchPilot has run the controlled version on a single site: adding the current month and year to title tags in a split test, with a reported 5% lift in organic sessions [5]. The observational study measured ranking position across a thousand results pages, while the split test measured traffic on one site. The two answer different questions, and the controlled one is the stronger evidence of cause for that site [5].
How long should an SEO holdout test run?
An SEO holdout test should run until Google has recrawled the changed pages and the gap between the groups has had time to settle, which for link tests usually means weeks to months. No published standard sets a fixed length, and anyone quoting a universal figure is guessing.
Set the length before starting, based on how often Google crawls the pages being tested and how long the links take to be indexed. Run the full period even if the result looks clear early.
When should you use a holdout test?
Use a holdout test whenever a decision depends on knowing whether an SEO change worked, and the site has enough comparable pages to split. It is the right tool for:
- Link building budgets. Before scaling a link building program, test it on half of a page set and measure the gap.
- Template changes. Title tag formats, schema, internal linking blocks and content additions on templated pages.
- Checking published advice. A correlation between a feature and rankings is a reason to test, not a reason to roll out. Search Insider's AI Overview study found a raw citation advantage for year-titled pages that disappeared once pages of the same age were compared.
A holdout test is not practical for a single page, a small site with no repeated templates, or a change that has to go sitewide at once. In those cases, say plainly that the result is a before-and-after observation and treat it as weaker evidence.