What is incrementality testing?
Incrementality testing measures the extra result an action caused, beyond what would have happened without it. The extra amount is called incremental lift.
Take a link building campaign. Organic traffic to the linked pages rises 20% over three months. A standard report credits the campaign with that 20%. An incrementality test asks a harder question: how much did comparable pages that received no links grow over the same three months? If they grew 15%, the incremental lift from the links is about 5 points, not 20. If they also grew 20%, the incremental lift is zero.
The comparison group is what makes it an incrementality test. Without one, there is no estimate of what would have happened anyway, and every change in the numbers gets credited to the last thing someone did.
How does incrementality testing work?
Incrementality testing works by withholding an action from a comparable group and measuring the gap between that group and the group that received the action. The withheld group stands in for "what would have happened anyway," which can never be observed directly for the same page at the same time.
The basic formula:
incremental lift = (change in test group) - (change in holdout group)The groups must be assigned at random, or at least matched closely, before the action starts. Randomized experiments are the standard way to do this, and they rely on statistical testing to judge whether a difference between groups is real [1]. Google measures its own Search changes this way: a change is enabled for a small share of people, usually starting at 0.1%, and compared with a control group that did not get it [2].
What is the difference between incrementality and attribution?
Attribution assigns credit for results that already happened, while incrementality measures how many of those results the action caused. The two often disagree.
| Attribution | Incrementality | |
|---|---|---|
| Question | Which channel or page touched this conversion? | Would this conversion have happened without the action? |
| Method | Rules or models applied to tracking data | A test group compared with a holdout group |
| Needs a control group? | No | Yes |
| Typical error | Credits actions for results that would have happened anyway | Understates effects that spill into the holdout group |
| Good for | Day-to-day reporting | Decisions about whether to keep spending |
A page can be "attributed" thousands of conversions and have an incremental effect near zero, if those visitors would have found the business through another route. Attribution reports cannot tell the two cases apart, because they have no comparison group.
Why is incrementality testing rare in SEO?
Incrementality testing is rare in SEO because organic search cannot be switched off for a random half of visitors the way an ad campaign can. Google decides who sees which result, and every searcher sees the same version of a given page. SEO teams therefore cannot run the visitor-level tests common in website optimization, where each visitor is randomly shown one version [1].
The workaround is to split pages instead of people. A holdout test applies the change to a random set of comparable pages and leaves the rest alone. That requires a site with many similar pages, time for Google to recrawl, and a willingness to leave some pages untreated, which is why most SEO reporting stays at before-and-after. No published figure exists for how many SEO teams test incrementality, so claims that "few" or "most" do are impressions, not data.
Running one is a credibility signal for exactly that reason. A report that shows a holdout group, and a lift measured against it, is making a claim a before-and-after chart cannot.
How do you measure the incremental lift of SEO work?
To measure the incremental lift of SEO work, compare how much the changed pages moved with how much a matched set of unchanged pages moved over the same dates. The steps:
- Define the action and the metric. For example, "new backlinks" and "organic clicks from Google Search Console."
- Find comparable pages. Same template, similar traffic, similar history.
- Split them at random into a test group and a holdout group.
- Record a baseline period for both groups before anything changes.
- Apply the action to the test group only and keep everything else equal.
- Calculate lift as the test group's change minus the holdout group's change.
- Check the lift against normal noise. Rankings and traffic move on their own. Even the same keyword searched twice, seconds apart, changed about 1 in 10 websites on page one in Search Insider's stop word study [6]. See statistical significance in SERP testing.
If no holdout group is possible, the next best option is matching: comparing pages that share the factors most likely to explain the result. That gives weaker evidence than a randomized test, but far stronger evidence than a raw comparison.
What happens when you skip the holdout?
When you skip the holdout, the measured effect usually includes things the action did not cause. Search Insider's own studies show how large that gap can be, even without any intervention at all.
In the AI Overview study, pages with the current year in the title were 1.91 times more likely to be quoted than undated pages on the same search. Comparing only pages of the same age brought that to 1.05 times, with a range of 0.81 to 1.36, which is consistent with no difference [4]. The raw figure was almost entirely page age, not the year.
In the ranking study, the current year was worth about 0.62 positions within the same results page, and in results Google showed a date for, roughly a third of the advantage was explained by pages simply being newer [3]. Google runs freshness systems that favor newer content where it is expected [5], so freshness is a predictable source of lift that has nothing to do with the title.
Both studies are observational, and say so [3] [4]. They are a reminder of what a test without a holdout is measuring: the action plus everything that travels with it.
Where does incrementality testing come from?
Incrementality testing comes from direct-response advertising and controlled experiments, long before search existed. Search engines adopted it at scale from 2000, and SEO has adopted it far more slowly.
1923: coupons in Scientific Advertising
Claude Hopkins described using promotional coupons to measure advertising campaigns in his 1923 book Scientific Advertising, an early form of testing what an ad actually produced, though without modern statistical methods [1].
2000: Google's first A/B test
Google engineers ran their first A/B test in 2000, to find the best number of results to show on a results page [1].
2012: a Bing headline test
A 2012 Microsoft Bing experiment on ad headlines produced a 12% revenue increase with no impact on user-experience metrics [1].
2023: control groups behind every launch
Google reported 16,871 live traffic experiments and 4,781 launches in 2023, each experiment measured against a control group [2].
What is the difference between incrementality testing and a holdout test?
Incrementality is the quantity being measured, while a holdout test is the most common method for measuring it. A holdout test produces an estimate of incremental lift; incrementality testing is the broader practice of insisting every result is measured against what would have happened without the action. Other methods, such as matched comparisons, can estimate incrementality when a true holdout is not possible, with weaker evidence.
When should you run an incrementality test?
Run an incrementality test before scaling any SEO spend whose value you would otherwise take on trust. The clearest cases:
- Link building programs. Test on half of a comparable page set before buying links for the whole site.
- Sitewide template changes. Title formats, content blocks and internal link modules can be rolled out to half the pages first.
- Tactics borrowed from studies. A correlation between a feature and rankings is a reason to test the feature, not to assume it will work on your site.
- Overlap with paid search. Where organic and paid results compete for the same clicks, only a test with a comparison group shows how much of either is truly additional.
Skip it when the change is required anyway (a broken page, a legal fix), when the site has too few comparable pages to split, or when the decision would not change whatever the result. Say plainly in reporting when a figure is a before-and-after comparison rather than an incremental one.