What is sample size?
Sample size is the number of separate observations a study measures. In SEO research those observations are usually searches, results pages, ranking URLs or AI prompts. A study that pulls Google's top 20 results for 1,000 keywords has a sample of 1,000 results pages, and a sample of roughly 20,000 ranking URLs inside them.
Sample size is written as n. Search Insider's study of years in title tags, for example, reports n = 18,321 ranking URLs drawn from 1,000 Google results pages [1].
Sample size matters because every study measures a slice of something larger. Nobody can collect every search Google answers, so a study measures a sample and treats it as representative. The bigger and better chosen the sample, the closer the study's percentages sit to the true figure.
Why does sample size matter in SEO studies?
Sample size matters in SEO studies because it decides whether a reported difference is real or just noise. Google results move between two identical searches, so a small study can find a "pattern" that is simply chance.
Search Insider measured that noise directly in its stop words study. The same keyword searched twice, seconds apart, kept about 9 in 10 of the same websites on page one, not all 10 [3]. Any change smaller than that background movement cannot be told apart from Google's ordinary variation without a large sample. Google's own documentation makes the same point about individual checks: results are specific to the time, place, device and recent history of the person searching, so one search is a single observation, not a finding [4].
A large sample does two jobs:
- It tightens the estimate. A percentage measured across 1,000 searches has a far narrower range of plausible true values than the same percentage measured across 50.
- It lets a study detect small effects. Many SEO effects are small. A current year in a title tag was associated with about 0.62 ranking positions [1]. An effect that size is invisible in a study of a few dozen keywords.
How do you calculate the sample size you need?
You calculate the sample size you need from the margin of error you can accept. For a simple percentage, such as "what share of pages carry a year in the title", the 95% margin of error is 1.96 times the square root of p(1 − p)/n, which is widest when p is 50% and then comes to about 0.98 divided by the square root of n [5]. Turned around, the sample needed for a margin of error E is n = 1.96² × p(1 − p) / E², which is why about 1,000 observations give roughly ±3 points and about 10,000 give ±1 point [6]:
| Sample size (n) | Margin of error (95%, worst case) |
|---|---|
| 100 | about ±9.8 percentage points |
| 400 | about ±4.9 points |
| 1,000 | about ±3.1 points |
| 10,000 | about ±1.0 point |
Two things follow from that arithmetic:
- Precision gets expensive. Quadrupling the sample only halves the margin of error. Going from 1,000 to 10,000 searches buys about two points of precision.
- Subgroups need their own n. A study of 1,000 queries split across 20 industries has about 50 queries per industry. A finding about one industry carries the margin of error of 50, not of 1,000.
Comparing two groups needs more than measuring one. To detect a small gap between two percentages, each group needs enough observations on its own, and the smaller the gap, the larger the sample required.
Is 18,321 URLs a sample of 18,321?
No. When ranking URLs come from the same results pages, the honest sample size sits between the number of pages and the number of searches. Twenty URLs from one results page are not 20 independent observations: they share the same query, intent, competition and date.
Search Insider's year-in-title study collected 18,321 URLs, but from 1,000 results pages [1]. The study compared pages competing on the same results page for exactly this reason, so each comparison was made between pages facing identical conditions. A study that reports "18,321 pages" and treats each as independent would overstate how certain its findings are.
This is the distinction most SEO studies skip. When reading a headline figure, ask which unit the study counted:
| Unit | What it tells you |
|---|---|
| Queries or results pages | How many separate search situations were tested |
| Ranking URLs | How many pages were measured inside those situations |
| Domains | How many different websites the findings rest on |
| Citations or clicks | How many events were counted, often many per query |
A study with 1 million URLs from 500 keywords tells you a lot about those 500 keywords and much less about search in general.
What is the difference between sample size and effective sample size?
Sample size is how much data a study collected, while effective sample size is how much of it actually answers the question. The two often differ, and the effective figure is the one that sets precision.
Search Insider's studies show both ways the number shrinks:
- Data that cannot be used. The stop words study ran 610 live searches, built as 305 keyword pairs. In 89 pairs the data provider returned a US result for one half and a mis-localized result for the other, so the published percentages rest on 216 pairs [3].
- Tighter comparisons. The AI Overview study started with 17,240 ranking pages across 1,000 searches [2]. When it compared only pages Google reported as genuinely the same age, the comparison rested on 3,793 pages, less than half the 7,797 pages behind the study's earlier comparisons [2].
The second case is the normal price of a fair comparison. Each control removes a difference that has nothing to do with the question, and each one also removes data. A study that reports only its headline n, and not the n behind its key comparison, is hiding the number that matters.
How do you judge whether an SEO study's sample is big enough?
You judge an SEO study's sample by finding the unit it counted, the n behind each claim, and how the sample was chosen. A large n is not enough on its own.
Check these points before trusting a figure:
- What was counted? Queries, pages, domains or events. The number of independent search situations is the floor of the real sample.
- What is n for this specific claim? Industry, device or intent breakdowns usually rest on a fraction of the total.
- How were the queries chosen? A frame fixed before collection, spread across industries and intents, beats a list of keywords one tool already tracks. Search Insider's AI Overview study used 1,000 queries across 20 industries, 50 per industry, stratified by intent and competition [2].
- Is there a baseline? A test that compares two versions of a search needs to know how much results move with no change at all. Without that baseline, normal fluctuation looks like an effect [3].
- Is a range reported? A finding with a range, such as 1.05 times with a range of 0.81 to 1.36, tells you whether "no difference" is still possible [2]. A bare percentage does not.
What sample size do you need for AI visibility tracking?
AI visibility tracking needs a larger sample than most teams expect, because AI answers vary between runs of the same prompt and many prompts produce no useful answer at all. The unit is the prompt, run on a fixed engine and date range, and not the number of citations collected.
The same rules apply as in organic studies. In Search Insider's AI Overview study, an AI Overview appeared on 947 of the 1,000 searches, so the citation analysis rested on fewer searches than the query frame [2]. For tracking a brand's share of AI citations over time, see citation share of voice, which covers how many prompts a reliable measurement needs.
What are the limits of a large sample size?
A large sample size reduces random error but does nothing about bias. If the keyword list leans toward one industry, or the data was collected on one day during a Google update, a bigger sample just measures the same skew more precisely.
Three limits matter most in SEO research:
- Correlation is not causation. A large sample can show that two things move together very precisely without showing that one causes the other. Search Insider found a year-in-title advantage for AI Overview citations that disappeared once pages of the same age were compared [2]. See correlation vs causation in SEO.
- Results expire. A sample describes Google on the date it was collected. A study's n says nothing about whether the finding still holds after the next core update.
- Location and device are part of the sample. A US desktop study describes US desktop search. Search Insider's recent studies say so in their method sections [1][3].
When should you worry about sample size?
You should worry about sample size whenever a decision rests on a percentage or a difference between two groups. That covers ranking studies, A/B tests on title tags, AI visibility reports and any claim that one tactic "works".
For your own testing, two practical rules cover most cases. First, decide the unit and the n before collecting anything, and write it down, so the sample cannot be trimmed later until a result appears. Second, run a no-change baseline alongside the test, so you know how much your results move on their own. If the effect you find is smaller than that baseline, you have not found an effect.