Holdout test

Incrementality is the question. A holdout test is how you answer it.

You split a comparable population, advertise to one part and deliberately not to the other, then compare what happened. The gap is the effect. Everything else in measurement is inference; this is the only approach that produces evidence.

The three ways to split. By geography — some regions get the campaign, matched regions do not. By audience — a random slice is excluded from targeting. By time — the campaign goes off, then on, with enough of each to separate the effect from normal variation. Geographic tests are the easiest to explain and the hardest to run cleanly, because regions differ in ways that have nothing to do with your advertising.

What makes one useless. A holdout that is not comparable to the exposed group measures the difference between the groups, not the effect of the ads. A holdout that leaks — the excluded audience sees the campaign anyway, through another channel, another device, or someone else in the household — shrinks the measured effect toward zero. And a test stopped early because the numbers looked bad is not a test.

Who runs it matters. Several ad platforms offer built-in lift studies, which are convenient and are designed, executed and reported by the party selling the advertising. That does not make them wrong. It does mean an independent holdout and a platform lift study are different kinds of evidence, and the difference is worth saying out loud when the two disagree.

The uncomfortable part is structural: a holdout costs you whatever the excluded group would have bought through the ads. That cost is the price of knowing.