Ads that do nothing get a lift result half the time

A lift study feels like the end of the argument: a randomised holdout, run by the platform, with a certainty score next to the result. Google Ads reports Conversion Lift results from 50% certainty and calls 50% to 70% “a moderate chance that your results were caused by your ads”. Its own help page also says how the number is made: certainty of lift is 1 minus the p-value.

A certainty of 50% means a p-value of 0.5. In plain terms, the exposed group converted at any rate above the holdout, by any margin. When the ads do nothing at all, that happens by chance in half of all studies. We simulated 2.4 million lift studies. Of the ones where the ads had no effect, 50.7% came back with a lift result, and 9.7% came back at 90% certainty.

The test

A script simulated user-based lift studies with the volume a small Google Ads account brings: 100,000 users, a 1% conversion rate, so about 1,000 conversions per study. That is the same number as Google’s account threshold for Conversion Lift, 1,000 observed conversions. For each design it ran 200,000 studies, scored each the way Google’s help page describes (1 minus the one-sided p-value, rounded down to 5%, below 50% reported as “no lift”), and counted the outcomes.

The ads either did nothing or lifted conversions by 5%, 10% or 20%. With a 10% holdout:

True lift	90%+ certainty	50–85% (“directional”)	“No lift”	Median lift reported at 90%+
0% (ads do nothing)	9.7%	41.0%	49.3%	+20.3%
+5%	20.3%	48.3%	31.4%	+20.7%
+10%	35.6%	47.5%	17.0%	+21.7%
+20%	69.9%	27.0%	3.1%	+25.2%

Three things in that table matter more than the headline.

The directional band cannot tell the cases apart. Between 41% and 48% of studies land at 50% to 85% whether the true lift is zero, 5% or 10%. A directional result is what a study looks like when it did not measure anything.

A 10% lift is missed two times in three. With 1,000 conversions and a 10% holdout, the holdout has about 100 conversions. A real 10% lift reached 90% certainty in 35.6% of studies, and 17.0% of studies reported it as “no lift”.

The studies that clear 90% overstate the lift. With a true lift of 10%, the median study that reached 90% certainty reported +21.7%, more than double. With no lift at all, it reported +20.3%. A small study only clears the bar when chance pushed the result up, so the number you get from a “successful” study is inflated by how hard it was to clear.

What a bigger holdout buys

Holdout size is the one setting you choose. Google allows up to 50%. The same 100,000 users, with the true lift at 10%:

Holdout	Holdout conversions	90%+ certainty	“No lift”	Median lift reported at 90%+
10%	~100	35.6%	17.0%	+21.7%
20%	~200	48.1%	10.4%	+16.4%
50%	~500	60.6%	5.9%	+13.6%

The false results do not move: with no true lift, 9.7% to 10.0% of studies reached 90% at every holdout size, and 49.5% to 50.7% reported some lift. That is what a p-value threshold does by design. A larger holdout makes a real lift easier to see. It does nothing to the rate at which noise is reported as lift.

Turned around, the worked example asks what lift a study this size can see. For a real lift to reach 90% certainty in 80% of studies, it has to be at least 22.3% with a 10% holdout, 16.7% with 20%, and 13.4% with 50%. Most campaigns that are worth testing are worth testing because nobody knows whether their lift is 5% or 15%. At 1,000 conversions, this test cannot separate the two.

Why “certainty” reads stronger than it is

Google’s page describes certainty of lift as “the likelihood that the amount of lift was driven by your campaigns and not due to chance”. The calculation it gives, 1 minus the p-value, is a different quantity. A p-value says how likely data this positive would be if the ads did nothing. It is not the probability that the ads worked. That probability depends on how many of the campaigns you test have any effect, which the study does not know. This is the most common misreading of statistical significance, and the label invites it.

The 90% level is also looser than it sounds. Google’s intervals are 80% two-sided, so 90% certainty is a one-sided p-value under 0.10. One in ten campaigns with no effect clears it. Run ten studies a year on campaigns that do nothing and expect one “very good chance” result.

Google’s page does say the useful parts: a “no lift” result “doesn’t necessarily mean that your ads were ineffective”, and the segment with the highest certainty is not necessarily the best one. Both follow from the table above.

What to do differently

1. Decide the lift that would change your budget before you set up the study. Then read the feasibility estimate for that lift. If the setup shows less than 90%, change the design (a longer study, up to 56 days, a larger holdout, more of the campaigns) or do not run it.
2. Read 50% to 85% as “this study could not tell”. It is not a weak yes. Half of the studies on ads with no effect land at 50% or more.
3. Read the lower bound, not the point estimate. A study that just cleared 90% has a point estimate pushed up by noise. The lower bound of the interval is the number to plan a budget on.
4. Do not slice a small study. Every segment has fewer conversions than the study, and with several segments, one will clear 90% by chance.
5. Count the tests you ran, not only the ones that worked. One study at 90% out of ten is about what ten campaigns with no incremental effect would produce.

A lift study at this volume checks one thing well: whether the ad platform’s own conversion count is far from what the ads cause. That count is not the number in your analytics tool, and the two are built to do different jobs.