Attribution answers "which touchpoint gets credit for this conversion?" Incrementality answers a harder, more useful question: "would this conversion have happened anyway?" The gap between those two questions is where marketing budgets are frequently wasted on channels that look good in a dashboard but add little real revenue.
How Do You Measure True Incremental Impact?
Randomly withhold the activity — an ad channel, a feature, a promotion — from a control group while the rest of your comparable audience receives it normally, then compare the conversion outcome of both groups over the identical period.
The difference between the exposed group's conversion rate and the holdout group's conversion rate is the incremental effect — the portion of conversions that genuinely would not have happened without the activity. Anything converted by the holdout group would have converted anyway, regardless of what your attribution model claims.
What Is Incrementality, and Why Does Attribution Miss It?
Incrementality is the portion of a conversion outcome that a specific action actually caused, as opposed to attribution, which assigns credit based on which touchpoints a converting user happened to interact with regardless of causation.
A branded search ad, for example, often gets attributed heavy conversion credit because it's frequently the last click before purchase — but many of those same buyers were already searching your brand name because they intended to buy regardless of the ad. This is the same limitation discussed in conversion attribution models: attribution describes correlation along a path, not causation.
How Do Geo and Audience Holdout Tests Work?
An audience holdout randomly excludes a percentage of eligible users from seeing the activity; a geo holdout instead pauses the activity in selected regions while running it as usual elsewhere, then compares aggregate conversion trends between exposed and unexposed geographies.
Geo holdouts are useful when individual-level randomization isn't possible — for instance, with TV, radio, or out-of-home advertising that can't be targeted at the user level. Choose geographic regions with similar historical performance so the comparison is fair, and run the test long enough to smooth out normal regional variance before comparing results.
What Is a Public Service Announcement (PSA) Test?
A PSA test is a form of audience holdout used by ad platforms where the control group is served a neutral placeholder ad instead of your actual ad, keeping ad-serving conditions identical between groups while isolating the effect of your specific message.
This design controls for factors like ad-auction dynamics and platform algorithm behavior that a simple "don't show any ad" holdout might not equalize. Many major ad platforms offer PSA-based incrementality testing natively, which is often the most practical entry point for teams without in-house experimentation infrastructure.
When Should You Run an Incrementality Test?
Run one before significantly scaling spend on a channel, when attributed conversions look strong but overall business growth doesn't reflect it, or whenever a budget decision depends on proving a specific channel or feature caused real additional conversions.
Incrementality tests take real time and often real forgone conversions in the control group, so they're worth the cost primarily for decisions with meaningful budget or roadmap consequences — not for every minor optimization, where a standard A/B test is a faster and sufficient tool.
How Do You Interpret Incrementality Results?
Calculate the incremental conversion rate difference between exposed and holdout groups, express it as an incremental cost per conversion, and compare that figure — not the attributed cost per conversion — against your actual profitability threshold.
It's common to find that a channel's incremental cost per conversion is two to five times higher than its attributed cost per conversion, because a large share of "attributed" conversions would have happened regardless. That gap is exactly the information a budget decision needs and attribution alone cannot supply.
What Are the Limitations of Incrementality Testing?
Incrementality tests require enough scale to detect a statistically reliable difference, they measure one channel or activity at a time rather than full-funnel interactions, and holding out a group has a real, temporary revenue cost during the test.
Small businesses or low-volume channels often lack the traffic to run a properly powered holdout test — apply the same sample-size discipline used in A/B test sample size calculation before committing to a design, and be prepared to accept a wider confidence interval or a longer test window rather than an underpowered result.
Summary
Attribution tells you where credit falls; incrementality testing tells you what would have happened anyway. Use randomized holdouts or geo tests before major budget decisions, and judge channels by incremental cost per conversion rather than attributed cost per conversion.