What it means
Every campaign gets credit for results. Some of those results would have happened anyway. The people who already loved you, the fans who were going to stream the new season, the buyers searching your name: they show up in the campaign report, but the campaign didn’t bring them. Incrementality is what’s left when you take them out.
The two-sentence version for your boss: we ran the campaign in some regions and not in others. The difference in results between the two groups is what the campaign did.
It works the same whatever you count. For a studio, the result might be tickets in each region. For a streaming service, sign-ups or hours viewed. For a drinks brand, sales through retail partners by region. As long as you can see the result by region, or by some other group you can split, you can run the test.
Why matching isn’t enough
Matching every reported result to one in your own numbers removes double counting. That’s a big step. It tells you which results are possible. It can’t tell you which ones the campaign caused.
Take people already searching your brand’s name. They click an ad on their way to a purchase they’d already decided on. Matching finds obvious cases like that. It can’t find the quieter ones: the fan who saw a trailer and would have watched anyway, or the regular who bought her usual case the week your ad ran. Only a comparison against people who didn’t see the campaign can.
This matters most for the channels that sit closest to the purchase. Brand search, retargeting and last-click credit all reach people at the final step. Their reported results can look excellent while adding little, because the buyers were already on their way. Those are usually the first channels worth testing.
The test, step by step
- Write the question. “Does paid social drive sign-ups in our top markets?” One channel, one result.
- Pick regions that look alike: similar size, similar history of sales, tickets or sign-ups.
- Split them into two groups. Run the campaign in one. Leave the other alone. This is the holdout.
- Agree the length before you start, long enough for results to move, usually a few weeks.
- Change nothing else. No new promotions in one group only, no price changes, no surprise press push.
- Compare results between the two groups using your own numbers, not the platform’s.
- Write down what you’ll do with each outcome before you look.
A worked example
Made up to show how a test reads, not from a client.
Your search ads get credit for a large share of results, and most of what holds up after matching came through brand search. The question: if you stopped paying for clicks on your own name, would those buyers still find you? The test pauses brand search in half your regions for three weeks and keeps it on in the other half.
- If results hold in the paused regions, people were finding you without the paid click. That budget can move somewhere that earns it.
- If results drop, the channel works, and the size of the drop tells you by how much.
Either way you learn something no report could tell you. And the test itself is cheap compared with a year of spending on a guess: three weeks, half the regions, and a comparison the whole team can read.
When it’s worth running: a checklist
- You’re about to move a meaningful amount of budget.
- A channel’s numbers look too good, like brand search or view-throughs.
- Two channels report the same results and you need to know which one earned them.
- You can hold out regions without hurting a launch you can’t repeat.
How long and how big: long enough for results to move and settle, and big enough that normal week-to-week swings don’t drown out the difference. For most brand campaigns that means a few weeks and a meaningful share of your regions. If your results are small or noisy, run longer rather than smaller. Agree all of this in writing before the test starts, with everyone who touches the campaign.
Who should run it? Whoever buys your media can set up the regions. Reading the result should start from your own numbers, with the same counting rule for every channel. That keeps the conversation afterward about what the numbers say.
Write the plan on one page: the question, the regions in each group, the dates, the result you’ll compare, and what you’ll do if results hold, rise or drop. Share it with everyone before the test starts.
Questions worth asking your team: Which channel should we test before we increase spend? Which regions would we hold out, and for how long? What result would make us cut the channel?
Common mistakes, and what a test doesn’t prove
- Stopping too early. Results take time to move. Agree the length up front and keep to it.
- Choosing regions that aren’t alike. A big city against small towns tells you about geography.
- Changing other things mid-test. One extra promotion in one group spoils the comparison.
- Reading the platform’s numbers. Use your own. The platform can’t see the regions where it didn’t run.
- Testing during your one big launch. Pick a period you can repeat.
A test measures one campaign, in one set of regions, over one period. It doesn’t tell you what the campaign will do at twice the budget, in a different season, or with new creative. It also doesn’t replace a media mix model, which estimates every channel at once from history. A test measures one channel directly, and it’s the best way to check what a model says.
We design and read these tests as part of Arcus, on your own numbers. To start smaller, try it free with a question like which spend paid off, before you test anything. If the answer points to a channel worth testing, we’ll say so.