What is holdout test?

A test where a campaign runs in some places and not others, so you can compare.

A holdout test is the most direct way to find out whether something caused a result. You run a campaign in some places, hold it back in similar places, and compare what happens. The difference is the campaign’s effect.

For brand teams, regions work well: cities, states or TV markets. You pick matched groups, run for a few weeks, and keep everything else steady. Each region needs enough volume to show a difference, and you have to go dark somewhere for a while, which is the part teams find hardest. It’s also the part that makes the answer trustworthy.

Why it matters

Run ads in half your regions for a few weeks and leave the other half alone. The difference in sales is the campaign’s real effect.

An example

Example (made-up numbers): you pause brand search ads in 8 of 16 regions for four weeks. Sales in the paused regions dip by 2%, while the search ads had been credited with 20% of sales. Most of those buyers found you anyway.

Where it misleads

Calling a test too early or on too few regions. A small difference over two weeks can be noise, and a confident read on noise is worse than no read.

How Arcus handles it

We design the test with your team, pick matched regions from your own numbers, and read the result. If your volume can’t support a clear answer, we say so before you go dark anywhere.

More in our guide: Incrementality, explained without the math