How Do You Run a Holdout Test on Meta?

A holdout test asks a different question from an A/B test: whether your ads caused conversions at all, not which version wins. It fits when you want to check what your reported results owe to ads and you meet Meta’s spend and conversion guide. It does not fit when you only want to compare two creatives or audiences. On Meta, the self-serve route is a conversion lift test in Experiments.

What is a holdout group on Meta?

Meta defines it as the people who match your ad audience but are intentionally withheld from seeing your ads. Meta also calls it the control group. A lift test works like this:

1. You choose a business objective and a campaign to test against a hypothesis.
2. Meta divides the campaign’s intended audience randomly into test and holdout groups. People stay in their group for the whole test.
3. The test group has the opportunity to see your ads. The holdout group is withheld.
4. Meta measures actions such as purchases in both groups and compares them to calculate lift and a confidence percentage.

How is it different from an A/B test?

Meta says a lift test is not the same as an A/B test, because A/B tests do not include randomized holdout groups. An A/B test compares two versions that are both shown. A lift test compares showing your ads with withholding them.

Is Meta’s older holdout test still available?

Meta’s page on disabling holdout tests says holdout tests were paused in the Experiments tool on 26 April 2021, after Apple’s iOS 14.5 changes, and that A/B tests with a holdout and holdout tests in Experiments are no longer available.

Meta’s separate conversion lift pages describe a lift test you can create in Experiments. If the pages seem to disagree, trust what the Experiments tool offers you; Meta notes its Get started button may be disabled if you miss the spend or conversion requirements.

What do you need before you start?

Meta gives these as a guide:

• A campaign that started in the past year with at least $5,000 USD in spend and 500 conversions on a 1-day click, 7-day click or 1-day view setting, prorated for campaigns over 90 days.
• Signal quality: Conversions API sending an event with an event match quality score above 5, or a supported data source sending a low-funnel or app install event. Meta says that second option is unavailable for advertisers primarily targeting countries affected by the ePrivacy Directive.

How do you set one up?

1. In Ads Manager, open All tools and select Experiments under Analyze and report.
2. Click Create test, then Get started under Conversion lift.
3. Select your learning objective, then the ad accounts, campaigns or ad sets to include. Meta says choosing an ad account includes all its active campaigns; select campaigns or ad sets instead to test only part of your spend.
4. Add cells if you want to compare strategies, then set the dates, name the test and choose your data source and events.
5. Review the details and click Create test.

Meta recommends a minimum budget of $5,000 USD and a duration of at least 28 days to increase the likelihood of statistical significance.

How do you read the result?

Lift is the difference in conversions between the test and control groups. Meta calls its approach conservative and says it is designed to provide its best estimate of the causal impact of ads. It also provides intervals showing the range of most probable outcomes, so read the range, not only the midpoint.

Results appear in Experiments once the study has finished. Meta does not recommend using the post-test conversion window to calculate them. Meta says lift results are unique to your test’s conditions and are not meant to be compared with campaign results in Ads Manager, because the two attribute conversions differently.

What if you do not qualify?

Meta’s pages say you can also contact your Meta account representative about a conversion lift test, if you have one; the pages cited here do not say that route waives the spend or conversion guide. You could also design your own holdout outside Meta’s tools, for example by pausing spend in a few regions. The Meta pages cited here do not cover that; the posts linked below discuss it.

FAQ

Does a holdout test cost extra?

Meta says there is no additional cost for a conversion lift test. The campaigns or ad sets you use may have budget requirements, and a holdout group means your ads are withheld from part of the audience for the length of the test.

Can people in the holdout group see my ads?

No. Meta says the control group is not eligible to be exposed to your ads during the study. The test group is eligible, but some of its people may not see an ad because of delivery limits or because they never scrolled to it.

Does a holdout test replace attribution settings?

The Meta pages cited here do not say that. They list, as one question a lift test’s results can answer, whether your current attribution model aligns with the results of a controlled experiment. They also say lift results are not meant to be compared with Ads Manager results, so treat the two as separate measures.