Statistical Significance in Meta A/B Tests Explained

When a Meta A/B test ends, the winner comes with a confidence percentage: Meta’s measure of how likely the same result is if the test were run again under the same conditions. Before the test starts, it may also show an estimated power figure. The two numbers answer different questions: power is a forecast about whether the test can find a result, and confidence describes the result it found.

What does the confidence percentage mean?

Meta says it shows a confidence percentage to help represent the likelihood of a causal result, which it defines as the likelihood that a test’s outcome would be consistent if repeated under the same conditions. It says it uses the same statistical rigor common in the field of statistical testing.

“Statistical significance” is the general term; in Meta’s test results, the number to read is the confidence percentage.

How does Meta calculate it for an A/B test?

For A/B tests without a holdout, Meta’s process is:

1. Compare each campaign on cost per result, where the result is the event you picked when creating the test, such as cost per click or cost per purchase.
2. Simulate possible outcomes tens of thousands of times, based on the activity observed during the test.
3. Count how often each version would have won across those simulations.
4. Report the winner with a confidence percentage, the chance of similar results if the test were repeated.

Once a winner is determined, Meta sends a notification in Ads Manager and an email with the results.

What thresholds does Meta use?

Test type	Confidence Meta treats as meaningful
A/B test	65% or higher represents a winning result
Lift test	90% or higher represents a statistically reliable result

Meta adds that if you have older test results, you may see a range, with plus and minus values or a single ± value, instead of one result with a confidence figure.

How is confidence different from estimated power?

Estimated power comes before the test. Meta describes it as a calculation that suggests the likelihood of a causal result based on your objectives, budget and other conditions of the test. Meta suggests tests have an estimated power of 80% or higher, and says the figure is only an estimate that doesn’t guarantee a causal result. Confidence comes after, and describes how reliable the result you got is.

Why can the winner have a higher cost?

Meta lists three reasons a winning campaign might still show higher costs:

• The test ran for too short a time.
• There weren’t enough results to calculate an accurate winner.
• Best practices for study length weren’t followed or met.

FAQ

Is 65% confidence enough to call an A/B test?

It is Meta’s threshold for a winning A/B test result. Meta defines it as a 65% chance of similar results if the test were repeated. By simple arithmetic, not a Meta figure, that leaves roughly one repeat in three turning out differently. For lift tests, Meta sets the bar at 90% for a statistically reliable result.

Does high estimated power guarantee a winner?

No. Meta says the power calculation is only an estimate and doesn’t guarantee a causal result from your study. It is based on your objectives, budget and other test conditions, and Meta suggests 80% or higher to increase the chance of a causal result. The confidence percentage after the test is what describes the result.

Can I compare A/B test versions on incremental purchases?

Meta says you can add a holdout to an A/B test to compare the cost per incremental purchase of each campaign. Without a holdout, versions are compared on cost per result for the event you chose.