Generalizability of minimized divergent delivery in strictly configured A/B tests

Determine whether the absence of statistical imbalance in impressed user characteristics across cells—observed in awareness-optimized Meta A/B tests configured with a common ad optimization goal, fixed target audience definitions, identical budgets and bidding strategies, distinct static-image creatives, and approximately one impression per user—generalizes to other advertising campaigns on Meta’s platforms.

Background

The paper shows that most A/B tests exhibit audience imbalance due to Meta’s ad delivery algorithms, but that careful configuration can mitigate this. After progressively filtering to tests with identical objectives, audiences, budgets, and bidding—and further restricting to single static-image creatives with impression frequency near one—the authors find no evidence of imbalance in a very small sample of three tests.

Because this sample is limited, the authors explicitly note uncertainty about whether these results generalize. They then expand the window and identify 17 additional tests meeting the strict criteria, again finding similar results, but they continue to caution against drawing strong conclusions from limited samples and emphasize that absolute balance cannot be guaranteed.