Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
194 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

A/B testing under Interference with Partial Network Information (2404.10547v1)

Published 16 Apr 2024 in cs.LG

Abstract: A/B tests are often required to be conducted on subjects that might have social connections. For e.g., experiments on social media, or medical and social interventions to control the spread of an epidemic. In such settings, the SUTVA assumption for randomized-controlled trials is violated due to network interference, or spill-over effects, as treatments to group A can potentially also affect the control group B. When the underlying social network is known exactly, prior works have demonstrated how to conduct A/B tests adequately to estimate the global average treatment effect (GATE). However, in practice, it is often impossible to obtain knowledge about the exact underlying network. In this paper, we present UNITE: a novel estimator that relax this assumption and can identify GATE while only relying on knowledge of the superset of neighbors for any subject in the graph. Through theoretical analysis and extensive experiments, we show that the proposed approach performs better in comparison to standard estimators.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (65)
  1. Graph-based methods for analysing networks in cell biology. Briefings in bioinformatics, 7(3):243–255.
  2. A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts: treatments for myocardial infarction. Jama, 268(2):240–248.
  3. Estimating average causal effects under general interference, with application to a social network experiment. The Annals of Applied Statistics, 11(4):1912–1947.
  4. The local approach to causal inference under network interference. arXiv preprint arXiv:2105.03810.
  5. Heterogeneous treatment and spillover effects under clustered network interference. arXiv preprint arXiv:2008.00707.
  6. Model-assisted design of experiments in the presence of network-correlated outcomes. Biometrika, 105(4):849–858.
  7. Causal inference under interference and network uncertainty. In Adams, R. P. and Gogate, V., editors, Proceedings of The 35th Uncertainty in Artificial Intelligence Conference, volume 115 of Proceedings of Machine Learning Research, pages 1028–1038. PMLR.
  8. Cluster randomized designs for one-sided bipartite experiments. Advances in Neural Information Processing Systems, 35:37962–37974.
  9. Social networks and the decision to insure. American Economic Journal: Applied Economics, 7(2):81–108.
  10. Double/debiased machine learning for treatment and structural parameters.
  11. Chin, A. (2019). Regression adjustments for estimating the global treatment effect in experiments with interference. Journal of Causal Inference, 7(2).
  12. Graph agnostic estimators with staggered rollout designs under network interference. Advances in Neural Information Processing Systems.
  13. Cox, D. R. (1958). Planning of experiments.
  14. D’Amour, A. (2019). On multi-cause causal inference with unobserved confounding: Counterexamples, impossibility, and alternatives. arXiv preprint arXiv:1902.10286.
  15. A flexible, interpretable framework for assessing sensitivity to unmeasured confounding. Statistics in Medicine, 35(20):3453–3470.
  16. Design and analysis of experiments in networks: Reducing bias from interference. Journal of Causal Inference, 5(1).
  17. On the evolution of random graphs. Publ. math. inst. hung. acad. sci, 5(1):17–60.
  18. Fine, P. E. (1993). Herd immunity: history, theory, practice. Epidemiologic reviews, 15(2):265–302.
  19. Network a/b testing: From sampling to estimation. In Proceedings of the 24th International Conference on World Wide Web, pages 399–409. International World Wide Web Conferences Steering Committee.
  20. Dependent happenings: a recent methodological review. Current epidemiology reports, 3(4):297–305.
  21. The statistical analysis of local structure in social networks.
  22. A generalization of sampling without replacement from a finite universe. Journal of the American statistical Association, 47(260):663–685.
  23. Toward causal inference with interference. Journal of the American Statistical Association, 103(482):832–842.
  24. Directed-graph epidemiological models of computer viruses. In Computation: the micro and the macro view, pages 71–102. World Scientific.
  25. Private causal inference. In Artificial Intelligence and Statistics, pages 1308–1317. PMLR.
  26. Introduction to spatial econometrics. Chapman and Hall/CRC.
  27. Leung, M. P. (2016). Treatment and spillover effects under network interference. Review of Economics and Statistics, pages 1–42.
  28. Leung, M. P. (2020). Treatment and spillover effects under network interference. Review of Economics and Statistics, 102(2):368–380.
  29. Interference, bias, and variance in two-sided marketplace experimentation: Guidance for platforms. In Proceedings of the ACM Web Conference 2022, pages 182–192.
  30. Causal inference under network interference with noise. arXiv preprint arXiv:2105.04518.
  31. An introduction to sensitivity analysis for unobserved confounding in nonexperimental prevention research. Prevention Science, 14(6):570–580.
  32. Identifying effects of multiple treatments in the presence of unmeasured confounding. arXiv preprint arXiv:2011.04504.
  33. Neyman, J. (1923). On the Application of Probability Theory to Agricultural Experiments: Essay on Principles. Statistical Science, 5:465–80. Section 9 (translated in 1990).
  34. Causal inference for social network data. arXiv preprint arXiv:1705.08527.
  35. Causal inference with spatio-temporal data: Estimating the effects of airstrikes on insurgent violence in iraq. arXiv preprint arXiv:2003.13555.
  36. Optimal design of experiments on connected units with application to social networks. Journal of the Royal Statistical Society: Series C (Applied Statistics), 66(3):455–480.
  37. Pearl, J. (2009). Causality. Cambridge university press.
  38. Variance reduction in bipartite experiments through correlation clustering. Advances in Neural Information Processing Systems, 32.
  39. Testing for arbitrary interference on experimentation platforms. arXiv preprint arXiv:1704.01190.
  40. Herd immunity: understanding covid-19. Immunity, 52(5):737–741.
  41. Multiple causal inference with latent confounding. arXiv preprint arXiv:1805.08273.
  42. Nonparametric bounds and sensitivity analysis of treatment effects. Statistical Science: A Review Journal of the Institute of Mathematical Statistics, 29(4):596.
  43. Ross, N. (2011). Fundamentals of stein’s method. Probability Surveys, 8:210–293.
  44. Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology, 66(5):688.
  45. Rubin, D. B. (1987). The calculation of posterior distributions by data augmentation: Comment: A noniterative sampling/importance resampling alternative to the data augmentation algorithm for creating a few imputations when fractions of missing information are modest: The sir algorithm. Journal of the American Statistical Association, 82(398):543–546.
  46. Community structure and scale-free collections of erdos-renyi graphs. Physical Review E, 85(5):056109.
  47. Direct inference of effect of treatment (diet) for a cookieless world. In Proceedings of the The 26th International Conference on Artificial Intelligence and Statistics (AISTATS).
  48. Privacy aware experiments without cookies. In Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining, WSDM ’23. Association for Computing Machinery.
  49. A/B testing: The most powerful way to turn clicks into customers. John Wiley & Sons.
  50. Spirtes, P. (2010). Introduction to causal inference. Journal of Machine Learning Research, 11(5).
  51. Elements of estimation theory for causal effects in the presence of network interference. arXiv preprint arXiv:1702.03578.
  52. Data-efficient off-policy policy evaluation for reinforcement learning. In International Conference on Machine Learning, pages 2139–2148. PMLR.
  53. Importance sampling with unequal support. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31.
  54. Estimation of causal peer influence effects. In International Conference on Machine Learning, pages 1489–1497.
  55. Tukey, H. (1956). Conditional monte carlo for normal samples. In Proc. Symp. on Monte Carlo Methods, pages 64–79. John Wiley and Sons.
  56. Graph cluster randomization: Network exposure to multiple universes. In Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 329–337. ACM.
  57. Sense and sensitivity analysis: Simple post-hoc analysis of bias due to unobserved confounding. arXiv preprint arXiv:2003.01747.
  58. Viviano, D. (2020). Experimental design under network interference. arXiv preprint arXiv:2003.08421.
  59. The blessings of multiple causes. Journal of the American Statistical Association, 114(528):1574–1596.
  60. Epidemic spreading in real networks: An eigenvalue viewpoint. In 22nd International Symposium on Reliable Distributed Systems, 2003. Proceedings., pages 25–34. IEEE.
  61. Wasserman, L. (2006). All of nonparametric statistics. Springer Science & Business Media.
  62. Wong, J. C. (2020). Computational causal inference. arXiv preprint arXiv:2007.10979.
  63. Bounds on the conditional and average treatment effect with unobserved confounding factors. arXiv preprint arXiv:1808.09521.
  64. Estimating total treatment effect in randomized experiments with unknown network structure. Proceedings of the National Academy of Sciences.
  65. Causal network motifs: identifying heterogeneous spillover effects in a/b tests. In Proceedings of the Web Conference 2021, pages 3359–3370.
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets