Anytime-valid t-tests and confidence sequences for Gaussian means with unknown variance (2310.03722v5)
Abstract: In 1976, Lai constructed a nontrivial confidence sequence for the mean $\mu$ of a Gaussian distribution with unknown variance $\sigma2$. Curiously, he employed both an improper (right Haar) mixture over $\sigma$ and an improper (flat) mixture over $\mu$. Here, we elaborate carefully on the details of his construction, which use generalized nonintegrable martingales and an extended Ville's inequality. While this does yield a sequential t-test, it does not yield an "e-process" (due to the nonintegrability of his martingale). In this paper, we develop two new e-processes and confidence sequences for the same setting: one is a test martingale in a reduced filtration, while the other is an e-process in the canonical data filtration. These are respectively obtained by swapping Lai's flat mixture for a Gaussian mixture, and swapping the right Haar mixture over $\sigma$ with the maximum likelihood estimate under the null, as done in universal inference. We also analyze the width of resulting confidence sequences, which have a curious polynomial dependence on the error probability $\alpha$ that we prove to be not only unavoidable, but (for universal inference) even better than the classical fixed-sample t-test. Numerical experiments are provided along the way to compare and contrast the various approaches, including some recent suboptimal ones.
- Time-uniform confidence spheres for means of random vectors. arXiv preprint arXiv:2311.08168, 2023.
- D. A. Darling and H. Robbins. Confidence sequences for mean, variance, and median. Proceedings of the National Academy of Sciences, 58(1):66–68, 1967.
- J. C. Duchi and S. Haque. An information-theoretic lower bound in time-uniform estimation. arXiv preprint arXiv:2402.08794, 2024.
- J. Ghosh. On some properties of sequential t-test. Calcutta Statistical Association Bulletin, 9(3):77–86, 1960.
- Safe testing. Journal of the Royal Statistical Society: Series B (to appear with discussion), 2023.
- P. G. Hoel. On a property of the sequential t-test. Scandinavian Actuarial Journal, 1954(1):19–22, 1954.
- S. R. Howard and A. Ramdas. Sequential estimation of quantiles with applications to A/B testing and best-arm identification. Bernoulli, 28(3):1704 – 1728, 2022. doi: 10.3150/21-BEJ1388. URL https://doi.org/10.3150/21-BEJ1388.
- Time-uniform Chernoff bounds via nonnegative supermartingales. Probability Surveys, 17:257–317, 2020.
- Time-uniform, nonparametric, nonasymptotic confidence sequences. The Annals of Statistics, 49(2):1055–1080, 2021.
- Always valid inference: Continuous monitoring of A/B tests. Operations Research, 70(3):1806–1821, 2022.
- Dealing with unknown variances in best-arm identification. In International Conference on Algorithmic Learning Theory, pages 776–849. PMLR, 2023.
- A. Klenke. Probability theory: a comprehensive course. Springer Science & Business Media, 2013.
- T. L. Lai. On confidence sequences. The Annals of Statistics, pages 265–280, 1976.
- The numeraire e-variable. arXiv preprint arXiv:2402.18810, 2024.
- Anytime-valid linear models and regression adjusted causal inference in randomized experiments. arXiv preprint arXiv:2210.08589, 2022.
- E-statistics, group invariance and anytime valid testing. arXiv preprint arXiv:2208.07610, 2022.
- Testing exchangeability: Fork-convexity, supermartingales and e-processes. International Journal of Approximate Reasoning, 141:83–109, 2022.
- H. Robbins. Statistical methods related to the law of the iterated logarithm. The Annals of Mathematical Statistics, 41(5):1397–1409, 1970.
- Bayesian t tests for accepting and rejecting the null hypothesis. Psychonomic bulletin & review, 16:225–237, 2009.
- A composite generalization of Ville’s martingale theorem. Electronic Journal of Probability, 2023.
- S. Rushton. On a sequential t-test. Biometrika, 37(3/4):326–333, 1950.
- J. Sacks. A note on the sequential t-test. The Annals of Mathematical Statistics, 36(6):1867–1869, 1965.
- P. Sprent and N. C. Smeeton. Applied nonparametric statistical methods. CRC press, 2007.
- Student. The probable error of a mean. Biometrika, pages 1–25, 1908.
- J. Ville. Etude critique de la notion de collectif. Bull. Amer. Math. Soc, 45(11):824, 1939.
- V. Vovk and R. Wang. Efficiency of nonparametric e-tests. arXiv preprint arXiv:2208.08925, 2022.
- A. Wald. Sequential tests of statistical hypotheses. The Annals of Mathematical Statistics, 16(2):117–186, 1945.
- A. Wald. Sequential analysis. 1947.
- H. Wang and A. Ramdas. The extended Ville’s inequality for nonintegrable nonnegative supermartingales. arXiv preprint arXiv:2304.01163, 2023a.
- H. Wang and A. Ramdas. Catoni-style confidence sequences for heavy-tailed mean estimation. Stochastic Processes and their Applications, 163:168–202, 2023b.
- Universal inference. Proceedings of the National Academy of Sciences, 117(29):16880–16890, 2020.
- I. Waudby-Smith and A. Ramdas. Estimating means of bounded random variables by betting. Journal of the Royal Statistical Society: Series B (Methodological), with discussion, 2023.
- Time-uniform central limit theory and asymptotic confidence sequences. arXiv preprint arXiv:2103.06476, 2021.
- A. Zellner and A. Siow. Posterior odds ratios for selected regression hypotheses. Trabajos de estadística y de investigación operativa, 31:585–603, 1980.