On Learning-Curve Monotonicity for Maximum Likelihood Estimators

Abstract: The property of learning-curve monotonicity, highlighted in a recent series of work by Loog, Mey and Viering, describes algorithms which only improve in average performance given more data, for any underlying data distribution within a given family. We establish the first nontrivial monotonicity guarantees for the maximum likelihood estimator in a variety of well-specified parametric settings. For sequential prediction with log loss, we show monotonicity (in fact complete monotonicity) of the forward KL divergence for Gaussian vectors with unknown covariance and either known or unknown mean, as well as for Gamma variables with unknown scale parameter. The Gaussian setting was explicitly highlighted as open in the aforementioned works, even in dimension 1. Finally we observe that for reverse KL divergence, a folklore trick yields monotonicity for very general exponential families. All results in this paper were derived by variants of GPT-5.2 Pro. Humans did not provide any proof strategies or intermediate arguments, but only prompted the model to continue developing additional results, and verified and transcribed its proofs.

• The paper establishes that MLEs show monotonic improvement with larger data sets under well-specified conditions in Gaussian and Gamma models.
• It utilizes advanced techniques like Wishart analysis and Laplace transforms to derive closed-form expressions for expected KL divergences.
• The study extends the findings to general exponential families, highlighting the robustness of MLEs and suggesting avenues for further research in complex models.

Detailed Summary of "On Learning-Curve Monotonicity for Maximum Likelihood Estimators" (2512.10220)

Introduction and Problem Statement

The paper "On Learning-Curve Monotonicity for Maximum Likelihood Estimators" addresses a significant question in the context of machine learning and statistical estimation. The primary focus is on the concept of learning-curve monotonicity, which posits that the performance of an algorithm should consistently improve as it receives more data. This notion is intuitively appealing but has been shown not to hold universally, especially in mis-specified models. The core question explored in this research is whether Maximum Likelihood Estimators (MLEs) demonstrate monotonic behavior under well-specified conditions.

Main Contributions

The authors provide several key results regarding the monotonicity of learning curves for MLEs:

1. Gaussian Vectors with Unknown Covariance: The paper establishes the monotonicity of the forward KL divergence in Gaussian models with unknown covariance matrices. This result applies to both cases where the mean is known and unknown. The dimension of Gaussian vectors explored in these results was previously noted as an open problem in the literature.
2. General Exponential Families: The study extends the monotonicity results to more general exponential families under the reverse KL divergence. This broader application suggests that monotonicity might be a more pervasive property than previously understood.
3. Gamma Distributions: Similar monotonicity properties are proven for Gamma distributions with unknown scale parameters, offering insights into the behavior of MLEs for non-Gaussian families.

Methodology and Theoretical Advancements

The paper uses advanced probabilistic tools and adopts a rigorous mathematical approach to establish monotonicity. The key methodologies include:

• Wishart Distribution Analysis: For Gaussian vectors, the authors analyze the Wishart distribution properties to understand the behavior of MLEs, employing properties of determinants and traces under this distribution.
• Laplace Transform Techniques: The authors leverage Laplace transform representations to derive complete monotonicity of the forward KL divergence functions in certain domains. This analytical technique not only establishes simple monotonicity but also provides stronger complete monotonicity results.
• Convexity in Exponential Families: Utilizing the convexity properties of the Bregman divergence in exponential families, the paper derives monotonicity for the reverse KL divergence, a result that benefits from the elegant interplay between convex analysis and probabilistic estimations.

Numerical and Theoretical Implications

The authors highlight specific formulas for expected KL divergences in Gaussian and Gamma settings, offering both practical insight and theoretical underpinnings for the monotonic behavior:

• Closed-form Expressions: Explicit formulas are derived for the expected forward and reverse KL divergences, elucidating how these expectations evolve with sample size.
• Complete Monotonicity: The Laplace transform approach underscores a complete understanding of monotonicity, suggesting that each additional data point contributes positively, albeit decreasingly, to model accuracy.

Conclusions and Future Directions

The paper concludes that MLEs exhibit desirable monotonicity properties under well-specified conditions across a range of statistical families. These findings have substantial implications for the deployment of MLEs in practical scenarios, advocating for their robust use given ample data. Future research could explore extending these monotonic findings to broader classes of estimators and investigating if similar monotonicity holds in more complex models, including neural networks and other nonparametric estimators. The study serves as a foundation for deeper exploration into learning curves' behavior in diverse statistical and machine learning contexts, providing a pathway for advancing both theoretical insights and practical algorithms.