Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations, and Forecast Rankings (1503.08195v2)

Abstract: In the practice of point prediction, it is desirable that forecasters receive a directive in the form of a statistical functional, such as the mean or a quantile of the predictive distribution. When evaluating and comparing competing forecasts, it is then critical that the scoring function used for these purposes be consistent for the functional at hand, in the sense that the expected score is minimized when following the directive. We show that any scoring function that is consistent for a quantile or an expectile functional, respectively, can be represented as a mixture of extremal scoring functions that form a linearly parameterized family. Scoring functions for the mean value and probability forecasts of binary events constitute important examples. The quantile and expectile functionals along with the respective extremal scoring functions admit appealing economic interpretations in terms of thresholds in decision making. The Choquet type mixture representations give rise to simple checks of whether a forecast dominates another in the sense that it is preferable under any consistent scoring function. In empirical settings it suffices to compare the average scores for only a finite number of extremal elements. Plots of the average scores with respect to the extremal scoring functions, which we call Murphy diagrams, permit detailed comparisons of the relative merits of competing forecasts.

Consistent Scoring Functions for Quantiles and Expectiles

The paper "Of Quantiles and Expectiles: Consistent Scoring Functions, Choquet Representations, and Forecast Rankings" by Werner Ehm, Tilmann Gneiting, Alexander Jordan, and Fabian Krüger provides a comprehensive investigation into scoring functions for quantiles and expectiles, shedding light on their representation through Choquet type mixtures. The research is a significant contribution to the evaluation and comparison of probabilistic forecasts, particularly in decision-making scenarios where point predictions are necessary. The paper is situated within the domain of computational statistics, addressing both theoretical developments and practical applications.

The primary focus of the paper is to establish that any scoring function consistent for a quantile or expectile functional can be decomposed into a mixture of extremal scoring functions. This is achieved through the construction of a linearly parameterized family of elementary scores, providing a unified framework for understanding and applying these scoring rules. The paper extends classical results by Savage (1971) and connects to decision theory through economic interpretations of these scoring functions.

Among the key findings is the demonstration that the quantile and expectile functionals, along with their corresponding extremal scoring functions, offer appealing economic interpretations, akin to threshold-based decision-making problems. For quantiles, this involves payoffs from betting scenarios, while expectiles relate to investment decisions under differential taxation conditions. This interpretation not only enriches the theoretical understanding but also emphasizes the practical relevance of these scoring functions in real-world economic contexts.

The research further explores the implications of these results for dominance in forecast comparisons. By leveraging the Choquet representations, the authors propose a simplified criterion for forecast dominance using Murphy diagrams. These diagrams plot the empirical scores of forecasts against evaluation metrics, providing a clear visual representation to compare forecast performance across different scoring functions. The concept of dominance explored here implies that one forecast outperforms another under any consistent scoring function.

From an empirical perspective, the paper underscores the practical utility of these findings through applications in synthetic and real-world data scenarios. By analyzing economic and meteorological data, the paper illustrates how these theoretical constructs are pertinent to actual forecasting tasks. Particularly, the examination of U.S. economic forecasts and wind speed predictions underscore the broad applicability of the framework.

In conclusion, this research sets the stage for further explorations into the role of consistent scoring functions beyond quantiles and expectiles, possibly extending into multi-dimensional functional spaces. Future work may explore the optimal choice of scoring functions for specific empirical tasks, drawing on insights from the mixture representations. Furthermore, the connection between scoring rules and decision-making frameworks suggests potential integrations of these methods within risk management and regulatory contexts, offering a promising avenue for subsequent inquiries into the utility and applicability of scoring functions in diverse domains of statistical forecasting.