All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously (1801.01489v5)

Abstract: Variable importance (VI) tools describe how much covariates contribute to a prediction model's accuracy. However, important variables for one well-performing model (for example, a linear model $f(\mathbf{x})=\mathbf{x}^{T}\beta$ with a fixed coefficient vector $\beta$) may be unimportant for another model. In this paper, we propose model class reliance (MCR) as the range of VI values across all well-performing model in a prespecified class. Thus, MCR gives a more comprehensive description of importance by accounting for the fact that many prediction models, possibly of different parametric forms, may fit the data well. In the process of deriving MCR, we show several informative results for permutation-based VI estimates, based on the VI measures used in Random Forests. Specifically, we derive connections between permutation importance estimates for a single prediction model, U-statistics, conditional variable importance, conditional causal effects, and linear model coefficients. We then give probabilistic bounds for MCR, using a novel, generalizable technique. We apply MCR to a public data set of Broward County criminal records to study the reliance of recidivism prediction models on sex and race. In this application, MCR can be used to help inform VI for unknown, proprietary models.

• The paper introduces Model Class Reliance (MCR), a measure quantifying variable importance uncertainty across an entire class of high-performing predictive models, addressing the Rashomon effect.
• The authors derive theoretical connections for permutation-based VI estimates, linking them to U-statistics, causal effects, and linear model coefficients, and provide probabilistic bounds for MCR estimates.
• MCR is applied to criminal recidivism prediction using the COMPAS system, demonstrating its utility in evaluating reliance on sensitive demographic variables for fairness and transparency.

Model Class Reliance: An In-Depth Analysis

The paper "All Models are Wrong, but Many are Useful: Learning a Variable's Importance by Studying an Entire Class of Prediction Models Simultaneously" by Aaron Fisher, Cynthia Rudin, and Francesca Dominici, addresses a critical challenge in the field of machine learning and statistical modeling: the uncertainty and variability in variable importance across different, yet well-performing models. This is encapsulated in the concept the authors term as Model Class Reliance (MCR).

Overview of Model Class Reliance (MCR)

Variable Importance (VI) is a common metric used for understanding the role of covariates in the predictive accuracy of a model. Traditional approaches, such as those used in Random Forests, measure VI by permuting feature values and observing changes in model accuracy. However, these approaches often overlook the Rashomon effect, where many different models, potentially with different important variables, can fit a dataset similarly well. MCR addresses this by providing a range of VI values across all well-performing models within a specified class. This introduces a spectrum rather than a point estimate for VI, offering a more nuanced insight into which covariates are indispensable across different modeling hypotheses.

Key Contributions

1. Permutation-based VI Estimates: The authors derive connections for VI measures, especially focusing on permutation-based estimates, linking them with U-statistics, causal effects, and linear model coefficients. This theoretical groundwork demonstrates the robustness of their approach across different statistical paradigms.
2. Probabilistic Bounds: Probabilistic bounds for MCR are derived, offering insights into the finite-sample uncertainties associated with these estimates. These bounds provide a statistical assurance that enhances the practical utility of MCR for modeling and inference.
3. Application to Criminal Recidivism Prediction: A practical application of MCR is demonstrated in studying recidivism prediction models' reliance on demographic variables such as sex and race. This application to the COMPAS system highlights the tool's potential in evaluating and ensuring fairness and transparency in predictive systems used in critical real-world applications.

Implications and Future Directions

Theoretical Implications: MCR brings a shift from the traditional focus on individual model parameters to an emphasis on entire model classes. This shift acknowledges the inherent uncertainty in model specification and encourages a more holistic view of variable importance.

Practical Implications: The ability to discern which variables maintain their importance across multiple models can guide the design of more robust predictive systems. Moreover, in fields sensitive to fairness and ethical considerations, like criminal justice or healthcare, MCR can serve as a diagnostic tool for detecting potential biases indirectly linked through proxy variables.

Future Research: Future directions may focus on extending MCR computational techniques to complex model classes, such as deep neural networks, where the model space is vast and heterogeneous. Additionally, exploring MCR's role in automated variable selection and model auditing could open new avenues for its application in machine learning pipelines.

In conclusion, this paper introduces Model Class Reliance as a stepping-stone toward more interpretable and reliable machine learning models. By accounting for variability across model classes, MCR enhances our understanding of variable importance, offering a richer interpretative framework that can adapt to diverse applications and modeling complexities.