A comparative study of conformal prediction methods for valid uncertainty quantification in machine learning (2405.02082v1)

Abstract: In the past decades, most work in the area of data analysis and machine learning was focused on optimizing predictive models and getting better results than what was possible with existing models. To what extent the metrics with which such improvements were measured were accurately capturing the intended goal, whether the numerical differences in the resulting values were significant, or whether uncertainty played a role in this study and if it should have been taken into account, was of secondary importance. Whereas probability theory, be it frequentist or Bayesian, used to be the gold standard in science before the advent of the supercomputer, it was quickly replaced in favor of black box models and sheer computing power because of their ability to handle large data sets. This evolution sadly happened at the expense of interpretability and trustworthiness. However, while people are still trying to improve the predictive power of their models, the community is starting to realize that for many applications it is not so much the exact prediction that is of importance, but rather the variability or uncertainty. The work in this dissertation tries to further the quest for a world where everyone is aware of uncertainty, of how important it is and how to embrace it instead of fearing it. A specific, though general, framework that allows anyone to obtain accurate uncertainty estimates is singled out and analysed. Certain aspects and applications of the framework -- dubbed conformal prediction' -- are studied in detail. Whereas many approaches to uncertainty quantification make strong assumptions about the data, conformal prediction is, at the time of writing, the only framework that deserves the titledistribution-free'. No parametric assumptions have to be made and the nonparametric results also hold without having to resort to the law of large numbers in the asymptotic regime.

• The paper demonstrates that Mondrian CP enhances conditional validity by applying conformal prediction on specific data subgroups.
• It explains how dividing data into taxonomies enables tailored prediction intervals that better capture uncertainty across heterogeneous datasets.
• The study highlights that improved subgroup-specific reliability drives targeted model refinements for more trustworthy machine learning outcomes.

Understanding Conditional Validity in Conformal Prediction

What is Conformal Prediction?

Conformal Prediction (CP) is a statistical approach used in machine learning to assess the reliability of predictions. It provides a way to compute prediction intervals that, with high probability, cover the true outcome. CP is especially known for its ability to provide guarantees about the error rates of predictions, which is crucial for making the model reliable and trustworthy.

The Challenge of Conditional Validity

While CP is effective, it generally assures validity globally across the entire data set. This means it ensures that the prediction errors fall within a certain expected range when considering all predictions as a whole. However, this global view often overlooks the fact that different subsets of the data might have different levels of uncertainty. For instance, data points from one region may be more predictable compared to those from another due to less variability in measurements or differing conditions.

Example Scenario

Imagine a scenario where we predict the yield from two types of crops grown under different conditions. Conventional CP might suggest that our predictions are reliable overall. But, when we look closely, predictions for one type of crop in a specific condition could be far less accurate than the rest. This discrepancy calls for assessing prediction reliability more granularly - a concept known as "conditional validity."

Introducing Mondrian Conformal Prediction

To address the limitations of traditional CP, Mondrian Conformal Prediction (MCP) comes into play. MCP extends the standard CP framework and introduces a way to assess the validity conditionally on different subsets or "taxonomies" of the data. This is particularly useful when we suspect that the data's properties might vary significantly across different groups or conditions.

Workflow of MCP

1. Setting Up Taxonomies: Data is divided into different subgroups (taxonomies) based on certain features or conditions. For instance, crops could be subgrouped by type or climate conditions.
2. Applying CP to Each Group: Instead of applying CP to the entire dataset as a whole, MCP applies CP individually within each subgroup. This way, MCP ensures that the predictions are reliable within each specific context or condition.
3. Assessment of Prediction Intervals: For each subgroup, prediction intervals are calculated and assessed to ensure that they adequately cover the true outcomes within that subgroup.

Benefits of MCP

• Enhanced Reliability: By ensuring that predictions hold valid in specific conditions or subgroups, MCP enhances the overall reliability of the model across diverse conditions.
• Targeted Improvement: Helps in identifying and improving prediction models for subgroups where the performance may be lacking.
• Flexibility and Granularity: Offers a more nuanced understanding of model performance, which is vital in critical applications like healthcare or agriculture.

Future Directions

With the growing complexity of datasets and models, the need for robust methods like MCP will only increase. Future research could explore:

• Automated Taxonomy Generation: Leveraging machine learning to automate the grouping of data into meaningful taxonomies.
• Integration with Other ML Techniques: Combining MCP with other machine learning advancements to enhance prediction accuracy and reliability.
• Real-world Applications: Extensive testing of MCP in varied sectors to understand its practical benefits and limitations fully.

Conclusion

Mondrian Conformal Prediction marks a significant step forward in the continual effort to enhance the reliability of predictive models, especially in scenarios with heterogeneous data. By validating predictions within specific contexts, MCP not only boosts confidence in model outputs but also highlights areas for model refinement and improvement.