TorchCP: A Python Library for Conformal Prediction

Abstract: Conformal Prediction (CP) has attracted great attention from the research community due to its strict theoretical guarantees. However, researchers and developers still face challenges of applicability and efficiency when applying CP algorithms to deep learning models. In this paper, we introduce \torchcp, a comprehensive PyTorch-based toolkit to strengthen the usability of CP for deep learning models. \torchcp implements a wide range of post-hoc and training methods of conformal prediction for various machine learning tasks, including classification, regression, GNN, and LLM. Moreover, we provide user-friendly interfaces and extensive evaluations to easily integrate CP algorithms into specific tasks. Our \torchcp toolkit, built entirely with PyTorch, enables high-performance GPU acceleration for deep learning models and mini-batch computation on large-scale datasets. With the LGPL license, the code is open-sourced at \url{https://github.com/ml-stat-Sustech/TorchCP} and will be continuously updated.

• The paper introduces TorchCP, a PyTorch library that implements GPU-accelerated conformal prediction techniques for reliable uncertainty estimation.
• It details the use of split conformal prediction with innovative non-conformity scores and predictors for both classification and regression tasks.
• Empirical benchmarks on ImageNet and time-series data validate TorchCP's ability to achieve valid coverage rates and robust performance under distribution shifts.

Exploring TorchCP: A PyTorch-Based Library for Conformal Prediction

Introduction to Conformal Prediction and TorchCP

Conformal Prediction (CP) offers a framework for generating uncertainty intervals around the predictions of machine learning models, ensuring a predefined validity under the assumption of data exchangeability or the more restrictive i.i.d. condition. Recognizing the potential of CP in enhancing the reliability of deep learning models, the paper introduces TorchCP, a Python library built on PyTorch. This tool is designed to facilitate research in CP by providing efficient, GPU-accelerated implementations of various CP techniques for classification and regression tasks, including models with multi-dimensional outputs.

Methodology

The core of TorchCP's methodology revolves around the process of split conformal prediction. The authors outline this process distinctly for both classification and regression settings:

• For classification, TorchCP implements numerous non-conformity score functions and conformal prediction algorithms (predictors), alongside specific loss functions for model training. Notable score functions like THR, APS, RAPS, SAPS, and Margin, and predictors like SplitPredictor and ClassWisePredictor, are highlighted for their roles in calibrating and generating prediction sets under given significance levels.
• In the domain of regression, TorchCP addresses both one-dimensional and multi-dimensional output scenarios. It introduces predictors such as SplitPredictor, CQR, and ACI, designed to adapt CP methodologies to the challenges of regression analysis, including time-series problems characterized by distribution shifts.

Benchmark Results

The paper presents empirical evaluations to demonstrate TorchCP's capabilities in handling both classification and regression tasks:

• Classification Benchmark: Using the ImageNet dataset with models like ResNet101, the authors discuss the performance of various CP algorithms and score functions in providing marginal and conditional coverage. Metrics such as coverage rate, average set size, and CovGap are used for performance evaluation.
• Regression Benchmark: Focusing on a distribution-shift time series problem, the efficacy of the QuantileLoss and ACI predictor is analyzed. The comparison highlights ACI's superior performance in achieving valid marginal coverage rates, demonstrating TorchCP's utility in addressing challenging regression scenarios.

Discussing Practical and Theoretical Implications

TorchCP represents a significant step towards standardizing and simplifying the integration of conformal prediction with deep learning models. The library not only streamlines the implementation of CP techniques but also opens avenues for exploring novel applications of CP in ensuring model reliability. From a practical standpoint, TorchCP could enhance the robustness of machine learning applications in critical domains where prediction uncertainty quantification is paramount.

Theoretically, TorchCP's open-source nature and its foundational basis in PyTorch encourage further academic exploration and advances in CP methodologies. The library's ability to handle multi-dimensional outputs and adapt to complex problem settings like time-series analysis under distribution shifts prompts critical questions about the limits of conformal prediction and its adaptability to evolving machine learning challenges.

Looking Ahead

TorchCP is poised to serve as a pivotal resource for researchers and practitioners alike, enabling more robust uncertainty estimation in predictive modeling. As the library evolves, future work might include expanding its repository of CP techniques, enhancing its scalability, and incorporating feedback from the research community to address broader sets of prediction challenges. The integration of TorchCP with emerging deep learning paradigms could further augment its utility, making it a cornerstone tool in the pursuit of reliable machine learning predictions.

In summary, TorchCP's introduction offers a promising advancement in the field of conformal prediction, pushing the boundaries of statistical guarantees in machine learning predictions and fostering a richer understanding and application of CP techniques.