Localized Conformal Prediction for Image Classification with Vision-Language Models

Abstract: Conformal predictions have attracted significant attention in the field of uncertainty quantification, mainly because of their strong marginal coverage guarantees. Full conditional guarantee is not an attainable goal, a well known fact in conformal predictions literature. As a result, several approaches have tried to approximate this behavior by adapting the conformal sets of test-time samples according to their similarity to calibration examples. Although the latter has gained traction and shown impressive performances for regression problems, its application to image classification remains under-explored. We conduct an extensive benchmarking on natural image classification tasks with vision-LLMs (VLMs), using our open source implementation of a recent localized conformal prediction algorithm. We show that straightforward usage of the cosine similarity between test-time and calibration visual features, an intuitive choice for VLMs, is not sufficient to improve over the non-local baselines. In response, we propose a simple non-linear transformation of the cosine similarities, which conserves marginal coverage guarantees and achieves statistically significant mean set sizes reduction. Code is available at https://github.com/cfuchs2023/lcp-vlm/.

• The paper introduces a nonlinear sigmoid-based localizer that adapts calibration weights to test instances in VLM latent spaces.
• The paper demonstrates statistically significant reductions in mean prediction set sizes across multiple datasets and architectures compared to non-local CP baselines.
• The paper shows that improved set efficiency is achieved without compromising marginal or class-conditional coverage, motivating further research on conditional validity.

Localized Conformal Prediction for Image Classification with Vision-LLMs

Introduction

Uncertainty quantification in deep visual classification models is essential for critical domains such as medical imaging, surveillance, and automated decision systems. Conformal Prediction (CP) offers formal marginal coverage guarantees for uncertainty sets but lacks conditional guarantees, which are vital for heterogeneous data. Recently, Localized Conformal Prediction (LCP) has been proposed to address this by weighting calibration examples based on test instance similarity, but its effectiveness for visual classification tasks—particularly in combination with vision-LLMs (VLMs)—has been largely unexplored.

Conformal Prediction and Its Limitations

CP frameworks construct prediction sets based on non-conformity scores evaluated on a calibration set. For image classification, variants such as TopK, LAC, APS, and RAPS utilize scores aligned with the model’s predictive structure to balance coverage and efficiency. Despite the theoretical assurance of marginal coverage, substantial conditional coverage gaps persist. These arise from the failure of the global quantile approach to adapt to the local structure of the feature space, leading to mis-calibration within subpopulations or domains with distributional shifts.

Localized Conformal Prediction in Classification

LCP generalizes CP by introducing a similarity-based weighting scheme over the calibration set, resulting in test-specific quantile thresholds for prediction sets. This approach relies on a localizer function $$H(\boldsymbol{f}_{\text{test}},\, \boldsymbol{f}_i)$$, where $\boldsymbol{f}_{\text{test}}$ and $\boldsymbol{f}_i$ are embeddings of the test and calibration examples, respectively. In the context of VLMs, image and text are jointly embedded into a common high-dimensional, normalized latent space (e.g., via CLIP with ViT or CNN visual encoders).

Shortcomings of Naïve Similarity Weighting

A straightforward choice for $H$ is the cosine similarity, $\boldsymbol{f}_{\text{test}}^\top \boldsymbol{f}_i$. However, empirical analysis demonstrates that this naïve localizer does not consistently improve mean set size relative to non-local baselines and may even degrade set efficiency on several datasets and architectures. Notably, statistically significant increases in prediction set size are observed in cases such as DTD and UCF101 with RN50, indicating that cosine similarity is a suboptimal test-time localizer in VLM-based image classification.

Proposed Nonlinear Localizer and Its Implementation

The authors introduce a nonlinear localizer by applying a parametrized sigmoid transformation to the cosine similarity:

$$H(\boldsymbol{f}_i, \boldsymbol{f}_j) = \frac{1 + \exp(-m(1 - \tau))}{1 + \exp(-m(\boldsymbol{f}_i^\top \boldsymbol{f}_j - \tau))}$$

where $m$ controls the sharpness and $\tau$ the inflection of the sigmoid. These hyperparameters are selected via 5-fold cross-validation on calibration data to minimize average prediction set size. This nonlinear transformation compensates for the concentration characteristics of VLM latent spaces and sharpens the influence of highly similar samples on test-time quantile estimation.

Figure 1: Examples of optimal sigmoid transformations $H$ used as localizers in LCP and the contrast with the naïve identity function (dotted line).

Experimental Protocol

The methodology was extensively benchmarked on nine diverse classification datasets (UCF101, DTD, Pets, EuroSAT, StanfordCars, Flower102, Aircraft, SUN397, Food101), using calibration splits that fully preserve the overall class label distribution:

Figure 2: Empirical label distributions for all datasets, confirming the uniform sampling protocol for calibration sets.

VLM backbones included ViT-B/16, ViT-L/14, RN50, and RN101. Multiple CP and LCP variants were evaluated using the standard metrics:

• Mean prediction set size (set efficiency)
• CovGap (mean absolute deviation from target coverage per-class)
• MCCC (min class-conditional coverage)

Results: Set Efficiency and Coverage

LCP with the proposed sigmoid localizer delivers consistent, statistically significant reductions in mean prediction set size relative to non-local CP baselines across all datasets and most conformal scores. For instance, on Aircraft (ViT-B/16, LAC) the mean set size is reduced from 18.11 to 17.43. This improvement is robust across all backbones and datasets, with paired tests confirming significance at $p < 0.01$ in most cases. By contrast, the naïve cosine localizer does not consistently yield improvements and frequently increases set size.

Classification coverage metrics behave differently. CovGap and MCCC show minimal sensitivity to localization: LCP and non-local CP variants manifest nearly identical coverage gaps and minimum class-conditional coverage, indicating that improved set efficiency via localization does not come at the expense of marginal or worst-case coverage. However, localization does not strongly reduce conditional coverage disparity, a theoretically expected outcome given the inherent limitations of CP frameworks under finite calibration.

Implications and Future Directions

This work establishes that principled nonlinear transformations of the similarity function are necessary for effective LCP in high-dimensional VLM latent spaces. The consistent empirical reduction in prediction set sizes enables more efficient uncertainty quantification and may enhance human-in-the-loop system usability in practical deployments. However, the marginal effect on CovGap suggests that further advances—potentially involving adaptive calibration or post-hoc groupwise rebalancing—are needed for strong conditional validity, particularly as VLMs become increasingly foundational in visual perception.

The open-source implementation provided extends the accessibility of LCP for vision researchers and facilitates reproducibility in both academic and industrial pipelines. Future research could focus on localized methods that leverage distribution-aware similarity metrics, domain adaptation techniques, and the integration of LCP with prompt engineering or test-time adaptation in VLMs.

Conclusion

Localized Conformal Prediction, when properly parameterized with nonlinear similarity transformations, offers statistically significant improvements in prediction set efficiency for VLM-based image classification while preserving formal coverage guarantees. The findings delineate the limitations of naïve similarity-based approaches and motivate refined algorithmic design for uncertainty quantification in vision. The public codebase will likely stimulate further community exploration of localization in conformal inference for large-scale, heterogeneous visual tasks.