CalibPrompt: Med-VLM Calibration Tuning

• CalibPrompt is a calibration-aware prompt learning framework that integrates SMAC for probability matching and Angular Separation for feature distinction.
• It optimizes a small set of learnable prompt vectors while keeping the pre-trained image and text encoders frozen, ensuring parameter efficiency in few-shot settings.
• Empirical results demonstrate significant calibration improvements with reduced Expected Calibration Error across diverse medical imaging datasets without compromising accuracy.

CalibPrompt is a calibration-aware prompt learning framework for Medical Vision-LLMs (Med-VLMs) that performs calibration during prompt tuning under a scarce labeled data regime rather than as a post-hoc correction or via full-model fine-tuning. It optimizes a small set of learnable prompts while keeping the encoders frozen, and introduces two calibration objectives: Smoothed Accuracy and Confidence Matching (SMAC) in probability space, and Angular Separation (AS) in feature space. The framework is presented as the first method to calibrate Med-VLMs during prompt tuning, and is evaluated on four publicly available Med-VLMs and five medical imaging datasets, where it consistently improves calibration without drastically affecting clean accuracy (Basu et al., 18 Sep 2025).

1. Problem setting and scope

Med-VLMs leverage large-scale image-text pretraining and have shown strong performance across medical imaging tasks, but their confidence calibration is largely unexplored. In this setting, miscalibrated predictions are consequential because overconfident errors undermine clinical trust and decision-making reliability. CalibPrompt is designed specifically for prompt tuning in the few-shot medical regime, where prompt vectors are optimized while the pretrained model remains frozen, making the method parameter-efficient and compatible with low-data adaptation (Basu et al., 18 Sep 2025).

The framework targets a specific failure mode of prompt-based adaptation in medical imaging: prompt tuning can improve task fit while degrading the reliability of confidence estimates. The method therefore treats calibration as intrinsic to prompt learning rather than as a separate post-processing stage. This distinguishes it from post-hoc procedures such as temperature scaling and from full-model adaptation strategies that update substantially more parameters (Basu et al., 18 Sep 2025).

A central premise is that medical labels are often ambiguous and that prompt-tuned Med-VLMs can become overconfident when class text embeddings collapse geometrically or when predicted confidences are forced to align with hard empirical frequencies. CalibPrompt addresses both effects with complementary regularizers operating on different parts of the pipeline: output probabilities and class-wise text features (Basu et al., 18 Sep 2025).

2. Prompt-learning formulation

CalibPrompt assumes the standard Med-VLM architecture of frozen image and text encoders mapping images and class prompts into a joint feature space. Prompt learning introduces learnable prompt vectors inserted into or prepended to the text prompt, and these prompt vectors are the only trainable parameters for the downstream task. Predictions are made from image-text feature similarity and converted to probabilities with the softmax function (Basu et al., 18 Sep 2025).

The method augments the standard classification objective with two calibration regularizers. The first is SMAC, which matches average predicted confidence to a smoothed empirical class frequency rather than to a hard frequency. Its loss is

$$\mathcal{L}_{\text{SMAC}} = \frac{1}{K}\sum_{c=1}^{K}\left| \frac{1}{N}\sum_{n=1}^{N}p_n^{(c)} - \left[(1-\alpha)f_c+\frac{\alpha(1-f_c)}{K-1}\right] \right|,$$

where $K$ is the number of classes, $p_n^{(c)}$ is the predicted probability for class $c$ on input $n$, $f_c=\frac{1}{N}\sum_{n=1}^{N}\mathbb{I}[y_n=c]$ is the empirical class frequency, and $\alpha\in[0,1)$ is a smoothing hyperparameter (Basu et al., 18 Sep 2025).

The second is Angular Separation, defined as the mean off-diagonal cosine similarity of the normalized class text embeddings:

$$\mathcal{L}_{\text{AS}} = \frac{1}{K(K-1)} \sum_{i=1}^{K} \sum_{j \neq i} S_{ij},$$

where $S_{ij}$ is the cosine similarity between text embeddings for classes $i$ and $K$0. Minimizing this term encourages inter-class text embeddings to become angularly separated (Basu et al., 18 Sep 2025).

The overall training objective combines the standard cross-entropy loss $K$1 with $K$2 and $K$3, weighted by $K$4, $K$5, and $K$6. In implementation terms, prompt tuning trains only about $K$7 of the parameters, namely the prompts, which keeps the method lightweight and feasible for large Med-VLMs (Basu et al., 18 Sep 2025).

3. Calibration mechanisms

SMAC is motivated by the observation that hard accuracy-confidence alignment can enforce overly sharp boundaries in ambiguous medical-label settings. By aligning mean predicted confidence with a smoothed class frequency, the regularizer penalizes systematic mismatch between confidence and soft empirical occurrence rather than demanding direct agreement with hard outcomes. In the formulation given for CalibPrompt, this is intended to reduce overconfidence while remaining robust to the ambiguity and overlap that are common in medical imaging labels (Basu et al., 18 Sep 2025).

Angular Separation addresses a distinct geometric effect. Prompt tuning without calibration can increase cosine similarity between class text embeddings, reducing separability in the shared multimodal space. CalibPrompt therefore minimizes the average inter-class cosine similarity, which encourages class prompts to become geometrically distinct. The reported interpretation is that similar class features amplify misplaced confidence, whereas larger angular gaps improve the reliability of confidence estimates (Basu et al., 18 Sep 2025).

The combination of SMAC and AS yields a two-level calibration strategy. SMAC regularizes probability space, while AS regularizes feature space. The paper characterizes this as enforcing both probabilistic and geometric margin, leading to more reliable confidences during prompt learning. A plausible implication is that CalibPrompt treats calibration error not as a purely output-level pathology but as a joint consequence of decision statistics and prompt-induced text geometry (Basu et al., 18 Sep 2025).

This emphasis on geometry places CalibPrompt in proximity to later calibration-aware prompt-tuning research for vision-LLMs that exploits text-feature dispersion, inter-/intra-class structure, dominant dimensions, or angular diversity. However, those works focus primarily on unlabeled test-time prompt tuning for CLIP-like models under natural distribution shift, whereas CalibPrompt is formulated for few-shot medical prompt learning with labeled data (Yoon et al., 2024, Hebbalaguppe et al., 28 Jun 2025, Han et al., 10 Oct 2025, Ahamed et al., 30 Oct 2025).

4. Experimental protocol

The empirical study evaluates CalibPrompt on four Med-VLMs and five medical imaging datasets in an 8-shots-per-class regime. The experimental setup is summarized below (Basu et al., 18 Sep 2025).

Category	Items
Med-VLMs	PLIP; QuiltNet; MedCLIP; BioMedCLIP
Datasets	COVIDX; RSNA18; Kather; PanNuke; DigestPath
Metrics	ACC; ECE; ACE; MCE; $K$8

The tasks span chest X-ray, histopathology, and pancancer histopathology. Baselines include standard prompt-tuning losses such as Cross-Entropy, Focal Loss, and Label Smoothing, as well as post-hoc and train-time calibration methods including Temperature scaling, DCA, MMCE, MDCA, Margin-based LS, LogitNorm, and ZS-Norm (Basu et al., 18 Sep 2025).

All experiments are run on NVIDIA RTX A6000 GPUs. The parameter-efficiency claim is explicit: prompt tuning updates only the prompts, amounting to roughly $K$9 of the parameters. This is significant in medical imaging settings where compute budgets, annotation budgets, and deployment constraints are often tightly coupled (Basu et al., 18 Sep 2025).

5. Empirical results and ablations

The principal empirical finding is that CalibPrompt consistently improves calibration across models and datasets without significant loss of accuracy. On PLIP for histopathology, Expected Calibration Error drops from $p_n^{(c)}$0 with Cross-Entropy prompt tuning to $p_n^{(c)}$1 with CalibPrompt. On BioMedCLIP for radiology, ECE drops from $p_n^{(c)}$2 to $p_n^{(c)}$3. Across datasets, accuracy remains on par with or slightly better than baseline prompt tuning (Basu et al., 18 Sep 2025).

The paper reports that CalibPrompt improves not only ECE but also ACE, MCE, and $p_n^{(c)}$4. This matters because the gains are not confined to a single binning-based calibration metric. The method is also described as consistently effective across multiple Med-VLM architectures and imaging modalities, which suggests that the two regularizers are not tied to a single encoder family or dataset idiosyncrasy (Basu et al., 18 Sep 2025).

The ablation studies emphasize several operational properties. Calibration benefits grow with more few-shot samples. Prompt token length exhibits an optimal range, while too many prompt tokens introduce instability. Angular Separation improves calibration synergistically when combined with other prompt-regularization techniques such as PromptSmooth. The authors also note a qualitative correlation between increased cosine similarity among class text features and calibration error, consistent with the AS design rationale (Basu et al., 18 Sep 2025).

The limitations are stated narrowly. Evaluation is confined to classification, and extension to more complex medical vision-language tasks such as generation, reasoning, and retrieval is left as future work. This boundary is important because the reported calibration improvements should not be read as a general claim about all medical multimodal tasks (Basu et al., 18 Sep 2025).

6. Relation to adjacent calibration research

CalibPrompt belongs to a broader line of work that treats prompt optimization as a calibration problem rather than solely an accuracy problem. In vision-LLMs, C-TPT introduced Average Text Feature Dispersion (ATFD) as a label-free regularizer for calibration during test-time prompt tuning, showing that higher text feature dispersion correlates with lower ECE (Yoon et al., 2024). Prompting without Panic proposed attribute-aware prompt initialization and a contrastive regularization that combines intra-class contraction with inter-class dispersion, reporting an average ECE of $p_n^{(c)}$5 compared with $p_n^{(c)}$6 for vanilla TPT on fine-grained classification with CLIP-RN50 (Hebbalaguppe et al., 28 Jun 2025).

Later work further refined the geometry. D-TPT attributes miscalibration in contrastive VLMs to dominant feature dimensions and introduces dimensional entropy maximization to regularize intra-feature distributions toward uniformity, achieving the lowest ECE on fine-grained and natural distribution-shift datasets in most reported cases (Han et al., 10 Oct 2025). A-TPT instead optimizes angular diversity by maximizing the minimum pairwise angular distance between normalized class text features on the unit hypersphere, and reports lower aggregate average ECE than C-TPT and O-TPT across several settings (Ahamed et al., 30 Oct 2025).

Within medical multimodal systems, Prompt4Trust extends calibration-aware prompting to clinical multimodal LLMs using reinforcement learning to generate auxiliary prompts that penalize overconfident errors asymmetrically and improve high-confidence accuracy on PMC-VQA (Kriz et al., 12 Jul 2025). Compared with these lines, CalibPrompt is distinguished by its focus on Med-VLM prompt learning under scarce labeled data and by its explicit coupling of smoothed confidence matching with angular separation in class text embeddings (Basu et al., 18 Sep 2025).

A common misconception is that calibration in prompt-based medical models is equivalent to applying temperature scaling after training. CalibPrompt rejects that view by treating calibration as a property to be optimized during prompt learning itself. Another misconception is that calibration gains necessarily require sacrificing task accuracy; the reported results instead show substantial ECE reductions with accuracy that is on par with or slightly better than baseline prompt tuning, although the evidence is limited to the evaluated classification tasks (Basu et al., 18 Sep 2025).