Calibration Diversity in Model Reliability

• Calibration diversity is a concept linking model output reliability with heterogeneity in models, data, prompts, and calibration sets to capture the distribution of uncertainty.
• It is applied through methods like ensemble regularization, member-specific augmentation, and multilingual data calibration to reduce calibration errors such as ECE.
• The approach highlights that reliable calibration depends on balancing confidence and disagreement, underscoring both the benefits and challenges of diverse uncertainty representations.

Calibration diversity denotes a family of research ideas that connect the reliability of model outputs to some form of heterogeneity in models, data, prompts, calibration sets, or decision panels. The arXiv literature suggests that the term is not standardized: in some papers it means output-level diversity inside an ensemble to improve probability calibration, in others it means calibrating a diversity criterion itself, or using diverse calibration data, or coupling confidence with disagreement as a reliability signal (Shui et al., 2018, Jin et al., 29 Sep 2025, Chimoto et al., 26 Jan 2026, Zhao et al., 4 Aug 2025). Across these usages, the recurring claim is that calibration is rarely governed by a scalar temperature or a single confidence score alone; it is shaped by how uncertainty is distributed across hypotheses, samples, or contexts.

1. Terminological scope

The literature suggests that “calibration diversity” is best treated as a polysemous technical term rather than a single invariant construct.

Research setting	Meaning of calibration diversity	Representative paper
Deep ensembles	Explicit diversity among members improves predictive calibration	(Shui et al., 2018)
Active learning	Final selection trades local representativeness against global diversity	(Jin et al., 29 Sep 2025)
Multilingual PTQ	Diversity of calibration data, especially language composition	(Chimoto et al., 26 Jan 2026)
Qualitative coding	Confidence combined with vote-entropy diversity predicts reliability	(Zhao et al., 4 Aug 2025)
Subjective supervision	Signed diversity regularization shapes calibration to annotator distributions	(Cui et al., 9 Jul 2026)
Confidence learning	Calibration requires non-collapsed confidence spread, but this may be impossible under binary supervision	(Nair et al., 17 Sep 2025)

In ensemble-based classification, calibration diversity usually refers to deliberately inducing disagreement or decorrelation so that averaged probabilities better match empirical accuracy. In active learning and prompt tuning, the term often shifts from probability calibration to the calibration of a diversity criterion itself. In multilingual quantization, calibration diversity refers to the composition of the calibration corpus used to estimate quantization statistics. In subjective and generative settings, it becomes a question of whether disagreement is preserved, suppressed, or redistributed in a way that better matches human uncertainty or valid continuation structure.

2. Ensemble-based predictive calibration

The most direct formulation appears in “Diversity regularization in deep ensembles” (Shui et al., 2018). There, a classifier is perfectly calibrated if, for any class $k$ and confidence level $p\in[0,1]$, the empirical probability of correctness conditioned on predicting class $k$ with confidence $p$ equals $p$. The paper uses reliability diagrams, prediction-confidence histograms, accuracy, and Expected Calibration Error (ECE) as its principal diagnostics. Its empirical finding is two-sided: a single deep network is overconfident, whereas a “pure” deep ensemble often corrects that overconfidence but drifts toward underconfidence as ensemble size $M$ grows. The proposed remedy is a negative-correlation regularizer applied directly to member outputs, with per-member loss

$$E_i\big((\mathbf{x}_j,y_j);\mathrm{h}_i\big) = L\big(y_j,\mathrm{h}_i(\mathbf{x}_j)\big) + \lambda\,\mathrm{div}\big(\mathrm{h}_i(\mathbf{x}_j);\mathrm{h}_{1:M}\big),$$

where $\bar{\mathrm{h}(\mathbf{x})}$ is treated as constant with respect to $\mathrm{h}_i$ during backpropagation. On CIFAR-100 with VGG-11 and $M=7$, the pure ensemble reaches accuracy $p\in[0,1]$0 with ECE $p\in[0,1]$1, whereas the negative-correlation ensemble reaches accuracy $p\in[0,1]$2 with ECE $p\in[0,1]$3; at $p\in[0,1]$4, ECE falls from $p\in[0,1]$5 to $p\in[0,1]$6 under the same regularization. The per-class “people” superclass analysis shows the average $p\in[0,1]$7 dropping from $p\in[0,1]$8 for a single model to $p\in[0,1]$9 for the pure ensemble and $k$0 for the negative-correlation ensemble (Shui et al., 2018).

Subsequent work broadened the mechanism used to induce that diversity. “Diverse Ensembles Improve Calibration” uses member-specific augmentation distributions rather than an explicit correlation penalty (Stickland et al., 2020). The ensemble prediction remains the arithmetic mean of member predictive distributions, but each member sees a different augmentation severity. On CIFAR-10 test data, the best combined method, AugMix with Bernoulli mixing at $k$1 plus adversarial perturbations inside BatchEnsemble, reaches error $k$2, ECE $k$3, and ECE-rms $k$4; on CIFAR-10-C it reaches error $k$5, ECE $k$6, and ECE-rms $k$7. The paper also shows that mixing augmented and un-augmented inputs is important: always augmenting ($k$8) tends to hurt in-distribution calibration relative to $k$9 (Stickland et al., 2020).

“Increasing robustness and calibration in ensembles with diversity regularization” moves the diversity signal to explicitly out-of-distribution samples (Mehrtens et al., 2022). Its Sample Diversity regularizer maximizes

$p$0

where $p$1 stacks normalized logits of ensemble members on uniform-noise OOD inputs. The training loss is the usual cross-entropy on the ensemble mean minus a weighted diversity term. The main empirical pattern is strongest for parameter-sharing architectures: on corrupted CIFAR-10, BatchEnsemble improves from accuracy $p$2 and ECE $p$3 under independent training to accuracy $p$4 and ECE $p$5 with Sample Diversity; TreeNet improves from $p$6 to $p$7. The paper further reports that a 2-member TreeNet or BatchEnsemble with diversity regularization can outperform the unregularized 5-member counterpart under corruption (Mehrtens et al., 2022).

A low-data variant appears in “Something for (almost) nothing: Improving deep ensemble calibration using unlabeled data” (Pitas et al., 2023). Each ensemble member is trained on the same labeled data but receives a different random labeling of an unlabeled pool. The theoretical bound controls test ensemble NLL by average empirical NLL minus a diversity term plus a complexity term, and the expected diversity induced by random labels is $p$8. On CIFAR-10 with a ResNet22 and only $p$9 labeled samples, standard ensembles achieve ECE $p$0, NLL $p$1, and mutual information $p$2, whereas $p$3-ensembles achieve ECE $p$4, NLL $p$5, and mutual information $p$6, with nearly unchanged accuracy (Pitas et al., 2023).

3. Why diversity helps, and when it fails

A more explicit theoretical link between diversity and calibration is given in “Confidence Calibration for Convolutional Neural Networks Using Structured Dropout” (Zhang et al., 2019). Interpreting MC dropout as an ensemble $p$7, the paper uses the error–ambiguity decomposition

$p$8

and derives an expression for calibration error in which ambiguity enters with a negative sign. The practical conclusion is that higher ambiguity, hence higher diversity, reduces ECE for fixed individual accuracy. Structured dropout operationalizes this by dropping channels, blocks, or layers rather than individual activations. On CIFAR-100, ECE $p$9 falls from $M$0 for the deterministic model and $M$1 for standard dropout to $M$2 for DropChannel and $M$3 for DropLayer, compared with $M$4 for a 5-model Deep Ensemble (Zhang et al., 2019).

The converse result is that efficient ensembles that fail to realize functional diversity do not inherit ensemble-like calibration. “Is BatchEnsemble a Single Model?” reports that on CIFAR-10, a Deep Ensemble reaches NLL $M$5, ECE $M$6, and JSD $M$7, while BatchEnsemble reaches NLL $M$8, ECE $M$9, and JSD $$E_i\big((\mathbf{x}_j,y_j);\mathrm{h}_i\big) = L\big(y_j,\mathrm{h}_i(\mathbf{x}_j)\big) + \lambda\,\mathrm{div}\big(\mathrm{h}_i(\mathbf{x}_j);\mathrm{h}_{1:M}\big),$$0, close to the single model at NLL $$E_i\big((\mathbf{x}_j,y_j);\mathrm{h}_i\big) = L\big(y_j,\mathrm{h}_i(\mathbf{x}_j)\big) + \lambda\,\mathrm{div}\big(\mathrm{h}_i(\mathbf{x}_j);\mathrm{h}_{1:M}\big),$$1 and ECE $$E_i\big((\mathbf{x}_j,y_j);\mathrm{h}_i\big) = L\big(y_j,\mathrm{h}_i(\mathbf{x}_j)\big) + \lambda\,\mathrm{div}\big(\mathrm{h}_i(\mathbf{x}_j);\mathrm{h}_{1:M}\big),$$2. The MNIST study further finds near-zero disagreement and cosine similarity approximately $$E_i\big((\mathbf{x}_j,y_j);\mathrm{h}_i\big) = L\big(y_j,\mathrm{h}_i(\mathbf{x}_j)\big) + \lambda\,\mathrm{div}\big(\mathrm{h}_i(\mathbf{x}_j);\mathrm{h}_{1:M}\big),$$3 between BatchEnsemble members in parameter space, leading to the paper’s conclusion that it behaves more like a single model than a true ensemble (Zamyatin et al., 23 Jan 2026).

The same principle governs distillation. “Diversity Matters When Learning From Ensembles” argues that standard distillation transfers accuracy more readily than function diversity, and proposes Output Diversified Sampling (ODS) to generate perturbed inputs where teacher members disagree (Nam et al., 2021). For BE-4 on CIFAR-10, KD plus ODS improves NLL from $$E_i\big((\mathbf{x}_j,y_j);\mathrm{h}_i\big) = L\big(y_j,\mathrm{h}_i(\mathbf{x}_j)\big) + \lambda\,\mathrm{div}\big(\mathrm{h}_i(\mathbf{x}_j);\mathrm{h}_{1:M}\big),$$4 to $$E_i\big((\mathbf{x}_j,y_j);\mathrm{h}_i\big) = L\big(y_j,\mathrm{h}_i(\mathbf{x}_j)\big) + \lambda\,\mathrm{div}\big(\mathrm{h}_i(\mathbf{x}_j);\mathrm{h}_{1:M}\big),$$5, ECE from $$E_i\big((\mathbf{x}_j,y_j);\mathrm{h}_i\big) = L\big(y_j,\mathrm{h}_i(\mathbf{x}_j)\big) + \lambda\,\mathrm{div}\big(\mathrm{h}_i(\mathbf{x}_j);\mathrm{h}_{1:M}\big),$$6 to $$E_i\big((\mathbf{x}_j,y_j);\mathrm{h}_i\big) = L\big(y_j,\mathrm{h}_i(\mathbf{x}_j)\big) + \lambda\,\mathrm{div}\big(\mathrm{h}_i(\mathbf{x}_j);\mathrm{h}_{1:M}\big),$$7, and the Deep Ensemble Equivalent from $$E_i\big((\mathbf{x}_j,y_j);\mathrm{h}_i\big) = L\big(y_j,\mathrm{h}_i(\mathbf{x}_j)\big) + \lambda\,\mathrm{div}\big(\mathrm{h}_i(\mathbf{x}_j);\mathrm{h}_{1:M}\big),$$8 to $$E_i\big((\mathbf{x}_j,y_j);\mathrm{h}_i\big) = L\big(y_j,\mathrm{h}_i(\mathbf{x}_j)\big) + \lambda\,\mathrm{div}\big(\mathrm{h}_i(\mathbf{x}_j);\mathrm{h}_{1:M}\big),$$9. On CIFAR-100-C, BE-4 + KD + ConfODS reduces NLL from $\bar{\mathrm{h}(\mathbf{x})}$0 to $\bar{\mathrm{h}(\mathbf{x})}$1 and ECE from $\bar{\mathrm{h}(\mathbf{x})}$2 to $\bar{\mathrm{h}(\mathbf{x})}$3, nearly matching the teacher ensemble (Nam et al., 2021). This suggests that calibration gains are tied not merely to averaging, but to preserving diversity in function space.

4. Geometry, selection, and adaptive variants

Outside classical probability calibration, the term is repurposed for diversity criteria that are themselves “calibrated” against local structure. In DECERN for active fine-grained image classification, calibration diversity is the final cluster-wise acquisition score applied after uncertainty filtering and uncertainty-weighted clustering (Jin et al., 29 Sep 2025). For a sample $\bar{\mathrm{h}(\mathbf{x})}$4 in cluster $\bar{\mathrm{h}(\mathbf{x})}$5, the score is

$\bar{\mathrm{h}(\mathbf{x})}$6

so the selected sample is close to the cluster centroid but far from labeled anchors. The first term encodes local representativeness; the second encodes global diversity. The paper emphasizes that this is not probability calibration in the standard sense. It is a calibrated diversity trade-off within the selection pipeline, and ablations show that removing either the clustering or the anchor term worsens performance (Jin et al., 29 Sep 2025).

In long-tailed recognition, “Dual-Branch Temperature Scaling Calibration for Long-Tailed Recognition” treats calibration diversity as diversity in temperature parameters across classes and confidence bins (Guo et al., 2023). Class Adaptive Temperature Scaling assigns a temperature $\bar{\mathrm{h}(\mathbf{x})}$7 to each class, Equal Size Bin Temperature Scaling assigns a temperature $\bar{\mathrm{h}(\mathbf{x})}$8 to each confidence bin, and Dual-TS fuses them in a per-sample temperature. The paper also introduces Equal Sample Bin Expected Calibration Error (Esbin-ECE), which uses equal-size bins rather than equal-width bins. On CIFAR-100-LT with imbalance factor $\bar{\mathrm{h}(\mathbf{x})}$9, Dual-TS reaches accuracy $\mathrm{h}_i$0, ECE $\mathrm{h}_i$1, and Esbin-ECE $\mathrm{h}_i$2, compared with the baseline at $\mathrm{h}_i$3, $\mathrm{h}_i$4, and $\mathrm{h}_i$5, and standard TS at ECE $\mathrm{h}_i$6 (Guo et al., 2023).

“A-TPT” extends the idea to test-time prompt tuning in vision-LLMs, where the relevant diversity is angular separation of class-wise text features on the unit hypersphere (Ahamed et al., 30 Oct 2025). Defining

$\mathrm{h}_i$7

the method maximizes the average nearest-neighbor angle and adds $\mathrm{h}_i$8 as a regularizer to the standard TPT loss. On fine-grained benchmarks with CLIP ViT-B/16, average ECE rises from about $\mathrm{h}_i$9 to $M=7$0 under vanilla TPT, falls to $M=7$1 under O-TPT, and further to $M=7$2 under A-TPT, while maintaining comparable accuracy (Ahamed et al., 30 Oct 2025). Here calibration diversity is geometric: uniformly dispersed text prototypes yield better-calibrated similarities.

5. Diversity of calibration data

In multilingual post-training quantization, calibration diversity shifts from model diversity to data diversity. “Calibrating Beyond English: Language Diversity for Better Quantized Multilingual LLM” studies GPTQ and AWQ for 4-bit weight-only quantization of multilingual LLMs (Chimoto et al., 26 Jan 2026). The paper’s central claim is that static one-size-fits-all English calibration is suboptimal because different languages induce different activation ranges, Hessians, and salient channels. Calibration diversity therefore means varying the language composition of the calibration set, including multilingual mixtures and code/math augmentation.

The empirical gains are substantial for GPTQ. On Llama3.1 8B with Wikipedia evaluation, English calibration yields average perplexity $M=7$3, multi10 yields $M=7$4, multi yields $M=7$5, multimix yields $M=7$6, and codemath-multi10 yields $M=7$7. AWQ shows smaller but consistent improvements, with English at $M=7$8, multi at $M=7$9, and math-multi at $p\in[0,1]$00 (Chimoto et al., 26 Jan 2026). The paper also reports that language-matched calibration often gives the best per-language perplexity, whereas balanced multilingual calibration gives the best average performance across languages. Failure cases are traced to activation range mismatch: a monolingual calibration set can underestimate the activation tails encountered at test time, leading to clipping and degraded quantization (Chimoto et al., 26 Jan 2026).

This usage broadens the concept considerably. Calibration diversity is no longer a property of predictors; it is a property of the calibration dataset itself. The common thread is still the same: calibration quality depends on representing the relevant variability of deployment conditions rather than compressing them into a single dominant mode.

6. Subjective judgments, generative validity, and hard limits

In “Confidence-Diversity Calibration of AI Judgement Enables Reliable Qualitative Coding,” the central variables are mean self-confidence $p\in[0,1]$01 and normalized Shannon entropy $p\in[0,1]$02 of panel votes (Zhao et al., 4 Aug 2025). At the category level, confidence alone predicts agreement with

$p\in[0,1]$03

with $p\in[0,1]$04, but adding diversity yields

$p\in[0,1]$05

with $p\in[0,1]$06. The operational risk score is

$p\in[0,1]$07

Using this signal, the workflow auto-accepts $p\in[0,1]$08 of coding points with $p\in[0,1]$09 audit-detected error and estimates up to $p\in[0,1]$10 reduction in manual effort; the three-tier workflow reaches $p\in[0,1]$11 and residual error $p\in[0,1]$12 (Zhao et al., 4 Aug 2025). In this setting, calibration diversity means that disagreement entropy corrects the blind spots of self-confidence.

“Ensemble Diversity Optimization for Subjective Supervision” formalizes a closely related but distinct objective: calibration to the full annotator distribution rather than to a single label (Cui et al., 9 Jul 2026). With soft labels $p\in[0,1]$13, ensemble prediction $p\in[0,1]$14, class-weighted cross-entropy, a soft F1 surrogate, and a signed diversity regularizer $p\in[0,1]$15, the method can either preserve or suppress disagreement. EDO-Random achieves the best cross-entropy on all four evaluated datasets; on ConvAbuse, CE is $p\in[0,1]$16 compared with $p\in[0,1]$17 for Soft-CE and $p\in[0,1]$18 for WEL, while Brier scores also improve substantially (Cui et al., 9 Jul 2026). Here calibration diversity is controlled disagreement: diversity is useful when it reflects human subjectivity and harmful when it reflects structural noise.

A generative-language-model analogue appears in “Sampling More, Getting Less: Calibration is the Diversity Bottleneck in LLMs” (Banayeeanzade et al., 11 May 2026). The paper defines validity

$p\in[0,1]$19

and conditional diversity

$p\in[0,1]$20

where $p\in[0,1]$21 is the model distribution conditioned on valid outputs. It then identifies two token-level bottlenecks. Order calibration requires all valid next tokens to be ranked above invalid ones; shape calibration requires token probabilities to be proportional to the number of valid completions they permit. The reported local precision–recall frontiers show that valid and invalid tokens are interleaved in rank and that the trade-off worsens with depth, so local miscalibration compounds into sequence-level diversity collapse (Banayeeanzade et al., 11 May 2026).

The strongest negative result is “Disproving the Feasibility of Learned Confidence Calibration Under Binary Supervision” (Nair et al., 17 Sep 2025). There calibration diversity means having both well-calibrated confidences and a non-collapsed confidence spread. The paper’s theorem states that, under binary correct/incorrect supervision, no gradient-based learning algorithm can simultaneously learn well-calibrated confidence with ECE $p\in[0,1]$22 and a diverse confidence distribution with standard deviation $p\in[0,1]$23 during training. Its empirical results further report universal failures for training-based methods across MNIST, Fashion-MNIST, and CIFAR-10, while post-hoc calibration attains low ECE only by compressing the confidence distribution (Nair et al., 17 Sep 2025). This suggests that diversity can function as a calibration resource only when the supervision signal is rich enough to identify multiple confidence levels.

Overall, calibration diversity is best understood as a research program rather than a single definition. In some settings it is explicit disagreement between ensemble members; in others it is angular, linguistic, or cluster-level dispersion; in still others it is the controlled preservation of subjective disagreement or the shape of a valid continuation set. What unifies these formulations is the claim that reliability is inseparable from how a system distributes uncertainty across alternatives, and that calibration therefore depends on the structure of diversity as much as on the numerical confidence assigned to any one prediction.