Rethinking Post-Hoc Calibration in Semantic Segmentation

Abstract: Reliable confidence estimates are essential in semantic segmentation, especially in safety-critical settings where overconfident errors can mislead downstream decisions. Yet modern segmentation models often remain miscalibrated. Post-hoc calibration offers a practical way to correct confidence estimates without retraining the segmentation model, but its use in dense prediction raises structural issues that are often overlooked. We study two such issues. First, adding a constant to all logits leaves the softmax probabilities unchanged, but several standard calibrators can still depend on this arbitrary offset. As a result, two logit representations encoding the same predictive distribution may yield different calibrated probabilities. We define translation-invariant (TI) calibrators as those whose outputs are unchanged under such shifts, characterize which common calibrators satisfy this property, and construct TI counterparts of shift-sensitive calibrators to isolate the effect of removing representation dependence. Second, post-hoc calibration is typically fitted by minimizing a likelihood-based objective, whereas segmentation models are trained with task-specific metrics such as Dice. This mismatch can cause calibration to alter class orderings and degrade the deployed segmentation map. We study decision-preserving calibration under argmax- and order-preservation constraints. Since enforcing these constraints collapses affine softmax calibrators to temperature scaling, we introduce class-conditional affine calibrators that can be made argmax- or order-preserving while retaining greater expressivity, allowing us to quantify the calibration-segmentation trade-off induced by decision preservation. Across natural-image and medical segmentation benchmarks, and under corruption-based covariate shift, matched comparisons show that TI variants generally improve calibration metrics, while decision-preserving variants prevent segmentation degradation and retain strong calibration performance. These results provide practical design principles for well-defined post-hoc calibration pipelines in semantic segmentation.

• The paper introduces key constraints—translation invariance and decision preservation—to robustly calibrate segmentation models.
• It demonstrates that enforcing translation invariance avoids arbitrary logit shifts and improves metrics such as ECE and NLL.
• The study shows that decision-preserving calibrators maintain segmentation accuracy (DSC) while enhancing reliability under distribution shifts.

Rethinking Post-Hoc Calibration in Semantic Segmentation

Introduction and Motivation

Reliable confidence estimation is essential for safety-critical semantic segmentation applications. Modern deep segmentation models, despite achieving high Dice scores (DSC), are frequently miscalibrated: their confidence scores do not reliably reflect empirical accuracy. This is problematic in downstream tasks where miscalibrated confidence can induce overconfident errors, undermine uncertainty-aware post-processing, or impair failure detection.

Post-hoc calibration pipelines attempt to improve calibration of neural networks' predictive probabilities without retraining the underlying segmentation models. Most such methods have been directly adapted from image-level classification, typically operating on network logits or softmax probabilities. However, the dense spatial setting of semantic segmentation exposes two overlooked structural issues:

1. Logit Non-Identifiability and Translation Invariance: Adding a constant to all class logits at a pixel leaves softmax probabilities unchanged, but many conventional calibrators are sensitive to such shifts. This non-identifiability introduces undesirable representation dependence in calibration, where equivalent logits can yield different calibrated confidences.
2. Segmentation Map Preservation: Standard calibration is usually likelihood-driven (e.g., via cross-entropy loss) and can alter the argmax classification at spatial locations, thus modifying the deployed segmentation. This degrades the main segmentation metric (DSC) when calibration shifts class orderings.

These challenges motivate two structural desiderata for post-hoc calibration in segmentation: translation invariance (TI) and decision preservation. Addressing these not only ensures intrinsic well-posedness of the calibration pipeline but empirically improves practical calibration outcomes.

Structural Constraints: Translation Invariance and Decision Preservation

Translation Invariance in Post-hoc Calibration:

A translation-invariant calibrator $g$ satisfies $g(\mathbf{z}) = g(\mathbf{z} + c\mathbf{1})$ for all logit vectors $\mathbf z$ and scalars $c$. TI ensures calibration outputs depend only on the predictive distribution, not on arbitrary logit representatives. Figure 1 illustrates the problems arising from non-TI calibrators and the spatial variability of logit offsets in real segmentation models.

Figure 1: Translation invariance: calibrators may vary under logit shifts even if underlying softmax predictions are identical (panel a). Right: spatial free energy variability in segmentation logits (panel b) and calibration-induced segmentation flips disturbing anatomical boundaries (panel c).

The authors rigorously characterize the invariance properties of standard calibration methods:

• TS (Temperature Scaling) and ETS (Ensemble TS): Naturally TI.
• VS (Vector Scaling) and MS (Matrix Scaling): Only TI under specific constraints (all scaling weights summed equally). Otherwise, calibration is shift-sensitive and can depend arbitrarily on the logit embedding.
• LTS (Local TS): TI if and only if the temperature predictor is shift-invariant. This is not guaranteed if the predictor accesses raw logits.
• DC (Dirichlet Calibration): A TI affine mapping applied to log-probabilities.

To enforce TI, they propose either constraining calibrator parameters (e.g., equal row sums in MS), or providing canonicalized logits (subtracting free energy to obtain log-probabilities) as input to the calibrator.

Decision Preservation and Argmax/Order-Preserving Calibration:

Calibration fitted via cross-entropy may alter the argmax, thus changing class predictions (and lowering DSC). Enforcing decision preservation—either argmax- or order-preserving mappings—can prevent such degradation, but strongly constrains the calibrator. The authors show that imposing such constraints on standard affine calibrators reduces their capacity to temperature scaling.

To regain expressivity, they propose class-conditional affine calibrators: each class has a dedicated calibrator applied in a canonicalized logit-permutation space. Constraints are enforced per-class, achieving argmax or order preservation without collapsing to scalar temperature scaling.

Experimental Design

Benchmarks:

Experiments are conducted on diverse settings:

• BraTS 2024 Glioma (3D MRI)
• Cityscapes (urban RGB image parsing)
• Massachusetts Roads (binary aerial road extraction)

Data setup includes held-out calibration, validation, and test splits. Representative examples are shown in Figure 2.

Figure 2: Input examples and overlaid ground-truth segmentations for all three benchmark datasets.

Pipeline Variants:

They implemented a wide range of pipelines, contrasting:

• Baseline approaches: uncalibrated single/ensemble models, probability and logit pooling.
• TI vs. non-TI variants: e.g., standard MS versus constrained TI MS and DC.
• Argmax/order-preserving versus unconstrained calibration (CDC vs. CMS).
• Conventional temperature-based calibrators (TS/ETS—as standard TI, order-preserving solutions).
• Local, spatially adaptive calibrators (LTS with raw logits vs. log-probabilities).

Metrics:

• Calibration: NLL, ECE, BA-ECE (boundary-aware error), ACE (averaged error).
• Segmentation: DSC; also report flip rate (fraction of altered argmax compared to baseline).

Inner Optimization:

All calibrators are optimized on held-out sets, with detailed Dirichlet-style regularization to mitigate overfitting. Calibrator expressiveness is analyzed in terms of identifiable degrees of freedom (i.e., up to the softmax gauge).

Key Results

Spatial Variability and the Need for TI:

Spatial maps of free energy confirm that additive logit offsets vary substantially across locations (Figure 3). This directly supports the need for TI calibrators in spatially structured outputs.

Figure 3: Spatial maps of pooled logit free energy in representative segmentation volumes, showing substantial spatial variability.

TI Pipelines and Calibration Performance:

TI calibrators (e.g., MS$_c$, DC, LTS with log-probabilities) consistently improve or match non-TI variants in NLL, ECE, BA-ECE, and ACE, particularly on datasets where spatial logit variability is pronounced. In matched comparisons, TI is universally preferable; translation invariance is thus established as a fundamental structural constraint for well-defined calibration.

Decision Preservation and Calibration-Segmentation Trade-off:

When unconstrained calibration is allowed (e.g., CDC), class label flips are frequent (notably, $>5\%$ of pixels in Cityscapes), eroding segmentation accuracy. Argmax- and order-preserving variants (CMS), though sometimes less expressive in binary settings, retain DSC by construction and typically match or even improve calibration (notably, no clear calibration penalty in high-class settings like Cityscapes or BraTS).

Spatial patterns of label flips (Figure 4) indicate that calibration-induced changes cluster around object boundaries, which are critical regions for both calibration and segmentation utility.

Figure 4: Label flips under non-decision-preserving affine calibration, concentrated at semantic boundaries.

Calibration Set Size and Binning Robustness:

Sensitivity analyses (Figure 5, Figure 6) demonstrate that calibration remains stable for moderate calibration set sizes ($\gtrsim 25$) and is robust to the number of bins used in ECE/ACE metrics.

Figure 5: Calibration performance is stable for calibration sets with more than 25 images.

Figure 6: Variation of ECE and ACE as confidence bin count changes, showing stable metric ranking.

Robustness under Distribution Shift:

Benchmarks on corrupted (OOD) Cityscapes images (Figure 7) confirm that TI and decision-preserving calibrators are more robust, preserving calibration and segmentation quality better under covariate shift.

Figure 7: Example test images under increasing corruption severity, showing that pixel-label correspondence is preserved.

Comprehensive Reliability Analysis:

Reliability diagrams (Figure 8) highlight that selection of a TI, sufficiently expressive, and where necessary argmax-preserving calibrator allows closing calibration gaps as measured by ECE/ACE, often beyond what conventional methods can achieve.

Figure 8: Reliability curves for test images show improved calibration using TI and decision-preserving methods.

Implications and Future Directions

Theoretical and Practical Impact:

This work establishes two core axes for post-hoc calibration design in segmentation: (1) enforcing translation invariance to avoid arbitrary dependence on non-identifiable logit degrees of freedom, and (2) allowing for decision preservation to protect deployed segmentation maps. The findings have both theoretical relevance (calibration as a well-posed mapping on distributions, not raw logits) and clear practical utility (effective calibration even under distribution shift, safety-critical reliability, and no need for retraining the underlying segmenter).

Future Directions:

Potential lines of research include:

• Extending these constraints to more flexible non-parametric or learned calibrators (e.g., neural network or kernel-based), ensuring analogues of TI and decision preservation are satisfied.
• Integrating train-time and post-hoc calibration synergistically for improved robustness to domain shift.
• Systematic benchmarking under real-world domain and covariate shifts—proven critical in this work for understanding calibration failure modes.

Conclusion

This investigation clarifies structural prerequisites for well-defined, effective post-hoc calibration in semantic segmentation. Enforcing translation invariance eliminates representation ambiguity inherited from non-identifiable logits, while decision-preserving calibration balances improved confidence estimation with unaltered segmentation accuracy. The empirical findings provide actionable guidelines: translation invariance is universally desirable; decision preservation can be imposed when downstream safety or analysis requires unwavering predictions. These insights set a principled foundation for robust calibration in dense prediction tasks, advancing the reliability and interpretability of deployed segmentation models.

---

Reference:

"Rethinking Post-Hoc Calibration in Semantic Segmentation" (2607.01902)