Probing Routing-Conditional Calibration in Attention-Residual Transformers

Abstract: Post-hoc calibration is usually evaluated as a function of logits or softmax confidence alone, even as routing-augmented architectures increasingly accompany predictions with sample-specific internal routing traces and pair them with claims of calibration-relevant uncertainty. We ask a basic question: do these traces provide stable routing-specific evidence for post-hoc calibration beyond confidence? We study this in Attention-Residual transformers (Kimi Team, 2026) through a matched-confidence diagnostic suite that stratifies examples by routing-derived state, compares subgroup gaps against within-bin routing-permutation nulls, and evaluates matched post-hoc probes differing only in their auxiliary feature. Across our completed AR runs, scalar routing summaries do not provide stable evidence of routing-conditional miscalibration: weighted gaps remain small or seed-sensitive, and only $1$ of $30$ within-bin permutation tests rejects the conditional-null at $α=0.05$ (only on one seed; not stable across seeds in that cell). AR-CondCal, a minimal $2$-D Nadaraya--Watson probe on confidence and routing-depth variance, lies within the seed-variance band of matched confidence-only and predictive-entropy controls and does not reliably improve worst-routing-tertile ECE; bandwidth-sensitivity checks (Scott multiples, CV-NLL, global-ECE oracle) do not change this. A full-vector MLP over $(c, H_1, \ldots, H_L)$ can appear to improve over a linear confidence baseline, but the apparent gain disappears once a capacity-matched confidence-only MLP is included as a control, and shuffled routing profiles achieve comparable performance. Apparent routing-aware calibration gains in this AR setting should not be read as internal-state calibration until matched-confidence, bandwidth, capacity, and permutation controls rule out common confounds.

• The paper finds no stable calibration benefit from scalar routing summaries: only 1 of 30 permutation tests rejected at p < 0.05, and the result failed correction and replication.
• The paper’s AR-CondCal probe matched or underperformed confidence-only controls, with worst-tertile ECE of 0.0194 versus 0.0180 for confidence-only calibration and 0.0168 with predictive entropy.
• The paper shows that apparent routing-aware gains can result from model capacity and projection noise, as a capacity-matched confidence MLP outperformed the full routing-profile MLP and shuffling routing features left performance unchanged.

Motivation and central question

Post-hoc calibration methods—temperature scaling, vector scaling, histogram binning, isotonic regression, and kernel-based calibrators—are, by construction, functions of the softmax output or logits alone. This design rests on an implicit assumption: that all sample-level information relevant to miscalibration is already contained in the classifier's confidence. Routing-augmented architectures (MoE layers, adaptive computation, dynamic-token transformers) expose per-sample internal routing traces that are not part of the softmax, and a growing body of work implicitly assumes these traces carry calibration-relevant uncertainty. The paper asks whether this assumption holds in Attention-Residual (AR) transformers (Team et al., 16 Mar 2026), where each sub-layer produces learned depth-wise soft-mixture weights $\alpha_l(x)$ over prior states. The answer is a carefully controlled negative result: across the evaluated AR setting, scalar routing summaries provide no stable evidence of routing-conditional miscalibration beyond confidence.

Diagnostic protocol

The authors formalise three complementary falsification instruments. First, a matched-confidence gap analysis: examples are stratified into tertiles of a routing-derived scalar (aggregate routing entropy $r_{\mathrm{agg}}$), binned by top-class confidence within each tertile, and the low-vs-high tertile accuracy gap is compared against a within-bin routing-permutation null ($P = 5000$ shuffles). Second, worst-tertile ECE, an architecture-induced subgroup stress test in the spirit of multicalibration (Biehl et al., 2018) and verified calibration (Thulasidasan et al., 2019): equal-width 15-bin ECE computed within each of three tertiles of the routing feature $r_{\mathrm{std}}$, reporting the maximum. Third, AR-CondCal, a deliberately minimal 2-D Nadaraya–Watson probe on $(c, r_{\mathrm{std}})$ with Scott's-rule bandwidths, designed not as a state-of-the-art calibrator but as a matched-kernel diagnostic whose only difference from its controls is the second feature. The kernel-predicted correctness probability is projected onto the range achievable by per-sample temperature scaling and inverted by bisection; argmax and accuracy are preserved.

Supporting theory clarifies what any positive result would require. Proposition 1 shows the oracle gain from adding $r$ equals the conditional explained variance $\mathbb{E}[(\eta_{c,r} - \eta_c)^2]$. Proposition 2 establishes that no confidence-only calibrator can drive worst-subgroup mismatch to zero wherever subgroup conditional accuracies separate at matched confidence—formalising why the hypothesis is worth testing at all. Proposition 4 gives a stylised bandwidth-shrinkage result: a Gaussian kernel smoother attenuates a true subgroup gap by $\tanh(d^2/4h^2)$, which vanishes when the bandwidth exceeds subgroup spacing. On Block-AR, $\sigma(r_{\mathrm{std}}) \approx 0.03$ places Scott's-rule bandwidth squarely in the high-shrinkage regime, predicting exactly the small point-estimate movements observed.

Fragility of scalar routing projections

The headline diagnostic finding is starkly negative. Across 30 completed AR runs (Swin-Tiny, DeiT-Small, ViT-B/16; Block-AR and Full-AR; CIFAR-10/CIFAR-100, plus descriptive single-seed Tiny-ImageNet rows), only 1 of 30 within-bin permutation tests rejects the conditional null at $\alpha = 0.05$: Swin-Tiny + Block-AR seed-0 on CIFAR-10, with max-gap 0.346 and $r_{\mathrm{agg}}$0. That rejection does not survive Bonferroni correction ($r_{\mathrm{agg}}$1), does not replicate on the other two seeds of the same cell ($r_{\mathrm{agg}}$2 and $r_{\mathrm{agg}}$3), and sits under a global null expectation of roughly 1.5 nominal rejections. Full-AR on the same seed yields max-gap 0.052 with $r_{\mathrm{agg}}$4. Per-run max-gap spans 0.049 to 0.500 while weighted-integrated gaps span only 0.010 to 0.035, and the max-gap statistic is strongly seed-sensitive (0.346 at seed-0 vs 0.100 at seed-2 in the same cell). The implication is direct: subgroups defined by a single routing-derived scalar are not consistently separable from the permutation null, so the max-over-bins gap reads as noise from a doubly compounded projection (tertile indicator, then extreme confidence bin), not as a stable substrate-level failure mode.

A seven-candidate feature ablation reinforces this: aggregate entropy, last-layer entropy, depth-variance, concentration, entropy-weighted confidence, predictive entropy, and confidence-only produce point-estimate ECE ranges of at most ~0.007 within every substrate, with no candidate separating from confidence-only under the protocol's confidence intervals. The winning scalar changes across substrates, indicating no fixed 1-D summary captures the hypothesised signal.

AR-CondCal fails matched controls

On the main benchmark (Swin-Tiny Block-AR, CIFAR-10, three training seeds, 50/50 cal/test split), AR-CondCal achieves global ECE of $r_{\mathrm{agg}}$5, statistically indistinguishable from its matched controls Conf-only ($r_{\mathrm{agg}}$6) and Conf+PredEntropy ($r_{\mathrm{agg}}$7). On the core metric, worst-tertile ECE, AR-CondCal ($r_{\mathrm{agg}}$8) does not improve on Conf-only ($r_{\mathrm{agg}}$9) or Conf+PredEntropy ($P = 5000$0); the three matched-kernel rows sit within a narrow ~0.003 std band. Bandwidth-sensitivity sweeps over Scott multiples, CV-NLL selection, and even a deliberately optimistic global-ECE oracle change worst-tertile ECE by at most 0.0009—well inside cross-run standard deviation (~0.007)—so the failure is not attributable to the default bandwidth choice. The pattern replicates across eight 3-seed cells and seven additional full-method per-cell tables. Notably, non-parametric kernel and binning methods exhibit worst-tertile seed-std an order of magnitude wider than TS/ETS, itself a practical caution for anyone evaluating subgroup-calibration claims with such calibrators.

Capacity-controlled audit closes the false-positive route

The most methodologically consequential result concerns how routing-aware calibration claims can be falsely supported. A two-layer MLP on the full routing profile $P = 5000$1 predicting $P = 5000$2 shows a naive held-out $P = 5000$3 uplift of $P = 5000$4 over a linear confidence-only baseline on three Block-AR cells—an apparent confirmation of internal-state signal. Two controls dissolve it:

Regressor	Input	Pooled held-out $P = 5000$5
conf-lin	$P = 5000$6	$P = 5000$7
conf-mlp	$P = 5000$8	$P = 5000$9
full-lin	$r_{\mathrm{std}}$0	$r_{\mathrm{std}}$1
shuf-full-mlp	shuffled routing profile	$r_{\mathrm{std}}$2
full-mlp	$r_{\mathrm{std}}$3	$r_{\mathrm{std}}$4

The capacity-matched confidence-only MLP beats the full-vector MLP in pooled $r_{\mathrm{std}}$5 and on 9 of 10 substrate aggregates, and shuffling the routing profile within the training fold leaves held-out $r_{\mathrm{std}}$6 essentially unchanged ($r_{\mathrm{std}}$7). Consistently, the Spearman correlation between $r_{\mathrm{std}}$8 and per-sample calibration error is $r_{\mathrm{std}}$9 ($(c, r_{\mathrm{std}})$0), and the $(c, r_{\mathrm{std}})$1 distributions for correct and incorrect predictions overlap almost entirely. The apparent uplift therefore reflects non-linear modelling of confidence and increased input dimensionality, not routing-specific information. Any future claim of routing-aware calibration gains must clear capacity-matched and permutation controls of exactly this kind before being read as internal-state evidence.

Scope constraints and limitations

The authors are explicit that conclusions are bounded to AR routing under the evaluated probes; no safety, OOD, robustness, or universal-absence claim is made. Several qualifications bear directly on interpretation. The single-seed ViT-B/16 pilot exceeds a pre-registered matched-gap threshold (Block-AR 0.1255, Full-AR 0.2222) but fails the accompanying feature-ablation criterion, so it does not upgrade the claim. The bandwidth audit covers only isotropic Gaussian kernels within the NW family; anisotropic grids or objectives tuning worst-tertile ECE directly remain untested. The propositions are population-level statements, not finite-sample guarantees, and the external appendix-only runs use incompatible protocols and cannot strengthen main-paper claims. Whether calibration-relevant routing content exists in richer (higher-dimensional or non-scalar) descriptors of internal dynamics, or in non-AR routing mechanisms such as MoE or dynamic token pruning (Shazeer et al., 2017, Li et al., 2021), remains open under this diagnostic.

Conclusion

This paper delivers a disciplined negative result with reusable methodology. Its matched-confidence permutation test, worst-tertile ECE stress metric, matched-kernel probe family, and capacity/shuffle-controlled probing audit together show that scalar summaries of AR routing trajectories carry no stable calibration-relevant signal beyond softmax confidence, and that apparent counterexamples arise from projection noise and model-capacity confounds. The practical contribution is a falsification protocol that any subsequent routing-aware calibration claim should be required to pass.

---

Note: citations above reference arXiv IDs where available; the AR backbone itself is "Attention Residuals" (Team et al., 16 Mar 2026) and DynamicViT is (Li et al., 2021).