LACE: Language-Aware Confidence Ensemble

• LACE is a training-free ensemble method that averages intermediate layer probabilities to significantly improve confidence calibration in multilingual LLMs.
• It selects optimal layers per language based on validation calibration metrics, addressing the English-centric bias of final-layer confidence.
• Empirical results on MMMLU and Belebele benchmarks show LACE dramatically reduces ECE and narrows confidence gaps for non-English inputs.

Language-Aware Confidence Ensemble (LACE) is a training-free, language-specific ensemble method for multilingual confidence calibration in LLMs. It was introduced in the context of the finding that, in multilingual settings, the final decoding layer is often not the best source of confidence for non-English inputs. Instead, late-intermediate layers frequently provide a better-calibrated signal, and the optimal set of such layers varies by language. LACE operationalizes this observation by selecting, for each language, the layers whose calibration is better than that language’s final-layer calibration, averaging their predictive distributions, and optionally applying language-specific post-hoc calibration (Zhou et al., 3 Oct 2025).

1. Multilingual calibration as the problem setting

LACE addresses multilingual confidence calibration: the alignment between predicted confidence and empirical correctness probability. The motivating empirical result is that non-English languages are systematically less well calibrated than English across six model families—LLaMA3, Qwen2.5, Mistral, Aya, DeepSeek, and Phi—over more than 100 languages, using the multilingual multiple-choice benchmarks MMMLU and Belebele. On MMMLU for LLaMA3, English has ECE $= 4.61\%$, whereas the average non-English ECE is $23.12\%$; Aya shows the same direction, with English ECE $= 20.66\%$ and average non-English ECE $= 26.77\%$ (Zhou et al., 3 Oct 2025).

The associated disparities are not limited to raw calibration error. In LLaMA3, $78.8\%$ of non-English correct predictions are underconfident with confidence $< 50\%$, versus $25.7\%$ for English. The average confidence gap between correct and incorrect predictions is $23.8\%$ in English but only $6.3\%$ in non-English languages. On Belebele, calibration also correlates with language resource availability for LLaMA3, with Spearman $\rho = -0.59$, Kendall $23.12\%$0, and Pearson $23.12\%$1 (Zhou et al., 3 Oct 2025).

Within this framing, LACE is one response to a broader diagnosis: current alignment and trustworthiness pipelines are English-centric. A plausible implication is that multilingual confidence estimation cannot be treated as a simple extension of English calibration, because both the quality of the confidence signal and its location in the network vary by language.

2. Why LACE looks beyond the final layer

The central technical premise of LACE is that final-layer confidence is often inadequate in multilingual settings. The layerwise analysis applies the model’s original language modeling head $23.12\%$2 to every layer hidden state $23.12\%$3, producing

$23.12\%$4

and then defines confidence at layer $23.12\%$5 for input $23.12\%$6 as the probability that layer $23.12\%$7 assigns to the final-layer predicted token $23.12\%$8: $23.12\%$9 The correctness target is always whether the final prediction is correct; only the confidence source changes (Zhou et al., 3 Oct 2025).

The resulting depth profiles differ sharply by language. For English, calibration generally improves monotonically with depth and is best at the final layer. For non-English languages, calibration often improves in late-intermediate layers and then degrades again near the final layer. The paper localizes the main “sweet spot” to layers $= 20.66\%$0 for LLaMA3, with best average around layer 29, and layers $= 20.66\%$1 for Aya, with best average around layer 28. Choosing layer 29 gives average AECE of $= 20.66\%$2 for LLaMA3; choosing layer 28 gives $= 20.66\%$3 for Aya (Zhou et al., 3 Oct 2025).

The per-language gains are large. For LLaMA3, German ECE drops from $= 20.66\%$4 to $= 20.66\%$5 at layer 29; Hindi from $= 20.66\%$6 to $= 20.66\%$7; Japanese from $= 20.66\%$8 to $= 20.66\%$9; Korean from $= 26.77\%$0 to $= 26.77\%$1; and Chinese from $= 26.77\%$2 to $= 26.77\%$3. English is the clearest counterexample: its ECE worsens from $= 26.77\%$4 at the final layer to $= 26.77\%$5 at layer 29. This asymmetry is the immediate motivation for a language-aware method rather than a single globally fixed intermediate layer (Zhou et al., 3 Oct 2025).

3. Formal definition and mechanism

LACE is a language-specific layer-ensemble confidence estimator built from intermediate-layer predictive distributions. It uses intermediate hidden states $= 26.77\%$6, the model’s original language modeling head $= 26.77\%$7, per-layer probability distributions $= 26.77\%$8, language identity $= 26.77\%$9, and per-language layerwise calibration statistics computed on a validation set (Zhou et al., 3 Oct 2025).

For language $78.8\%$0, LACE defines the set of “good” layers as those whose ECE is better than the final layer’s ECE for that same language: $78.8\%$1 It then forms a uniform average over the predictive distributions of those selected layers: $78.8\%$2 The paper does not describe learned layer weights; selection is discrete and weighting is equal across selected layers (Zhou et al., 3 Oct 2025).

An optional second stage applies language-specific post-hoc calibration: $78.8\%$3 In the multiple-choice setting, confidence is then read out in the standard form

$78.8\%$4

The method is training-free in the sense that no model parameters are updated and no additional neural module is trained. It reuses the model’s existing hidden states and output head, and only computes layerwise probabilities plus layer selection from validation-set calibration statistics. Language awareness enters through explicit access to language identity and through the per-language construction of $78.8\%$5 (Zhou et al., 3 Oct 2025).

4. Evaluation protocol and empirical performance

The reported LACE experiments use multilingual MCQA data with held-out validation and separate test sets. MMMLU contains $78.8\%$6K examples and Belebele $78.8\%$7K examples; the appendix reports $78.8\%$8k validation examples from MMMLU and $78.8\%$9k from Belebele for post-hoc calibration. All experiments use eight-shot prompting in the respective language. The main comparison set is final layer baseline, best layer, good layers ensemble, and LACE, each optionally combined with Temperature Scaling or Isotonic Regression. Metrics include ECE, Brier score, AUROC, and accuracy, with macro-averaged results across languages (Zhou et al., 3 Oct 2025).

Benchmark / model	Final layer ECE	LACE ECE	LACE + IR ECE
MMMLU / LLaMA3	22.44	5.96	3.09
MMMLU / Aya	24.39	11.42	3.45
Belebele / LLaMA3	17.68	7.05	5.79
Belebele / Aya	15.73	10.22	4.80

These results show a consistent progression from final layer, to a single best intermediate layer, to a global good-layers ensemble, to the language-aware good-layers ensemble. On MMMLU for LLaMA3, the sequence is $< 50\%$0. On MMMLU for Aya, it is $< 50\%$1. The table headings keep accuracy fixed—for example, MMMLU LLaMA3 is reported with accuracy $< 50\%$2 and Belebele LLaMA3 with $< 50\%$3—which indicates that these methods change confidence estimation rather than task predictions (Zhou et al., 3 Oct 2025).

Post-hoc calibration remains complementary rather than redundant. On MMMLU, LLaMA3 final layer plus Temperature Scaling worsens from $< 50\%$4 to $< 50\%$5, while final layer plus Isotonic Regression improves to $< 50\%$6; both remain much worse than LACE at $< 50\%$7. By contrast, combining LACE with post-hoc calibration gives the best overall numbers, including $< 50\%$8 for LLaMA3 LACE + TS and $< 50\%$9 for LLaMA3 LACE + IR. The paper explicitly interprets this as evidence that LACE reshapes the confidence signal rather than merely rescales it (Zhou et al., 3 Oct 2025).

5. Relation to earlier multilingual and ensemble confidence research

LACE belongs to a broader development in which multilingual confidence estimation moved from monolingual baselines toward explicitly language-conditioned aggregation. A direct precursor is MlingConf, which studied multilingual confidence estimation on translated factual QA and found that confidence quality varies substantially by language, that self-verbalized numerical confidence is the strongest monolingual signal, and that averaging confidence across semantically equivalent multilingual inputs improves AUROC beyond monolingual English, temperature sampling, and prompt-based multi-output baselines. In GPT-4, English Verbal Number confidence is $25.7\%$0, while the cross-lingual configuration “en fr de it id” reaches $25.7\%$1 (Xue et al., 2024).

A plausible connection is that MlingConf demonstrates language-dependent confidence quality at the input and prompting level, whereas LACE relocates the same multilingual calibration problem into the model’s internal depth structure. Both lines of work support the claim that a single English-default confidence signal is inadequate. MlingConf also reports that more languages are not always better and that shared-family aggregation can be especially effective, which suggests a family resemblance to LACE’s per-language layer selection rather than global pooling (Xue et al., 2024).

Related ensemble methods also illuminate what LACE is not. The lexical-variant ensemble for classification in “Ensemble based approach to quantifying uncertainty of LLM based classifications” aggregates predictions across $25.7\%$2 rephrasings of the same latent intent and uses majority-vote frequency as a confidence statistic; it is language-aware in the sense of linguistic perturbation, but it ensembles prompts rather than layers (Rajamohan et al., 12 Feb 2025). LENS learns per-model confidence predictors from layerwise neural states for multi-LLM answer integration and then selects the answer from the most confident model, which is closer to instance-level model routing than to LACE’s training-free, intra-model, per-language layer selection (Guo, 31 Jul 2025). ORCE, by contrast, treats confidence as a decoupled verbalized-generation problem and optimizes relative ordering of confidence across responses, which is particularly relevant when logits are unavailable; this suggests a distinct family of confidence-elicitation methods rather than an alternative to LACE’s layerwise readout (Li et al., 12 May 2026).

6. Scope, limitations, and interpretation

LACE is strongest precisely where final-layer multilingual calibration is weakest. Its clearest failure case is English: when final-layer confidence is already strong, moving to intermediate layers can hurt, as in the LLaMA3 example where English ECE rises from $25.7\%$3 to $25.7\%$4 at layer 29. This is why the method is explicitly language-specific rather than globally fixed (Zhou et al., 3 Oct 2025).

The reported scope is also narrow in identifiable ways. The experiments are on mid-scale $25.7\%$5B–$25.7\%$6B models, restricted to MCQA tasks, and the paper states that it is unclear whether the same gains transfer directly to open-ended generation. LACE requires access to all hidden states and the original language modeling head, which limits applicability to systems that expose intermediate representations. Relative to final-layer confidence, it also requires reading probabilities from multiple layers, storing or processing layerwise outputs, and maintaining per-language selected layer sets, although the method is still described as modular and low-overhead because it avoids retraining (Zhou et al., 3 Oct 2025).

The broader significance claimed for LACE is diagnostic as much as methodological. The method supports the thesis that English-centric alignment leaves non-English trustworthiness behind, and that the final layer is especially affected by this asymmetry. In that interpretation, LACE is not merely a calibration heuristic. It is evidence that multilingual reliability signals persist inside the network but are partially obscured at the final layer, and that language-aware confidence estimation can recover them by looking beyond the final decoding layer (Zhou et al., 3 Oct 2025).