Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals

Abstract: Activation oracles aim to make the activations of other models legible to humans and yield promising results compared to white-box interpretability techniques. However, uncertainty quantification (UQ) for the natural-language outputs of such activation oracles is so far understudied. Here, we investigate 6 different methods for estimating the confidence of activation oracles and evaluate how well-calibrated their confidence scores are. Our experiments on 6,000 samples per oracle (varying verbalizer and context prompts) reveal that bootstrap mode frequency is the best-calibrated method among those tested (ECE 5.7% vs. 25.5% for the answer-word log-probability on Qwen3-8B; 10.3% vs. 13.1% on Qwen3.6-27B), and that the log-prob baseline can serve as a fast triage signal at a fraction of the cost. Code and the patched trainer are available at https://github.com/federicotorrielli/probabilistic_activation_oracles.

• The paper establishes that bootstrap mode frequency yields superior calibration compared to other uncertainty quantification methods.
• It rigorously benchmarks six UQ techniques using Qwen-family activation oracles and evaluates metrics such as ECE, AUROC, and NLL.
• The work provides practical guidelines for deploying calibrated activation oracles in model auditing, emphasizing temperature tuning and controlled interventions.

Confidence and Calibration of Activation Oracles: An Expert Overview

Activation Oracles: Mechanism and Motivation

Activation oracles are LLMs fine-tuned to map the hidden states of a target model into natural-language descriptions, thereby rendering otherwise opaque neural activations legible and interpretable. The architecture applies a controlled intervention where vectors extracted from the target model's residual stream are injected at specific token positions and layers of the oracle's residual stream. Typically, this is accomplished using LoRA adapters trained on mixed-task regimes such as latent QA, binary classification, and self-supervised context prediction. The injection is norm-matched to avoid distributional shifts.

This paradigm supports applications in alignment auditing, deception detection, and elicitation of latent objectives, necessitating reliable uncertainty quantification (UQ) for actionable downstream decisions. However, calibrating confidence scores for natural-language outputs in such steered settings remains underexplored.

Figure 1: Top: activation oracle setup with hidden-state reading; bottom: ECE of six evaluated UQ methods, highlighting newly introduced approaches.

Uncertainty Quantification Methods and Evaluation Protocol

The paper benchmarks six confidence estimation methods for activation oracles in a steered setting:

• Log-probability (M1): Joint probability of answer-word tokens under steering.
• Temperature bootstrap (M2): Agreement rate across $k=20$ samples at various temperatures, analogous to short-answer self-consistency.
• Direct numeric self-report (M3): Elicited model confidence via free-form numeric response.
• MCMC acceptance (M4): Empirical acceptance ratio from Metropolis-Hastings power-sampling chains.
• MCMC agreement (M5): Mode frequency across multiple independent power-sampling chains.
• Steering-coefficient sensitivity (M6): Stability of answer decoding across grid-swept steering coefficients.

The experimental protocol applies these methods to two Qwen-family activation oracles (Qwen3-8B and Qwen3.6-27B), using the secret-word taboo task (20 LoRA adapters, 3 verbalizer prompts, 100 context prompts, yielding 6,000 samples per oracle). Calibration and discrimination are assessed using accuracy, expected calibration error (ECE), Brier score, negative log-likelihood (NLL), and AUROC.

Scorecard and Comparative Analysis

Bootstrap mode frequency emerges as the most reliable calibration method across both models. Optimal sampling temperature for bootstrap aligns with overall task accuracy: $T=1.0$ for Qwen3-8B (accuracy $\sim0.41$) yields ECE $5.7\%$, while $T=1.5$ for Qwen3.6-27B (accuracy $\sim0.22$) yields ECE $10.3\%$. Log-probability serves as a low-cost triage signal with competitive AUROC but notably higher native ECE.

Direct numeric self-report is consistently anti-calibrated; it exhibits nearly identical confidence for correct and incorrect outputs and, on 27B, marks higher confidence for wrong answers. Raw MCMC acceptance similarly fails to capture correctness, as mode-peaked steered distributions accept proposals indiscriminately.

Figure 2: Calibration versus ranking trade-off for Qwen3-8B and Qwen3.6-27B across UQ methods; Pareto front dominated by bootstrap and log-prob.

Figure 3: Reliability diagram for Qwen3-8B; bootstrap tracks the diagonal, log-prob underconfident, self-report and MCMC pinned to overconfident regime.

Figure 4: Heatmap of method rankings per metric for both models; bootstrap variants superior, direct self-report and MCMC acceptance inferior.

Mechanistic Insights and Calibration Dynamics

Calibration of bootstrap mode frequency is tied to empirical accuracy distribution: the mean mode frequency over samples is a consistent estimator when set to match per-item accuracy. On harder tasks, optimal temperature for calibration migrates toward flatter distributions. Power-sampling methods do not enhance signal—acceptance ratio saturates in mode-peaked settings. Multi-chain agreement only marginally improves AUROC at significant computational cost; ECE remains suboptimal.

Direct numeric self-report fails primarily due to the dominance of language-modeling priors in free-form elicitation, replicating probe-verbalizer gaps documented in recent work (Yuan et al., 10 May 2026, Miao et al., 26 Mar 2026). Introspective UQ is recoverable via constrained label scoring (AUROC $0.957$ in pilot), demonstrating the necessity of bypassing the free-text decoding channel.

Figure 5: Confidence separation for correct/wrong predictions; direct numeric self-report is anti-calibrated on Qwen3.6-27B (inverted markers).

Per-Word and Layer Scaling Dynamics

Accuracy breakdowns by target word highlight substantial variability (order-of-magnitude spread). Wrong answers on 8B cluster as topical neighbors; 27B frequently refuses or emits non-content tokens, resulting in higher AUROC and better calibration despite lower accuracy. Controlled target-set scaling experiments reveal bootstrap calibration improves with larger vocabularies only on 8B.

Layer sweep across four oracle bases (Qwen3-8B, Qwen3.6-27B, Gemma-2-9B, Gemma-3-27B) demonstrates probe-readable bands vary widely with architecture. The trained layers sparsely intersect the optimal probe regions.

Figure 6: Per-layer log-prob accuracy sweep; probe-readable regions vary by architecture, with Gemma-3-27B exhibiting the broadest and most accurate band.

Practical Recommendations and Implications

Strong claims substantiated:

• Bootstrap mode frequency offers superior native calibration.
• MCMC acceptance is unsuitable for confidence; explanation is mechanistic mode-peakedness.
• Direct numeric self-report is anti-calibrated; constrained label scoring recovers signal.

For practical deployment:

• Adopt bootstrap mode frequency for confidence assessment, tuning $T$ so mean mode frequency matches empirical accuracy.
• Use log-prob as a triage when cost constraints preclude sampling.
• Avoid direct numeric self-report and raw MCMC acceptance for production confidence estimation.

Theoretically, the results reinforce the diagnostic—but not causal—nature of internal confidence representations (Yuan et al., 10 May 2026, Miao et al., 26 Mar 2026), echoing boundaries observed in mechanistic interpretability (diagnostic error signaling vs. actionable correction) (Basu et al., 18 Mar 2026).

Limitations and Future Directions

Evaluation covers only Qwen-family oracles with a fixed training recipe. Method ranking is invariant to answer-extraction pipeline, but absolute accuracy may shift under embedding-based soft metrics (BERTScore-style similarity). Extending systematic calibration to broader architectures, more nuanced metrics, and larger, real-world vocabularies is a natural progression.

Post-hoc calibration (Platt, isotonic, beta scaling) closes the ECE gap between methods when validation labels are available, reducing practical distinction to computational efficiency.

Conclusion

This work presents a comprehensive benchmark and mechanistic analysis of uncertainty quantification for activation oracles. Temperature-bootstrap mode frequency is the unequivocal best-calibrated method, with calibration optimality tracking task accuracy. MCMC power-sampling acceptance and direct numeric self-report fail to deliver useful confidence signals; calibration can be rescued by constrained label scoring. These findings underpin reliable calibration for activation-oracle-based interpretability pipelines and provide actionable guidance for model auditing and monitoring practices.