Hidden Failures in Robustness: Why Supervised Uncertainty Quantification Needs Better Evaluation
Abstract: Recent work has shown that the hidden states of LLMs contain signals useful for uncertainty estimation and hallucination detection, motivating a growing interest in efficient probe-based approaches. Yet it remains unclear how robust existing methods are, and which probe designs provide uncertainty estimates that are reliable under distribution shift. We present a systematic study of supervised uncertainty probes across models, tasks, and OOD settings, training over 2,000 probes while varying the representation layer, feature type, and token aggregation strategy. Our evaluation highlights poor robustness in current methods, particularly in the case of long-form generations. We also find that probe robustness is driven less by architecture and more by the probe inputs. Middle-layer representations generalise more reliably than final-layer hidden states, and aggregating across response tokens is consistently more robust than relying on single-token features. These differences are often largely invisible in-distribution but become more important under distribution shift. Informed by our evaluation, we explore a simple hybrid back-off strategy for improving robustness, arguing that better evaluation is a prerequisite for building more robust probes.
Summary
- The paper demonstrates that supervised UQ probes, trained on in-distribution data, significantly drop to near-chance performance under OOD conditions, particularly on long-form tasks.
- The study finds that specific probe configurations—using middle-layer hidden states and aggregated response tokens—are crucial for better OOD robustness.
- The work introduces hybrid back-off strategies that improve short-form UQ performance while highlighting persistent challenges in evaluating long-form generative outputs.
Systematic Analysis of Uncertainty Quantification Probe Robustness in LLMs
Motivation and Context
Supervised uncertainty quantification (UQ) probes leveraging LLM internal states (e.g., hidden states, attention maps) have seen increasing adoption for hallucination and reliability prediction. Prior works claim robust generalization, but there has been limited consensus on how probe design choices (representation layer, feature type, token aggregation) affect OOD robustness. The paper "Hidden Failures in Robustness: Why Supervised Uncertainty Quantification Needs Better Evaluation" (2604.11662) delivers a thorough empirical study—over 2,000 probes across models (Llama-3.1, Gemma-2), tasks, and OOD regimes—to identify intrinsic robustness shortcomings and chart the path to practical UQ evaluation.
Empirical Evaluation: Hidden Failures in Robustness
The primary finding is a systematic deterioration in probe robustness outside the training distribution, especially for long-form generative tasks.
Figure 1: Uncertainty Quantification methods quickly lose robustness outside of their training distribution, especially for long-form generations. These failures are often not visible in existing OOD evaluation.
Probes trained on in-distribution data exhibit strong discrimination of output correctness, but their performance degrades to near-chance OOD, with the effect most pronounced on long-form outputs. Robustness claims in previous works often stem from restricted or synthetic evaluation settings, masking these failures.
Layer Representation and Feature Selection
The paper rigorously disentangles probe configuration variables:
- Layer Selection: Probes trained on middle-layer hidden states consistently generalize better OOD than those trained on final-layer or early-layer states, despite in-distribution equivalence.
- Feature Type: Hidden-state probes (especially mid-layer) outperform attention-based and logit-based probes OOD. Attention-derived features, mean pooling, and ratio-of-context attention offer less robust signals.
- Token Aggregation Strategy: Averaging over response tokens (rather than using last-token or context tokens) yields superior generalization for correctness prediction. Aggregating response token features helps overcome the noisiness induced by distribution shift.
Figure 2: PLS regression visualization on TriviaQA shows that as OOD-ness increases, hidden states of more correct (blue) and less correct (red) generations become less separable.
Figure 3: PRR by layer for Llama and Gemma, highlighting robust performance for middle-layer probes under OOD settings; final-layer representations drop sharply OOD.
Token-Level Structure and OOD Collapse
Detailed representation visualizations further corroborate that ID correctness signals are linearly separable in probe space, but this separation collapses as the shift widens (leave-one-out, cross-task, etc.). The same pattern appears across datasets, including SciQ (Figure 4) and PubmedQA (Figure 5). Correct and incorrect generations become entangled in representation space, undermining probe reliability.
Figure 4: Hidden states on SciQ: OOD shift erodes separation between correct and incorrect generations.
Figure 5: Hidden states on PubmedQA: OOD shift causes even more drastic collapse, particularly for long-form answers.
Probe Architecture vs. Input Choices
Architectural advances (MLP depth, transformer heads) have only marginal impact on robustness. The dominant factor is the input representation configuration, not the probe's internal complexity. Even sophisticated hybrid probe designs fail to recover generalization unless the underlying features are appropriately chosen.
Hybrid and Adaptive UQ Strategies
Inspired by the poor OOD robustness, the paper explores hybrid back-off (HBO) strategies mixing supervised probe predictions and unsupervised (maximum sequence probability, MSP) scores, dynamically weighted by Mahalanobis distance from the training data. HBO significantly improves OOD PRR for short-form generations—outperforming prior hybrids—by defaulting to logit-based uncertainty as examples move further OOD. However, this does not solve the robustness challenge for long-form tasks, where all approaches remain close to chance level.
Evaluation Best Practices and Recommendations
The study exposes several methodological gaps in prior literature:
- Robustness must be evaluated across both near and far OOD shifts (leave-one-out and true task transfer), with consistency across test sets.
- Strong baselines (middle-layer MLP, logit-based MSP) must be included.
- Claims supported only by synthetic or narrow evaluation are often invalid OOD.
- Short-form generalization can be improved with hybrid strategies; long-form remains an open challenge.
The authors formalize recommendations for future experimentation: robust baseline inclusion, evaluation breadth, and negative robustness results disclosure are necessary to avoid masked limitations.
Implications and Future Directions
Practically, the results advise caution in deploying probe-based UQ for LLM reliability assurance—supervised probes trained on final-layer or single-token features can fail silently OOD. Selection of probe input features (middle-layer hidden states, aggregated response tokens) is paramount. Theoretically, the work motivates research into more invariant uncertainty signals and distribution-agnostic probe mechanisms, as well as improved correctness functions for long-form outputs.
There is also a strong implication for AI safety workflows: uncertainty estimation in generative models is nontrivial and cannot be trivially extrapolated across tasks or domain shifts. Future development of robust, scalable, and interpretable UQ methods must prioritize comprehensive dataset coverage and more advanced hybridization or latent structure modeling.
Conclusion
This work systematically demonstrates the hidden non-robustness of supervised UQ probes in LLMs, especially for long-form generative tasks. Robustness hinges on probe input representation choices rather than probe architecture complexity. Hybrid strategies can improve short-form robustness OOD, but fundamental failure persists in long-form cases. Evaluation practices must evolve to expose these limitations and guide the development of more resilient uncertainty quantification techniques in LLMs (2604.11662).
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Explain it Like I'm 14
Overview: What this paper is about
This paper asks a simple question: can we tell when an AI is unsure, before it gives a wrong or made‑up answer? The authors study “uncertainty probes,” which are small tools trained to read a LLM’s “internal signals” (like its hidden thoughts) and predict whether the model’s answer will be reliable. They test how well these tools work not just on familiar problems, but also on new, different ones. Their main message: many current methods look good on familiar data but quietly fail on new situations—especially for long, paragraph‑style answers. They also show simple ways to make these methods more reliable and offer better ways to test them.
Key questions the paper explores
- Can we train small “probes” to read a model’s internal signals and tell when its answer might be wrong?
- Do these probes still work when the model faces new kinds of questions or topics it wasn’t trained on (called “out‑of‑distribution,” or OOD)?
- Which design choices matter most for building reliable probes (e.g., which internal layer to read, what signals to use, how to combine info from words)?
- Can we combine different uncertainty signals to make a stronger, more dependable method?
How the researchers studied it (in everyday terms)
Think of a LLM as a student that writes answers and keeps lots of notes to itself while thinking. A “probe” is like a small grader that looks at those notes to guess whether the student’s answer will be right or wrong.
What they varied and tested:
- Where to “listen” inside the model: They tried reading different internal layers—like listening to thoughts in the middle of the process vs. at the very end.
- What kind of signals to use:
- Hidden states: the model’s internal notes about meaning.
- Attention: where the model “looked” while answering (which words it focused on).
- Token probabilities: how confident the model sounds based on its own math.
- Density-based features: how “unusual” a new question looks compared to training examples (think: does this look familiar or weird?).
- How to combine word information:
- Only the last word of the answer, or
- Average across all the words in the answer (like judging the whole sentence instead of just the final word).
What they tested on:
- Short answers (like a single word or sentence) and long answers (multiple sentences or summaries).
- Both familiar topics (in‑distribution) and new topics or tasks (out‑of‑distribution). They looked at “near” shifts (slightly different) and “far” shifts (very different).
How they measured success:
- They used an automatic “correctness scorer” that checks how well the AI’s outputs match facts.
- They used a score called PRR (Prediction–Rejection Ratio), which asks: if you sort answers by “most suspicious” to “least suspicious” and discard the suspicious ones, does the average quality of what remains improve a lot? A higher PRR means the uncertainty measure is doing a good job at finding risky answers.
Main findings and why they matter
Here are the most important takeaways, explained simply:
- Probes look good on familiar data but often fail on new data.
- When models face new topics or tasks, many probes drop to chance‑level performance—especially for long answers like summaries or explanations.
- This means that methods that seem strong in lab tests may not help much in the real world.
- The input signals matter more than the probe’s design.
- Middle‑layer signals are more reliable than final‑layer signals when the topic changes.
- Analogy: the model’s mid‑process thoughts generalize better than its last‑second thoughts.
- Averaging across all answer tokens (all words in the reply) is more robust than looking at just one last word.
- Analogy: judge the whole paragraph, not just the final sentence.
- Probe architecture (how fancy the probe is) matters less than what you feed it.
- More complex probes didn’t fix the robustness problem by themselves.
- Choosing better inputs (which layer, which signals, how to combine words) matters more.
- Some earlier “robustness” claims were based on limited tests.
- Different papers used different test setups, so results looked better than they actually are when you try broader, harder tests.
- The authors argue for clearer, more consistent evaluations across both near and far distribution shifts.
- A simple “hybrid” method helps on short answers.
- They propose a Hybrid Back‑Off (HBO) strategy: if a question looks familiar, trust the trained probe; if it looks unusual, lean more on a simple, unsupervised confidence measure (like how confident the model sounds from its probabilities).
- This hybrid approach improved reliability for short answers across different shifts.
- It did not yet solve the long‑answer problem.
Why this matters:
- In practice, we want AI that knows when it might be wrong and can flag risky answers. If methods don’t hold up on new topics, people could be misled. These findings push the field toward more realistic testing and better design choices.
What this could change going forward
- Better testing standards: The authors recommend evaluating both “near” and “far” new situations, using both short and long answers, and including strong simple baselines.
- Smarter probe design: Use middle‑layer signals and combine information across the whole answer, rather than focusing on the last token or the final layer.
- Practical systems: For short answers, a hybrid approach (mixing trained probes with simple confidence measures) can make systems more trustworthy. Long‑answer reliability remains a big open challenge.
- Research focus: Instead of making probes more complicated, focus on better inputs and more careful evaluation. That’s likely the fastest path to dependable systems.
In one sentence
The paper shows that many tools for telling when an AI is unsure work fine on familiar problems but break on new ones—especially for long answers—and that choosing the right internal signals (middle layers, whole‑answer info) and using smart hybrids can make uncertainty estimation more reliable, as long as we test it properly.
Knowledge Gaps
Knowledge gaps, limitations, and open questions
Below is a focused list of what remains missing, uncertain, or unexplored in the paper, phrased to be directly actionable for future research.
- Long-form robustness remains unsolved: Why do all supervised probes degrade to near chance under long-form OOD settings, and what design choices (e.g., hierarchical aggregation, claim-level reasoning, span-level verification) could restore robustness?
- Correctness function sensitivity: How sensitive are the conclusions to the choice of correctness metric (AlignScore vs. LLM-as-judge vs. human annotation), especially for long-form tasks where metric biases (e.g., length sensitivity) are pronounced?
- Human evaluation coverage: Are the reported OOD robustness failures replicated under human-annotated correctness labels at scale, and how do metric-induced errors translate to real human judgments?
- Probe training data size and diversity: How does OOD robustness vary with training set size, domain diversity, and multi-task training (beyond the constant-size setting used), and can targeted data augmentation improve generalisation?
- Domain shift typology: Which specific shift types (topic, style, temporal drift, adversarial prompts, instruction format changes) most harm probe robustness, and can shift-specific mitigations be designed?
- Real-world shifts: Do results hold under naturally occurring distribution shifts (e.g., time-evolving corpora, deployment logs), rather than curated academic datasets?
- Cross-model transfer: Can a probe trained on one base model (or family) transfer to another (e.g., Llama → Gemma), and what representation alignment or domain adaptation techniques would enable cross-model generalisation?
- Larger models and instruction tuning: Do robustness patterns change for larger or differently instruction-tuned LLMs, and how do alignment/RLHF regimes affect uncertainty signals in middle vs. final layers?
- Feature space beyond hidden states/attention: Are there more robust internal features (e.g., KV cache dynamics, MLP activations per head, gradients/logit-lens features, neuron-level sparsity patterns) that improve OOD reliability?
- Layer choice generality: Is the “middle-layer is best” result stable across architectures, scales, and tasks, and can an adaptive per-instance layer selection or learned layer fusion improve robustness?
- Token aggregation beyond averaging: Can learned token-weighting, hierarchical pooling, or rationale/claim-aware aggregation outperform simple mean pooling for long-form sequences?
- Architecture vs. inputs: Given that architecture complexity had limited impact, can representation learning strategies (e.g., domain-adversarial training, invariance penalties, contrastive or IRM-style objectives) extract shift-stable uncertainty features?
- Hybrid back-off design: The HBO weighting is heuristic and MD-based; can learned gating, calibration-aware mixtures, or alternative OOD scores (energy, kNN density, ensembles, temperature scaling) yield stronger and more general hybrids?
- Mahalanobis distance reliability: How sensitive is the OOD ranking to choice of layer, covariance estimation, dimensionality reduction, and sample size, and does MD robustly track OOD-ness across tasks and models?
- Uncertainty dimensions: To what extent do probes capture epistemic vs. aleatoric uncertainty in generative tasks, and can disentangling these components improve OOD behaviour?
- Decoding parameters: How do temperature, top-p, and sampling strategies affect both unsupervised baselines (MSP) and probe-based UQ, especially under OOD conditions and for long-form outputs?
- Calibration and decision utility: Beyond PRR (ranking-focused), how do methods fare on calibration metrics (e.g., ECE, risk-coverage), abstention policies, and downstream decision utility under OOD?
- Subpopulation robustness: Which input attributes (length, entity density, compositionality, novelty, domain expertise) drive failures, and can per-attribute robustness auditing inform probe design?
- Multilingual and code tasks: Do findings generalise to multilingual settings and non-natural language domains (e.g., code generation), where uncertainty signals and correctness functions differ?
- Multi-sample uncertainty: How do self-consistency, multi-sample disagreement, or ensemble LLMs compare or combine with probe-based UQ for OOD robustness, especially on long-form tasks?
- Training signal quality: Since supervision relies on automated correctness scoring, how do label noise and bias impact probe learning, and can noise-robust training (e.g., bootstrapping, label smoothing) help?
- Cost and practicality: What are the compute/latency trade-offs for deploying robust probes at scale, and can lightweight alternatives (e.g., cached features, low-rank adapters) maintain robustness?
- Benchmark standardisation: The paper calls for better evaluation but does not release a standardised, plug-and-play OOD suite. Can a fixed, community benchmark (datasets, shifts, baselines, metrics) be established to ensure comparability?
- Long-form hybrid methods: HBO is effective for short-form but not long-form. What hybrid architectures or correctness-aware mixtures can deliver reliable long-form UQ under distribution shift?
- Causal analysis of representation drift: What mechanisms cause the “uncertainty subspace” to rotate across distributions, and can causal or mechanistic interpretability guide invariant feature extraction?
Practical Applications
Immediate Applications
Below are actionable, sector-linked uses that can be deployed now by adopting the paper’s empirically supported design choices (middle-layer features, response-token aggregation) and evaluation practices, and by leveraging the hybrid back-off (HBO) strategy for short-form generations.
- Robust uncertainty gating for short-form LLM outputs using Hybrid Back-Off (HBO)
- Sectors: customer support, software/dev tools (code completion, code review), search/Q&A, fintech chatbots, e-commerce assistants.
- Workflow/product:
- Train a simple MLP probe (SAPLMA-style) on middle-layer hidden states with mean aggregation over response tokens using in-domain short-form data.
- Compute Maximum Sequence Probability (MSP) per output.
- Estimate how OOD an example is using Mahalanobis distance ranks over mid-layer features.
- Combine supervised probe and MSP with the paper’s back-off weighting to produce a robust confidence score.
- Gate actions based on confidence: answer/abstain, switch to retrieval, escalate to human, or trigger additional verification.
- Assumptions/dependencies:
- Access to middle-layer hidden states (not always available in closed APIs).
- Availability of labeled correctness signal for short-form tasks to train the probe (e.g., automatic metrics or modest human annotation).
- The HBO design is validated on short-form tasks and midsized models (Llama 3.1-8B, Gemma-2-9B).
- Default to MSP as a strong, training-free baseline for short-form uncertainty
- Sectors: any LLM product performing short answers or structured responses (finance Q&A, support bots, developer assistants).
- Tools/workflows:
- Integrate MSP into the LLM serving stack for instant confidence scoring when supervised probes are unavailable or for far-OOD traffic.
- Use MSP-led abstention policies and result-ordering.
- Assumptions/dependencies:
- Works best on short-form; underperforms on long-form outputs.
- Requires calibrated token probabilities from the LLM.
- Configure probes for better OOD robustness (when training is feasible)
- Sectors: platform teams building internal LLM services, research labs, vendors.
- Practices:
- Use middle-layer representations rather than final-layer features.
- Aggregate across all response tokens (mean pooling) versus last-token features.
- Keep probe architectures simple (MLP is sufficient); focus on inputs rather than complex probe models.
- Assumptions/dependencies:
- Access to internal representations; logging added compute and memory overhead.
- Training data must be representative enough for the target short-form use cases.
- OOD evaluation harness for product readiness and model governance
- Sectors: enterprise AI platforms, QA teams, academic labs, evaluators/auditors.
- Tools/workflows:
- Adopt Prediction–Rejection Ratio (PRR) and prediction–rejection curves for continuous correctness.
- Test both near-OOD (leave-one-out datasets) and far-OOD (different task) settings.
- Evaluate on both short- and long-form datasets; include robust baselines (MSP and mid-layer MLP probe).
- Complement AlignScore with LLM-as-a-judge when domain-specific correctness is required.
- Assumptions/dependencies:
- Need curated datasets with known shifts; PRR requires correctness scoring infrastructure.
- AlignScore and LLM-as-a-judge each have domain constraints/biases.
- Drift and health monitoring using feature-space diagnostics
- Sectors: any production LLM stack with changing user inputs.
- Workflows/tools:
- Maintain Mahalanobis-distance rank dashboards to monitor OOD rates over time.
- Use PLS-based 2D projections to visualize separability of correct vs. incorrect generations during A/B tests or model upgrades.
- Assumptions/dependencies:
- Requires periodic embedding extraction and storage policies; privacy considerations for hidden states/logs.
- Human-in-the-loop escalation for long-form outputs
- Sectors: healthcare/clinical notes (non-decisional support), legal summarization, research/biomed summarization, journalism support.
- Workflow/product:
- Route long-form outputs with low confidence to human review; show confidence flags in the UI.
- Add post-hoc verification steps (e.g., citation checking or claim-spot checks).
- Assumptions/dependencies:
- Current methods show weak OOD robustness on long-form tasks; rely on human oversight until better methods emerge.
- Data acquisition and annotation prioritization via UQ
- Sectors: model training/finetuning teams, dataset vendors, labs.
- Workflow/product:
- Use uncertainty scores to prioritize examples for human annotation, error analysis, and domain expansion (active learning).
- Assumptions/dependencies:
- Requires logging uncertainty signals and linking them to data pipelines and labeling tools.
- Procurement and policy checklists that require robust OOD UQ evidence
- Sectors: regulated industries (finance, healthcare, public sector), enterprise procurement.
- Practices:
- Vendors must show PRR performance under near- and far-OOD tests on short- and long-form tasks, with standardized baselines.
- Assumptions/dependencies:
- Availability of shared benchmarks and willingness of vendors to expose evaluation artifacts.
- User-facing confidence displays for short-form assistance
- Sectors: productivity apps, IDE copilots, customer support portals.
- Workflow/product:
- Present a simple confidence indicator derived from HBO/MSP; toggle auto-abstain below a threshold; offer “verify” buttons that trigger retrieval or additional reasoning.
- Assumptions/dependencies:
- Careful UX design to avoid overreliance; calibration may be needed per domain.
Long-Term Applications
These require additional research, scaling, or ecosystem changes to realize fully.
- Robust long-form UQ methods and segment-level verification
- Sectors: healthcare (clinical summaries), legal, scientific reporting, media.
- Products/workflows:
- Develop hybrid methods tailored to long-form, e.g., segment/chunk-level uncertainty aggregation, better correctness functions beyond AlignScore, domain-specific LLM judges, and claim-level verification integrated into UQ.
- Assumptions/dependencies:
- New correctness metrics and annotations; scalable evaluation datasets; efficient extraction and aggregation of long outputs.
- Standards and certifications for OOD UQ in AI systems
- Sectors: regulators, industry consortia, audit firms.
- Products:
- Common benchmark suites defining near- and far-OOD protocols; PRR-based pass/fail criteria; profiles for short vs. long-form tasks.
- Assumptions/dependencies:
- Community convergence on datasets, metrics, and acceptable baselines; governance frameworks to enforce reporting.
- Model training strategies that expose stable uncertainty signals
- Sectors: model developers, foundation model labs.
- Products:
- Pretraining/fine-tuning objectives that enhance OOD-stable uncertainty representations; training dedicated “uncertainty heads” accessible via API (with standardized outputs and calibration).
- Assumptions/dependencies:
- Requires large-scale experimentation; careful measurement to avoid overfitting UQ to ID distributions.
- API and platform support for UQ features
- Sectors: cloud AI providers, LLM API vendors.
- Products:
- APIs that expose calibrated confidence or internal middle-layer embeddings (or privacy-preserving surrogates) and lightweight OOD scores (e.g., MD ranks).
- Assumptions/dependencies:
- Privacy/security constraints on exposing internals; standardized interfaces and customer education.
- UQ-driven agentic control and tool selection
- Sectors: autonomous agents for software ops, finance ops, enterprise workflows.
- Products/workflows:
- Controllers that use robust UQ to decide between reasoning depth, external tool invocation, retrieval augmentation, or human escalation.
- Assumptions/dependencies:
- Robust UQ across tasks and longer outputs; integration with orchestration frameworks.
- Comprehensive UQ tooling suites
- Sectors: MLOps/LLMOps vendors, open-source ecosystems.
- Products:
- Libraries implementing HBO, SAPLMA variants, mid-layer feature extraction wrappers, Mahalanobis rankers, PRR evaluation harnesses, and dataset packs for near/far OOD tests.
- Assumptions/dependencies:
- Cross-model compatibility; maintenance as base models evolve; reproducibility infrastructure.
- Efficient/on-device UQ for edge deployments
- Sectors: mobile assistants, embedded AI, robotics.
- Products:
- Lightweight MSP- and mid-layer–based probes optimized for limited compute and memory; caching strategies for feature statistics.
- Assumptions/dependencies:
- Efficient access to internal states on-device; quantization-friendly implementations.
- Sector-specific UQ-integrated workflows
- Healthcare: UQ-gated clinical summaries and triage; audit logs for compliance.
- Finance: UQ-based abstention for advisory outputs; escalation to compliance checks.
- Education: UQ-assisted grading/feedback systems flagging low-confidence answers; instructor review queues.
- Energy/industrial: UQ-gated procedure assistants that abstain or require operator confirmation under high uncertainty.
- Assumptions/dependencies:
- Domain-tailored correctness signals; strong human oversight; regulatory alignment; secure data handling.
Notes on Feasibility and Dependencies
- Hidden-state access: Many findings depend on access to middle-layer hidden states; closed-model APIs may not expose them. Workarounds include:
- Use MSP-only methods where hidden states are unavailable (short-form).
- Request vendor-provided confidence APIs or embeddings.
- Correctness supervision: Supervised probes need correctness labels (AlignScore, LLM-judge, or human). Domain-specific labeling may be necessary for high-stakes use.
- Model and task scope: Results were demonstrated on midsized models and specific datasets; generalization to larger models, other tasks, and domains requires validation.
- Long-form limitations: Current methods are not robust OOD for long-form outputs; rely on human-in-the-loop and verification until improved methods are developed.
- Privacy/security: Logging and analyzing hidden states may raise privacy concerns; consider privacy-preserving surrogates or on-the-fly feature computations.
- Evaluation rigor: Adopt both near- and far-OOD tests, strong baselines, and report negative results to avoid overestimating robustness.
Glossary
- AlignScore: An automatic metric that scores factual consistency of generated text by leveraging entailment judgments. "We use AlignScore, estimating factual consistency using the entailment predictions of a RoBERTa model fine-tuned on NLI and claim verification."
- Aleatoric uncertainty: The component of uncertainty arising from inherent noise or ambiguity in the data. "The term 'hybrid methods' has previously been used to describe methods that combine aleatoric and epistemic uncertainty estimates."
- Attention maps: Matrices capturing how each token attends to others in a transformer, used here as features for uncertainty probes. "including attention maps"
- Density-based features: Features derived from how far representations lie from training data distributions, often via distances like Mahalanobis. "The SATMD and SATRMD baselines use density-based features, training probes using the Mahalanobis distance of hidden states from a subset of training examples, using an average distance per layer."
- Distribution shift: A change between training and test data distributions that can degrade model or probe performance. "which probe designs provide uncertainty estimates that are reliable under distribution shift."
- Entailment predictions: Outputs from an NLI model indicating whether a hypothesis is supported by a premise, used here to score factual consistency. "estimating factual consistency using the entailment predictions of a RoBERTa model fine-tuned on NLI and claim verification."
- Epistemic uncertainty: The component of uncertainty due to limited knowledge or model parameters, reducible with more data or better models. "methods that combine aleatoric and epistemic uncertainty estimates"
- Hidden states: Internal vector representations produced by model layers that can carry signals about correctness and uncertainty. "the hidden states of LLMs contain signals useful for uncertainty estimation and hallucination detection"
- Hybrid Back-Off (HBO): A method that adaptively combines supervised and unsupervised uncertainty signals based on how OOD an example is. "we explore Hybrid Back-Off (HBO) as an illustrative strategy for improving robustness"
- Kernel density estimate (KDE): A non-parametric way to estimate probability density, used here to visualize separability of correct vs. incorrect generations. "Kernel density estimate (KDE) contour lines show the density of generations above and below the median AlignScore."
- Leave-one-out (LOO) evaluation: An OOD testing setup where models are trained on several datasets and evaluated on a held-out dataset. "leave-one-out evaluation across a range of datasets"
- Lookback-lens: A probe method that uses the proportion of attention paid to context tokens as features for uncertainty estimation. "Lookback-lens uses attention, training probes with the proportion of attention paid to the context tokens."
- Mahalanobis distance: A distance metric that accounts for covariance structure, used to quantify how far a point is from a distribution. "training probes using the Mahalanobis distance of hidden states from a subset of training examples"
- Maximum Sequence Probability (MSP): An unsupervised uncertainty baseline computed by multiplying token probabilities across the sequence. "Maximum Sequence Probability (MSP), an unsupervised baseline that multiplies token probability scores from the model."
- Mean pooling: An aggregation technique that averages token representations, often used over response tokens for robust probing. "mean pooling over response tokens"
- Natural Language Inference (NLI): A task of determining entailment/contradiction between sentence pairs; used to train the scorer for factual consistency. "fine-tuned on NLI and claim verification."
- Oracle (in evaluation): The best possible predictor given the available generations, used to normalize evaluation metrics. "This score is normalised by an oracle (the best possible AUC given the generations), adjusting for a baseline that randomly predicts uncertainty"
- Out-of-distribution (OOD): Test conditions where inputs differ from those seen during training, challenging robustness. "not robust out-of-distribution (OOD)."
- Partial Least Squares (PLS) regression: A dimensionality reduction and regression method that projects features to components predictive of a target. "two-component Partial Least Squares (PLS) regression models that are fitted to predict the rank-normalised AlignScore."
- Prediction-Rejection curves: Curves plotting performance as uncertain examples are progressively rejected, used to evaluate uncertainty estimates. "Prediction-Rejection curves, where uncertain examples are progressively rejected to measure the average correctness of the remaining examples."
- Prediction-Rejection Ratio (PRR): A normalized AUC metric that measures how effectively uncertainty scores separate reliable from unreliable outputs. "Performance is evaluated using the Prediction-Rejection Ratio (PRR), which measures how effectively an uncertainty estimate identifies unreliable outputs."
- Probe (probing): A supervised model trained on internal representations (e.g., hidden states or attention) to predict properties like correctness or confidence. "Supervised Uncertainty Quantification (UQ) methods train probes to predict the reliability of the model outputs from model internals such as hidden states or attention"
- Rank-normalised: A normalization that maps ranks to [0,1], used here to combine supervised and unsupervised uncertainty scores. "both the rank-normalised supervised and unsupervised UQ estimates"
- Relative Mahalanobis distance: A normalized variant of Mahalanobis distance that uses a separate reference set for scaling. "SATRMD is a modification of SATMD using the Relative Mahalanobis distance, which normalises the Mahalanobis distance with a separate reference set."
- SAPLMA: A hidden-state probe that trains an MLP on model representations to estimate uncertainty. "SAPLMA trains a multi-layer perceptron (MLP) using the model hidden states"
- SATMD: A density-based probing method that aggregates Mahalanobis distances across layers as features. "The SATMD ... baselines use density-based features, training probes using the Mahalanobis distance of hidden states from a subset of training examples, using an average distance per layer."
- SATRMD: A variant of SATMD that employs Relative Mahalanobis distance for improved normalization. "SATRMD is a modification of SATMD using the Relative Mahalanobis distance"
- Sequence likelihoods: Probabilistic scores over entire generated sequences used as unsupervised uncertainty signals. "using token probabilities or sequence likelihoods"
- Token aggregation: Strategies for combining token-level signals into example-level features (e.g., last-token, mean over tokens). "token aggregation strategy"
- Uhead: A transformer-based probe that uses attention maps and token probabilities to predict uncertainty. "Uhead, a transformer-based probe using attention maps and token probabilities as features."
- Uncertainty Quantification (UQ): Methods for estimating how confident a model should be in its outputs. "Supervised Uncertainty Quantification (UQ) methods train probes to predict the reliability of the model outputs"
Collections
Sign up for free to add this paper to one or more collections.