"I Don't Know" -- Towards Appropriate Trust with Certainty-Aware Retrieval Augmented Generation

Abstract: Achieving the right amount of trust in AI systems is important, but challenging. The problem is exacerbated with the rise of LLMs as they provide human-level communication capabilities, but potentially hallucinate in the content that they generate. Moreover, they express over-confidence in their answers, making it difficult for users to judge their truthfulness. An important human value that users seek is benevolence, which can be met by LLM's self-reflection leading to reliable and honest answers. Accordingly, this paper proposes conveying appropriate levels of self-reflected certainty to build appropriate trust. Our contributions are twofold: 1) We develop CERTA (Certainty Enhanced RAG for Trustworthy Answers), a specialized Retrieval Augmented Generation (RAG) system that incorporates the relevance between question, context, and answer to reflect its uncertainty in answering questions; 2) We create the Certainty Benchmark with 90 question-context pairs of non-objective questions, divided over four categories (factuality, preference, sycophancy, morality) and three types of contexts (relevant, incomplete, irrelevant). We run experiments with a baseline RAG system and three CERTA settings using two LLMs. Our evaluations indicate that CERTA helps identify answers that are uncertain, decreases the cases of over-agreeing, and provides cautious behavior when prompted for moral judgments.

• The paper presents CERTA, a framework that integrates explicit, relevance-driven certainty scores into retrieval augmented generation for calibrated uncertainty estimates.
• It employs multi-dimensional relevance assessments (question-context, context-answer, answer-question) to compute numerical and verbal uncertainty, ensuring more accurate responses.
• Empirical results show that CERTA reduces overconfidence in non-objective queries, yielding improved alignment and trustworthiness in AI outputs.

Certainty-Aware Retrieval Augmented Generation for Appropriately Calibrated Trust in LLMs

Introduction and Motivation

LLMs have reached impressive capabilities in natural language understanding and generation, yet suffer from a lack of calibrated uncertainty estimation. This manifests as overconfident answers—often to non-objective or ill-posed queries—thereby fostering misplaced user trust, exacerbating risks of both over-reliance and undue skepticism. The paper "I Don't Know" -- Towards Appropriate Trust with Certainty-Aware Retrieval Augmented Generation (2605.00957) investigates this phenomenon and proposes the CERTA (Certainty Enhanced Retrieval Augmented Generation for Trustworthy Answers) framework. The core objective is to integrate explicit self-reflection and uncertainty quantification into LLM-based retrieval-augmented systems, aligning LLM responses with the human value of benevolence and ultimately aiming for appropriate—not excessive or insufficient—user trust.

Background and Related Work

Prior research highlights the dangers of LLM hallucinations and sycophancy, especially in domains where factual accuracy or adherence to user values is essential. While value alignment and uncertainty estimation have been considered separately, there is a notable gap in merging these concerns for RAG-based LLM setups. Standard approaches to uncertainty—such as using LLM self-assessments or confidence scores—are limited by the circularity and biases inherent to LLM self-judgment. The RAG Triad paradigm, comprising question-context, context-answer, and answer-question relevance, is a well-motivated alternative. By computing objective relevance scores (e.g., through embedding-based similarity), it becomes feasible to generate calibrated uncertainty estimates usable for both automated and user-facing interfaces.

Figure 1: Schematic of the RAG setup augmented with the Triad, explicitly highlighting the three axes of relevance assessment among question, context, and answer.

The CERTA Framework

CERTA extends a baseline RAG pipeline by introducing systematic self-reflection grounded in multi-dimensional relevance. The notable innovations are:

• Explicit Relevance-Driven Certainty: Instead of relying on LLM-internal knowledge or subjective self-assessment, CERTA computes certainty scores derived from objective measures between the input question, retrieved context, and generated answer (i.e., the RAG Triad: QCR, CAR, AQR).
• Dashboard Presentation: CERTA presents not only a verbal response but numerical certainty metrics and reasoning support, allowing users to observe and interpret the model's knowledge boundaries.
• Flexible Operation Modes: Several configurations (CERTA-0, CERTA-1, CERTA-2) modulate the stringency with which the model expresses uncertainty. This permits tuning towards user preference for risk-aversion versus answer directness.

Certainty Benchmark: Task and Evaluation

To rigorously evaluate CERTA, the authors introduce a certainty benchmark with 90 question-context pairs. These span not only factual queries but also preference, sycophancy, and moral categories—each articulated with relevant, incomplete, or irrelevant contexts to challenge the system's ability to detect when confident answering is unwarranted. The baseline approach (plain RAG) and CERTA variants were tested using both GPT-based and Mistral-based LLMs.

Figure 2: An example of a non-objective question ("Is it more important to be honest or kind?") answered with unwarranted confidence by ChatGPT-4o.

Empirical Results

The experiments show that CERTA systematically increases the use of uncertain/cautious responses (e.g., "I don't know"), though not indiscriminately. Strong findings include:

• Factuality: When context is incomplete or irrelevant, CERTA models more appropriately indicate uncertainty. Notably, CERTA-1 and CERTA-2 provide cautious responses where baseline RAG continues to assert, sometimes inappropriately, plausible answers. This effect is especially pronounced for subjective or ambiguous questions.
• Preference, Sycophancy, and Morality: CERTA reduces over-agreeing (alignment to perceived user preference without evidence), refuses to issue unsubstantiated moral judgments more often, and provides detailed reasoning when opting for "I don't know."
• Consistency and Side-effect Analysis: Crucially, the introduction of CERTA's uncertainty-aware mechanisms does not degrade performance on questions where confident answers are justified.

  Figure 3: CERTA-2$_{\text{Mistral}}$ dashboard response to a factual query, displaying calibrated certainty scores and a reserved verbal response.

  

  Figure 4: GPT-baseline RAG dashboard response to a personal preference query, illustrating failure to reflect low relevance, leading to overconfident answering.

  

  Figure 5: CERTA-1$_{\text{GPT}}$ dashboard example for a moral choices scenario, demonstrating nuanced uncertainty expression and principled withholding of moral judgment when context is incomplete.

Quantitatively, CERTA-1 and CERTA-2 produce "I don't know" responses in up to 149/240 cases (vs 106/240 for baseline RAG), with the trend stronger for non-objective and poorly-supported contexts.

Theoretical and Practical Implications

CERTA advances the alignment of AI systems with user expectations by giving explicit, transparent signals of uncertainty calibrated via relevance metrics instead of LLM self-report. This approach responds to recognized limitations of both overconfidence and excessive indecisiveness, finding an empirically validated middle ground. The modularity and retriever-agnostic design support potential integration with both closed and open-source models. Presenting both verbal and numerical uncertainty aids users in making informed trust decisions, particularly improving system utility in high-stakes or safety-critical applications.

Pragmatically, CERTA's inclusion of explicit "I don't know" and uncertainty rationale functionality can mitigate legal liability and reputational risks of LLMs producing harmful advice or misinformation. From a theoretical standpoint, the integration of value-sensitive AI paradigms with robust epistemic humility marks a compelling direction for future alignment and trustworthiness research in AI agents.

Limitations and Future Directions

The primary limitations include the lack of user studies on the dashboard's effect on trust calibration and the relatively constrained scope of the certainty benchmark (90 examples, limited diversity in ambiguous moral/sycophantic queries). The impact of CERTA on other LLMs (including smaller-scale and highly fine-tuned models), alternate embedding/relevance metrics, and retrieval strategies remains to be fully characterized.

Planned future extensions include in-situ user evaluation, scaling the benchmark dataset to more domains, benchmarking with adversarial and noisy retrieval conditions, and developing formal representations of certainty as a configurable value. Layering self-reflection with actionable suggestions (e.g., "refine your query") could further enhance collaborative LLM-user interactions.

Conclusion

CERTA constitutes a substantive step toward value-aligned, uncertainty-aware RAG systems, demonstrating significant gains in appropriate uncertainty communication, overconfidence mitigation, and moral judgment calibration in LLMs. By leveraging multi-faceted relevance scores to drive epistemic self-reflection, CERTA enables more nuanced and trustworthy AI advice systems and provides a practicable framework for explicit uncertainty presentation. Ongoing research should explore broader model applicability, richer user-centered benchmarks, and formalization of self-reflection and certainty as tunable AI values.