TrustEHRAgent: Confidence-Aware EHR QA

• TrustEHRAgent is a confidence-aware clinical QA system that integrates step-wise and final confidence estimation to manage uncertainty.
• It enhances clinical safety by selectively answering queries, abstaining when evidence is weak to prevent hallucinations.
• Evaluated with HCAcc@k% metrics, it demonstrates superior reliability and effectiveness compared to traditional EHR agents.

TrustEHRAgent is a confidence-aware agent for clinical question answering over Electronic Health Records (EHRs), proposed to address a central safety problem in medical LLM deployment: a clinically useful system must not only answer correctly when evidence is sufficient, but also recognize when uncertainty is high and abstain rather than hallucinate (Song et al., 26 Aug 2025). The framework builds on the code-based reasoning style of EHRAgent while adding step-wise confidence estimation and a final confidence estimator, and it is evaluated with a reliability-aware metric, Hallucination Controlled Accuracy at $k\%$ (HCAcc@$k\%$), designed to capture the trade-off among correctness, abstention, and hallucination control in high-stakes healthcare settings (Song et al., 26 Aug 2025).

1. Clinical problem formulation and design rationale

TrustEHRAgent is motivated by the observation that LLMs are increasingly used for medical question answering and EHR querying, including text-to-SQL and agentic reasoning systems, yet these models can hallucinate factual details, generate incorrect SQL, misread abbreviations, or answer confidently when the underlying EHR evidence is weak (Song et al., 26 Aug 2025). In the clinical setting, such errors are treated not as ordinary model mistakes but as patient-safety risks, because an incorrect answer about a diagnosis, medication, lab value, or timing issue can mislead clinicians.

The conceptual distinction at the center of the framework is between accuracy and reliability. Accuracy asks how often the model is correct. Reliability asks whether the model knows when it is correct and whether it can abstain when uncertain. TrustEHRAgent is therefore organized around selective answering rather than unconditional response generation: answer when confidence is high, reject or abstain when confidence is low, and expose uncertainty instead of hallucinating (Song et al., 26 Aug 2025).

A common misconception addressed by the paper is that higher aggregate exact-match accuracy is sufficient evidence of clinical readiness. The reported results argue otherwise. Two systems can exhibit similar unconstrained accuracy while differing sharply once hallucination control is imposed, which suggests that raw accuracy can obscure the operational safety profile of an EHR agent (Song et al., 26 Aug 2025).

2. Architecture and inference workflow

TrustEHRAgent augments EHRAgent with confidence awareness at two levels: step-wise confidence estimation during reasoning and a final confidence estimator that governs answer acceptance (Song et al., 26 Aug 2025). The agent reasons over a clinical query in a code/SQL-style interaction, but it does not wait until the final answer to assess uncertainty. Instead, at each major reasoning step it produces a self-rated confidence score on a 0–10 scale. The prompting instruction quoted in the paper requires the model to place a comment of the form # Confidence: X at the top of each code solution and to use a “highly critical standard,” explicitly considering data limitations, interpretation ambiguities, and error propagation risks.

These step-wise signals are produced especially when the system generates SQL or code, selects tables or joins, interprets query wording, handles abbreviations, or combines intermediate results. The resulting trace exposes how uncertainty evolves across the reasoning chain rather than collapsing uncertainty into a single terminal judgment (Song et al., 26 Aug 2025).

After the reasoning process, a separate confidence estimator receives three inputs: the original clinical question, the reasoning history enriched with step-wise confidence signals, and the final answer. It outputs a final confidence score $C \in [0,1]$, which is compared to a threshold $\tau$. If $C \ge \tau$, the system returns the answer; if $C < \tau$, it rejects or abstains. The estimator uses 5 discrete confidence levels and relies on token log probabilities rather than forcing a hard discrete output, with the stated goal of obtaining a finer-grained confidence signal and avoiding coarse scoring bias (Song et al., 26 Aug 2025).

The workflow can therefore be summarized as a selective-decision pipeline: receive a clinical question over an EHR database, reason through the query using code/SQL-style interaction, record confidence at each reasoning step, aggregate the reasoning traces and confidence signals in the confidence estimator, compute the final score, threshold the score, and evaluate correctness under reliability constraints rather than only under unconditional answering (Song et al., 26 Aug 2025).

3. HCAcc@$k\%$ and the formalization of reliability-aware evaluation

The principal evaluation contribution paired with TrustEHRAgent is Hallucination Controlled Accuracy at $k\%$, denoted HCAcc@$k\%$ (Song et al., 26 Aug 2025). The metric asks not how accurate the model is when it answers every question, but what maximum overall accuracy can be achieved while keeping hallucination below a specified bound. In this construction, $k\%$ is the required reliability level. HCAcc@90% means hallucinations must be at or below 10%; HCAcc@70% means hallucinations must be at or below 30%.

Let $k\%$0 be the full set of questions and let

$k\%$1

be the set of questions answered at confidence at least $k\%$2. Let $k\%$3 denote the subset answered correctly. The paper defines the hallucination rate as

$k\%$4

where the conditional accuracy is

$k\%$5

Overall accuracy is then

$k\%$6

with response rate

$k\%$7

The optimization objective is

$k\%$8

This metric makes abstention explicit. A model that answers less often may be safer; a model that answers more often may be more dangerous if its errors increase. HCAcc@$k\%$9 therefore combines utility through overall accuracy, safety through hallucination control, and abstention behavior through the response rate (Song et al., 26 Aug 2025). In the context of EHR QA, this is the formal mechanism by which the paper operationalizes the claim that reliability is distinct from accuracy.

4. Experimental setting, datasets, and baselines

The evaluation uses MIMIC-III and eICU, both real ICU EHR databases, with a benchmark based on EHRSQL / EHRAgent data (Song et al., 26 Aug 2025). The datasets contain 580 question-answer pairs for MIMIC-III and 580 for eICU, for a total of 1,160 QA pairs. The questions were originally collected from 200 hospital staff members and span demographics, prescriptions, lab results, diagnosis-related queries, time-sensitive EHR questions, and multi-table reasoning. Each example includes the clinical question, the gold answer, and a reference SQL query.

The task is to answer EHR questions by querying structured data, ranging from simple retrieval to multi-hop reasoning, joins, calculations, and temporal constraints. The backbone model is GPT-4.1-mini with temperature 0.0, and all methods share the same EHR metadata, schema descriptions, tools, four-shot demonstrations, interaction step limits, and answer-matching procedure. Exact match against the gold answer is the main metric, while HCAcc@$C \in [0,1]$0 provides the reliability-aware analysis (Song et al., 26 Aug 2025).

The baseline set has two layers. Confidence-aware baselines are SQLAgent and PythonAgent augmented with the same confidence estimation pipeline. Baselines without confidence estimation, taken from EHRAgent and related work, are CoT, ReACT, Reflexion, DIN-SQL, SQLAgent, PythonAgent, and EHRAgent (Song et al., 26 Aug 2025). This design isolates the effect of the proposed confidence mechanism from the underlying task formulation.

5. Quantitative performance and empirical interpretation

The headline numerical result is that TrustEHRAgent remains operational under reliability constraints that cause baseline methods to collapse (Song et al., 26 Aug 2025). At HCAcc@0%, corresponding to standard unconstrained accuracy, TrustEHRAgent achieves 62.13% on MIMIC-III and 59.82% on eICU. EHRAgent achieves 58.97% and 53.10%, so the reported improvements are +3.16 points and +6.72 points, respectively. The paper interprets this as evidence that adding confidence estimation does not harm baseline question-answering performance.

At HCAcc@70%, the paper reports the critical reliability result: TrustEHRAgent achieves 44.23% on MIMIC-III and 25.34% on eICU, while baseline methods fall to 0% at this threshold (Song et al., 26 Aug 2025). At HCAcc@90%, performance is much lower—3.27% on MIMIC-III and 2.24% on eICU—but the paper treats this as evidence that extreme reliability constraints remain difficult rather than as a failure of the framework.

Setting	MIMIC-III	eICU
TrustEHRAgent HCAcc@0%	62.13%	59.82%
TrustEHRAgent HCAcc@70%	44.23%	25.34%
TrustEHRAgent HCAcc@90%	3.27%	2.24%
EHRAgent HCAcc@0%	58.97%	53.10%
Baselines at HCAcc@70%	0%	0%

The ablation results indicate that step-wise confidence estimation contributes most strongly at higher confidence thresholds. The paper also compares three confidence-estimation strategies: a weighted sum over token log probabilities, binary classification confidence, and discrete scoring. The weighted probability-based approach yields the smoothest and most useful confidence curve; binary classification produces sharp drops because it is too coarse; discrete scoring lacks sufficient granularity; and continuous confidence estimation is reported as better aligned with threshold-based selective answering (Song et al., 26 Aug 2025).

The case studies illustrate the intended operating mode. In a successful complex clinical analytics query, confidence remains high throughout multiple reasoning steps and the final answer is correct. In a case involving an ambiguous medical abbreviation, confidence declines across reasoning steps and the final confidence falls to about 0.499, causing the system to reject the query rather than hallucinate (Song et al., 26 Aug 2025). This is significant because the rejection mechanism is not treated as a defect but as the clinically appropriate response under insufficient evidence.

6. Limitations, deployment implications, and relation to adjacent trust frameworks

The paper identifies three explicit limitations. First, performance at very high thresholds remains poor: even TrustEHRAgent is weak at HCAcc@90%, indicating that extreme safety-aware QA remains difficult. Second, the evaluation uses only MIMIC-III and eICU, which may not capture the full diversity of real clinical workflows or EHR systems. Third, the current weighted log-probability estimator is useful but not ideal, and the paper proposes temperature scaling, ensembles, Bayesian calibration, and domain-specific calibration methods as future directions (Song et al., 26 Aug 2025).

The practical implication is that LLM-based clinical agents should be deployed as selective decision-support tools rather than unconditional answer machines. The paper argues that systems should expose confidence, support rejection or abstention, and tune thresholds to risk level—for example, lower thresholds for low-risk administrative queries and higher thresholds for high-risk clinical decision support (Song et al., 26 Aug 2025). This position directly counters the misconception that abstention is merely reduced capability; within the TrustEHRAgent framework, abstention is a safety mechanism.

TrustEHRAgent also sits within a broader research program on trustworthy agentic systems. AEMA treats evaluation itself as a process-aware, auditable workflow with human oversight (Lee et al., 17 Jan 2026). AgentTrace provides runtime observability across operational, cognitive, and contextual logging surfaces (AlSayyad et al., 7 Feb 2026). AgentTrust introduces pre-execution interception for tool calls with verdicts such as allow, warn, block, or review (Yang, 6 May 2026). EchoTrust applies an evidence-based Actor–Verifier design to echocardiographic VQA rather than EHR querying (Huang et al., 7 Apr 2026). This suggests that TrustEHRAgent addresses one specific trust axis—confidence-calibrated selective answering in clinical EHR QA—while adjacent work addresses complementary axes such as verifiable evaluation, execution tracing, and runtime intervention.

In that broader context, the central contribution of TrustEHRAgent is not merely a higher score on medical question answering. Its significance lies in recasting clinical EHR agents as systems that must couple answer generation with calibrated uncertainty, explicit abstention, and evaluation under hallucination constraints (Song et al., 26 Aug 2025). For high-stakes medical AI, that reframing is the paper’s defining claim.