CalVerT: Augmenting Agents with Calibrated Verifier Telemetry Improves Action and Learning in Knowledge-Intensive Tasks

Abstract: LLM agents in knowledge intensive question answering take retrieval and reasoning actions with incomplete knowledge about whether their current answer is uncertain, unsupported, or already complete. This produces two failure modes: committing to confident but unsupported answers, which hurts accuracy, and over-retrieving when the evidence in hand already suffices, resulting in wasted compute. To give agents a more complete picture of the state space they are operating in, we introduce calibrated verifier telemetry (CalVerT), which augments the agent's state with additional telemetry: a calibrated self-confidence score and a grounding verifier score. We show that CalVerT can improve agents in both training-free and training-based settings. On four QA benchmarks, we find that CalVerT raises F1 by triggering retrieval in cases where agents over-rely on parametric knowledge, while cutting redundant retrieval in cases where agents have sufficient context to answer. We show that CalVerT can augment existing QA frameworks without training. Moreover, CalVerT also improves trained systems: by simply augmenting an agent's state with telemetry, we observe improvements after reinforcement learning, as compared to an agent with identical training but no CalVerT telemetry.

• The paper introduces CalVerT, which integrates dual calibrated telemetry signals to guide LLM agents in making more accurate and efficient decisions.
• The paper demonstrates that calibrated self-confidence and grounding verification reduce over-committing and over-retrieval, achieving significant F1 improvements in multi-hop QA benchmarks.
• The paper shows that both training-free prompt augmentation and RL fine-tuning with CalVerT yield robust performance gains across diverse agent frameworks.

CalVerT: Augmenting Agents with Calibrated Verifier Telemetry for Knowledge-Intensive Tasks

Motivation and Problem Formulation

The paper addresses a core limitation in the operation of LLM agents for knowledge-intensive QA: the inability to accurately gauge, on each turn, whether the currently proposed answer is both well-grounded in external evidence and accompanied by a correctly calibrated self-confidence estimate. Existing agents exhibit two primary failure modes: (1) over-committing—delivering highly confident yet evidence-unsupported answers, and (2) over-retrieval—needlessly issuing retrieval calls when extant evidence is already sufficient. These inefficiencies lead to either degraded QA performance or inflated computational cost, and stem from the lack of direct telemetry providing interpretable, calibrated per-turn information about answer veracity and support.

To resolve this, CalVerT introduces a two-channel verifier telemetry signal. This complements the agent's state at each decision point with (a) a verbalized self-confidence score and (b) a fact-grounding verifier score. Both signals are independently calibrated and derived from specialized models, ensuring their reliability as state features for the agent's action selection process.

CalVerT Framework and Telemetry Signals

CalVerT integrates into standard ReAct-style agent pipelines, where an agent, for each subgoal or step in a reasoning decomposition, must choose among actions: commit to an answer, retrieve more external evidence, refine the current answer, or decompose the subgoal further. At each such decision node, CalVerT injects the following signals into the agent's state:

• Calibrated Self-Confidence: Obtained via DiNCo, combining normalized verbal confidence (NVC) and self-consistency (SC)—the latter derived from the level of agreement among multiple sampled outputs given the same question.
• Calibrated Grounding Verifier: Computed by MiniCheck, which scores the entailment of each atomic claim in the candidate answer against the currently retrieved evidence pool. Both mean grounding ($g_{\text{mean}}$) and minimum grounding ($g_{\min}$, reflecting the weakest claim) are exposed as telemetry.

Agents perceive these as additional quantitative state info in their per-turn prompt (for prompt-based settings), or as input channels during RL fine-tuning. Critically, CalVerT refrains from enforcing explicit action gates or thresholds; the agent must learn to internalize and act upon these signals in a manner best aligned with downstream reward.

Figure 1: Verifier telemetry enables the agent to avoid both over-retrieval and premature commitment—by dynamically responding to state information about confidence and grounding at each reasoning turn.

Empirical Evaluation

Training-Free Prompt Augmentation

A comprehensive evaluation is conducted on four benchmarks (HotpotQA-distractor, 2WikiMultihopQA, MuSiQue, WiTQA), with CalVerT compared to baseline agents under both prompt-only augmentation (frozen generator with and without telemetry in the prompt) and training-based (RL fine-tuned) regimes.

Results indicate that, in harder multi-hop QA settings, telemetry exposure yields consistent F1 improvements—up to $\mathbf{+4.7}$ points on WiTQA and $\mathbf{+3.7}$ on 2Wiki for Qwen3-32B, with corresponding reductions in average turns per example, evidencing more efficient traversal of the reasoning/search space. In single-hop or factoid settings (e.g., WiTQA with Mistral-24B), telemetry triggers increased (and warranted) retrieval rates, correcting parametric over-trust.

Figure 2: F1 gains attributable to calibrated telemetry rise with question hop count (multi-hop QA) or with decreasing subject/entity frequency (factoid settings), demonstrating effective action allocation particularly on challenging or data-sparse queries.

Ablation analysis confirms the necessity of both telemetry channels: using only confidence or grounding impairs efficiency and accuracy, particularly in multi-hop environments where isolated signals are ambiguous for optimal routing.

Plug-and-Play Portability

To test the generalizability of CalVerT, the telemetry signal was integrated into five established adaptive retrieval frameworks (e.g., TARG, SeaKR, Self-Ask, SUGAR, Verify-and-Edit), substituting their original uncertainty gates without retraining. Across four of five frameworks, telemetry yielded significant F1 gains (e.g., $\mathbf{+15.4}$ on TARG, $\mathbf{+7.8}$ on SeaKR). Retrieval costs either decreased or remained stable except in the SUGAR baseline, where aggressive pruning was observed.

RL-Finetuned Agents

In reinforcement learning experiments with GRPO optimization on HotpotQA-distractor, agents equipped with telemetry as input features outperformed telemetry-free baselines given identical reward structure and training protocol. For instance, Qwen3-8B saw a $\mathbf{+5.9}$ F1 gain; Qwen3-30B-A3B achieved $\mathbf{+3.3}$ F1 improvement post-GRPO training. Notably, prompt-only telemetry can yield negative or negligible effects at small model scales (e.g., Qwen3-8B), but RL can compensate, enabling agents to exploit the signals productively.

Calibration Integrity

Reliability diagrams and calibration metrics confirm both DiNCo and MiniCheck provide well-calibrated scoring, evidenced by near-diagonal empirical accuracy curves and low ECE/Brier scores over respective validation regimes (TriviaQA for confidence, LLM-AggreFact for grounding).

Figure 3: Reliability diagrams illustrate strong calibration: agent self-confidence and grounding scores map closely to empirical answer correctness across a wide operating range.

Theoretical and Practical Implications

The demonstration that dual-signal telemetry systematically improves both action efficiency and answer accuracy, and that it is robust to integration into a variety of agent architectures without retraining, establishes a strong case for modular telemetry-based agent design. The separation of parametric confidence and explicit evidence grounding guarantees that robust action policies can emerge even under action-agnostic prompting, provided model scale or RL adaptation is sufficient. This reduces both the risk of hallucination (especially in multi-hop and tail-entity settings) and the computational expense associated with blind search/retrieval.

The approach is especially salient for settings where cost/latency constraints are acute, or where agents must abstain or defer reliably when evidence is ambiguous or lacking—crucial for scientific, medical, or engineering applications demanding traceability and calibrated abstention.

Limitations and Future Directions

CalVerT's framework, focusing on commit-on-support policies, is tailored for settings where single answer extraction per question is appropriate. Generalization to answer recall or multi-answer tasks (FanOutQA, QAMPARI, AmbigQA) is nontrivial and would require extensions to the action/state space and potentially richer forms of telemetry aggregation. Further, while calibration integrity is substantiated for the current signal sources, continued advances in both confidence estimation and grounding verification architectures could yield even more potent telemetry.

An interesting avenue is leveraging richer forms of belief-state summarization and integrating more granular or historical uncertainty signals (e.g., from belief decoupling or reflective agent paradigms) alongside calibrated external telemetry.

Conclusion

CalVerT provides a general, framework-agnostic method for augmenting knowledge-intensive QA agents with reliable, turn-level confidence and grounding signals. Experimental evidence across multiple models, benchmarks, and agent paradigms demonstrates that calibrated telemetry substantially improves answer accuracy, cost efficiency, and adaptiveness in both zero-shot and RL-finetuned settings. This work substantiates a principled path toward measurable and controllable LLM agents for complex knowledge reasoning, and motivates ongoing research into more expressive and nuanced uncertainty architectures.