Papers
Topics
Authors
Recent
Search
2000 character limit reached

The "I Don't Know" Filter: Enhancing Agentic Reliability in Function Calling

Published 4 Jul 2026 in cs.SE and cs.AI | (2607.04034v1)

Abstract: The LLMs that underpin agents have seen a rapid rise in performance on function calling benchmarks. However, the metrics used in the training and evaluation of these models often encourage models to make positive claims even when the answer is uncertain, leading to hallucinations. Such hallucinations can be disastrous when LLMs are trusted to use function calls to make decisions in high stakes applications. To that end, we propose an agent evaluation metric that takes into account the negative outcomes associated with incorrect function calls. Further, to catch hallucinations before they can cause harm, we propose a lightweight trainable filter that can quantify a LLM's uncertainty and remove potentially harmful function calls. By training that filter to detect and suppress uncertain function calls without modifying the underlying model, we demonstrate a practical path toward agents that know when to say "I don't know," a property we argue is essential to production reliability.

Summary

  • The paper introduces the 'I Don't Know' filter, a classifier-based system that lets language models abstain from uncertain function calls to prevent hallucinations.
  • It leverages whitebox, graybox, and blackbox feature analyses through engineered metrics to calibrate uncertainty and improve output reliability.
  • Experimental evaluations reveal consistent IDKS gains and cross-model generalization, underscoring its practical value in high-stakes, API-limited settings.

Enhancing Reliability in LM Function Calling with the "I Don't Know" Filter

Introduction and Motivation

The paper "The 'I Don't Know' Filter: Enhancing Agentic Reliability in Function Calling" (2607.04034) addresses a critical concern in deploying agentic LMs: the overproduction of hallucinated or incorrect function calls in task-oriented settings. Existing benchmarks prioritize raw accuracy and often fail to penalize confidently-incorrect outputs, which poses substantial risks in high-stakes applications such as infrastructure automation and healthcare. The authors propose a new evaluation metric and an auxiliary classifier-based filtering approach to enable agents to abstain (“I don't know,” or IDK) when appropriate, reducing errant function executions.

IDK Score: A Penalty-Weighted Metric for Agent Reliability

Traditional benchmarks inadequately incentivize restraint, making no distinction between a wrong answer and a safe non-answer. The paper introduces the "I Don't Know Score" (IDKS):

IDKS=C−INIDKS = \frac{C - I}{N}

where CC is the count of correct function calls, II is the count of incorrect ones, and NN is the total. Abstentions do not influence the score, which ranges from -1 (all incorrect) to 1 (all correct). By explicitly penalizing mistakes more harshly than refusals to answer, IDKS better aligns evaluation with the operational risks of agentic systems.

The Classifier Filter: Design and Integration

To bridge the production-benchmark gap, the authors propose a lightweight, trainable filter that assesses agent confidence by inspecting output variability across repeated invocations for a given input. The filter is instantiated as a random forest classifier, operating on engineered features derived from three levels of model access:

  • Whitebox: Internal activations and states (available when full model access is possible)
  • Graybox: Log-probabilities output by the model (accessible via certain APIs)
  • Blackbox: Textual outputs (always available)

Features include measures such as semantic volume, variation ratio, pairwise normalized distances, and entropy. Abstention is triggered when the classifier deems the uncertainty threshold is exceeded, replacing the proposed function call with an explicit “I don’t know.”

Figure 1

Figure 1: A classifier-based filter processes repeated agent function call outputs, suppressing likely-incorrect calls via abstentions to mitigate hallucinations.

Synthetic Data Generation for Robust Training

Realistic training data for function calling tasks is scarce. The authors construct a synthetic protocol: starting from structured function-calling datasets (specifically, the Berkeley Function Calling Leaderboard datasets), they employ "constrained prompting" with GPT-4.1-mini to generate consistent user questions aligned to given functions and expected outputs. This approach is contrasted with open-ended prompting, which frequently produces misaligned task-API mappings.

Figure 2

Figure 2: Constrained prompting for synthetic data generation ensures alignment between user questions, available functions, and expected function calls, outperforming open-ended generation strategies.

Furthermore, the design enables compatibility with any function-calling dataset, promoting extensibility.

Experimental Evaluation

Cross-Model and Cross-Benchmark Validation

The filter was evaluated on three families of small LMs (SLMs): Phi-4, Qwen2.5, and Llama-3.2, on a battery of real and synthetic function calling benchmarks (including APIGen). Validation included scenarios with varying numbers of repeat outputs and different feature access modes (blackbox, graybox, whitebox), as well as the transferability of classifiers trained on one LM to predicting uncertainty in others.

Tuning Abstention Thresholds

The abstention threshold balances filtering aggressiveness. Too low, and insufficient hallucinations are suppressed; too high, and correct calls are excessively abstained from, reducing utility.

Figure 3

Figure 3: The effect of the classifier's decision threshold on agent reliability, as measured by IDKS, highlights the need for principled threshold calibration to optimize performance.

Results: Numerical Impact and Generalization

Three major findings characterize the results:

  • Consistent IDKS Gains: Across all datasets and agent models, applying the classifier yields measurable improvements in IDKS over the unfiltered baseline.
  • Feature Invasiveness: While whitebox features occasionally boost classification, blackbox+graybox access suffices for large IDKS gains, beneficial for API-gated deployment.
  • Cross-LM Portability: Filters trained on one model generalize with only modest degradation when used on others, indicating that uncertainty signals are not idiosyncratic to specific models.

Figure 4

Figure 4: IDK Score improvements for Llama with varying numbers of repeated outputs, both with and without the classifier filter, demonstrate that filtering efficacy is robust to repeat count variations.

Figure 5

Figure 5: Broad IDK Score improvements for Llama, Phi, and Qwen agents, validating cross-model generality of the classifier-based filter, regardless of the model on which it was trained.

Notably, longer lines in Figure 4 and consistent gaps in Figure 5 highlight the strong, model-agnostic performance gap induced by classifier filtering.

Failure Modes and Limitations

Despite its utility, the approach faces limits:

  • Confident Consistency in Error: The filter cannot detect cases where the agent repeatedly makes the same wrong call; semantic invariance is assumed to encode correctness.
  • Superficial Formatting Variance: Agents producing functionally-equivalent calls with minor syntactic changes may be over-filtered due to spurious output variance.
  • Thresholding: Optimal abstention threshold selection is nontrivial and task-dependent.

Implications and Future Directions

The framework’s plug-and-play nature—filtering extraneous to the base LM—renders it feasible for real-world integration, particularly with closed-source or API-limited models. The pronounced improvements in reliability, even with limited model introspection, suggest applicability to contemporary LLM APIs and future agentic deployments at scale.

For theory, the findings reinforce the value of output consistency as a predictor of uncertainty in function calling as opposed to open-ended language tasks. The classifier’s cross-model generalizability suggests deep, architecture-independent signals present in repeated LM behavior.

Future research could target:

  • Sophisticated threshold calibration methods (domain-adaptive abstention).
  • Hybrid filtering combining semantic normalization and external type-checks.
  • Extension to closed/proprietary LLMs or larger agentic frameworks.
  • Deeper exploration of adversarial and robustness queries for abstention mechanisms.

Conclusion

The "I Don't Know" filter presents an effective, generalizable solution for enhancing reliability in LM-based agents performing function calling. By introducing a penalty-weighted metric (IDKS) and a classifier-based filter acting on repeated outputs, the framework robustly reduces hallucinations and unsafe actions in agentic contexts. The approach delivers transferable gains across models and benchmarks, requiring only superficial access to agent predictions, and can be integrated into safety-critical systems without architectural overhaul. These results lay the groundwork for future, more nuanced interventions addressing the gap between LLM capability benchmarks and reliable, real-world agent deployment.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Tweets

Sign up for free to view the 1 tweet with 1 like about this paper.