PaladinEval: Dual Evaluation Benchmarks

• PaladinEval is a term defining two evaluation benchmarks: one for document–claim grounding and one for tool failure recovery in language agents.
• The grounding benchmark employs a formal entailment framework with balanced accuracy metrics across domains like General, Logical, Prices/Math, and Time/Dates.
• The tool-agent benchmark simulates deterministic tool failures using seven error classes, measuring recovery competence, honesty, and efficiency.

Searching arXiv for the relevant PALADIN/PaladinEval papers and closely related benchmark context. PaladinEval denotes two distinct evaluation resources in the 2025 literature. In the grounding literature, the term corresponds to the Qualifire-grounding-benchmark and its associated protocol for binary groundedness classification in document–claim pairs, as used to evaluate Paladin-mini (Ivry et al., 25 Jun 2025). In the agentic tool-use literature, PaladinEval is the explicit name of a deterministic failure-injection benchmark for measuring recovery competence, honesty, and efficiency of tool-augmented LLM agents under tool malfunction (Vuddanti et al., 25 Sep 2025). The shared label masks a substantive divergence in task definition, supervision signal, metrics, and intended deployment setting.

1. Terminological scope and disambiguation

The two uses of the term target different failure modes of LLM systems. The grounding-oriented usage asks whether a claim is supported by a document; the agentic usage asks whether an agent can recover when tool execution fails. This suggests that “PaladinEval” functions less as a single benchmark family than as a name collision across two contemporaneous PALADIN-related research programs.

Usage	Underlying resource	Core evaluation target
Grounding	Qualifire-grounding-benchmark / grounding-benchmark	Grounded vs ungrounded claim classification
Tool agents	PaladinEval	Recovery under deterministic tool failure injection

In the first usage, the benchmark is hosted on Hugging Face as qualifire/grounding-benchmark and is central to the PALADIN grounding framework around Paladin-mini (Ivry et al., 25 Jun 2025). In the second, PaladinEval is built from ToolBench evaluation tasks and instrumented with controlled failures aligned with ToolScan’s seven error classes (Vuddanti et al., 25 Sep 2025).

2. PaladinEval as the Qualifire-grounding-benchmark

Within "Paladin-mini: A Compact and Efficient Grounding Model Excelling in Real-World Scenarios" (Ivry et al., 25 Jun 2025), PaladinEval corresponds functionally to the Qualifire-grounding-benchmark plus its evaluation protocol. It is defined as a grounding benchmark, a specialized benchmark suite, and an application-focused alternative to broad aggregated fact-checking benchmarks. Its purpose is to assess whether a model can judge if a claim is grounded in a given document or context.

The task is a binary grounding classification task: given a document $D$ and a claim $c$, the model must decide whether the claim is grounded or ungrounded. The benchmark is explicitly organized into four reasoning categories. General targets abstract logical structures such as entailment and contradiction across arbitrary topics. Logical targets fine-grained reasoning in technical or industry-specific contexts. Prices / Math evaluates multi-step arithmetic grounded in textual numerical information, including prices, rates, and taxes. Time / Dates evaluates temporal reasoning, including parsing, comparing, and ordering temporal expressions.

The benchmark is described as “real-world” because these categories are chosen to match high-stakes enterprise use cases: finance and e-commerce for Prices/Math, scheduling and operations for Time/Dates, technical documentation for Logical, and broad factual consistency for General. The paper contrasts this design with aggregated academic benchmarks such as LLM-AggreFact, arguing that disaggregation is necessary to expose domain-specific weaknesses, particularly in numerical, logical, and temporal reasoning (Ivry et al., 25 Jun 2025).

The paper does not provide a full annotation pipeline, detailed corpus provenance, or a benchmark-wide statistics table. It also does not describe train/validation/test splits for this resource, because the benchmark is used as an evaluation benchmark, while Paladin-mini is trained on separate public and proprietary synthetic data.

3. Formal grounding semantics and evaluation protocol

The grounding formulation underlying the benchmark is stated in a formal entailment framework (Ivry et al., 25 Jun 2025). The evidence set is a finite set of sentences from a document,

$D = \{s_1, s_2, \ldots, s_m\},$

and the claim is a logical proposition that may be decomposed into atomic facts,

$$c \leftrightarrow (a_1 \land a_2 \land \ldots \land a_n).$$

A claim $c$ is grounded in $D$ iff

$D \models c,$

and ungrounded iff

$D \not\models c.$

The benchmark examples are therefore triplets $(D, c, \text{label})$, where the label is grounded or ungrounded under this entailment notion. For the synthetic data generation framework that informs model training, the paper also imposes a minimal support condition: each atomic fact $a_i$ is supported by a subset $c$0, and removing the evidence for any atomic component breaks entailment,

$c$1

Evaluation uses Balanced Accuracy (BACC) rather than plain accuracy, because the paper emphasizes that label imbalance can make raw accuracy misleading (Ivry et al., 25 Jun 2025). BACC gives equal weight to the grounded and ungrounded classes through Sensitivity and Specificity. Results are reported per category—General, Logical, Prices/Math, and Time/Dates—and then averaged for summary reporting.

The evaluation setup compares Paladin-mini (3.8B, open source), Paladin-large (14B, private), and Bespoke-MiniCheck-7B, with Gemini-2.0-flash included qualitatively. The task remains binary classification throughout: no multi-label output, no externally specified decomposition, and no open-ended generation metric are used in the benchmark protocol (Ivry et al., 25 Jun 2025).

4. PaladinEval as a deterministic failure-injection benchmark for tool agents

In "PALADIN: Self-Correcting LLM Agents to Cure Tool-Failure Cases" (Vuddanti et al., 25 Sep 2025), PaladinEval is the paper’s primary benchmark for execution-level robustness of tool-augmented LLM agents. It is described as “a deterministic failure-injection benchmark designed to evaluate recovery competence, honesty, and efficiency under the seven ToolScan error classes.”

PaladinEval is derived from ToolBench’s evaluation dataset. The authors begin from ToolBench tasks and tool schemas, extract the original user prompt and toolset, and then inject failures into tool calls during simulated execution. The benchmark is therefore simultaneously a dataset of failure-augmented ToolBench-style episodes and an evaluation framework consisting of a simulator and grading protocol (Vuddanti et al., 25 Sep 2025).

The failure taxonomy follows ToolScan’s seven canonical error classes: Tool Hallucination, Argument Hallucination, Invalid Tool Invocation, Partial Execution, Tool Output Hallucination, Invalid Intermediate Reasoning, and Re-entrant Failures. At a predetermined step in an episode, the simulator injects a single labeled failure such as HTTP 400/401/403/404/429/500/503 errors, timeouts, DNS failures, malformed JSON, schema violations, or partial outputs. Seeds, injection positions, and subsequent tool outputs are fixed to ensure determinism across model comparisons.

Episodes use the same serialized format as ToolBench and the PALADIN training traces: system messages, user prompts, assistant messages in a Thought → Action → Action Input pattern, and function messages for tool outputs, including injected errors. The simulator logs the dialogue context $c$2, observed error $c$3, recovery action $c$4, and final outcome. These traces are then passed to an automated GPT-5 API grader that computes the benchmark metrics (Vuddanti et al., 25 Sep 2025).

From the agent’s perspective, each instance provides the task prompt $c$5 and toolset $c$6, but no explicit advance warning that a failure will occur. The expected behavior is to detect failure, diagnose whether it is transient or client-side, choose an appropriate recovery action—retry with backoff, correct parameters, switch tools, or terminate candidly—and either complete the task or fail honestly (Vuddanti et al., 25 Sep 2025).

5. Metrics, baselines, and reported empirical behavior

The grounding benchmark and the tool-failure benchmark use different metric suites because they operationalize different notions of reliability.

For the grounding benchmark, the core metric is Balanced Accuracy. Table 2 in the paper reports, for Paladin-mini, 91.97 on General, 97.1 on Logical, 82.0 on Time/Dates, and 96.0 on Prices/Math, with an AVG. of 91.767; for Bespoke-MiniCheck-7B, the corresponding values are 84.02, 92.8, 90.0, and 46.0, with an AVG. of 78.205 (Ivry et al., 25 Jun 2025). The reported interpretation is that Paladin-mini beats MiniCheck-7B on three of four categories, with the most pronounced gap in Prices/Math, while underperforming on Time/Dates.

The same paper also reports a broader model summary across both the grounding-benchmark and LLM-AggreFact subsets: paladin-large at 83.480 Avg Bacc and ~150ms latency, paladin-mini at 79.31 Avg Bacc and ~70ms, and Bespoke-MiniCheck-7B at 77.868 Avg Bacc and ~7sec (Ivry et al., 25 Jun 2025). This is the basis for the paper’s characterization of Paladin-mini as a “realtime guardrail against ungrounded claims.”

For the tool-agent PaladinEval, the metric suite consists of Task Success Rate (TSR), Recovery Rate (RR), Catastrophic Success Rate (CSR), and Efficiency Score (ES) (Vuddanti et al., 25 Sep 2025). TSR measures solved tasks over all tasks. RR measures failures recovered over failures encountered. CSR is defined so that higher values indicate fewer hallucinated successes after failure. ES is the reciprocal of the average number of steps to complete the task.

The main PaladinEval result reported in the paper is on Gemma-32B-Instruct: Vanilla RR 23.75%, ToolBench agent RR 32.76%, ToolReflect RR 65.86%, CRITIC RR 76.34%, and PALADIN RR 89.68% (Vuddanti et al., 25 Sep 2025). The same section reports strong gains across additional backbones. On PaladinEval, Qwen-2.5-4B improves from 81.74% RR under CRITIC to 94.67% under PALADIN; LLaMA-3.1-8B from 63.44% to 79.77%; and AM-Thinking V1 from 89.91% to 96.08%. TSR and CSR show corresponding improvements. The paper further reports correlations among the metrics: RR and TSR are strongly positively correlated at $c$7, CSR and ES are negatively correlated at $c$8, and TSR and ES correlate weakly at $c$9 (Vuddanti et al., 25 Sep 2025).

Taken together, the two PaladinEval usages measure distinct but related forms of reliability. One measures whether a claim is entailed by retrieved evidence; the other measures whether an agent can avoid cascading failure after tool malfunction.

6. Relation to adjacent benchmarks, availability, and limitations

The grounding-oriented PaladinEval is positioned against LLM-AggreFact, TofuEval, ClaimVerify, and FactCheck-GPT (Ivry et al., 25 Jun 2025). Its main distinguishing property is specialization rather than aggregation: instead of collapsing heterogeneous fact-verification phenomena into a single score, it disaggregates numerical, logical, temporal, and general reasoning. The paper uses the strong LLM-AggreFact standing of Bespoke-MiniCheck-7B together with its 46.0% BACC on Prices/Math to argue for a benchmark–utility gap in real-world grounding evaluation.

The tool-agent PaladinEval is positioned against ToolBench, ToolReflectEval, and ToolScan (Vuddanti et al., 25 Sep 2025). ToolBench supplies the underlying task distribution but assumes clean tool behavior. ToolReflectEval focuses on call-level self-correction. ToolScan provides the error taxonomy rather than a recovery benchmark. PaladinEval is distinct in centering full-episode recovery under explicit, deterministic failure injection, with honesty and efficiency measured alongside task completion.

Availability also differs across the two uses. The grounding benchmark is publicly hosted as https://huggingface.co/datasets/qualifire/grounding-benchmark, and the paper lists public training resources such as C2D-and-D2C MiniCheck, LLM-AggreFact, TofuEval, FactCheck-GPT, ExpertQA, RAGTruth, and AggreFact-CNN & XSum (Ivry et al., 25 Jun 2025). The tool-agent work releases a recovery-annotated training dataset on Hugging Face and the runtime error catalog plus recovery dictionary on GitHub, with PaladinEval implemented through the simulator, deterministic injection policy, and GPT-5-based grading pipeline (Vuddanti et al., 25 Sep 2025).

The reported limitations are likewise domain-specific. For the grounding benchmark, the paper identifies weaker performance on Time/Dates, bounded domain coverage outside the four target categories, reliance on synthetic data, and incomplete disclosure of annotation guidelines and benchmark statistics (Ivry et al., 25 Jun 2025). For the tool-agent benchmark, the paper highlights the safety–efficiency trade-off captured by CSR and ES and uses a separate generalization set to probe transfer to unseen tools and errors; ablations indicate that removing inference-time retrieval lowers RR by 20–30 points across backbones (Vuddanti et al., 25 Sep 2025).

A recurrent source of confusion is lexical overlap with the 2021 kernel-security paper "In-Kernel Control-Flow Integrity on Commodity OSes using ARM Pointer Authentication" (Yoo et al., 2021). That work introduces PAL, not PaladinEval, and concerns ARM Pointer Authentication for in-kernel CFI rather than either grounding evaluation or tool-failure robustness. The name overlap is terminological only.

In current arXiv usage, then, PaladinEval is best treated as a disambiguation label covering two unrelated benchmark constructs: a groundedness benchmark for document–claim entailment and a failure-injection benchmark for robust tool-using agents. Their shared significance lies not in a common protocol, but in a common evaluative ambition: replacing single-number capability summaries with stress tests tailored to the failure modes that matter in deployment.