---
title: 'PaladinEval: Dual Evaluation Benchmarks'
url: https://www.emergentmind.com/topics/paladineval
type: topic
---

# PaladinEval: Dual Evaluation Benchmarks

Searching arXiv for the relevant PALADIN/PaladinEval papers and closely related benchmark context.
PaladinEval denotes two distinct evaluation resources in the 2025 literature. In the grounding literature, the term corresponds to the **Qualifire-grounding-benchmark** and its associated protocol for binary groundedness classification in document–claim pairs, as used to evaluate Paladin-mini [2506.20384]. In the agentic tool-use literature, **PaladinEval** is the explicit name of a **deterministic failure-injection benchmark** for measuring recovery competence, honesty, and efficiency of tool-augmented language model agents under tool malfunction [2509.25238]. The shared label masks a substantive divergence in task definition, supervision signal, metrics, and intended deployment setting.

## 1. Terminological scope and disambiguation

The two uses of the term target different failure modes of LLM systems. The grounding-oriented usage asks whether a claim is supported by a document; the agentic usage asks whether an agent can recover when tool execution fails. This suggests that “PaladinEval” functions less as a single benchmark family than as a name collision across two contemporaneous PALADIN-related research programs.

| Usage | Underlying resource | Core evaluation target |
|---|---|---|
| Grounding | Qualifire-grounding-benchmark / grounding-benchmark | Grounded vs ungrounded claim classification |
| Tool agents | PaladinEval | Recovery under deterministic tool failure injection |

In the first usage, the benchmark is hosted on Hugging Face as `qualifire/grounding-benchmark` and is central to the PALADIN grounding framework around Paladin-mini [2506.20384]. In the second, PaladinEval is built from ToolBench evaluation tasks and instrumented with controlled failures aligned with ToolScan’s seven error classes [2509.25238].

## 2. PaladinEval as the Qualifire-grounding-benchmark

Within "Paladin-mini: A Compact and Efficient Grounding Model Excelling in Real-World Scenarios" [2506.20384], PaladinEval corresponds functionally to the **Qualifire-grounding-benchmark** plus its evaluation protocol. It is defined as a **grounding benchmark**, a **specialized benchmark suite**, and an application-focused alternative to broad aggregated fact-checking benchmarks. Its purpose is to assess whether a model can judge if a claim is **grounded** in a given document or context.

The task is a **binary grounding classification task**: given a document \(D\) and a claim \(c\), the model must decide whether the claim is **grounded** or **ungrounded**. The benchmark is explicitly organized into four reasoning categories. **General** targets abstract logical structures such as entailment and contradiction across arbitrary topics. **Logical** targets fine-grained reasoning in technical or industry-specific contexts. **Prices / Math** evaluates multi-step arithmetic grounded in textual numerical information, including prices, rates, and taxes. **Time / Dates** evaluates temporal reasoning, including parsing, comparing, and ordering temporal expressions.

The benchmark is described as “real-world” because these categories are chosen to match high-stakes enterprise use cases: finance and e-commerce for Prices/Math, scheduling and operations for Time/Dates, technical documentation for Logical, and broad factual consistency for General. The paper contrasts this design with aggregated academic benchmarks such as LLM-AggreFact, arguing that disaggregation is necessary to expose domain-specific weaknesses, particularly in numerical, logical, and temporal reasoning [2506.20384].

The paper does not provide a full annotation pipeline, detailed corpus provenance, or a benchmark-wide statistics table. It also does not describe train/validation/test splits for this resource, because the benchmark is used as an **evaluation benchmark**, while Paladin-mini is trained on separate public and proprietary synthetic data.

## 3. Formal grounding semantics and evaluation protocol

The grounding formulation underlying the benchmark is stated in a formal entailment framework [2506.20384]. The evidence set is a finite set of sentences from a document,
\[
D = \{s_1, s_2, \ldots, s_m\},
\]
and the claim is a logical proposition that may be decomposed into atomic facts,
\[
c \leftrightarrow (a_1 \land a_2 \land \ldots \land a_n).
\]

A claim \(c\) is **grounded** in \(D\) iff
\[
D \models c,
\]
and **ungrounded** iff
\[
D \not\models c.
\]
The benchmark examples are therefore triplets \((D, c, \text{label})\), where the label is grounded or ungrounded under this entailment notion. For the synthetic data generation framework that informs model training, the paper also imposes a **minimal support** condition: each atomic fact \(a_i\) is supported by a subset \(D_i \subseteq D\), and removing the evidence for any atomic component breaks entailment,
\[
\forall i \in \{1, \ldots, n\} : (D \setminus D_i) \not\models c.
\]

Evaluation uses **Balanced Accuracy (BACC)** rather than plain accuracy, because the paper emphasizes that label imbalance can make raw accuracy misleading [2506.20384]. BACC gives equal weight to the grounded and ungrounded classes through Sensitivity and Specificity. Results are reported per category—General, Logical, Prices/Math, and Time/Dates—and then averaged for summary reporting.

The evaluation setup compares **Paladin-mini (3.8B, open source)**, **Paladin-large (14B, private)**, and **Bespoke-MiniCheck-7B**, with **Gemini-2.0-flash** included qualitatively. The task remains binary classification throughout: no multi-label output, no externally specified decomposition, and no open-ended generation metric are used in the benchmark protocol [2506.20384].

## 4. PaladinEval as a deterministic failure-injection benchmark for tool agents

In "PALADIN: Self-Correcting Language Model Agents to Cure Tool-Failure Cases" [2509.25238], PaladinEval is the paper’s **primary benchmark for execution-level robustness** of tool-augmented language model agents. It is described as “a deterministic failure-injection benchmark designed to evaluate recovery competence, honesty, and efficiency under the seven ToolScan error classes.”

PaladinEval is derived from **ToolBench’s evaluation dataset**. The authors begin from ToolBench tasks and tool schemas, extract the original user prompt and toolset, and then inject failures into tool calls during simulated execution. The benchmark is therefore simultaneously a dataset of failure-augmented ToolBench-style episodes and an evaluation framework consisting of a simulator and grading protocol [2509.25238].

The failure taxonomy follows ToolScan’s seven canonical error classes: **Tool Hallucination**, **Argument Hallucination**, **Invalid Tool Invocation**, **Partial Execution**, **Tool Output Hallucination**, **Invalid Intermediate Reasoning**, and **Re-entrant Failures**. At a predetermined step in an episode, the simulator injects a single labeled failure such as HTTP 400/401/403/404/429/500/503 errors, timeouts, DNS failures, malformed JSON, schema violations, or partial outputs. Seeds, injection positions, and subsequent tool outputs are fixed to ensure determinism across model comparisons.

Episodes use the same serialized format as ToolBench and the PALADIN training traces: `system` messages, `user` prompts, `assistant` messages in a Thought → Action → Action Input pattern, and `function` messages for tool outputs, including injected errors. The simulator logs the dialogue context \(C_t\), observed error \(e_t\), recovery action \(a_t\), and final outcome. These traces are then passed to an automated **GPT-5 API** grader that computes the benchmark metrics [2509.25238].

From the agent’s perspective, each instance provides the task prompt \(P\) and toolset \(\mathcal{A}\), but no explicit advance warning that a failure will occur. The expected behavior is to detect failure, diagnose whether it is transient or client-side, choose an appropriate recovery action—retry with backoff, correct parameters, switch tools, or terminate candidly—and either complete the task or fail honestly [2509.25238].

## 5. Metrics, baselines, and reported empirical behavior

The grounding benchmark and the tool-failure benchmark use different metric suites because they operationalize different notions of reliability.

For the grounding benchmark, the core metric is **Balanced Accuracy**. Table 2 in the paper reports, for **Paladin-mini**, **91.97** on General, **97.1** on Logical, **82.0** on Time/Dates, and **96.0** on Prices/Math, with an AVG. of **91.767**; for **Bespoke-MiniCheck-7B**, the corresponding values are **84.02**, **92.8**, **90.0**, and **46.0**, with an AVG. of **78.205** [2506.20384]. The reported interpretation is that Paladin-mini beats MiniCheck-7B on three of four categories, with the most pronounced gap in **Prices/Math**, while underperforming on **Time/Dates**.

The same paper also reports a broader model summary across both the grounding-benchmark and LLM-AggreFact subsets: **paladin-large** at **83.480 Avg Bacc** and **~150ms** latency, **paladin-mini** at **79.31 Avg Bacc** and **~70ms**, and **Bespoke-MiniCheck-7B** at **77.868 Avg Bacc** and **~7sec** [2506.20384]. This is the basis for the paper’s characterization of Paladin-mini as a **“realtime guardrail against ungrounded claims.”**

For the tool-agent PaladinEval, the metric suite consists of **Task Success Rate (TSR)**, **Recovery Rate (RR)**, **Catastrophic Success Rate (CSR)**, and **Efficiency Score (ES)** [2509.25238]. TSR measures solved tasks over all tasks. RR measures failures recovered over failures encountered. CSR is defined so that higher values indicate fewer hallucinated successes after failure. ES is the reciprocal of the average number of steps to complete the task.

The main PaladinEval result reported in the paper is on **Gemma-32B-Instruct**: **Vanilla RR 23.75%**, **ToolBench agent RR 32.76%**, **ToolReflect RR 65.86%**, **CRITIC RR 76.34%**, and **PALADIN RR 89.68%** [2509.25238]. The same section reports strong gains across additional backbones. On PaladinEval, **Qwen-2.5-4B** improves from **81.74%** RR under CRITIC to **94.67%** under PALADIN; **LLaMA-3.1-8B** from **63.44%** to **79.77%**; and **AM-Thinking V1** from **89.91%** to **96.08%**. TSR and CSR show corresponding improvements. The paper further reports correlations among the metrics: RR and TSR are strongly positively correlated at \(r = 0.91\), CSR and ES are negatively correlated at \(r = -0.72\), and TSR and ES correlate weakly at \(r = 0.12\) [2509.25238].

Taken together, the two PaladinEval usages measure distinct but related forms of reliability. One measures whether a claim is entailed by retrieved evidence; the other measures whether an agent can avoid cascading failure after tool malfunction.

## 6. Relation to adjacent benchmarks, availability, and limitations

The grounding-oriented PaladinEval is positioned against **LLM-AggreFact**, **TofuEval**, **ClaimVerify**, and **FactCheck-GPT** [2506.20384]. Its main distinguishing property is **specialization rather than aggregation**: instead of collapsing heterogeneous fact-verification phenomena into a single score, it disaggregates numerical, logical, temporal, and general reasoning. The paper uses the strong LLM-AggreFact standing of Bespoke-MiniCheck-7B together with its **46.0%** BACC on Prices/Math to argue for a benchmark–utility gap in real-world grounding evaluation.

The tool-agent PaladinEval is positioned against **ToolBench**, **ToolReflectEval**, and **ToolScan** [2509.25238]. ToolBench supplies the underlying task distribution but assumes clean tool behavior. ToolReflectEval focuses on call-level self-correction. ToolScan provides the error taxonomy rather than a recovery benchmark. PaladinEval is distinct in centering full-episode recovery under explicit, deterministic failure injection, with honesty and efficiency measured alongside task completion.

Availability also differs across the two uses. The grounding benchmark is publicly hosted as `https://huggingface.co/datasets/qualifire/grounding-benchmark`, and the paper lists public training resources such as C2D-and-D2C MiniCheck, LLM-AggreFact, TofuEval, FactCheck-GPT, ExpertQA, RAGTruth, and AggreFact-CNN & XSum [2506.20384]. The tool-agent work releases a recovery-annotated training dataset on Hugging Face and the runtime error catalog plus recovery dictionary on GitHub, with PaladinEval implemented through the simulator, deterministic injection policy, and GPT-5-based grading pipeline [2509.25238].

The reported limitations are likewise domain-specific. For the grounding benchmark, the paper identifies weaker performance on **Time/Dates**, bounded domain coverage outside the four target categories, reliance on synthetic data, and incomplete disclosure of annotation guidelines and benchmark statistics [2506.20384]. For the tool-agent benchmark, the paper highlights the safety–efficiency trade-off captured by CSR and ES and uses a separate generalization set to probe transfer to unseen tools and errors; ablations indicate that removing inference-time retrieval lowers RR by **20–30 points** across backbones [2509.25238].

A recurrent source of confusion is lexical overlap with the 2021 kernel-security paper "In-Kernel Control-Flow Integrity on Commodity OSes using ARM Pointer Authentication" [2112.07213]. That work introduces **PAL**, not PaladinEval, and concerns ARM Pointer Authentication for in-kernel CFI rather than either grounding evaluation or tool-failure robustness. The name overlap is terminological only.

In current arXiv usage, then, **PaladinEval** is best treated as a disambiguation label covering two unrelated benchmark constructs: a **groundedness benchmark** for document–claim entailment and a **failure-injection benchmark** for robust tool-using agents. Their shared significance lies not in a common protocol, but in a common evaluative ambition: replacing single-number capability summaries with stress tests tailored to the failure modes that matter in deployment.

Source: https://www.emergentmind.com/topics/paladineval