- The paper introduces a synthetic benchmark that integrates multi-hop entity identification, factual property retrieval, and quantitative computation.
- It employs a rigorous pipeline that utilizes knowledge graphs, programmatic templates, and diversity filters to generate contamination-resistant questions.
- Empirical evaluations reveal significant gaps in current LLM performance, highlighting challenges in multi-skill, agentic reasoning tasks.
DrBencher: Evaluating Multi-Skill Agentic Reasoning with Synthetic Benchmarks
Motivation and Context
Evaluation of agentic LLM-based systems increasingly requires tasks integrating multi-hop entity identification, factual property retrieval, and quantitative computation—capabilities that modern web-enabled agents must master for robust, real-world performance. Previous benchmarks predominantly assess these skills in isolation, resulting in a lack of coverage for the true compositional and agentic complexity encountered in deployed scenarios. Furthermore, static benchmarks with publicly available answer keys suffer from contamination and memorization effects, severely limiting their utility for the evaluation of advanced models. The "DRBENCHER" framework directly addresses these gaps by generating synthetic, contamination-resistant benchmarks—each instance requiring both browsing and quantitative reasoning, with verifiable, model-independent programmatic gold solutions (2604.09251).
Pipeline Architecture and Criteria
DrBencher's pipeline follows a strictly answer-first paradigm, in which all candidate questions are instantiated by first fetching ground-truth data from knowledge graphs (primarily Wikidata), then composing a parameterized computational template over these values to produce a reference gold answer. The subsequent question generation stages enforce four explicit, jointly optimized design criteria:
- Verifiability: All gold answers are deterministically derived by executing domain-specific code over KG-sourced properties; no step in the pipeline relies on subjective or unverifiable judgments.
- Complexity: Each question requires multi-hop entity identification, correlation of at least three distinct KG chains into clue facts, property retrieval, and domain-specific computation. This is quantified using the Compositional Complexity Index (CCI), defined as the sum of entity and property retrieval operations required.
- Difficulty: A two-stage, self-referential verification filter discards questions deemed solvable by the generating LLM with access to tools ("open-book") or without ("closed-book"), thus calibrating extrinsic difficulty relative to the capabilities of the current frontier models.
- Diversity: A graph-based max-min embedding filter (MiniLM-L6-v2 as the default backbone) is applied to maximize representational coverage across entities, templates, and linguistic formulations.
This architecture is domain-agnostic and is instantiated across biochemistry, financial, geophysical, security, and history domains, leveraging domain-specific data sources as needed.
Synthetic Benchmark Generation Algorithms
DrBencher's generation algorithm begins with programmatic discovery of entities (e.g., via SPARQL queries or domain-resolved identifiers), extraction of multi-hop knowledge-graph chains, and retrieval of candidate properties. Template instantiation is conditioned on compatibility of entity types with quantitative computation templates (e.g., population density, atmospheric modeling, financial ratios).
LLMs are used for two key roles: (i) extracting natural language clues for each chain, ensuring indirect entity descriptions across multiple paths; and (ii) composing the final question by integrating clues with the reasoning template prompt and suppressing any direct leakage of the entity or answer.
Programmatic validators ensure the correct entity is uniquely recoverable, the gold answer is perfectly reproducible, and there is no ambiguity. Questions surviving these filters are then passed through the two-stage difficulty filter, excluding all instances that the base model finds trivial before diversity is enforced.
Evaluation: Human and Automatic
Human evaluation was conducted by domain experts, measuring both entity resolvability and question clarity. DrBencher achieves 76% overall validity (rising to 84% when errors from stale knowledge-graph data are excluded), with the predominant error categories being outdated KG values, ambiguity, and LLM-induced hallucinations in clue generation. Notably, 35% of errors derive from data staleness, emphasizing a limitation in KG-based pipelines operating over rapidly evolving domains.
Automatic evaluation was performed using three proprietary and three open-weight models. The best-performing closed model (Claude Opus 4.6) reached 86% entity identification accuracy but only 20% answer accuracy, with error analysis indicating that property retrieval and subsequent computation, rather than entity resolution, are the main rate-limiting factors. Model accuracy drops monotonically as CCI increases, a trend confirmed by Jonckheere–Terpstra and Spearman correlation analyses. Domains with high CCI and property-obscure data sources (e.g., financial, security) show particularly abysmal performance (as low as 5–6% accuracy even at CCI=2), suggesting a paucity of pretraining signal for these data types in foundation models.
Contamination Resistance and Benchmark Diversity
A major contribution of DrBencher is dynamic, instancewise benchmark generation. As each sample is uniquely synthesized over live data and has a programmatically checked gold answer, the framework is not susceptible to answer key leakage, unlike static or manually curated benchmarks.
Empirical analysis using Self-BLEU and embedding-based semantic dissimilarity (MiniLM, BGE, Granite, E5) demonstrates that DrBencher exhibits higher diversity than classic manually constructed datasets (BrowseComp+, MATH-500, GPQA). This outcome is robust to choice of embedding model, showing the efficacy of the diversity filter and the compositional process.
Implications and Future Directions
DrBencher exposes major gaps in the agentic and compositional reasoning abilities of current frontier LLMs. The marked separation between entity identification and answer accuracy, especially in high-CCI or domain-obscure settings, suggests that integration between retrieval-augmented architectures and robust tool-use policies remains insufficient for complex agentic question answering.
Practically, DrBencher enables honest assessment of evolving agent systems without contamination risks. Theoretically, it motivates research in multi-hop retrieval architectures, cross-modal entity linking, robust numerical tool use, dynamic program induction, and techniques to track and leverage up-to-date structured knowledge under data drift. Reliance on external KGs also highlights bottlenecks related to coverage, factual drift, and subjectivity in indirect entity referencing.
Future directions include integrating multi-model pipelines for phase decoupling (to avoid generator–verifier coupling pathologies), expanding reasoning templates across more scientific and engineering domains, and exploring augmentation with more temporally live knowledge sources. There is also scope for applying augmented consistency filters, improving clue chain grounding to manage KG incompleteness, and extending complexity calibration to account for real-time tool failures or search engine volatility.
Conclusion
DrBencher represents a comprehensive approach to synthetic, contamination-resistant benchmark generation for multi-skill question answering in agentic LLMs. It formalizes the joint enforcement of verifiability, complexity, difficulty, and diversity, and provides a robust empirical basis for measuring true agentic reasoning capability. The current state of closed- and open-weight models on DrBencher demonstrates substantial unsolved challenges, setting a demanding future agenda for both practical system design and foundational research in agentic AI evaluation (2604.09251).