DRBench: Multifaceted AI and Software Benchmark

• DRBench is a multifaceted evaluation suite that assesses AI robustness, enterprise research agents, citation reliability, and data race detection in parallel programs.
• It employs dynamic protocols including blind image perturbations and multi-horizon research tasks to identify failure modes like language bias, prompt sensitivity, and race conditions.
• Key metrics such as the Dynamic Robustness Score, insight recall, non-resolving URL rate, and precision in race detection provide actionable, reproducible benchmarks.

DRBench denotes multiple benchmarks with significant impact in AI robustness, enterprise deep research, citation validation, and parallel program analysis. The term refers most prominently to: (1) Dynamic Robustness Benchmark (for vision-LLMs), (2) a benchmark for enterprise deep research agents, (3) a large-scale testbed for citation hallucination assessment in LLMs, and (4) DataRaceBench for data race detection in parallel programs. Each instantiation addresses distinct domains but shares methodological rigor and relevance as a reproducibility-driven evaluation suite.

1. Dynamic Robustness Benchmark for Vision-LLMs

The Dynamic Robustness Benchmark ("DRBench") (Tang et al., 8 Mar 2026) is a model-specific evaluation suite designed to diagnose and quantify robustness failures in Large Vision-LLMs (LVLMs), targeting two principal categories: language bias and language sensitivity. It is motivated by empirical evidence that modern LVLMs exhibit idiosyncratic failure modes—such as “blind-image bias” (reliance on linguistic priors over image content) and “irrelevant-prompt sensitivity”—that evade static, model-agnostic benchmarks.

Failure Modes and Formal Definitions

Given a collection of test triples $\{(v^0_n, q^0_n, a_n^\mathrm{gt})\}_{n=1}^N$, DRBench identifies:

• Language-Bias Failure: For some sample $n$, if

$$\arg\max_a\,p^0_n(a) = \arg\max_a\,p^j_n(a) \neq a_n^{\mathrm{gt}}$$

for any $j \in \{1,\dots,J\}$, where $v^j_n$ is a blind (fully masked or blurred) version of $v^0_n$ and $p^j_n(a) = p(a|v^j_n, q^0_n)$. This captures over-reliance on the question alone.

• Language-Sensitivity Failure: For some sample $n$, if

$\arg\max_a\,p^0_n(a) \neq \arg\max_a\,r^i_n(a)$

for any $i \in \{1,\dots,I\}$, where $n$0 is a semantically irrelevant prompt variant and $n$1. This detects instability under innocuous prompt edits.

Indicator variables $n$2 (language bias) and $n$3 (language sensitivity) are aggregated so that each “hard” sample flagged forms the DRBench set for a model.

Construction and Protocol

The DRBench protocol is model-specific and dynamic:

• For each sample, generate $n$4 blind images and $n$5 counterfactual prompts via masking, paraphrase, or distraction.
• Run the LVLM for $n$6 rounds, updating the flagged samples in each iteration.
• The benchmark grows monotonically with $n$7 and adapts to each model, revealing both fresh and persistent vulnerabilities.

Typical hyperparameters are $n$8 (maximally occluded image), $n$9 (prompt variants), $$\arg\max_a\,p^0_n(a) = \arg\max_a\,p^j_n(a) \neq a_n^{\mathrm{gt}}$$0.

Metrics

• Language-Bias Rate (LB): $$\arg\max_a\,p^0_n(a) = \arg\max_a\,p^j_n(a) \neq a_n^{\mathrm{gt}}$$1
• Language-Sensitivity Rate (LS): $$\arg\max_a\,p^0_n(a) = \arg\max_a\,p^j_n(a) \neq a_n^{\mathrm{gt}}$$2
• Dynamic Robustness Score (DRS): $$\arg\max_a\,p^0_n(a) = \arg\max_a\,p^j_n(a) \neq a_n^{\mathrm{gt}}$$3

Empirical Findings

Application to Qwen2-VL shows that multiple rounds of Self-Critical Inference (SCI) incrementally reduce both LB and LS, improving DRS and transferring to other robustness benchmarks. For instance, SCI$$\arg\max_a\,p^0_n(a) = \arg\max_a\,p^j_n(a) \neq a_n^{\mathrm{gt}}$$4 achieves up to 2 percentage points gain on DRBench and up to 3 points on external suites over the baseline, highlighting generalization.

Deployment Recommendations

• Always report LB, LS, and DRS with standard accuracy scores.
• Construct DRBench per model; optionally, utilize cross-model construction with hold-outs to avoid circularity.
• Fix perturbation-generation seeds for reproducibility.

2. Benchmark for Enterprise Deep Research Agents

DRBench (Abaskohi et al., 30 Sep 2025) offers a challenging assessment framework for AI agents tasked with complex, open-ended research in enterprise environments. Unlike conventional web-only, single-step QA benchmarks, it operationalizes multi-horizon, multi-source tasks that require aggregation across public and private (e.g., SharePoint, chats, emails) knowledge silos.

Task Ontology and Synthesis

Fifteen deep research tasks are generated via a five-stage human-in-the-loop LLM pipeline:

1. Company and persona generation (ensuring realistic business context).
2. Collection of domain-specific public insights, verified for topical relevance.
3. Question drafting and selection, ensuring part of the question is answerable from public sources while requiring private knowledge for completeness.
4. Internal insight and distractor synthesis, mapped to specific modalities (PDFs, spreadsheets, chats, etc.).
5. File generation and manual verification for plausibility and balance.

The result is a highly heterogeneous search space combining public web documents, productivity files, emails, and chat logs.

Evaluation Axes

DRBench scoring quantifies agent capability across four dimensions:

• Insight Recall (IR): Fraction of ground-truth injected insights retrieved.
• Distractor Avoidance (DA): $$\arg\max_a\,p^0_n(a) = \arg\max_a\,p^j_n(a) \neq a_n^{\mathrm{gt}}$$5, where $$\arg\max_a\,p^0_n(a) = \arg\max_a\,p^j_n(a) \neq a_n^{\mathrm{gt}}$$6 is recall on distractor (false) insights.
• Factuality (F): Proportion of agent-claimed insights actually supported by citations, as assessed by a Retriever-Augmented Generation (RAG) and LLM-judge pipeline.
• Report Quality (Q): LLM-judged mean over six axes (depth, relevance, consistency, coherence, contradiction-free, completeness).
• Composite Score: Harmonic mean of (IR, F, DA, Q).

Experimental Results

Evaluations indicate that advanced planning strategies (e.g., Adaptive Action Planning, AAP) notably improve both recall and report quality. Closed-source LLMs (e.g., GPT-5) outperform open-source alternatives in insight recall and overall composite score; AAP yields a harmonic mean of 47.4 compared to 41.7–42.5 for less adaptive strategies. Navigating realistic application environments decreases recall and factuality relative to sandboxed local settings.

Limitations and Directions

The benchmark comprises 15 tasks with 114 expert-injected insights across three industries, and currently lacks video/image modalities or extensive cross-domain integration. Planned expansions include multimodal file support, privacy-aware scenario design, and hybrid human-LLM rating of outputs.

3. DRBench for Reference Hallucination and Citation Reliability

DRBench (Rao et al., 3 Apr 2026) is constructed as a large-scale testbed to systematically analyze citation reliability in output from deep research agents and retrieval-augmented LLMs. It is composed of 100 expert-level queries (in English and Chinese) spanning Finance, Science, and Technology, each issued to 23 models (10 evaluated for URL liveness), generating over 53,000 unique citation URLs.

Reliability Metrics

• Non-resolving Rate: Proportion of URLs returning HTTP errors or timeouts.
• Hallucination Rate: Fraction of non-resolving URLs with no previous existence as per Wayback Machine snapshots.
• Stale URL Rate: Fraction of non-resolving URLs with an archived snapshot (link rot, not fabrication).

Formally,

$$\arg\max_a\,p^0_n(a) = \arg\max_a\,p^j_n(a) \neq a_n^{\mathrm{gt}}$$7

• Non-resolving: $$\arg\max_a\,p^0_n(a) = \arg\max_a\,p^j_n(a) \neq a_n^{\mathrm{gt}}$$8–$$\arg\max_a\,p^0_n(a) = \arg\max_a\,p^j_n(a) \neq a_n^{\mathrm{gt}}$$9
• Hallucination: $j \in \{1,\dots,J\}$0–$j \in \{1,\dots,J\}$1

Notably, some GPT-4.1 search-augmented models produce exclusively hallucinated non-resolving URLs, while others (e.g., deep research agents) present a significant fraction of stale, non-hallucinated failures.

Taxonomy and Mitigation

Failure analysis partitions non-resolving links as either hallucinated (never existed) or stale (previously existed, now dead). The open-source “urlhealth” tool supports automated status determination via a combination of HTTP checks and Wayback Machine queries.

Embedding urlhealth in an agentic self-correction loop yields drastic reductions in non-resolving links (e.g., GPT-5.1 from $j \in \{1,\dots,J\}$2 after correction), with typical final live rates approaching $j \in \{1,\dots,J\}$3–$j \in \{1,\dots,J\}$4 per system. Correction effectiveness and iteration count vary by model architecture.

Domain Variation

Citation reliability is highly domain-dependent, with ExpertQA benchmarks showing business fields at $j \in \{1,\dots,J\}$5 non-resolving versus $j \in \{1,\dots,J\}$6 in theology—suggesting model and domain-specific content rot effects.

4. DataRaceBench and DataRaceBench-ML

While outside the AI agent and citation context, DataRaceBench ("DRBench") (Chen et al., 2023) is a widely adopted microbenchmark suite for systematic evaluation of data race detection algorithms in parallel C/C++ programs. It supports both static and dynamic tool evaluation, now also serving the machine learning and LLM-based code analysis community through its augmentation as DataRaceBench-ML.

Design and Growth

Originating in 2017, the benchmark contains 181 concise C/C++ kernels, each annotated with:

• Race diagnosis (yes/no)
• Detailed source-code locations and variables
• Ground-truth root cause (e.g., critical section mismatch, lock misuse, omitted barriers)

Recent versions add new edge-case races, e.g., DRB193, featuring races triggered by critical-section name mismatches.

Machine Learning Extension: DRB-ML

DRB-ML reformulates each test as a machine learning–ready JSON record with

• Full and trimmed code
• Explicit race classification (“data_race”: 0/1)
• Structured access pairs (variable names, lines, columns, read/write labels)
• Prompt–response pairs enabling LLM-based fine-tuning for supervised race localization and explanation

Formalization and Metrics

The canonical HPC data race definition—overlapping accesses (one write, unordered via happens-before)—is adopted. Tool evaluation is oriented around precision, recall, and F1-score for both detection and localization.

Use Cases

DRBench and DRB-ML serve as the principal validation suite for race detectors and, increasingly, as a pretraining/evaluation resource for code-understanding LLMs in HPC domains.

5. Comparative Table: Major DRBench Instantiations

Domain/Context	Purpose	Notable Metrics/Protocols
LVLM Robustness (Tang et al., 8 Mar 2026)	Diagnose/model-specific language bias/sensitivity	LB, LS, DRS; multi-round image/prompt perturbations
Enterprise Deep Research (Abaskohi et al., 30 Sep 2025)	Evaluate AI research agents in real-world business scenarios	IR, DA, F, Q, composite HM; multi-source reports
Citation Reliability (Rao et al., 3 Apr 2026)	Quantify and correct hallucinated/stale citations in LLM outputs	Non-resolving, Hallucination, Stale URL rates; urlhealth
Data Race Detection (Chen et al., 2023)	Benchmark static/dynamic race detectors, LLMs	Precision, Recall, F1; variable-pair localization

6. Impact and Future Challenges

The multiple DRBench initiatives set modern standards for evaluation fidelity:

• Model-specific, failure-targeted protocol design (LVLMs) enables dynamic adaptation as models evolve.
• Multi-modality and real-world complexity (enterprise research) reveal agentic and planning limitations beyond synthetic QA tasks.
• At-scale citation reliability testing surfaces critical reliability challenges for applied LLMs, with practical mitigation via agentic correction and tool integration.
• Rigorous source-code benchmarking grounds progress in program analysis and catalyzes LLM advances in high-stakes HPC contexts.

Limitations across DRBench instances include coverage constraints (cycle of synthetic tasks, lack of multimodal sources), dependence on automated or LLM-derived metrics, and evolving relevance as new architectures emerge. Planned expansions involve richer source integration, privacy-aware scenarios, and broader cross-domain applicability.