DiscoBench: Multidomain Evaluation Suite

• DiscoBench is a composite evaluation suite featuring three benchmarks for discourse-aware language modeling, algorithm discovery agents, and clarification-aware deep search.
• It employs tailored datasets, standardized protocols, and diagnostic tests to assess document-level cohesion, multi-domain algorithm performance, and interactive ambiguity resolution.
• Empirical analyses reveal that domain-specific pretraining and strategic clarification significantly outperform large general-purpose LLMs, underscoring the need for specialized evaluation methods.

DiscoBench refers to three distinct evaluation suites in contemporary research, each addressing critical gaps in robust model assessment for LLMs, algorithm discovery agents, and clarification-aware search agents. Though sharing a common ethos in rigorous multi-faceted evaluation, these DiscoBench benchmarks target fundamentally different challenges in natural language processing, machine learning, and interactive agent reasoning. The following sections provide a comprehensive account of each DiscoBench and their impact across their respective domains.

1. DiscoBench for Discourse-Aware Language Modeling

Motivation and Scope

Disco-Bench, introduced by Wang et al., addresses the paucity of evaluation suites measuring document-level discourse competence in NLP. Unlike prior benchmarks—such as GLUE, CLUE, and XGLUE—that focus primarily on sentence-level or limited inter-sentence properties, Disco-Bench probes cohesion and coherence phenomena across literary texts. Targeted phenomena include anaphora, coreference, zero pronoun recovery, lexical repetition, synonymy, collocation, discourse relations, and tree-structured coherence (e.g., RST) (Wang et al., 2023).

Dataset Structure

Disco-Bench consists of nine document-level test sets drawn exclusively from the literature domain (novel, classical, and poetry), in Chinese (zh), English (en), or bilingual classical→modern Chinese (czh→mzh). Tasks are as follows:

Category	Task (Acronym)	Language(s)	Description	Key Metrics
Understanding	SI	zh	Speaker identification in paragraphs	F1, EM
Understanding	ZPR	zh	Zero pronoun recovery (30 forms)	micro-F1, P, R
Understanding	MRC	mzh+czh	MCQ reading comprehension (5–10 sentence context)	Accuracy
Translation	NT	zh→en	Novel translation (entity/anaphora consistency)	d-BLEU, BLEU, TER, METEOR, COMET
Translation	CCT	czh→mzh	Classical→modern Chinese	BLEU, TER, COMET
Translation	PT	zh→en	Poetry translation (Shi, Ci, Qu, Fu)	BLEU, TER, METEOR, COMET
Generation	TE	en	Text expansion from parse-induced skeletons	BLEU, PPL
Generation	TI	zh	Text infilling (predicting missing sentence)	BLEU, PPL, BERTScore, Dist-2/4
Generation	TC	zh	Text completion (long context generation)	[as TI]

Diagnostic Test Suite

A hand-crafted contrastive suite contains 1,294 instances targeting six cohesion categories: repetition, synonyms, ellipsis, substitution, reference, and conjunction. Each includes a correct and a perturbed candidate, with models evaluated by forced-choice preference ($$P(x_{correct} \mid \text{context}) > P(x_{incorrect} \mid \text{context})$$).

Evaluation Protocol

Standardized train/validation/test splits are provided. All models are fine-tuned on train+dev and evaluated on test. Automatic and diagnostic metrics are reported for holistic coverage:

• Understanding: F1, EM, accuracy.
• Translation: document-level BLEU, BLEU$_4$, TER, METEOR, COMET.
• Generation: BLEU, BERTScore, perplexity (PPL), Dist-2/4.
• Diagnostic: forced-choice accuracy.

Empirical Findings

Fine-grained, domain-specific pretraining on the Disco-Bench corpus consistently improves discourse knowledge acquisition (e.g., RoBERTa(large) ZPR F1 33.0→34.3; BART(large) TE BLEU 33.8→36.2). Large general-purpose LLMs (e.g., GPT-4) underperform compared to in-domain models on cohesion-sensitive tasks, and commercial LLMs rank below 50% on most cohesion types in the diagnostic suite, underscoring lasting challenges in discourse modeling. Document-level pretraining emerges as a robust strategy for capturing cross-sentential phenomena (Wang et al., 2023).

2. DiscoBench for Algorithm Discovery Agents

Background and Motivation

DiscoBench, by Goldie et al., is an evaluation subset derived from DiscoGen—a procedural generator spanning millions of combinatorially varied algorithm discovery tasks in machine learning. Its primary purpose is the principled measurement of algorithm discovery agents (ADAs), circumventing prior limitations such as task contamination, poor train/test separation, and limited domain diversity (Goldie et al., 18 Mar 2026).

Benchmarked Task Taxonomy

DiscoBench encompasses approximately 75 hand-designed tasks, structured as follows:

• For each ML domain (10 total, e.g., RL, computer vision, Bayesian optimization), for $m$ editable modules (e.g., optimizer, loss):
  • $m$ "Single" tasks (only one module editable) and one "All" task (all modules editable).
  • Standardized meta-train/meta-test dataset splits for rigorous comparability.
  • Prohibits ADA optimization on DiscoBench tasks to ensure generalization assessment.

Illustrative task domains and modules:

Domain	Editable Modules	Meta-train Datasets	Meta-test Datasets
Bayesian Optimization	surrogate model, acquisition, etc.	Ackley1D, Branin2D, etc.	Ackley2D, Drop-Wave2D, Griewank5D, etc.
Computer Vision	loss, optimizer, network, preprocessing	CIFAR-10, MNIST, etc.	CIFAR-100, TinyImageNet, etc.
RL (on-policy)	loss, optimizer, network, train-loop	Breakout, Freeway	Asterix, SpaceInvaders

Task Configuration Parameters

Each task is specified as a configuration over:

1. Domain
2. Editable modules ($m$)
3. Meta-train datasets
4. Meta-test datasets
5. Backend (e.g., recurrent)
6. Evaluation type (performance, efficiency, speed)
7. Initialization (“empty” or “baseline”)

Evaluation Methodology

• Inner-loop: Algorithm is trained on individual datasets.
• Meta-loop: ADA iteratively proposes algorithms, guided by inner-loop feedback over meta-train data.
• ADA evaluation: Performance on unseen meta-test data for each task.
• Aggregation: Elo rating across datasets, success@3 (≥1 successful run in 3 seeds), and task-specific metrics.

Experimental Results

Tasks with all modules editable are substantially harder; baseline Elo meta-test is 1377±383, with the leading LLM-powered agents performing significantly lower (e.g., Deepseek v3.2, 940±362). Single-module tasks feature higher success rates (e.g., Deepseek v3.2, 80.0%; GPT-OSS 68.2%). Statistical analysis confirms high diversity—mean Spearman $r\approx 0.4$ across tasks, vanishing correlation between meta-train and meta-test when datasets change. Prompt optimization on procedural DiscoGen tasks measurably improves performance on DiscoBench (Goldie et al., 18 Mar 2026).

A plausible implication is that current LLM-based ADAs lack generalization robustness required for open-ended, multi-domain algorithm discovery, especially when simultaneously optimizing multiple code modules.

3. DiscoBench for Clarification-Aware Deep Search

Motivation and Scope

Traditional deep search benchmarks, such as GAIA and AgentBench, unrealistically presuppose fully specified, error-free user queries. DiscoBench, as introduced by an unnamed group (Tao et al., 26 Jun 2026), introduces controlled ambiguity into multi-hop information-seeking environments, evaluating whether LLM-powered agents can:

1. Proactively detect ambiguity,
2. Ask targeted clarification questions, and
3. Recover correct reasoning trajectories following user guidance.

Dataset and Ambiguity Taxonomy

DiscoBench comprises 211 multi-hop samples (2–5 checkpoints each) across 11 domains (Film/TV, Games, Sports, Geography, Academic, Music, Art, Medicine, Finance, Technology, Policy/Law). Across these, 463 ambiguity instances are injected at precise “checkpoints,” stratified by difficulty.

Ambiguity types include:

1. Entity Ambiguity: Underspecified references requiring resolution among alternatives.
2. Factual Inaccuracy: Incorrect premises necessitating correction.
3. Version Ambiguity: Temporal/logical uncertainty (e.g., which instance or edition).
4. Criteria Ambiguity: Unspecified selection criteria (e.g., "top three cities" by which metric or scope).

Interaction Protocol

Agents interact in a turn-based framework with three permitted actions per checkpoint: Search (web query), Ask (clarification), or Answer (propose solution). A deterministic user simulator provides pre-specified disambiguation only for appropriately targeted Asks. Clarification is mandatory at ambiguous checkpoints; otherwise, successful Search→Answer suffices.

Evaluation Measures

• Task Utility: End-to-end accuracy and checkpoint pass rate.
• Ambiguity Detection: Accuracy, precision, recall, F1 on whether an Ask was invoked at ambiguous checkpoints.
• Clarification Effectiveness: Fraction of Asks that correctly resolve ambiguity.
• Cost Efficiency: Task utility normalized by interaction cost, parametrized by user-tunable $\lambda$.

Empirical Analysis

Under neutral prompting (no hint), leading LLM agents attain ≤43.1% end-to-end accuracy, with ambiguity detection F1 typically in the 45–70% range. Correctly targeted Ask actions (CE-A) are often >85%, yet timely and strategic invocation remains lacking (CE-B). Behavioral modes include DirectGuess, SearchHeavyGuess, and SearchThenAsk, with the latter yielding the highest checkpoint pass rate (~93.4%). Repeated blind searching is inferior to direct asking when ambiguity is present. Guided prompting improves both F1 and end-to-end accuracy; however, agents continue to frequently fail at recognizing ambiguous contexts.

A key insight is that ambiguity detection and clarification are dissociable competencies. High search intensity does not substitute for efficient, strategic clarification; models lacking dynamic uncertainty estimates are susceptible to cascading errors in deep-search scenarios (Tao et al., 26 Jun 2026).

4. Comparative Synthesis Across DiscoBench Variants

The three research efforts independently named “DiscoBench” reflect a convergent need for multidimensional, stress-testing benchmarks in the era of advanced AI agents and systems. Each targets:

• Latent, cross-cutting challenges (discourse, algorithmic creativity, ambiguity resolution) not sufficiently addressed by prevailing benchmarks.
• Rich data and protocol structures to rigorously isolate specific failure modes and performance dimensions.
• Comprehensive metric suites spanning automatic scores, contrastive diagnostics, Elo/task-level aggregation, and cost-sensitive interaction evaluation.

By employing complex, hand-crafted, or procedurally generated datasets and suites, these benchmarks instantiate best practices—clear train/test separation, avoidance of leakage, and cross-domain generalizability—for the principled assessment of learning and reasoning agents.

5. Impact and Open Directions

DiscoBench benchmarks have had a marked impact on shaping evaluation strategies in their respective areas. For language modeling, document-level and diagnostic probing have challenged the adequacy of token-level or n-gram metrics for discourse-sensitive tasks, spurring further work on pretraining and adaptation strategies. In algorithm discovery, procedural generation of tasks and hand-designed benchmarking have exposed persistent generalization limits of current LLM ADAs, motivating meta-meta-learning, autocurricula, and world model approaches (Goldie et al., 18 Mar 2026). Finally, clarification-aware benchmarks for deep search have revealed the disconnect between retrieval competence and interactive problem-solving, motivating further integration of online ambiguity estimators, clarification planners, and joint retrieval–clarification policies (Tao et al., 26 Jun 2026).

DiscoBench thus exemplifies the movement toward realistic, adversarially designed, and cross-disciplinary benchmarks that not only evaluate aggregate performance but also diagnose and spur progress on the most challenging open questions in machine learning and AI.