DiscoGen: ADA Benchmarking Framework

• DiscoGen is a procedural generation framework enabling automated creation of millions of unique, specification-driven machine learning tasks.
• It improves ADA evaluation by eliminating overfitting, task contamination, and saturation through dynamic meta-train/meta-test splits.
• The framework supports diverse domains and modular configurations, encouraging systematic research in meta-learning, AutoML, and AI agent optimization.

DiscoGen is a procedural generation framework for large-scale benchmarking and optimization of algorithm discovery agents (ADAs) in machine learning. Unlike conventional task suites that rely on a fixed collection of hand-crafted problems, DiscoGen enables automated creation of millions of unique, specification-driven tasks that challenge an ADA to design or improve core ML pipeline components—such as loss functions, optimizers, or neural architectures—across multiple domains. By design, DiscoGen addresses critical bottlenecks in previous ADA evaluation regimes, such as poor generalization, contamination risk, and task saturation, thereby providing a robust foundation for research at the intersection of meta-learning, automated machine learning, and AI agent optimization (Goldie et al., 18 Mar 2026).

1. Motivation and Limitations of Existing Algorithm Discovery Benchmarks

Prevailing task suites in ADA research—including MLGymBench, MLE-Bench, and MLAgentBench—comprise only tens or hundreds of manually constructed problems. These suffer from:

• Overlap between discovery (meta-train) and evaluation (meta-test) sets, inducing overfitting and leaking evaluation difficulty;
• Template-centric workflows that bias agents toward rote code completion or software engineering, instead of genuine algorithmic innovation;
• Static, public task sets that permit pre-training or data contamination, undermining fair generalization assessment;
• "Saturated" or nearly-solved problems that contribute minimal learning signal.

These deficiencies together constrain not only the scale but also the authenticity of ADA evaluation and progress. DiscoGen's procedural foundation confronts these issues by enabling essentially unbounded, non-overlapping, and customizable task instantiation (Goldie et al., 18 Mar 2026).

2. Architecture and Task Generation Pipeline

DiscoGen is implemented as a Python library that, when provided a concise configuration, generates a new ADA discovery task. Task construction comprises:

• Task Domains: Defines the primary problem type and code scaffold—such as On-Policy Reinforcement Learning, Image Classification, Language Modeling, or Bayesian Optimization—each supporting a suite of interchangeable modules.
• Editable Modules: Specified code components accessible to the ADA (e.g., loss function, optimizer, architecture, training loop); all other code is fixed.
• Meta-Train vs. Meta-Test Splits: Datasets are partitioned so that meta-test codebases are generated only after the meta-train phase, eliminating train-test leakage by construction.
• Backends: Optional implementation variants (e.g., policy head type in RL, model classes in LM), multiplying the combinatorial task space.
• Evaluation Types: Task metrics can be set to performance, energy to threshold, or time to solution, selected independently per instantiation.
• Initializations: Tasks may start with only function signatures (empty initialization) or a working baseline (facilitating both de novo and incremental discovery).

The overall ADA meta-learning protocol comprises repeated inner-loop optimization by the ADA across sampled meta-train environments, followed by evaluation on held-out meta-test suites, and is itself iterated as part of a "meta-meta" workflow for ADA optimization (Goldie et al., 18 Mar 2026).

3. Task Parameterization and Complexity Control

The dimension and diversity of the task space is rigorously specified by discrete parameters:

• $m$: Number of editable modules in the pipeline.
• $d$: Number of available datasets per task domain.
• $b$: Number of backend variants.
• 2 initializations $(\text{empty, baseline}) \times$ 3 evaluation types.

Dataset assignments to meta-train or meta-test yield $3^d$ combinations; module editability is $2^m$. Excluding degenerate cases (no editable modules, no train/test datasets), the full task count per domain is

$$N_{tasks} = 2 \cdot 3 \cdot b \cdot (2^m-1) \left(3^d - 2^{d+1} + 1\right)$$

In model unlearning variants, dataset combinatorics are further extended. Difficulty is controlled primarily by $m$: tasks with more editable modules (greater algorithmic degrees of freedom) are empirically harder for ADAs, with observed success rates decreasing rapidly as $m$ increases from 1 to 4. Task variety occurs via orthogonal axes: domains, datasets, modules, backends, and evaluation types. Rank-correlation analyses across task outcomes (mean Spearman $\rho\approx0.4$) reveal a diversity spectrum from highly correlated to anti-correlated pairs, confirming the richness and non-redundancy of the generated suite (Goldie et al., 18 Mar 2026).

4. Task Instantiation and Benchmarking Methodology

DiscoGen can generate, for example:

• On-Policy RL: "Develop a new loss function and network architecture for a JAX PPO agent on Breakout/Freeway (train), test on Asterix/SpaceInvaders (test)." Editable module signatures are provided.
• Image Classification: "Discover a novel loss for ResNet-18 on CIFAR-10/Flowers (train), evaluate on CIFAR-100/CIFAR-10-C (test)." Modular code snippets evaluate, e.g., $d$0.
• Bayesian Optimization: "Design surrogate model, acquisition, and optimizer for 2D Branin (train), test on Drop-Wave and Griewank (test)."
• Language Modeling: "Improve a JAX LLM’s cross-entropy loss, test on TinyStories (train) vs. OPC-FineWebCode (test)."

Random sampling over modules/datasets/backends yields tasks spanning trivial parameter modification to open-ended algorithm discovery.

To facilitate fair, consistent evaluation detached from ongoing optimization, DiscoBench defines a fixed, hand-picked subset: for each domain, $d$1 tasks (one per editable module, one "all modules editable") with rigorously partitioned meta-train/meta-test datasets. Only previously unseen ADAs and prompt variations may be fairly benchmarked on DiscoBench (Goldie et al., 18 Mar 2026).

5. Evaluation Metrics and Empirical Results

ADA performance on DiscoGen tasks and DiscoBench is assessed via:

• Success Rate: Fraction of runs yielding a valid, compiling, and interface-correct solution.
• Elo Rating: Bootstrapped score from pairwise win/draw/loss comparisons of agents, penalizing unreliable solutions.
• Success@3: At least one successful run in three random seeds.
• Time/Energy: Secondary metrics optionally tracked per task.

On DiscoBench, benchmarking three open-source LLM-based ADAs (Deepseek-v3.2, Devstral2, GPT-OSS 120B) revealed that Deepseek-v3.2 achieved the best Elo on "Single" tasks (Meta-Train 1079 [1050–1108], Meta-Test 1053 [1020–1082]) and the highest success rate (80%). Performance on "All" tasks (all modules editable) is markedly lower (success < 35%, Elo < 1200), which evidences the inherent hardness of multi-component algorithm discovery (Goldie et al., 18 Mar 2026).

Prompt optimization experiments (e.g., prompt evolution using Claude 4.5 Sonnet) show that systematic prompt tuning across randomized DiscoGen samples monotonically improves meta-test Elo (957 → 1071) and overall success (70.6% → 78.7%), with performance gains robustly confirmed by inner-loop metrics.

6. Applications and Future Research Enabled

DiscoGen unlocks several high-impact lines of inquiry in ADA and meta-learning:

• Pathology Analysis: Systematic characterization of ADA failures (e.g., failure mode distributions, spurious solutions) across vast task variation.
• Meta-Meta Learning: Training and adaptation of ADAs themselves via serialized or curriculum-based meta-training on procedurally generated tasks, analogous to LLM fine-tuning for downstream reasoning.
• Autocurricula & Curriculum Learning: Regret-based or learnability-weighted adversarial sampling to dynamically construct hard-yet-learnable task sequences.
• Algorithm World Models & LLM-Judges: Development of predictive models over algorithm performance for informed agent search or synthetic value estimation.
• Symbolic / Evolutionary Algorithms: Application of program templates and modular code scaffolds for hybrid search (genetic programming, neuroevolution).
• Self-Improving Agents: Enabling ADAs to recursively adapt prompts or code templates in a Gödelian self-modification loop, sampling new DiscoGen tasks to circumvent degenerate overfitting.
• Tooling: Provision of procedural-scale diagnostics (e.g., static analysis for hard-coded or identity exploits) for in-depth research on ADA generalization and creativity (Goldie et al., 18 Mar 2026).

7. Accessibility, Usage, and Extensibility

DiscoGen is publicly available at https://github.com/AlexGoldie/discogen. Installation requires a suite of core libraries, including pip install discogen gymnax mlbench nle brax jax jaxlib flax transformers. Key usage workflows include:

• Sampling a random task via the Python API, building meta-train/meta-test splits, and launching ADA meta-learning loops.
• Domain-uniform sampling to avoid overpopulation of the task space by large-parameter domains.
• Benchmarked evaluation via provided DiscoBench protocols.
• Extending the framework with new domains by populating domain modules, editable component definitions, and dataset logic.

DiscoGen’s design—rigorous dataset/codebase partitioning, combinatorial task space, and modular extensibility—establishes it as the reference platform for genuinely reproducible, scalable, and creative algorithm discovery research in machine learning (Goldie et al., 18 Mar 2026).