AlphaResearchComp: LLM Algorithm Discovery

• AlphaResearchComp is a benchmark and evaluation platform that tests LLM-driven research agents in discovering novel algorithms through dual evaluation environments.
• It employs execution-based verification alongside a simulated peer-review system to ensure both functional correctness and research innovation.
• The platform demonstrates competitive performance against human baselines across diverse algorithmic challenges, highlighting the potential of autonomous AI research.

AlphaResearchComp is a benchmark and evaluation platform designed to systematically assess the capabilities of LLM–driven, autonomous research agents in discovering new algorithms for open-ended mathematical and computational problems. It provides a dual-environment evaluation framework, rigorous programmatic and simulated peer-review vetting, and a curated suite of diverse algorithmic challenges. AlphaResearchComp serves as the testbed for the AlphaResearch agent, which demonstrated the potential for LLM-guided discovery of novel algorithms that can outperform strong human and baseline benchmarks in selected cases (Yu et al., 11 Nov 2025).

1. Benchmark Architecture and Dual Environment

AlphaResearchComp is structured to address both the feasibility and innovation of algorithm discovery in LLM agents. The architecture comprises:

• Execution-Based Verification: Every candidate solution is executed against problem-specific evaluation scripts to check constraint adherence and compute a scalar performance metric $r$. Statistically sound program-based pipelines ensure that only functionally correct and precisely measured outputs are accepted.
• Simulated Peer-Review Environment: To capture qualitative innovation, a reward model (AlphaResearch-RM-7B, fine-tuned on ICLR 2017–2024 reviewer records) is used. Each proposed research idea $i$ is rated with reviewer-style scores (0–10); proposals failing to meet a threshold (5.5/10) are rejected without consuming computational resources on code generation.

This dual-environment design enables both strict enforceability of technical constraints and filtering for research originality or insight.

2. Problem Suite and Formal Evaluation Protocols

AlphaResearchComp encompasses eight open-ended algorithmic problems, spanning continuous and discrete optimization, combinatorial construction, and signal processing. Each problem is specified with formal mathematical notation, exact programmatic evaluation, and best-known human (or literature) baselines:

Problem	Metric $r$	Human Baseline	Best AlphaResearch
Packing circles (n=26)↑	$\sum r_i$	2.634	2.636
Packing circles (n=32)↑	$\sum r_i$	2.936	2.939
Max–min distance (n=16)↓	$\max_{i<j} / \min_{i<j}$	12.89	12.92
Spherical code (n=30)↑	$\min_{i<j}\arccos\,\langle p_i,p_j\rangle$	0.67365	0.6735
Littlewood poly (n=512)↓	$1/\|P\|_\infty$	1/32	1/32
Third autocorr. ineq.↓	Peak ratio	1.4581	1.546
Autoconv. peak min.↓	Peak ratio	0.755	0.756
MSTD (n=30)↑	$|A+A|/|A-A|$	1.04	1.04

Notation: “$\uparrow$” indicates higher is better; “$\downarrow$” means lower is better. The Excel@Best metric reports relative excess over the human baseline.

Problems are chosen for their open-endedness, strict objective quantifiability, and verifiable programmatic assessment.

3. Iterative Research Agent Protocol

The AlphaResearch agent operates through an iterative loop involving state sampling, idea generation, peer-review filtering, code synthesis, execution, and performance-driven optimization:

1. Trajectory Sampling: Sample prior tuples $(i_t, p_t, r_t)$ from agent history.
2. Idea Generation: Propose a new research idea $i_k$, conditioned on current trajectory.
3. Peer Review: If $\mathrm{RM}(i_k)$ < 5.5, skip idea; else, proceed.
4. Code Generation and Execution: Generate new program $p_k$, execute in evaluation environment $\mathcal{E}$, compute performance $r_k$.
5. Bookkeeping: If $r_k > r_{\mathrm{best}}$, update best result; append $(i_k, p_k, r_k)$ to history.
6. Termination: Repeat until either surpassing the best-known human result or reaching a preset iteration bound.

This reinforcement protocol synergizes proposal creativity with strict feasibility and avoids stagnation on non-innovative ideas or incorrect code. Ablation studies show that the dual-environment agent discovers high-quality solutions more efficiently than execution-only agents.

4. Metrics, Reproducibility, and Reward Model

All evaluations rely on fully reproducible, program-based pipelines: candidate programs are executed in isolated environments, with outputs parsed and ranked via objective metrics. For innovation, the reward model is a Qwen2.5-7B-Instruct model fine-tuned to output (human-like) peer review scores. Empirical validation on the ICLR 2025 holdout set yields 72.0% reward model accuracy, exceeding human reviewer agreement (65.0%).

The primary global metric, excel@best, is defined as

$$\mathrm{excel@best} = \mathbb{E}_{\text{problems}}\left[\frac{\left|r_\text{best} - r_\text{human}\right|}{r_\text{human}}\text{ sign}(d) \right],$$

where $d=+1$ for maximize, $d=-1$ for minimize. Positive values indicate algorithmic discovery outperforming the human literature.

5. Results and Comparative Analysis

AlphaResearchComp’s canonical round led to the following findings:

• Best-known across problems: AlphaResearch outperformed both human experts and strong recent baselines (e.g., AlphaEvolve) in packing circles ($n=26,32$), achieving new top values (+0.32% and +0.10% over human records, respectively).
• Human parity: Equaled human best in four additional problems (e.g., Littlewood polynomials, MSTD set discovery).
• Marginal deficit on remaining: For the remainder, performance was within 0.01–6% of the best-known human/professional values.
• Filter efficiency: The reward model filtered ≈30% of unverifiable or low-value ideas; retrospective analysis indicates ≈71.5% of rejections were appropriate (the remainder included 43 viable—but rejected—solutions).

The aggregate outcome was a 2/8 "win" rate in head-to-head problem comparison, with superior convergence efficiency for the dual-reward environment.

6. Implications, Limitations, and Future Directions

AlphaResearchComp demonstrates that LLM-driven research agents, when coupled with executable verification and simulated peer-review filtering, can autonomously discover nontrivial, sometimes state-of-the-art, solutions in problems with strict objective and innovation requirements. However, key limitations include:

• Missed viable ideas: Reward-model filtering, while efficient, discards a subset of promising solutions.
• Stagnation on complex or low-innovation domains: For problems where the solution space is highly constrained, improvement beyond human achievements is minimal.
• No domain-specific transfer or meta-learning: The framework treats each problem independently, with no mechanism for cross-problem algorithmic abstraction.
• Dependence on code execution reliability and restricted problem domains: All problems must admit unambiguous evaluation scripts.

A plausible implication is that future autonomous research workflows would benefit from combining execution-based and linguistic/semantic models, but require adaptive, context-aware reward shaping and more granular control over research trajectory branching.

7. Significance for AI Research and Algorithmic Discovery

AlphaResearchComp sets a methodological precedent for evaluating LLM assistants in open-ended scientific discovery. It integrates rigorous programmatic validation with a theoretically grounded, data-driven approach to simulated novelty assessment. As LLM capabilities increase, benchmarks of this type will be crucial both for measuring genuine progress in autonomous science and for partitioning agent gains into those emergent from reasoning, search, synthesis, or language-based generalization. The paradigm informs the design of next-generation scientific discovery agents, with implications for automated theorem proving, chemical/materials search, and complex engineering optimization—all contexts where feasibility and innovation must be jointly verifiable.