AutoSOTA: An End-to-End Automated Research System for State-of-the-Art AI Model Discovery

Abstract: Artificial intelligence research increasingly depends on prolonged cycles of reproduction, debugging, and iterative refinement to achieve State-Of-The-Art (SOTA) performance, creating a growing need for systems that can accelerate the full pipeline of empirical model optimization. In this work, we introduce AutoSOTA, an end-to-end automated research system that advances the latest SOTA models published in top-tier AI papers to reproducible and empirically improved new SOTA models. We formulate this problem through three tightly coupled stages: resource preparation and goal setting; experiment evaluation; and reflection and ideation. To tackle this problem, AutoSOTA adopts a multi-agent architecture with eight specialized agents that collaboratively ground papers to code and dependencies, initialize and repair execution environments, track long-horizon experiments, generate and schedule optimization ideas, and supervise validity to avoid spurious gains. We evaluate AutoSOTA on recent research papers collected from eight top-tier AI conferences under filters for code availability and execution cost. Across these papers, AutoSOTA achieves strong end-to-end performance in both automated replication and subsequent optimization. Specifically, it successfully discovers 105 new SOTA models that surpass the original reported methods, averaging approximately five hours per paper. Case studies spanning LLM, NLP, computer vision, time series, and optimization further show that the system can move beyond routine hyperparameter tuning to identify architectural innovation, algorithmic redesigns, and workflow-level improvements. These results suggest that end-to-end research automation can serve not only as a performance optimizer, but also as a new form of research infrastructure that reduces repetitive experimental burden and helps redirect human attention toward higher-level scientific creativity.

• The paper introduces an eight-agent system that grounds AI papers in executable resources, reproduces reported baselines, generates optimization ideas, and validates improvements under explicit integrity constraints.
• AutoSOTA exceeded the original reported performance on all 105 selected papers, averaging about 10% improvement in roughly five hours per paper, although the benchmark excludes costly, theoretical, or unusable repositories.
• The system finds gains through bug fixes, data cleaning, architectural changes, sampling, and test-time optimization, while case studies show that higher quality can require substantial increases in inference cost and still depend partly on LLM-based auditing.

AutoSOTA is an end-to-end multi-agent system that takes a research paper from a top-tier AI conference as input and produces an executable repository whose empirical performance surpasses the method reported in the paper. The system is positioned as an instantiation of the broader OmniScientist vision of autonomous scientific discovery (Shao et al., 21 Nov 2025), but with a deliberately narrow and demanding target: the full pipeline from unstructured literature to validated state-of-the-art (SOTA) improvement. The authors formalize this as an open-ended discovery function $$\mathcal{F}: \mathcal{P} \rightarrow \mathcal{R}^{*}$$ mapping a multi-modal paper $\mathcal{P}$ to an improved repository $\mathcal{R}^{*}$ satisfying $$\text{eval}(\mathcal{R}^{*}) > \text{eval}(\mathcal{R}_{rep})$$, where $\mathcal{R}_{rep}$ is a faithful reproduction of the original baseline.

Problem formulation

The task is decomposed into four coupled sub-problems. Resource preparation grounds the paper in concrete artifacts: repository $\mathcal{C}$, datasets $\mathcal{D}$, base models $\mathcal{M}$, and a target metric $g^*$. Experiment evaluation synthesizes a runnable reproduction such that $\text{eval}(\mathcal{R}_{rep}) \approx g^*$, which the authors characterize as a long-horizon reasoning-and-acting problem under partial observability, plagued by missing scripts, broken dependencies, and hallucinated bug-fixing loops. Code optimization translates abstract ideas into executable candidate repositories, and reflection and ideation performs open-ended hypothesis generation under strict validity constraints. A central claim of the framing is that prior systems—AlphaEvolve (Novikov et al., 16 Jun 2025), FunSearch, AutoResearch—either operate on bounded code modules within sanitized sandboxes or bypass environment setup entirely; AutoSOTA's contribution is closing this gap end-to-end.

Architecture: eight specialized agents

The architecture distributes the workload across eight agents organized into three macro-modules, with evaluation and reflection running as a tightly coupled iterative loop rather than a sequential pipeline.

In resource preparation, AgentResource performs paper-to-repository grounding over papers from eight 2025 conferences (ICML, ICLR, NeurIPS, AAAI, NAACL, ICCV, ACL, CVPR), using an LLM classifier for methodological typing, LLM-based repository selection, and a permissive readiness assessment based on abstracts, file trees, and README prefixes. A decoupled two-phase external acquisition module first builds a symbolic Global Resource Registry under a zero-download constraint, then executes physical downloads with size gating, polymorphic retrieval (rule-based HuggingFace/HTTP modes plus LLM-generated custom scripts), and breakpoint resumption. AgentObjective constructs tree-structured evaluation rubrics via BFS decomposition with weight-conserving termination, supported by tiered on-demand context injection (shallow strategic, intermediate technical, deep implementation layers) that injects PDF content, VLM-extracted table/figure evidence, and repository-level logic at appropriate depths.

In experiment evaluation, AgentInit maps the grounded task tuple into an initial executable state through a preset workflow (environment construction, dependency resolution, command discovery) backed by a skill library including Repo2Run-based setup, manual Docker fallback, and protocol-preserving glue-logic synthesis. AgentMonitor operates outside the execution loop, parsing streamed traces to detect dead-end trajectories and issuing high-level supervisory actions (continue, fallback, terminate, rollback), while maintaining persistent external memory in three documents: code_analysis.md, idea_library.md, and research_report.md. AgentFix implements retrieval-before-repair: failures are normalized into reusable signatures matched against structured repair skills (pip failures, network constraints, CUDA toolchain mismatches), with cross-paper failure memory converting one-off debugging episodes into procedural knowledge.

In reflection and ideation, AgentIdeator constructs a constrained hypothesis library categorized by granularity (PARAM/CODE/ALGO) and risk level, filtered by an admissibility predicate that rejects proposals violating evaluation integrity. AgentScheduler manages the optimization lifecycle across four phases—baseline measurement anchored to empirically measured rather than reported metrics, code understanding with explicit hard-constraint enumeration, idea library construction with red-line audit, and a seven-step iterative loop. Two mechanisms stand out: a bifurcation between a Normal Path and a forced Leap Path when the last three iterations were all PARAM-type adjustments (an anti-stagnation mechanism granting structural changes a five-iteration "honeymoon period" before rollback), and container-internal version control with atomic _best tag advancement. AgentSupervisor enforces a Red Line System of seven non-negotiable constraints covering metric parameterization, evaluation script integrity, prohibition of output fabrication, cross-metric trade-offs, dataset split integrity, dataset immutability, and paper-specific comparability requirements. Enforcement is layered across constraint enumeration during code analysis, post-ideation audits, Leap Path checks, and prompt-level absolute prohibitions.

Evaluation

The benchmark consists of exactly 105 method-driven papers surviving a four-stage filter: methodological relevance, public code availability, repository readiness, and an execution cost cap of 4 GPU hours per replication. This filtering is a substantive scope restriction—the evaluated distribution excludes theoretical work, unreleased artifacts, and expensive experiments—and all headline claims should be read against it.

The principal result is that AutoSOTA discovers new SOTA models surpassing the original reported methods for all 105 papers, averaging roughly five hours per paper and yielding an average improvement near 10%. Representative improvements span several orders of magnitude:

Paper	Conference	Improvement
Certified Unlearning for Neural Networks	ICML 2025	63.64%
Multi-Task Vehicle Routing Solver (MoSES)	NeurIPS 2025	51.40%
Tree Ensemble Explainability (TreeHFD)	NeurIPS 2025	36.60%
Circuit Stability Characterizes LM Generalization	ACL 2025	35.80%
Proxy-SPEX	NeurIPS 2025	30.30%
Least squares variational inference	NeurIPS 2025	0.02%

Cross-domain analysis over CV, ML, NLP, Optimization, and TimeSeries categories (with log-transformed gains spanning 3–33%) indicates consistent behavior, with the authors claiming lower variance than typical human-reported results. The claim that improvements were obtained "in all tested cases" is strong; it holds by construction only within the curated 105-paper set, and no failure or partial-success statistics outside this set are reported.

Case studies

Six case studies illustrate the qualitative range of interventions beyond hyperparameter tuning. In the FR-Spec speculative decoding case, AutoSOTA traced abnormal token acceptance to incorrect reuse of tree_draft_ids after verification—a latent implementation bug—and combined its removal with GPU-side terminal-token checking and reduced CPU–GPU synchronization, raising MT-Bench throughput from 674.58 to 765.27 tok/s (+13.44%). In the NLP case, gains came from replacing single-scale graph propagation $\mathcal{P}$0 with a six-step multi-scale resolvent composition plus edge-weight smoothing, improving moral-value correlation from 0.4577 to 0.4970. The biology case is notable for a data-hygiene discovery: AutoSOTA found a mutation-string formatting inconsistency limiting FoldX/StaB-ddG data intersection to 63%, fixed it to 100% row matching, and applied a 60/40 physics–ML ensemble, lifting per-interface Pearson correlation from 0.4897 to 0.5655 (+15.48%) in one iteration. The CV case introduced three-way CLS/mean/max pooling fusion in a ViT IQA pipeline (+2.68% PLCC). The time series case on Neural MJD combined antithetic variates, compute-aware sampling-size selection (rejecting values beyond 300), and loss reweighting to reduce S&P 500 MAE by ~19%, while explicitly discarding pseudo-optimizations that improved NLL but not MAE. The MoSES vehicle routing case achieved a 51.4% optimality-gap reduction through LoRA gating changes, test-time augmentation scaling from 8× to 128× (with autonomous batch downsizing to avoid OOM), and classical 2-opt/Or-opt post-processing—at the cost of increasing inference time from ~3.7 to ~103 seconds, a deliberate compute-for-quality trade-off the authors acknowledge openly.

Limitations and open questions

Several limitations are conceded or evident. First, the evaluation rubric pipeline currently emphasizes Result Match categories; the authors note this may overlook structural and semantic fidelity, deferring fuller use of the rubric tree as dense reward signals to future work. Second, the benchmark's filters (code availability, readiness, ≤4 GPU-hour replication cost) mean the reported success rate does not characterize behavior on the broader literature, on repositories without usable code, or on computationally heavy training regimes. Third, the AgentSupervisor's red lines are enforced largely through LLM self-auditing at multiple checkpoints rather than independent programmatic verification, so the guarantee against invalid gains depends on the reliability of those audits—an assumption the paper states but does not quantify. Fourth, some large gains involve substantial inference-time increases (e.g., MoSES), raising the question of how improvements should be normalized against compute budgets. Finally, extension to hardware-in-the-loop and robotic modalities remains unexplored, and integration with macro-level ideation frameworks such as OmniScientist is proposed but not demonstrated.

Conclusion

AutoSOTA demonstrates that a decomposed eight-agent architecture with explicit validity supervision can convert top-tier conference papers into reproduced baselines and then surpass them automatically, achieving improvements on all 105 curated papers at approximately five hours each. Its distinguishing contributions relative to evolutionary code-optimization systems are the treatment of environment initialization and long-horizon debugging as first-class problems, and the Red Line System constraining what counts as a legitimate gain. The evidence supports the system as research infrastructure for empirical iteration within its filtered scope; whether the same reliability extends to unfiltered literature, expensive experiments, and non-software scientific domains remains the central open question.