AutoResearch-RL: Perpetual Self-Evaluating Reinforcement Learning Agents for Autonomous Neural Architecture Discovery

Abstract: We present AutoResearch-RL, a framework in which a reinforcement learning agent conducts open-ended neural architecture and hyperparameter research without human supervision, running perpetually until a termination oracle signals convergence or resource exhaustion. At each step the agent proposes a code modification to a target training script, executes it under a fixed wall clock time budget, observes a scalar reward derived from validation bits-per-byte (val-bpb), and updates its policy via Proximal Policy Optimisation (PPO). The key design insight is the separation of three concerns: (i) a frozen environment (data pipeline, evaluation protocol, and constants) that guarantees fair cross-experiment comparison; (ii) a mutable target file (train.py) that represents the agent's editable state; and (iii) a meta-learner (the RL agent itself) that accumulates a growing trajectory of experiment outcomes and uses them to inform subsequent proposals. We formalise this as a Markov Decision Process, derive convergence guarantees under mild assumptions, and demonstrate empirically on a single GPU nanochat pretraining benchmark that AutoResearch-RL discovers configurations that match or exceed hand-tuned baselines after approximately 300 overnight iterations, with no human in the loop.

• The paper introduces a transformer-based RL agent that autonomously edits ML training scripts to refine network architectures, achieving improved validation bits-per-byte.
• The approach features a real-time self-evaluation module that aborts poor-performing runs, resulting in a 2.4× boost in experiment throughput.
• The framework proves theoretical convergence with super-martingale behavior and outperforms traditional baselines on key performance metrics.

AutoResearch-RL: A Perpetual RL Agent for Autonomous Neural Architecture Discovery

Motivation and Context

AutoResearch-RL addresses the inefficiency and human-dependence of neural architecture and training algorithm research by formulating the entire experimental loop as a continual RL process. The key innovation is to allow an RL agent—parameterized as a transformer-based policy fine-tuned with PPO—to edit an ML training script (train.py), evaluate resultant validation performance under a fixed wall-clock budget, and iteratively refine its code-editing strategy. This approach conceptualizes the agent as not merely a hyperparameter optimizer but as an open-ended research agent capable of modifying all aspects of the training pipeline—including network structure, optimizer implementation, and scheduler logic—until an external oracle (i.e., a termination signal or resource exhaustion) intervenes.

Conventional NAS and AutoML are constrained by fixed search spaces and static evaluation protocols, typically optimizing over architectural definitions or hyperparameter grids. In contrast, AutoResearch-RL explores a superset of these by operating directly on source code. Recent advances in LLM-based code generation and autonomous software agents have demonstrated code-editing capability but previously lacked formal reinforcement learning formulations or empirical analysis in the context of autonomous ML research.

Methodological Framework

MDP Formalization

The system's outer loop is cast as a Markov Decision Process $$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma)$$, where the state $s_t$ incorporates the current source code, indexed experiment history, and system diagnostics; the action $a_t$ is an atomic source code diff; the transition function captures deterministic code modification and stochastic training dynamics; and the reward $r_t$ is a combination of validation bits-per-byte (val-bpb) improvement and a compute-efficiency bonus.

A salient design decision is conducting all experiments under a fixed wall-clock time per configuration. This scheme ensures comparability by controlling for implementation-dependent speedups and batch size effects—yielding direct correspondence between reward signals and architectural merit. The use of val-bpb as the primary scalar reward ensures evaluation is invariant to tokenization granularity and vocabulary size.

Policy Architecture

The agent's policy $\pi_\theta$ is implemented as a transformer LM (claude-sonnet-4 fine-tuned with LoRA), conditioned on:

1. Immutable research instructions,
2. Current train.py code snapshot,
3. A sliding window of $K=32$ recent experiments (including the best-ever configuration), each annotated with val-bpb and self-evaluations.

Actions are code diffs; invalid diffs incur significant penalties. PPO is employed for policy improvement, with entropy regularization to promote sustained exploration given the high-dimensional, discrete edit space. Novelty bonuses based on normalized edit distance are used to encourage semantic diversity in generated proposals.

Self-Evaluation and Early Stopping

The self-evaluation module is a real-time monitoring subsystem that fits a power-law forecast to loss curves during training and aborts runs whose predicted final bpb is statistically worse than a pessimistic threshold (based on historical outcomes and a user-controlled tolerance parameter $\alpha$). This module functions as an adaptive best-arm bandit, yielding up to $2.4\times$ improvement in experiment throughput by aborting ~54% of subpar runs early.

Theoretical Results

A key theoretical claim is that the trace of best-obtained val-bpb values forms a super-martingale process and thus monotonically improves (or remains unchanged). The process converges almost surely to the minimum attainable val-bpb given a non-trivial probability of sampling improvements at each step and sufficient exploration, as proved by monotone convergence. Sample complexity bounds detail the number of runs required to achieve $\epsilon$-proximity to the minimum, given minimal improvement probability $p_{\min}(\epsilon)$.

The exploration–exploitation dilemma is addressed via entropy regularization and explicit novelty bonuses, with empirical analysis supporting policy diversity in edit proposals.

Experimental Evaluation

Benchmark and Baselines

Experiments are conducted on a single-GPU (NVIDIA H100) nanochat pretraining benchmark with a 5-minute per-experiment wall-clock cap, held-out validation set, and SoTA baselines. Baselines include:

• Hand-tuned expert configuration,
• Random search over an extensive hyperparameter grid,
• Greedy LLM agent (GPT-4o) without RL adaptation,
• The proposed AutoResearch-RL agent.

Main Results

AutoResearch-RL achieves the lowest validation bpb (2.681) in 8 GPU-hours, outperforming both the hand-tuned baseline (2.847) and the LLM baseline (2.734). The agent's learning curve demonstrates accelerated improvement and sustained search efficacy relative to all baselines.

Key discoveries, diverging from the human-expert baseline, include:

• Increased Muon learning rate and adapted AdamW weight decay for better convergence,
• Injection of per-head $l_2$ query-key normalization to stabilize attention and enlarge batch size,
• Scheduled gradient clipping for improved optimization stability,
• Deeper transformer architectures within the compute envelope.

These changes mirror recent state-of-the-art architecture and optimizer advances, attesting to the agent's non-trivial search capability.

Perpetual operation experiments reveal continued improvement (down to 2.608 val-bpb after 2147 experiments over one week), with diminishing returns at scale but no evidence of premature convergence.

Throughput Improvement

Integration of the self-evaluation/early stopping module leads to a 1.35× gain in experiments per wall-clock hour and 2.4× cumulative efficiency, empirically validating the utility of real-time loss-curve monitoring.

Implications and Future Directions

AutoResearch-RL demonstrates the practical feasibility of perpetual, self-improving agents for automated neural architecture and training algorithm discovery. The main implications are:

• Automated meta-research: The agent learns and internalizes high-level research heuristics that guide exploration—beyond simple hyperparameter or architectural search.
• Compute-constrained rapid discovery: Bottlenecks shift from human iteration rates to available computational resources, suggesting fundamentally new paradigms for scalable meta-learning.
• Continuous, open-ended research: The loop is provably non-degrading, theoretically safe for indefinite operation, and practically effective across night-to-week compute windows.
• Towards broader autonomy: Current limits include single-file mutation and single-node execution; extensions to multi-node, multi-file ("multi-research-thread") and more expressive code modifications (e.g., data pipeline, vocabulary change) are immediate targets.
• Agent reliability: Introducing sophisticated safety instrumentation and enumerating search bounds are essential for broader deployment.

Future research could examine joint optimization of architectures and data preprocessing, generalization to different training domains (RL, vision), and stronger theoretical convergence or regret guarantees in high-dimensional code-edit spaces. A further avenue is integration with human-in-the-loop or hybrid researcher-agent paradigms for maximizing system creativity and compliance.

Conclusion

AutoResearch-RL formalizes and empirically validates a perpetual, self-evaluating RL agent for the autonomous discovery of neural architectures and training algorithms. By directly integrating real training metrics, unrestricted code-edit actions, and RL-based exploration strategies, the framework demonstrates measurable superiority to both random and human-driven search within a formal performance regime. Theoretical convergence, sample complexity guarantees, and empirical sample efficiency collectively mark a decisive step toward continual, compute-bound, autonomous algorithmic discovery in machine learning.

Reference: "AutoResearch-RL: Perpetual Self-Evaluating Reinforcement Learning Agents for Autonomous Neural Architecture Discovery" (2603.07300).