AutoResearch-RL: Perpetual RL for Code Discovery

• AutoResearch-RL is a reinforcement learning framework that autonomously refines code to optimize neural architectures in an open-ended, self-improving loop.
• It formalizes the problem as a Markov Decision Process and leverages PPO with transformer models to propose and evaluate atomic code edits.
• Empirical results highlight its superior performance and enhanced sample efficiency compared to traditional methods, validating its perpetual research capabilities.

AutoResearch-RL denotes a class of perpetual, self-evaluating reinforcement learning (RL) agents designed for autonomous, open-ended discovery in neural architecture and hyperparameter spaces without human supervision. In the canonical instantiation, an RL agent proposes structured modifications to an editable codebase (e.g., a training script), executes each altered configuration under standardized computational and evaluation conditions, observes scalar reward signals reflecting downstream learning outcomes, and iteratively refines its proposal policy using policy-gradient methods. The system continues until a termination oracle, driven by convergence or resource constraints, intervenes. This closed-loop, experiment-generating architecture is a rigorous formulation of the “neural researcher” paradigm and has been validated on neural sequence modeling benchmarks (Jain et al., 7 Mar 2026).

1. Formal Markov Decision Process for Open-Ended Code Research

AutoResearch-RL is cast as a discrete-time Markov Decision Process (MDP)

$$\mathcal{M} = (\mathcal{S}, \mathcal{A}, \mathcal{T}, \mathcal{R}, \gamma)$$

with the following specifications:

• State space $\mathcal{S}$: Each state $s_t = (c_t, h_t; d_t)$ contains the full source code of the trainable target (train.py), a trajectory buffer $h_t = \{(c_i, r_i)\}_{i < t}$ of prior proposals and observed rewards, and real-time diagnostics $d_t$ (e.g., GPU memory footprint, wall-clock metrics).
• Action space $\mathcal{A}$: Each action $a_t$ is a code-level unified diff (insert, replace, or delete), atomically applied to $c_t$ to generate $c_{t+1}$ as the new policy candidate.
• Transition kernel $\mathcal{T}$: Deterministic at the code-edit step and stochastic during training/evaluation (due to run-time factors and early termination).
• Reward function $\mathcal{R}$: The principal reward is based on the signed change in a scalar downstream validation metric (“bits per byte” or val-bpb). Auxiliary terms include (i) a bonus for computational efficiency, (ii) penalties for syntax errors or wasted runs.

$$r_t = -(\mathrm{bpb}_t-\mathrm{bpb}_{t-1}) + \lambda_{\mathrm{eff}}\eta_t - \mathbf{1}_{\mathrm{syntax}\ \mathrm{fail}}p_{\mathrm{syntax}} - \mathbf{1}_{\mathrm{early}\ \mathrm{abort}}p_{\mathrm{waste}}$$

• Discount $\gamma$: Typically set to $[0,1)$, controlling lookahead.

The environment, including data loaders, validation protocol, and other global constants, is frozen to guarantee cross-experiment comparability (see (Jain et al., 7 Mar 2026), Proposition 1).

2. Policy Optimization via Proximal Policy Optimization (PPO)

AutoResearch-RL employs transformer LLMs to parameterize both the policy $\pi_\theta(a|s)$ and value function $V_\phi(s)$, updated according to the PPO algorithm. Roll-outs $(s_t, a_t, r_t, s_{t+1})$ are collected and PPO’s clipped surrogate objective is used for stable gradient updates: $$\mathcal{L}^{\mathrm{CLIP}}(\theta) = \mathbb{E}_t\Big[\min\big(\rho_t\hat{A}_t,\,\mathrm{clip}(\rho_t, 1-\varepsilon, 1+\varepsilon)\hat{A}_t\big)\Big]$$ with importance ratio $\rho_t$ and GAE-style advantage estimates. Auxiliary value-function and entropy penalties

$$\mathcal{L}(\theta) = \mathcal{L}^{\mathrm{CLIP}}(\theta) - c_1\mathcal{L}^{\mathrm{VF}} + c_2\mathbb{E}_t\left[\mathcal{H}\left(\pi_\theta(\cdot\mid s_t)\right)\right]$$

ensure balanced, monotonic improvement. This approach is compatible with large memories and non-sequential action sets, as required for high-level code modification tasks.

3. System Architecture: Separation of Three Concerns

The AutoResearch-RL system architecture decomposes agent-environment design as follows:

1. Frozen environment: All aspects of dataset construction, evaluation split, and computational constants are immutable. This assures that each candidate policy edit is evaluated under identical conditions, enabling stochastic improvement guarantees.
2. Mutable target file: Only a single source of truth — typically a file such as train.py — is subject to automated edits. Diffs are validated pre-execution; invalid code triggers negative reward and re-sampling.
3. Meta-learner: A transformer-based agent maintains a rolling history window of prior code-reward pairs (empirically $K=32$ suffices for strong performance) and, at each iteration, proposes the next atomic edit. Memory is updated after code compilation and execution. A run-time self-evaluator observes the target’s loss curve in real time, fits a decay model, and applies an early-abort protocol (sequential probability ratio tests) if training appears unpromising, increasing overall experiment throughput by a factor up to 2.4 (Jain et al., 7 Mar 2026).

4. Theoretical Properties and Convergence Guarantees

The system is a form of nonparametric, perpetually improving stochastic hill-climber over a directed acyclic graph of code configurations. Under the assumption that each code state has nonzero probability of leading to a better validation metric, monotonic improvement in the best-seen metric $B_t$ is guaranteed (Theorem 1, (Jain et al., 7 Mar 2026)): $$B_{t+1} = \min(B_t, \mathrm{bpb}_{t+1}) \implies \mathbb{E}[B_{t+1}] \le B_t,\quad B_t \to B^*_{\min} \text{ a.s.}$$ Sample complexity bounds for achieving $\epsilon$-optimality can be established in terms of minimum-improvement probability: $$T \le \frac{\log\delta}{\log(1 - p_{\min}(\epsilon))}$$ enforcing finite convergence in expectation. The self-evaluation and early-abort subroutine further guarantee that time/resources are not wasted on unpromising runs, respecting the perpetual-learning assumption.

5. Empirical Validation and Comparative Analysis

The prototype AutoResearch-RL agent was validated on a single-GPU “nanochat” neural pretraining benchmark using the FineWeb data subset and a 5M-token validation split (Jain et al., 7 Mar 2026). The following experimental results summarize the core findings:

Method	val-bpb	# Experiments
Human Expert (GPT-2-small)	2.847	1
Random Search	2.791	93
Greedy LLM (no RL)	2.734	88
AutoResearch-RL	2.681	101

AutoResearch-RL rapidly achieves and surpasses the best-found configurations of both human and ablation baselines, reaching strong performance after approximately 100–300 iterations. Qualitative architecture and optimization discoveries (“Muon optimiser” scheduling, per-head QK-norm, gradient-clip scheduling, and depth increases) are automatically discovered. Early-abort self-evaluation enables a practical 1.35×–2.4× increase in wall-clock sample efficiency by terminating unpromising trials before full budget expiration.

In longer-term scaling runs, the best-seen metric continues to improve sublinearly with experiment count, demonstrating perpetual, but diminishing, returns.

6. Stopping Criteria and Perpetual Operation

By design, AutoResearch-RL lacks an intrinsic finite-horizon stopping point and can operate indefinitely (“perpetual agent”). Termination oracles may be supplied for resource bounding, convergence detection (e.g., improvement rate below threshold), or external objectives (target metric met). In the absence of such oracles, theoretical guarantees ensure that while experimentation continues, the best-seen validation metric is never worsened. This characteristic is crucial for fully unsupervised, open-ended neural architecture research.

7. Broader Significance and Implications

AutoResearch-RL unifies recent trends in experiment-generating RL agents, perpetual neural architecture search, and meta-optimization under a formal stochastic MDP framework with policy-gradient optimization and self-supervised feedback. The system explicitly separates environment regularization, proposal space, and adaptive memory, mitigating confounds from drift or direct overfitting. Empirical evidence demonstrates that such agents can autonomously match or outperform both exhaustively searched and expert-crafted configurations in nontrivial learning domains with minimal intervention (Jain et al., 7 Mar 2026). This formalism enables rigorous study of perpetual experimentation, automatic algorithm discovery, and the asymptotics of autonomous research pipelines.