Dream-RSI: Recursive Self-Improvement through Evolving Worlds

Abstract: Recursive self-improvement is becoming increasingly vital for autonomous AI agents, where progress hinges on discovering high-value solutions across complex domains. The driver of this process is effective exploration, however, managing and improving exploration strategies remains a major bottleneck. Current systems face a fundamental dilemma: fixed strategies fail to adapt as search spaces scale, while online policy optimization requires navigating vast meta-search spaces under delayed and expensive feedback over long-horizon rollouts. We introduce \textsc{Dream-RSI}, a framework for scalable and recursively self-improving exploration. A lightweight orchestration layer makes exploration explicit and programmable while leaving the underlying coding agent unchanged. Our key insight is that accumulated discovery history can serve as a replay simulator over the realized search space. By performing dreaming in the replay simulator constructed from historical discovery trees, \textsc{Dream-RSI} secures immediate, low-cost off-policy feedback to evaluate and refine exploration policies without invoking repetitive, expensive online evaluations. The improved policy is subsequently redeployed online to drive further discovery, continuously expanding the simulator pool in a self-improving loop. Across algorithm engineering, mathematical optimization, and GPU kernel engineering, \textsc{Dream-RSI} achieves competitive or improved discovery quality while substantially reducing discovery cost in several settings.

• The paper introduces Dream-RSI, a framework that enhances the exploration policy in recursive self-improvement by using replay simulators to efficiently evaluate different search strategies offline.
• Across eight tasks in three domains and discovery agents’ Gemini 3.1 and Gemini 3.7, Dream-RSI consistently reduces discovery agent calls (e.g., 1.7 ≈ 86% and 1.7 ⨟ 40 to -20), and approximately mean runtime improvement by 21% and 35 % in Lasso tasks
• The framework’s non-worsening property ensures that deployed policies cannot perform worse on a replay history, but extended evaluations with longer timeframes, other inputs, or contexts are necessary for robust online improvement.

Problem setting and central thesis

“Dream-RSI: Recursive Self-Improvement through Evolving Worlds” (2609.14858) addresses a specific bottleneck in recursive self-improvement (RSI): improving the exploration policy that allocates computation during long-horizon discovery. Existing discovery systems typically use manually specified orchestration strategies, while online optimization of those strategies is expensive because their quality is observable only after extended sequences of candidate generation and evaluation. The paper’s central claim is that completed discovery histories can be converted into replay simulators, enabling large numbers of alternative exploration policies to be evaluated off-policy before deployment.

The proposed framework separates the discovery system into a fixed underlying coding agent and evaluator, and a programmable orchestration layer. The orchestration layer determines which search branches to extend, how many workers to use, how to schedule parallel attempts, and when to stop. Dream-RSI recursively modifies only this layer. Its operating loop consists of online exploration, construction of replay simulators from the resulting discovery trees, and offline policy improvement through replay.

Figure 1: Dream-RSI alternates online exploration, replay-simulator construction, and offline improvement of an executable exploration policy.

The paper’s principal conceptual move is to treat history not merely as prompt context, memory, or training data, but as an empirical simulator of the portion of the search space already realized. This distinction is important. A textual summary of previous attempts can influence the next action, but it does not generally support counterfactual evaluation of alternative scheduling decisions. A structured discovery tree, by contrast, preserves parent-child relationships, workspaces, artifacts, diagnostics, and scores, allowing candidate controllers to traverse different subsets of previously observed outcomes.

Discovery histories as replay simulators

Each online rollout constructs a rooted discovery tree. The root represents the initial workspace, while each non-root node records one generation-evaluation attempt and retains its inherited workspace state. A node includes the generated artifact, filesystem snapshot, evaluator diagnostics, and scalar score. The policy can select either the root or currently exposed leaves as continuation points. With a worker budget $W$, an action is a batch of at most $W$ eligible nodes, so the policy jointly controls branch selection and parallelism.

Online transitions are stochastic: extending a node invokes the coding agent and evaluator, producing a new candidate and a new child. Replay transitions are deterministic: selecting a node reveals a previously recorded child, without invoking the coding agent or evaluator. The replay trajectory therefore represents a counterfactual policy’s traversal of an existing tree rather than a newly generated discovery process.

Figure 2: A recorded discovery tree supports low-cost off-policy evaluation of alternative branch-selection, scheduling, parallelism, and stopping policies.

This replay construction gives a single online rollout substantial reuse value. A historical tree can be traversed according to many candidate policies, each of which may choose different branches, branch orders, batch sizes, and termination points. Since all node outcomes are already stored, these evaluations incur no additional generation or evaluation calls. The framework consequently replaces repeated online meta-level experimentation with rapid simulation over realized experience.

The approximation is deliberately empirical rather than generative. Replay cannot expose outcomes for branches that were never explored, and it cannot model the stochastic consequences of a policy outside the recorded tree. Its value therefore depends on the coverage and structural diversity of accumulated histories. Within those constraints, it provides an unusually direct form of off-policy supervision for exploration control.

Dream-RSI algorithm

Dream-RSI maintains an outer sequence of policies $\pi_1,\pi_2,\ldots$. At iteration $t$, $\pi_t$ conducts a fresh online rollout for at most $K_1$ decision rounds. The resulting tree $\mathcal{T}_t$ is appended to the history $\mathcal{H}_t$. The offline phase then evaluates a sequence of policy revisions on every historical tree.

The candidate policy set begins with the currently deployed policy, $\pi_t^0=\pi_t$. A policy-development agent receives replay trajectories, scores, and feedback from previous revisions, then edits the executable exploration-policy code to produce $\pi_t^{m+1}$. Each version is replayed independently on all trees in $W$0, with a replay horizon of at most $W$1 rounds. After $W$2 revisions, the system deploys the highest-scoring evaluated version.

The replay objective combines three terms:

1. Discovery quality: the maximum score among revealed nodes.
2. Execution cost: a penalty proportional to the number of represented generation-evaluation attempts.
3. Parallelism: a bonus for completing more attempts per decision round.

In simplified form, the score rewards high-quality discoveries, penalizes unnecessary attempts, and favors useful batching. Because the incumbent policy is included among the candidates, selection guarantees that the chosen revision is no worse than the incumbent on the fixed replay history. This is a replay-set monotonicity guarantee, not a guarantee of monotonic online improvement. Distribution shift between replay trees and future online rollouts remains possible.

The framework leaves the underlying discovery model, evaluator, and execution interface unchanged. This design isolates the contribution to meta-exploration and avoids conflating improvements in orchestration with improvements in code-generation capability. The policy itself remains an executable program, so revisions can express explicit control logic rather than only changing textual prompts.

Experimental design

The evaluation covers eight tasks in three domains: Lasso regularization-path engineering, mathematical optimization, and GPU kernel engineering. The principal controlled comparison is Recursive Fixed Exploration, which uses the same discovery agent, evaluator, initialization, and per-round budgets but retains the initial orchestration policy unchanged. Discovery cost is measured primarily by cumulative discovery-agent calls.

The initial policy is a parallel refinement strategy with multiple independent workspaces. Both methods therefore behave identically during the first discovery round. Differences emerge only when Dream-RSI uses accumulated histories to revise its exploration controller. This design makes the comparison more informative than comparing systems with different agents or initial search spaces, although it does not eliminate all variability associated with stochastic generation.

Algorithm engineering results

The Lasso task requires discovering an efficient and numerically correct implementation of the complete regularization path. Search uses 17 synthetic instances, while final evaluation uses six held-out biological and non-biological datasets. Correctness is checked separately from timing, and any correctness failure receives a zero search score.

Dream-RSI improves both performance and discovery cost for both tested backbones:

Discovery agent	Method	Calls	Mean held-out runtime
Gemini-3.1 Pro	Recursive Fixed Exploration	550	3587.1 ms
Gemini-3.1 Pro	Dream-RSI	317	2931.0 ms
Gemini-3.7 Flash	Recursive Fixed Exploration	3200	2516.7 ms
Gemini-3.7 Flash	Dream-RSI	1879	2350.6 ms

With Gemini-3.1 Pro, Dream-RSI reduces calls by approximately $W$3 while reducing mean runtime by approximately 18.3%. With Gemini-3.7 Flash, it reduces calls by approximately $W$4 and improves mean runtime by approximately 6.6%. The paper further reports lower average downstream runtime than SimpleTES, despite using only hundreds or thousands of calls rather than SimpleTES’s reported 51,200 generations. This is the paper’s strongest efficiency result, although the comparison involves different model families and system configurations and should not be interpreted as a pure orchestration-only ablation.

The discovered solver also differs algorithmically from the SimpleTES characterization. It combines strong-rule screening with Cauchy–Schwarz-based KKT pruning, adaptively recomputes exact gradients when bounds are inconclusive, and falls back to full refreshes when pruning loses effectiveness. Lazy Gram-matrix construction, active-set bookkeeping, and hardware-aware implementation are integrated into the resulting program. Thus, the orchestration improvement ultimately affects not only search efficiency but also which algorithmic mechanisms are discovered.

Mathematical optimization results

The mathematical evaluation covers Sum–Difference, Circle Packing, and Autocorrelation Inequalities. Dream-RSI is compared with several automated discovery systems, including AlphaEvolve, OpenEvolve, CodeEvolve, ShinkaEvolve, ThetaEvolve, EvoX, and SimpleTES.

Task	Dream-RSI	Recursive Fixed Exploration	Strong comparison
Sum–Difference, higher better	1.145427	1.144047	SimpleTES: 1.143975
Autocorrelation, lower better	1.456375	1.456001	SimpleTES: 1.453675
Circle Packing, higher better	2.635983	2.635983	Several methods: 2.635983

Dream-RSI improves the Sum–Difference score over both controlled baselines and matches the strongest reported Circle Packing result. On Autocorrelation, it remains competitive but does not match SimpleTES’s best score. The paper emphasizes that SimpleTES reaches its Autocorrelation result with 51,200 generations, whereas Dream-RSI uses fewer than 1,000 generations. This supports the more precise claim that Dream-RSI provides a favorable quality-cost trade-off, not that it dominates every method on every objective.

The mathematical results also expose an important asymmetry: replay-based policy improvement can reduce the cost of reaching a given solution quality even when it does not improve the absolute best score. This is consistent with the framework’s objective, which explicitly trades off discovery quality and represented execution cost.

GPU kernel engineering results

The KernelBench evaluation tests VGG16, LayerNorm, ConvDiv, and ConvMax using Gemini-3.1 Pro. Candidate kernels must remain numerically correct, and performance is measured as inverse runtime. Dream-RSI is again compared with Recursive Fixed Exploration under matched initialization and evaluation conditions.

The reported scaling results are:

• VGG16: comparable performance with $W$5 fewer generations.
• LayerNorm: comparable performance with $W$6 fewer generations.
• ConvDiv: $W$7 higher performance at a comparable budget.
• ConvMax: $W$8 higher performance at a comparable budget.

These results are significant because kernel optimization requires coordinated decisions about memory access, parallelization, algorithmic decomposition, and hardware-specific implementation. The improvements therefore suggest that the learned controller is not limited to abstract mathematical search or straightforward code refinement. At the same time, the paper reports four tasks from one benchmark, so the breadth of hardware and kernel coverage remains limited.

Behavioral analysis of the learned controller

The paper examines whether accumulated history is more useful as an interactive simulator than as semantic prompt guidance. In an additional ConvDiv comparison, historical information is summarized into explicit directional guidance and injected into both fixed and Dream-RSI exploration. The guided variants underperform their unguided counterparts under equivalent budgets.

The authors interpret this result as evidence that strong semantic guidance can over-constrain parallel long-horizon search. The replay simulator retains concrete alternative branches and their observed consequences, whereas a high-level textual direction compresses the history into a narrower inductive bias. The result does not establish that semantic guidance is generally harmful; it shows that, in this setting, direct replay-based interaction with historical branches is more effective than the tested prompt-level abstraction.

The evolution of the exploration policy on ConvDiv further indicates that Dream-RSI learns a nonmonotonic compute-allocation strategy. In early rounds, the policy reduces evaluated attempts from 110 to 50 while performance improves. When progress later plateaus, it increases effort again, coinciding with additional performance gains. This behavior is consistent with adaptive allocation between exploitation and renewed exploration. It also demonstrates that the controller changes not only branch priorities but the amount and granularity of search effort.

Limitations and open questions

The replay simulator is restricted to realized discovery trees. It cannot evaluate the consequences of unexplored branches, and replay outcomes are deterministic even though corresponding online generation is stochastic. Consequently, replay scores may be optimistic for policies that exploit artifacts of historical coverage or may fail to predict policies whose behavior depends on new discoveries. The paper does not provide a formal bound on this counterfactual bias.

The non-worsening property also applies only to average replay score on the fixed history. It does not imply monotonic improvement on future online rollouts, and the experiments do not isolate how often a selected policy regresses after deployment. A more complete evaluation would report replay-to-online rank correlation, policy-selection regret, and the frequency and magnitude of online regressions.

The policy-development agent is itself fixed and LLM-based, while the exploration-policy representation, replay objective, worker model, and stopping rules are hand-designed. It is therefore unclear how much of the observed gain derives from the replay abstraction, the particular prompts, or the chosen cost and parallelism coefficients. The experiments also use relatively small task sets and a limited number of recursive rounds. Whether the approach remains stable as histories become very large, heterogeneous, or stale is left open.

Finally, comparisons with external systems such as SimpleTES are informative but not fully controlled because they use different discovery agents and resource configurations. The controlled comparison with Recursive Fixed Exploration supports the orchestration claim more directly; cross-system comparisons mainly establish competitive cost-quality performance.

Conclusion

Dream-RSI (2609.14858) presents a concrete mechanism for recursive improvement of exploration policies in long-horizon discovery. Its key contribution is to transform structured discovery histories into replay simulators that support inexpensive off-policy evaluation of branch selection, scheduling, parallelism, and stopping behavior. Across Lasso engineering, mathematical optimization, and GPU kernel engineering, the method generally matches or improves discovery quality while reducing generation costs, with particularly strong reported savings in Lasso and KernelBench.

The central empirical conclusion is narrower and more defensible than a general claim about RSI: when discovery histories preserve executable structure and outcome traces, replay-based policy improvement can provide useful meta-level supervision without repeatedly rerunning expensive online discovery. The principal unresolved issue is the reliability of this supervision outside the realized search trees, particularly as future policies pursue branches and distributions not represented in the accumulated history.