Dream-RSI: Framework Overview

• Dream-RSI is a recursive self-improvement framework for AI-driven discovery systems that leverages historical discovery data to enhance exploration policies through offline replay simulations, focusing on efficient allocation of computational resources.
• Improvement is achieved by converting completed discovery histories into replay simulators, evaluating candidate exploration policies offline, and redeploying the best policy online to maximize search efficiency, rather than directly modifying the underlying coding model.
• The framework excels in tasks with long-horizon and expensive evaluations, such as algorithm engineering, mathematical optimization, and GPU kernel tasks, demonstrating reduced discovery cost and improved performance compared with fixed exploration policies.

Dream-RSI is a framework for recursive self-improvement through evolving replay environments. It improves an exploration controller rather than the underlying coding agent: completed discovery histories are converted into replay simulators, candidate exploration policies are evaluated offline on those historical trees, and the selected policy is redeployed online to generate further discoveries. The framework targets the allocation of search effort—branch selection, parallelism, continuation, and stopping—under long-horizon and expensive evaluation. Its formal name is “Dream-RSI: Recursive Self-Improvement through Evolving Worlds” (Zheng et al., 14 Sep 2026).

1. Motivation and scope

AI-driven discovery systems commonly iterate between candidate generation and evaluation. A coding agent generates or modifies a program, the candidate is executed, and its result determines subsequent exploration. As discovery tasks become more difficult, the exploration strategy itself becomes consequential: it determines which workspaces are continued, when new branches are opened, how many workers are used, and when search terminates.

Dream-RSI addresses a tension between fixed and online-optimized exploration. Fixed policies are inexpensive and stable but cannot adapt their allocation of computation according to accumulated evidence. Online optimization is adaptive in principle, but evaluating a candidate exploration policy may require an expensive, long-horizon rollout involving many coding-agent calls and delayed feedback.

The framework therefore treats completed discovery histories as replay simulators, also called “worlds.” Instead of rerunning the coding agent and evaluator for every candidate policy, Dream-RSI replays previously observed discovery trees. Candidate policies can thereby be compared off-policy before one is deployed online. The central loop is:

$$\text{online exploration} \rightarrow \text{record discovery history} \rightarrow \text{construct replay worlds} \rightarrow \text{dream over candidate policies} \rightarrow \text{redeploy the selected policy}.$$

The method performs recursive improvement at the level of exploration orchestration. The coding agent, evaluator, model weights, and execution interface remain fixed in the experiments. Consequently, improvement in discovered artifacts results from better search allocation rather than direct self-modification of the coding model.

2. System architecture

Dream-RSI consists of a lightweight orchestration layer surrounding an unchanged discovery system. The orchestration layer makes exploration programmable but does not itself generate candidate solutions. It selects previously observed nodes, determines batch size and parallelism, decides whether to open or continue branches, and can terminate exploration.

The coding agent generates or modifies candidates from selected workspaces. The experiments use LLM-based coding agents such as Gemini-3.1 Pro and Gemini-3.7 Flash. The evaluator executes candidates, verifies correctness, and returns task-specific scores. Both components remain fixed during exploration-policy optimization.

Each online rollout produces a discovery tree containing workspaces or filesystem states, generated artifacts, observations and diagnostics, evaluation scores, parent–child relationships, exploration decisions, and computationally relevant information. If $\mathcal{T}_t$ is the tree produced at outer iteration $t$, the accumulated history is

$$\mathcal{H}_t=\mathcal{H}_{t-1}\cup\{\mathcal{T}_t\}.$$

Every completed tree becomes a replay world. The simulator pool at iteration $t$ is therefore

$\{\mathcal{T}_1,\ldots,\mathcal{T}_t\}.$

A separate policy-development agent examines replay trajectories and scores, then edits executable exploration-policy code. The resulting policy versions are denoted

$\pi_t^0,\pi_t^1,\ldots,\pi_t^{M-1},$

with $\pi_t^0$ equal to the policy currently deployed online.

3. Discovery trees and online exploration

Let $r$ denote the root of a discovery tree, representing the initial workspace. Every non-root node has one primary parent and stores the result of one generation–evaluation attempt, including its score $s_v$. The eligible continuation nodes are

$\mathcal{T}_t$0

The root remains eligible because selecting it opens another independent branch; a non-root leaf is eligible because it can be refined. If $\mathcal{T}_t$1 is the maximum number of parallel workers, the feasible action set is

$\mathcal{T}_t$2

Thus, an exploration-policy action is a batch $\mathcal{T}_t$3 of nodes. It simultaneously specifies which workspaces are continued, how many attempts are issued, and the degree of parallelism. The empty batch, $\mathcal{T}_t$4, means that exploration stops.

At outer iteration $\mathcal{T}_t$5, the online tree begins as

$\mathcal{T}_t$6

For each online round, the deployed policy observes the currently available tree and selects a batch. Each selected node is assigned to a worker; the coding agent generates a candidate; the evaluator executes and scores it; and the resulting children are attached to the tree:

$\mathcal{T}_t$7

The rollout terminates when the policy selects the empty batch or when the maximum number of online rounds, $\mathcal{T}_t$8, is reached. The completed tree is then added to the history and becomes available for future replay.

Online transitions are stochastic because the coding agent may generate different candidates from the same workspace. This distinguishes online discovery from replay: online exploration can encounter novel candidates and states, whereas replay is restricted to recorded outcomes.

4. Replay simulation and dreaming

A historical tree is converted into a replay world by preserving its recorded structure and outcomes. Replay does not rerun the coding agent, evaluator, candidate execution, or underlying discovery process. Instead, it reveals previously recorded children when a policy requests them.

For policy $\mathcal{T}_t$9 and historical tree $t$0, replay begins with

$t$1

The full tree $t$2 remains fixed, while $t$3 denotes the portion revealed to the policy. At each replay round, the policy selects

$t$4

For a non-root node, replay reveals its unique recorded child if one exists. For the root, it reveals the earliest-created previously unrevealed child. If no continuation exists, no node is revealed. This rule preserves parent–child order while permitting policies to choose different branches, orders, batch sizes, and stopping points.

Replay terminates when the policy selects the empty batch, when $t$5 replay rounds have elapsed, or when all recorded nodes have been revealed. If $t$6 is the number of completed replay rounds, the number of represented non-root attempts is

$t$7

The quantity $t$8 measures how much historical discovery computation a replay trajectory would have consumed, even though no new candidate is executed.

“Dreaming” refers to running candidate exploration policies through these replay worlds. Every policy version is evaluated on every historical tree. The policy-development agent receives replay trajectories, scores, successful and unsuccessful branch decisions, and feedback from previous revisions. It then produces another executable policy, which is evaluated on the same fixed replay worlds.

This procedure differs from using history merely as textual context, a memory bank, semantic lessons, or static training data. Replay retains the decision structure of the discovery tree and allows alternative policies to interact with recorded branches under different allocation rules.

5. Replay objective and recursive policy selection

The replay score for policy $t$9 on world $$\mathcal{H}_t=\mathcal{H}_{t-1}\cup\{\mathcal{T}_t\}.$$0 is

$$\mathcal{H}_t=\mathcal{H}_{t-1}\cup\{\mathcal{T}_t\}.$$1

where $$\mathcal{H}_t=\mathcal{H}_{t-1}\cup\{\mathcal{T}_t\}.$$2 is the recorded quality score, $$\mathcal{H}_t=\mathcal{H}_{t-1}\cup\{\mathcal{T}_t\}.$$3 is the number of revealed non-root nodes, $$\mathcal{H}_t=\mathcal{H}_{t-1}\cup\{\mathcal{T}_t\}.$$4 is the number of replay rounds, and $$\mathcal{H}_t=\mathcal{H}_{t-1}\cup\{\mathcal{T}_t\}.$$5 are fixed coefficients.

The objective contains three terms. The discovery-quality term favors trajectories reaching high-scoring historical solutions. The execution-cost term penalizes revealing many attempts. The parallelism term rewards policies that accomplish more attempts per decision round. The average policy score over the current history is

$$\mathcal{H}_t=\mathcal{H}_{t-1}\cup\{\mathcal{T}_t\}.$$6

The next deployed policy is selected as

$$\mathcal{H}_t=\mathcal{H}_{t-1}\cup\{\mathcal{T}_t\}.$$7

Because the candidate set includes the current policy, $$\mathcal{H}_t=\mathcal{H}_{t-1}\cup\{\mathcal{T}_t\}.$$8, selection guarantees

$$\mathcal{H}_t=\mathcal{H}_{t-1}\cup\{\mathcal{T}_t\}.$$9

on the fixed replay history. This is not a guarantee of improvement on future online trees, because replay cannot represent unobserved branches or future stochastic outcomes.

The complete recursive loop alternates between online exploration and offline policy revision:

1. The current policy conducts an online rollout.
2. The resulting discovery tree is added to the history.
3. Candidate policy versions are evaluated on all historical replay worlds.
4. The policy-development agent revises the exploration code.
5. The highest-scoring replay policy is redeployed.
6. The new online rollout expands the simulator pool.

The process improves the exploration controller, including exploration breadth, branch continuation, parallelism, stopping, and allocation of coding-agent calls. It does not update the coding agent’s model weights, evaluator, execution engine, or candidate-generation mechanism.

6. Experimental evaluation

Dream-RSI is evaluated in algorithm engineering, mathematical optimization, and GPU kernel engineering. The main comparison is between Recursive Fixed Exploration, which retains a fixed exploration policy, and Dream-RSI, which updates the policy through replay after each recursive round. Both methods use the same coding agent, evaluator, initial exploration policy, and per-round resource constraints.

For Gemini-3.1 Pro, the per-round budget is

$t$0

coding-agent calls. For Gemini-3.7 Flash, it is

$t$1

calls. Discovery cost is measured primarily by cumulative coding-agent calls or generations.

Algorithm engineering

The Lasso task concerns efficient computation of the complete Lasso regularization path. Dream-RSI discovers implementations combining strong-rule screening, Cauchy–Schwarz-based KKT pruning, selective exact-gradient recomputation, full gradient refresh when pruning becomes ineffective, active-set bookkeeping, lazy Gram-matrix construction, and hardware-aware implementation.

For Gemini-3.1 Pro, Dream-RSI reduces average held-out runtime from $t$2 ms under Fixed Exploration to $t$3 ms while reducing calls from $t$4 to $t$52516.7$t$62350.6$t$73200$t$81879. The reported generalization datasets are Gisette, RCV1, DNA, Leukemia, Colon, and Duke Breast.

Mathematical optimization

The mathematical experiments cover Sum–Difference, circle packing, and autocorrelation inequalities. Dream-RSI achieves a Sum–Difference score of $t$9, exceeding the reported SimpleTES and Fixed Exploration results; it matches the strongest reported circle-packing result of $\{\mathcal{T}_1,\ldots,\mathcal{T}_t\}.$0; and it remains competitive on autocorrelation inequalities, where SimpleTES retains the strongest reported score.

Dream-RSI uses fewer than $\{\mathcal{T}_1,\ldots,\mathcal{T}_t\}.$1 generations in the reported mathematical setting, compared with $\{\mathcal{T}_1,\ldots,\mathcal{T}_t\}.$2 for SimpleTES. The evidence therefore supports a cost-efficient and competitive discovery process rather than uniform superiority on every objective.

GPU kernel engineering

The GPU experiments use the KernelBench tasks VGG16, LayerNorm, ConvDiv, and ConvMax. Dream-RSI obtains comparable performance with $\{\mathcal{T}_1,\ldots,\mathcal{T}_t\}.$3 fewer generations on VGG16 and $\{\mathcal{T}_1,\ldots,\mathcal{T}_t\}.$4 fewer generations on LayerNorm. It achieves $\{\mathcal{T}_1,\ldots,\mathcal{T}_t\}.$5 higher performance at comparable budget on ConvDiv and $\{\mathcal{T}_1,\ldots,\mathcal{T}_t\}.$6 higher performance at comparable budget on ConvMax.

On ConvDiv, the reported number of evaluated attempts across recursive rounds is

$\{\mathcal{T}_1,\ldots,\mathcal{T}_t\}.$7

while round-best performance increases approximately as

$\{\mathcal{T}_1,\ldots,\mathcal{T}_t\}.$8

This pattern is interpreted as adaptive effort allocation: the policy conserves computation when progress is available, increases exploration after plateaus, and can obtain further improvements after renewed exploration.

7. Limitations and interpretation

Replay worlds are empirical representations of realized discovery histories. They cannot invent outcomes absent from the recorded tree, generate novel candidates, or reproduce all stochasticity of online execution. A policy may therefore overfit to the historical simulator pool. The non-degradation guarantee applies only to replay score on the fixed history and does not establish monotonic improvement in future online discovery.

The framework’s effectiveness may depend on the initial exploration policy, coding model, evaluator reliability, discovery-tree structure, replay coefficients, online and replay horizons, number of policy revisions, and worker count. The reported experiments do not provide a broad component-by-component ablation over all of these factors.

The principal cost metric is coding-agent calls or generations. Replay has no new discovery-execution cost, but it still incurs orchestration, storage, policy-execution, and policy-development computation. Consequently, “zero-execution-cost” replay does not mean zero total system cost.

Dream-RSI is best characterized as sample-efficient optimization of a search controller. Its central contribution is the conversion of discovery trees from passive records into executable replay environments in which alternative exploration policies can be evaluated before deployment. The method does not recursively improve the underlying coding model in the reported experiments; it recursively improves the allocation and organization of search.