RSIAgent: Autonomous Exploration for Recursive Self-improvement in New Environments

Abstract: Digital agents must often adapt to new environments whose interfaces, tools, and failure modes are not fully captured by pretrained models. We introduce \textbf{RSIAgent}, a training-free multi-agent framework for recursive self-improvement through autonomous memory construction. RSIAgent coordinates curriculum, actor, and verifier agents to continually explore the environment, validate outcomes, and retain environment-specific knowledge, including reusable causal relationships between actions, conditions, and consequences. It further adopts a \textbf{broad-then-deep} exploration strategy, combining parallel broad recursive self-exploration for discovering diverse environment structures with focused deep self-exploration for uncovering hard cases, hidden constraints, boundary conditions, and previously unknown causal dependencies. The resulting memory is frozen and can be directly reused for downstream tasks without updating model parameters. Experiments on OSWorld-v2 and Agent's Last Exam show that RSIAgent substantially improves strong open-source models, enabling Kimi-K3 and GLM-5.3 to outperform frontier closed-source models including GPT-6.

• RsIAgent improves digital agents' learning in new environments through recursive self-exploration and memory consolidation, significantly boosting performance on benchmark tasks by delivering a partial score of 78.98% on OSWorld 2.0 and 84.82% on Agents' Last Exam, among other metrics.
• The architecture incorporates an actor, verifier, and curriculum agent, ensuring flexible information handling and a secure learning process.
• The proposed BRS and DRS stages account for broad and deep recursive self-exploration, with BRS focusing on generating and executing multiple independent projects while DRS leveraging verified experiences.

Problem formulation and central claim

RSIAgent addresses adaptation of digital agents to software environments whose interfaces, workflows, latent state transitions, and failure modes are not adequately represented in pretrained model parameters. Its central claim is that substantial agent improvement can be obtained without parameter updates, human demonstrations, or external task labels by constructing and reusing environment-specific memory through autonomous interaction and verification (2609.15364).

The paper formalizes an agent as a code-generating policy operating in an environment through a sequence of executable actions. Short-lived interaction history is reset between tasks, whereas durable memory persists across tasks and stores procedures, constraints, failure diagnoses, and environment-specific causal relations. This distinction is important: RSIAgent does not merely retrieve successful trajectories. It attempts to consolidate rules of the form “under condition $c$, action $a$ produces consequence $y$,” together with the evidence and uncertainty surrounding those rules.

The method is therefore positioned between conventional test-time prompting and parameter-level adaptation. The model weights remain fixed, but the effective policy changes because later executions are conditioned on an evolving memory corpus. The paper’s stronger methodological claim is that the memory can be acquired recursively: exploration changes the agent’s knowledge, the changed knowledge affects the next exploration decision, and the resulting evidence is again consolidated into memory.

Multi-agent harness and memory lifecycle

RSIAgent decomposes the adaptation loop into three roles: an actor, a verifier, and a curriculum agent. The actor executes code-based actions, produces artifacts, and proposes memory updates. The verifier independently inspects the resulting environment state, artifacts, and execution evidence. The curriculum agent selects subsequent practice tasks based on the target query, prior outcomes, memory, and diagnosed uncertainties.

This separation establishes explicit information boundaries. The verifier does not receive the actor’s private reasoning or memory, reducing the likelihood that it simply ratifies the actor’s interpretation. The curriculum agent selects experiences but does not directly author canonical memory. The actor that generated an experience performs its distillation and reconciliation after receiving the verifier’s report. Parallel projects begin from an immutable memory snapshot, while their updates are reconciled sequentially into a canonical memory bank.

The verifier interface supports PASS, FAIL, and UNVERIFIED. The third state is consequential: unresolved evidence is not converted into either a successful or failed learning example. In principle, this prevents ambiguous observations from becoming durable rules. In practice, the paper’s failure analysis shows that the effectiveness of this safeguard depends on the verifier’s ability to identify all task requirements and distinguish local evidence from global correctness.

The resulting architecture is summarized by the broad-then-deep exploration procedure:

Figure 1: RSIAgent coordinates curriculum generation, code-based execution, independent verification, and frozen-memory reuse across environments.

Broad and deep recursive self-exploration

Broad Recursive Self-exploration (BRS) is the coverage-oriented stage. The curriculum agent generates multiple independent projects spanning prerequisite skills, application variants, failure patterns, and stress cases. These projects execute in parallel, allowing the system to collect heterogeneous experience before updating memory. After each wave, the curriculum agent identifies remaining gaps and generates a new wave or stops.

Deep Recursive Self-exploration (DRS) is sequential and target-conditioned. It begins with an attempt at the target task using the memory acquired through BRS. The verifier’s outcome and the actor’s learning diagnosis determine whether the curriculum agent should propose a supplementary exercise, a contrastive case, or a boundary-condition test. Each experience is verified and consolidated before the next practice decision. The process terminates when the curriculum agent judges that further practice has insufficient expected value, not merely when the agent obtains one successful execution.

Figure 2: BRS provides parallel coverage, while DRS sequentially investigates hard cases and consolidates each verified experience before the next curriculum decision.

The division is empirically motivated. In the four-task ablation, full RSI obtains a mean partial score of 74.54%, compared with 65.52% for BRS alone, 56.50% for DRS alone, and 56.50% for the reported empty-memory baseline comparison as described in the paper’s stage analysis. More specifically, the paper reports that deep-only exploration falls below the baseline on T085 and T089, whereas broad-only exploration improves over the baseline on every task. The result supports a specific ordering hypothesis: DRS is more effective when it receives a broad procedural substrate, while BRS alone lacks sufficient target-specific refinement.

The method’s memory is frozen after exploration. Test-time execution disables curriculum decisions and memory writeback, so the final evaluation measures reuse of an immutable artifact rather than continued online learning. This separation is essential for interpreting the reported gains as memory-mediated adaptation rather than unbounded interaction with the benchmark.

Aggregate benchmark results

RSIAgent is evaluated on 82 tasks from OSWorld 2.0’s 0808 offline split and all 67 tasks in the Near-term subset of Agents’ Last Exam. The primary metrics are partial credit and binary accuracy. Partial credit measures graded task completion, whereas binary accuracy requires full credit.

System	OSWorld partial	OSWorld binary	ALE partial	ALE binary
RSIAgent without RSI	71.97	37.80	83.75	49.25
RSIAgent	78.98	42.68	84.82	50.75
GPT-6 Astra	72.60	Not reported	82.26	52.24
Claude Opus 5	70.19	34.72	79.54	46.27

Relative to the same harness without recursive exploration, RSIAgent improves OSWorld partial score by 7.01 percentage points and binary accuracy by 4.88 points. On Agents’ Last Exam, the corresponding gains are 1.07 and 1.50 points. The paper reports partial-credit advantages over GPT-6 Astra of 6.38 points on OSWorld and 2.56 points on ALE.

These comparisons support the paper’s principal claim that open-source models equipped with recursive exploration and frozen memory can exceed the reported partial scores of stronger closed-source systems. They do not establish a uniformly superior system: on ALE binary accuracy, RSIAgent reaches 50.75%, below GPT-6 Astra’s 52.24%. Moreover, the cross-system comparison is not protocol-matched. Published baselines retain their original harnesses, budgets, and evaluation procedures, while the RSIAgent aggregates replace baseline scores with RSI results for selected tasks and retain baseline scores for others. The numerical comparison is therefore evidence of performance potential, not a controlled model-to-model causal comparison.

The aggregate reporting protocol is particularly important. The OSWorld RSI row contains 41 reported non-diagnostic RSI entries, with the remaining tasks retaining baseline scores; the ALE row combines 19 RSI-column scores with 48 baseline scores. The improvements should consequently be interpreted as improvements to the reported benchmark aggregate under a mixed replacement protocol, not as a complete independent rerun of every task under frozen-memory evaluation.

Task-level effects and memory reuse

The task trajectories illustrate how memory changes execution decisions rather than merely supplying generic advice. In representative OSWorld tasks, scores reach 100% for T044 video editing and T065 railway booking, and 80% for T049 presentation repair after recursive exploration. The trajectories are discontinuous: a long sequence of memory accumulation may produce little immediate score change until one missing procedure or constraint is resolved.

T049 demonstrates the value of retaining failure diagnoses. The first DRS attempt incorrectly interprets a preservation constraint as prohibiting movement of corrupted figure boxes. The resulting failure is consolidated into a rule distinguishing preservation of dimensions and appearance from preservation of erroneous positions. A supplementary presentation exercise then tests the revised interpretation before the target is retried.

The final memory-free and frozen-memory evaluations are especially informative. Both runs recover the broad slide arrangement, but the baseline leaves connector 69 misaligned by 121,104 EMU horizontally and 145,345 EMU vertically, exceeding the scorer’s 100,000-EMU per-axis tolerance. The memory run places the connector endpoint exactly at the required anchor. The task score therefore increases from 0.40 to 0.80, with the entire gain attributable to the slide-3 arrow component. This is a constrained artifact-level improvement rather than a general improvement across all presentation defects: slide 2 remains uncredited in both runs.

Figure 3: T049’s memory-free and frozen-memory outputs differ primarily in the precision of a connector anchor, which changes the task score from 0.40 to 0.80.

T044 shows a different mechanism. The memory-free baseline removes a watermark using generic video filters, but the benchmark checks for an application-native crop operation and a valid crop range. The memory run retrieves Shotcut-specific procedures, applies a native crop with a 76-pixel top offset and center disabled, and produces a smaller valid export. The score rises from 0.40 to 1.00: the memory run satisfies the native crop criterion and the center-off criterion in addition to the size and resolution requirements already satisfied by the baseline.

The gain is not primarily visual. Both runs remove the watermark and preserve the image dimensions, but only the memory run expresses the operation through the application-specific editable structure expected by the scorer. This case demonstrates that reusable memory can encode provenance and tool semantics, not merely visual transformations.

T085 extends the analysis to REAPER audio production. The frozen-memory evaluation increases the partial score from 0.6800 to 0.9417. Sentence-gap credit rises from 0.8250 to 1.0000, while processed-final-sentence credit rises from 0.2837 to 0.8996. Source-order and closing-sting scores remain unchanged. The paper appropriately qualifies these results: the evaluator approximates sentence boundaries through acoustic activity, so the component scores measure the benchmark’s recorded criteria rather than complete semantic or perceptual quality.

Generalization to game development

The paper also evaluates RSIAgent on 40 randomly sampled GameCraft-Bench tasks. In this setting, the system iteratively develops playable games, uses playtesting feedback, and modifies executable game artifacts. RSIAgent is compared with Play2Code and with its own no-RSI configuration across multiple base-game generators.

The full method improves the overall score for every generator:

Base generator	Baseline	Play2Code	RSIAgent without RSI	RSIAgent
Codex + GPT-5.5	52.77	51.05	57.84	61.28
Kimi-K2.6	31.28	36.02	42.61	46.37
GLM-5.3-Flash	30.55	38.25	44.73	48.72
Qwen3.8-27B	41.30	47.67	53.82	57.46

The comparison makes two claims beyond simple improvement. First, the no-RSI RSIAgent harness already outperforms Play2Code in all four groups. Second, recursive experience evolution adds a further 3.64–5.90 points over the no-RSI configuration. Play2Code can improve weaker base games but slightly degrades the strongest generator, whereas RSIAgent improves both. This suggests that the framework’s contribution is not reducible to repeated playtesting; the persistent memory and verification protocol provide reusable diagnostic knowledge across game types.

The game results nevertheless use a distinct configuration from the main computer-use experiments: GLM-5.3-Flash serves as both development actor and verifier, Qwen3.8-27B performs playtesting, and a 15-round development budget is imposed. The evidence therefore supports cross-domain portability of the framework design, but does not isolate which role decomposition or memory mechanism produces the improvement.

Failure mechanisms

The paper’s failure analysis is one of its most important contributions because it rejects the assumption that more exploration automatically produces reliable self-improvement. Three failure mechanisms recur.

Insufficiently targeted exploration occurs when practice expands task diversity without testing the decision responsible for an observed weakness. For example, exploration may cover document processing and persistence while failing to test how unavailable user information should be handled. The resulting memory can appear broad while leaving a decisive target-specific ambiguity unresolved. The implication is that curriculum quality must be assessed by uncertainty reduction and hypothesis discrimination, not by episode count or memory volume.

Incomplete verification occurs when local evidence is mistaken for satisfaction of the complete task specification. A verifier may confirm that an artifact exists or that a transaction completed while failing to check unsupported field values, hidden constraints, or rubric-specific requirements. Because verifier output controls memory consolidation, this failure affects both current execution and all later tasks that retrieve the resulting rule.

Unreliable memory consolidation occurs when a locally accepted interpretation is generalized beyond its evidence. In the audited form-completion cases, the actor retains rules that treat missing-data markers as valid answers or interpret unavailable information as a negative answer. The problem is not simply incorrect retrieval; it is an epistemic failure in which uncertainty and applicability conditions are discarded during distillation.

Figure 4: Audited failure modes show that exploration targeting, verification completeness, and memory consolidation interact as coupled sources of error.

The case studies also reveal that memory growth is not equivalent to append-only accumulation. T044 grows from 109,647 bytes after BRS to 316,541 bytes at the final freeze, T049 from 89,086 to 141,312 bytes, and T065 from 159,065 to 281,085 bytes. Some updates revise existing files or reduce memory size by removing or reorganizing material. This is consistent with the paper’s view of memory as an evolving knowledge base rather than a trajectory archive.

Figure 5: Memory expands through both new records and revisions, with failed projects sometimes producing useful updates.

Limitations and open questions

The reported improvements require substantial additional test-time computation. BRS executes multiple projects, sometimes concurrently, while DRS performs sequential target attempts and supplementary exercises. The paper’s execution limits include up to 86,400 seconds for practice and curriculum configurations and 36,000 seconds for a target run in the OSWorld reference setup. Consequently, the method trades parameter-training cost for inference-time exploration cost; the paper does not provide a systematic quality-per-compute comparison against fine-tuning, retrieval augmentation, or alternative test-time adaptation methods.

The benchmark evidence also has protocol limitations. The main aggregate mixes newly obtained RSI evaluations with retained baseline scores, and the selected RSI tasks are those with room for improvement. The comparisons with frontier systems are drawn from heterogeneous published configurations rather than matched budgets and harnesses. The selected case studies use archival baselines rather than matched-seed ablations, so they establish plausible mechanisms of improvement but do not quantify average treatment effects over task populations.

The verifier remains a model-based component and can issue incorrect judgments. The paper explicitly documents cases in which local verification accepted artifacts that failed official rubric checks. Since verifier judgments influence curriculum selection and memory updates, verification errors can propagate recursively. The method also leaves open how memory should represent uncertainty, scope conditions, contradictory evidence, and environment drift when a previously valid procedure becomes stale.

Finally, the paper does not fully isolate the contributions of role separation, code-as-policy actions, memory representation, curriculum generation, stopping policy, or the BRS/DRS ordering. The game-development evaluation changes the role configuration, and some reported benchmark results use model-based official grading. The open technical question is therefore not whether memory can help in selected environments, but under what verification and curriculum conditions recursive memory construction remains reliable as task distributions, interfaces, and evaluation criteria change.

Conclusion

RSIAgent presents a training-free framework for environment-specific agent adaptation through coordinated curriculum generation, code-based execution, independent verification, and persistent memory consolidation. Its broad-then-deep strategy yields substantial reported gains: OSWorld partial score increases from 71.97% to 78.98%, Agents’ Last Exam partial score reaches 84.82%, and selected artifact-level cases improve from 0.40 to 0.80, 0.40 to 1.00, and 0.6800 to 0.9417.

The strongest evidence concerns procedural and application-specific knowledge that is difficult to infer reliably in a single attempt: native editing operations, geometric anchors, dynamic-interface observation rules, and audio-rendering constraints. At the same time, the paper demonstrates that recursive self-improvement is limited by curriculum misspecification, incomplete verification, and overgeneralized memory. RSIAgent therefore establishes a concrete framework for inference-time adaptation, while leaving the reliability and cost of autonomous knowledge consolidation as the central unresolved problems.