SPADE: Self-Play in Adaptive Synthetic Executable Environments

Abstract: Continuous self-improvement requires an ever-expanding pool of self-generated, diverse, adaptive goals. For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales. We introduce SPADE (Self-Play in Adaptive Synthetic Executable Environments), a self-play RL framework in which a single LLM plays two roles: an Environment Designer that writes complete, long-horizon training environments as executable code with an OpenAI Gym-style reset()/step() interface, and a Reasoning Agent that learns to act in them. Each is a stateful, multi-turn environment (state transitions, reward functions, and verification code), so one interface spans reasoning problems and multi-step agentic tool use. The Reasoning Agent's regret is estimated using the gap between its reward with and without privileged hints; in optimizing this regret signal the Environment Designer learns to target environments at the edge of the agent's capabilities while keeping them feasible. Through extensive experimentation, we find several components critical to success: grounding the Environment Designer on documents sampled from a large pretraining corpus, and giving it an accumulated environment memory. Scaling to 30B-parameter models, SPADE improves over the strongest fixed-environment baseline by +5.3 on average across eight held-out math, science, code, and reasoning benchmarks, and lifts the tool-use setting by +5.7 on BFCL-v4 multi-turn and +13.9 on ACEBench-Agent; on the games setting, the margin over the strongest baseline grows with model scale. By making environment design itself a learnable component, SPADE takes a concrete step toward open-ended self-improvement.

• The paper introduces SPADE, a shared LLM framework in which an Environment Designer generates executable Python MDPs while a Reasoning Agent learns through hinted and unhinted self-play.
• The method uses hint-based regret, corpus grounding, and environment memory to target learnable difficulty, maintain semantic diversity, and adapt curricula as the agent improves.
• The paper reports gains of up to 8.1 points on games benchmarks and 13.9 points on ACEBench-Agent, while showing that adaptive design transfers to procedural reasoning and multi-turn tool use.

SPADE addresses a specific limitation of reinforcement-learning post-training for language agents: the training environment distribution is usually fixed while the policy changes. Its central claim is that environment generation should itself be optimized as part of the learning system. The framework therefore uses one LLM in two role-conditioned modes: an Environment Designer (ED) that generates executable environments and a Reasoning Agent (RA) that interacts with them. The environments are not merely prompts or question–answer pairs. They are Python programs implementing a Gym-style reset()/step() interface, with state transitions, reward functions, termination conditions, and verification logic. This permits a common training formulation for single-turn reasoning games, stateful multi-turn games, and simulated tool-use workflows (2608.19197).

Problem formulation and contribution

The paper situates SPADE within unsupervised environment design (UED), asymmetric self-play, and synthetic environment generation. Existing approaches address environment scarcity through human-curated pools, frozen synthetic generators, adaptive sampling from static environments, or self-play task generation. SPADE argues that these approaches leave two important constraints unresolved. First, a fixed pool eventually becomes mastered or saturated. Second, task-level synthesis generally produces a terminal problem and verifier rather than a complete interactive MDP with stateful dynamics.

SPADE changes the object being generated. The ED produces an executable environment $e$ together with a privileged hint $h$. The RA then plays the same environment in two independent conditions: with and without $h$. The RA is trained using the environment’s correctness reward. The ED is trained using the return difference between the hinted and unhinted plays. Both roles share parameters, although role-specific prompts determine which behavior is elicited.

The code-as-environment representation is consequential because it expands the design space beyond a fixed parameterization such as maze size or terrain friction. In principle, any computable MDP expressible in the execution sandbox can be proposed, including environments with hidden state, branching interaction, partial rewards, tool schemas, and multi-turn user instructions. In practice, this space remains bounded by the model’s ability to write valid, solvable code, a limitation the paper explicitly acknowledges.

Hint-based regret as the curriculum signal

The ED reward is based on privileged-hint regret:

$r_D(e) = \bar r_A(e \mid h) - \bar r_A(e),$

where $\bar r_A(e)$ is the RA’s mean return without the hint and $\bar r_A(e \mid h)$ is its mean return with the hint. The intended regime is an environment that the RA can solve when supplied with an informative strategy but cannot reliably solve unaided. Such an environment is neither already mastered nor infeasible.

This signal distinguishes three cases. If the RA succeeds both with and without the hint, the environment is too easy. If it fails under both conditions, the environment is too difficult or invalid. A positive gap identifies a potentially learnable frontier. The design is therefore more constrained than a purely adversarial ED objective: an adversary maximizing difficulty can generate unsolvable tasks, whereas hint-regret rewards tasks for which privileged information closes a measurable performance gap.

The paper supplements regret with a flat-top difficulty anchor. The anchor favors environments for which the RA’s no-hint win rate lies in a target band, principally $[0.4,0.6]$, while the floored regret term selects environments with especially useful hint-induced improvement inside that band. In the deployed reward mixture, the difficulty anchor has weight $0.6$ and the normalized regret term weight $0.4$. The implementation also uses per-role advantage normalization, delayed ED updates, truncated importance sampling for the resulting off-policy objective, asymmetric clipping, and a nonnegative regret floor.

The theoretical analysis establishes a stronger result only under idealized assumptions. If hints always provide the optimal unhinted value and hinted behavior can be internalized by some policy in the attainable policy class, then every pure Nash equilibrium has zero hint-regret and the RA is optimal without hints on every valid environment. The conclusion follows because any environment with positive regret would be a profitable point-mass deviation for the ED. These assumptions are not guaranteed in the practical system: hints may be incomplete or misleading, the policy class is finite in effective capacity, returns are estimated from finite rollouts, and generated environments are only approximately sound. The theorem therefore provides an incentive interpretation rather than an optimality guarantee for the implemented algorithm.

The qualitative examples show that the gap is behaviorally meaningful rather than merely a scalar artifact. In one environment, the hint exposes the relevant probe pattern; in another, it narrows the search over a hidden parameter. The hinted and unhinted agents consequently follow different action sequences and attain different returns.

Figure 1: Privileged hints alter the Reasoning Agent’s interaction strategy and produce positive return gaps on stateful environments.

Corpus grounding and environment memory

SPADE relies on two mechanisms with distinct functions. Corpus grounding supplies semantic breadth. In the games setting, the ED conditions on documents sampled from mathematics and science corpora; in tool use, it uses a code and API-oriented corpus. The document is not directly presented to the RA and is not itself treated as a training answer. Instead, it serves as an external source from which the ED derives an executable environment.

Environment memory supplies longitudinal adaptation. The memory stores previously generated environments with regret scores and skill labels, including high-regret examples and environments that were too easy or too difficult. The ED can therefore vary difficult precedents rather than repeatedly designing from scratch. The corpus primarily affects what the environments are about; memory primarily affects where their difficulty lies relative to the current RA.

The ablations separate these effects. With corpus grounding, the normalized Vendi diversity is approximately $0.68$ for full SPADE and $h$0 for a fixed-designer control that retains corpus access. Removing the corpus reduces the value to $h$10.94$h$20.34$$. The no-corpus run eventually emits the same <code>RotatingMazeEnv</code> family 41 consecutive times. Thus, <strong>the corpus, rather than ED learning or memory, is the principal source of semantic diversity</strong>. ED learning changes the difficulty of a broad stream; it does not by itself prevent <a href="https://www.emergentmind.com/topics/mode-collapse" title="" rel="nofollow" data-turbo="false" class="assistant-link" x-data x-tooltip.raw="">mode collapse</a>.</p> <p>The adaptive curriculum is visible over training. Early environments are often simple or reveal their solution structure directly. Later environments increasingly require hidden-state discovery, sequential interaction, and state-gated decisions. The formula-reveal rate in the physics subset falls from$$h$3 to $h$4 across 473 environments. At the same time, the mean number of distinct reward levels increases from $h$5 to $h$6, including strictly partial reward levels increasing from $h$7 to $h$$8. These changes suggest that ED training sharpens task structure and reward granularity without simply increasing program length or reducing validity.</p> <p><img src="https://images.emergentmind.com/paper-images/2608-19197/env_curriculum_filmstrip.png" alt="Figure 2" title="" class="markdown-image" loading="lazy"></p> <p><p class="figure-caption">Figure 2: The generated curriculum shifts from simpler tasks toward state-gated, multi-turn environments as the Reasoning Agent improves.</p></p> <p>The fraction of generated environments in the learnable win-rate interval$$h$9 rises from $h$0 early in training to $h$1 late, while the RA’s mean win rate increases from $h$2 to $h$397$h$498\%$h$590$h$693\%$, and generated programs remain around 320 lines with roughly 13 hidden-state variables. The implication is that the increase in useful training supply is primarily attributable to improved difficulty targeting, not to the ED learning to emit shorter or less demanding programs.

Training procedure and experimental design

The canonical experiments use GRPO with 400 training rollouts, 24 environments per rollout, and groups of 16 RA trajectories per environment. The ED regenerates environments every four rollouts. The RA plays each environment repeatedly without hints and also performs hinted plays used to estimate regret. Candidate code undergoes syntax and execution checks; tool-use environments receive additional reset-gate and semantic solvability checks.

The games setting rotates among six cognitive skills: mathematical reasoning, logical deduction, spatial reasoning, pattern recognition, optimization, and causal inference. Three skills are active at a time, with eight environments per skill in each batch. Evaluation includes AIME 2025 and 2026, GPQA-Diamond, LiveCodeBench-v6, and four hard Reasoning-Gym categories. The tool-use setting generates simulated APIs, backend state, and three to five sequential user instructions. Success requires completing all instructions, making the environments structurally closer to BFCL multi-turn, $h$7-bench, and ACEBench-Agent than to single-turn function-calling prompts.

The evaluation protocol compares SPADE with fixed-environment GRPO and Fixed-env RLVE. Both fixed baselines are retrained from the same backbone for the same nominal 400-iteration budget. This is an important control because SPADE’s advantage could otherwise be attributed simply to additional RL rather than to adaptive environment generation.

Games-setting results

At 30B-A3B, SPADE achieves an eight-benchmark mean of $h$8, compared with $h$9 for the untrained base and $r_D(e) = \bar r_A(e \mid h) - \bar r_A(e),$0 for the strongest fixed-environment baseline, Fixed-env RLVE. Thus, SPADE improves over base by $r_D(e) = \bar r_A(e \mid h) - \bar r_A(e),$1 points and over Fixed-env RLVE by $r_D(e) = \bar r_A(e \mid h) - \bar r_A(e),$2 points. The paper reports improvements over the strongest fixed baseline on all three Qwen3 backbones, with the margin increasing with model scale.

Backbone	Base average	Fixed-env RLVE	SPADE	SPADE gain over base
Qwen3-4B	38.9	42.5	44.1	+5.2
Qwen3-8B	49.8	53.8	55.5	+5.7
Qwen3-30B-A3B	50.2	53.0	58.3	+8.1

The strongest improvements occur in procedural reasoning. At 30B-A3B, SPADE raises the four Reasoning-Gym categories by $r_D(e) = \bar r_A(e \mid h) - \bar r_A(e),$3 points in RG-Math, $r_D(e) = \bar r_A(e \mid h) - \bar r_A(e),$4 in RG-Algorithmic, $r_D(e) = \bar r_A(e \mid h) - \bar r_A(e),$5 in RG-Cognition, and $r_D(e) = \bar r_A(e \mid h) - \bar r_A(e),$6 in RG-Logic relative to the base. The gains are not confined to the generated game distribution: GPQA-Diamond rises by $r_D(e) = \bar r_A(e \mid h) - \bar r_A(e),$7 points and LiveCodeBench-v6 by $r_D(e) = \bar r_A(e \mid h) - \bar r_A(e),$8, while competition mathematics is preserved or modestly improved. This transfer supports the paper’s claim that the generated environments train general procedural and interactional competencies rather than merely memorized game-specific strategies.

Figure 3: Synthetic games improve science, code, and procedural reasoning while preserving competition-math performance.

The training trajectories provide a behavioral account of these results. At the beginning, the RA often derives an answer before interacting and is unable to recover from interface or format errors. Later, it probes the environment, updates hypotheses from feedback, and derives only after sufficient evidence has been collected. This evidence-first behavior is especially relevant to stateful environments, where the optimal policy depends on information acquired through action rather than on static deduction from the initial prompt.

Figure 4: Reasoning behavior shifts from front-loaded derivation toward evidence-driven interaction over training.

Curriculum breadth is also material. Restricting the curriculum from six skills to two reduces the best eight-benchmark mean from $r_D(e) = \bar r_A(e \mid h) - \bar r_A(e),$9 to $\bar r_A(e)$0 and captures only about half of the Reasoning-Gym gains. The result argues against attributing SPADE’s performance solely to a particular game family. Broad skill coverage appears to be an important condition for transfer to science, code, and heterogeneous procedural benchmarks.

Tool-use results

The tool-use experiments test whether the same framework can train multi-turn function-calling behavior. Each environment contains simulated tools, mutable backend state, sequential user instructions, and per-instruction criteria. The privileged hint provides a high-level plan but not the exact arguments, so the RA must still inspect state and perform the appropriate calls.

At 30B-A3B, SPADE improves the BFCL v4 multi-turn average from $\bar r_A(e)$1 to $\bar r_A(e)$2 ($\bar r_A(e)$3), $\bar r_A(e)$4-bench from $\bar r_A(e)$5 to $\bar r_A(e)$6 ($\bar r_A(e)$7), and ACEBench-Agent from $\bar r_A(e)$8 to $\bar r_A(e)$9 ($\bar r_A(e \mid h)$0). The largest gain occurs on ACEBench-Agent, whose database-like state, tool schema, and multi-call workflows most closely match the generated environments. At 4B, the BFCL gain is larger, $\bar r_A(e \mid h)$1, while the gains at 8B and 30B-A3B are $\bar r_A(e \mid h)$2 and $\bar r_A(e \mid h)$3, respectively.

The uneven benchmark gains are informative. SPADE does not produce a uniform improvement across all tool-use evaluations; transfer is strongest when the evaluation’s interaction structure resembles the synthetic training MDPs. This supports a structural-transfer interpretation, but it also limits the claim of domain-general tool-use improvement. Comparisons with dedicated synthesis systems are additionally qualified because the reference systems use different base models, data, budgets, benchmark versions, and simulator configurations.

Adaptation, ablations, and scaling

The component ablations indicate that the complete system is necessary for sustained improvement. Removing memory while retaining ED training yields a best eight-benchmark mean of $\bar r_A(e \mid h)$4. Removing corpus grounding yields $\bar r_A(e \mid h)$5. A fixed GPT-5.5 ED with corpus grounding and memory reaches $\bar r_A(e \mid h)$6. By contrast, removing both ED training and memory produces a mean of $\bar r_A(e \mid h)$7, which is $\bar r_A(e \mid h)$8 points below the untrained base. The frozen-designer controls often peak early and decline later, whereas full SPADE remains strongest late in the 400-step run.

These results support the paper’s strong claim that a fixed frontier model is not an adequate substitute for an adapting ED. The claim should be interpreted cautiously because the frozen controls do not isolate every factor perfectly: one removes both ED training and memory, while another swaps the shared model for GPT-5.5. Nevertheless, the pattern is consistent with the proposed mechanism: a static designer can produce useful environments, but it does not track the evolving RA as effectively as joint co-adaptation.

The reward ablation further distinguishes hint-regret from an EMA-based learning-potential signal. Hint-regret reaches an eight-benchmark mean of $\bar r_A(e \mid h)$9 ($[0.4,0.6]$0 over base), whereas the EMA signal reaches $[0.4,0.6]$1 ($[0.4,0.6]$2). The EMA reward uses unsigned deviation from a running per-skill mean, so it can assign similar scores to mastered and hopeless environments. It also requires history before its statistics become informative. Hint-regret measures the current policy’s information-sensitive performance gap directly and therefore identifies the frontier earlier and more sharply.

Scaling favors SPADE. Average gain over base rises from $[0.4,0.6]$3 at 4B to $[0.4,0.6]$4 at 8B and $[0.4,0.6]$5 at 30B-A3B, while matched-budget Fixed-env GRPO remains near $[0.4,0.6]$6. At smaller scales, empirical hint-regret estimates can become negative because hints may mislead the current policy or because finite-sample estimates are noisy. The models nevertheless improve, indicating that the generated environments can remain useful even when the formal regret interpretation is imperfect.

Figure 5: SPADE’s gain increases with model scale, whereas fixed-environment GRPO remains nearly flat.

A second-backbone experiment on Nemotron-30B-A3B-BF16 provides a limited cross-family check. All four Reasoning-Gym categories finish above the untrained base, with gains of $[0.4,0.6]$7 in RG-Cognition, $[0.4,0.6]$8 in RG-Algorithmic, $[0.4,0.6]$9 in RG-Math, and $0.6$0 in RG-Logic. Because the authors do not report the full games benchmark suite on Nemotron, this result establishes transfer of training dynamics rather than a fully matched cross-family comparison.

Limitations and open questions

SPADE’s theoretical guarantee depends on assumptions that are substantially stronger than the implementation. In particular, the hint must attain the optimal unhinted value, and hinted behavior must be representable without the hint. Real generated hints can be incomplete, misleading, or strategically misaligned with the verifier. The authors observe negative empirical regret on smaller models, directly demonstrating that the ideal nonnegativity property does not hold for finite-sample current-policy estimates.

The environment space is also bounded by the ED’s model capacity, context window, generation budget, code reliability, and sandbox. Raw generated-code executability is only $0.6$1 before repair and $0.6$2 after stripping Markdown fences; the reported post-filter validity reaches $0.6$3 because the training pipeline sanitizes or rejects failures. This distinction matters: the system’s usable environment stream is not equivalent to unconstrained raw generation.

The evaluation remains fixed-task evaluation. It measures transfer to held-out benchmarks rather than open-ended capability growth, and no formal result establishes that the generated curriculum is globally optimal. The paper also uses a human-designed GRPO optimizer and does not demonstrate self-modification of the learning rule. Finally, the tool-use comparisons are not fully controlled against prior systems, and several ablations confound ED training, memory, and designer identity. A specific unresolved question is whether in-context accumulation of environment-design strategies could replace or complement gradient updates to the ED while preserving the same late-training adaptability.

Conclusion

SPADE makes environment design an RL-trained role in a shared LLM self-play system. Its main technical elements are executable MDP generation, corpus-grounded diversity, memory-based frontier targeting, and hint-based regret that rewards environments solvable with privileged information but not yet reliably solvable without it. Across Qwen3 backbones, the method improves the eight-benchmark games suite by $0.6$4, $0.6$5, and $0.6$6 points over base at 4B, 8B, and 30B-A3B, respectively, and improves multi-turn tool-use benchmarks by up to $0.6$7 points. The ablations indicate that corpus grounding sustains breadth, while ED training and memory sustain adaptive difficulty. The evidence supports adaptive environment generation as a substantive component of LLM post-training, while leaving the robustness of the regret objective, the limits of executable environment synthesis, and the relationship between benchmark transfer and genuinely open-ended improvement unresolved.