SPADE: Self-Play in Adaptive Synthetic Executable Environments
SPADE transforms environment design into a learned component of reinforcement learning by training a language model to generate executable Python environments that adapt to an agent's evolving skill frontier. Using hint-based regret as a curriculum signal, the system produces stateful multi-turn tasks that improve reasoning, code generation, and tool use by up to 8 points over fixed-environment baselines while sustaining diversity through corpus grounding and memory-based adaptation.Script
Most reinforcement learning systems train agents in fixed environments, but what if the training world itself could learn? SPADE addresses this by making environment design an adaptive role inside a self-play loop, where one language model generates executable Python programs as training tasks and another model learns to solve them.
The system generates complete Markov decision processes, not just question and answer pairs. Each environment is a Gym-style Python program with state transitions, rewards, and verification logic. The environment designer is rewarded when a privileged hint closes a performance gap, identifying tasks the agent can solve with guidance but not yet reliably master alone.
The curriculum adapts as the agent improves. Early environments often reveal their solution structure directly, but later tasks increasingly require hidden state discovery and sequential interaction. The fraction of learnable environments rises from 16 percent to 31 percent, while formula reveal rates in physics tasks drop from 25 to 5 percent.
SPADE improves over the strongest fixed-environment baseline across all three model scales, with gains rising from 5.2 points at 4 billion parameters to 8.1 points at 30 billion. The improvement transfers beyond generated games: science reasoning rises by 5.4 points and code generation by 4.1 points, suggesting the system trains general procedural competencies rather than narrow game-specific strategies.
Component ablations reveal that the complete system is necessary. Removing environment designer training and memory together drops performance 9.7 points below the untrained base. Corpus grounding sustains semantic diversity, preventing mode collapse, while memory and adaptive training sustain difficulty targeting. A frozen frontier model reaches only 53 points compared to 58.3 for full SPADE, demonstrating that static designers cannot track an evolving agent as effectively as joint co-adaptation.
SPADE demonstrates that environment design can be learned rather than fixed, opening a new axis for scaling reinforcement learning systems. The framework achieves its strongest transfer when evaluation structures match the generated training tasks, yet the limits of executable synthesis and the gap between benchmark transfer and open-ended capability growth remain unresolved. To explore how adaptive curricula might reshape language model training and to create your own research videos, visit EmergentMind.com.
