EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments

Abstract: Autonomous agents are increasingly expected to improve executable policies through feedback, yet existing evaluations often collapse this process into a final score or confound it with open-ended software-engineering progress. We introduce Autonomous Policy Evolution, a controlled evaluation setting in which a harness-model agent repeatedly edits an executable policy system under a fixed interaction budget. We instantiate this setting in EvoPolicyGym, a benchmark built from compact interactive RL environments that evaluates how agents iteratively improve explored policies. On the EvoPolicyGym suite, GPT-5.5 achieves the strongest aggregate rank score and top-two performance on all 16 environments. Beyond leaderboard results, EvoPolicyGym also provides trajectory-level diagnostics that distinguish how agents allocate budget, convert feedback into parametric tuning. These analyses show that strong autonomous policy evolution depends not only on isolated task wins, but on discovering task-appropriate mechanisms and refining policies under bounded feedback.

• The paper introduces Autonomous Policy Evolution to evaluate sequential policy improvements with strict feedback and episode budgets.
• It employs a comprehensive protocol across varied environments, distinguishing synthesis-driven structural edits from mere parameter tuning.
• Experimental results highlight significant performance differences among agents, emphasizing the need for robust structural reasoning and effective policy-edit strategies.

EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments

Problem Formulation and Benchmark Motivation

EvoPolicyGym introduces "Autonomous Policy Evolution" as a formal problem setting assessing an agent's capacity for iterative self-improvement under strict feedback and budget constraints. Unlike prior RL agent benchmarks—which typically conflate final performance with software engineering progress or focus on one-shot task completion—EvoPolicyGym isolates the policy revision process: agents sequentially edit executable policies, receiving feedback after each submit, under a fixed episode interaction budget. The evaluation object is not static policy performance but the dynamic sequence of policy improvements, budget decisions, and mechanisms by which agents convert feedback into generalizable policy modifications.

This formulation targets key dimensions neglected by open-ended coding or long-horizon software benchmarks, such as SWE-bench or SLopcodeBench. Those focus on evolving codebases or repository maintenance, often conflating engineer-like tool use, spec drift, and patch validation. By contrast, EvoPolicyGym constrains the evolving object to the decision policy itself, enforces a fixed (and limited) opportunity for feedback acquisition, and hides validation and held-out splits during policy search. This strict protocol prevents overfitting and diagnostic leakage, enabling controlled measurement of how agents trade off exploration, exploitation, and mechanism synthesis when improving agent behavior in diverse RL environments.

Protocol and Environment Suite

The benchmark protocol uses reproducible Gymnasium-style environments, spanning classic control, MuJoCo, MiniGrid, robotics, and simulated driving—covering both low-dimensional continuous control and high-level symbolic planning. Each agent iteration consists of:

• Maintaining a live policy workspace.
• Submitting executable policy systems for episodic rollouts within a global episode budget.
• Receiving granular feedback: per-episode rewards, trajectory records, error diagnostics, and aggregate statistics.
• Revising code to address observed failures or optimize performance.
• Repeating the loop until the budget is exhausted, at which point the server evaluates all submitted checkpoints on hidden validation and held-out splits.

Environments are designed for both "synthesis-dominant" tasks (requiring new structural mechanisms, e.g., pixel-based driving with perception, symbolic navigation) and "tuning-dominant" tasks (amenable to parametric adjustment, e.g., linear control, physical gaits).

Experimental Results and Diagnostic Analyses

The Core16 suite, containing 16 heterogeneous environments, is employed for systematic comparison. Four agent systems (GPT-5.5 with Codex harness, Claude Opus 4.7, MiniMax-M3, DeepSeek-V4-Pro—all with Claude Code harness) are evaluated under identical 128-episode interaction budgets.

Numerical Findings

• GPT-5.5 achieves the highest aggregate ranking (Core16 score 0.891), securing top-two placement in all environments. Claude Opus 4.7 ranks second overall (0.750) and leads on MiniGrid tasks. MiniMax-M3 and DeepSeek-V4-Pro achieve isolated environment wins but lack broad coverage.
• Aggregate rank-based scores expose the stringency of the benchmark: only agents delivering reliably high performance across synthesis- and tuning-dominant tasks achieve high scores.
• Raw held-out means (environment native scale) reveal that agent superiority is not uniform; for example, Claude Opus 4.7 dominates key symbolic tasks, while GPT-5.5 is more robust on continuous control and driving.
• The normalized improvement scale (random-to-best anchor): On synthesis-dominant tasks, GPT-5.5 and Claude Opus 4.7 reach 0.98–1.00, while the others are near-random; on tuning-dominant tasks, the gap narrows substantially.

Trajectory- and Edit-Level Diagnostics

EvoPolicyGym enables efficient post-hoc diagnostic analysis unavailable in prior agent benchmarks:

• Trajectory Analysis: By reconstructing best-so-far validation scores over the episode-budget trajectory, agents are distinguishable not just by final policy quality but also by the temporal signature of improvement events—i.e., early synthesis jumps versus late, incremental parameter tuning.
• Edit-Type Attribution: High-performing agents (GPT-5.5, Claude Opus 4.7) make successful "synthesis edits" (introducing novel control or perception structures) at high rates in synthesis tasks, while others exhibit "structural churn" with little effect. In tuning tasks, performance converges quickly once a viable family is instantiated, and most gains arise from parametric edits.
• Qualitative Feedback Utilization: Case studies show that stronger agents attribute performance failures to specific structural deficiencies, make targeted policy edits, and manage candidate checkpoints efficiently. Weaker agents exhibit rollback loops or repeated mechanism swaps with little validation gain.

Mechanistic Policy Insights

Selected policy excerpts illustrate the nature of evolved solutions:

• CarRacing: Strong agents synthesize vision-based state extraction, combine curvature- and center-line-based control, and implement explicit recovery logic based on failed rollouts.
• HalfCheetah: Oscillatory open-loop controllers, modulated with safety scaling and tunable phase/amplitudes, emerge via systematic parametric refinement following structural discovery.
• MiniGrid (ObstructedMaze): Successful policies construct persistent egocentric maps and use BFS planning—agent must maintain memory across steps and plan symbolic action sequences.
• FetchPush: Task-specific manipulation sequences synthesized as phase-based geometric controllers; subsequent parameter adjustment tunes efficiency and robustness.

Implications and Future Directions

EvoPolicyGym concretizes an agent evaluation paradigm emphasizing feedback-driven iterative policy synthesis and tuning under realistic interaction budgets and strict visibility boundaries. This paradigm exposes limitations in current agent architectures:

• Effective self-improvement depends on hierarchical exploration: Agents must first synthesize viable high-level mechanism (perception, planning, phase machines), then tune for task-specific robustness.
• Persistent performance gaps are diagnostic: Failures to realize mechanism-level improvements on synthesis tasks cannot be offset by parameter sweeps, revealing agent brittleness in generalization and code-structural understanding.
• Leaderboard rankings alone are insufficient: Temporal analysis of policy evolution, candidate management, and feedback integration pathways are essential for understanding and improving agent architectures.

The framework highlights the need for advances in agents' structural reasoning, feedback attribution, and generalization from sparse feedback, informing research into robust tool-using coding agents, program synthesis, and agentic RL interfaces. Future directions include automated analysis of code-edit graphs, richer environment task types, scaling budgets with complexity, and integration of conventional RL baselines under comparable feedback constraints.

Conclusion

EvoPolicyGym sets a new standard for evaluating agent-driven autonomous policy evolution, focusing on how agents exploit bounded feedback and make mechanism-level improvements over time. It demonstrates that robust, generalizable self-improvement requires both effective structural synthesis and strategic budget allocation, motivating future work on language-model-driven feedback attribution and policy search heuristics. The benchmark and tooling provide a concrete, extensible protocol for tracking the evolution of executable policies—paving the way for diagnostics-driven improvement of AI agents in dynamic, interactive settings (2607.02440).