RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experiments

Abstract: For most of scientific history, researchers studying behavior could only infer hidden mechanisms from outward actions: an inverse problem that becomes more tractable when observation is augmented by targeted intervention. We pose a computational analogue: given only behavioral traces of an agent in a game environment, can a learner reconstruct the underlying decision program as executable code, and how much does this reconstruction improve with the ability to design controlled experiments? We introduce RevengeBench, a benchmark of 75 LLM generated, Elo-calibrated policies across five game environments, drawn from CodeClash tournament trajectories. The learner observes the hidden target policy play against sampled opponents and designs behavioral probes in the form of custom opponent policies that elicit informative behavior. It then submits an executable hypothesis, which is evaluated using continuous action-distance metrics. We further validate that recovered code carries informative signal in downstream player-versus-player tournaments. Across twelve frontier LLMs, recovery quality varies substantially (34 to 72% of initial distance closed), with reconstructed policies yielding measurable competitive advantage, particularly for weaker models that otherwise struggle to design effective counter-strategies. Our benchmark positions behavioral recovery of programmatic policies as a tractable inverse problem in code-space, opening a path to opponent modeling, policy interpretability, and the broader question of inferring latent mechanisms from observations.

• The paper introduces RevengeBench, a benchmark for reverse-engineering game policies using LLM agents and controlled probing experiments.
• It employs iterative refinement and active behavioral probing across five competitive arenas to quantify recovery accuracy with measurable improvements.
• Results demonstrate that targeted intervention strategies significantly boost policy reconstruction fidelity, enhancing counter-strategy effectiveness.

Reverse Engineering Executable Game Policies: Detailed Analysis of RevengeBench

Problem Formulation and Benchmark Design

The paper introduces REVENGEBENCH, a benchmark targeting the inverse problem of reconstructing an agent's underlying decision policy as executable code, given only behavioral traces from competitive game environments. The benchmark formalizes behavioral policy recovery as a constrained intervention task: a learner observes trajectories generated by a fixed, hidden policy and is permitted only to probe through custom-designed opponent policies, which elicit informative actions from the target in a closed-loop interaction protocol. The reconstructed hypothesis must match the target’s behavior in held-out states and is scored on a continuous action-distance metric specific to each game.

REVENGEBENCH consists of 75 programmatic target policies sampled from frontier LLM-generated CodeClash artifacts across five arenas—BattleSnake, Halite, Poker, RoboCode, and RobotRumble—spanning discrete and continuous state/action spaces, multi-agent and sequential decision processes, and four programming languages. Targets are Elo-calibrated to enforce difficulty stratification, and the learner operates in a sandboxed environment (mini-SWE-agent), interacting via command-line file/script operations with explicit step and probe budgets. The recovery pipeline thus embodies algorithmic epistemology: passive observation yields ambiguous candidate mechanisms, while active intervention via opponent design can resolve behavioral uncertainties specific to the policy’s logic.

Experimental Evaluation: LLM Agents and Recovery Protocols

The benchmark evaluates twelve frontier LLMs as coding agents for the inverse-policy task, with all models executing iterative refinement loops and, optionally, active probing. Each model runs five rounds of refinement (30 edit steps per round) per target, with evaluations based on the fraction of action distance closed relative to a simplistic starter baseline. Recovery performance distinguishes model tiers: the best models (GPT-5.5, GPT-5) close over 70% of the action distance, mid-tier models (Kimi K2.6, DeepSeek v4. Pro, Gemma 4 31B) achieve 52–60%, and weakest models (e.g., GPT-oss-120b) only 34%. These results substantially outperform the pool-mean baseline, confirming genuine behavioral inference rather than accidental pool matching.

Recovery varies with policy complexity and game dynamics: in Poker, even weaker models achieve high fidelity (>60% recovery), while in arenas like RobotRumble and RoboCode, the model spread is substantial and predictive accuracy diminishes. Strikingly, there is no significant correlation between target policy Elo and recovery difficulty, indicating that policy complexity rather than gameplay performance constrains behavioral identifiability.

Effects of Intervention, Observation Channels, and Recovery Reliability

Active behavioral probing—designing targeted opponent policies—produces model-dependent gains: for high-capability models (DeepSeek v4. Pro, GPT-5), probing yields measurable improvement (up to +0.05 mean gain), especially in complex environments. In weaker models, the probe budget is frequently exhausted on invalid or minimally informative probes, sometimes degrading recovery. Probing success thus depends on the agent’s ability to design and execute strategically informative experiments rather than mere volume.

Alternative observation channels are analyzed: replacing raw traces with LLM-generated natural-language summaries generally weakens recovery fidelity except in specific games (Halite), and Bayesian Program Inference (best-of-N passive batch selection) underperforms iterative refinement for most models. Maintaining persistent context across rounds (with compaction) improves recovery and reduces API cost; per-round context resets uniformly degrade performance.

Reliability is domain-dependent: robustness analysis via multi-seed runs (Gemma 4 31B, 5 seeds per target) reveals high intra-target agreement in Poker and BattleSnake (ICC > 0.8), but substantial noise in Halite, RoboCode, and RobotRumble, making fine-grained ranking comparisons only valid when averaged over repeated runs.

Behavioral Analysis and Failure Modes

The agent action trajectories reveal distinct behavioral clusters, shaped by throughput and refinement strategy. High-throughput models exhaust their step budget on iterative testing cycles and frequent analysis scripts, while low-throughput models favor large edits and minimal test loops. Three primary failure modes emerge:

• Step-budget exhaustion: Repeated script failures preclude timely submission, especially in high-throughput models.
• Regression from best submission: Edit steps sometimes degrade behavioral fit; up to 55% of targets exhibit final-round regression.
• Submission failures: Invalid code crashes are most frequent in weaker models, necessitating stricter tool-use validation.

Strong models craft targeted probes (e.g., deterministic, safety-aware opponent policies in BattleSnake), while weaker models default to trivial baselines (e.g., stationary probes), underscoring the necessity of operational reasoning rather than random experimentation.

Downstream Strategic Utility: Counter-strategy Generation

To assess the practical value of reconstructed policies, the paper conducts player-vs-player tournaments where a challenger LLM writes counter-strategies against a fixed target under three conditions: blind (rules only), recovered (inverse pipeline code), and oracle (target’s true code). Across all models and games, access to recovered/opponent intelligence consistently boosts win rate relative to blind baseline, with a further ceiling achieved by oracle knowledge. Notably, weaker challengers derive the largest gains from opponent models; strong models converge to effective strategies over rounds via iterative adaptation, decaying the intelligence advantage.

This demonstrates that even imperfect behavioral reconstructions confer actionable strategic value, particularly for agents unable to independently synthesize effective counter-strategies—thus positioning executable policy modeling as a powerful tool for opponent modeling and strategy synthesis.

Methodological Implications and Limitations

REVENGEBENCH establishes behavioral recovery in code-space as a tractable inverse problem, providing mechanistic benchmarks for strategic agent modeling, interpretability, and experimental reasoning. The benchmark’s constrained intervention protocol parallels scientific inference where only certain perturbations are feasible, and assessments are stratified by action fidelity and downstream utility.

Limitations include the controlled, synthetic nature of targets (fixed policies, no adaptive or stochastic behavior), the proxy status of action distance as a measure of behavioral fidelity (ignoring rare/adversarial states), and domain-specific robustness nuances. Policy recovery is inherently underdetermined: multiple executable programs may match observed behavior, yielding equivalence classes rather than unique source recovery.

Conclusion

REVENGEBENCH rigorously evaluates the capability of LLM coding agents to reverse engineer executable policies from behavioral traces and controlled interactions in competitive environments (2606.26094). Numerical results confirm substantial but partial recovery of target policies across model tiers and domains. Active probing provides selective gains, conditioned on the agent’s experimental reasoning capabilities. Reconstructed policies are strategically valuable, enabling improved counter-strategy performance, especially in weaker agents. These findings advance the methodological basis for behavioral program inference, opponent modeling, and policy interpretability in agentic AI systems.

Future directions include extension to adaptive, stochastic, or concealing agents, more complex interactive domains (e.g., web or scientific workflows), and formalization of recovery methods to yield distributions or families of plausible policies beyond single-executable hypotheses. The intersection of behavioral evidence, constrained intervention, and executable policy modeling promises enhanced theoretical insight and practical advances for multi-agent AI systems.