RevengeBench: Inverse Code Recovery Benchmark

• RevengeBench is a benchmark for reconstructing executable agent strategies from behavioral traces, emphasizing the inverse problem of code recovery.
• It integrates passive trajectory observation with active probing, allowing iterative refinement of code hypotheses to enhance strategic performance.
• Empirical findings show that top LLMs recover 34–72% of agent logic, underscoring the benchmark’s potential in advancing machine-driven experimental reasoning.

RevengeBench is a benchmark and evaluation suite for studying the inverse problem of reconstructing an agent’s hidden decision program—expressed as executable code—solely from its behavioral traces in game-based environments, optionally augmented by constrained interventions. It asks whether, given only state–action trajectories of a black-box policy playing a game, a learner can recover the underlying programmatic strategy as runnable code, and how that recovery changes when the learner can design behavioral probes through gameplay rather than directly query latent states. The benchmark is introduced in "RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experiments" (Rahmani et al., 24 Jun 2026).

1. Benchmark definition and construction

RevengeBench is organized around 75 hidden target policies distributed across five arenas drawn from the CodeClash competition: BattleSnake, Halite, Poker/HuskyBench, RoboCode, and RobotRumble (Rahmani et al., 24 Jun 2026). These environments span discrete and continuous control, varying observability, and distinct action-space structures. The targets are described as LLM generated and Elo-calibrated, and the benchmark is grounded in CodeClash tournament trajectories.

From approximately 15,000 CodeClash submissions, policies are filtered, deduplicated, and stratified by structural complexity to produce a 40-policy pool per game. Swiss-system Elo tournaments are then held among these 40 policies; the top 15 policies in each arena become the hidden targets, yielding 75 targets in total, and the remaining 39 serve as the opponent pool for generating behavioral traces (Rahmani et al., 24 Jun 2026). This construction ties the benchmark to competitive policies rather than synthetic toy agents.

Environment	Language	Action modality
BattleSnake	Python	discrete 4-direction moves
Halite	C	per-cell discrete commands in a grid
Poker/HuskyBench	Python	discrete fold/check/call/raise with continuous raise sizes
RoboCode	Java	5-dimensional continuous control: velocity, body/gun/radar turn rates, fire power
RobotRumble	JavaScript	multi-unit per-unit type+direction assignments

The benchmark’s design positions policy recovery as a code-space problem rather than a pure imitation-learning problem. The target is not merely to match actions statistically, but to submit an executable hypothesis that captures latent strategic structure sufficiently well to generalize under evaluation and downstream competition.

2. Formalization of the code-space inverse problem

RevengeBench formulates recovery in terms of behavioral traces generated by an unknown target policy $\pi^* \colon \mathcal S \to \mathcal A$. For any pooled or probe opponent policy $o$, rolling out $\pi^*$ against $o$ produces a trajectory

$\tau = (s_0, a_0, \dots, s_T)$

with $a_t = \pi^*(s_t)$ (Rahmani et al., 24 Jun 2026). The learner passively observes batches of such trajectories and must infer an executable approximation to the target.

The benchmark’s central intervention model is constrained. In the active regime, the learner may author up to $B$ probe opponents per round. Each probe is itself executable code, designed to drive the hidden policy into informative regions of state space. The example given is cornering a BattleSnake agent against a wall to reveal tie-breaking heuristics (Rahmani et al., 24 Jun 2026). This is an important restriction: interventions occur only through the environment’s ordinary interaction channels, not through arbitrary access to latent memory, hidden variables, or internal activations.

This setup is explicitly motivated by scientific inverse problems in domains such as biology and psychology, where causes are unobservable and must be inferred from observable effects, and where targeted interventions can disambiguate competing hypotheses. In that sense, RevengeBench operationalizes what it calls an intermediate epistemic regime: the learner has more than passive observation, but less than privileged mechanistic access (Rahmani et al., 24 Jun 2026). This suggests that the benchmark is intended to test machine-driven experimental reasoning in addition to policy modeling.

3. Closed-loop reconstruction protocol

The learner operates over $R$ refinement rounds in a closed loop. In each round, it first observes trajectories generated against sampled opponents, then compares its current executable hypothesis to the target’s observed behavior, edits code, optionally designs probes, and finally emits an updated hypothesis $\hat\pi_r$ (Rahmani et al., 24 Jun 2026).

The mismatch representation is defined as

$$\mathcal M_r = \bigl\{ (s_t,\hat\pi_{r-1}(s_t),\pi^*(s_t)) : s_t \in \tau \in \mathcal T_r,\; \hat\pi_{r-1}(s_t)\neq \pi^*(s_t) \bigr\}.$$

At the start of round $o$0, the learner samples 20 opponents $o$1 and collects trajectories $o$2. It then enters an iterative edit–probe loop under a step budget: propose code changes to $o$3 via file writes, execute evaluation scripts to recompute $o$4, and, if the round’s probe budget has not been exhausted, write and submit a probe opponent, observe its new trace, and update $o$5 (Rahmani et al., 24 Jun 2026).

Implementation uses the mini-SWE-agent scaffold, described as a bash toolbox inside a sandboxed Docker environment. LLMs including GPT-5, GPT-5.4-mini, Gemma-4-31B, and DeepSeek-v4 are used to generate, debug, and refine code (Rahmani et al., 24 Jun 2026). The protocol therefore evaluates not only inference quality, but also tool use, code editing, execution, debugging, and evidence integration under iterative feedback.

4. Evaluation metrics and downstream validation

RevengeBench evaluates recovered hypotheses with continuous action-distance metrics defined separately for each game. Across environments, a symmetric game-specific distance $o$6 is used, satisfying $o$7 and $o$8 if either action is invalid. Trajectory-level fidelity is measured by the mean per-step distance

$o$9

Per target and model, the benchmark reports

$\pi^*$0

with lower values better, and normalized recovery

$\pi^*$1

where $\pi^*$2 is the starter policy’s distance (Rahmani et al., 24 Jun 2026).

The benchmark specifies environment-dependent distance functions. In BattleSnake, the four-direction metric is the indicator $\pi^*$3. In Poker, fold versus continuation is treated as maximally distant, while non-fold actions use a graded logarithmic distance based on raise fraction $\pi^*$4 and stack $\pi^*$5 (Rahmani et al., 24 Jun 2026). These choices matter because they encode partial similarity among actions rather than reducing all disagreements to a zero-one mismatch.

RevengeBench also includes a downstream player-versus-player validation. A challenger LLM writes a counter-policy over five rounds against a fixed target under three information conditions: Blind, Recovered, and Oracle. Blind provides rules only; Recovered provides the code output by the inverse pipeline; Oracle provides the true target source code (Rahmani et al., 24 Jun 2026). The benchmark measures win rate and defines lift relative to the blind condition. This auxiliary evaluation tests whether recovered code is strategically useful even when it is not behaviorally exact.

5. Empirical findings

Across twelve frontier LLMs, normalized recovery $\pi^*$6 spans 34% to 72% of initial behavioral distance closed (Rahmani et al., 24 Jun 2026). The top cluster consists of GPT-5.5-high, GPT-5, and GPT-5.5-low/medium, each exceeding approximately 68%. Mid-tier models such as Kimi K2.6, DeepSeek v4 Pro, and Gemma-4 31B reach 52–60%, while the remainder fall in the 42–50% range.

Difficulty varies substantially by environment. Poker is reported as the easiest, with recovery up to 82%, whereas RobotRumble is the most variable, spanning 5–89% (Rahmani et al., 24 Jun 2026). A plausible implication is that action-space structure and the informativeness of observed trajectories materially affect whether behavioral traces suffice to constrain the latent program.

The benchmark also reports that passive iterative refinement already closes most of the gap. Active probing yields significant gains for strong models, with DeepSeek v4 Pro receiving a +5.2% mean boost and GPT-5 a +1.3% boost, but the effect is negligible or mixed for weaker models (Rahmani et al., 24 Jun 2026). The reported explanation is procedural rather than merely statistical: probe success requires valid code generation and targeted experimental design. This makes probing a compound capability, not a guaranteed source of additional signal.

In downstream competition, the ordering Oracle > Recovered > Blind holds across all challenger models (Rahmani et al., 24 Jun 2026). Weaker challengers obtain larger win-rate gains from opponent intelligence, and recovered code confers measurable advantage even when imperfect. This directly rebuts the common assumption that only exact source-code recovery is useful: RevengeBench reports that partial recovery is already strategically actionable.

6. Interpretation, scope, and future directions

The benchmark’s main conclusion is that behavioral signatures of programmatic policies are surprisingly rich: passive observation alone recovers a substantial fraction of agent logic (Rahmani et al., 24 Jun 2026). It also concludes that active intervention helps only when the learner can plan, implement, and integrate experiments, because probing demands tool use, debugging, and evidence assimilation. Together, these findings frame policy recovery as an inverse problem in code-space rather than solely a behavioral cloning problem.

RevengeBench is explicitly positioned as a tractable code-space inverse problem for opponent modeling and policy interpretability (Rahmani et al., 24 Jun 2026). The benchmark therefore sits at the intersection of behavioral inference, executable program synthesis, and multi-agent evaluation. Its use of constrained intervention through gameplay makes it relevant to settings where direct introspection is unavailable but experimental manipulation is possible.

The paper identifies several future directions: adaptive or stochastic targets, richer domains such as web agents and scientific workflows, and methods that recover distributions or equivalence classes of plausible policies rather than a single best-guess executable (Rahmani et al., 24 Jun 2026). This suggests a shift from exact reconstruction toward structured uncertainty over latent mechanisms. It also suggests that the benchmark’s current deterministic executable-hypothesis format may be only one point in a broader design space for mechanism discovery from behavior.

Taken together, RevengeBench defines a concrete evaluation framework for inferring hidden decision programs from trajectories, measuring the value of both passive observation and constrained intervention, and testing whether recovered mechanisms have downstream competitive utility. Within that scope, it offers a formal and empirical basis for studying machine-driven experimental reasoning in multi-agent systems.