---
title: 'RevengeBench: Inverse Code Recovery Benchmark'
url: https://www.emergentmind.com/topics/revengebench
type: topic
---

# RevengeBench: Inverse Code Recovery Benchmark

RevengeBench is a benchmark and evaluation suite for studying the inverse problem of reconstructing an agent’s hidden decision program—expressed as executable code—solely from its behavioral traces in game-based environments, optionally augmented by constrained interventions. It asks whether, given only state–action trajectories of a black-box policy playing a game, a learner can recover the underlying programmatic strategy as runnable code, and how that recovery changes when the learner can design behavioral probes through gameplay rather than directly query latent states. The benchmark is introduced in "RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experiments" [2606.26094].

## 1. Benchmark definition and construction

RevengeBench is organized around 75 hidden target policies distributed across five arenas drawn from the CodeClash competition: BattleSnake, Halite, Poker/HuskyBench, RoboCode, and RobotRumble [2606.26094]. These environments span discrete and continuous control, varying observability, and distinct action-space structures. The targets are described as LLM generated and Elo-calibrated, and the benchmark is grounded in CodeClash tournament trajectories.

From approximately 15,000 CodeClash submissions, policies are filtered, deduplicated, and stratified by structural complexity to produce a 40-policy pool per game. Swiss-system Elo tournaments are then held among these 40 policies; the top 15 policies in each arena become the hidden targets, yielding 75 targets in total, and the remaining 39 serve as the opponent pool for generating behavioral traces [2606.26094]. This construction ties the benchmark to competitive policies rather than synthetic toy agents.

| Environment | Language | Action modality |
|---|---|---|
| BattleSnake | Python | discrete 4-direction moves |
| Halite | C | per-cell discrete commands in a grid |
| Poker/HuskyBench | Python | discrete fold/check/call/raise with continuous raise sizes |
| RoboCode | Java | 5-dimensional continuous control: velocity, body/gun/radar turn rates, fire power |
| RobotRumble | JavaScript | multi-unit per-unit type+direction assignments |

The benchmark’s design positions policy recovery as a code-space problem rather than a pure imitation-learning problem. The target is not merely to match actions statistically, but to submit an executable hypothesis that captures latent strategic structure sufficiently well to generalize under evaluation and downstream competition.

## 2. Formalization of the code-space inverse problem

RevengeBench formulates recovery in terms of behavioral traces generated by an unknown target policy $\pi^* \colon \mathcal S \to \mathcal A$. For any pooled or probe opponent policy $o$, rolling out $\pi^*$ against $o$ produces a trajectory
$$
\tau = (s_0, a_0, \dots, s_T)
$$
with $a_t = \pi^*(s_t)$ [2606.26094]. The learner passively observes batches of such trajectories and must infer an executable approximation to the target.

The benchmark’s central intervention model is constrained. In the active regime, the learner may author up to $B$ probe opponents per round. Each probe is itself executable code, designed to drive the hidden policy into informative regions of state space. The example given is cornering a BattleSnake agent against a wall to reveal tie-breaking heuristics [2606.26094]. This is an important restriction: interventions occur only through the environment’s ordinary interaction channels, not through arbitrary access to latent memory, hidden variables, or internal activations.

This setup is explicitly motivated by scientific inverse problems in domains such as biology and psychology, where causes are unobservable and must be inferred from observable effects, and where targeted interventions can disambiguate competing hypotheses. In that sense, RevengeBench operationalizes what it calls an intermediate epistemic regime: the learner has more than passive observation, but less than privileged mechanistic access [2606.26094]. This suggests that the benchmark is intended to test machine-driven experimental reasoning in addition to policy modeling.

## 3. Closed-loop reconstruction protocol

The learner operates over $R$ refinement rounds in a closed loop. In each round, it first observes trajectories generated against sampled opponents, then compares its current executable hypothesis to the target’s observed behavior, edits code, optionally designs probes, and finally emits an updated hypothesis $\hat\pi_r$ [2606.26094].

The mismatch representation is defined as
$$
\mathcal M_r
=
\bigl\{
(s_t,\hat\pi_{r-1}(s_t),\pi^*(s_t))
:
s_t \in \tau \in \mathcal T_r,\;
\hat\pi_{r-1}(s_t)\neq \pi^*(s_t)
\bigr\}.
$$
At the start of round $r$, the learner samples 20 opponents $O_r \subset \mathcal O$ and collects trajectories $\mathcal T_r$. It then enters an iterative edit–probe loop under a step budget: propose code changes to $\hat\pi_{r-1}$ via file writes, execute evaluation scripts to recompute $\mathcal M_r$, and, if the round’s probe budget has not been exhausted, write and submit a probe opponent, observe its new trace, and update $\mathcal M_r$ [2606.26094].

Implementation uses the mini-SWE-agent scaffold, described as a bash toolbox inside a sandboxed Docker environment. LLMs including GPT-5, GPT-5.4-mini, Gemma-4-31B, and DeepSeek-v4 are used to generate, debug, and refine code [2606.26094]. The protocol therefore evaluates not only inference quality, but also tool use, code editing, execution, debugging, and evidence integration under iterative feedback.

## 4. Evaluation metrics and downstream validation

RevengeBench evaluates recovered hypotheses with continuous action-distance metrics defined separately for each game. Across environments, a symmetric game-specific distance $d_g(a_1,a_2)\in[0,1]$ is used, satisfying $d_g(a,a)=0$ and $d_g=1$ if either action is invalid. Trajectory-level fidelity is measured by the mean per-step distance
$$
D(\hat\pi,\pi^*;\tau)
=
\frac{1}{T}\sum_{t=1}^T d_g\bigl(\hat\pi(s_t),\pi^*(s_t)\bigr).
\tag{1}
$$
Per target and model, the benchmark reports
$$
D_{\min}=\min_{r\le R}\mathbb E_{\tau\sim\mathcal T_r}D(\hat\pi_r,\pi^*;\tau),
$$
with lower values better, and normalized recovery
$$
\Delta = 1 - D_{\min}/D_0,
$$
where $D_0$ is the starter policy’s distance [2606.26094].

The benchmark specifies environment-dependent distance functions. In BattleSnake, the four-direction metric is the indicator $d(a,b)=\mathbf 1[a\neq b]$. In Poker, fold versus continuation is treated as maximally distant, while non-fold actions use a graded logarithmic distance based on raise fraction $\rho_i$ and stack $S$ [2606.26094]. These choices matter because they encode partial similarity among actions rather than reducing all disagreements to a zero-one mismatch.

RevengeBench also includes a downstream player-versus-player validation. A challenger LLM writes a counter-policy over five rounds against a fixed target under three information conditions: Blind, Recovered, and Oracle. Blind provides rules only; Recovered provides the code output by the inverse pipeline; Oracle provides the true target source code [2606.26094]. The benchmark measures win rate and defines lift relative to the blind condition. This auxiliary evaluation tests whether recovered code is strategically useful even when it is not behaviorally exact.

## 5. Empirical findings

Across twelve frontier LLMs, normalized recovery $\Delta$ spans 34% to 72% of initial behavioral distance closed [2606.26094]. The top cluster consists of GPT-5.5-high, GPT-5, and GPT-5.5-low/medium, each exceeding approximately 68%. Mid-tier models such as Kimi K2.6, DeepSeek v4 Pro, and Gemma-4 31B reach 52–60%, while the remainder fall in the 42–50% range.

Difficulty varies substantially by environment. Poker is reported as the easiest, with recovery up to 82%, whereas RobotRumble is the most variable, spanning 5–89% [2606.26094]. A plausible implication is that action-space structure and the informativeness of observed trajectories materially affect whether behavioral traces suffice to constrain the latent program.

The benchmark also reports that passive iterative refinement already closes most of the gap. Active probing yields significant gains for strong models, with DeepSeek v4 Pro receiving a +5.2% mean boost and GPT-5 a +1.3% boost, but the effect is negligible or mixed for weaker models [2606.26094]. The reported explanation is procedural rather than merely statistical: probe success requires valid code generation and targeted experimental design. This makes probing a compound capability, not a guaranteed source of additional signal.

In downstream competition, the ordering Oracle > Recovered > Blind holds across all challenger models [2606.26094]. Weaker challengers obtain larger win-rate gains from opponent intelligence, and recovered code confers measurable advantage even when imperfect. This directly rebuts the common assumption that only exact source-code recovery is useful: RevengeBench reports that partial recovery is already strategically actionable.

## 6. Interpretation, scope, and future directions

The benchmark’s main conclusion is that behavioral signatures of programmatic policies are surprisingly rich: passive observation alone recovers a substantial fraction of agent logic [2606.26094]. It also concludes that active intervention helps only when the learner can plan, implement, and integrate experiments, because probing demands tool use, debugging, and evidence assimilation. Together, these findings frame policy recovery as an inverse problem in code-space rather than solely a behavioral cloning problem.

RevengeBench is explicitly positioned as a tractable code-space inverse problem for opponent modeling and policy interpretability [2606.26094]. The benchmark therefore sits at the intersection of behavioral inference, executable program synthesis, and multi-agent evaluation. Its use of constrained intervention through gameplay makes it relevant to settings where direct introspection is unavailable but experimental manipulation is possible.

The paper identifies several future directions: adaptive or stochastic targets, richer domains such as web agents and scientific workflows, and methods that recover distributions or equivalence classes of plausible policies rather than a single best-guess executable [2606.26094]. This suggests a shift from exact reconstruction toward structured uncertainty over latent mechanisms. It also suggests that the benchmark’s current deterministic executable-hypothesis format may be only one point in a broader design space for mechanism discovery from behavior.

Taken together, RevengeBench defines a concrete evaluation framework for inferring hidden decision programs from trajectories, measuring the value of both passive observation and constrained intervention, and testing whether recovered mechanisms have downstream competitive utility. Within that scope, it offers a formal and empirical basis for studying machine-driven experimental reasoning in multi-agent systems.

Source: https://www.emergentmind.com/topics/revengebench