Coding Agents with Environment Interaction: A Theoretical Perspective

Abstract: Coding agents are increasingly utilized in test-driven software development, yet the theoretical mechanisms behind their environment-interaction strategies remain underexplored. We provide a probabilistic framework for two dominant paradigms: code selection after generation using the execution environment, and code generation conditioned on environment feedback. First, we formalize several well-established selection heuristics as environment-aware estimators of code correctness. We theoretically prove that estimators based on fuzzy functional similarity add an inductive bias and strictly dominate estimators based on functional equivalence in terms of signal-to-noise ratio. Second, we frame backprompting as an in-context approximation of Thompson sampling. We derive a novel regret bound for reward functions with unobservable components, theoretically explaining why the effectiveness of backprompting is limited by the ambiguity of the informal task description (an irreducible regret). Using three state-of-the-art open weight models, we corroborate these findings across BigCodeBenchHard, LeetCodeDataset, and QiskitHumanEvalSim. Our formalization also suggests how to improve task descriptions effectively, leading to a new benchmark, QiskitHumanEvalSimX.

• The paper unifies post-generation selection and backprompting as probabilistic methods for using execution environments to improve coding agents.
• Soft functional-similarity estimators consistently outperform hard equivalence counting, with reported gains including 79.56% versus 78.60% for CodeT and 42.30% versus 39.59% for MaxPass.
• Backprompting behaves like in-context Thompson sampling, but ambiguous task descriptions create irreducible regret that execution feedback cannot eliminate, making richer specifications essential.

Overview and motivation

This paper by Menet, Hersche, Krause, and Rahime (IBM Research Zurich / ETH Zurich) develops a probabilistic theory of how LLM-based coding agents exploit an execution environment, covering the two dominant paradigms: post-generation selection (choosing among sampled implementations using test suites) and in-loop backprompting of execution feedback. The central contributions are (i) a formalization of selection heuristics as estimators of a "probability of correctness" over behavioral equivalence classes, with a proof that fuzzy functional similarity strictly dominates functional equivalence in signal-to-noise ratio (SNR), and (ii) an interpretation of backprompting as approximate Thompson sampling, yielding a regret bound that exposes an irreducible regret caused by ambiguity in the informal task description (2602.06098).

Probabilistic setup

The generative model posits a latent abstract algorithm $a \in \mathcal A$ and environment $e \in \mathcal E$, from which an informal task description $d$ is produced as a lossy compression. The agent seeks both a test suite $t$ (executable specification) and implementation $c$, modeled jointly as $p(c,t \mid \mathcal H, d)$. Crucially, $a$ is unobservable beyond $d$, while $e$ can be probed through scalar rewards $R(c,t\,\vert\,e) = \frac{1}{|t|}\sum_i O_i$ or textual reports $e \in \mathcal E$0.

Post-generation selection as correctness estimation

The paper defines functional similarity $e \in \mathcal E$1, proves it is a PSD kernel via one-hot feature maps, extends it to $e \in \mathcal E$2-similarity (also PSD for all $e \in \mathcal E$3, converging to functional equivalence as $e \in \mathcal E$4), and lifts these to fuzzy neighborhoods whose probability measures smooth the distribution $e \in \mathcal E$5 over implementations. Measure smoothing provides an inductive bias: if two codes are similar within $e \in \mathcal E$6, their smoothed probabilities differ by at most $e \in \mathcal E$7.

The key quantitative result is SNR dominance: with true similarity $e \in \mathcal E$8, the ratio of SNRs between the smooth estimator $e \in \mathcal E$9 (averaging per-suite similarities) and the sharp equivalence estimator is at least $d$0. This means aggregating similar behaviors is provably more robust than counting identical outputs, with dominance strongest when generations are diverse ($d$1 small). A dual expression under code-test calibration equates $d$2 with $d$3, and a well-specification assumption yields that greedy selection by $d$4 maximizes Pass@k — establishing this quantity as the canonical target of all selection heuristics. The paper then reinterprets MBR-Exec, AlphaCode, FunCoder, MaxPass, and CodeT uniformly as hard or soft variants of these estimators, introducing new soft versions of MaxPass and CodeT.

Empirically, across BigCodeBenchHard, QiskitHumanEvalSim, and LeetCodeDataset with Qwen3-235B-A22B-Instruct-2507, GPT-OSS-120B, and MiniMax-M2.1, soft variants consistently beat hard ones; e.g., on LeetCode with MiniMax-M2.1, CodeT Soft reaches 79.56% versus 78.60% hard, and MaxPass Soft reaches 42.30% vs 39.59% hard on BigCodeBenchHard with MiniMax-M2.1. Notably, GPT-OSS shows almost no benefit from selection (37.16% random → ~38% best), which the authors attribute to poor test synthesis producing noisy signals.

Backprompting as in-context Thompson sampling

The paper argues that pretraining on description–history–solution trajectories with cross-entropy loss minimizes KL divergence to the conditional distribution over the optimal solution, so conditioning autoregressive generation on past failed attempts approximates posterior sampling — i.e., Thompson sampling. They derive a novel Bayesian regret bound for reward functions with an unobservable component $d$5:

$d$6

where $d$7. The sub-linear term captures learnable uncertainty about the environment; the linear term $d$8 is the irreducible regret: no amount of execution feedback can compensate for an ambiguous task description. Experiments match this prediction precisely: with oracle tests, in-context Thompson sampling achieves Pass@1 of 63.78%, 72.31%, and 72.11% on BigCodeBenchHard, QiskitHumanEvalSim, and LeetCodeDataset — exceeding even unconditioned Pass@10 (46.35%, 67.83%, 28.60%). With self-generated tests, gains transfer fully on LeetCode (whose descriptions contain input–output examples, reducing $d$9), partially on Qiskit, and not at all on BigCodeBenchHard. To confirm causality, the authors construct QiskitHumanEvalSimX by enriching descriptions with natural-language test specifications (via Gemini 3 Pro); the accuracy delta over round one rises from 3.21% to 7.13%. Ablations show test-first factorization $t$0 works best, and summary concatenation beats insight reformulation for feedback compression.

Limitations and open questions

The Gaussianity assumption on rewards over token strings is acknowledged as stylized, adopted only for closed-form bounds. Selection requires cross-executing $t$1 codes against $t$2 suites ($t$3): on QiskitHumanEvalSimX, post-generation selection consumed 21h32m versus 12h42m for generation itself. Context growth from summary concatenation is linear per round, interacting quadratically with attention cost (118.9M tokens processed, only 17.0M newly generated). The strong dependence of all results on high-quality self-generated tests — exposed starkly by GPT-OSS-120B and MiniMax-M2.1 failing to benefit from feedback — remains unresolved. Open questions include whether architectures with linear attention complexity can sustain long interaction histories, and how to reduce $t$4 without leaking solutions into prompts.

Conclusion

The paper supplies the first unified theoretical account of environment-aware selection and backprompting for coding agents: soft similarity-based estimators provably dominate hard equivalence counting, and backprompting's ceiling is set by task-description ambiguity through a linear irreducible-regret term. Both claims are validated across three benchmarks and three open-weight frontier models, and the derived benchmark QiskitHumanEvalSimX operationalizes the prescription that richer task specifications, not just stronger models, drive agentic coding performance.