---
title: Stateful SWE-Bench Evaluation
url: https://www.emergentmind.com/topics/stateful-swe-bench
type: topic
---

# Stateful SWE-Bench Evaluation

Stateful SWE-bench is a class of software engineering (SWE) benchmarks and evaluation protocols that incorporate persistent, temporally evolving state — either in the user/interaction context, system environment, or reasoning history — into the assessment of autonomous coding agents. Unlike stateless, one-shot code-generation challenges, stateful SWE-bench settings require agents to model or leverage accumulated context, maintain or manipulate execution/workspace state, and reason over multi-turn or multi-phase solution trajectories. Modern stateful SWE-bench variants encompass multi-session user modeling, end-to-end environment–implementation–verification pipelines, continual-learning task streams, and state-aware benchmarking methodologies.

## 1. Conceptualization and Motivation

Stateful SWE-bench emerges in response to the inadequacy of traditional stateless evaluations for realistic, autonomous software agent assessment. Legacy benchmarks such as HumanEval, MBPP, and the original SWE-bench [2310.06770] treat each task as an independent, atomic event: the agent receives a textual issue description and a static codebase snapshot, produces code edits, and is immediately scored by a test suite. These paradigms do not reflect the ongoing nature of real-world software engineering, where agents must:

- Retain and exploit user or system preferences across sessions
- Coordinate sequential development phases (e.g., environment setup, implementation, test generation)
- Transfer knowledge between temporally ordered tasks
- Manage the cumulative effects of actions in dynamic interactive settings

Stateful SWE-bench protocols thus simulate or instrument these longitudinal phenomena, enabling rigorous evaluation of agent capabilities in realistically evolving contexts [2510.21903, 2605.13139, 2507.00014, 2606.17261].

## 2. Formal Task Definitions and State Representations

Leading stateful SWE-bench protocols formally specify tasks as Markov Decision Processes (MDPs), with domain-specific definitions of state $S$, action space $A$, and deterministic or stochastic transition functions $T$. The "SWE-Cycle" benchmark [2605.13139] exemplifies this approach:

- **Environment Reconstruction (Env):**
  - $S_{env}$: On-disk repository snapshot (files, folder tree), no dependencies
  - $A_{env}$: System/environment mutating commands (e.g., install, write, configure)
  - $T_{env}(s_t, a_t) = s_{t+1}$: Transition to state where $a_t$’s side effects are applied

- **Code Implementation (Impl):**
  - $S_{impl}$: Pre-configured codebase, issue description $I$, test suite $T_{gold}$
  - $A_{impl}$: Patch operations over AST or text (insert, delete, modify code)
  - $T_{impl}(s_t, a_t) = s_{t+1}$: Updated codebase after applying $a_t$

- **Verification Test Generation (TestGen):**
  - $S_{test}$: Patched codebase, issue description $I$, reference tests
  - $A_{test}$: Add/modify test files, configure runners
  - $T_{test}(s_t, a_t) = s_{t+1}$: State updated to contain agent-authored discriminative tests

- **FullCycle (End-to-End):**
  - $S_{full}$: Bare repository, issue description $I$, empty execution environment
  - $A_{full}$: $A_{env} \cup A_{impl} \cup A_{test}$ in unified, uninterrupted session
  - $T_{full}(s_t, a_t) = s_{t+1}$ for entire sequence, storing all state in a single container with zero external resets

State passing is explicit: each phase serializes its output (environment snapshot, repo state, test suite) as input for downstream phases, enforcing strict continuity and prohibiting human intervention.

## 3. User and Memory Modeling in Stateful Interactions

Stateful SWE-bench methodologies extend state tracking to user-agent interaction history and persistent user preferences. The ToM-SWE framework [2510.21903] introduces:

- **Developer Profiles ($p$):** Encapsulate interaction traits (verbosity, question timing), coding preferences (frameworks, libraries)
- **Session Histories ($H_p$):** Sets of prior user-agent transcripts, exposing temporal dependencies
- **Persistent User Model ($M_p$):** Inferred via theory-of-mind (ToM) agents; consumed by the SWE agent when choosing actions

A typical policy incorporates both in-session context ($c^i_t$) and long-term user modeling:

$$ a_t = \pi(a | c^i_t, M_p) $$

Tasks are evaluated on agents' ability to (1) infer persistent preferences, (2) respect user styles, and (3) minimize unnecessary clarifications. This simulation is typically realized by a profile-conditioned user simulator and task generator, with satisfaction and efficiency metrics scored by LLM-powered evaluators.

## 4. Evaluation Protocols and Metrics

Stateful SWE-bench evaluation metrics measure phase-resolved correctness, state integrity, and learning/dynamic adaptation:

- **Phase Scores (SWE-Cycle) [2605.13139]:**
  - Static and dynamic sub-scores $\mathrm{Stat}_{i,p}, \mathrm{Dyn}_{i,p} \in \{0,1,2\}$ per instance/phase
  - Normalized aggregate: $\overline{\rm Score}_p=\frac1N\sum_{i=1}^N\text{Score}_{i,p}$ and solved rate
  - End-to-end: $\overline{\rm Score}_{\rm full}$ and $\mathrm{SolveRate}_{\rm full}$ computed as means over composite phase scores

- **Continual Learning Metrics (SWE-Bench-CL) [2507.00014]:**
  - **Average Accuracy (AA):** Mean final performance over sequence
  - **Forgetting (F):** Loss from peak prior performance on earlier tasks
  - **Forward/Backward Transfer (FT/BWT):** Gains/losses on new/prior tasks due to incremental learning
  - **Composite Continual-Learning Score (CCLS):** Weighted sum of above, with stable-plasticity F$_\beta$ harmonics

- **Stateful Decision-Problem Benchmarking [2606.17261]:**
  - Replaces naive absolute metrics with contrast (delta) estimators: $\Delta_{f,g} = \bar{T}_f - \bar{T}_g$
  - Experiments use randomized or blocked trial designs for consistent identification of optimal programs, accounting for uncontrolled environmental state drift

## 5. Implementation Strategies and Agent Architectures

Modern stateful SWE-bench protocols leverage system architectures and memory systems capable of retaining, retrieving, and condensing long-run interaction or reasoning histories:

- **Graph-based Agent State (LangGraph) [2507.00014]:** Encodes agent observations, actions, plans, and tool invocations as nodes in a persistent, updatable graph structure at each turn
- **FAISS-based Semantic Memory [2507.00014]:** Stores representations (embeddings) of prior solved tasks, facilitating retrieval of analogous experience for new tasks
- **Dynamic Reasoning Contexts (SWE-AGILE) [2604.11716]:** Maintains a sliding window of detailed reasoning over N recent steps, storing earlier reasoning as digests. Compression functions $\phi$ optimize semantic fidelity under token constraints. This approach mitigates context window limitations and preserves deep System-2 reasoning chains

Pseudocode implementations exemplify strict state-carryover, memory-prompting, and hybrid retrieval-training structures.

## 6. Empirical Findings and Performance Analysis

Empirical studies across benchmarks reveal that:

- **Cross-phase or cross-session state dependencies substantially increase task difficulty.** In SWE-Cycle, the solve rate for FullCycle (simultaneous end-to-end, stateful execution) drops below 14%, even as best-in-class agents achieve 78%–97% per-phase accuracy in isolation. Most of the dynamic boost in end-to-end setting is offset by cumulative static errors and verification bottlenecks [2605.13139].
- **User modeling and memory improve objective and subjective outcomes.** ToM-SWE achieves 59.7% success on stateful scenarios, far exceeding the 18.1% baseline, and earns consistently higher satisfaction scores via persistent preference tracking [2510.21903].
- **Continual learning agents with external memory show improved accuracy, reduced forgetting, and better transfer** compared to memoryless baselines. Memory-aware architectures support faster resolution of recurring bug types and more efficient adaptation to evolving codebases, as measured by AA, F, FT, and CCLS [2507.00014].
- **Contrast-based estimators yield consistent decisions** even under unobservable, stateful environmental drift, avoiding the misleading bias of traditional averaging protocols [2606.17261].

## 7. Practical Recommendations and Future Directions

Best practices for constructing and deploying stateful SWE-bench protocols include:

- Instrument all benchmarks to pass state explicitly between phases or sessions; prohibit human resets or ad hoc re-initializations.
- Utilize explicit user simulators or developer profiles to test agents' ability to recognize, retain, and exploit long-term user context.
- Cap memory context to avoid overflow, and filter prior experiences for relevance and correctness.
- Adopt randomized or block-based experiment designs for performance benchmarking under stateful system dynamics, always relying on relative (contrast) metrics rather than absolute averages.
- Incorporate continual fine-tuning, curriculum ordering, and diagnostic logging to enable adaptive, robust agent training.

Further research is exploring algorithmic improvements in memory compression, adaptive window sizing, theory-of-mind augmentation, and cross-language or cross-domain generalization.

---

Stateful SWE-bench defines a suite of rigorous, state-dependent evaluations for autonomous code agents, encompassing environment manipulation, persistent user modeling, multi-turn continual learning, and robust benchmarking under dynamic system state. State formalization, explicit memory, and hybrid evaluation protocols are essential for measuring, and ultimately achieving, practical agent autonomy in real-world software engineering [2310.06770, 2510.21903, 2604.11716, 2507.00014, 2605.13139, 2606.17261].

Source: https://www.emergentmind.com/topics/stateful-swe-bench