---
title: 'SE-Bench: Benchmarking Self-Evolution'
url: https://www.emergentmind.com/topics/se-bench
type: topic
---

# SE-Bench: Benchmarking Self-Evolution

SE-Bench refers to a family of benchmarks centered on software engineering (SE) tasks, with a particular emphasis on the recently introduced "SE-Bench: Benchmarking Self-Evolution with Knowledge Internalization" [2602.04811]. This work represents a shift from traditional static knowledge benchmarks to a focus on lifelong learning, ability for knowledge internalization, and self-evolving behavior in large language models (LLMs) and AI coding agents. The SE-Bench ecosystem now encompasses both classical benchmarks for software issue resolution and diagnostic frameworks for evaluating lifelong learning in artificial agents.

## 1. Historical Context and Motivation

The SE-Bench initiative arises in a landscape dominated by benchmarks such as SWE-Bench and its derivatives (e.g., SWE-Sharp-Bench for C#, Multi-SWE-Bench for multilingual tasks), which have focused on repository-level issue resolution and patch generation. These benchmarks have spurred progress in AI-driven code synthesis and automated bug fixing, providing granular metrics on agent performance in real-world software repositories [2511.02352]. However, previous efforts have not directly measured an agent's ability to internalize genuinely novel knowledge—i.e., learning new APIs or conceptual primitives de novo rather than through dataset contamination or context-window retrieval.

The core motivation for SE-Bench is to rigorously diagnose *self-evolution*: the ability of an autonomous agent to continually update its internal parameters with newly encountered knowledge in a setting where prior training provides no relevant information and inference-time context is unavailable. This problem is complicated by two confounds:

- **Prior knowledge entanglement:** distinguishing true learning of novelty from recollection of pre-existing information.
- **Reasoning complexity entanglement:** separating failures due to lack of knowledge from failures due to general cognitive or computational limits.

SE-Bench creates a needle-in-the-haystack diagnostic where success is impossible without true internalization and trivial once the mapping is learned, providing an unambiguous testbed for self-evolution [2602.04811].

## 2. Formal Framework and Diagnostic Environment

SE-Bench defines *self-evolution* formally: An agent $A$ self-evolves if, after parameter updates, its responses to new tasks improve on instances where all external references are absent. *Knowledge internalization* is the property that a piece of knowledge $K$ is encoded in weights such that closed-book performance (no context) equals performance with context.

### Obfuscation Protocol

- Selects 268 NumPy functions, each obfuscated into a pseudo-novel Python package (`zwc`) with randomized API identifiers.
- Prevents trivial escapes by encapsulating all arrays in a custom class (`ZWCArray`) with only the obfuscated interface exposed.
- Zero-shot baseline performance is rigorously 0%; no agent can succeed without first internalizing the documentation mapping.

### Task Generation

- **Single-function tasks:** LLMs generate isolated coding problems solvable by exactly one obfuscated API call, with at least 8 test cases per instance.
- **Multi-function tasks:** Randomized composite tasks requiring compositions of at least three distinct `zwc` API calls.
- Conservative consensus filtering ensures only tasks solvable by multiple expert models are retained.

### Train/Test Partition

- Training set: 718 single-call tasks (every function appears at least once).
- Test set: 259 new single-call tasks (knowledge retention) and 440 compositional (multi-call) tasks (compositional generalization).

## 3. Learning Paradigms and Experimental Methodology

SE-Bench is explicitly designed to compare core agents and optimization regimes under controlled novelty.

### Supervised Fine-Tuning (SFT)

- *Trajectory collection* phase includes both problem text and relevant docstring.
- *Parameter update* phase (closed-book): docstring is removed, enforcing compression of the mapping into model parameters.
- The SFT loss is
  $$
  \mathcal{L}_{\mathrm{SFT}}(\theta) = -\mathbb{E}_{(x,y)\sim\mathcal{D}}\sum_{t=1}^T\log\pi_\theta(y_t|x, y_{<t}).
  $$

### Reinforcement Learning (RL, PPO-based)

- PPO objective with clipped probability ratios and normalized advantages:
  $$
  \mathcal{L}_{\rm RL}(\theta) =\mathbb{E}_{x\sim\mathcal{D}}\biggl[\frac{1}{N}\sum_{i=1}^N\rho_i(\theta)A_i\biggr]
  $$
  where $\rho_i(\theta) = \min\left(r_i(\theta), \mathrm{clip}(r_i(\theta), 1-\epsilon, 1+\epsilon)\right)$.
- Rewards: +1 for exact correctness, 0 otherwise.

### Self-Play

- Inspired by strategies like Absolute-Zero, alternates between curriculum self-generation and closed-SFT optimization.
- Demonstrates that LLMs can bootstrap knowledge via self-generated data, but only closed-SFT injects the mapping into model weights.

### Metrics

- **Rigor:** Solutions must (i) pass all cases, (ii) demonstrate AST-level use of only `zwc.*`, and (iii) prohibit NumPy access.
- **Pass@k:** Top-64 solution acceptance rate reported for context upper bound, with closed-book zero-shot always 0%.
- **Retention vs. Generalization:** Single-call evaluates recall, multi-call probes ability to compose internalized knowledge.

## 4. Key Findings and Algorithmic Insights

SE-Bench experiments uncover several mechanisms and limitations underpinning self-evolving LLMs:

### Open-Book Paradox

- Supervised fine-tuning (SFT) with documentation present at update time leads to 0% knowledge retention: the agent learns to depend on context rather than encode mappings in weights.
- Closed-book SFT (doc removed during updates) achieves up to 39.6% single-call accuracy on Qwen3-8B and loses no performance if doc is reintroduced at test time. This demonstrates that explicit context removal is necessary for knowledge internalization.

### The RL Gap

- PPO-style RL fails entirely (0% closed-book), regardless of whether training is open- or closed-book.
- Ablations highlight that
  - PPO's probability-ratio clipping impedes large weight updates needed for rare or entirely novel tokens.
  - Negative advantage gradients can erase partial associations during early learning phases.
- Modifying PPO to remove these obstacles restores performance to SFT levels, suggesting that canonical RL objectives are poorly suited for acquisition of atomic, novel mappings.

### Viability of Self-Play with SFT

- RL-driven self-play yields 0% retention.
- Self-play with closed-SFT achieves 22.5% on single-call and 8.7% on multi-call test tasks, establishing that LLMs can self-generate effective learning curricula if paired with the correct optimization protocol.

### Error Dynamics

- SFT-trained models primarily suffer from “hallucination” failures (inventing nonexistent API attributes).
- Additional RL consolidation effectively collapses hallucinations by biasing generations toward robust, previously acquired mappings.

## 5. Implications for Lifelong Learning and Benchmarking

SE-Bench provides a unique, contamination-free, and highly diagnostic environment for assessing whether and how agents convert context or demonstration into parametrized skill. This fills a critical gap left by static benchmarks, which often conflate memory, context-retrieval, and true internal learning [2507.09108, 2505.23419].

### Extensibility and Future Research

- The obfuscation and closed-book paradigm can be systematically transferred to other libraries (Pandas, SciPy) or even to language semantics, enabling a scalable family of synthetic lifelong learning benchmarks.
- Hybrid optimization strategies are suggested: generate data with rich context but enforce context removal during parameter updates ("Open-Book Paradox" inspired hybrid SFT).
- RL may be reintegrated for late-stage consolidation once knowledge fragments have been internalized.
- SE-Bench acts as a "unit test" for when an LLM transitions from context-reliance to direct parameter mastery.

## 6. Relation to Classical SE-Benchmarks

The nomenclature "SE-Bench" is overloaded. Beyond the self-evolution benchmark [2602.04811], SE-Bench (shorthand for "Software Engineering Benchmarks") frequently denotes the family of repository-level benchmarks such as SWE-Bench, SWE-Sharp-Bench, and their successors [2511.02352, 2505.23419]. These traditional SE-Bench tasks assess an agent's ability to resolve issues in real-world repositories—typically by producing patches that cause failing tests to pass under strict, reproducible environment constraints.

While these benchmarks are essential for tracking progress in practical code synthesis, they do not—by construction—disambiguate true learning of novelty from retrieval or prior exposure. Recent variants (e.g., SWE-bench-Live) have moved toward continuous, contamination-resistant evaluation but still principally target code-generation, bug-fixing, or feature-fulfillment under known APIs [2505.23419]. Thus, the self-evolution flavor of SE-Bench occupies a complementary position in the benchmarking ecosystem, specifically targeting intrinsic learning dynamics under strictly novel conditions.

| Benchmark class        | Core Objective                  | Novelty Isolation | Example Reference    |
|-----------------------|---------------------------------|-------------------|---------------------|
| SWE-Bench             | Issue/pull request resolution   | No                | [2505.23419]        |
| SWE-Sharp-Bench       | Repository-level C# bug-fixing  | No                | [2511.02352]        |
| SE-Bench (Self-Evol.) | Knowledge internalization       | Yes (obfuscated)  | [2602.04811]        |

*This taxonomy underscores the methodological innovation of SE-Bench (Self-Evolution) in the broader context of software engineering evaluation frameworks.*

## 7. Significance and Future Directions

SE-Bench's closed-book, obfuscated protocol constitutes a gold-standard diagnostic for tracking the transition from context dependence to true weight-based skill acquisition in LLMs. This is directly relevant for research communities aiming to build lifelong learning agents, develop safe and generalizable AI systems, and understand intrinsic model limitations. SE-Bench also exposes concrete optimization bottlenecks—particularly with canonical RL techniques—clarifying necessary adaptations for effective self-evolving AI.

Planned research extensions include:

- Obfuscation of additional APIs or full language semantics to probe granularity of skill acquisition.
- Construction of benchmarks with more complex compositionality, memory, or planning demands.
- Algorithmic innovation in optimization routines for parameterizing genuinely novel functionality.
- Integration into the broader SE-Bench family to better measure downstream impact of knowledge internalization on real-world software engineering tasks.

SE-Bench thus establishes a reproducible, scalable, and unambiguously interpretable framework at the intersection of lifelong machine learning, agentic reasoning, and software engineering evaluation [2602.04811].

Source: https://www.emergentmind.com/topics/se-bench