Socratic-SWE: Self-Evolving Coding Agents

• Socratic-SWE is a closed-loop self-evolution framework that transforms agents' historical solving traces into structured skills for adaptive task generation.
• It employs a dual role system where the Generator creates repair tasks and the Solver validates and refines them through execution-based checks.
• Experimental results reveal consistent improvements across multiple SWE benchmarks, highlighting the effectiveness of trace-derived skills and curriculum evolution.

Searching arXiv for the specified paper and closely related Socratic SWE/tutoring work. arxiv_search({"query":"(Xiao et al., 5 Jun 2026) Socratic-SWE Self-Evolving Coding Agents via Trace-Derived Agent Skills", "max_results": 5, "sort_by": "submittedDate", "sort_order": "descending"}) arxiv_search({"query":"Socratic-SWE Self-Evolving Coding Agents via Trace-Derived Agent Skills", "max_results": 10, "sort_by": "relevance", "sort_order": "descending"}) Socratic-SWE is a closed-loop self-evolution framework for software engineering agents in which an agent’s own historical solving traces are reused as training substrate rather than treated merely as rollout logs or reward evidence. Introduced in “Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills” (Xiao et al., 5 Jun 2026), the framework distills traces into structured agent skills, uses those skills to generate targeted repair tasks in real repositories, filters candidate tasks through execution-based validation, and scores them with a solver-gradient alignment reward so that admitted tasks are both verifiable and useful for improving the Solver. Under a shared compute budget across multiple self-evolving baselines, the method reports consistent gains on SWE-bench Verified, SWE-bench Lite, SWE-bench Pro, and Terminal-Bench 2.0, reaching 50.40% on SWE-bench Verified after three iterations (Xiao et al., 5 Jun 2026).

1. Motivation and problem formulation

Socratic-SWE addresses a bottleneck in training LLM-driven software engineering agents: the scarcity of high-quality, executable SWE tasks. The paper argues that existing synthetic data pipelines often rely on fixed mutation or bug-injection procedures, with task distributions that are largely independent of the agent’s own weaknesses and training progress. As a result, the training distribution is not adaptive: as the model improves, the useful signal from a fixed task pool shrinks, and learning plateaus. By contrast, every training run already produces rich solving traces consisting of repository inspection, code edits, command execution, test runs, and verification outcomes; these traces expose recurring failure modes, brittle repair behavior, localization errors, and repository structures that confuse the model (Xiao et al., 5 Jun 2026).

The central move is therefore curricular rather than merely diagnostic. Instead of using traces only to compute reward, Socratic-SWE converts them into structured “skills” and then uses those skills to generate the next wave of tasks directly inside real repositories. The resulting curriculum is adaptive because it is conditioned on the current policy’s capability frontier. This makes task synthesis part of the learning loop itself rather than an external preprocessing step.

2. Closed-loop self-evolution pipeline

The framework alternates between two roles implemented with a shared policy $\pi_\theta$: a Generator, which proposes new repair tasks in real repositories, and a Solver, which attempts to solve those tasks by editing code and running tests. The loop has three stages: trace-to-skill distillation, skill-guided task generation plus validation, and Solver training with feedback. The paper summarizes the mechanism as a closed task–trace–skill–task loop: traces become skills, skills guide task creation, generated tasks produce new traces, and the cycle repeats (Xiao et al., 5 Jun 2026).

Given a repository $r \in \mathcal{R}$ and a skill $s \in \mathcal{S}$, the Generator samples a candidate task and verifier,

$$(\tau, v) \sim \pi_\theta(\cdot \mid r, s, \mathcal{E}_t, \mathrm{role}=G),$$

where $\tau$ is the repair objective, $v$ is the executable verification signal, and $\mathcal{E}_t$ is Solver-side evidence gathered from the current iteration. The Solver then samples a patch trajectory,

$$\hat{y} \sim \pi_\theta(\cdot \mid \tau, r, v, \mathrm{role}=S),$$

applies the patch, runs the verifier, and feeds the resulting traces back into the next round. This architecture makes generation adaptive to recent Solver behavior rather than independent of it.

3. Trace-to-skill distillation and the Agent Skill Registry

A defining component of Socratic-SWE is the Agent Skill Registry $\mathcal{S}$. A skill is a structured document with four fields: name, natural-language description, applicability conditions, and ordered list of operations. Skills are therefore not free-form summaries but reusable abstractions that can be retrieved and used programmatically during subsequent task generation (Xiao et al., 5 Jun 2026).

The distillation pipeline has three stages. First, the current policy is run on a seed task set to produce a trace corpus,

$\mathcal{T} = \{\tau_1,\ldots,\tau_N\},$

which is partitioned into successful traces $r \in \mathcal{R}$0, where $r \in \mathcal{R}$1, and failed traces $r \in \mathcal{R}$2, where $r \in \mathcal{R}$3. The paper explicitly retains both: success traces expose reusable repair strategies, while failure traces reveal capability gaps and corrective lessons. Second, a distillation model extracts candidate skills,

$r \in \mathcal{R}$4

Third, the candidate set is deduplicated by semantic similarity and filtered by trace coverage,

$r \in \mathcal{R}$5

The resulting skills capture search-and-localize strategies, debugging patterns, repository-specific semantic conventions, common repair failure modes, and corrective heuristics. The appendix example described for OAuthLib is notably concrete, including scope-conversion contracts, constructor storage pitfalls, OAuth1 parameter confusion, and OIDC inheritance issues. This specificity is important: the registry is intended to encode trace-grounded SWE behavior rather than generic “coding ability.”

4. Repository-grounded task generation, validation, and rewards

Generated tasks are admitted only after a staged validation pipeline executed in a repository sandbox. The validation stages are Format, Grounding, Execution, and Semantics: the task and verifier must be well-formed and parseable; they must reference real repository artifacts; the verifier must run without infrastructure errors and remain stable across repeated runs; and the verifier must separate failing from repaired states with at least one valid repair. The validation indicator is written as

$r \in \mathcal{R}$6

Only tasks that satisfy all stages are added to the curriculum,

$r \in \mathcal{R}$7

This validation regime enforces that generated tasks are executable, reproducible, solvable, and semantically meaningful (Xiao et al., 5 Jun 2026).

The Solver reward is decomposed into full-suite success, partial repair, and regression avoidance:

$r \in \mathcal{R}$8

Here $r \in \mathcal{R}$9 denotes originally failing tests, $s \in \mathcal{S}$0 originally passing tests, $s \in \mathcal{S}$1 failing tests that now pass, and $s \in \mathcal{S}$2 passing tests that remain passing. Training uses GDPO, with each reward component normalized separately before aggregation.

The Generator is trained with a solver-gradient alignment reward. A held-out validation set $s \in \mathcal{S}$3 defines a target gradient direction $s \in \mathcal{S}$4 through per-task Solver gradient estimates,

$s \in \mathcal{S}$5

and

$s \in \mathcal{S}$6

For a candidate generated task with Solver gradient $s \in \mathcal{S}$7, the Generator reward is

$s \in \mathcal{S}$8

The paper’s first-order justification is

$s \in \mathcal{S}$9

Cosine similarity is preferred to the unnormalized inner product because large gradient magnitude can be induced by long or complex tasks that are not necessarily useful.

5. Experimental setup, benchmarks, and results

The experimental configuration uses Qwen3.5-9B as the base model for both Generator and Solver, Qwen3.6-27B as the skill extractor, GDPO for Solver training, GRPO for Generator training, mini-swe-agent as the harness for SWE benchmarks, and little-coder as the harness for Terminal-Bench 2.0. Training runs for 3 iterations with 12k validated instances per iteration, for 36k total, and uses a held-out subset of BeyondSWE, size 100, for gradient alignment. Evaluation is by pass rate on SWE-bench Verified, SWE-bench Lite, SWE-bench Pro Public, and Terminal-Bench 2.0 (Xiao et al., 5 Jun 2026).

Benchmark	Pass rate after 3 iterations
SWE-bench Verified	50.40%
SWE-bench Lite	36.67%
SWE-bench Pro	22.85%
Terminal-Bench 2.0	14.61%
Overall mean	31.13%

Relative to the base agent, these correspond to gains of +7.80 on Verified, +7.00 on Lite, +5.61 on Pro, +4.50 on Terminal-Bench 2.0, and +6.22 overall. Among baselines, R-Zero improves briefly and then regresses below the base agent; SPIRAL and Absolute-Zero improve early and then saturate or regress; Socratic-Zero is stronger than many baselines but peaks early and then declines slightly; and SSR is the strongest baseline, reaching 47.00% on Verified after 3 iterations but still trailing Socratic-SWE by 3.40 points. On Terminal-Bench 2.0, only SSR and Socratic-SWE show meaningful gains, with Socratic-SWE performing better, which the paper interprets as evidence that trace-derived skills capture general agent behaviors such as file manipulation, command chaining, and environment navigation (Xiao et al., 5 Jun 2026).

The scaling experiment on SWE-bench Verified extends training to 5 iterations. Socratic-SWE continues improving to 52.00%, whereas SSR plateaus earlier at 48.00%. The ablation study shows that removing the Skill Registry causes the largest drop, removing trace distillation also hurts, and replacing GDPO with GRPO degrades performance. The paper presents these ablations as support for the claim that the effective unit is not trace reuse alone, but the combination of trace-derived skills, grounded task generation, and decomposed reward learning.

6. Relation to the broader “Socratic” literature

The term “Socratic” has been used across several distinct LLM research programs. In prompt engineering, “Prompting LLMs With the Socratic Method” identifies methods such as definition, elenchus, dialectic, maieutics, generalization, and counterfactual reasoning, and proposes the CRIT workflow for stepwise inspection and verification (Chang, 2023). In reasoning evaluation, SocREval uses Definition, Maieutics, and Dialectic to improve GPT-4 as a reference-free evaluator (He et al., 2023). In education-oriented systems, SocratiCode shifts an adaptive programming tutor toward “guided questioning, reflection prompts, misconception checks, incremental hints, and mandatory pauses for learner input” for K-12 Python learning (Lucas et al., 18 May 2026); SocraticLLM decomposes mathematics tutoring into Review, Guidance / heuristic, Rectification, and Summarization (Ding et al., 2024); and TreeInstruct applies multi-turn, hierarchical questioning to code debugging, estimating a student’s knowledge state and asking sibling or child questions depending on whether the student’s response is incorrect or partially correct (Kargupta et al., 2024).

Against that background, Socratic-SWE occupies a different level of the stack. It is not primarily a learner-facing tutoring interface, a teacher-training simulator, or an LLM-as-judge. Its “Socratic” mechanism lies in self-evolution through structured reflection on traces: the agent’s prior successes and failures are distilled into skills, those skills generate new tasks, and the resulting tasks induce new traces. A plausible implication is that the broader Socratic literature now spans at least three layers: user-facing dialogue and scaffolding, evaluator-side reasoning inspection, and training-time curriculum evolution for agents. Socratic-SWE is characteristic of the third layer because it operationalizes reflection not as questioning in conversation, but as trace-derived curriculum design.

A frequent misconception is therefore to treat Socratic-SWE as a debugging tutor for human programmers. That description fits TreeInstruct more closely than it fits Socratic-SWE. Socratic-SWE is instead a framework for improving SWE agents by mining their own histories.

7. Limitations, constraints, and future directions

The paper is explicit about several constraints. First, the framework operates in a closed-world repository pool; as the skill registry becomes more complete, later iterations discover fewer new gaps, producing redundancy and eventual saturation. Second, the gradient-alignment reward depends on a held-out trusted validation set. If that validation set is unrepresentative, curriculum quality may degrade. Third, the method assumes executable verification and sandboxing: tasks must be testable under deterministic execution. Fourth, evaluation is limited to four benchmarks, so broader transfer across languages, repository styles, and workflows remains unproven (Xiao et al., 5 Jun 2026).

The forward-looking directions named in the paper are dynamic repository augmentation, cross-repository skill transfer, online skill discovery, and more open-ended self-evolution beyond a fixed repository corpus. More broadly, Socratic-SWE reframes training data generation for SWE agents. Instead of treating task synthesis as an external synthetic-data problem, it makes task synthesis endogenous to learning and grounds it in the agent’s own traces, structured skills, executable validation, and validation-aligned gradients. This suggests a shift in how SWE-agent improvement is conceptualized: not as scaling a static task pool, but as iteratively converting behavior into curriculum.