---
title: 'Socratic-SWE: Self-Evolving Coding Agents'
url: https://www.emergentmind.com/topics/socratic-swe
type: topic
---

# Socratic-SWE: Self-Evolving Coding Agents

Searching arXiv for the specified paper and closely related Socratic SWE/tutoring work.
arxiv_search({"query":"2606.07412 Socratic-SWE Self-Evolving Coding Agents via Trace-Derived Agent Skills", "max_results": 5, "sort_by": "submittedDate", "sort_order": "descending"})
arxiv_search({"query":"Socratic-SWE Self-Evolving Coding Agents via Trace-Derived Agent Skills", "max_results": 10, "sort_by": "relevance", "sort_order": "descending"})
Socratic-SWE is a closed-loop self-evolution framework for software engineering agents in which an agent’s own historical solving traces are reused as training substrate rather than treated merely as rollout logs or reward evidence. Introduced in “Socratic-SWE: Self-Evolving Coding Agents via Trace-Derived Agent Skills” [2606.07412], the framework distills traces into structured agent skills, uses those skills to generate targeted repair tasks in real repositories, filters candidate tasks through execution-based validation, and scores them with a solver-gradient alignment reward so that admitted tasks are both verifiable and useful for improving the Solver. Under a shared compute budget across multiple self-evolving baselines, the method reports consistent gains on SWE-bench Verified, SWE-bench Lite, SWE-bench Pro, and Terminal-Bench 2.0, reaching 50.40% on SWE-bench Verified after three iterations [2606.07412].

## 1. Motivation and problem formulation

Socratic-SWE addresses a bottleneck in training LLM-driven software engineering agents: the scarcity of high-quality, executable SWE tasks. The paper argues that existing synthetic data pipelines often rely on fixed mutation or bug-injection procedures, with task distributions that are largely independent of the agent’s own weaknesses and training progress. As a result, the training distribution is not adaptive: as the model improves, the useful signal from a fixed task pool shrinks, and learning plateaus. By contrast, every training run already produces rich solving traces consisting of repository inspection, code edits, command execution, test runs, and verification outcomes; these traces expose recurring failure modes, brittle repair behavior, localization errors, and repository structures that confuse the model [2606.07412].

The central move is therefore curricular rather than merely diagnostic. Instead of using traces only to compute reward, Socratic-SWE converts them into structured “skills” and then uses those skills to generate the next wave of tasks directly inside real repositories. The resulting curriculum is adaptive because it is conditioned on the current policy’s capability frontier. This makes task synthesis part of the learning loop itself rather than an external preprocessing step.

## 2. Closed-loop self-evolution pipeline

The framework alternates between two roles implemented with a shared policy $\pi_\theta$: a **Generator**, which proposes new repair tasks in real repositories, and a **Solver**, which attempts to solve those tasks by editing code and running tests. The loop has three stages: trace-to-skill distillation, skill-guided task generation plus validation, and Solver training with feedback. The paper summarizes the mechanism as a closed task–trace–skill–task loop: traces become skills, skills guide task creation, generated tasks produce new traces, and the cycle repeats [2606.07412].

Given a repository $r \in \mathcal{R}$ and a skill $s \in \mathcal{S}$, the Generator samples a candidate task and verifier,
$$
(\tau, v) \sim \pi_\theta(\cdot \mid r, s, \mathcal{E}_t, \mathrm{role}=G),
$$
where $\tau$ is the repair objective, $v$ is the executable verification signal, and $\mathcal{E}_t$ is Solver-side evidence gathered from the current iteration. The Solver then samples a patch trajectory,
$$
\hat{y} \sim \pi_\theta(\cdot \mid \tau, r, v, \mathrm{role}=S),
$$
applies the patch, runs the verifier, and feeds the resulting traces back into the next round. This architecture makes generation adaptive to recent Solver behavior rather than independent of it.

## 3. Trace-to-skill distillation and the Agent Skill Registry

A defining component of Socratic-SWE is the **Agent Skill Registry** $\mathcal{S}$. A skill is a structured document with four fields: **name**, **natural-language description**, **applicability conditions**, and **ordered list of operations**. Skills are therefore not free-form summaries but reusable abstractions that can be retrieved and used programmatically during subsequent task generation [2606.07412].

The distillation pipeline has three stages. First, the current policy is run on a seed task set to produce a trace corpus,
$$
\mathcal{T} = \{\tau_1,\ldots,\tau_N\},
$$
which is partitioned into successful traces $\mathcal{T}^+$, where $r(\tau)=1$, and failed traces $\mathcal{T}^-$, where $r(\tau)=0$. The paper explicitly retains both: success traces expose reusable repair strategies, while failure traces reveal capability gaps and corrective lessons. Second, a distillation model extracts candidate skills,
$$
\hat{\mathcal{S}} = \mathcal{M}_{\mathrm{distill}}(\mathcal{T}^+, \mathcal{T}^-).
$$
Third, the candidate set is deduplicated by semantic similarity and filtered by trace coverage,
$$
\mathcal{S} = \mathrm{Dedup}(\hat{\mathcal{S}}, \delta_{\mathrm{sim}}).
$$

The resulting skills capture search-and-localize strategies, debugging patterns, repository-specific semantic conventions, common repair failure modes, and corrective heuristics. The appendix example described for OAuthLib is notably concrete, including scope-conversion contracts, constructor storage pitfalls, OAuth1 parameter confusion, and OIDC inheritance issues. This specificity is important: the registry is intended to encode trace-grounded SWE behavior rather than generic “coding ability.”

## 4. Repository-grounded task generation, validation, and rewards

Generated tasks are admitted only after a staged validation pipeline executed in a repository sandbox. The validation stages are **Format**, **Grounding**, **Execution**, and **Semantics**: the task and verifier must be well-formed and parseable; they must reference real repository artifacts; the verifier must run without infrastructure errors and remain stable across repeated runs; and the verifier must separate failing from repaired states with at least one valid repair. The validation indicator is written as
$$
\mathcal{V}_{\mathrm{alid}}(\tau, v, r)=\prod_{l=1}^{4} f_l(\tau,v,r)\in\{0,1\}.
$$
Only tasks that satisfy all stages are added to the curriculum,
$$
\mathcal{D}_{t+1} = \mathcal{D}_t \cup \{(r,\tau',v')\mid \mathcal{V}_{\mathrm{alid}}(\tau',v',r)=1\}.
$$
This validation regime enforces that generated tasks are executable, reproducible, solvable, and semantically meaningful [2606.07412].

The Solver reward is decomposed into full-suite success, partial repair, and regression avoidance:
$$
r_S = \lambda_1\, \mathbf{1}\!\left[\mathcal{F}_{\checkmark} = \mathcal{F} \wedge \mathcal{P}_{\checkmark} = \mathcal{P}\right] + \lambda_2\, \frac{|\mathcal{F}_{\checkmark}|}{|\mathcal{F}|} + \lambda_3\, \frac{|\mathcal{P}_{\checkmark}|}{|\mathcal{P}|}.
$$
Here $\mathcal{F}$ denotes originally failing tests, $\mathcal{P}$ originally passing tests, $\mathcal{F}_{\checkmark}$ failing tests that now pass, and $\mathcal{P}_{\checkmark}$ passing tests that remain passing. Training uses GDPO, with each reward component normalized separately before aggregation.

The Generator is trained with a solver-gradient alignment reward. A held-out validation set $\mathcal{V}_{\text{val}}$ defines a target gradient direction $G_v$ through per-task Solver gradient estimates,
$$
g_j^v = \frac{1}{K}\sum_{k=1}^{K} \hat{A}_{j,k}\, \nabla_\theta \log \pi_\theta(\hat{y}_{j,k} \mid \tau_j^v, r_j^v, v_j^v),
$$
and
$$
G_v = \frac{1}{|\mathcal{V}_{\text{val}}|}\sum_j g_j^v.
$$
For a candidate generated task with Solver gradient $g_\tau$, the Generator reward is
$$
R_G(\tau, v, r) = \mathcal{V}_{\mathrm{alid}}(\tau, v, r)\cdot \cos(g_\tau, G_v).
$$
The paper’s first-order justification is
$$
\Delta J_{\mathrm{val}} \approx \eta \langle g_\tau, G_v \rangle = \eta \|g_\tau\|\,\|G_v\|\,\cos(g_\tau,G_v).
$$
Cosine similarity is preferred to the unnormalized inner product because large gradient magnitude can be induced by long or complex tasks that are not necessarily useful.

## 5. Experimental setup, benchmarks, and results

The experimental configuration uses **Qwen3.5-9B** as the base model for both Generator and Solver, **Qwen3.6-27B** as the skill extractor, **GDPO** for Solver training, **GRPO** for Generator training, **mini-swe-agent** as the harness for SWE benchmarks, and **little-coder** as the harness for Terminal-Bench 2.0. Training runs for **3 iterations** with **12k validated instances per iteration**, for **36k total**, and uses a held-out subset of **BeyondSWE**, size **100**, for gradient alignment. Evaluation is by pass rate on **SWE-bench Verified**, **SWE-bench Lite**, **SWE-bench Pro Public**, and **Terminal-Bench 2.0** [2606.07412].

| Benchmark | Pass rate after 3 iterations |
|---|---:|
| SWE-bench Verified | 50.40% |
| SWE-bench Lite | 36.67% |
| SWE-bench Pro | 22.85% |
| Terminal-Bench 2.0 | 14.61% |
| Overall mean | 31.13% |

Relative to the base agent, these correspond to gains of **+7.80** on Verified, **+7.00** on Lite, **+5.61** on Pro, **+4.50** on Terminal-Bench 2.0, and **+6.22** overall. Among baselines, **R-Zero** improves briefly and then regresses below the base agent; **SPIRAL** and **Absolute-Zero** improve early and then saturate or regress; **Socratic-Zero** is stronger than many baselines but peaks early and then declines slightly; and **SSR** is the strongest baseline, reaching **47.00%** on Verified after 3 iterations but still trailing Socratic-SWE by **3.40 points**. On **Terminal-Bench 2.0**, only SSR and Socratic-SWE show meaningful gains, with Socratic-SWE performing better, which the paper interprets as evidence that trace-derived skills capture general agent behaviors such as file manipulation, command chaining, and environment navigation [2606.07412].

The scaling experiment on SWE-bench Verified extends training to **5 iterations**. Socratic-SWE continues improving to **52.00%**, whereas SSR plateaus earlier at **48.00%**. The ablation study shows that removing the **Skill Registry** causes the largest drop, removing **trace distillation** also hurts, and replacing **GDPO** with **GRPO** degrades performance. The paper presents these ablations as support for the claim that the effective unit is not trace reuse alone, but the combination of trace-derived skills, grounded task generation, and decomposed reward learning.

## 6. Relation to the broader “Socratic” literature

The term “Socratic” has been used across several distinct LLM research programs. In prompt engineering, “Prompting Large Language Models With the Socratic Method” identifies methods such as **definition**, **elenchus**, **dialectic**, **maieutics**, **generalization**, and **counterfactual reasoning**, and proposes the CRIT workflow for stepwise inspection and verification [2303.08769]. In reasoning evaluation, SocREval uses **Definition**, **Maieutics**, and **Dialectic** to improve GPT-4 as a reference-free evaluator [2310.00074]. In education-oriented systems, SocratiCode shifts an adaptive programming tutor toward “guided questioning, reflection prompts, misconception checks, incremental hints, and mandatory pauses for learner input” for K-12 Python learning [2605.17857]; SocraticLLM decomposes mathematics tutoring into **Review**, **Guidance / heuristic**, **Rectification**, and **Summarization** [2407.17349]; and TreeInstruct applies multi-turn, hierarchical questioning to code debugging, estimating a student’s knowledge state and asking sibling or child questions depending on whether the student’s response is incorrect or partially correct [2406.11709].

Against that background, Socratic-SWE occupies a different level of the stack. It is not primarily a learner-facing tutoring interface, a teacher-training simulator, or an LLM-as-judge. Its “Socratic” mechanism lies in self-evolution through structured reflection on traces: the agent’s prior successes and failures are distilled into skills, those skills generate new tasks, and the resulting tasks induce new traces. A plausible implication is that the broader Socratic literature now spans at least three layers: user-facing dialogue and scaffolding, evaluator-side reasoning inspection, and training-time curriculum evolution for agents. Socratic-SWE is characteristic of the third layer because it operationalizes reflection not as questioning in conversation, but as trace-derived curriculum design.

A frequent misconception is therefore to treat Socratic-SWE as a debugging tutor for human programmers. That description fits TreeInstruct more closely than it fits Socratic-SWE. Socratic-SWE is instead a framework for improving SWE agents by mining their own histories.

## 7. Limitations, constraints, and future directions

The paper is explicit about several constraints. First, the framework operates in a **closed-world repository pool**; as the skill registry becomes more complete, later iterations discover fewer new gaps, producing redundancy and eventual saturation. Second, the gradient-alignment reward depends on a held-out trusted validation set. If that validation set is unrepresentative, curriculum quality may degrade. Third, the method assumes **executable verification and sandboxing**: tasks must be testable under deterministic execution. Fourth, evaluation is limited to four benchmarks, so broader transfer across languages, repository styles, and workflows remains unproven [2606.07412].

The forward-looking directions named in the paper are **dynamic repository augmentation**, **cross-repository skill transfer**, **online skill discovery**, and more open-ended self-evolution beyond a fixed repository corpus. More broadly, Socratic-SWE reframes training data generation for SWE agents. Instead of treating task synthesis as an external synthetic-data problem, it makes task synthesis endogenous to learning and grounds it in the agent’s own traces, structured skills, executable validation, and validation-aligned gradients. This suggests a shift in how SWE-agent improvement is conceptualized: not as scaling a static task pool, but as iteratively converting behavior into curriculum.

Source: https://www.emergentmind.com/topics/socratic-swe