SERA: Soft-Verified Efficient Repository Agents

Abstract: Open-weight coding agents should hold a fundamental advantage over closed-source systems: they can be specialized to private codebases, encoding repository-specific information directly in their weights. Yet the cost and complexity of training has kept this advantage theoretical. We show it is now practical. We present Soft-Verified Efficient Repository Agents (SERA), an efficient method for training coding agents that enables the rapid and cheap creation of agents specialized to private codebases. Using only supervised finetuning (SFT), SERA achieves state-of-the-art results among fully open-source (open data, method, code) models while matching the performance of frontier open-weight models like Devstral-Small-2. Creating SERA models is 26x cheaper than reinforcement learning and 57x cheaper than previous synthetic data methods to reach equivalent performance. Our method, Soft Verified Generation (SVG), generates thousands of trajectories from a single code repository. Combined with cost-efficiency, this enables specialization to private codebases. Beyond repository specialization, we apply SVG to a larger corpus of codebases, generating over 200,000 synthetic trajectories. We use this dataset to provide detailed analysis of scaling laws, ablations, and confounding factors for training coding agents. Overall, we believe our work will greatly accelerate research on open coding agents and showcase the advantage of open-source models that can specialize to private codebases. We release SERA as the first model in Ai2's Open Coding Agents series, along with all our code, data, and Claude Code integration to support the research community.

• The paper introduces a soft-verified training paradigm that achieves comparable performance to hard-verified methods while reducing costs by up to 57x.
• It leverages a two-rollout Soft Verified Generation process to synthesize high-quality training data from repository code changes without relying on expensive unit tests.
• Extensive scaling laws and ablation studies demonstrate that repository specialization significantly improves sample efficiency and practical agent deployment.

SERA: Soft-Verified Efficient Repository Agents – An Expert Analysis

Introduction and Motivation

The paper "SERA: Soft-Verified Efficient Repository Agents" (2601.20789) addresses the persistent bottleneck in LLM-based coding agents: the high cost and infrastructure complexity of open-weight model specialization to private repositories. The core claim is that, by systematically stripping away components in existing agent training pipelines (notably unit-test-based hard verification and bug injection), the authors demonstrate that cheaper, simpler supervised finetuning with “soft verification” enables practical, high-performance, repository-specialized agents—substantially lowering compute and engineering requirements relative to prior RL and synthetic data pipelines.

SERA, a 32B coding agent, achieves state-of-the-art results among fully open-source models on SWE-bench Verified, matching or exceeding open-weight baselines such as Devstral-Small-2 at a small fraction of the previous cost. The work substantiates its claims with rigorous scaling laws, cross-seed robustness, extensive ablation, and repository specialization studies.

The Soft Verified Generation (SVG) Pipeline

SERA’s data generation (“SVG”) comprises a two-rollout process:

1. First Rollout: A teacher model is prompted with a vague bug-type and a randomly selected function in the codebase, producing a change (“trajectory” and patch), which is used to synthesize a PR.
2. Second Rollout: The teacher is tasked to reproduce the change using only the generated PR description, producing a second trajectory and patch.
3. Soft Verification: The two patches are compared via line-level recall (no unit test execution), and samples are selected above a recall threshold.

   Figure 1: Diagrammatic overview of the Soft Verified Generation two-rollout protocol and line-level recall-based soft verification.

This soft verification methodology decouples data quality check from expensive or fragile test suites, making synthetic data generation applicable to arbitrary repositories. The diversity emergent from vague instruction prompts is also highly emphasized: these instructions yield a significant proportion of non-bug-related changes (e.g., refactoring, doc improvements), broadening the skill coverage of the training data.

Efficiency Analysis: Scaling Laws and Cost Reductions

The empirical evaluation applies thorough cost and sample efficiency analysis comparing SERA against reinforcement learning-based systems (e.g., SkyRL, DeepSWE) and synthetic data approaches (notably SWE-smith, BugPilot).

Figure 2: [Left] Cost/performance scaling of various approaches under self-hosted vLLM inference. [Right] Specialization sample-efficiency scaling for Django.

Figure 3: Cost to match SoTA (Devstral-Small-2, GLM-4.5-Air) via API and vLLM cost models. Horizontal baselines denote predicted cost thresholds for parity.

The main results:

• SERA is 26x cheaper than RL and 57x cheaper than SWE-smith (self-hosted) in reaching equivalent SWE-bench Verified performance at 32K context.
• At API pricing regimes, SERA can be 53–115x more cost efficient for a given performance threshold.
• Repository specialization to parity with (teacher) GLM-4.5-Air requires only 8k samples ($\sim$\$1300), a 3.5x sample efficiency improvement over training on general data.

Extensive scaling laws, fit with high predictive R², provide strong evidence that SERA maintains predictable, monotonic improvement with increased compute/data, and is not reliant on idiosyncratic sample selection or chance.

Repository Specialization

A significant practical contribution is demonstrating the feasibility and efficiency of specializing open-weight models to specific codebases. The specialization studies use Django, Sympy, and Sphinx repositories from SWE-bench Verified.

Figure 4: Specialization scaling law on Django; $\alpha$ denotes fraction of Django-specific data in the training mixture.

Findings:

• With full specialization ($\alpha = 1.0$), the student matches teacher performance at approximately 8k samples; general data ($\alpha = 0.0$) requires 25k samples for equivalent performance, confirming strong returns to repo specificity.
• Specialization consistently enables the fine-tuned student to match or slightly surpass the teacher LLM, which lacks static repo-specific knowledge in its weights (being limited to what fits in context).
• This advantage is unattainable for API-locked closed models, and further underscores the utility of open-weight solutions for organizations with private or proprietary repositories.

Ablation Studies: Verification, Truncation, and Data Curation

The authors conduct systematic ablations on several axes:

Verification Stringency

Figure 5: Scaling curves for various soft/hard verification thresholds; no significant differential performance—strict verification does not improve results.

Contrary to standard pipeline assumptions, training on unverified or softly verified data achieves identical downstream performance to hard verification (unit tests, $r=1$). This result is consistent across scales and sample sizes up to the studied regime. It echoes similar insights from recent reasoning task literature, indicating that the learning signal required for agent improvements lies in skill coverage present even in data not strictly passing test-based checks.

Truncation and Trajectory Curation

Figure 6: Performance versus truncation ratio of agent rollouts; high-ratio (i.e., minimally truncated) trajectories perform best.

The effect of slicing trajectories to fit model context windows is shown to be nontrivial: picking high-ratio (i.e., minimally truncated) segments is superior to random selection, refuting prior practice of naive arbitrary slicing in SFT pipelines.

Data Filtering

Filtering of extremely large edits or tool call outputs is beneficial but codebase-specific; no universal filtering heuristic was identified that improved all cases, indicating the importance of repository-adaptive curation strategies.

Robustness, Statistical Analysis, and Best Practices

The authors stress the high-variance, low-SNR (signal-to-noise ratio) regime in coding agent evaluation (per-run std devs $1-3\%$; marginal improvements commensurate with this noise). As a consequence, the practice of single-seed reporting is shown to be unreliable. The authors recommend at least 3 seeds (6–15 for effects $<1–2\%$), careful SNR benchmarking, and routine use of scaling laws for robust extrapolation and ablation.

Theoretical and Practical Implications

On a theoretical level, this work reinforces a growing perspective in data-centric AI that verification stringency and “perfectly labeled” data may not be necessary, or even optimal, especially in settings with high data diversity, long-horizon tasks, and agent-environment interactions. The result that suboptimal and noisy data can offer sufficient and equally performant training signal invites reevaluation of costly data generation and curation strategies, especially in LLM agent workflows.

Practically, SERA provides:

• A cost-effective pipeline for synthesizing high-quality training data without reliance on test infrastructure.
• A foundation for rapid, low-resource specialization to arbitrary or private codebases—closing a critical real-world gap for smaller organizations and open-source development.
• An open, reproducible agent baseline accompanied by thorough ablation, scaling, and deployment documentation.

The “soft verification” paradigm is likely to influence other agent domains where rigid correctness verification is infeasible or resource-prohibitive.

Future Directions

Open questions spurred by this research include:

1. Limits of Soft Verification: Whether equivalence between soft/hard/unverified data holds at larger scales and higher model capacities, or if eventually, correctness-based verification becomes necessary to further advance SoTA frontiers.
2. Transferability to Other Programming Languages/Benchmarks: The approach is validated on SWE-bench (Python); its utility across other real-world settings, including low-resource languages or radically different software stacks, is not tested here.
3. Systematic Exploration of Curation and Specialization Strategies: As filtering effectiveness is codebase dependent, algorithmic or automated approaches to selecting optimal curation policies remain an open problem.
4. Robustness across Model Backbones: This work utilizes Qwen-3 as the base and GLM as the teacher; generalization across families (Llama, Mistral, etc.) is not exhaustively established, and may uncover further nuances.

Conclusion

SERA's pipeline and empirical analysis concretely demonstrate that repository-specialized, high-quality coding agents can be trained within modest compute and resource constraints, using pragmatically “soft-verified” and diverse synthetic data. This work marks a transition to agent training architectures that maximize the practical advantages of open-weight models, moving beyond the test-suite and unit-testing orthodoxy. Its release, code, and data notably enhance the reproducibility and accessibility of state-of-the-art coding agent research.

The main implications are both practical—enabling rapid, private, low-barrier agent development—and conceptual, challenging cost and complexity axioms long-held in LLM agent training methodology. The findings will undoubtedly inform the next generation of data-efficient, highly specialized, and reproducible agent pipelines in software engineering and adjacent domains.