SWE-Future: Forecast-Conditioned Data Synthesis for Future-Oriented Software Engineering Agents

Abstract: Realistic coding-agent benchmarks often replay public GitHub issues and pull requests, making them vulnerable to overlap with model pretraining, fine-tuning, synthetic-data generation, or benchmark-driven model selection. Fully synthetic tasks avoid direct historical replay, but can drift away from real repository needs. We propose SWE-Future, a forecast-conditioned data synthesis method for future-oriented coding tasks. Given a forecast snapshot at time $T_0$, the method uses only pre-$T_0$ repository evidence to forecast future feature implementation/enhancement, bugfix, and refactor task families. We first validate this forecasting step retrospectively: after forecasts are fixed, later pull requests are used only to measure whether the predicted task families match future repository work. In an 80-repository study, the forecaster achieves 58.1\% future-work relevance under the main semantic matching metric. We then use validated forecast families as conditioning signals to synthesize a 200-task coding-agent dataset across 61 repositories from a task-generation snapshot, rather than replaying the later pull requests used for validation. SWE-Future shows that repository-evolution forecasts can guide realistic, future-oriented coding-task synthesis while reducing direct dependence on historical pull-request replay.

• The paper introduces forecast-conditioned data synthesis to generate future-oriented coding-agent tasks using repository evolution signals.
• It employs a three-phase methodology—forecasting, retrospective validation, and task synthesis—to ensure dataset relevance and executable validation.
• Experimental results on 80 repositories show strong match rates, particularly in bugfix tasks, reducing overfitting and contamination risks.

Forecast-Conditioned Data Synthesis for Future-Oriented Coding-Agent Evaluation

Background and Motivation

Rapid advancements in coding-agent benchmarks and datasets for software engineering have revealed substantial challenges in dataset construction. Historical issue and pull request (PR) replay, as seen in benchmarks such as SWE-Bench, achieves realism but creates contamination risk for model development, owing to overlap in pretraining, fine-tuning, and benchmarking data selection. Synthetic task generation circumvents direct replay but often loses repository-specific relevance, leading to incongruities between synthetic tasks and the genuine pressures faced by maintainers. SWE-FUTURE introduces forecast-conditioned data synthesis, aiming to bridge this gap by leveraging repository evolution forecasting as a conditioning signal for realistic, future-oriented task construction.

Methodological Framework

SWE-FUTURE organizes task synthesis around three distinct temporal boundaries: a forecast snapshot ($T_0$), a retrospective validation window ($(T_0, T_1]$), and a task-generation snapshot ($T_{gen}$). The evidence bundle is constructed exclusively from repository signals visible at or before $T_0$—issues, PRs, labels, and metadata—eschewing any reliance on post-$T_0$ historical artifacts during task generation. Forecasting is performed at the family level, concentrating on feature implementation/enhancement, bugfix, and refactor categories. The cluster-to-family method identifies repository-specific anchors via repeated signals and constructs coarse future-work family specifications, rather than precise PRs.

Retrospective validation is performed by matching forecast families against post-$T_0$ PR metadata using a semantic judge, which labels matches as strong, related, or rejected. Crucially, post-$T_0$ PRs are never used as task sources. Only families validated as strong or related matches condition downstream task synthesis. All synthetic tasks are generated from the $T_{gen}$ repository snapshot and grounded in forecasted directions, with strict gate-based admission restrictively enforcing executable validation.

Experimental Outcomes

The retrospective validation pool comprises 80 repositories, spanning dominant Python projects and software engineering ecosystem domains. The forecaster emits 260 families across 76 repositories. Of these, 58.1% are validated as strong or related matches under the primary semantic metric, with bugfix families exhibiting higher validation rates (64.0%) compared to feature implementation/enhancement (48.4%). The dataset construction grounded in 151 strong or related families yields 200 executable tasks across 61 repositories. Each task undergoes a multi-agent construction pipeline ensuring separation of context selection, task writing, oracle design, patch construction, and verification, thereby mitigating leakage and disguised historical replay.

Executable validation involves constructing a task-specific test patch and a gold patch from $T_{gen}$ rather than historical PRs, with the test patch serving as a FAIL_TO_PASS target and the gold patch demonstrating maintainability and solution viability. Preservation gates leveraging repository-native tests further enforce behavioral correctness.

Implications of SWE-FUTURE

SWE-FUTURE demonstrates that repository evolution forecasting, when retrospectively validated, serves as a robust conditioning mechanism for task synthesis, thereby offering contamination-aware, future-facing coding-agent datasets. The 58.1% future-work relevance metric corroborates the efficacy of forecast family identification, with bugfix directions proving more amenable to prediction than feature implementation/enhancement. The task synthesis workflow, which grounds forecast families in repository state and enforces gate-based executable validation, sets a new standard for reducing direct replay and benchmark contamination risk in dataset construction.

Practically, SWE-FUTURE equips the community with datasets that challenge coding agents on repository-natural demands reflective of anticipated evolution, supporting more temporally valid agent evaluation. Theoretically, it advances the methodology of snapshot-based forecasting and semantic family matching in the context of software engineering evolution.

Limitations and Prospective Directions

Forecast fidelity remains constrained by noisy repository signals, anchor selection ambiguity, and label tracker artifacts. Test patch quality may lag behind maintainer-written tests, especially in the refactor domain where behavioral preservation is subtle. SWE-FUTURE’s validation pipeline surfaces the need for more granular retrieval algorithms, enhanced anchor de-noising, and richer test synthesis aligned with project conventions.

Future work should focus on optimizing semantic cluster formation, improving family-to-task mapping precision, and developing automated test-generation strategies that yield patches indistinguishable from those produced by experienced maintainers. The forecasting regime could be further augmented by incorporating dynamic code analysis, cross-repository evolution patterns, and contextual maintainability signals.

Conclusion

SWE-FUTURE formalizes forecast-conditioned data synthesis as a principled route for developing coding-agent benchmarks that balance realism, repository specificity, and contamination control. By freezing repository history, forecasting coarse future demand families, grounding tasks in current repository state, and enforcing strict executable validation, it robustly reduces overfitting and historical replay risk. Continued refinement of forecasting and validation methodologies will further enhance the reliability and utility of future-oriented software engineering evaluation datasets.