TeleSWEBench: A Commit-Driven Benchmark for Evaluating LLM-Powered Software Engineering in Telecommunications

Abstract: With the telecommunications field embracing zero touch management alongside novel O-RAN and AI-RAN frameworks, contemporary telecom networks now function as immensely intricate and heavily softwareized codebases. While automated software engineering (ASE) tools and Software Engineering (SWE) Agents hold the potential to alleviate the critical code generation bottleneck in this domain, their ability to navigate and modify specialized, mathematically rigorous wireless stacks like srsRAN 5G remains unverified. General-purpose coding benchmarks fail to capture the stateful logic and strict requirements of telecommunications, leaving a critical evaluation gap. In this paper, we introduce TeleSWEBench, the first commit-driven benchmark specifically designed to measure an agent's performance in the telecom domain. We mine real developer commits from the srsRAN 5G repository and distill them into structured test cases across three difficulty tiers (Easy, Medium, and Difficult). Our benchmark consists of 734 questions that are accompanied by executable unit tests. To avoid the rigidity of test cases, we further propose a hierarchical LLM as a Judge framework called TeleJudge that scores agent outputs at the file level and aggregates verdicts holistically. This follows an evaluation based on context and semantic similarity in parallel to a standard unit test-based evaluation. Using this benchmark, we evaluate AIDER, OpenHands, and the ClaudeCode frameworks, powered by state-of-the-art reasoning LLMs, including Qwen3, GPT OSS, Gemma 4, Kimi, and Qwencoder 2.5. Our two-stage evaluation reveals that models suffer from a lack of both localization accuracy and functional correctness, with the strongest ASE tools achieving up to 25% of shippable changes.

• The paper introduces TeleSWEBench as a commit-driven benchmark tailored for evaluating LLM-based ASE agents in complex telecom codebases.
• It details a hierarchical task generation and evaluation pipeline with scaled and verified splits that address multi-file, cross-layer challenges.
• The results reveal that advanced models achieve up to 25% shippable patches, highlighting limitations in context handling and semantic coherence.

TeleSWEBench: Commit-Driven Benchmarking of LLM-Based Software Engineering in Telecommunications

Motivation and Domain Challenges

The telecommunications domain is rapidly shifting toward fully autonomous, software-defined architectures, exemplified by O-RAN and AI-RAN paradigms. Modern telecom stacks such as srsRAN 5G represent highly complex, deeply interdependent C++ codebases governed by rigorous real-time, asynchronous, and standardization constraints. The primary bottleneck in the progress of next-generation wireless networks is not hardware, but automated code generation and maintenance. Despite advances from general-purpose Automated Software Engineering (ASE) frameworks and LLM-powered Software Engineering (SWE) agents, existing benchmarks (e.g., SWE-Bench, HumanEval, MBPP) fundamentally fail to reflect the multi-file, cross-layer semantic complexity and hardware-adjacency of telecom repositories.

TeleSWEBench is introduced as the first commit-driven benchmark tailored specifically to evaluate LLM-based ASE agents in real-world telecom environments. By mining authentic developer commits from the srsRAN 5G repository, TeleSWEBench distills genuine bug fixes, feature additions, and protocol updates into 734 structured test cases categorized by three difficulty tiers. Its design rigorously stresses repository-scale reasoning, cascading dependencies, and strict conformance to domain-specific requirements.

Figure 1: Task count comparison across SWE benchmarks; TeleSWEBench’s coverage contextualized against SWE-Bench, HumanEval, and MBPP.

Benchmark Creation and Evaluation Pipeline

TeleSWEBench leverages srsRAN's extensive commit history, characterized by an average of 5.57 files changed per commit, demonstrating the necessity of synchronized, cross-layer modifications. Benchmark item generation follows a hierarchical cascade: hard tasks generated first (high-level objectives with minimal hints), medium tasks require structural understanding and selective file naming, and easy tasks specify exact paths and line-level edit instructions. A validator LLM ensures clarity, difficulty-appropriate density, and faithfulness to the original patch, retaining only high-confidence cases.

Figure 2: TeleSWEBench pipeline: Commit mining → task generation/validation → ASE inference → holistic evaluation.

Two evaluation splits are defined:

• Scaled: 1,500 questions evaluated via hierarchical LLM judge for statistical assessment.
• Verified: 140 questions accompanied by executable unit tests for gold-standard correctness.

To address the limitations of rigid unit tests and context overload in large-scale patch evaluation, TeleSWEBench introduces TeleJudge—a hierarchical LLM-as-a-Judge system that reasons both at file and holistic patch levels. This framework permits semantic evaluation in parallel to traditional unit testing, critical for cross-file and structurally creative solutions.

ASE Frameworks and Model Suite

Evaluation is conducted using three open-source CLI-powered ASE frameworks: AIDER, OpenHands, and ClaudeCode. Backbones range from small-scale to trillion-parameter LLMs (Qwen3, Qwen3.5, Gemma4, Kimi-K2.5, GPT-OSS, QwenCoder-2.5). All inference is strictly time-bound per task (e.g., 300s maximum) to reflect practical agent utility. Models run from 1.5B up to 1T parameters and context windows up to 1M tokens, allowing systematic analysis of parameter scaling and context utilization effects.

Localization, Failure Modes, and Analysis

Stage 1 localization assesses the agent's ability to identify which files require modification, decoupling architectural navigation from code generation. Metrics include Exact Match (EM), Partial Match (PM), Over Addressed (OA), No Match (NM), and No Changes (NC).

Performance degrades with increasing task difficulty. Decisive models (e.g., QwenCoder-2.5) localize well in easy tasks but fail with high context or ambiguous instructions, suffering from context exhaustion or timid behavior. Larger, highly-aligned models exhibit extreme caution, frequently abstaining from any modification (high NC rates), reflecting over-alignment to safety objectives.

Figure 3: Distribution of three principal failure modes across No Changes outcomes: pseudocode suggestion, verbosity/counter-queries, and context exhaustion.

Stage 2: Functional Correctness and End-to-End Deployability

Functional correctness is assessed only for exact-match outputs. Three metrics capture quality:

• Unit Test Acceptance Rate (UAR): Patch passes repository-native unit/integration tests.
• TeleJudge Acceptance Rate (TAR): Patch accepted by hierarchical semantic judge.
• Ship-Ready Percentage (SRP): Patch passes both UAR and TAR.

Results indicate persistent divergence between executable correctness and semantic coherence. For instance, GLM-4.7 achieves highest SRP at 25%, while QwenCoder-2.5 exhibits high UAR (57.73%) but zero TAR, revealing a pattern of brittle, narrow solutions that exploit unit test specificity but violate domain conventions.

Figure 4: Model scatter plot: UAR vs. TAR, annotated with SRP; illustrating the gap between semantic and test-driven correctness.

OpenHands and ClaudeCode further demonstrate that agent architecture impacts efficacy, with OpenHands yielding higher SRP than AIDER, but the bottleneck remains primarily at the backbone model's understanding of telecom logic.

Figure 5: Comparative performance between OpenHands and ClaudeCode frameworks using the qwen3 backbone; demonstrating framework-driven variance in localization and correctness.

Implications, Limitations, and Future Directions

TeleSWEBench provides a systematic foundation for evaluating, analyzing, and improving LLM-based SWE agents in telecom—a domain typified by extreme codebase complexity, intricate dependencies, and strict functional constraints. The results highlight strong claims: even the best current open-source frameworks achieve only 25% shippable patch rates, with model failures dominated by context exhaustion, timidity, or non-executable outputs.

Practically, this exposes critical deficiencies in existing models, revealing the necessity for domain-specific instruction fine-tuning, agent retraining, and context-aware hierarchical evaluation tools. Theoretically, it underscores the limits of general-purpose coding benchmarks for specialized software stacks and motivates deeper work in repository-scale reasoning, memory management, and LLM-as-a-Judge protocols. Immediate research agenda includes expanding TeleSWEBench to additional repositories, importing commercial agent baselines, and fully automating multi-agent cooperative patch refinement.

Conclusion

TeleSWEBench sets a rigorous and quantitative baseline for LLM-powered ASE agents in telecommunications, demonstrating persistent incapacity of current tools to address complex, multi-file, protocol-driven code changes required for real-world deployments. Its hierarchical evaluation and commit-driven structure not only highlight structural and functional correctness gaps but also provide actionable metrics to guide future model development and practical agent adoption in mission-critical telecom environments.