TeleSWEBench: Telecom ASE Benchmark

• TeleSWEBench is a telecom-native benchmark that evaluates LLM-based automated software engineering by using real commits from the srsRAN 5G repository.
• It applies a two-stage evaluation pipeline that decouples localization from synthesis, using natural language tasks, unit tests, and hierarchical semantic judgment to validate patches.
• The benchmark targets telecom-specific challenges such as cross-file dependencies, asynchronous constraints, and strict protocol requirements to ensure domain-relevant performance.

Searching arXiv for TeleSWEBench and closely related benchmarking papers. TeleSWEBench is a commit-driven benchmark for evaluating LLM-powered software engineering in telecommunications, introduced to measure whether agentic ASE systems can operate effectively on a specialized wireless codebase rather than on generic software tasks (Gajjar et al., 3 Jun 2026). It is built from real developer commits mined from the srsRAN 5G repository and formulates those commits as natural-language tasks with executable tests, while also supplying a parallel semantic evaluation path through a hierarchical LLM-as-a-Judge framework called TeleJudge (Gajjar et al., 3 Jun 2026). In the broader benchmarking landscape, TeleSWEBench is telecom-native rather than general-purpose: it targets a softwareized 5G stack with asynchronous timing constraints, layered protocol logic, cross-file dependencies, and strict 3GPP-oriented requirements that are not well represented in standard software engineering benchmarks (Gajjar et al., 3 Jun 2026). A related executable benchmarking suite provides a telemetry, replay, provenance, and evidence-admission substrate for SWE-bench-compatible evaluation across code, web, and micro-task settings; its description explicitly presents that substrate as suitable for operationalizing a TeleSWEBench-style evaluation when the emphasis is auditable execution and admitted evidence rather than a solver leaderboard (Zhong et al., 10 May 2026).

1. Domain motivation and benchmark rationale

TeleSWEBench is motivated by the increasing softwareization of telecommunications systems under zero-touch management, O-RAN, and AI-RAN, where network functions become cloud-native software components deployed on commodity hardware rather than proprietary appliances (Gajjar et al., 3 Jun 2026). In this setting, modern wireless networks constitute large, tightly coupled codebases spanning PHY through core, and the principal bottleneck shifts from hardware to software engineering and code generation (Gajjar et al., 3 Jun 2026).

The benchmark is designed around the observation that existing coding benchmarks do not capture the properties of telecom software stacks. General-purpose benchmarks such as HumanEval, MBPP, SWE-bench, and related variants primarily emphasize Python or Java ecosystems and generic library or web tasks, whereas telecom software involves hardware-adjacent C++ idioms, stateful protocol logic, and strict layered semantics (Gajjar et al., 3 Jun 2026). Telecom-specific evaluations prior to TeleSWEBench had mainly focused on multiple-choice question answering over standards or on static repository understanding, rather than end-to-end code modification within a real stack (Gajjar et al., 3 Jun 2026).

This positioning distinguishes TeleSWEBench from both generic software engineering benchmarks and non-executable telecom evaluations. The benchmark is therefore intended to test whether LLM-based ASE tools can localize, modify, and functionally repair or extend code within srsRAN 5G, a repository characterized by cross-file coupling and mathematically rigorous protocol behavior (Gajjar et al., 3 Jun 2026). A plausible implication is that TeleSWEBench is aimed less at broad coding aptitude and more at domain-grounded software engineering competence under telecom-specific operational constraints.

2. Construction from srsRAN 5G commits

TeleSWEBench is derived from real developer commits in the srsRAN 5G repository, licensed under AGPL-3.0, with the source history described as containing 15k+ commits from 2023–2025 (Gajjar et al., 3 Jun 2026). The average commit touches 5.57 files, which is used as evidence of the cross-file and tightly coupled nature of the target domain (Gajjar et al., 3 Jun 2026). Tasks are mined from bug fixes, feature additions, and protocol updates (Gajjar et al., 3 Jun 2026).

Task generation follows a commit-driven pipeline. For each candidate commit, tasks are generated and validated through an LLM cascade, proceeding from smaller to larger models with early stopping once a valid instance is produced (Gajjar et al., 3 Jun 2026). A second heuristic attempts difficulty levels in the order Difficult $\rightarrow$ Medium $\rightarrow$ Easy, also with early stopping: if Difficult fails, the commit is abandoned; if Difficult succeeds but Medium fails, the process does not proceed to Easy (Gajjar et al., 3 Jun 2026). A separate validator LLM checks clarity, difficulty-appropriate information density, IDE realism, faithfulness to the original commit diff, and internal consistency, and only items marked valid with confidence at least $T=0.9$ are retained (Gajjar et al., 3 Jun 2026).

The resulting benchmark contains 734 verified tasks with executable tests (Gajjar et al., 3 Jun 2026). The difficulty distribution is 313 Difficult, 279 Medium, and 142 Easy tasks (Gajjar et al., 3 Jun 2026). File scope is broad: Easy tasks have mean 29.82 files and median 7, Medium tasks have mean 19.57 and median 4, Difficult tasks have mean 18.42 and median 5, and across all questions the scope ranges from 1 to 300 files (Gajjar et al., 3 Jun 2026).

Quantity	Value
Total verified tasks with executable tests	734
Difficult tasks	313
Medium tasks	279
Easy tasks	142
Scope across all questions	1 to 300 files
Average files touched per source commit	5.57

These statistics support the benchmark’s claim to repository-scale rather than function-local evaluation. The means and medians indicate that even easier tasks may involve wide file exposure, while the upper bound of 300 files underscores the localization burden imposed by telecom codebases (Gajjar et al., 3 Jun 2026).

3. Task formulation and telecom-specific difficulty structure

Each TeleSWEBench task is a natural-language instruction derived from an actual commit context (Gajjar et al., 3 Jun 2026). The benchmark enforces a strict difficulty taxonomy that is defined by how much localization information is disclosed.

Easy tasks are near-mechanical edits that include exact file paths and line references, exact removals and insertions, and full self-containment so that a search-and-replace-style procedure is sufficient (Gajjar et al., 3 Jun 2026). Medium tasks explain the “what” and “why,” mention affected files or subsystems but not exact line numbers, and provide salient facts such as constants, function names, or version identifiers; these tasks require structural understanding in order to localize edits (Gajjar et al., 3 Jun 2026). Difficult tasks specify high-level objectives with minimal hints and must not name specific file paths or functions, thereby requiring autonomous localization and planning within the repository (Gajjar et al., 3 Jun 2026).

The benchmark provides the task description and uses the original commit to define a ground-truth target file set for localization evaluation (Gajjar et al., 3 Jun 2026). For Stage 2 functional verification, repository-native unit and integration tests are executed where available, and TeleJudge supplies an additional semantic verdict at file and patch level (Gajjar et al., 3 Jun 2026).

Representative task types include authentic bug fixes, feature additions, and protocol updates touching areas such as ASN.1 utilities, DCI packing, schedulers, and MAC/RLC logic (Gajjar et al., 3 Jun 2026). The examples described in the source material include refactors that replace enum-based VRB-to-PRB mapping with boolean flags across encoder and decoder logic, builders, and unit tests; configurable scheduler parameters; and ASN.1 octet string API changes (Gajjar et al., 3 Jun 2026). The benchmark is therefore not restricted to isolated syntax corrections; it targets semantically entangled modifications in C++ code under protocol and systems constraints.

4. Evaluation protocol: localization, execution, and semantic judgment

TeleSWEBench uses a two-stage evaluation pipeline that explicitly decouples localization from synthesis (Gajjar et al., 3 Jun 2026). Stage 1 asks whether the agent understood where changes should be made. With $T$ denoting the ground-truth target file set and $P$ the files modified by the agent, five localization outcomes are defined (Gajjar et al., 3 Jun 2026):

• Exact Match (EM): $P = T$
• Partial Match (PM): $P \subset T$
• Over Addressed (OA): $T \subset P$
• No Match (NM): $P \cap T = \varnothing$ and $P \neq \varnothing$
• No Changes (NC): $\rightarrow$0

Stage 2 measures functional correctness, but only for EM cases, specifically to separate code synthesis from localization error (Gajjar et al., 3 Jun 2026). It applies two parallel checks. The execution-driven path applies the patch, compiles srsRAN 5G, and runs repository-native unit and integration tests over the 734-task set (Gajjar et al., 3 Jun 2026). The semantic path runs TeleJudge, a hierarchical LLM-as-a-Judge framework that delivers a holistic pass/fail verdict after file-level analysis (Gajjar et al., 3 Jun 2026).

The execution harness mines tests from srsRAN’s native test hierarchy and pairs implementation files with tests through mirrored directory structure and filename patterns such as the _test suffix, together with thresholded similarity intended to reduce spurious matches (Gajjar et al., 3 Jun 2026). Because complete test vectors are consistently available at merge commits, the benchmark primarily leverages merges to ensure compilability and executability (Gajjar et al., 3 Jun 2026). Tasks are mapped to tests if any ground-truth modified file matches the implementation file associated with a test suite (Gajjar et al., 3 Jun 2026).

The formal Stage 2 metrics are defined over the EM subset. Let $\rightarrow$1 be the number of tasks and $\rightarrow$2 the number of Exact Match localizations. Let $\rightarrow$3 be the number of EM patches that compile and pass all associated tests; then

$\rightarrow$4

Let $\rightarrow$5 be the number of EM patches accepted by TeleJudge; then

$\rightarrow$6

Let $\rightarrow$7 count EM patches that pass both TeleJudge and tests; then

$\rightarrow$8

These definitions make SRP the benchmark’s strictest criterion and the closest proxy to “shippable changes” in the benchmark’s own terminology (Gajjar et al., 3 Jun 2026).

5. TeleJudge and hierarchical semantic evaluation

TeleJudge is a hierarchical LLM-as-a-Judge framework introduced to complement execution-only evaluation, which the benchmark characterizes as rigid because structurally different yet functionally valid solutions may be penalized by tests alone (Gajjar et al., 3 Jun 2026). TeleJudge therefore supplies a semantic, context-aware judging path parallel to unit testing (Gajjar et al., 3 Jun 2026).

Its first tier is the File-Level Judge. For each file touched either in the ground-truth patch or in the agent’s patch, the judge receives three inputs: the original task description, the ground-truth diff for that file, and the agent’s diff for the same file (Gajjar et al., 3 Jun 2026). It must accept if the modifications are functionally equivalent in the sense of addressing the same root cause or feature, even when stylistically different; otherwise it rejects (Gajjar et al., 3 Jun 2026). The output is JSON of the form: $T=0.9$8

The second tier is the Holistic Meta-Judge. It receives the task together with all per-file verdicts and reasons, and makes a final accept or reject decision for the entire patch (Gajjar et al., 3 Jun 2026). Importantly, a single file rejection does not automatically force overall failure; the meta-judge weighs the importance of any rejected file before issuing its verdict (Gajjar et al., 3 Jun 2026). It returns the same JSON schema (Gajjar et al., 3 Jun 2026).

The paper reports that single-shot judges suffered from context overload and unparsable outputs, whereas the two-tier design preserved context across files and models more effectively (Gajjar et al., 3 Jun 2026). In experiments, the two-tier judge delivered accurate verdicts 64.05% of the time and up to 96.9% for specific models, and a third tier that attempted to abstract semantics across files reduced accuracy and was abandoned (Gajjar et al., 3 Jun 2026). Gemma 4 (31B) was selected as the local judge model (Gajjar et al., 3 Jun 2026).

TeleJudge’s quantitative “semantic” metric is TAR, the TeleJudge Acceptance Rate (Gajjar et al., 3 Jun 2026). There are no additional bespoke similarity formulas beyond the accept/reject decisions and their aggregation (Gajjar et al., 3 Jun 2026). This design reflects a commitment to semantic equivalence under repository context rather than lexical similarity to a reference patch.

6. Evaluated systems, quantitative findings, and failure modes

TeleSWEBench evaluates three ASE frameworks: AIDER, OpenHands, and ClaudeCode (Gajjar et al., 3 Jun 2026). Primary experiments use AIDER for efficiency and openness, while follow-up ablations substitute OpenHands and ClaudeCode with the best-performing backbone to assess framework effects (Gajjar et al., 3 Jun 2026).

The LLM backbones are open-weight and locally served via NAUTILUS, except QwenCoder 2.5 via Ollama (Gajjar et al., 3 Jun 2026). The paper lists Qwen 3.5, Qwen 3.5 Small, GPT-OSS, Gemma 4, Kimi K2.5, and QwenCoder 2.5 as the principal backbones, and GLM-4.7 appears in Stage 1 localization results (Gajjar et al., 3 Jun 2026). Hardware is specified as Intel i9-14900KF, 62 GiB RAM, and NVIDIA RTX 4090 (24 GB), and each task is subject to a hard cap of $\rightarrow$9 seconds; failure to produce a final patch within that limit counts as failure (Gajjar et al., 3 Jun 2026).

Stage 1 localization results show a general degradation of EM from Easy to Difficult, with NM and especially NC increasing as task difficulty rises (Gajjar et al., 3 Jun 2026). The benchmark highlights a recurring behavioral pattern of “timidness,” in which larger models identify likely edits but do not emit concrete code changes, instead returning plain-text suggestions or lengthy explanations (Gajjar et al., 3 Jun 2026). It also notes verbose non-action and context exhaustion during iterative repository exploration (Gajjar et al., 3 Jun 2026).

Selected cumulative localization outcomes are as follows (Gajjar et al., 3 Jun 2026):

Model	Cumulative EM (%)	Cumulative NC (%)
Qwen 3.5	14.0	58.7
Qwen 3.5 Small	3.5	82.6
GPT-OSS	11.0	65.7
Gemma 4	2.8	86.8
Kimi K2.5	5.3	77.3
QwenCoder 2.5	17.4	37.8
GLM-4.7	1.5	92.2

At the per-difficulty level, QwenCoder 2.5 attains EM of 37.8 on Easy, 20.4 on Medium, and 5.3 on Difficult tasks, whereas Qwen 3.5 achieves 24.0, 25.0, and 1.3 respectively (Gajjar et al., 3 Jun 2026). These figures indicate that localization remains difficult even before synthesis is considered, and that EM on Difficult tasks is uniformly low across the tested models (Gajjar et al., 3 Jun 2026).

Stage 2, computed only on EM cases, reveals that unit-test success and semantic correctness often diverge (Gajjar et al., 3 Jun 2026). GLM-4.7 is reported as the strongest relative end-to-end performer with $T=0.9$0, despite having Stage 1 NC above 90%, which the paper interprets as extreme caution paired with relatively reliable edits when the model does act (Gajjar et al., 3 Jun 2026). QwenCoder 2.5 has $T=0.9$1, $T=0.9$2, and $T=0.9$3, indicating solutions that pass narrow tests while failing semantic and cross-file correctness (Gajjar et al., 3 Jun 2026). Kimi K2.5 records $T=0.9$4, $T=0.9$5, and $T=0.9$6 (Gajjar et al., 3 Jun 2026). Qwen3-small and Gemma 4 reach approximately 44–47% UAR but only single-digit SRP because judge acceptance remains low (Gajjar et al., 3 Jun 2026).

The benchmark’s central empirical conclusion is that the strongest ASE tools achieve up to 25% of shippable changes (Gajjar et al., 3 Jun 2026). In context, this means that exact localization, semantic acceptability, and test validation jointly remain challenging in the telecom setting. A plausible implication is that benchmark difficulty arises not only from synthesis but from the interaction between repository navigation, protocol semantics, and cross-file coherence.

7. Relation to adjacent benchmarking infrastructure, limitations, and use

TeleSWEBench is distinct from general-purpose SWE benchmarks and from telemetry-first execution substrates, but these are adjacent rather than mutually exclusive categories. The executable benchmarking suite in "An Executable Benchmarking Suite for Tool-Using Agents" provides a SWE-Gym slice with SWE-bench-compatible verification, shared adapters, manifests, event schemas, replay classes, declared drivers, and reporting pipelines, together with an evidence-admission contract that separates admitted paper-facing rows from smoke, fixture, preflight, and diagnostic rows (Zhong et al., 10 May 2026). Its description states that this provides a drop-in execution and evidence layer to operationalize a TeleSWEBench-style evaluation, including per-step, per-tool, and verifier telemetry, replay/freeze bindings, and auditable reporting (Zhong et al., 10 May 2026). This suggests a methodological complement: TeleSWEBench contributes a telecom-native task set and evaluation protocol, while the suite contributes a generic executable substrate for auditable, replayable agent evaluation (Zhong et al., 10 May 2026).

TeleSWEBench’s present scope is limited in several ways. It covers a single repository, srsRAN 5G, so architectural idiosyncrasies may limit generalization to other telecom stacks (Gajjar et al., 3 Jun 2026). Only three ASE frameworks were evaluated, due to cost and infrastructure constraints, and performance may differ for other open or closed systems (Gajjar et al., 3 Jun 2026). Reliance on repository-native tests means that some edge cases may be missed and that structurally unconventional but correct code may be penalized, which is partly why TeleJudge is included (Gajjar et al., 3 Jun 2026). Infrastructure limitations also mattered: NAUTILUS serving incurred occasional API drops and premature timeouts, and Stage 2 is computed only for EM cases where generation completed within the allotted time budget (Gajjar et al., 3 Jun 2026).

For practical use, the benchmark recommends an end-to-end workflow centered on the two-stage pipeline: apply a task time budget such as $T=0.9$7 seconds; compute EM, PM, OA, NM, and NC from edited-file sets; then, for EM cases, compile and run srsRAN tests and execute TeleJudge to obtain UAR, TAR, and SRP (Gajjar et al., 3 Jun 2026). The benchmark further recommends using the provided prompts for generation, validation, and TeleJudge to preserve consistency in reruns or extensions, and it advises caution when interpreting large-context models because they may exhibit timidness and context exhaustion, whereas smaller code-specialized models may produce more decisive but semantically weaker patches (Gajjar et al., 3 Jun 2026).

TeleSWEBench therefore occupies a specific position within the evaluation ecosystem. It is a telecom-native, commit-driven benchmark built to expose whether LLM-based ASE systems can produce localized, semantically coherent, and test-verified modifications in a real 5G codebase (Gajjar et al., 3 Jun 2026). Its core contribution lies in separating localization from synthesis, pairing executable verification with hierarchical semantic judgment, and documenting a substantial capability gap under telecom-specific constraints (Gajjar et al., 3 Jun 2026).