---
title: 'TeleSWEBench: Telecom ASE Benchmark'
url: https://www.emergentmind.com/topics/teleswebench
type: topic
---

# TeleSWEBench: Telecom ASE Benchmark

Searching arXiv for TeleSWEBench and closely related benchmarking papers.
TeleSWEBench is a commit-driven benchmark for evaluating LLM-powered software engineering in telecommunications, introduced to measure whether agentic ASE systems can operate effectively on a specialized wireless codebase rather than on generic software tasks [2606.05001]. It is built from real developer commits mined from the srsRAN 5G repository and formulates those commits as natural-language tasks with executable tests, while also supplying a parallel semantic evaluation path through a hierarchical LLM-as-a-Judge framework called TeleJudge [2606.05001]. In the broader benchmarking landscape, TeleSWEBench is telecom-native rather than general-purpose: it targets a softwareized 5G stack with asynchronous timing constraints, layered protocol logic, cross-file dependencies, and strict 3GPP-oriented requirements that are not well represented in standard software engineering benchmarks [2606.05001]. A related executable benchmarking suite provides a telemetry, replay, provenance, and evidence-admission substrate for SWE-bench-compatible evaluation across code, web, and micro-task settings; its description explicitly presents that substrate as suitable for operationalizing a TeleSWEBench-style evaluation when the emphasis is auditable execution and admitted evidence rather than a solver leaderboard [2605.11030].

## 1. Domain motivation and benchmark rationale

TeleSWEBench is motivated by the increasing softwareization of telecommunications systems under zero-touch management, O-RAN, and AI-RAN, where network functions become cloud-native software components deployed on commodity hardware rather than proprietary appliances [2606.05001]. In this setting, modern wireless networks constitute large, tightly coupled codebases spanning PHY through core, and the principal bottleneck shifts from hardware to software engineering and code generation [2606.05001].

The benchmark is designed around the observation that existing coding benchmarks do not capture the properties of telecom software stacks. General-purpose benchmarks such as HumanEval, MBPP, SWE-bench, and related variants primarily emphasize Python or Java ecosystems and generic library or web tasks, whereas telecom software involves hardware-adjacent C++ idioms, stateful protocol logic, and strict layered semantics [2606.05001]. Telecom-specific evaluations prior to TeleSWEBench had mainly focused on multiple-choice question answering over standards or on static repository understanding, rather than end-to-end code modification within a real stack [2606.05001].

This positioning distinguishes TeleSWEBench from both generic software engineering benchmarks and non-executable telecom evaluations. The benchmark is therefore intended to test whether LLM-based ASE tools can localize, modify, and functionally repair or extend code within srsRAN 5G, a repository characterized by cross-file coupling and mathematically rigorous protocol behavior [2606.05001]. A plausible implication is that TeleSWEBench is aimed less at broad coding aptitude and more at domain-grounded software engineering competence under telecom-specific operational constraints.

## 2. Construction from srsRAN 5G commits

TeleSWEBench is derived from real developer commits in the srsRAN 5G repository, licensed under AGPL-3.0, with the source history described as containing 15k+ commits from 2023–2025 [2606.05001]. The average commit touches 5.57 files, which is used as evidence of the cross-file and tightly coupled nature of the target domain [2606.05001]. Tasks are mined from bug fixes, feature additions, and protocol updates [2606.05001].

Task generation follows a commit-driven pipeline. For each candidate commit, tasks are generated and validated through an LLM cascade, proceeding from smaller to larger models with early stopping once a valid instance is produced [2606.05001]. A second heuristic attempts difficulty levels in the order Difficult $\rightarrow$ Medium $\rightarrow$ Easy, also with early stopping: if Difficult fails, the commit is abandoned; if Difficult succeeds but Medium fails, the process does not proceed to Easy [2606.05001]. A separate validator LLM checks clarity, difficulty-appropriate information density, IDE realism, faithfulness to the original commit diff, and internal consistency, and only items marked valid with confidence at least $T=0.9$ are retained [2606.05001].

The resulting benchmark contains 734 verified tasks with executable tests [2606.05001]. The difficulty distribution is 313 Difficult, 279 Medium, and 142 Easy tasks [2606.05001]. File scope is broad: Easy tasks have mean 29.82 files and median 7, Medium tasks have mean 19.57 and median 4, Difficult tasks have mean 18.42 and median 5, and across all questions the scope ranges from 1 to 300 files [2606.05001].

| Quantity | Value |
|---|---:|
| Total verified tasks with executable tests | 734 |
| Difficult tasks | 313 |
| Medium tasks | 279 |
| Easy tasks | 142 |
| Scope across all questions | 1 to 300 files |
| Average files touched per source commit | 5.57 |

These statistics support the benchmark’s claim to repository-scale rather than function-local evaluation. The means and medians indicate that even easier tasks may involve wide file exposure, while the upper bound of 300 files underscores the localization burden imposed by telecom codebases [2606.05001].

## 3. Task formulation and telecom-specific difficulty structure

Each TeleSWEBench task is a natural-language instruction derived from an actual commit context [2606.05001]. The benchmark enforces a strict difficulty taxonomy that is defined by how much localization information is disclosed.

Easy tasks are near-mechanical edits that include exact file paths and line references, exact removals and insertions, and full self-containment so that a search-and-replace-style procedure is sufficient [2606.05001]. Medium tasks explain the “what” and “why,” mention affected files or subsystems but not exact line numbers, and provide salient facts such as constants, function names, or version identifiers; these tasks require structural understanding in order to localize edits [2606.05001]. Difficult tasks specify high-level objectives with minimal hints and must not name specific file paths or functions, thereby requiring autonomous localization and planning within the repository [2606.05001].

The benchmark provides the task description and uses the original commit to define a ground-truth target file set for localization evaluation [2606.05001]. For Stage 2 functional verification, repository-native unit and integration tests are executed where available, and TeleJudge supplies an additional semantic verdict at file and patch level [2606.05001].

Representative task types include authentic bug fixes, feature additions, and protocol updates touching areas such as ASN.1 utilities, DCI packing, schedulers, and MAC/RLC logic [2606.05001]. The examples described in the source material include refactors that replace enum-based VRB-to-PRB mapping with boolean flags across encoder and decoder logic, builders, and unit tests; configurable scheduler parameters; and ASN.1 octet string API changes [2606.05001]. The benchmark is therefore not restricted to isolated syntax corrections; it targets semantically entangled modifications in C++ code under protocol and systems constraints.

## 4. Evaluation protocol: localization, execution, and semantic judgment

TeleSWEBench uses a two-stage evaluation pipeline that explicitly decouples localization from synthesis [2606.05001]. Stage 1 asks whether the agent understood where changes should be made. With $T$ denoting the ground-truth target file set and $P$ the files modified by the agent, five localization outcomes are defined [2606.05001]:

- **Exact Match (EM)**: $P = T$
- **Partial Match (PM)**: $P \subset T$
- **Over Addressed (OA)**: $T \subset P$
- **No Match (NM)**: $P \cap T = \varnothing$ and $P \neq \varnothing$
- **No Changes (NC)**: $P = \varnothing$

Stage 2 measures functional correctness, but only for EM cases, specifically to separate code synthesis from localization error [2606.05001]. It applies two parallel checks. The execution-driven path applies the patch, compiles srsRAN 5G, and runs repository-native unit and integration tests over the 734-task set [2606.05001]. The semantic path runs TeleJudge, a hierarchical LLM-as-a-Judge framework that delivers a holistic pass/fail verdict after file-level analysis [2606.05001].

The execution harness mines tests from srsRAN’s native test hierarchy and pairs implementation files with tests through mirrored directory structure and filename patterns such as the `_test` suffix, together with thresholded similarity intended to reduce spurious matches [2606.05001]. Because complete test vectors are consistently available at merge commits, the benchmark primarily leverages merges to ensure compilability and executability [2606.05001]. Tasks are mapped to tests if any ground-truth modified file matches the implementation file associated with a test suite [2606.05001].

The formal Stage 2 metrics are defined over the EM subset. Let $N$ be the number of tasks and $N_{EM}$ the number of Exact Match localizations. Let $N_{UT}$ be the number of EM patches that compile and pass all associated tests; then
$$
UAR = \frac{N_{UT}}{N_{EM}}.
$$
Let $N_{pass}$ be the number of EM patches accepted by TeleJudge; then
$$
TAR = \frac{N_{pass}}{N_{EM}}.
$$
Let $N_{SR}$ count EM patches that pass both TeleJudge and tests; then
$$
SRP = \frac{N_{SR}}{N_{EM}}, \qquad
N_{SR} = \sum_{i=1}^{N_{EM}} 1[\mathrm{JudgeAccept}_i \wedge \mathrm{TestPass}_i].
$$
These definitions make SRP the benchmark’s strictest criterion and the closest proxy to “shippable changes” in the benchmark’s own terminology [2606.05001].

## 5. TeleJudge and hierarchical semantic evaluation

TeleJudge is a hierarchical LLM-as-a-Judge framework introduced to complement execution-only evaluation, which the benchmark characterizes as rigid because structurally different yet functionally valid solutions may be penalized by tests alone [2606.05001]. TeleJudge therefore supplies a semantic, context-aware judging path parallel to unit testing [2606.05001].

Its first tier is the File-Level Judge. For each file touched either in the ground-truth patch or in the agent’s patch, the judge receives three inputs: the original task description, the ground-truth diff for that file, and the agent’s diff for the same file [2606.05001]. It must accept if the modifications are functionally equivalent in the sense of addressing the same root cause or feature, even when stylistically different; otherwise it rejects [2606.05001]. The output is JSON of the form:
```json
{ "verdict": "accept"/"reject", "confidence": 0-100, "reasons": [...] }
```

The second tier is the Holistic Meta-Judge. It receives the task together with all per-file verdicts and reasons, and makes a final accept or reject decision for the entire patch [2606.05001]. Importantly, a single file rejection does not automatically force overall failure; the meta-judge weighs the importance of any rejected file before issuing its verdict [2606.05001]. It returns the same JSON schema [2606.05001].

The paper reports that single-shot judges suffered from context overload and unparsable outputs, whereas the two-tier design preserved context across files and models more effectively [2606.05001]. In experiments, the two-tier judge delivered accurate verdicts 64.05% of the time and up to 96.9% for specific models, and a third tier that attempted to abstract semantics across files reduced accuracy and was abandoned [2606.05001]. Gemma 4 (31B) was selected as the local judge model [2606.05001].

TeleJudge’s quantitative “semantic” metric is TAR, the TeleJudge Acceptance Rate [2606.05001]. There are no additional bespoke similarity formulas beyond the accept/reject decisions and their aggregation [2606.05001]. This design reflects a commitment to semantic equivalence under repository context rather than lexical similarity to a reference patch.

## 6. Evaluated systems, quantitative findings, and failure modes

TeleSWEBench evaluates three ASE frameworks: AIDER, OpenHands, and ClaudeCode [2606.05001]. Primary experiments use AIDER for efficiency and openness, while follow-up ablations substitute OpenHands and ClaudeCode with the best-performing backbone to assess framework effects [2606.05001].

The LLM backbones are open-weight and locally served via NAUTILUS, except QwenCoder 2.5 via Ollama [2606.05001]. The paper lists Qwen 3.5, Qwen 3.5 Small, GPT-OSS, Gemma 4, Kimi K2.5, and QwenCoder 2.5 as the principal backbones, and GLM-4.7 appears in Stage 1 localization results [2606.05001]. Hardware is specified as Intel i9-14900KF, 62 GiB RAM, and NVIDIA RTX 4090 (24 GB), and each task is subject to a hard cap of $Time_m = 300$ seconds; failure to produce a final patch within that limit counts as failure [2606.05001].

Stage 1 localization results show a general degradation of EM from Easy to Difficult, with NM and especially NC increasing as task difficulty rises [2606.05001]. The benchmark highlights a recurring behavioral pattern of “timidness,” in which larger models identify likely edits but do not emit concrete code changes, instead returning plain-text suggestions or lengthy explanations [2606.05001]. It also notes verbose non-action and context exhaustion during iterative repository exploration [2606.05001].

Selected cumulative localization outcomes are as follows [2606.05001]:

| Model | Cumulative EM (%) | Cumulative NC (%) |
|---|---:|---:|
| Qwen 3.5 | 14.0 | 58.7 |
| Qwen 3.5 Small | 3.5 | 82.6 |
| GPT-OSS | 11.0 | 65.7 |
| Gemma 4 | 2.8 | 86.8 |
| Kimi K2.5 | 5.3 | 77.3 |
| QwenCoder 2.5 | 17.4 | 37.8 |
| GLM-4.7 | 1.5 | 92.2 |

At the per-difficulty level, QwenCoder 2.5 attains EM of 37.8 on Easy, 20.4 on Medium, and 5.3 on Difficult tasks, whereas Qwen 3.5 achieves 24.0, 25.0, and 1.3 respectively [2606.05001]. These figures indicate that localization remains difficult even before synthesis is considered, and that EM on Difficult tasks is uniformly low across the tested models [2606.05001].

Stage 2, computed only on EM cases, reveals that unit-test success and semantic correctness often diverge [2606.05001]. GLM-4.7 is reported as the strongest relative end-to-end performer with $SRP = 25.0\%$, despite having Stage 1 NC above 90%, which the paper interprets as extreme caution paired with relatively reliable edits when the model does act [2606.05001]. QwenCoder 2.5 has $UAR = 57.73\%$, $TAR = 0.0\%$, and $SRP = 0.0\%$, indicating solutions that pass narrow tests while failing semantic and cross-file correctness [2606.05001]. Kimi K2.5 records $UAR = 36.36\%$, $TAR = 40.91\%$, and $SRP = 13.64\%$ [2606.05001]. Qwen3-small and Gemma 4 reach approximately 44–47% UAR but only single-digit SRP because judge acceptance remains low [2606.05001].

The benchmark’s central empirical conclusion is that the strongest ASE tools achieve up to 25% of shippable changes [2606.05001]. In context, this means that exact localization, semantic acceptability, and test validation jointly remain challenging in the telecom setting. A plausible implication is that benchmark difficulty arises not only from synthesis but from the interaction between repository navigation, protocol semantics, and cross-file coherence.

## 7. Relation to adjacent benchmarking infrastructure, limitations, and use

TeleSWEBench is distinct from general-purpose SWE benchmarks and from telemetry-first execution substrates, but these are adjacent rather than mutually exclusive categories. The executable benchmarking suite in "An Executable Benchmarking Suite for Tool-Using Agents" provides a SWE-Gym slice with SWE-bench-compatible verification, shared adapters, manifests, event schemas, replay classes, declared drivers, and reporting pipelines, together with an evidence-admission contract that separates admitted paper-facing rows from smoke, fixture, preflight, and diagnostic rows [2605.11030]. Its description states that this provides a drop-in execution and evidence layer to operationalize a TeleSWEBench-style evaluation, including per-step, per-tool, and verifier telemetry, replay/freeze bindings, and auditable reporting [2605.11030]. This suggests a methodological complement: TeleSWEBench contributes a telecom-native task set and evaluation protocol, while the suite contributes a generic executable substrate for auditable, replayable agent evaluation [2605.11030].

TeleSWEBench’s present scope is limited in several ways. It covers a single repository, srsRAN 5G, so architectural idiosyncrasies may limit generalization to other telecom stacks [2606.05001]. Only three ASE frameworks were evaluated, due to cost and infrastructure constraints, and performance may differ for other open or closed systems [2606.05001]. Reliance on repository-native tests means that some edge cases may be missed and that structurally unconventional but correct code may be penalized, which is partly why TeleJudge is included [2606.05001]. Infrastructure limitations also mattered: NAUTILUS serving incurred occasional API drops and premature timeouts, and Stage 2 is computed only for EM cases where generation completed within the allotted time budget [2606.05001].

For practical use, the benchmark recommends an end-to-end workflow centered on the two-stage pipeline: apply a task time budget such as $Time_m = 300$ seconds; compute EM, PM, OA, NM, and NC from edited-file sets; then, for EM cases, compile and run srsRAN tests and execute TeleJudge to obtain UAR, TAR, and SRP [2606.05001]. The benchmark further recommends using the provided prompts for generation, validation, and TeleJudge to preserve consistency in reruns or extensions, and it advises caution when interpreting large-context models because they may exhibit timidness and context exhaustion, whereas smaller code-specialized models may produce more decisive but semantically weaker patches [2606.05001].

TeleSWEBench therefore occupies a specific position within the evaluation ecosystem. It is a telecom-native, commit-driven benchmark built to expose whether LLM-based ASE systems can produce localized, semantically coherent, and test-verified modifications in a real 5G codebase [2606.05001]. Its core contribution lies in separating localization from synthesis, pairing executable verification with hierarchical semantic judgment, and documenting a substantial capability gap under telecom-specific constraints [2606.05001].

Source: https://www.emergentmind.com/topics/teleswebench