TeleCom-Bench: Telecom Operations Benchmark
- TeleCom-Bench is a telecom benchmark that measures LLMs' ability to handle both theoretical comprehension and operational execution in network environments.
- It is structured in a two-tiered hierarchy featuring Multi-dimensional Knowledge Comprehension and End-to-End Knowledge Application across 12 evaluation sets.
- Empirical findings reveal a pronounced execution gap: models excel in diagnosis tasks but struggle with procedural solution generation in live network operations.
Searching arXiv for TeleCom-Bench and closely related telecom benchmark papers to ground the article. Searching arXiv for "TeleCom-Bench industrial telecommunication applications benchmark". Searching arXiv for related telecom benchmark papers: TeleCom-Bench, TeleMath, TeleResilienceBench, SWE-Bench 5G, TeleSWEBench, MM-Telco, CeBed, and τ²-Bench. TeleCom-Bench is a telecommunications benchmark designed to measure how far LLMs are from real industrial telecommunication applications. It comprises 12 evaluation sets with 22,678 curated samples and is organized as a two-level hierarchy: Multi-dimensional Knowledge Comprehension, which combines telecommunication fundamentals, 3GPP protocols, 5G network architecture, and proprietary product knowledge, and End-to-End Knowledge Application, which formalizes six core tasks from authentic trajectories in live network agent workflows across network optimization and fault maintenance scenarios (Xiao et al., 18 May 2026).
1. Definition, motivation, and problem setting
TeleCom-Bench was introduced to address a specific deficiency in telecom evaluation: earlier benchmarks were strong on public standards or isolated reasoning skills, but weak on equipment-specific documentation, vendor manuals, configuration guides, diagnostics docs, field cases, and multi-step operational workflows. The benchmark is therefore constructed as a bridge between knowing telecom facts and carrying out telecom work in production-relevant settings (Xiao et al., 18 May 2026).
The motivating critique has three parts. First, existing resources exhibit a static knowledge bias: they measure whether a model can recall telecom theory, protocol terminology, or 5G concepts, but not whether it can function in an operator environment. Second, they suffer from atomic skill fragmentation: real telecom workflows are not single-step tasks, but chains involving intent understanding, entity grounding, event checking, tool invocation, root-cause diagnosis, and remediation. Third, they underrepresent product-specific and operational documentation, even though industrial telecom knowledge is often embedded in proprietary sources rather than only in public standards (Xiao et al., 18 May 2026).
This positioning is central to the benchmark’s role. TeleCom-Bench does not treat telecommunications as a purely linguistic domain. It treats it as an operational domain tied to live network operations, where safety, structured tool use, and executable actions matter. A plausible implication is that the benchmark is less a telecom QA set than a telecom agent diagnostic, because it explicitly tests the transition from semantic understanding to production-relevant behavior.
2. Data sources and construction pipeline
The benchmark is built from two distinct but connected pipelines: one for knowledge comprehension and one for knowledge application. For comprehension, the authors process 1.52 TB of raw material into 17,442 samples. The sources include 3GPP specifications, IETF RFCs, ITU-T recommendations, IEEE standards, textbooks, academic papers, patents, engineering encyclopedias, certified exam banks, vendor manuals, field operational cases, and live service data from intelligent customer support scenarios (Xiao et al., 18 May 2026).
Preprocessing includes format normalization from CHM/DOCX/EPUB/PDF into Markdown, OCR and layout-aware parsing for PDF documents, and subsequent sanitization, deduplication, and segmentation. The documents are also assigned metadata for language, quality, toxicity, topic, and license. Sample generation is driven by a task-adaptive LLM distillation method, with document-type-specific strategies: technical standards use paradigm-constrained extraction, manuals/configuration guides use JSON structural generation, academic papers use progressive prompt-chain mining, and heterogeneous documents use cross-document knowledge mining (Xiao et al., 18 May 2026).
A major design feature is knowledge graph-driven synthesis. The graph represents telecom entities and relations such as NetworkFunction, Interface, KPI, [causes], [flow_pair], and [impacts_kpi]. Two domain-specific graph structures are emphasized: an Equipment-Scenario Hierarchy Tree, which maps equipment attributes to application scenarios, and a Causal Fault Chain, structured as Symptom → Intermediate Mechanism → Root Cause → Solution. These structures generate atomic QA, multi-hop QA, and aggregate QA, and are used together with graph-based distractor generation and GraphRAG-style correction to reduce hallucination and improve discriminative quality (Xiao et al., 18 May 2026).
For application data, the benchmark uses real commercial network data from Network Element Management Systems and Unified Management Environments, acquired through MML commands, SFTP, and SNMP. It integrates Performance Management (PM), Configuration Management (CM), and Fault Management (FM) data. Sensitive identifiers such as IMSI and GPS coordinates are anonymized using irreversible hashing, and PM/FM/CM streams are temporally aligned into coherent network state observations. The construction process is explicitly Agent-in-the-Loop: domain-specific agents solve actual network issues, their trajectories are mined into samples, the trajectories are field-verified using closed-loop execution, and senior engineers audit and refine the traces (Xiao et al., 18 May 2026).
3. Benchmark structure and task taxonomy
TeleCom-Bench is divided into six knowledge-comprehension sets and six knowledge-application sets. The comprehension layer covers both public telecom theory and proprietary product knowledge; the application layer formalizes a sequential workflow for telecom agents (Xiao et al., 18 May 2026).
| Evaluation set | Samples | Layer |
|---|---|---|
| Basic Knowledge | 2,662 | Knowledge Comprehension |
| 5G Network | 2,564 | Knowledge Comprehension |
| 3GPP Protocols | 4,043 | Knowledge Comprehension |
| Wireless Network | 3,725 | Knowledge Comprehension |
| Wired Network | 3,488 | Knowledge Comprehension |
| Core Network | 960 | Knowledge Comprehension |
| Intent Recognition | 2,174 | Knowledge Application |
| Entity Extraction | 365 | Knowledge Application |
| Tool Invocation | 585 | Knowledge Application |
| Event Verification | 146 | Knowledge Application |
| Root Cause Diagnosis | 983 | Knowledge Application |
| Solution Generation | 983 | Knowledge Application |
The Multi-dimensional Knowledge Comprehension layer covers Basic Theory, 5G network architecture, 3GPP protocols, and product knowledge spanning wireless network, wired network, and core network products. This is the benchmark’s main departure from public-standards-only evaluation: it incorporates proprietary product knowledge across multiple network segments, using knowledge graph-driven synthesis, knowledge distillation-driven generation, and structured extraction from official manuals, configuration guides, patents, and operational documents (Xiao et al., 18 May 2026).
The End-to-End Knowledge Application layer defines six tasks from real operational trajectories. Intent Recognition classifies the business scenario or user intent. Entity Extraction extracts structured entities such as network elements, locations, timestamps, or identifiers. Event Verification verifies whether a network event or state is valid using structured management-system data. Tool Invocation selects the appropriate external tool or API and its parameters. Root Cause Diagnosis infers the cause of faults from alarms, performance metrics, and related signals. Solution Generation produces an actionable remediation plan that matches the diagnosis and is executable in field operations (Xiao et al., 18 May 2026).
The sequentiality of these tasks matters. The benchmark does not merely test whether a model can answer telecom questions; it tests whether the model can participate in a workflow that begins in language and ends in operational action. This suggests that the application layer functions as a staged abstraction of telecom agent pipelines rather than a set of unrelated tasks.
4. Evaluation protocol and measurement design
TeleCom-Bench evaluates eight state-of-the-art LLMs: Qwen3-32B, Qwen3-235B-A22B, DeepSeek-V3.2, Gemini 2.5, Grok 4.1, GLM-4.7, Doubao-pro, and Kimi K2. All models are tested under the same inference setup: temperature = 0.7, reasoning mode enabled, 3 independent sampling trials per item, and majority vote for the final answer (Xiao et al., 18 May 2026).
The metrics vary by task type. Macro-F1 is used for multiple-response questions. Exact Match is used for structured QA. Open-ended responses are evaluated by LLM-as-a-Judge with three expert judges using a 5-point Likert scale, with reported human-alignment quality of Krippendorff’s alpha = 0.82. The appendix specifies three question formats: Multiple-Select Question, Subjective QA, and Structured QA. Multiple-select questions are used for architecture, interfaces, and protocol knowledge; subjective QA supports open-ended reasoning such as network optimization; structured QA is used for JSON-formatted extraction tasks (Xiao et al., 18 May 2026).
An important methodological point is that the benchmark does not collapse heterogeneous tasks into a single simplistic score. Instead, it measures different aspects of telecom competence with task-appropriate protocols. In practical terms, this allows TeleCom-Bench to distinguish models that are good at linguistic interface tasks from models that can support procedural telecom operations. That distinction becomes decisive in the empirical findings.
5. Empirical findings and the “Execution Wall”
The paper’s central result is the Execution Wall. Models perform well on linguistic interface tasks but collapse on procedural execution tasks. The headline summary is approximately 90% accuracy on tasks such as intent recognition and entity extraction, versus approximately 30% accuracy on procedural tasks such as solution generation (Xiao et al., 18 May 2026).
The main table makes this contrast concrete. Intent Recognition is reported at around 93–95% and Entity Extraction at around 95–100%. By contrast, Root Cause Diagnosis falls in the range of roughly 49–71%, and Solution Generation falls to roughly 4–31%. The most striking example is Qwen3-235B, which achieves 71.49% on Root Cause Diagnosis but only 4.67% on Solution Generation, producing a 66.82-point gap. DeepSeek-V3.2 shows a similar pattern, with 63.00% on diagnosis and 5.61% on solution generation. Even the strongest model on solution generation, Doubao-pro at 30.72%, remains far below what the paper regards as necessary for autonomous production deployment (Xiao et al., 18 May 2026).
The benchmark uses this gap to argue that current LLMs act as competent diagnosticians but not as field engineers. They can often parse user intent, extract structured information, and infer causes from telecom data, but they do not reliably choose the correct tools, follow operational procedures, or produce executable and safety-compliant remediation steps. The paper further reports that tool context is not enough: even when models are given a 23-tool library during fault-resolution tests, generalist models tend to ignore tools, misuse them, or generate non-executable commands (Xiao et al., 18 May 2026).
Several secondary findings refine this diagnosis. Scaling is non-monotonic: larger models are not always better, and Qwen3-32B can outperform Qwen3-235B on product knowledge, suggesting that coverage of vendor documentation matters more than parameter count in some telecom settings. The authors also report a positive correlation between Mixture-of-Experts architecture and tool-calling performance, with Pearson correlation . A plausible implication is that TeleCom-Bench is measuring a specific mixture of domain grounding, procedural memory, and tool-conditioned reasoning rather than generic language-model scale (Xiao et al., 18 May 2026).
A common misconception addressed by these results is that strong telecom knowledge accuracy implies operational readiness. TeleCom-Bench shows that diagnosis and execution are separable capabilities, and that the hard boundary lies not at interpretation but at procedural synthesis.
6. Position within the telecom benchmark ecosystem
TeleCom-Bench is explicitly framed against earlier telecom benchmarks such as SPEC5G, TeleQnA, TSpec-LLM, ORAN-Bench-13K, TelecomGPT, TeleTables, TeleLogs, OpsEval, and TeleMath, which are described as capturing only part of the overall problem space (Xiao et al., 18 May 2026). Its distinctive claim is breadth across both knowledge comprehension and knowledge application, especially through incorporation of proprietary product documentation and live operational trajectories.
At the same time, the broader telecom benchmark ecosystem has become increasingly specialized. TelecomGPT extends telecom evaluation with Telecom Math Modeling, Telecom Open QnA, and Telecom Code Tasks, emphasizing math modeling, standards classification, and telecom code generation (Zou et al., 2024). TeleMath isolates numerical mathematical reasoning in telecommunications through 500 question-answer pairs with numerical answers only (Colle et al., 12 Jun 2025). CeBed standardizes deep data-driven OFDM channel estimation through a scenario taxonomy, standardized protocols, and ten baselines (Feriani et al., 2023). MM-Telco expands the benchmark space into multimodal tasks involving 3GPP standards, network logs, packet captures, technical diagrams, and documentation (Gupta et al., 17 Nov 2025).
Other efforts target specific deployment regimes. TeleResilienceBench measures reasoning resilience, namely whether a model can recover after inheriting a partially completed, already flawed reasoning trace in telecom tasks (Gajjar et al., 11 May 2026). -Bench, also written as TeleCom-Bench in its telecom-domain formulation, evaluates conversational agents in a dual-control telecom technical-support environment modeled as a Dec-POMDP, where both agent and user act in a shared state (Barres et al., 9 Jun 2025). SWE-Bench 5G studies whether coding agents can diagnose and fix real bugs in 5G core network software (Chen et al., 29 Apr 2026), while TeleSWEBench evaluates repository-scale software engineering on srsRAN 5G with 734 questions and executable unit tests, together with the hierarchical TeleJudge framework (Gajjar et al., 3 Jun 2026). TeleAntiFraud-Bench moves into audio-text telecom fraud detection with a standardized benchmark derived from 28,511 speech-text pairs (Ma et al., 31 Mar 2025).
Within this landscape, TeleCom-Bench occupies the role of a broad industrial-operational benchmark rather than a narrow capability test. It asks whether models can move from telecom literacy to telecom work. The benchmark’s main contribution is therefore diagnostic: it localizes the present deficit not in recognition or diagnosis, but in the transition from diagnosis to executable action. In the paper’s own formulation, current models are competent diagnosticians, not field engineers (Xiao et al., 18 May 2026).