6G-Bench: Semantic Reasoning for AI 6G

Updated 5 July 2026

6G-Bench is an open benchmark that evaluates semantic communication and network-level reasoning in AI-native 6G networks through 30 standardization-aligned tasks.
It constructs a high-confidence evaluation set of 3,722 difficult multiple-choice questions curated from 10,000 generated scenarios, emphasizing robust multi-step reasoning.
It applies a worst-case regret minimization strategy over multi-turn horizons to simulate realistic network uncertainties and optimize AI agent performance.

6G-Bench is an open benchmark for evaluating semantic communication and network-level reasoning with foundation models in AI-native 6G networks. It defines a taxonomy of 30 decision-making tasks extracted from ongoing 6G and AI-agent standardization activities in 3GPP, IETF, ETSI, ITU-T, and the O-RAN Alliance, organizes them into five standardization-aligned capability categories, and builds a high-confidence evaluation set of 3,722 very-hard multiple-choice questions from an initial pool of 10,000 questions generated out of 113,475 scenarios (Ferrag et al., 9 Feb 2026). Its stated purpose is not telecom question answering in the narrow sense, but evaluation of whether a model can act as a semantic reasoning and control layer that interprets intents, policies, slices, trust relations, and multi-agent context under uncertain 6G conditions (Ferrag et al., 9 Feb 2026).

1. Definition and conceptual scope

6G-Bench is centered on network-level decision-making rather than on factual recall or specification lookup. The benchmark is framed against earlier telecom-oriented evaluations such as TeleQnA, TeleTables, TelAgentBench, LDOT, and CovertComBench, which do not systematically test semantic communication and network-level reasoning under uncertainty and future risk (Ferrag et al., 9 Feb 2026). In this setting, “semantic communication” denotes operation on meaning-bearing abstractions—mission intent, network policy, SLA state, trust relationships, and agent roles—rather than on bit-level recovery alone.

The underlying episode model is expressed as

$\mathcal{E}=\{d_t\}_{t=1}^{T}, \qquad d_t=(r_t,a_t,o_t,n_t),$

where $r_t$ is a natural-language request or intent, $a_t$ an agent action, $o_t$ an observation, and $n_t$ the network state (Ferrag et al., 9 Feb 2026). The network state is represented as

$n_t=(\text{slice}_t,\ell_t,j_t,\rho_t,\tau_t,e_t),$

with slice, latency, jitter, packet loss, throughput, and normalized edge load. The benchmark further defines a semantic state

$s_t=(i_t,n_t,p_t,x_t),$

where $i_t$ is mission or network intent, $p_t$ policy or SLA constraints, and $x_t$ operational state such as battery, speed, or location (Ferrag et al., 9 Feb 2026).

A central design choice is that the correct action is not defined by greedy instantaneous optimization. Instead, the oracle action is obtained by worst-case regret minimization over a finite horizon: $r_t$ 0 This formulation makes each question a robust control problem over future uncertainty rather than a static classification item (Ferrag et al., 9 Feb 2026). The benchmark therefore tests whether a model can choose the least bad action over multi-turn horizons when every option has downsides.

2. Task taxonomy and standardization alignment

The 30 tasks are grouped into five capability categories derived from current architectural directions in 3GPP, IETF, ETSI, ITU-T, and O-RAN (Ferrag et al., 9 Feb 2026).

Group	Scope	Task IDs
G1	Intent & Policy Reasoning	T1, T2, T3, T12, T15
G2	Network Slicing & Resource Management	T4, T5, T6, T7, T8, T10, T11, T13, T14, T29
G3	Trust, Security & SLA Awareness	T9, T16, T18, T26, T30
G4	AI-Native Networking & Agentic Control	T17, T18, T19, T20, T21, T27, T28
G5	Distributed Intelligence & Emerging 6G Use Cases	T22, T23, T24, T25, T28, T29

G1 comprises T1 “Intent Feasibility Assessment,” T2 “Intent Conflict Resolution,” T3 “Intent Drift Detection,” T12 “Conservative Continuation Decision,” and T15 “Decision Consistency under Replanning.” These tasks target semantic consistency under evolving network state and policy constraints, including cases where intent must be adapted rather than directly executed (Ferrag et al., 9 Feb 2026).

G2 covers slice selection, switching, fairness, compute placement, graceful degradation, SLA violation prediction, autonomy downgrade, swarm-level slice negotiation, scheduler adaptation, and immersive or AR prioritization. The task names are T4 “Slice Selection Reasoning,” T5 “Slice Switching Decision,” T6 “Slice Fairness vs Safety,” T7 “Compute Placement Decision,” T8 “Graceful Degradation under Edge Overload,” T10 “SLA Violation Prediction,” T11 “Preemptive Autonomy Downgrade,” T13 “Swarm-Level Slice Negotiation,” T14 “Scheduler Reconfiguration Adaptation,” and T29 “Immersive/AR Resource Prioritization” (Ferrag et al., 9 Feb 2026).

G3 addresses normative and policy-constrained security reasoning: T9 “Trust-Aware Offloading,” T16 “Network-Exposed Compute Marketplace,” T18 “AI Agent Identity & Onboarding,” T26 “Trust-Aware Third-Party Agent Exposure,” and T30 “Network Security Detection & Response Automation.” The emphasis is on trust-aware exposure, authorization reasoning, and automated security response under SLA and regulatory constraints (Ferrag et al., 9 Feb 2026).

G4 concerns the lifecycle and interaction of AI agents themselves. It includes T17 “Network-Knowledge RAG Augmentation,” T18 “AI Agent Identity & Onboarding,” T19 “AI Agent Interoperability & Federation,” T20 “Agent-to-Agent Communication Management,” T21 “Device-Network Task Offload Arbitration,” T27 “Agent Lifecycle & Management,” and T28 “6G Model Training-as-a-Service Decision” (Ferrag et al., 9 Feb 2026).

G5 addresses distributed intelligence and emerging 6G use cases: T22 “Federated / Collaborative Learning Orchestration,” T23 “Network-Assisted Digital Twin Control,” T24 “Sensing-Enhanced Decisioning (ISAC),” T25 “Disaster / Public-Safety Coordination,” T28 “Training-as-a-Service Decision,” and T29 “Immersive/AR Resource Prioritization” (Ferrag et al., 9 Feb 2026). Across these groups, the benchmark covers numeric, combinatorial, policy-driven, and multi-turn reasoning modes.

3. Dataset construction and validation pipeline

6G-Bench builds on $r_t$ 1-Bench and UAVBench. Each source episode is a multi-turn dialogue with network telemetry, mission context, actions, and observations. From 113,475 such scenarios, the benchmark generation pipeline reconstructs semantic states, identifies decision turns, computes oracle actions by the regret criterion, and turns those states into four-option multiple-choice questions (Ferrag et al., 9 Feb 2026).

Question generation is task-conditioned. The prompting process includes the formal definition of the target task and enforces “very hard” difficulty: at least four reasoning steps, at least four numeric parameters from multiple categories, explicit uncertainty, and selection of the option that minimizes worst-case regret. The generation stage was used to produce a balanced pool of 10,000 questions across the 30 tasks (Ferrag et al., 9 Feb 2026).

The filtering process proceeds in two stages. Automated filtering enforces structural and logical conditions such as exactly four options, one clear correct answer, no duplicates or near-duplicates, sufficient numeric grounding, and explicit uncertainty or future reasoning. Remaining items undergo expert human validation for semantic correctness, uniqueness of the correct answer, and alignment with 6G and standards-oriented reasoning. The result is a 3,722-question “high-confidence” evaluation set, while the full 10,000-question pool is released for training and fine-tuning of 6G-specialized models (Ferrag et al., 9 Feb 2026).

The released assets include the evaluation set, the full training pool, episode summaries derived from $r_t$ 2-Bench and UAVBench, and evaluation scripts. The repository is public at https://github.com/maferrag/6G-Bench (Ferrag et al., 9 Feb 2026).

4. Evaluation protocol and core metrics

The benchmark uses a standardized prompt that provides TASK_ID, TASK_NAME, a formal task definition, an episode summary, a question, and four answer options. Models are instructed to return a single JSON object of the form {"answer": "A"}. Parsing first attempts JSON extraction and then falls back to regex or standalone answer token extraction; if extraction fails, the prediction is marked incorrect (Ferrag et al., 9 Feb 2026).

The main metric is deterministic single-shot accuracy, or pass@1: $r_t$ 3 Per-task accuracy is

$r_t$ 4

and group-level accuracy is the unweighted mean over tasks in a capability category (Ferrag et al., 9 Feb 2026).

For a reasoning-intensive subset, the benchmark also reports stochastic pass@k: $r_t$ 5 This metric is used to distinguish deterministic failure from recoverable reasoning uncertainty under multiple samples (Ferrag et al., 9 Feb 2026).

The first paper evaluates 22 foundation models spanning dense and mixture-of-experts architectures, open-weight and proprietary systems, and context windows up to 1M tokens (Ferrag et al., 9 Feb 2026). A later scaling study reuses 6G-Bench as a “standardization-aligned benchmark comprising 30 decision-making tasks across five capability domains” to study compact LLMs from 135M to 7B parameters, adding deterministic accuracy $r_t$ 6, stochastic robustness $r_t$ 7, the instability gap $r_t$ 8, and an edge-oriented efficiency metric

$r_t$ 9

where $a_t$ 0 is mean single-query latency and $a_t$ 1 is peak VRAM (Ferrag et al., 2 Mar 2026).

5. Empirical findings and scaling behavior

Across the 22-model evaluation, deterministic pass@1 ranges from 0.228 to 0.829. The highest reported pass@1 is 0.829 for meta-llama/llama-4-maverick, followed by 0.818 for qwen/qwen3-coder-next, 0.811 for mistral-small-creative, 0.795 for mistral-14b-2512, 0.794 for openai/gpt-5.2-chat, 0.790 for openai/gpt-5.2-codex, and 0.789 for deepseek-v3.2-exp (Ferrag et al., 9 Feb 2026).

Performance is not uniform across task categories. Intent and policy reasoning is the strongest group: qwen/qwen3-coder-next reaches $a_t$ 2, mistral-small-creative 0.885, llama-4-maverick 0.881, and gpt-5.2-chat 0.878. In G2, llama-4-maverick reports 0.805. In G3, deepseek-v3.2 reports 0.838. In G4, llama-4-maverick reports 0.855. In G5, llama-4-maverick reports 0.806 (Ferrag et al., 9 Feb 2026). The published interpretation is that leading models approach high accuracy on intent and policy reasoning, while trust/security and distributed-intelligence reasoning remain materially harder.

Selective robustness analysis via pass@5 shows that some difficult tasks are recoverable under multiple samples even when deterministic pass@1 is imperfect. For example, pass@5 on certain reasoning-intensive tasks reaches approximately 0.91–0.97 for some models, indicating that the correct answer is often present in the sample distribution but not consistently selected under deterministic decoding (Ferrag et al., 9 Feb 2026).

The compact-model scaling study adds a second empirical layer. On 6G-Bench, deterministic accuracy rises from 0.224 for SmolLM2-135M to 0.707 for Qwen2.5-7B, but the increase is highly non-uniform. A pronounced stability transition occurs in the 1 to 1.5B range, where pass@1 rises from 0.373 for Llama-3.2-1B to 0.531 for Qwen2.5-1.5B, and the instability gap $a_t$ 3 contracts from 0.356 to 0.138. Beyond 3B parameters, the gain diminishes to +0.064 from 3B to 7B. On the deployment side, the Edge Score shows that semantic reliability per unit latency and memory does not scale monotonically with parameter count; the reported conclusion is that models in the approximate 1.5 to 3B range offer the most favorable balance between deterministic stability and computational efficiency (Ferrag et al., 2 Mar 2026).

6. Position in the broader 6G benchmarking landscape

6G-Bench is one layer in a broader and still-evolving 6G evaluation ecosystem. The benchmark itself is explicitly focused on semantic communication and network-level reasoning with foundation models (Ferrag et al., 9 Feb 2026). A domain-specific extension is $a_t$ 4-Bench, which evaluates LLM-based UAV agents as multi-turn conversational controllers under dynamic 6G conditions, using slice state, latency, jitter, packet loss, throughput, and edge load as first-class variables (Ferrag et al., 1 Jan 2026). That benchmark operationalizes safety, robustness, and efficiency for UAV autonomy, whereas 6G-Bench abstracts such environments into standardization-aligned reasoning tasks.

The name “6G-Bench” has also appeared in a looser sense in semantic communication research. In the context of GenSC-6G, the term can point to a prototype large-AI semantic communication testbed and public dataset for goal-oriented communication, generative AI integration, and hybrid quantum-classical processing. In that usage, the benchmark object is a modular semantic communication testbed with vehicle imagery, encoded features, and noise-corrupted feature variants under controlled channel conditions rather than a reasoning-centric MCQ benchmark (Arfeto et al., 17 Jan 2025).

A plausible implication is that “6G-Bench” is best understood not as a single monolithic artifact but as an emerging family of benchmark layers: semantic reasoning and policy alignment at the foundation-model level, domain-specific agentic benchmarks under explicit 6G network dynamics, and semantic communication or air-interface testbeds for encoded features, noise, and task-aware decoding. The architecture and slicing literature further suggests that a fuller 6G benchmark suite would eventually need to combine AI-native reasoning, cloud-edge deployment, multi-domain slicing, Network-of-Networks integration, and sustainability or trust metrics rather than treat them in isolation (Liebhart et al., 2024).

The current benchmark retains important limitations. Its question set is derived from UAV-centered source episodes; it is text-only and MCQ-based; it does not use live network traces; and it does not yet cover all possible 6G verticals or continuous-control settings. The stated future directions include adding more verticals, integrating network simulators and real traces, and extending beyond MCQs toward continuous-control evaluation (Ferrag et al., 9 Feb 2026).