WirelessBench: Benchmark for LLM Wireless Agents

• WirelessBench is a comprehensive benchmark that evaluates LLM agents in wireless network management, emphasizing tolerance-aware scoring, tool integration, and cascading error detection.
• It is organized into a three-tier cognitive hierarchy covering domain knowledge reasoning, intent-driven resource allocation, and proactive multi-step decision-making under mobility.
• The benchmark addresses operational risks by penalizing unit and magnitude mismatches, enabling failure localization through detailed chain-of-thought traceability.

WirelessBench is a benchmark for evaluating LLM agents for wireless network intelligence in a way that reflects real engineering risk rather than abstract question-answering ability. It is presented as the first tolerance-aware, tool-integrated benchmark for LLM-based wireless agents, and is organized as a three-tier cognitive hierarchy spanning domain knowledge reasoning, intent-driven resource allocation, and proactive multi-step decisions under mobility. Its central premise is that autonomous wireless network management depends on the composition of telecom knowledge, external tool use, structured decision-making, and chained reasoning, and that benchmarks which evaluate those capabilities in isolation miss deployment-relevant failure modes such as cascaded-chain failures and catastrophic unit confusions (Tong et al., 22 Mar 2026).

1. Motivation and problem gap

WirelessBench was introduced against the background of prior telecom and wireless benchmarks that probe useful capabilities such as knowledge recall, math reasoning, semantic reasoning, or network-level understanding, but largely evaluate these capabilities in isolation. The benchmark’s argument is that wireless failures in practice are rarely single-step failures. They are usually cascaded-chain failures: an agent misreads intent, misses a tool call, propagates a wrong CQI, allocates the wrong bandwidth, and then incorrectly certifies QoS feasibility. A benchmark that scores each isolated step independently misses this deployment-relevant failure mode.

A second motivation is the misalignment between conventional exact-match metrics and wireless engineering. WirelessBench treats small throughput deviations caused by rounding as potentially operationally irrelevant, while confusions such as dB vs. dBm or kbps vs. Mbps are treated as operationally dangerous. The paper emphasizes examples such as confusing dBW and dBm, which can cause a $1000\times$ power misestimate. In this framing, a model that is “almost right” numerically should not be ranked the same way as one that is dangerously wrong in units or scale.

The benchmark therefore addresses a broader evaluation gap than telecom fact recall. It asks whether an agent can autonomously manage parts of a wireless network workflow: interpret free-text service requests, call the right external tools, integrate returned values, reason over structured state, and make sequential resource-management decisions. This suggests a benchmark philosophy in which engineering tolerance, catastrophic failure modes, tool dependence, and reasoning-chain traceability are first-class evaluation objects rather than afterthoughts.

2. Design principles and risk-sensitive scoring

WirelessBench is built around three design principles: tolerance-aware scoring with catastrophic-error detection, tool-necessary evaluation, and Chain-of-Thought-traceable items.

The tolerance-aware component begins with the relative error

$\epsilon = \frac{|\hat{y}-y|}{\max(|y|,\delta)},$

where $\hat{y}$ is the prediction, $y$ is the reference, and $\delta$ is a small constant to avoid division by zero. Numeric credit is then assigned by tiers: $$s_{\mathrm{num}}(\epsilon)= \begin{cases} 1.0, & \epsilon \le 0.01,\ 0.9, & 0.01 < \epsilon \le 0.05,\ 0.7, & 0.05 < \epsilon \le 0.10,\ 0.0, & \epsilon > 0.10. \end{cases}$$ Predictions within $1\%$ relative error receive full credit; those within $5\%$ still receive high credit; those within $10\%$ receive reduced credit; and anything beyond $10\%$ gets zero. Before this numeric credit is applied, WirelessBench separately checks for catastrophic errors. Two categories are automatically zeroed regardless of numeric proximity: unit mismatch and order-of-magnitude mismatch. A prediction like “30 dBm” when the answer is “30 dB” gets zero, and a scientific-notation exponent mismatch of at least one order, such as $\epsilon = \frac{|\hat{y}-y|}{\max(|y|,\delta)},$0 versus $\epsilon = \frac{|\hat{y}-y|}{\max(|y|,\delta)},$1, is treated as catastrophic.

The tool-necessary component makes the benchmark explicitly agentic. In Tier 2 and Tier 3, correct answers require invocation of an external radio model through a 3GPP-compliant ray-tracing query: $\delta$0 This returns a deterministic CQI in $\epsilon = \frac{|\hat{y}-y|}{\max(|y|,\delta)},$2. In WCNS the agent must query the user’s current location; in WCMSA it must first predict a future position and then query at that predicted position. The prompt does not contain the relevant CQI, so the benchmark is designed so that an agent that skips the tool must hallucinate a CQI and will usually fail.

The Chain-of-Thought-traceable component means that every benchmark item ships with a complete Chain-of-Thought trajectory. In WCHW the trajectory is a linear chain $\epsilon = \frac{|\hat{y}-y|}{\max(|y|,\delta)},$3. In WCNS it becomes $\epsilon = \frac{|\hat{y}-y|}{\max(|y|,\delta)},$4. In WCMSA it branches as $\epsilon = \frac{|\hat{y}-y|}{\max(|y|,\delta)},$5. The benchmark thereby supports diagnosis of where in the reasoning chain an agent fails, rather than only whether a final answer is wrong.

3. Three-tier cognitive hierarchy

WirelessBench is organized as a three-tier cognitive hierarchy intended to mirror increasing levels of competence needed in wireless network automation.

Tier	Benchmark	Role and size
1	WCHW	Domain knowledge reasoning, 1,392 items
2	WCNS	Intent-driven resource allocation, 1,000 items
3	WCMSA	Proactive multi-step decisions under mobility, 1,000 items

WCHW (“Wireless Communication Homework”) evaluates whether a model can correctly apply wireless fundamentals such as Shannon capacity, BER computation, modulation, coding, SNR conversion, fading analysis, and related communication-theory concepts. The tasks span nine categories: modulation/demodulation, digital communication, analog communication, information theory, wireless channels, noise analysis, multiplexing, multiple access/cellular systems, and error-control coding. Inputs are textbook-style questions; outputs may be numeric values with units, formulas, scientific notation, or short technical text. The benchmark’s position is that if an agent fails here, it cannot be trusted with downstream operational decisions.

WCNS (“Wireless Communication Network Slicing”) evaluates intent-driven resource allocation. Each item provides network state, user position, and a free-text service request. The agent must infer service intent, call the ray-tracing tool to obtain CQI, and produce a four-field structured output: slice type, CQI, bandwidth, and throughput. Bandwidth is computed under a proportional-fairness rule and throughput is derived via the 3GPP CQI-to-spectral-efficiency table.

WCMSA (“Wireless Communication Mobile Service Assurance”) is the most advanced tier. Each item contains a short historical trajectory, base-station parameters, network state, a service request with minimum-rate QoS requirement, and latency sensitivity. The agent must predict the user’s future position, query ray tracing at that predicted position, classify service intent, allocate bandwidth, compute throughput, and determine whether QoS will be satisfied. The output has six fields: predicted position, CQI, slice type, bandwidth, throughput, and QoS feasibility. This tier makes explicit the benchmark’s chain-based philosophy: decisions are sequential and interdependent, and an early mistake propagates downstream.

4. Physical environment, structured evaluation, and task-specific metrics

The benchmark’s tool integration is grounded in a campus-scale digital twin built from OpenStreetMap data of the HKUST campus. Ray-tracing results are precomputed over three regions—North, Center, and South—using a propagation setup conforming to 3GPP TR 38.901 Urban Micro assumptions. The deployment uses a 3.5 GHz n78 configuration with 30 kHz subcarrier spacing. For each location, the environment stores received power, path loss, SNR, LOS status, and the derived CQI obtained through the SNR-to-CQI mapping from 3GPP TS 38.214 Table 5.2.2.1-3.

Throughput is defined as

$\epsilon = \frac{|\hat{y}-y|}{\max(|y|,\delta)},$6

where $\epsilon = \frac{|\hat{y}-y|}{\max(|y|,\delta)},$7 is bandwidth in MHz and $\epsilon = \frac{|\hat{y}-y|}{\max(|y|,\delta)},$8 is the spectral efficiency from the 3GPP table. In WCMSA, mobility prediction uses a Kalman filter with state vector $\epsilon = \frac{|\hat{y}-y|}{\max(|y|,\delta)},$9, process noise $\hat{y}$0 m, and measurement noise $\hat{y}$1 m.

WCNS uses a weighted composite score

$\hat{y}$2

with default weights $\hat{y}$3. WCMSA uses

$\hat{y}$4

with $\hat{y}$5 assigned respectively to position, CQI, slice type, bandwidth, throughput, and QoS verification. Position is scored continuously by distance: $\hat{y}$6 where $\hat{y}$7 is Euclidean distance and $\hat{y}$8 m.

The evaluation stack is more elaborate than exact match. Numeric answers are parsed with unit normalization into base SI units. Supported conversion families include frequency, linear power, logarithmic power, data rate, and distance. Formula answers are normalized and checked using structural similarity and symbolic equivalence. Text answers are scored by weighted keyword overlap and semantic consistency rather than exact lexical identity. Structured outputs are decomposed into fields and each field is scored with a type-specific evaluator before recombination through the weighted formulas above.

5. Diagnostic value and empirical findings

WirelessBench’s diagnostic contribution lies not only in final scores but in failure localization. The paper organizes observed failures into four actionable diagnostic categories: formula misapplication, reasoning-path break, unit/magnitude confusion, and arithmetic error. Formula misapplication includes selecting the wrong equation or applying it to the wrong scale or variables. Reasoning-path break refers to cases where some intermediate steps are correct but the chain fails at a junction, such as computing bandwidth correctly and then using it directly as throughput. Unit/magnitude confusion includes errors such as dB vs. dBm, MHz vs. Hz, or mW vs. W. Arithmetic error denotes numerically incorrect computation despite a conceptually correct path.

Empirically, the benchmark reports that the direct-prompting model GPT-4o scores $\hat{y}$9, whereas the tool-integrated reference agent WirelessBench-Ref scores $y$0, a 16.64 percentage-point gap. Under exact match, GPT-4o falls from $y$1 to $y$2, a drop of 16.18 pp, while WirelessBench-Ref falls from $y$3 to $y$4, a drop of 5.59 pp. The benchmark also reports that 23% of errors are catastrophic failures invisible to exact-match metrics (Tong et al., 22 Mar 2026).

For GPT-4o direct prompting, the failure-mode breakdown is reported as 31% formula misapplication, 28% reasoning-path break, 23% unit/magnitude confusion, and 18% arithmetic error. The distribution varies by tier: formula misapplication dominates WCHW, while reasoning-path breaks dominate WCNS and WCMSA, where dependencies and tool interactions matter more. The paper further notes that catastrophic errors are “nearly eliminated” in tool-using agents, indicating that external verification and deterministic tools act as safeguards.

6. Construction, reproducibility, limitations, and subsequent use

WirelessBench contains 3,392 items in total, with fixed validation/test splits: WCHW 348/1,044, WCNS 250/750, and WCMSA 250/750. Its construction pipeline combines seed data collection, psychometric-inspired cleaning, knowledge-anchored augmentation, and human validation. Seeds come from textbooks, research papers, and 3GPP specifications. Cleaning uses multi-model consensus and a hierarchical grading pipeline, followed by psychometric filters such as item-total correlation, a Mokken-style proxy, and inter-item consistency. Suspicious items are audited by a stronger reasoning model and reviewed by humans.

Human validation was performed on 150 stratified samples with three graduate annotators, yielding 96% question clarity, 96% answer correctness, 92.7% CoT validity, and inter-annotator agreement around $y$5. The released artifacts include the dataset in JSONL, scoring scripts, prompt templates, ray-tracing data, Kalman predictor, experiment logs, and statistical-analysis scripts. The project resources are listed as https://wirelessbench.github.io/ and https://github.com/jwentong/WirelessBench.

The benchmark is not presented as complete. The digital twin covers only one campus environment—HKUST at 3.5 GHz under a 3GPP UMi model—so absolute performance may not transfer directly to other deployment conditions. The baseline set is limited and includes mostly closed-source models. The scoring thresholds and field weights are engineering heuristics rather than standards-derived quantities. A large fraction of the dataset is LLM-augmented, which raises potential concerns about style leakage and diversity. WCMSA uses a constant-velocity Kalman model, and the benchmark remains focused on a single-agent offline setting rather than multi-agent, streaming, or online-control scenarios.

Subsequent work has already treated WirelessBench as an optimization substrate rather than only a leaderboard. “WirelessAgent++: Automated Agentic Workflow Design and Benchmarking for Wireless Networks” (Tong et al., 28 Feb 2026) uses WirelessBench as a standardized multi-dimensional benchmark suite comprising WCHW, WCNS, and WCMSA, and reports test scores of $y$6, $y$7, and $y$8 respectively, with a total search cost below $y$9 per task. This suggests that WirelessBench has become not only an evaluation set for wireless LLM agents, but also a search objective for automated workflow design.