WirelessMathBench-XL Benchmark Suite

• WirelessMathBench-XL is a benchmark suite designed to rigorously evaluate mathematical reasoning in wireless communications by curating problems from nearly a thousand peer-reviewed papers.
• It employs diverse task formats—MCQ, PFITB, and FEC—to assess both formula recall and advanced symbolic derivation with machine-verifiable correctness.
• The use of reinforcement learning via GRPO and deterministic rewards demonstrates that compact, domain-specialized models can achieve significant performance gains over generalist LLMs.

WirelessMathBench-XL is a large-scale benchmark suite developed for rigorous evaluation and training of mathematical reasoning in wireless communications. Constructed from nearly a thousand peer-reviewed papers in wireless systems, it covers optimization, information-theoretic and signal processing formulations that reflect the complexity of real-world wireless research. WirelessMathBench-XL is designed to measure and catalyze progress in both LLM mathematical reasoning and wireless-system-specific derivation tasks, supporting development of domain-specialized LLMs.

1. Construction and Scope

WirelessMathBench-XL was built through a semi-automated pipeline that starts with approximately 47,000 arXiv papers across 24 categories including core wireless communications and adjacent fields such as AI/ML. After initial filtering for mathematical density using a relevance scoring function and a secondary GPT-4o-based relevance classifier, the set was narrowed to 970 high-value source papers. Key system equations and variable tables are extracted using a customized LLM (DeepSeek-R1), and then human-curated for physical correctness, symbol/unit consistency, and context appropriateness.

The dataset contains 4,027 problems, each comprising a system scenario, notational definitions, and a formal question in one of three formats:

• Multiple Choice Question (MCQ): Select the physically and dimensionally correct expression from several highly similar alternatives.
• Progressive Fill-In-the-Blank (PFITB): Complete equations with masks that obscure 25–75% of the original content, requiring intermediate steps to be reconstructed from context.
• Full Equation Completion (FEC): Only a problem description and symbol list is provided; the entire core equation must be derived from first principles.

All tasks are written in LaTeX-based markup and demand rigorous symbolic manipulation and knowledge of the underlying systems engineering constraints.

2. Problem Domains and Mathematical Content

WirelessMathBench-XL questions span the breadth of advanced wireless communications research, including:

• Large-scale and extremely large-scale (XL) MIMO systems, with channel modeling that departs from uniform plane wave (UPW) assumptions and addresses near-field, spherical wave, and projected aperture effects.
• Interference management and multi-user systems (e.g., beamforming with distance-dependent decorrelation, inter-user pilot collision, spatial nonstationarity).
• Reconfigurable surfaces (RIS/IRS), including channel estimation and deployment optimization (such as anchor-based estimation, double-sided visibility regions, and element-wise masking).
• Modular array architectures, double-sided nonstationarity, and deployment trade-offs (beamfocusing vs. spatial multiplexing gain).
• Domain-specific mathematical challenges such as maintaining physical and dimensional correctness, handling conjugate and matrix transpose operations (e.g., $\mathbf{H}_\text{BR}^H$), and reconstructing full derivation chains from partial information.
• Equation types are drawn directly from peer-reviewed literature ("no formula invented"), e.g., free-space path loss, signal-to-noise and sum-rate expressions, convex optimization for beamforming, and effective degrees of freedom (EDoF) heuristics. The benchmark’s problems are designed to precisely mirror the forms, notation, and logical steps seen in up-to-date arXiv wireless papers.

3. Evaluation Protocol and Model Performance

The evaluation protocol distinguishes between question types:

• MCQs test formula recall and discrimination under intentionally confusing distractors.
• PFITB tasks assess the model's ability to reconstruct equations with missing terms, sometimes requiring symbolic derivation steps.
• FECs challenge the model to generate the correct equation from description and symbols alone, testing the most advanced domain-compositional reasoning skills.

Performance assessments on WirelessMathBench-XL show a steep drop-off from recall to derivation:

• State-of-the-art LLMs (e.g., DeepSeek-R1 with 671B parameters) achieve 57.4% overall, with only 7.8% accuracy on full equation completion.
• The 7B-parameter WirelessMathLM trained solely via reinforcement learning with group relative policy optimization (GRPO) and deterministic verification rewards achieves 39.5%, near GPT-4o's 40.4%, with large relative improvements at smaller model sizes. Failure cases are dominated by partial fill errors (misplacing a variable or term), misinterpretation of notational symmetries (such as missing Hermitian transpose), and propagation of early mistakes through multi-step derivations.

A distinguishing property of this domain is "verifiable correctness": answers are automatically checked with exact, machine-verifiable criteria for solution acceptance, enabling precise, high-confidence performance measurement and reward computation.

4. Training Paradigm: Reinforcement Learning via Verifiable Rewards

WirelessMathLM models are trained end-to-end with GRPO, without supervised warm-start. For each question, G = 8 samples are drawn, and advantages are computed as group-mean-corrected normalized gaps in correctness. A binary reward is assigned per sample using a two-level verification protocol (format then content). The policy is updated by maximizing the conservative GRPO surrogate objective—critical for providing useful gradients even when the initial model success rate is low.

This reward mechanism leverages the unique verifiability of mathematical problems in wireless systems (absence of ambiguity, presence of explicit ground-truth), circumventing the need for costly or inconsistent human annotation. When trained with this approach, compact models (0.5B to 7B) exhibit large performance gains across all task types and positive transfer to unrelated mathematical reasoning datasets (MATH, OlympiadBench, AMC), indicating a general strengthening of symbolic mathematical skills.

5. Implications for Specialized Reasoning and Wireless Research

WirelessMathBench-XL demonstrates that current generalist LLMs are fundamentally limited on technical mathematical reasoning in specialized domains, with high-profile models plateauing far below expert-level proficiency except on recall-style questions. It also shows that compact models, when trained with domain-specific RL using verifiable rewards, can achieve efficiency and generalization unmatched by parameter scaling alone.

This suggests several future directions:

• Extension of the benchmarking and RL frameworks to other verifiable scientific and engineering domains (e.g., circuits, coding theory, quantum information).
• Use of WirelessMathBench-XL as a standard testbed for specialized LLM development and self-improving reasoning systems in STEM.
• Increased focus on techniques (curriculum, prompt merit functions, chaining intermediate steps) that directly address the core symbolic and dimensional consistency challenges exposed by the dataset.

6. Public Release and Research Utility

WirelessMathBench-XL, as described in the literature (Li et al., 27 Sep 2025), is publicly available with a comprehensive toolkit at https://lixin.ai/WirelessMathBench. The release includes:

• The complete, curated dataset with question, solution, and verification templates.
• Automated scoring utilities for model evaluation on MCQ, fill-in, and completion tasks.
• Scripts and protocols suitable for benchmarking, ablation studies, and RL training loops.

Researchers and developers can use WirelessMathBench-XL to evaluate the reliability and expert-level technical reasoning of LLMs in wireless communications, to refine in-domain pre-training recipes, and to test novel reinforcement learning strategies grounded in deterministic, verifiable objectives. The methodology may serve as a blueprint for similar efforts in neighboring technical fields with explicit symbolic constraints.

---

In summary, WirelessMathBench-XL establishes a rigorous, scalable, and verifiable benchmarking platform for mathematical reasoning in wireless communications, catalyzes advances in compact domain-specialized LLMs, and sets a technical standard for domain-specific AI evaluation and training.