When Does Multimodal AI Help? Diagnostic Complementarity of Vision-Language Models and CNNs for Spectrum Management in Satellite-Terrestrial Networks

Abstract: The adoption of vision-LLMs (VLMs) for wireless network management is accelerating, yet no systematic understanding exists of where these large foundation models outperform lightweight convolutional neural networks (CNNs) for spectrum-related tasks. This paper presents the first diagnostic comparison of VLMs and CNNs for spectrum heatmap understanding in non-terrestrial network and terrestrial network (NTN-TN) cooperative systems. We introduce SpectrumQA, a benchmark comprising 108K visual question-answer pairs across four granularity levels: scene classification (L1), regional reasoning (L2), spatial localization (L3), and semantic reasoning (L4). Our experiments on three NTN-TN scenarios with a frozen Qwen2-VL-7B and a trained ResNet-18 reveal a clear taskdependent complementarity: CNN achieves 72.9% accuracy at severity classification (L1) and 0.552 IoU at spatial localization (L3), while VLM uniquely enables semantic reasoning (L4) with F1=0.576 using only three in-context examples-a capability fundamentally absent in CNN architectures. Chain-of-thought (CoT) prompting further improves VLM reasoning by 12.6% (F1: 0.209->0.233) while having zero effect on spatial tasks, confirming that the complementarity is rooted in architectural differences rather than prompting limitations. A deterministic task-type router that delegates supervised tasks to CNN and reasoning tasks to VLM achieves a composite score of 0.616, a 39.1% improvement over CNN alone. We further show that VLM representations exhibit stronger cross-scenario robustness, with smaller performance degradation in 5 out of 6 transfer directions. These findings provide actionable guidelines: deploy CNNs for spatial localization and VLMs for semantic spectrum reasoning, rather than treating them as substitutes.

• The paper demonstrates that CNNs excel in supervised spatial tasks, achieving 72.9% scene classification accuracy and superior localization performance.
• The paper shows that VLMs uniquely enable semantic reasoning, with few-shot prompting yielding a 235% F1 improvement over zero-shot performance.
• The paper introduces a deterministic task router that combines both models, resulting in a 39.1% composite performance boost for spectrum management.

Diagnostic Complementarity of Vision-LLMs and CNNs in Spectrum Management

Introduction

This paper provides a rigorous comparative analysis of vision-LLMs (VLMs) and convolutional neural networks (CNNs) for the domain of spectrum heatmap understanding in cooperative satellite-terrestrial networks. The research is grounded in a novel benchmark, SpectrumQA, encompassing a broad range of tasks at four semantic granularity levels crucial for wireless spectrum management. Results demonstrate that VLMs and CNNs possess complementary strengths, challenging common assumptions about the universal superiority of foundation models for operational spectrum analysis. The findings have immediate implications for practical deployment strategies and future architecture development in multimodal AI for telecommunications.

Granularity Framework and Benchmark Construction

Spectrum management tasks are decomposed into four levels:

• L1: Scene Classification – Interference severity categorization.
• L2: Regional Reasoning – Quadrant-level identification of maximum interference.
• L3: Spatial Localization – Pixel-wise interference mask prediction for spatial mapping.
• L4: Semantic Reasoning – Open-ended natural language analysis, contextual explanations, and mitigation recommendations.

The SpectrumQA benchmark features 108K question-answer pairs, generated from a physics-calibrated NTN-TN simulator, spanning three coverage scenarios (dense urban LEO, rural GEO, and mixed configurations). Questions are categorized as descriptive, localization, reasoning, and prescriptive, reflecting realistic operator needs. Each QA pair is strictly mapped to ground truth metadata, ensuring high fidelity in evaluation.

Implementation and Models

• CNN Baseline: ResNet-18, trained on domain-specific images for supervised tasks (L1–L3), equipped with a spatial probe for mask prediction and classification heads for categorical tasks.
• VLM: Qwen2-VL-7B, fully frozen, operating with zero-shot and few-shot prompting protocols for classification, reasoning, and spatial tasks. For spatial tasks (L3), an MLP probe is applied to early visual token embeddings.

Routing logic is empirically derived: supervised spatial tasks are delegated to CNN, semantic reasoning tasks to VLM.

Diagnostic Comparison of Model Performance

A detailed diagnostic comparison reveals strict task-dependent complementarity:

• L1–L3 Supervised Tasks: CNN consistently achieves highest accuracy and spatial localization performance. Specifically, scene classification accuracy reaches 72.9%, regional accuracy 65.7%, and spatial IoU 0.552. VLM, when evaluated for strict 3-class severity, achieves only 0.6%; for binary severity, accuracy reaches 84.6%, indicating calibration discrepancies between numerical thresholds and human judgment.
• L4 Semantic Reasoning: CNN architectures lack language generation capacity; VLM achieves F1 = 0.576 with three in-context examples, reflecting a 235% improvement over zero-shot F1.

  Figure 1: CNN vs. VLM performance across four granularity levels; CNN dominates supervised tasks, while VLM uniquely enables semantic reasoning.

Few-shot prompting unlocks latent reasoning abilities in VLMs, demonstrating the transferability of pre-trained multimodal knowledge to spectrum management.

Figure 2: Few-shot scaling for VLM reasoning; three in-context spectrum analysis examples yield a substantial F1 improvement.

Chain-of-thought (CoT) prompting selectively improves semantic reasoning (L4) by over 10%, but leaves spatial tasks (L2–L3) unaffected, reinforcing that architectural features, not prompting protocol, define capability boundaries.

Task-Type Routing and Composite Performance

A deterministic task-type router leveraging the diagnostic complementarity achieves a composite score of 0.616, a 39.1% improvement over a CNN-only system. Naive assumptions about VLM handling all reasoning-like tasks are empirically refuted; optimal routing maximizes operational performance by combining the strengths of both model families.

Cross-Scenario Robustness

VLMs demonstrate stronger transfer robustness compared to CNNs in cross-scenario spatial localization evaluations. In 5 out of 6 transfer directions (between simulation scenarios), VLMs suffer smaller degradation in IoU performance.

Figure 3: Cross-scenario generalization; VLMs experience smaller performance drops during scenario transfer in most cases.

This suggests pre-trained VLMs encode more generalizable spatial priors, supporting their use in rapidly evolving spectrum environments or new deployments with limited domain-specific data.

VLM Representation Analysis and Ablation

Layer-wise analysis reveals early hidden layers in VLMs preserve spatial information more effectively than later layers, which become increasingly diluted through repeated self-attention fusion geared towards language generation.

Figure 4: Spatial probe IoU and Pearson correlation vs. VLM hidden layer index; Layer 0 achieves superior spatial preservation.

LoRA fine-tuning for text generation demonstrably degrades spatial representations, with a 22.8% drop in spatial IoU, highlighting a fundamental objective conflict in joint spatial-language tasks.

Physics-informed text prompts provided to VLMs exert zero influence on spatial probes extracting embeddings at Layer 0, confirming independence of visual and textual modalities at early representation stages in frozen VLMs.

Qualitative and Cost Analysis

Qualitative outputs from VLM reasoning illustrate operationally valuable analyses—cause identification, spatial awareness, mitigation recommendations—that CNNs cannot produce due to architectural constraints. These outputs, while not numerically precise, are judged as actionable for human operators.

VLM inference is 30× slower and 16× more memory intensive than CNN, underscoring its deployment for infrequent, high-value semantic queries rather than real-time spatial monitoring.

Practical Deployment Guidelines and Limitations

Key practical takeaways:

• Use lightweight CNNs for all spatial and supervised categorization tasks.
• Use frozen VLMs with minimal prompt engineering for semantic spectrum reasoning, particularly during initial assessments in new environments.
• Combine both models via a deterministic router for comprehensive situational awareness.

Limitations include the use of simulated data; real-world validation is required to confirm transferability. Single VLM architecture evaluation further prompts future work on broader model comparisons and adaptive routing policies.

Conclusion

The comparative analysis establishes a formal boundary for model deployment in spectrum management: CNNs excel at spatial categorization and localization, while VLMs uniquely enable semantic reasoning. Complementary deployment through task-type routing is quantitatively and qualitatively superior to monolithic approaches. Future directions include real-world validation, multi-family VLM evaluation, adaptive routing optimization, and effective fine-tuning strategies that preserve spatial and semantic capacity.