AirTF: Over-the-Air Token Fusion for Task-Oriented Multi-Modal Token Communications

Abstract: In the Internet of Vehicles (IoV), transmitting high-dimensional multi-modal sensory data to edge servers for time-sensitive tasks faces severe spectrum bottlenecks. To address this, we propose a foundation model-driven over-the-air token fusion (AirTF) framework for task-oriented multi-modal token communications. Unlike existing schemes for segmentation that rely on convolutional neural networks (CNNs) with limited local receptive fields, AirTF leverages vision transformer (ViT) encoders to extract globally contextualized semantic tokens from distributed heterogeneous sensors. By concurrently transmitting these spatially aligned tokens over a shared wireless channel, our framework exploits the superposition property of the multiple access channel to inherently fuse complementary multi-modal semantics (e.g., RGB and infrared) directly over the air. This mechanism significantly enhances spectral efficiency compared to orthogonal transmission. Furthermore, the integration of a pre-trained foundation model provides critical visual priors, effectively addressing the data-hungry nature of ViTs on limited, scenario-specific semantic segmentation datasets. Experiments demonstrate that AirTF consistently outperforms orthogonal transmission and CNN-based fusion baselines across AWGN and fading channels. Additional evaluations under a three-user setting, residual synchronization errors, and imperfect channel state information estimation further confirm its robustness. The source code will be made publicly available upon acceptance.

• The paper introduces AirTF, a framework that fuses multi-modal tokens over noisy channels for robust IoV semantic segmentation.
• It employs ViT-based encoders pre-trained with foundation models to achieve higher mIoU even under severe channel impairments.
• Experimental results show AirTF outperforms baselines across AWGN, fading channels, and multi-user settings, ensuring data efficiency and resilience.

AirTF: Over-the-Air Token Fusion for Task-Oriented Multi-Modal Token Communications

Motivation and Context

Conventional approaches to task-oriented communications in the Internet of Vehicles (IoV) rely heavily on resource-intensive transfer of high-dimensional sensory data from distributed heterogeneous sensors such as RGB cameras, infrared (IR) sensors, and LiDAR. Limited vehicular spectrum, strict latency requirements, and collaborative inference demands pose significant bottlenecks, especially for time-critical tasks like semantic segmentation required for autonomous driving. The AirTF framework addresses these constraints by merging semantic communications and over-the-air computation (AirComp), facilitating efficient and robust multi-modal token-level fusion across noisy wireless channels. Unlike previous work using CNN-based local feature extraction, AirTF leverages vision transformers (ViTs) with foundation model-driven pre-training to obtain globally contextualized semantic tokens, enhancing both representation quality and data efficiency.

System Architecture and Token Fusion Mechanism

The AirTF pipeline comprises distributed ViT-based modality-specific encoders, a compression and packing module for token transmission, simultaneous token fusion over the multiple access channel (MAC), and a unified semantic decoder at the edge server. Each device encodes its modality (e.g., RGB, IR) as spatially indexed patch tokens, normalized and mapped to complex channel symbols. Leveraging the MAC's superposition property, tokens from multiple modalities are spatially aligned and analytically fused during wireless transmission. This approach inherently exploits modality complementarity (e.g., color-texture from RGB, heat signature from IR) without resource-intensive orthogonal transmission.

Notably, the ViT encoders are initialized with strong cross-modal priors from foundation models, mitigating the limitations of data scarcity and local receptive field bias. End-to-end task-oriented optimization ensures the system's received token superpositions are robust to channel corruption and preserve scene-level semantics essential for downstream segmentation.

Figure 1: Visual comparison of multi-modal segmentation on SemanticRT under low SNR and high compression, with AirTF outperforming baselines in object boundary and completeness preservation.

Experimental Results: Quantitative Analysis

AirTF demonstrates consistent superiority on the SemanticRT and PST900 datasets in both classical AWGN and severe fading channels. At SNR = 0 dB and $C=1/256$, AirTF achieves an 85.78% mean Intersection over Union (mIoU), outpacing TokenCom-OMA (84.17%) and MFNet (83.71%). Bandwidth ablation shows AirTF maintains its advantage even as compression rate increases ($C=1/512$), with orthogonal schemes suffering stronger degradation.

Multi-user scenarios with three modalities (RGB, thermal, depth) confirm generalizability: AirTF yields 79.33% mIoU at SNR = 5 dB ($C=1/256$), compared to significant drops in baselines. Fading channel tests reveal resilience: Rayleigh and Rician experiments show AirTF maintains higher mIoU than TokenCom-OMA and MFNet, with performance gains amplified under bandwidth constraints.

Robustness and Ablation Studies

Robustness evaluation covers residual synchronization errors, impaired channel state information (CSI), and spatial misalignment. AirTF retains stability under moderate phase/timing offsets and imperfect CSI, outperforming MFNet even under uncalibrated conditions. Spatial perturbations (translation/rotation) confirm robustness to small misalignments.

Encoder initialization ablation underscores foundation model utility: ImageNet-based pre-training delivers a 4.2% absolute boost in mIoU versus random initialization, with joint pre-training across modalities maximizing performance.

Qualitative Insights: Semantic Fusion and Attention Visualization

AirTF's token-level fusion mechanism is visually validated through comprehensive segmentation maps and self-attention overlays. Object boundaries, fine details, and semantic consistency with ground truth substantially exceed CNN and orthogonal baseline outputs under adverse channel conditions. Attention map analysis substantiates modality complementarity: RGB tokens concentrate on texture-dense, illuminated objects, while IR focuses on heat-emitting targets invisible to RGB, culminating in fused representations capturing richer semantics.

Figure 2: Visualization of self-attention, with AirTF's RGB and IR branches targeting complementary regions, enabling robust fusion of visual and thermal cues.

Computational Complexity and Latency

AirTF incurs elevated computational cost due to ViT-based encoding/decoding (756.69 GFLOPs total, 16.70 ms latency per sample), but latency is dominated by semantic encoding on edge devices rather than wireless transmission. Communication overhead at $C=1/256$ is negligible (<0.3 ms), confirming practical feasibility for real-time vehicular perception.

Practical and Theoretical Implications

AirTF advances the paradigm of semantic-aware over-the-air fusion, reducing bandwidth consumption while maximizing semantic utility for collaborative inference. It establishes the viability of ViT-based foundation models for edge-driven multi-modal semantic transmission, overcoming both the data-efficient training and robustness bottlenecks of prior CNN-centric methods. The framework is extensible to other perception tasks and additional modalities, with token selection/adaptation offering further optimization potential for cross-modal transmission.

Conclusion

AirTF exemplifies task-oriented, foundation model-driven over-the-air token fusion for robust multi-modal semantic communication in bandwidth-constrained IoV environments. The explicit exploitation of transformer-based global context, combined with simultaneous superposed transmission and end-to-end task training, delivers quantifiable gains in semantic segmentation accuracy and resilience. Future directions include broader perception task coverage and adaptive token selection, reinforcing AirTF's position as a scalable solution for collaborative AI in autonomous and connected vehicular networks (2607.03099).