V2X-ViT: Transformer-Based V2X Perception

• V2X-ViT is a unified Vision Transformer framework designed for cooperative 3D perception and blockage prediction in connected vehicular networks.
• It employs heterogeneous multi-agent self-attention and multi-scale window attention to mitigate challenges like asynchronous data, pose errors, and sensor heterogeneity.
• The framework achieves state-of-the-art performance by effectively integrating LiDAR, camera, and temporal features through CNN-ViT-GRU architectures for multimodal prediction.

V2X-ViT is a unified Vision Transformer-based framework for Vehicle-to-Everything (V2X) cooperative perception and prediction in connected and autonomous vehicular networks. It addresses the challenges of fusing multi-agent sensor information under real-world constraints—such as asynchronous communication, pose uncertainty, and data heterogeneity—to produce robust, high-fidelity 3D perception or network state prediction outputs. V2X-ViT denotes both the generic multi-agent cooperative perception transformer introduced in "V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer" (Xu et al., 2022), and, in a recent variant, a multimodal CNN-ViT-GRU architecture ("ViT LoS V2X: Vision Transformers for Environment-aware LoS Blockage Prediction for 6G Vehicular Networks" (Gharsallah et al., 2024)). Both systems leverage Vision Transformers as their core component, with architectural adaptations to support either agent-wise LiDAR fusion (for cooperative detection) or temporal multimodal fusion (for mmWave environment-aware blockage prediction).

1. Motivation and Problem Formulation

V2X-ViT targets two major domains within intelligent vehicular networks and wireless communications:

• Cooperative 3D Perception: Standalone perception using single-agent LiDAR is fundamentally limited by sensor field of view, occlusions (e.g., by other vehicles/buildings), and sparsity at long range. V2X cooperation, in which agents (vehicles and infrastructure) share sensor and feature data, can resolve occlusions and extend perception range, but brings unique technical constraints:
  • Asynchrony: Data are captured and transmitted at different times, introducing delay-aware spatial misalignment due to both ego and object motion.
  • Pose errors: GPS/INS-based localization uncertainty (typical std ≈ 0.2 m/0.2°) misaligns spatial registration among collaborators.
  • Heterogeneity: Vehicles and infrastructure have differing sensor extrinsics, observation geometries, and noise characteristics (Xu et al., 2022).
• Environment-aware Network State Prediction: mmWave vehicular networks promise high data rates/low latency but are sensitive to dynamic line-of-sight (LoS) blockages caused by obstacles (vehicles, trees, foliage). 6G architectures will integrate multimodal sensors (cameras, beamforming arrays, LiDAR), requiring precise, temporally robust blockage prediction for proactive handovers and scheduling (Gharsallah et al., 2024).

V2X-ViT, in both forms, provides deep learning-based solutions, using Transformer attention structures for spatial, agent-wise, and temporal fusion.

2. High-level Architectures

(A) Cooperative Perception V2X-ViT (Xu et al., 2022)

1. Metadata & Feature Extraction: Each agent broadcasts pose, calibration, and agent type metadata. Raw LiDAR point clouds are processed by PointPillar to generate a BEV (bird’s-eye-view) pseudo-image feature $\mathbf{F}_i \in \mathbb{R}^{H \times W \times C}$.
2. Communication-Efficient Feature Sharing: Features are channel-compressed ($C\rightarrow C'=8$) using $1\times1$ convolutions, broadcast over DSRC (~27 Mbps), and decompressed on receipt.
3. Ego-Frame Alignment: The Spatial-Temporal Correction Module (STCM) applies affine warping using SE(3) relative transforms to temporally and spatially align features from neighbors.
4. Positional Delay Encoding: Delay-aware positional encoding (DPE) is applied, mapping each agent’s communication lag to additional embedding channels.
5. Unified Transformer-based Fusion:
   • Heterogeneous Multi-agent Self-Attention (HMSA): Relates each agent’s spatial features to those of neighbors using edge/agent-type-specific attention. Inter-agent messages are constructed with edge-type-aware parameterizations.
   • Multi-Scale Window Attention (MSwin): For intra-agent spatial context, attention is computed within multi-scale local windows (window sizes 4, 8, 16).
   • Feedforward MLP and LayerNorm: Standard transformer feedforward processing and normalization.
   • Stacked Layers: Three blocks are stacked; the ego-agent’s updated features are passed to the detection head.
6. Detection Head: Two parallel $1\times1$ convolutional branches predict per-anchor object locations and classifications.

(B) Multimodal Temporal V2X-ViT / ViT LoS V2X (Gharsallah et al., 2024)

1. Multimodal Input: Inputs are $p$-length sequences of:
   • Beam-vector indices (from a 128-codebook mmWave beamformer, past timesteps, embedded to 64D)
   • Triplet RGB images per frame (frontal-left-right, resized to $224\times224$)
2. Feature Extraction:
   • CNN Branch: Four convolutional layers process the beam-vector feature sequence ($p\times64 \rightarrow p\times256$).
   • ViT Branch: Images are split into $16\times16$ patches, projected to 512D, plus positional embedding; passed through six transformer encoder layers (MSA: $h=8$ heads).
3. Fusion & Temporal Modeling: At each time-step, the CNN and ViT embeddings are concatenated ($256+512=768$), forming a $p\times768$ sequence.
4. GRU Sequence Modeling: A two-layer unidirectional GRU (hidden sizes 256, 128; dropout 0.3) models temporal dependencies.
5. Classification Head: Fully connected layer and sigmoid output the blockage probability for a future prediction horizon $f$.

3. Technical Formulations

Attention and Transformer Mechanics (Cooperative Perception)

• HMSA: Each feature vector $\mathbf{H}_i$ is projected to keys/queries/values via agent-type-specific dense layers. Attention energies are modulated by learnable edge-type bias matrices.
• MSwin: Inputs are partitioned into multiple window sizes; attention is local, with relative position bias per window. Outputs are fused via a split-attention module.
• Block Formula:

$$\mathbf{U}^{(\ell)} = \mathrm{HMSA}(\mathrm{LN}(\mathbf{Z}^{(\ell-1)})) + \mathbf{Z}^{(\ell-1)},\; \mathbf{V}^{(\ell)} = \mathrm{MSwin}(\mathrm{LN}(\mathbf{U}^{(\ell)})) + \mathbf{U}^{(\ell)},\; \mathbf{Z}^{(\ell)} = \mathrm{MLP}(\mathrm{LN}(\mathbf{V}^{(\ell)})) + \mathbf{V}^{(\ell)}.$$

• Loss: Weighted sum of smooth-$L_1$ regression ($\mathbf{x},y,z,w,l,h,\theta$) and focal classification losses.

Multimodal Patch and Sequence Embedding (LoS Blockage Prediction)

• Patch Embedding:

$$z_{p}^{(0),i} = x_i E + e_{pos}^i,\quad i=1\ldots N,$$

with $x_i$ the flattened patch, $E$ a learnable projection, $e_{pos}^i$ the positional encoding.

• ViT Encoder:

MSA and residual MLPs, as in original transformer, are applied with:

$$Q = ZW^Q,\quad K = ZW^K,\quad V = ZW^V,\;\; \mathrm{head}_j = \mathrm{softmax}(\frac{Q_j K_j^T}{\sqrt{d_k}}) V_j.$$

• GRU for Temporal Modeling: Standard single-directional GRU recurrence equations applied on the fused feature sequence.
• Loss: Binary cross-entropy for blockage state.

4. Dataset Design and Evaluation Protocols

V2X-ViT for 3D Cooperative Perception

• Dataset: V2XSet, synthesized in CARLA and OpenCDA, with 8 towns, 5 scenario types, including both vehicles and infrastructure, covering 11,447 frames/33,081 agent-frames under realistic localization noise ($\sigma_{pos}=0.2$ m, $\sigma_{head}=0.2^\circ$) and 100 ms communication delays.
• Evaluation Metrics: 3D vehicle detection AP@IoU 0.5/0.7 in the BEV range $x\in[-140,140]$, $y\in[-40,40]$. Performance compared under both perfect and noisy conditions.

ViT LoS V2X for Blockage Prediction

• Dataset: Generated via Wireless InSite ray-tracing on ViWi “ASUDT1” scenario (28 GHz, 128-element ULA, two base stations, three RGB cameras per station, 60 vehicles, 7000 total sequences). Time-ordered train/val/test split: 70%/15%/15%.
• Metrics: Accuracy, precision, recall, F1-score for blockage prediction.

Method (LoS Blockage)	Train Acc	Val Acc	Precision	Recall
Baseline (YOLOv7 + GRU)	0.860	0.729	88.4%	72.3%
ViT LoS V2X (CNN+ViT+GRU)	0.905	0.839	95.7%	85.0%

Method (3D Detection, Noisy)	[email protected]	[email protected]
No Fusion	0.606	0.402
Early Fusion	0.720	0.384
F-Cooper	0.715	0.469
V2VNet	0.791	0.493
DiscoNet	0.798	0.541
V2X-ViT	0.836	0.614

5. Ablations, Robustness, and Comparative Analysis

Cooperative Perception (Xu et al., 2022):

• Component ablations demonstrate the incremental gains:
  • Base MSA: 0.478 [email protected] (noisy); +MSwin: 0.519; +split-attention: 0.548; +HMSA: 0.601; +DPE: 0.614.
• Sensitivity: V2X-ViT degrades <8 pp [email protected] under severe pose error ($\sigma_{xyz}=0.5$ m; $\sigma_{head}=1^\circ$), compared to $>20$ pp drops for other methods. Robust to delays up to 400 ms, outperforming all fusion baselines under increased asynchrony.
• Compression: Retains performance with up to 128$\times$ feature compression ($<$5 pp drop), outperforming all prior methods under bandwidth stress.
• Infrastructure fusion: Including roadside infrastructure (V2X, not just V2V) provides ~5 pp AP gain due to superior viewpoints.

LoS Blockage Prediction (Gharsallah et al., 2024):

• Ablation (Val Acc): CNN+GRU: $\approx$0.78; ViT+GRU: $\approx$0.81; Fused: 0.839.
• Parameter sensitivity: Optimal $p=8$ past frames, $f=3$ prediction horizon. Performance degrades for $f>5$, suggesting current recurrent models' difficulty in extrapolating long-term temporal dynamics given available data.
• Error Profile: Dataset imbalance (fewer non-LoS samples) lowers achievable recall.

6. Significance and Future Research Trajectories

• Unified Multimodal Fusion: Both cooperative and end-to-end multimodal fusion tasks benefit from the Transformer’s ability to model non-local spatial/temporal dependencies, with explicit handling of agent-wise heterogeneity (HMSA), multi-scale spatial patterns (MSwin), and real-world asynchrony/uncertainty (STCM, DPE).
• Implications for Wireless Networks: LoS blockage prediction with multimodal ViT architectures offers a robust, detection-free avenue for proactive management in 6G mmWave vehicular communications (Gharsallah et al., 2024).
• Scalability/Robustness: Attention-based multi-agent fusion supports graceful degradation under noise, severe delays, localization errors, and aggressive feature compression. Benefits are most pronounced up to 4 collaborating agents.
• Generalization: V2X-ViT sets new state-of-the-art in both cooperative 3D detection and LoS prediction, outperforming all prior early/late/intermediate fusion designs.
• Open Challenges: Current limitations include partial modality coverage (LiDAR/camera only), fixed viewpoint/camera setups, and recency-limited temporal prediction. Future work will extend to LiDAR/radar fusion, graph-based cross-cell collaboration, longer-horizon modeling via temporal transformers or TCNs, and online continual adaptation to new domains and weather regimes.

7. Related Architectures and Position within V2X Research

V2X-ViT represents the first unified vision-transformer framework for cooperative V2X perception in the literature (Xu et al., 2022). It augments the basic transformer block with:

• Heterogeneity-aware multi-agent self-attention,
• Multi-scale window attention,
• Bandwidth- and delay-robust communication primitives.

Predecessors such as V2VNet, OPV2V, F-Cooper, and DiscoNet relied on spatial pooling, naïve or single-scale attention, or only early/late fusion. None matched V2X-ViT’s robustness to joint pose/delay/heterogeneity distortions or compression constraints. In blockage prediction, CNN-GRU and "object-then-detect" pipelines (e.g., YOLOv7+GRU) underperform compared to the CNN+ViT+GRU fusion adopted by ViT LoS V2X (Gharsallah et al., 2024).

A plausible implication is that transformer-based intra- and inter-agent context modeling, coupled with explicit handling of wireless and sensor uncertainties, represents a dominant paradigm for forthcoming 6G/cmWave vehicular applications and multi-agent cooperative autonomy.