V2XFormer: Transformer Multi-Agent Fusion

• V2XFormer is a transformer-based multi-agent fusion model for autonomous driving that enhances motion prediction, accident anticipation, and 3D object detection.
• It employs a modular architecture with per-agent BEV feature extraction, spatial alignment, and sparse transformer attention to integrate data from vehicles and infrastructure.
• Empirical evaluation on the DeepAccident dataset demonstrates significant performance gains, especially in occlusion-heavy urban scenarios with multi-agent collaboration.

V2XFormer is a baseline model proposed for V2X (vehicle-to-everything) autonomous driving, demonstrating high performance in motion and accident prediction as well as 3D object detection by leveraging transformer-based multi-agent fusion. Designed and evaluated on the DeepAccident dataset, V2XFormer fuses information from up to four vehicles and optionally an infrastructure agent to address the challenges of perception and prediction under partial observability, particularly in occlusion-heavy urban scenarios (Wang et al., 2023).

1. Architectural Overview

V2XFormer adopts a modular architecture with several key components:

• Per-agent feature extraction: Each participating agent (vehicles and infrastructure) processes its own camera streams using a shared Swin-Transformer (Tiny) backbone applied to the most recent $N = 3$ frames, capturing a total of 1 second of temporal history at 10 Hz. The extracted features are projected (or "lifted") into a bird's-eye-view (BEV) representation using the BEV-feature head from BEVerse, producing features $F^a \in \mathbb{R}^{C \times H \times W}$ with $H = W \approx 128$.
• Spatial Alignment: Each agent’s BEV map is transformed using the relative 6-DoF pose to align all features to the ego-vehicle's coordinate frame. The aligned BEV features are concatenated along the channel axis.
• Cross-agent (V2X) fusion: Four candidate fusion modules are considered (Average-pooling, DiscoNet, V2X-ViT, and CoBEVT), with the final V2XFormer employing the CoBEVT module. CoBEVT integrates the multi-agent BEV features using a sparse transformer architecture that alternates between self-attention within an agent and cross-attention between agents.
• Task-specific heads:
  • 3D object detection head: Outputs both classification (including centerness) and bounding box regression (2D position offsets, object height, extent, and orientation).
  • End-to-end motion prediction head: Predicts BEV segmentations for each future instance, future center offsets, future dense displacement fields ("flow"), and centerness for observed past segments. The ego-vehicle’s own prospective trajectory is jointly predicted within this structure.

2. Mathematical and Algorithmic Formulation

V2XFormer fuses BEV features via transformer-style attention mechanisms:

• Cross-agent attention: For a stack of $M$ agent-aligned BEV features $$X = [F^1; F^2; \ldots; F^M] \in \mathbb{R}^{MHW \times C}$$, each transformer block computes queries, keys, and values as

$Q = X W_Q,\quad K = X W_K, \quad V = X W_V$

Attention is then calculated as

$$\mathrm{Attention}(X) = \mathrm{softmax}\left( \frac{Q K^\top}{\sqrt{d_k}} \right) V$$

Each CoBEVT block alternates between sparse self-attention (within each agent's set of patch tokens) and cross-agent attention, as defined in the original CoBEVT work.

• Temporal aggregation: V2XFormer does not employ explicit recurrence; instead, the most recent $N$ BEV features for each agent are concatenated along the channel dimension to form the spatio-temporal representation.
• Loss functions: The total training loss combines detection and motion prediction objectives, matching BEVerse/FIERY:

$L_\text{total} = L_\text{det} + L_\text{motion}$

where $L_\text{det}$ consists of classification, bounding box regression, and centerness losses; $F^a \in \mathbb{R}^{C \times H \times W}$0 sums segmentation, flow, offset, and centerness (for motion) losses. Standard binary cross-entropy is used for segmentation and centerness, and $F^a \in \mathbb{R}^{C \times H \times W}$1 loss for regression and flow.

3. Training Regimen and Dataset Parameters

V2XFormer is trained on the DeepAccident dataset, which provides 57,000 V2X frames (285,000 total samples), following a 70/15/15 split: 203,000 for training, 41,000 each for validation and testing. Input BEV ranges and grid sizes are set as follows:

• Detection: $F^a \in \mathbb{R}^{C \times H \times W}$2 with 0.8 m grid size, totaling $F^a \in \mathbb{R}^{C \times H \times W}$3 cells.
• Motion: $F^a \in \mathbb{R}^{C \times H \times W}$4 with 0.5 m grid size, totaling $F^a \in \mathbb{R}^{C \times H \times W}$5 cells.

The temporal window is a 1 s history sampled at 10 Hz, predicting 2 s into the future with 4 frames at 2 Hz. Optimization settings (Adam with warmup and polynomial decay) and hyperparameters follow those from BEVerse/FIERY. Batch size and learning rate are not restated; training comprises 20 epochs over the training split.

4. Empirical Performance and Ablation Results

V2XFormer is evaluated on a comprehensive battery of tasks, using metrics such as mIOU and VPQ for motion prediction, APA (Accident Prediction Accuracy) and associated errors for accident prediction, and mAP for 3D detection. The APA is defined as: $F^a \in \mathbb{R}^{C \times H \times W}$6 where $F^a \in \mathbb{R}^{C \times H \times W}$7 indexes different distance thresholds.

Fusion module comparison (5-agent setup):

Fusion Module	mIOU	VPQ	APA	mAP
Average-pooling	52.1	39.5	67.1	36.2
DiscoNet	54.2	42.0	68.9	38.5
V2X-ViT	55.1	43.2	69.1	40.1
CoBEVT (final)	56.2	44.0	69.5	40.8

Agent configuration ablation:

Configuration	mIOU	VPQ	APA	IDerr	Pos err (m)	mAP
Single-vehicle	43.8	31.6	61.9	0.12	3.20	26.5
4 vehicles + infra	56.2	44.0	69.5	0.06	2.45	40.8

In cases where the accident agent is occluded from the ego-vehicle for over 50% of the history, V2XFormer demonstrates substantial gains (mIOU improved from 39.6 to 55.5, APA from 57.3 to 69.5). Longer horizon prediction remains challenging: single-vehicle APA drops from 61.9 (2 s) to 35.4 (4 s).

Robustness experiments indicate that performance degrades less in the presence of pose noise (Gaussian error on position) or communication delays when more V2X agents are present. Sim2Real adaptation, combining DeepAccident and nuScenes data, yields marginal gains in VPQ and mAP relative to nuScenes-only training.

5. Significance and Insights

V2XFormer demonstrates that multi-agent fusion via transformer-based cross-agent attention is effective for both accident prediction and conventional 3D detection under challenging occlusions. The contribution of infrastructure agents (high-mounted sensors) is notably higher than that of follow-vehicle agents. Performance improves monotonically with additional agents, with greater robustness to pose inaccuracies and transmission delays.

A key insight is that explicit fusion via sparse transformers, as implemented in CoBEVT, is superior to both mean-pooling and learned graph-based approaches (DiscoNet), as well as to vanilla Vision Transformers adapted for V2X (V2X-ViT). This confirms the importance of capturing heterogeneous viewpoint relationships in dynamic driving environments.

6. Connections and Context

V2XFormer builds upon several architectural innovations from BEVerse (for BEV lifting and detection), FIERY (for motion prediction head design), and CoBEVT (for sparse transformer-based fusion). The model is evaluated on the DeepAccident dataset, which is substantially larger and more diverse in accident scenarios compared to nuScenes, enabling direct and explainable safety evaluation for autonomous driving (Wang et al., 2023).

The adoption of transformer-based V2X fusion evidences a critical transition within autonomous driving research away from single-view or naive feature combination, towards explicit, learned, attention-based mechanisms that model the spatial and temporal inter-agent dependencies essential for robust scene understanding and accident anticipation.

7. Limitations and Future Directions

Certain model and training hyperparameters (such as BEV feature channel dimensionality, transformer depth, and learning rate schedules) are inherited from prior work and not fully specified in the documentation. The approach does not deploy explicit recurrence for temporal history, instead concatenating past BEV feature tensors along the channel axis. Longer-horizon motion prediction remains a difficult challenge, as performance degrades significantly over 4-second prediction horizons in single-agent settings.

A plausible implication is that expanding the density of agent participation and incorporating more advanced temporal modeling could further enhance accident prediction capabilities and robustness under real-world operational noise. The model architecture and dataset enable broad evaluation of V2X fusion strategies, offering a reference point for subsequent research into scalable, safety-critical autonomous driving systems (Wang et al., 2023).