V2X Cooperative Perception

• V2X cooperative perception is the process in which connected vehicles and roadside units share and fuse sensor data to create a unified scene model extending beyond individual sensor limits.
• It employs diverse fusion methods—early, intermediate, and late—to find a practical balance between communication bandwidth and information richness for improved detection and forecasting.
• Real-world implementations highlight that low-latency, reliable communication and precise synchronization are critical for mitigating occlusions and ensuring safe autonomous driving.

Vehicle-to-Everything (V2X) cooperative perception is the process by which multiple V2X agents—most commonly connected vehicles and roadside infrastructure—share, align, and fuse observational data to extend situational awareness beyond the ego vehicle’s field of view and sensor limits. Its immediate purpose is to overcome occlusions, limited field of view, sparse long-range perception, and field-of-view fragmentation by transforming isolated observations into a collective scene model, typically through a workflow of selection, communication, alignment, fusion, and decoding (Huang et al., 2023, Song et al., 30 Apr 2025). In recent work, cooperative perception is also framed as an observation-centric middle layer in a broader Sensor–Perception–Decision loop, where the fused scene representation becomes forecast-ready and decision-relevant rather than remaining a stand-alone detection output (Zhang et al., 29 Nov 2025, Zhou et al., 2024).

1. Problem formulation and operational scope

The canonical V2X cooperative perception setting assumes an ego vehicle whose onboard sensing is degraded by occlusion, range limits, or viewpoint restrictions, while other connected vehicles or infrastructure units observe complementary portions of the same scene. Cooperative perception therefore targets the recovery of blind spots, extension of perception range, and improvement of 3D object detection under multi-agent sensing (Huang et al., 2023, Xiang et al., 2024). In the cooperative driving formulation emphasized by one line of work, the driving policy is written as

$$\pi(a_t \vert O_{t\_ego}, O_{t\_1}, \ldots, O_{t\_{N_t}})$$

so that the control action depends on the ego observation together with observations from neighboring vehicles (Li et al., 2023).

A recurring theme is that the technical problem is not limited to fusing information from multiple viewpoints. V2X cooperative perception also requires deciding whether cooperation is needed, which collaborators are worth querying, what information should be transmitted, how it should be aligned to a common reference frame, and whether the communication substrate is reliable enough for the fused result to remain actionable (Huang et al., 2023, Li et al., 2023). This broader systems framing is central to later work on online deployment, communication-aware fusion, and prediction-oriented extensions (Xiang et al., 13 Mar 2025, Zhou et al., 2024).

The scope of the field has expanded from single-frame cooperative 3D detection toward temporal perception and motion forecasting. V2XPnP explicitly argues that V2X systems should not be viewed as a single-frame, detection-only problem, because autonomous driving requires both current scene understanding and future motion prediction, and because objects may disappear and reappear over time due to occlusion or range limits (Zhou et al., 2024). A related conceptual framework, Collaborative Joint Perception and Prediction, similarly places collaborative scene completion upstream of joint perception and prediction to reduce cumulative perception errors and mitigate occlusion-driven failures in forecasting (Wan et al., 27 Jan 2025).

2. Information representation and fusion regimes

The dominant taxonomy of cooperative perception distinguishes early fusion, intermediate fusion, and late fusion. Early fusion transmits raw sensor data before encoding; late fusion transmits final detections or object-level outputs; intermediate fusion transmits latent features, usually in BEV form, after local encoding and before task heads (Huang et al., 2023, Song et al., 30 Apr 2025). Surveys consistently describe early fusion as the most information-rich but the most communication-heavy, late fusion as the lowest-bandwidth but most information-lossy, and intermediate fusion as the practical compromise between semantic richness and bandwidth efficiency (Huang et al., 2023, Zhang et al., 29 Nov 2025).

Real online measurements make this trade-off explicit. In V2X-ReaLO, detection outputs are reported as 3.63 KB, 10.0 ms, LiDAR points as 3.7 MB, 144.2 ms, and intermediate features (32×) as 840 KB, 41.7 ms; the system selects 32× compression as a practical balance between latency and fidelity (Xiang et al., 13 Mar 2025). The same framework demonstrates a unified ROS pipeline for Early Fusion, Late Fusion, and Intermediate Fusion, with MarkerArray used for boxes, PointCloud2 for raw LiDAR, and UInt8MultiArray bev for compressed BEV features (Xiang et al., 13 Mar 2025). This makes online intermediate fusion a systems problem involving GPU–host transfer, serialization, network transmission, deserialization, and decompression rather than merely a modeling choice (Xiang et al., 13 Mar 2025).

Although BEV features have become the standard intermediate representation, they are not the only proposed message unit. V2X-PC argues that dense BEV collaboration is affected by object feature destruction during packing, inefficient message aggregation for long-range collaboration, and implicit structure representation communication, and introduces the point cluster as a sparse object-centric message unit carrying point coordinates, a cluster center, and a cluster feature (Liu et al., 2024). The communication packet becomes

$\mathcal{M}^i=(\bm{M}^i,\xi^i),$

where $\bm{M}^i$ contains the positive point clusters and $\xi^i$ is the sender pose (Liu et al., 2024). This object-centric reformulation is explicitly intended to make aggregation depend on object count rather than communication range (Liu et al., 2024).

Another practical alternative is “late-early collaboration,” in which agents broadcast detections rather than raw point clouds, while the ego vehicle fuses the received detections with its own raw point cloud sequence before final detection. On V2X-Sim, this framework consumes 0.01 MB on average, the same bandwidth as late collaboration, and the paper states that its performance saturates at 98.2% of Early Collaboration in sync setting as the number of agents increases (Dao et al., 2023). A plausible implication is that message representation in V2X cooperative perception is now an active design dimension in its own right, rather than a byproduct of network architecture.

3. Communication, synchronization, and non-ideal operating conditions

A central systems result is that cooperative sensing is only useful if the communication link carrying sensor data is reliable enough and available often enough. “NLOS Dies Twice” makes this argument explicitly: non-line-of-sight harms perception once by creating blind zones in sensing and again by blocking the V2X links needed to share complementary sensor data (Li et al., 2023). The paper notes that sensor fusion can require roughly 2–10 Mbps, that cooperative perception needs sufficient packet reception rate or low packet error rate, and that poor V2X QoS can make the end-to-end driving system fail even when the perception model itself is strong (Li et al., 2023). From the information-sensor viewpoint, this aligns with the observation that real-world C-V2X throughput in cooperative scenarios can be less than 10 Mbps, making raw data exchange fundamentally difficult (Song et al., 30 Apr 2025).

To address communication unreliability under blockage, the same work proposes two mechanisms. Abstract Perception Matrix Matching (APMM) converts lidar point clouds into a 2D abstract perception matrix, enabling compact blind-zone matching; a 20 × 20 matrix can be broadcast in about 1600 bytes together with sender location (Li et al., 2023). Mobility-Height Hybrid Relay Determination (MoHeD) then scores direct links and relay candidates using a mobility similarity factor

$$S_{i} = \frac {1}{|v_{node} - v_{obstacle\_i}|} + \frac {1}{|v_{ego} - v_{obstacle\_i}|}$$

and total NLOS risk

$V_{NLOS} = \sum _{i=1}^{n} L_{v-shadow\_i}S_{i}.$

In the reported scenario, mobility-based relay selection achieves average packet reception rate: 87.98%, compared with 46.59% for signal-strength-based selection, 51.48% for random selection, and 39.22% for direct link, while reducing relay switching to 2.75 times versus 3.34 times and 3.65 times for signal-strength-based and random selection (Li et al., 2023).

Temporal asynchrony and pose error are equally central. V2X-ViT addresses these with a Spatial-Temporal Correction Module (STCM) for feature warping and a Delay-aware Positional Encoding (DPE) that injects message age into the Transformer pipeline (Xu et al., 2022). Practical Collaborative Perception relaxes strict synchronization by assuming only a common time reference, practically achievable via GPS time, and propagates stale detections by scene flow before transforming them into the ego frame (Dao et al., 2023). V2X-ReaLO realizes these requirements in a deployed system by synchronizing host clocks to GPS time, phase-locking LiDAR sensors with GPS signals, and buffering poses and timestamps so that collaborator data can be retrieved within a time window around the current timestamp (Xiang et al., 13 Mar 2025).

These mechanisms reflect a broader consensus in the literature: cooperative perception is not robust by default to latency, localization error, or packet loss; these factors must be explicitly represented in the model or the communication policy (Huang et al., 2023, Song et al., 30 Apr 2025).

4. Model architectures and learning paradigms

Architecturally, the field has moved from simple pooling and graph message passing toward heterogeneity-aware attention, spatio-temporal fusion, distillation, self-supervised pretraining, and domain generalization. V2X-ViT is a representative intermediate-fusion design built around alternating layers of heterogeneous multi-agent self-attention and multi-scale window self-attention to model inter-agent interaction and per-agent spatial relationships under asynchrony, pose errors, and heterogeneity (Xu et al., 2022). On V2XSet it reports 0.882 / 0.712 AP at IoU 0.5 / 0.7 in the perfect setting and 0.836 / 0.614 in the noisy setting, outperforming the compared baselines in both cases (Xu et al., 2022).

V2XPnP extends the architectural scope from single-frame perception to end-to-end perception and prediction. Its preferred design is intermediate fusion within one-step communication, where each agent compresses historical BEV features

$$\mathbf{F}_{i}^{seq}\in \mathbb{R}^{T \times H \times W \times C} \quad \to \quad \mathbf{F}'_{i}\in \mathbb{R}^{H \times W \times C},$$

and then a unified Transformer models interactions across frames, agents, and HD map (Zhou et al., 2024). Under 32× compression, one-step communication reduces data transmission from

$5 \times 0.269 \text{ Mb} \to 0.269 \text{ Mb},$

while improving performance by about 5.0% AP and about 12% EPA relative to multi-step communication (Zhou et al., 2024).

Several works target the quality of single-agent features before fusion. V2X-AHD uses an asymmetric heterogeneous distillation network in which a fused multi-view teacher transfers contour-aware latent representations to a single-view student, combined with the sparse-convolution-based Spare Pillar backbone and lightweight MSA fusion (He et al., 2023). On V2XSet, the full model reports 0.877 / 0.770 AP and the ablation states that, at IoU=0.7, SparePillar and AHD improve AP by 13.5% and 10.8%, respectively (He et al., 2023).

Recent work also addresses training data scarcity and cross-domain transfer. CooPre introduces self-supervised cooperative pretraining via multi-agent LiDAR reconstruction with BEV-guided masking, reporting a 4% mAP improvement on V2X-Real dataset and stronger robustness under localization errors and time delay (Zhao et al., 2024). V2X-DG studies domain generalization across OPV2V, V2XSet, V2V4Real, and DAIR-V2X, proposing CMAG and Cooperation Feature Consistency to simulate unseen cooperation and regularize generalized features (Li et al., 19 Mar 2025). For heterogeneous sensing, HeCoFuse is designed for nine mixed sensor configurations in which nodes may carry Cameras, LiDARs, or both; on TUMTraf-V2X it reports 43.22% 3D mAP on LC+LC and 43.38% 3D mAP on L+LC (Wei et al., 18 Jul 2025).

A common misconception is that V2X cooperative perception is reducible to designing a better fusion block. The model literature suggests otherwise: contour completion, self-supervised representation learning, domain shift, heterogeneous sensing, and temporal compression all materially affect downstream cooperative performance (He et al., 2023, Zhao et al., 2024, Li et al., 19 Mar 2025, Wei et al., 18 Jul 2025).

5. Datasets, co-simulation, and online evaluation

The empirical base of the field now spans simulation, real-world datasets, and online deployed systems. V2X-Real is presented as the first open large-scale real-world dataset for V2X cooperative perception, collected with 2 connected automated vehicles and 2 smart infrastructure units, and containing 33K LiDAR frames, 171K camera images, and over 1.2M annotated 3D bounding boxes across 10 object categories (Xiang et al., 2024). It explicitly supports Vehicle-Centric, Infrastructure-Centric, Vehicle-to-Vehicle, and Infrastructure-to-Infrastructure sub-datasets, making infrastructure a first-class cooperative agent rather than a special case (Xiang et al., 2024).

V2X-ReaLO moves beyond static evaluation by extending the V2X-Real test split into dynamic synchronized ROS bags with 25,028 frames and 6,850 fully annotated key frames, and by providing the first practical demonstration of online intermediate fusion’s feasibility and performance under genuine real-world conditions (Xiang et al., 13 Mar 2025). The framework is deployed on real connected vehicles, smart roadside infrastructure, ROS-based distributed nodes, and Wi-Fi communication, and demonstrates 10 Hz LiDAR stream with 10 Hz output detections (Xiang et al., 13 Mar 2025).

For temporal tasks, the V2XPnP Sequential Dataset is described as the first large-scale real-world sequential dataset for V2X cooperative perception and prediction, with 100 scenarios, 40k frames, 10 Hz capture rate, 4 agents, 24 intersections, 10 object categories, and an average of 136 tracked objects per scene (Zhou et al., 2024). This directly supports evaluation of temporal perception, traditional cooperative prediction, and end-to-end perception plus prediction (Zhou et al., 2024).

Simulation remains important for scale, controllability, and end-to-end coupling. V2XSet, built with CARLA and OpenCDA, provides 11,447 frames and 33,081 agent-wise samples with 2 to 7 intelligent agents per scene and explicit noisy settings for delay and pose (Xu et al., 2022). The NLOS study introduces a cross-platform co-simulation framework linking CARLA, AutoCastSim, and Veins + OMNeT++ + SUMO through TraCI so that traffic scene, wireless QoS, sensor fusion, and ego driving decisions form an end-to-end loop (Li et al., 2023). Multi-V2X further targets deployment realism by modeling CAV penetration rate, with 549k RGB frames, 146k LiDAR frames, 4,219k annotated 3D bounding boxes, up to 31 agents in communication range, and a highest possible CAV penetration rate of 86.21% (Li et al., 2024).

Evaluation has correspondingly diversified. Detection metrics include AP at IoU thresholds such as 0.3, 0.5, and 0.7; temporal tasks add ADE, FDE, MR, and EPA; online systems measure communication latency, packet reception rate, collision rate, driving score, and route completion rate (Xiang et al., 13 Mar 2025, Zhou et al., 2024, Guo et al., 29 Sep 2025). This suggests that cooperative perception is now evaluated as a coupled perception–communication–control system rather than a static detector alone.

6. Error patterns, safety effects, and emerging directions

The link between cooperative perception quality and driving safety has become an explicit object of study. “When Autonomous Vehicle Meets V2X Cooperative Perception: How Far Are We?” defines six prevalent cooperative perception error patterns: LCME, LCLE, LADE, CCME, CCLE, and CADE, distinguishing misleading cooperative errors from miscorrected cooperative errors (Guo et al., 29 Sep 2025). Across its large-scale study, LiDAR-based cooperation achieves the highest AP, outperforming multimodal cooperation by 21.7% on average and camera-based cooperation by 87.0% on average, while localization-related errors are reported as especially frequent (Guo et al., 29 Sep 2025).

The same study connects these errors to downstream driving violations. Under normal communication, cooperative systems outperform the single-agent baseline in driving score and collision rate; for example, V2XViT reports DS 81.0, CR 26.3, RCR 87.9, compared with Single Agent: DS 69.1, CR 36.8, RCR 74.7 (Guo et al., 29 Sep 2025). Yet the system is reported as not robust to communication interference when running online: under communication latency, DS decreases by 10.3%, RCR decreases by 6.3%, and CR increases by 7.9%; under pose error, DS decreases by 11.2%, RCR decreases by 7.2%, and CR increases by 7.9% (Guo et al., 29 Sep 2025). Under normal conditions, CCLE is the only statistically significant predictor in the reported logistic regression with $p = 0.047$ and OR = 1.019; under abnormal communication, LCLE becomes the dominant violation-related error (Guo et al., 29 Sep 2025).

A safety-oriented abstraction appears in coPEM, which models cooperative perception through object-level error models rather than full sensor simulation. In the reported urban intersection scenario, ego-only PEM achieves 426/500 = 85.2% success, while cooperative perception raises success to 499/500 = 99.8% at 0s latency and 474/500 = 94.8% at 0.5s, but drops to 323/500 = 64.6% at 1s and 0/500 = 0% at 1.5s (Piazzoni et al., 2022). The direct conclusion is that cooperative perception can nearly restore ground-truth-like safety if latency is sufficiently small, but stale cooperative information can become harmful (Piazzoni et al., 2022).

Emerging directions reflect this shift from isolated detection accuracy to system intelligence. Collaborative Joint Perception and Prediction proposes decoupled Collaborative Scene Completion and Joint Perception and Prediction modules, together with a collaboration trigger based on occlusion level, ego perception confidence, and communication conditions (Wan et al., 27 Jan 2025). The SPD survey generalizes this trend by requiring cooperative evidence to be calibrated, time-stamped, and provenance-aware, and by advocating evaluation beyond geometry toward earliness, calibration, coordination, and utility (Zhang et al., 29 Nov 2025). The information-sensor survey similarly argues that future V2X systems must be communication-aware, task-adaptive, and scalable, with message representation, fusion, and scheduling treated as first-class design variables rather than implementation details (Song et al., 30 Apr 2025).

Taken together, these developments indicate that V2X cooperative perception is no longer understood solely as multi-agent 3D detection. It is increasingly treated as a distributed inference layer whose validity depends on communication reliability, synchronization, heterogeneity handling, temporal reasoning, and safety-relevant evaluation (Li et al., 2023, Xiang et al., 13 Mar 2025, Zhou et al., 2024, Guo et al., 29 Sep 2025).