HeCoFuse: Unified V2X Perception

• HeCoFuse is a unified framework for vehicle-to-everything cooperative perception that supports nine sensor configurations by fusing camera and LiDAR data into a common BEV representation.
• It employs hierarchical attention fusion and adaptive spatial resolution to align cross-modality features and adapt processing based on sensor quality and deployment asymmetries.
• The framework leverages cooperative learning across diverse sensor setups, achieving robust 3D object detection performance with 3D mAP scores ranging from 21.74% to 43.38%.

Searching arXiv for the HeCoFuse paper and closely related V2X cooperative perception work to ground the article in current literature. HeCoFuse is a unified framework for Vehicle-to-Everything cooperative perception under heterogeneous sensor configurations, designed for vehicle–infrastructure 3D object detection when participating nodes may carry Cameras (C), LiDARs (L), or both (LC). Its central objective is to preserve cooperative perception performance across mixed deployments in which the vehicle and infrastructure do not necessarily expose identical sensing modalities, while addressing cross-modality feature misalignment, imbalanced representation quality, missing modalities at some nodes, and computational constraints. In the reported formulation, all participating nodes are mapped into a compatible bird’s-eye-view representation, followed by hierarchical attention-based inter-node fusion and configuration-aware resolution control (Wei et al., 18 Jul 2025).

1. Problem setting and deployment model

HeCoFuse addresses vehicle-to-everything cooperative 3D perception in a vehicle–infrastructure setup. The task is to fuse information from a vehicle node and a roadside infrastructure node so that 3D object detection is more robust under occlusion, limited field of view, and difficult environmental conditions. The framework is explicitly designed for heterogeneous sensor configurations, a setting motivated by cost constraints, deployment variability across vehicles and roadside units, incremental hardware upgrades, different sensing capabilities across fleets and infrastructure, and possible partial sensor failures (Wei et al., 18 Jul 2025).

In this setting, each node can have camera only, LiDAR only, or both. The cooperative pair can therefore take nine configurations: LC+LC, L+L, C+C, L+C, C+L, LC+C, LC+L, C+LC, and L+LC. Here, $X+Y$ denotes vehicle $=X$ and infrastructure $=Y$. The framework is presented as a single model spanning all nine cases rather than as a collection of per-configuration models.

The technical difficulty arises from several coupled asymmetries. LiDAR and camera features differ in geometry, resolution, density, and field of view; modality quality is scene-dependent; some nodes are unimodal while others are multimodal; and practical cooperative systems cannot assume retraining for every sensor combination. HeCoFuse is therefore organized around a common BEV feature interface, conditional intra-node fusion or adaptation, and a shared inter-node cooperative fusion module.

2. Architectural organization

At a high level, HeCoFuse proceeds in six stages: each node senses with its available modalities; LiDAR is encoded into a BEV feature map; camera images are encoded and transformed into BEV features; multimodal nodes perform BEVFusion while unimodal nodes pass through a PseudoFusion or feature-adapter path; vehicle and infrastructure BEV features are fused by Hierarchical Attention Fusion; and Adaptive Spatial Resolution adjusts feature-map scale around the inter-node fusion stage before the fused representation is sent to the perception head (Wei et al., 18 Jul 2025).

For LiDAR, the methodology describes dynamic voxelization, a 3D sparse convolutional backbone, and projection to the ground plane to obtain a dense BEV feature map $F_n^{lidar}$. The implementation section reports PointPillars as the LiDAR encoder. The paper notes this partly conceptual and partly implementation-level description without fully reconciling the two. For camera input, multi-view images are processed by a 2D convolutional backbone, a feature neck such as FPN, and a view transformation using camera intrinsics, camera-to-LiDAR transforms, and estimated depth distributions, yielding a camera BEV map $F_n^{camera}$. The implementation reports YOLOv8 for camera data processing.

For a node $n$ with sensor set $S_n \subseteq \{\text{camera}, \text{lidar}\}$, the feature definition is

$$F_n = \begin{cases} \text{Fuse}(F_n^{camera}, F_n^{lidar}), & \text{if } |S_n| = 2 \ F_n^s, & \text{if } |S_n| = 1,\ s \in S_n \end{cases}$$

This establishes the framework’s modular conditional architecture. If the node is multimodal, camera and LiDAR are fused at BEV level. If the node is unimodal, the available feature is retained and then mapped into a common format.

For LC nodes, the intra-node fusion operator is

$F_n = \text{BEVFusion}(F_n^{camera}, F_n^{lidar}),$

with

$$\text{BEVFusion}(F_1, F_2) = \phi_{act}\big(\phi_{norm}(\phi_{conv}([F_1, F_2]))\big),$$

where $=X$0 is channel-wise concatenation, $=X$1 is a convolution layer, $=X$2 is normalization, and $=X$3 is an activation function.

For L-only or C-only nodes, HeCoFuse introduces PseudoFusion through modality-specific adapters:

$=X$4

with

$=X$5

This common-shape requirement is the mechanism by which heterogeneous node outputs become compatible for subsequent cooperative fusion.

3. Hierarchical Attention Fusion and Adaptive Spatial Resolution

The principal inter-node fusion module is Hierarchical Attention Fusion. Let $=X$6 and $=X$7 denote the BEV features of the vehicle and infrastructure nodes. The first stage is channel-wise attention:

$=X$8

followed by

$=X$9

This assigns per-channel trust between the two nodes. The intended role is to handle modality imbalance and sensor-quality asymmetry, so that different semantic or geometric channels can favor whichever node is more informative in that subspace (Wei et al., 18 Jul 2025).

The second stage is spatial attention. For node $=Y$0,

$=Y$1

The final fused feature is then

$=Y$2

The fusion is termed hierarchical because weighting occurs at two levels: channel level and spatial level. This design is meant to suppress weak or noisy regions, emphasize locally reliable evidence, and adapt to asymmetric configurations such as L+C, C+LC, or L+LC.

Adaptive Spatial Resolution is inserted around the inter-node fusion stage. Downsampling is defined as

$=Y$3

where

$=Y$4

and restoration after fusion is

$=Y$5

The paper presents ASR as a mechanism for balancing computational cost and fusion effectiveness, and claims a reduction of computational overhead by $=Y$6 in heterogeneous configurations. However, it does not provide a dedicated runtime, FLOP, latency, or bandwidth table, so the efficiency claim is formulaic and qualitative rather than fully benchmarked (Wei et al., 18 Jul 2025).

4. Cooperative learning across nine sensor configurations

A defining feature of HeCoFuse is that it does not train a separate network for each sensor combination. Instead, it uses a cooperative learning strategy in which training randomly samples from all nine heterogeneous configurations, and the active fusion path is determined by which modalities are actually present (Wei et al., 18 Jul 2025).

This dynamic selection is described conceptually rather than through a dedicated routing equation. If a node has both camera and LiDAR, the model activates BEVFusion at that node. If it has only one modality, it activates the corresponding adapter-based PseudoFusion path. The same HAF and ASR modules are then used for inter-node fusion. The result is a unified architecture whose conditional behavior is induced by modality availability and multi-configuration training rather than by maintaining distinct model families.

The paper also specifies several operational assumptions. Fusion is performed in BEV as a shared intermediate space; calibration must be sufficient to project camera features into BEV and align vehicle and infrastructure features; sensing is synchronized by the dataset; and pose or geometric consistency is assumed to support cooperative BEV fusion. A plausible implication is that performance depends materially on calibration quality and geometric registration, even though the paper does not isolate those factors experimentally.

5. Dataset, implementation, and reported performance

HeCoFuse is evaluated on TUMTraf-V2X, a real-world urban intersection dataset collected in Munich. The paper states that the dataset contains 2,000 camera images, 5,000 LiDAR point clouds, 29,000 annotated 3D bounding boxes, and 8 object categories. Both vehicle and infrastructure have cameras and mechanical LiDARs of similar specifications, which permits simulation of the nine heterogeneous configurations (Wei et al., 18 Jul 2025).

The reported training setup is from scratch on the TUMTraf-V2X mini dataset, defined as half of the full dataset, for 20 epochs on a single GeForce RTX 4090 GPU. The paper does not specify learning rate, optimizer type, batch size, weight decay, data augmentation settings, scheduler, or communication packet size. It also does not provide explicit loss-function equations, although the architecture is embedded in a 3D detector pipeline.

The main metric emphasized is 3D mAP. The reported HeCoFuse results across all nine configurations are:

Configuration	3D mAP
LC+LC	43.22
L+L	42.10
C+C	21.74
L+C	30.04
C+L	32.68
LC+C	30.85
LC+L	39.17
C+LC	34.76
L+LC	43.38

In the full sensor configuration, HeCoFuse achieves $=Y$7 3D mAP, compared with $=Y$8 for the CoopDet3D baseline, an improvement of $=Y$9. The best-performing configuration is L+LC at $F_n^{lidar}$0 3D mAP. Across all nine cases, performance remains in the range $F_n^{lidar}$1 to $F_n^{lidar}$2.

The paper also reports additional metrics for each configuration, including Precision, Recall, 3D IoU, Position RMSE, Rotation RMSE, and 3D mAP. For LC+LC, the reported values are Precision $F_n^{lidar}$3, Recall $F_n^{lidar}$4, 3D IoU $F_n^{lidar}$5, Position RMSE $F_n^{lidar}$6, Rotation RMSE $F_n^{lidar}$7, and 3D mAP $F_n^{lidar}$8. For L+LC, they are Precision $F_n^{lidar}$9, Recall $F_n^{camera}$0, 3D IoU $F_n^{camera}$1, Position RMSE $F_n^{camera}$2, Rotation RMSE $F_n^{camera}$3, and 3D mAP $F_n^{camera}$4. For C+C, they are Precision $F_n^{camera}$5, Recall $F_n^{camera}$6, 3D IoU $F_n^{camera}$7, Position RMSE $F_n^{camera}$8, Rotation RMSE $F_n^{camera}$9, and 3D mAP $n$0.

6. Interpretation, significance, and limitations

Several empirical patterns are explicit. First, LiDAR contributes much more strongly than camera in this benchmark: L+L yields $n$1 3D mAP whereas C+C yields $n$2, a gap of $n$3 mAP. Second, infrastructure sensing is especially valuable: L+LC reaches $n$4, while LC+L reaches $n$5, even though both involve three total sensors. The paper attributes this asymmetry to the infrastructure’s elevated viewpoint and wider field of view. Third, the method maintains nontrivial performance under strongly asymmetric pairings such as L+C, C+L, C+LC, and LC+C, which supports the claim that one unified architecture can operate across heterogeneous sensor deployments (Wei et al., 18 Jul 2025).

The framework is also presented as the current state-of-the-art on the TUM-Traf V2X dataset, and the paper states that it achieved first place in the CVPR 2025 DriveX Workshop TUMTraf-V2X Challenge, with the L+LC configuration reaching $n$6 3D mAP. Under LC+LC, the paper lists challenge entries BUPTMM $n$7, CV123 $n$8, TEAM12138 $n$9, KaAI $S_n \subseteq \{\text{camera}, \text{lidar}\}$0, and HeCoFuse $S_n \subseteq \{\text{camera}, \text{lidar}\}$1, indicating that HeCoFuse is competitive with the top challenge methods in the full-configuration setting.

Several limitations are stated or directly implied. Evaluation is restricted to a two-node vehicle–infrastructure setup. The paper does not deeply validate behavior under adverse weather such as rain, fog, or snow. Camera-only performance remains substantially weaker than LiDAR-heavy configurations. Exact efficiency, bandwidth, and loss-function details are not deeply reported. The paper also does not provide a dedicated ablation table isolating the individual gains of HAF, ASR, and the cooperative learning strategy. This suggests that while the integrated framework is empirically strong, the marginal contribution of each architectural component is not numerically decomposed in the reported text.

A common misconception would be to interpret HeCoFuse as merely a multimodal fusion model for the fully equipped LC+LC case. The paper instead defines it as a heterogeneous cooperative perception framework whose central contribution is support for all nine configurations through a single conditional architecture. Another possible misconception is that ASR has been comprehensively benchmarked for systems efficiency; the paper claims computational reduction, but does not provide the dedicated timing or bandwidth tables that would fully characterize deployment cost.