Progressive Heterogeneous Collaborative Perception

• Progressive Heterogeneous Collaborative Perception (PHCP) is a framework that incrementally integrates diverse agents using staged feature alignment and fusion.
• It leverages intermediate Bird’s-Eye-View features to efficiently balance communication costs with enhanced detection accuracy in multi-agent systems.
• PHCP employs inference-time adaptation and lightweight alignment modules to reconcile heterogeneous sensor inputs and backbone disparities.

Progressive Heterogeneous Collaborative Perception (PHCP) denotes a line of collaborative perception research in which heterogeneous agents are incorporated stage-wise and their shared representations are progressively aligned across agents, feature scales, or deployment phases. In the literature, the term appears both as a general design lens for heterogeneous collaborative perception and as the name of a specific inference-time framework that formulates heterogeneity handling as few-shot unsupervised domain adaptation (Maleki et al., 7 Mar 2026, Si et al., 11 Sep 2025). Within the broader cooperative perception taxonomy, PHCP lies primarily in the intermediate-fusion regime, where agents share Bird’s-Eye-View (BEV) features rather than raw sensor streams or only final detections, in order to balance bandwidth, robustness, and perceptual utility (Bai et al., 2022, Ren et al., 2022).

1. Conceptual origins and scope

Collaborative perception (CP) is motivated by the fact that single-agent perception is limited by physical occlusion, long-range sparsity, and restricted field of view. The 2022 surveys define CP as perception based on “multiple spatially separated perception nodes” and organize it by node structure, sensor modality, and fusion scheme (Bai et al., 2022). They distinguish early collaboration in the input space, intermediate collaboration in the feature space, late collaboration in the output space, and mixed collaboration, and they emphasize that intermediate collaboration offers “economic communication bandwidth compared to early collaboration as well as upgraded perception ability compared to late collaboration” (Ren et al., 2022).

The surveys do not explicitly define “Progressive Heterogeneous Collaborative Perception,” but they provide the conceptual substrate from which it emerges. The Hierarchical Cooperative Perception framework scales cooperation from local to meso to macro levels, while the collaboration-mode taxonomy describes progressive escalation from singleton perception to peer-to-peer and infrastructure-assisted cooperation (Bai et al., 2022). This suggests a PHCP interpretation in which heterogeneity is not treated as a one-time nuisance variable but as a structural property of real deployments, requiring staged collaboration and staged adaptation as new sensor types, backbones, and agent roles enter the system.

Later work makes this interpretation explicit. HEAL formulates the “open heterogeneous collaborative perception problem” as continual accommodation of “continually emerging new heterogeneous agent types” while maintaining high perception performance and low integration cost (Lu et al., 2024). Faster-HEAL states that Progressive Heterogeneous Collaborative Perception addresses heterogeneity by “progressively refining and aggregating features across agents and scales, while progressively adapting the domain gaps via lightweight mechanisms” (Maleki et al., 7 Mar 2026). PHCP in the narrower 2025 sense further specializes the idea by asking whether heterogeneity can be handled “directly during inference, eliminating the need for joint training” (Si et al., 11 Sep 2025).

2. Representation, alignment, and fusion mechanics

PHCP inherits the standard CP pipeline in which each agent transforms local observations into a shared representation, aligns them to an ego or global frame, and fuses them for downstream detection. In the survey formulation, if agent $i$ has local frame data $x_i$ and rigid transform $T_i^g \in SE(3)$, then alignment to global or ego coordinates is written as $x_g = T_i^g x_i$ (Bai et al., 2022). In intermediate collaboration, raw inputs are encoded as features $\phi_i(x_i)$, fused by a learned operator $g$, and decoded by a task head $h$: $F = g(\{\phi_i(x_i)\}_{i=1}^N), \qquad y = h(F).$ This representation is central to PHCP because heterogeneity appears most acutely as a feature-space mismatch: identical BEV tensor shapes can still carry incompatible channel semantics and spatial statistics (Ren et al., 2022).

The dominant spatial substrate is BEV. HEAL and Faster-HEAL both use BEV features as the alignment target because BEV provides a unified spatial representation across LiDAR and camera pipelines (Lu et al., 2024, Maleki et al., 7 Mar 2026). In Faster-HEAL, the ego base uses a 64-channel LiDAR encoder with PointPillar, while heterogeneous neighbors may use SECOND or camera encoders based on EfficientNet or ResNet-50 together with Lift-Splat-Shoot; all are projected into the ego frame and aligned to a unified tensor shape $(C,H,W)$ before fusion (Maleki et al., 7 Mar 2026). The same logic appears in the 2025 PHCP framework, which assumes that heterogeneous encoders can expose compatible BEV tensors so that a lightweight adapter can map collaborator features into the ego semantic space (Si et al., 11 Sep 2025).

Progressive aggregation is typically realized by pyramid fusion. HEAL introduces a multi-scale foreground-aware Pyramid Fusion network that downsamples aligned BEV features through ResNeXt layers, estimates per-scale foreground maps, normalizes them into softmax fusion weights, and aggregates coarse and fine scales through weighted summation and upsampling (Lu et al., 2024). Faster-HEAL describes the same mechanism as “a progressive coarse-to-fine aggregation” whose “softmax-normalized occupancy weighting acts as attention over agents per location and scale,” with lower-resolution features improving robustness to pose and calibration error while higher-resolution features inject detail (Maleki et al., 7 Mar 2026). This multi-scale structure is one of the principal senses in which PHCP is progressive: refinement is distributed across scales rather than delegated to a single monolithic fusion block.

3. Progressive adaptation strategies

The central technical problem in PHCP is the domain gap induced by heterogeneity in sensor modality, encoder backbone, training distribution, and sometimes calibration quality. Different PHCP variants differ less in the existence of this problem than in when and how they adapt to it.

HEAL adopts a two-phase protocol. First, it trains a collaboration base on initial homogeneous agents to establish a unified feature space via Pyramid Fusion. Second, when a new agent type appears, it performs “backward alignment” by freezing the previously trained Pyramid Fusion and detection head and training only the new agent’s encoder locally with single-agent supervision (Lu et al., 2024). The progressive element is temporal: the alliance grows by repeated onboarding of new types without retraining existing agents.

Faster-HEAL retains the two-stage structure but replaces encoder retraining with a lighter mechanism. Stage 1 builds the unified pyramid feature space and detection head on homogeneous agents. Stage 2 freezes the new agent’s encoder or backbone, the ego pyramid fusion module, and the detection head, and trains only a compact ConvNeXt-based aligner together with a low-rank visual prompt (Maleki et al., 7 Mar 2026). If a new agent $k$ of type $x_i$0 produces aligned feature $x_i$1, the prompt $x_i$2 is added once before pyramid fusion: $x_i$3 The prompt is factorized with PARAFAC decomposition,

$x_i$4

reducing trainable parameters from $x_i$5 to $x_i$6, with $x_i$7 reported as the best accuracy–efficiency trade-off on OPV2V-H (Maleki et al., 7 Mar 2026).

The 2025 framework named “Progressive Heterogeneous Collaborative Perception” moves adaptation to inference time. During a short warm-up period, a new collaborator shares both intermediate BEV features and detection outputs; the ego vehicle treats collaborator detections as pseudo labels and self-trains a lightweight CBAM adapter using the first $x_i$8 unlabeled frames, with typical shot counts $x_i$9 (Si et al., 11 Sep 2025). After this Stage I warm-up, only features are shared in Stage II, and the ego applies the learned adapter before standard intermediate fusion. The framework does not use teacher–student networks, momentum encoders, adversarial discriminators, MMD, or contrastive alignment; adaptation is guided solely by the detector’s standard loss against pseudo labels (Si et al., 11 Sep 2025).

HyDRA interprets PHCP as a routing principle rather than an alignment-only principle. It integrates intermediate and late fusion within a domain-aware pipeline: a training-free domain classifier assigns compatible agents to intermediate fusion and heterogeneous ones to late fusion, then anchor-guided pose graph optimization refines heterogeneous poses using Stage 1 detections as fixed anchors before final late fusion (Song et al., 25 Mar 2026). This is progressive in the order of information use: feature-compatible agents are fused first, incompatible agents are incorporated later, and pose correction is inserted between the two.

GenComm departs further by replacing feature alignment with feature generation. In that paper, PHCP means “Pragmatic Heterogeneous Collaborative Perception,” but the mechanism still supports progressive integration because new agents can be added by fine-tuning only a lightweight Deformable Message Extractor per sender–receiver pair while keeping encoders, fusion modules, and task heads frozen (Zhou et al., 22 Oct 2025). Collaborators transmit compact spatial messages rather than full intermediate features; a conditional diffusion-based Spatial-Aware Feature Generator then generates collaborator features in the ego semantic space and a Channel Enhancer refines them before fusion.

4. Training protocols, communication design, and privacy

PHCP systems are typically organized around staged training or staged deployment. HEAL’s collaboration base training learns encoders, Pyramid Fusion, and a detection head on homogeneous multi-agent data, after which the fusion back-end is frozen and each new agent type is trained separately with single-agent supervision (Lu et al., 2024). Faster-HEAL follows the same logic, but its Stage 1 specifically trains the ego PointPillar encoder, a ResNeXt pyramid fusion network with 3 scales using $T_i^g \in SE(3)$0 channels and $T_i^g \in SE(3)$1 blocks per scale, 1×1 foreground estimators, and a detection head for 25 epochs with Adam at learning rate 0.002; Stage 2 freezes the new agent’s encoder or backbone, the ego pyramid fusion, and the detection head, and trains the ConvNeXt-based aligner and low-rank prompt with the same optimization settings (Maleki et al., 7 Mar 2026).

Inference-time PHCP uses a different schedule. The ego initializes from a homogeneous CP model, collects the first $T_i^g \in SE(3)$2 frames from a new collaborator as a support set, filters or soft-weights pseudo labels by confidence threshold $T_i^g \in SE(3)$3, freezes the encoder, fusion, and detection head, and optimizes only the per-collaborator CBAM adapter for 20 epochs with linear warm-up factor 0.001 for 8 epochs, base learning rate 0.005, decay at epochs 12 and 16, and batch size 1 (Si et al., 11 Sep 2025). The support buffer is not maintained indefinitely; adaptation is performed once when the collaboration relationship is established.

Communication in PHCP is usually feature-centric. The surveys emphasize that early fusion of raw LiDAR is often infeasible over V2V because of payload, whereas intermediate fusion drastically reduces communication relative to raw sharing while retaining richer information than late fusion (Bai et al., 2022). HEAL, Faster-HEAL, and the 2025 PHCP framework all rely on intermediate BEV features plus pose or alignment metadata (Lu et al., 2024, Maleki et al., 7 Mar 2026, Si et al., 11 Sep 2025). Faster-HEAL further states that privacy is maintained by design because only intermediate BEV features are shared, while raw sensor data and full model weights remain local; the ego provides a compact aligner to the neighbor for on-device projection and compression before transmission, but the framework does not use encryption or differential privacy (Maleki et al., 7 Mar 2026).

GenComm shifts the communication unit from features to compact spatial messages. Its transmitted messages have shape $T_i^g \in SE(3)$4 with $T_i^g \in SE(3)$5, versus 128-channel features in the reported LiDAR setting, yielding approximately 64× channel compression (Zhou et al., 22 Oct 2025). This mechanism is explicitly motivated by pragmatic deployment constraints: non-intrusiveness to pretrained stacks, low incremental cost, and lower communication burden.

5. Reported empirical behavior

Reported evidence for PHCP spans open heterogeneous onboarding, few-shot inference-time adaptation, training-free routing, and generative communication. The reported results are not directly interchangeable because the datasets, heterogeneity regimes, and training assumptions differ, but together they define the empirical profile of the field.

Framework	Setting	Reported result
HEAL (Lu et al., 2024)	OPV2V-H, integrating 3 new agent types	Training parameters reduced by 91.5%
Faster-HEAL (Maleki et al., 7 Mar 2026)	OPV2V-H	About 94% fewer trainable parameters; 30.44 samples/s; 4.3 GB peak GPU memory
PHCP (Si et al., 11 Sep 2025)	OPV2V, LS–LP heterogeneity	[email protected] improved from 53.0 to 85.9
HyDRA (Song et al., 25 Mar 2026)	V2X-Real	Comparable to state-of-the-art with no additional training; performance maintained as the number of agents increases
GenComm (Zhou et al., 22 Oct 2025)	OPV2V-H, DAIR-V2X, V2X-Real	81% reduction in computational cost and parameter count when incorporating new agents

HEAL reports progressive integration on OPV2V-H starting from $T_i^g \in SE(3)$6 and then adding $T_i^g \in SE(3)$7, $T_i^g \in SE(3)$8, and $T_i^g \in SE(3)$9, with AP50 progressing as 0.826 $x_g = T_i^g x_i$0 0.892 $x_g = T_i^g x_i$1 0.894 and AP70 as 0.726 $x_g = T_i^g x_i$2 0.812 $x_g = T_i^g x_i$3 0.813, while train throughput, model parameters, FLOPs, and peak memory all remain substantially lower than HM-ViT (Lu et al., 2024). Faster-HEAL, evaluated on the same OPV2V-H family of scenarios, reports [email protected] of 89.3%, 93.0%, and 92.7%, and [email protected] of 79.7%, 86.3%, and 85.7%, when progressively adding camera EfficientNet, LiDAR SECOND, and camera ResNet-50 collaborators to a PointPillar base; compared with HEAL trained under the same configuration, the improvements are approximately 2.1%, 1.7%, and 1.8% at IoU 0.5 and approximately 1.4%, 1.5%, and 1.2% at IoU 0.7 (Maleki et al., 7 Mar 2026).

The 2025 PHCP framework emphasizes the recovery of heterogeneous intermediate fusion without labeled joint training. On OPV2V, direct heterogeneous fusion yields [email protected] of 53.0, whereas PHCP raises it to 85.9; the homogeneous references are 90.5 for SECOND and 93.1 for PointPillars (Si et al., 11 Sep 2025). Even 1-shot adaptation is reported to yield approximately 50% improvement over the heterogeneous baseline at IoU 0.7, with Stage I training taking 1.49–2.39 s for 20 epochs depending on the number of shots and Stage II inference running at 0.07 s/frame (Si et al., 11 Sep 2025).

HyDRA’s reported contribution is different: it shows that training-free domain classification plus anchor-guided pose graph optimization can match or exceed state-of-the-art heterogeneity-aware methods in multiple AP metrics while remaining stable as the number of collaborating agents increases (Song et al., 25 Mar 2026). GenComm reports strong or best/second-best AP across static and dynamically growing heterogeneous settings on OPV2V-H, DAIR-V2X, and V2X-Real, while also reducing the reported “Comm. Volume (log2)” from 22.0 to 16.0 on OPV2V-H across fusion backbones (Zhou et al., 22 Oct 2025).

6. Terminological ambiguity, limitations, and research directions

A common misconception is that PHCP denotes a single settled architecture. The literature is less uniform. In the survey lineage, PHCP is an inferred extension of hierarchical and heterogeneous CP rather than a formally named framework (Bai et al., 2022, Ren et al., 2022). In HEAL and Faster-HEAL, it functions as a staged alignment-and-fusion paradigm for open heterogeneous onboarding (Lu et al., 2024, Maleki et al., 7 Mar 2026). In the 2025 work, it is the explicit name of an inference-time few-shot unsupervised domain adaptation framework (Si et al., 11 Sep 2025). In GenComm, the acronym PHCP is reassigned to “Pragmatic Heterogeneous Collaborative Perception,” with emphasis on non-intrusiveness and scalable deployment rather than on progressive adaptation as such (Zhou et al., 22 Oct 2025). The overlap is substantive, but the term is not semantically uniform.

The limitations reported across the literature are also consistent. Surveys identify calibration and synchronization across heterogeneous sensors and nodes, localization drift, communication reliability and bandwidth, privacy and security, trust in adversarial settings, time-varying topology, standardization, and the difficulty of scaling deep fusion to large systems as major open problems (Bai et al., 2022). HEAL notes BEV dependency, extreme modality or domain gaps, and dynamic calibration drift and time asynchrony as unresolved issues (Lu et al., 2024). Faster-HEAL reports remaining failure cases under severe calibration errors, extreme sensor noise, and rare appearance distributions, and states that statistical significance beyond point estimates was not reported; it also notes that current alignment is task-driven rather than based on explicit distributional matching, suggesting contrastive or MMD-based alignment as a possible extension (Maleki et al., 7 Mar 2026). The inference-time PHCP framework assumes compatible feature map shapes, rough extrinsics and time synchronization, and reasonably calibrated collaborator detections; severe domain gaps, noisy pseudo labels, heavy pose errors, and communication faults remain out of scope (Si et al., 11 Sep 2025). HyDRA highlights domain misclassification near threshold $x_g = T_i^g x_i$4, anchor scarcity, and severe calibration mismatch as failure modes for progressive routing (Song et al., 25 Mar 2026). GenComm identifies dependence on spatial message quality, diffusion overhead, and commercial frictions surrounding per-pair module deployment as practical limits (Zhou et al., 22 Oct 2025).

A plausible implication is that PHCP research is converging on a small set of recurring design principles rather than on a single dominant implementation. These principles include a shared BEV space, staged incorporation of new agents, preference for intermediate fusion when feature compatibility can be preserved, selective fallback to late fusion when it cannot, compact adaptation modules rather than full-stack retraining, and communication schemes that make privacy preservation and incremental deployment operational rather than purely rhetorical. Whether future work emphasizes progressive onboarding, inference-time self-training, domain-aware routing, or generative communication, the technical center of PHCP remains the same: preserving the utility of feature-level collaboration under the heterogeneity that real multi-agent autonomous driving systems inevitably exhibit.