Collaborative Cross-Modal Encoder (CCE)

• CCE is a neural architecture that integrates sensor-specific encoders and object-centric queries to achieve unified multi-modal feature extraction and fusion.
• It employs multi-head cross-attention and masked self-attention to optimize collaborative information exchange while reducing communication bandwidth.
• Empirical studies in vehicular perception and remote sensing demonstrate that CCE enhances detection accuracy and efficiency under diverse conditions.

A Collaborative Cross-Modal Encoder (CCE) is a neural architecture that unifies multi-sensor feature extraction, joint cross-modal fusion, and multi-agent collaborative information sharing anchored in object-centric queries. CCE has found application in distributed perception systems, continual multi-modal interpretation, and heterogeneous sensor communication, addressing bandwidth, robustness, and semantic consistency in collaborative settings. Notable instantiations include CoCMT for vehicular perception (Wang et al., 13 Mar 2025), CPP for continual remote sensing (Yuan et al., 2024), and CMSC for heterogeneous collaborative communication (Lu et al., 25 Nov 2025).

1. Architectural Principles

CCE structures integrate feature extraction from disparate sensor modalities, such as camera images and LiDAR point clouds, within a unified computational pipeline. In CoCMT (Wang et al., 13 Mar 2025), each connected autonomous vehicle (CAV) is equipped with dedicated modality-specific encoders: a ResNet-50 backbone for cameras and a PointPillar-SpConv2 stack for LiDAR. These generate modality-specific token sequences: $$F^\mathrm{cam}_i \in \mathbb{R}^{T^\mathrm{cam} \times D}$$ and $$F^\mathrm{lidar}_i \in \mathbb{R}^{T^\mathrm{lidar} \times D}$$, with $D$ typically set to 256. A fixed set of $N$ learnable object queries $Q^0_i \in \mathbb{R}^{N \times D}$ is refined through cross-modal Transformer layers, attending to both camera and LiDAR features to produce object-centric query features $Q_i$.

In remote sensing continual learning, CPP (Yuan et al., 2024) deploys a MaskFormer-derived encoder, projecting 2D input $\mathbf{x} \in \mathbb{R}^{C\times H\times W}$ via a base CNN and successive $1\times 1$ convolutions to produce a shared embedding $F$. Both pixel-level and caption-level decoders access $F$ through cross-attention, ensuring feature-level correspondence across tasks.

CMSC (Lu et al., 25 Nov 2025) leverages the CCE as a semantic converter: modality-specific features, e.g., from Lift-Splat (camera) or PointPillars (LiDAR), are projected into a unified semantic space $S$ via learned converters $\Gamma_{\alpha m \rightarrow s}$, enabling downstream communication and fusion irrespective of sensor origin.

2. Cross-Modal Fusion Mechanisms

CCE employs multi-head cross-attention to facilitate joint modality integration. In CoCMT (Wang et al., 13 Mar 2025), learnable queries form the $Q$ inputs, while $K$ and $V$ are constructed from concatenated modality-specific features using learned linear projections: $$K = W_K^c F^\mathrm{cam} + W_K^l F^\mathrm{lidar}, \qquad V = W_V^c F^\mathrm{cam} + W_V^l F^\mathrm{lidar}$$ Standard sinusoidal positional encodings are injected into modality-specific sequences. The multi-head mechanism ensures that visual and depth cues are jointly embedded into the latent feature queries.

In panoptic continual perception, CPP (Yuan et al., 2024) permits both segmentation (mask queries) and image-level captioning decoders to attend to a shared embedding $F$. Multi-head self- and cross-attention within each decoder allows joint representation learning and cross-modal semantic alignment.

CMSC (Lu et al., 25 Nov 2025) utilizes cross-modal semantic projection to bring modality-specific features into a joint semantic space, where subsequent fusion is realized both at the communication encoding level and during multi-agent aggregation.

3. Object-Centric Collaboration and Information Exchange

CCE architectures transmit object-centric intermediate representations for collaborative fusion. In CoCMT (Wang et al., 13 Mar 2025), each CAV selects top-$k$ queries based on the highest object-classification scores. Selected queries, along with predicted centers and class scores, are exchanged over the V2V network after coordinate alignment using Motion-Aware Layer Normalization (MLN). The spatial transformation ensures all queries are referenced to the receiver ego-vehicle’s frame. Aggregation is carried out by concatenating ego and remote queries, zero-padding to a fixed maximum agent count, and forming a unified $$Q_\mathrm{all} \in \mathbb{R}^{(A \cdot N) \times D}$$.

CMSC (Lu et al., 25 Nov 2025) employs sparse feature selection via an importance map for bandwidth-critical communication, gathering only top-$K$ locations for transmission. Semantic encoding, channel modulation, equalization, and decoding are all performed in the unified semantic space, facilitating cross-agent fusion and robust recovery, even under channel noise and heterogeneous sensor configurations.

4. Masked Attention and Query Selection

CCE leverages masked self-attention to enforce communication and fusion efficiency. CoCMT (Wang et al., 13 Mar 2025) introduces the Efficient Query Transformer (EQFormer), where a composite mask $M_\mathrm{all} \in \{0,1\}^{(AN) \times (AN)}$ guides self-attention across queries:

• Query Selective Mask ($M_\mathrm{QSM}$): suppresses zero-padded slots
• Proximity-Constrained Mask ($M_\mathrm{PCM}$): enforces spatial locality using a threshold $\tau$ (e.g., $\|C_i-C_j\|>10m$)
• Score-Selective Mask ($M_\mathrm{SSM}$): drops queries below confidence $\theta$ (e.g., $\theta=0.20$)

The self-attention operation is modified via the mask: $$\text{Attention}(Q,K,V;M) = \text{softmax}\left( \frac{QK^T}{\sqrt{d_k}} + M \right) V$$

Ablation studies show substantial accuracy drops with mask removal and less effective information fusion.

5. Training Objectives and Deep Supervision

CCE models employ multi-task supervision and intermediate representation distillation. In CoCMT (Wang et al., 13 Mar 2025), synergistic deep supervision (SDS) applies at every layer of both the single-agent encoder and collaborative EQFormer blocks. Predictions from intermediate query slots are matched via bipartite matching, with loss defined as a weighted sum of cross-entropy (classification) and $L_1$ (regression), weights typically unity ($w_\mathrm{sin}=w_\mathrm{co}=1$).

CPP (Yuan et al., 2024) uses a composite loss: $$L = L_\mathrm{cls} + L_\mathrm{seg} + \lambda L_\mathrm{cap} + L_\mathrm{TKD}$$, where $\lambda$ balances segmentation vs. captioning, and $L_\mathrm{TKD}$ encodes task-interactive knowledge distillation through intermediate latent alignment. Both instance and semantic mask losses are considered, with $\eta_1=20.0$, $\eta_2=1.0$ in the focal and dice components.

CMSC (Lu et al., 25 Nov 2025) applies cycle-consistency losses during semantic converter pre-training, with mean squared error terms enforcing modality-space alignment, alongside standard detection objectives.

6. Bandwidth Optimization and Practical Deployment

A central objective of CCE is communication efficiency. CoCMT (Wang et al., 13 Mar 2025) demonstrates that transmitting only top-$k$ object-centric queries ($k_\mathrm{inf}=50$, $D=256$, $L\approx10$) results in bandwidth costs of approximately 0.43 Mb per agent, representing an $83\times$ reduction compared to BEV feature map schemes (e.g., V2X-ViT, HEAL). Experimental validation on OPV2V and V2V4Real confirms detection performance gains with drastic bandwidth savings.

CMSC (Lu et al., 25 Nov 2025) further evidences robustness in harsh channel conditions and sensor heterogeneity. Importance-aware selection and unified semantic encoding allow consistent 3D detection AP performance over noisy channels, outperforming standard codec pipelines and vanilla JSCC approaches.

7. Empirical Performance and Ablation Insights

CCE frameworks have been statistically validated across perception tasks:

• CoCMT (Wang et al., 13 Mar 2025): On V2V4Real, AP70 increases by +1.1 points (0.471 vs. 0.419) compared to SOTA feature-map baselines, with communication reduced to 0.416 Mb.
• Mask ablations in CoCMT show proximity and confidence masking are essential; removing them causes losses of 2–5 points (AP70).
• CPP (Yuan et al., 2024): Joint training of segmentation and captioning via CCE improves both panoptic quality (PQ: +3.33 pts over MaskFormer-CL) and BLEU scores.
• CMSC (Lu et al., 25 Nov 2025): Heterogeneous sensor pairs maintain $\mathrm{AP}@0.7$ up to 0.904 (LiDAR–LiDAR) and 0.866 (LiDAR–Camera) in 20 dB AWGN; CMSC exhibits smoother AP curves across SNR than baseline protocols.

A plausible implication is that object-query-based cross-modal encoders, when coupled with masked attention and joint training, can simultaneously achieve high perception accuracy and near-optimal communication cost in collaborative settings.

---

Tables summarizing modality, architecture, and communication bandwidth (with data from the references):

Architecture	Backbone(s)	Bandwidth (per CAV)	Application Domain
CoCMT CCE (Wang et al., 13 Mar 2025)	ResNet-50 (Cam), PointPillar (LiDAR)	0.416 Mb	V2V collaborative perception
CPP CCE (Yuan et al., 2024)	MaskFormer (CNN/Transformer)	N/A	Continual remote sensing
CMSC CCE (Lu et al., 25 Nov 2025)	Lift-Splat/PointPillar + Converter	Adaptive (λ≈0.06)	Heterogeneous V2V comms

Conclusion

The Collaborative Cross-Modal Encoder (CCE) represents a generalizable paradigm for multi-agent, multi-modal perception and communication. Through the integration of specialized modality encoders, joint cross-modal fusion, object-centric query collaboration, and masked attention, CCE achieves order-of-magnitude reductions in information exchange bandwidth without sacrificing downstream task performance. Empirical evaluations confirm that carefully instantiated CCE modules are crucial for practical, scalable deployment in distributed autonomous systems, continual multi-task learning, and heterogeneous vehicular communication.