SiMO: Single-Modality-Operable Multimodal Perception

• SiMO is a collaborative perception framework that remains operational when one sensor modality fails, ensuring independent usability of each branch.
• It enforces semantic consistency between unimodal and fused features through the LAMMA module, preserving a common semantic space.
• The PAFR training strategy with Random Drop fine-tuning enables resilience under heterogeneous sensor conditions, outperforming conventional methods.

Single-Modality-Operable Multimodal Collaborative Perception (SiMO) is a collaborative perception framework for multimodal, multi-agent systems in which perception remains operational when one sensor modality is missing or broken, especially in vehicle-to-vehicle and vehicle-to-infrastructure settings. The term is introduced in "SiMO: Single-Modality-Operable Multimodal Collaborative Perception" [2603.08240], which argues that conventional multimodal collaborative perception pipelines are brittle under modal failure because downstream heads are trained to expect a fused multimodal representation whose semantic structure changes when one input branch disappears. SiMO addresses this by enforcing semantic consistency between unimodal and multimodal features, preserving the standalone usability of each modality branch, and supporting heterogeneous modal availability across agents.

1. Definition and problem setting

SiMO targets a specific operating condition in collaborative perception: a multimodal system should continue functioning when at least one modality remains available. In the formulation of [2603.08240], the desired behavior is not that every agent always possesses all sensors, but that the collaborative system remains effective under missing-modality conditions, including heterogeneous failures across agents. This differs from conventional multimodal collaborative perception, where local features are extracted from each sensor, transformed to BEV, shared, and fused, and where multimodal performance often depends on the assumption that all fusion inputs remain present.

The central claim of SiMO is that standard multimodal collaborative perception behaves like a "series circuit": if any modality fails, the entire perception stack becomes unreliable because the downstream modules were optimized for fused multimodal features rather than for unimodal branch outputs [2603.08240]. SiMO instead seeks "parallel circuit" behavior, in which each modality branch remains independently usable and all branches, whether isolated or fused, inhabit a compatible semantic space.

This operating principle is closely related to earlier reduced-modality and missing-modality work, but SiMO is more specific than generic multimodal robustness. CAML, for example, explicitly supports multimodal learning during training and reduced-modality inference at test time in multi-agent systems, with training-time collaboration and testing-time reduced modality sets [2502.17821]. BM2CP likewise remains usable when an agent lacks either LiDAR or camera, but it is a LiDAR-camera-specific collaborative perception design rather than a general semantic-consistency framework for unimodal and multimodal operation [2310.14702]. A plausible implication is that SiMO formalizes and systematizes a requirement that earlier methods addressed only partially or within narrower architectural assumptions.

2. Failure mode in conventional multimodal collaborative perception

The SiMO paper identifies two root causes of modal-failure brittleness. The first is feature-space mismatch between fused and unfused inputs. Common fusion operators such as concatenation, CNN-based fusion, graph fusion, attention, and Transformer-based fusion either merge modality dimensions into a new representation or map inputs into a fused latent space. As a result, the task heads are trained on post-fusion features; when one modality disappears at inference time, the remaining unimodal feature no longer matches the semantic expectations of those heads [2603.08240].

The second root cause is modality competition. Joint multimodal training often favors the easier modality, with LiDAR typically dominating because 3D geometry is more directly available from point clouds than from images. The weaker branch then remains undertrained and unusable on its own. In collaborative perception, this problem is amplified because different agents may lose different modalities, so the system must maintain inter-agent semantic consistency even when one agent contributes LiDAR features and another contributes camera features [2603.08240].

Earlier collaborative perception literature exposed adjacent but distinct failure modes. CAML argues that multimodal data are highly informative during training but may be computationally expensive, latency-heavy, unavailable, or unrealistic at deployment, motivating training with richer multimodal supervision and testing with reduced modalities [2502.17821]. "Latency-Aware Collaborative Perception" studies another degradation source: stale asynchronous features caused by communication latency, showing that naive fusion of delayed collaborator features can become worse than no collaboration at high latency [2207.08560]. BM2CP addresses partial missing-sensor conditions in LiDAR-camera collaboration, but explicitly notes that camera alone cannot accomplish the task independently in its setting, especially when each agent has only one camera [2310.14702]. These results collectively suggest that collaborative perception robustness depends not only on fusion strength under ideal sensing, but also on maintaining usable representations under sensor absence, asynchronous communication, and heterogeneous agent capability.

3. Architectural principle: semantic alignment and LAMMA

SiMO resolves the fused-versus-unfused mismatch by aligning modality-specific features into a unified semantic space before multimodal fusion. Its architecture is summarized as
$$
Z_m = g_m\left[f_m(X_m)\right], \quad m \in {L, C}
$$
where $X_L$ and $X_C$ are LiDAR and camera inputs, $f_m$ denotes the modality-specific feature extractor, $g_m$ the modality alignment module, and $Z_m$ the aligned BEV feature for modality $m$. The multimodal feature is then produced by
$$
Z_{mm} = \phi(Z_L, Z_C),
$$
followed by multi-agent fusion and task prediction,
$$
\hat{Y} = h\left[\psi(Z_{mm,i})\right], \quad i=1,2,\dots
$$
with $\phi$ the multimodal fusion module, $\psi$ the multi-agent fusion module, and $h$ the classification/regression/direction heads [2603.08240].

The key fusion module is Length-Adaptive Multi-Modal Fusion (LAMMA). The "length-adaptive" property refers to the varying number of available modalities: the same module can process two modalities, one modality, or heterogeneous modality configurations across agents without structural changes. Before fusion, SiMO applies separate ConvNeXt-based aligners $g_L$ and $g_C$ to map LiDAR and camera features into a shared semantic space, both channel-wise and pixel-wise. The paper states that modality fusion should happen after alignment; otherwise, the fused space shifts and the task head becomes brittle under missing modalities [2603.08240].

LAMMA then performs additive and attention-based fusion:
$$
Q = W_Q [Z_A; Z_B]
$$
$$
K_m = W_K Z_m
$$
$$
V_m = W_V Z_m
$$
$$
Z^{att}_m = \text{MHA}(Q, K_m, V_m)
$$
$$
Z^{fused}_m = \text{Sum}(\text{Split}(Z^{att}_m))
$$
$$
Z_{mm} = Z^{fused}_A + Z^{fused}_B.
$$
The paper emphasizes addition rather than concatenation for the final multimodal representation. The stated reason is to avoid shifting the feature space: if aligned modality features are combined additively, fused and unimodal features remain compatible with the same downstream heads [2603.08240].

When one modality is absent, LAMMA does not rely on a dedicated failure detector. The missing branch contributes an empty or zero query component, expressed as
$$
Q = [0; Q_B],
$$
so the operation degenerates to a self-attention pathway on the remaining modality:
$$
Z^{fused}_A = 0, \qquad Z^{fused}_B \approx SA(Q_B, K_B, V_B).
$$
This is the mechanism by which SiMO remains single-modality operable under modal failure [2603.08240].

The semantic-consistency claim is supported in the paper by Procrustes analysis and t-SNE visualizations. After LAMMA, the reported Procrustes disparity between camera and LiDAR features drops from (0.6747) to (0.0472), while t-SNE plots show that modality clusters remain distinct even as their internal structures become mirror-symmetric [2603.08240]. This suggests alignment without collapse: modality-specific information is preserved, but the geometric organization of the feature space becomes compatible across modalities.

4. Training strategy: PAFR and Random Drop

SiMO does not change the supervised detection objective. The paper uses the standard 3D detection loss
$$
\mathcal{L}(Y, \hat{Y}) = \mathcal{L}{\text{Focal}(Y^{cls}, \hat{Y}^{cls}) + \mathcal{L}{\text{Smooth-L1}(Y^{reg}, \hat{Y}^{reg})}.
$$
Its novelty lies in how the representations are trained, via the "Pretrain-Align-Fuse-RD" (PAFR) strategy [2603.08240].

The first stage is independent pretraining of each modality branch to convergence. In the reported implementation, PointPillar is used for LiDAR and Lift-Splat-Shot (LSS) for camera. This ensures that each branch learns a complete and usable representation before any joint optimization can induce modality competition [2603.08240].

The second stage aligns modalities into the same semantic BEV space. After pretraining, the feature extractors are frozen. The aligners $g_L$ and $g_C$ are then trained one modality at a time: one aligner is trained with its modality input, frozen, and then the other is trained. The paper presents this staged procedure as a way to avoid competition during alignment and to guarantee that both branches can map into a shared space [2603.08240].

The third stage trains the common modules—LAMMA, multi-agent fusion, and the task heads—with multimodal input while encoders and aligners remain frozen. These common modules are intentionally modality-independent. Because inputs have already been aligned, they can operate without modality-specific bias [2603.08240].

The final stage is Random Drop fine-tuning (RD). The reported probabilities are: both modalities with probability (0.5), LiDAR only with probability (0.25), and camera only with probability (0.25). This stage exposes the shared modules to both complete and partial input configurations. According to the ablations, RD is necessary for missing-modality resilience, but it is effective only after pretraining and alignment have already made each branch competent and semantically compatible [2603.08240].

This training logic differs from CAML’s teacher-student design. CAML trains a teacher with full multimodal access and a student with reduced modalities, using binary cross-entropy for the teacher and a combination of BCE and KL-based distillation for the student,
$$
\mathcal{L}\mathcal{S} = (1-\alpha) \mathcal{L}{BCE}(y, \mathcal{S}) + \alpha t^2 \mathcal{L}_{KD}(\mathcal{S}, \mathcal{T}),
$$
with (\alpha=0.5) and (t=3.0) [2502.17821]. A plausible implication is that CAML emphasizes knowledge transfer from rich multimodal collaboration to a reduced-modality student, whereas SiMO emphasizes semantic-space compatibility so that a single modality can directly drive the same downstream collaborative perception stack.

5. Experimental setting and empirical evidence

SiMO is evaluated on OPV2V-H, described as a heterogeneous collaborative perception dataset built on OPV2V with 73 scenes, 11,464 frames total, 2–7 connected automated vehicles per frame, multi-view RGB cameras, 32-channel LiDAR, and 3D bounding box annotations. The reported metrics are AP@30, AP@50, and AP@70. The camera encoder is Lift-Splat-Shot, the LiDAR encoder is PointPillar, the fusion module is LAMMA, the multi-agent fusion is often AttFusion or Pyramid Fusion, the aligners are ConvNeXt blocks, and optimization uses Adam with batch size 1 on RTX 3090, together with the staged training pipeline described above [2603.08240].

The comparison set includes BM2CP, BEVFusion, UniBEV, AttFusion, HEAL with Pyramid Fusion, and adapted versions of these methods for collaborative perception. The main qualitative result reported in the paper is that standard multimodal collaborative methods such as BM2CP, BEVFusion, and UniBEV fail under LiDAR missing conditions, whereas SiMO operates in LiDAR+camera, LiDAR-only, and camera-only regimes [2603.08240]. The paper further states that SiMO-PF and SiMO-AF retain state-of-the-art-level multimodal performance and become robust to modal failures after RD fine-tuning.

The evidence is not limited to complete LiDAR failure. As LiDAR quality degrades, SiMO reportedly degrades more slowly than the baselines and eventually converges to the camera-only level rather than collapsing. Under heterogeneous failure patterns, SiMO remains strong when different agents lose different modalities and outperforms HEAL in those settings without extra fine-tuning [2603.08240]. The ablations show that removing LAMMA prevents proper adaptation to missing modalities even with RD; removing PAFR or using naive training leaves branches insufficiently independent; and removing RD preserves better multimodal performance but weakens resilience under modal failure [2603.08240].

Related empirical findings from adjacent work contextualize these claims. CAML reports collaborative decision-making gains in CARLA + AutoCast accident-prone scenarios—overtaking, left turn, and red light violation—where training uses RGB + LiDAR for each vehicle and testing uses RGB only, with ADR improvements over AML of 13.2%, 32.6%, and 58.13%, respectively. In aerial-ground collaborative semantic segmentation on CoPeD, with RGB + depth during training and RGB only during testing, CAML reports mIoU values of 60.05 indoor and 66.83 outdoor, improving over AML by 8.88% and 10.61% [2502.17821]. BM2CP, on OPV2V and DAIR-V2X, reports 83.72 / 63.17 [email protected] / [email protected] on OPV2V and 64.03 / 48.99 on DAIR-V2X, while remaining operable when a modality is missing and using markedly lower communication volume [2310.14702]. These results indicate that SiMO emerges within an already active line of work on robust, bandwidth-aware, and missing-modality collaborative perception, but introduces a more explicit account of semantic-space consistency under modal failure.

6. Relation to predecessor methods, scope, and limitations

The following comparison summarizes the positioning of SiMO relative to closely related collaborative perception methods.

Method	Core mechanism	Relation to SiMO
SiMO [2603.08240]	LAMMA + PAFR + RD	Defines single-modality-operable multimodal collaborative perception
CAML [2502.17821]	Collaborative auxiliary modality learning with teacher-student distillation	Supports multimodal training and reduced-modality inference in multi-agent systems
BM2CP [2310.14702]	Cooperative depth generation, LiDAR-guided modal fusion, modality-guided collaborative fusion	LiDAR-camera collaborative perception that remains usable when one sensor is missing
SyncNet [2207.08560]	Feature-Attention Symbiotic Estimation + Time Modulation	Latency-aware feature synchronization, not a multimodal missing-modality framework

CAML is SiMO-like in the sense that it realizes the principle "train with more modalities and collaboration, deploy with fewer modalities," and it explicitly allows agents to collaborate and share multimodal data during training while operating on reduced modality sets at testing [2502.17821]. However, CAML is broader than a strict single-agent single-modality framing because each agent may have one modality or a subset, and the overall system can still construct a multimodal representation from multiple agents’ partial observations. BM2CP is likewise SiMO-compatible because it can cope with the special case where one of the sensors, same or different type, of any agent is missing, and it does so without retraining or fine-tuning for missing-modality cases [2310.14702]. Yet BM2CP is explicitly LiDAR-camera-specific, strongly LiDAR-guided, and the paper states that camera alone cannot reliably accomplish the task in its setting.

SyncNet addresses a different axis of robustness. It studies asynchronous communication latency rather than multimodal sensing, using a latency compensation module that estimates synchronized collaborator features and attention from delayed history buffers. It therefore supports collaborative perception when partner information is delayed or effectively missing at the current time stamp, but it does not address true single-modality multimodal inference or missing modality translation and reconstruction [2207.08560].

Several limitations and open constraints are also explicit in the source material. BM2CP degrades under pose noise because depth projection depends on localization accuracy, which motivates its BM2CP-robust variant [2310.14702]. CAML notes that modal misalignment can hurt fusion, auxiliary modalities or views may become noise or useless, system complexity increases with the number of agents, and communication overhead may cause delays [2502.17821]. SyncNet assumes discrete collaboration frames, a known latency value, and locally stored historical feature buffers, and it offers no explicit guarantee for completely absent collaborator streams [2207.08560]. For SiMO itself, the paper’s claims concentrate on maintaining semantic consistency and branch independence under modal failure; a plausible implication is that its effectiveness depends on the quality of alignment, staged training, and the availability of at least one competent modality branch per agent [2603.08240].

Within the collaborative perception literature, SiMO therefore denotes more than missing-sensor tolerance in the narrow engineering sense. It names a design criterion: unimodal branch outputs, multimodal fused features, and multi-agent aggregated features should remain semantically compatible so that downstream heads do not depend on the presence of any particular sensor. In that formulation, SiMO is both a specific architecture—LAMMA with PAFR and RD—and a broader principle for robust multimodal collaborative perception under heterogeneous sensor failure [2603.08240].