---
title: 'LAMMA: Length-Adaptive Fusion for Multimodal Perception'
url: https://www.emergentmind.com/topics/length-adaptive-multi-modal-fusion-lamma
type: topic
---

# LAMMA: Length-Adaptive Fusion for Multimodal Perception

Searching arXiv for the specified papers and directly related multimodal fusion work.
Length-Adaptive Multi-Modal Fusion (LAMMA) is an attention-based fusion mechanism introduced within Single-Modality-Operable Multimodal Collaborative Perception (SiMO) to maintain multimodal collaborative perception under arbitrary sensor failures while preserving a unified semantic space across full, partial, and single-modality inputs [2603.08240]. In SiMO, LAMMA addresses a specific failure mode of conventional multimodal fusion: once feature fusion maps modality-specific representations into a new joint space, downstream modules trained on that fused representation are no longer compatible with the original single-modality features when one sensor becomes unavailable. LAMMA is designed to fuse any subset of modalities using the same structural logic, keep the feature space before and after fusion identical through additive fusion in a unified space, process a variable number of modalities without changing parameters or control flow, and behave like a parallel circuit in which the system remains operational as long as one branch is healthy [2603.08240].

## 1. Problem formulation and motivation

In SiMO, the central requirement is that a multimodal collaborative perception system must continue functioning when any subset of sensors fails, while preserving each modality’s ability to function alone, avoiding semantic mismatch between single-modality features and the joint fusion and detection head, and enabling heterogeneous agents with different available modalities to collaborate in a unified feature space [2603.08240].

The motivating diagnosis is that common multimodal fusion induces a semantic space shift after fusion. The paper formalizes this as
\[
Z_{\text{mm}} = \phi\big(Z_L, Z_C\big),
\]
where \(Z_L\) and \(Z_C\) denote LiDAR and camera features, and \(Z_{\text{mm}}\) lies in a different space than either input feature. Task heads \(h(\cdot)\) are trained on \(Z_{\text{mm}}\); if one modality fails, the remaining modality feature does not constitute a valid input in that fused semantic space, so the head fails [2603.08240]. This is compounded by a hard dependency on all modalities, described as a series-circuit behavior, and by the collaborative setting in which different agents may retain different modalities, yet their BEV features must remain in a strictly unified semantic space for multi-agent fusion such as AttFusion or Pyramid Fusion [2603.08240].

This formulation distinguishes LAMMA from earlier adaptive multimodal fusion work that treated fusion as a learned function rather than a deterministic operator. "Adaptive Fusion Techniques for Multimodal Data" characterizes concatenation, sum, and tensor-product style constructions as static fusion, and proposes Auto-Fusion and GAN-Fusion as adaptive alternatives that learn how to combine modalities in latent space [1911.03821]. That work is not framed around collaborative perception or sensor failure, but it establishes the broader principle that multimodal fusion can itself be a learned transformation rather than a fixed operator [1911.03821]. A plausible implication is that LAMMA occupies a more constrained design point: it is not merely adaptive in weighting or compression, but explicitly engineered to preserve semantic compatibility across modality subsets.

## 2. Placement within the SiMO pipeline

In the SiMO architecture, each modality \(m \in \{L, C, \dots\}\) has a modality-specific encoder \(f_m\) and a modality-specific aligner \(g_m\). For the LiDAR and camera instantiations reported in the paper, the encoders are PointPillars for LiDAR and LSS for cameras, while the aligners are ConvNeXt blocks [2603.08240]. The aligned modality feature is
\[
Z_m = g_m\big(f_m(X_m)\big), \quad m \in \{L, C\}.
\]

LAMMA is applied after per-modality encoding and alignment:
\[
Z_{\text{mm}} = \phi(Z_L, Z_C),
\]
with \(\phi\) instantiated as LAMMA [2603.08240]. Multi-agent fusion \(\lambda\), such as AttFusion or Pyramid Fusion, is then performed on the fused BEV features from each agent:
\[
\hat{Y} = h\big(\{\lambda(Z_{\text{mm},i})\}_i\big).
\]
This makes LAMMA a middle-fusion module positioned between per-modality BEV formation and inter-agent fusion [2603.08240].

Its role in training is equally specific. Under the PAFR (Pretrain–Align–Fuse–RD) schedule, LAMMA is not learned jointly from scratch with all branches. It is trained only after single-modality pretraining of each branch and per-modality alignment training of the aligners, and is followed by fine-tuning with Random Modality Drop (RD) [2603.08240]. This staging reflects the paper’s claim that naive joint training induces modality competition, especially causing LiDAR dominance and weak camera-only behavior. Accordingly, LAMMA supplies the structural mechanism for semantics-consistent fusion, while PAFR supplies the optimization regime needed to maintain branch independence [2603.08240].

## 3. Fusion architecture and length adaptivity

After encoder and aligner processing, each modality feature is represented as
\[
Z_m \in \mathbb{R}^{B \times N \times d},
\]
where \(B\) is batch size, \(N\) is the number of BEV tokens, and \(d\) is feature dimension [2603.08240]. Before attention, SiMO adds a modality-agnostic positional embedding,
\[
Z'_m = Z_m + P, \quad P \in \mathbb{R}^{1 \times N \times d},
\]
and may project features through \(W_p : \mathbb{R}^{d} \to \mathbb{R}^{d}\) to reduce Transformer compute [2603.08240].

LAMMA is defined as attention-based and set-size invariant. For two modalities \(A\) and \(B\), queries are constructed by concatenating modality tokens along the length dimension:
\[
Q = W_Q [Z_A; Z_B] \in \mathbb{R}^{B \times 2N \times d}.
\]
For each modality \(m\), keys and values are
\[
K_m = W_K Z_m \in \mathbb{R}^{B \times N \times d}, \qquad
V_m = W_V Z_m \in \mathbb{R}^{B \times N \times d}.
\]
The module then applies per-modality multi-head attention:
\[
Z^{\text{att}}_m = \mathrm{MHA}(Q, K_m, V_m) \in \mathbb{R}^{B \times 2N \times d}.
\]
Because \(Q\) contains queries from both modalities while \(K_m\) and \(V_m\) derive from only one modality, the same MHA call simultaneously contains self-attention and cross-attention effects [2603.08240].

The output \(Z^{\text{att}}_m\) is split into modality-origin segments and summed:
\[
Z^{\text{fused}}_m = \mathrm{Sum}\big(\mathrm{Split}(Z^{\text{att}}_m)\big) \in \mathbb{R}^{B \times N \times d}.
\]
The final multimodal feature is then obtained by addition:
\[
Z_{\text{mm}} = Z^{\text{fused}}_A + Z^{\text{fused}}_B.
\]
In the general case,
\[
Z_{\text{mm}} = \sum_{m \in \mathcal{M}} Z^{\text{fused}}_m.
\]
The paper identifies this final addition as the critical operation that preserves the feature space of inputs and outputs, in contrast to concatenation-plus-convolution schemes that create a new semantic coordinate system [2603.08240].

The “length-adaptive” designation has two explicit meanings. First, the query length is proportional to the number of available modalities \(|\mathcal{M}|\), since queries are formed by concatenating all available modality tokens. If some modalities are absent, their tokens are simply omitted; no branch structure or learnable parameter changes [2603.08240]. Second, when one modality fails, the mechanism implicitly degenerates to self-attention on the remaining modality. For example, if modality \(A\) is absent, the paper describes
\[
Z^{\text{fused}}_m
=
\mathrm{Softmax}\left(\frac{Q_B K_m^\top}{\sqrt{d}}\right)V_m
+
\mathrm{Softmax}\left(\frac{Q_A K_m^\top}{\sqrt{d}}\right)V_m,
\]
and with \(Q_A \approx 0\), the second term vanishes, yielding
\[
Z^{\text{fused}}_B \approx \mathrm{SA}(Q_B, K_B, V_B).
\]
Thus, the module preserves the same control flow and parameterization under modality removal, while changing only the effective token length [2603.08240].

## 4. Semantic alignment and modality preservation

LAMMA depends on a pre-fusion semantic alignment stage because direct fusion of raw LiDAR and camera features produces large semantic discrepancies arising from different distributions and geometric structure [2603.08240]. SiMO therefore applies ConvNeXt-based aligners \(g_L\) and \(g_C\), each implemented as a 3-block ConvNeXt module, to align channel-wise and pixel-wise semantics and map modality features into a LAMMA-compatible space:
\[
Z_m = g_m\big(f_m(X_m)\big).
\]
These aligners are trained per modality, with the remaining components frozen, so that each modality is brought into a compatible space without competition from the others [2603.08240].

The paper identifies two design decisions as responsible for semantic consistency before and after fusion. The first is shared processing architecture inside fusion: queries are treated equally regardless of modality, keys and values are processed via shared linear layers \(W_K\) and \(W_V\), and the Transformer block sees all modalities symmetrically [2603.08240]. The second is additive fusion in a unified space:
\[
Z_{\text{mm}} = \sum_m Z^{\text{fused}}_m.
\]
Because the aligned per-modality features share a semantic basis, the channel meaning is preserved under addition; only activation content changes [2603.08240].

The paper provides both quantitative and qualitative evidence for this claim. In Procrustes analysis, disparity between LiDAR and camera features decreases from \(0.6747\) before LAMMA to \(0.0472\) after LAMMA alignment [2603.08240]. The reported t-SNE visualizations show that LiDAR, camera, and fused clusters remain modality-separate, which the authors interpret as preservation of modality-specific information, while their internal topologies exhibit mirror symmetry after LAMMA, indicating aligned semantic structure across modalities and fused features [2603.08240].

This combination of alignment without identity collapse distinguishes LAMMA from fusion schemes that aggressively homogenize modality representations. A plausible implication is that LAMMA is intended not to erase modality provenance, but to make modality-specific representations interoperable under a shared downstream head.

## 5. Training regime and modality independence

LAMMA is coupled to the PAFR training strategy because the paper argues that fusion design alone does not solve modality competition [2603.08240]. The four stages are:

| Stage | Operation | Purpose stated in the paper |
|---|---|---|
| Pretrain | Train each encoder \(f_m\) separately with its own detection head | Learn strong single-modality features without interference |
| Align | Freeze encoders and train aligners \(g_m\) one by one | Map each modality into a LAMMA-compatible space without competition |
| Fuse | Freeze encoders and aligners; train LAMMA and shared downstream modules on multimodal inputs | Make downstream modules modality-independent |
| RD | Fine-tune with random modality drop | Expose fusion and downstream heads to available-modality combinations |

In RD, the reported sampling probabilities are \(0.5\) for multimodal input, \(0.25\) for LiDAR-only, and \(0.25\) for camera-only in the basic setting, with additional patterns in heterogeneous settings [2603.08240]. The paper states that this stage strengthens single-modality operability by randomly dropping one modal feature in LAMMA and thereby exposing the shared downstream modules to the full range of modality availability patterns [2603.08240].

The ablation findings are sharply stated. Naive training, with or without RD, cannot adapt to modal failure; SiMO without LAMMA, using BEVFusion-style fusion, cannot adapt even with RD; and only the combination of PAFR, LAMMA, and RD yields high performance in multimodal, LiDAR-only, and camera-only conditions simultaneously [2603.08240]. In a normalized ablation table, the reported AP@70 values for the complete design are \(0.95\) for L+C, \(0.94\) for L-only, and \(0.45\) for C-only [2603.08240].

This training structure has a conceptual parallel with earlier adaptive fusion work that combined a fusion-specific loss with a task loss. In Auto-Fusion, the fused representation is constrained by a reconstruction objective \(\mathcal{J}_{\text{tr}} = \|z_m - \hat{z}_m\|_2^2\), while GAN-Fusion uses adversarial alignment losses, both optimized jointly with the downstream task [1911.03821]. LAMMA does not adopt those losses; instead, it relies on staged training and modality drop. This suggests that LAMMA’s contribution is chiefly architectural and optimization-structural rather than based on explicit latent-space regularization beyond the alignment modules [2603.08240; 1911.03821].

## 6. Empirical behavior, comparisons, and limitations

The principal empirical evaluation reported for LAMMA is on OPV2V-H. Under normal operation, SiMO-AF with RD achieves AP@70 values of \(84.81\) for L+C, \(82.35\) for L-only, and \(22.59\) for C-only; SiMO-PF with RD achieves \(94.64\) for L+C, \(80.81\) for L-only, and \(44.82\) for C-only [2603.08240]. The camera-only result for SiMO-PF exceeds the corresponding Pyramid Fusion camera-only result of \(39.07\), despite using the same camera encoder, which the paper interprets as evidence that LAMMA refines modality features through attention-based enhancement rather than merely combining them [2603.08240].

Under LiDAR degradation, the paper reports that Pyramid Fusion and BEVFusion degrade sharply and eventually fail as LiDAR points are reduced, whereas SiMO-PF degrades more slowly and converges to its camera-only performance as LiDAR becomes unusable [2603.08240]. In heterogeneous modal-failure patterns across agents, SiMO-PF matches or exceeds HEAL in AP@30/50/70 for both LiDAR-ego and camera-ego cases, with only minor drop from homogeneous to heterogeneous failures, which the authors attribute to cross-agent semantic consistency provided by alignment and LAMMA [2603.08240].

The comparison to other fusion strategies is explicit. Relative to concatenation plus convolution, exemplified by BEVFusion, LAMMA avoids constructing a new fused space incompatible with single-modality features and inherently supports a variable number of modalities [2603.08240]. Relative to Transformer-based fusion without length-adaptive design, LAMMA builds queries from only available modalities, uses the same structure regardless of modality presence, and degrades automatically to self-attention [2603.08240]. Relative to UniBEV’s channel-normalized weighted sum, LAMMA is presented as compatible with heterogeneous encoders such as PointPillars and LSS, rather than depending on identical encoder architectures, and as directly addressing multi-agent heterogeneous failure [2603.08240].

Two limitations are expressly noted. First, because LAMMA performs fusion through attention plus addition without convolutional smoothing, noisy modalities can propagate noise more directly; the paper contrasts this with BEVFusion’s convolutional fusion, which can implicitly denoise [2603.08240]. Second, the PAFR schedule increases training complexity and training time relative to naive end-to-end optimization [2603.08240].

Related multimodal fusion designs in other domains illustrate that “adaptive” need not mean the same thing across settings. In 4DRVO-Net, adaptive multimodal fusion is realized as an Adaptive Radar–Camera Fusion Module (A-RCFM) that uses radar-conditioned deformable attention to select image features around projected radar points at multiple scales [2308.06573]. That mechanism is adaptive in a spatially varying, point-wise, and scale-wise sense, whereas LAMMA is adaptive in the number of available modalities and in preserving semantic consistency under missing-modality conditions [2308.06573; 2603.08240]. This suggests that “length adaptivity” in LAMMA refers specifically to token-length variation induced by modality subsets, not to temporal sequence length or geometric scale per se.

## 7. Broader interpretation and relation to adaptive fusion literature

Within SiMO, LAMMA can be defined succinctly as a length-adaptive, attention-based, additive fusion module operating on aligned BEV features from arbitrary modality subsets, with modality-agnostic parameters and invariant downstream semantics [2603.08240]. Its distinguishing property is not simply that it learns fusion weights, but that it makes a multimodal collaborative perception system single-modality-operable by preserving semantic compatibility between single-modality and multimodal representations [2603.08240].

The wider literature supplies useful contrast. Auto-Fusion and GAN-Fusion propose lightweight adaptive networks that learn to compress and regularize multimodal latent spaces for tasks such as multimodal machine translation and emotion recognition [1911.03821]. Those methods treat adaptivity primarily as learned compression or adversarial latent alignment over fixed-dimensional modality embeddings. By contrast, LAMMA applies multi-head attention to BEV token sets and ties adaptivity to the variable cardinality of the modality set \(\mathcal{M}\), while enforcing additive closure in a shared semantic space [2603.08240]. 4DRVO-Net, in turn, demonstrates another form of adaptive fusion in which deformable cross-attention is applied point-wise and multi-scale to fuse sparse radar with dense camera features for odometry [2308.06573]. 

Taken together, these works indicate that adaptive multimodal fusion can be organized along at least three axes present in the cited papers: learned latent compression and alignment [1911.03821], spatially varying cross-modal sampling and confidence weighting [2308.06573], and semantic-space-preserving fusion under variable modality availability [2603.08240]. LAMMA belongs to the third category. Its specific contribution is to make the fused representation structurally compatible with downstream collaborative perception modules across multimodal, partial-modality, and single-modality operation, thereby serving as the architectural basis for single-modality-operable multimodal collaborative perception [2603.08240].

Source: https://www.emergentmind.com/topics/length-adaptive-multi-modal-fusion-lamma