Lag Fusion Mamba: Efficient 3D Multi-modal Fusion

• Lag Fusion Mamba is a state-space model-based fusion architecture that integrates camera and LiDAR data with precise height-fidelity encoding.
• It employs a hybrid Mamba block with local and global modules to maintain spatial continuity and achieve dense, efficient 3D object detection.
• Empirical evaluations demonstrate state-of-the-art NDS and mAP performance while significantly reducing computational complexity compared to traditional methods.

Lag Fusion Mamba is a class of state-space model–based fusion architectures designed for efficient, dense, and globally contextual multi-modal integration, with particular emphasis on camera–LiDAR 3D object detection. It leverages the linear-complexity selective state-space modeling first formalized in Mamba, augmented with innovations that preserve spatial fidelity across modalities and eliminate typical pitfalls in sequence alignment and global receptive field. This approach achieves state-of-the-art detection performance and efficiency by tightly coupling advanced spatial encoding (notably, height-fidelity LiDAR representation) with a hybrid local-global Mamba fusion block, ensuring that each 3D token—regardless of modality—participates in omniscient, order-corrected interactions using only linear time and memory resources (Wang et al., 6 Jul 2025).

1. Motivation and Problem Context

Camera–LiDAR fusion for 3D object detection presents inherent challenges: existing approaches struggle to achieve efficient, dense, global context modeling while preserving geometric fidelity, notably precise vertical (height) information. Discretized voxelization in common LiDAR backbones introduces height quantization error, misordering sequences when features are serialized, and consequently impedes the effectiveness of recurrence-based or attention-based fusion. Furthermore, quadratic-complexity models (e.g., Transformers) are computationally prohibitive for real-time deployment on large-scale 3D scenes, while window or local-only methods cannot fully exploit long-range inter-modal dependencies.

Lag Fusion Mamba addresses these deficiencies via (i) height-fidelity LiDAR encoding to restore 3D sequence alignment, and (ii) a linear-complexity, hybrid Mamba fusion block for both local and global contextual learning—all within a pure state-space model (SSM) framework (Wang et al., 6 Jul 2025).

2. Height-Fidelity LiDAR Encoding

Conventional LiDAR encoders aggregate 3D points into grid voxels; downsampling averages grid centers, compounding discretization error—especially in the z-axis. Height-fidelity LiDAR encoding instead computes voxel coordinates directly in continuous 3D space via scatter-mean operations on true point coordinates at each backbone stage. Mathematically, for stage $s$:

$$\mathcal{C}_{P_s}[q] = \mathrm{ScatterMean}( \mathcal{C}_{P_{s-1}}, \boldsymbol{v}_s )$$

where $\mathcal{C}_{P_s}$ is the matrix of continuous voxel coordinates and $\boldsymbol{v}_s$ denotes the depth-$s$ voxel assignment. This process guarantees that vertical geometry is preserved exactly through downsampling, ensuring downstream serialization (e.g., via a 3D Hilbert curve) reflects accurate spatial locality.

Conflict-test filtering excludes any voxel whose coordinates have already been assigned, especially when handling top-$k$ salient tokens that risk spatial overlap—a crucial operation in models such as LION.

Correct sequence ordering is restored by computing the Hilbert index $h_i = \mathrm{HilbertIndex}(x_i,y_i,z_i)$ for each LiDAR or camera token, establishing a fixed 1D serialization that faithfully reflects 3D position (Wang et al., 6 Jul 2025).

3. Hybrid Mamba Block: Architecture and Formalism

The Hybrid Mamba Block (HMB) processes multi-modal tokens (LiDAR and camera) concatenated in a unified coordinate system. It comprises three sub-components:

1. Modality Alignment: A shared Mamba block normalizes latent feature distributions.
2. Local Mamba Module: Partitions the spatial domain into non-overlapping $w \times w$ windows, serializes tokens within each window, and applies a localized SSM recurrence.
3. Global Mamba Module: Serializes all tokens across modalities via the 3D Hilbert curve (injecting learned positional embeddings), then applies a bidirectional Mamba recurrence for omniscient fusion.

The generic SSM recurrence is:

$$\mathbf{h}_n = \bar{\mathbf{A}} \mathbf{h}_{n-1} + \mathbf{B} \mathbf{x}_n, \qquad \mathbf{y}_n = \mathbf{C}^\top \mathbf{h}_n + \mathbf{D} \mathbf{x}_n$$

where $$\bar{\mathbf{A}}, \mathbf{B}, \mathbf{C}, \mathbf{D}$$ are learned projection matrices, and the recurrence operates linearly over the sequence (Wang et al., 6 Jul 2025).

Processing steps—expressed in algorithmic sketch—are:

def HybridMambaBlock(F_tokens, coords): F_aligned = MambaLayer(F_tokens, coords) partition_info = compute_windows(coords, w) F_local = LocalMamba(F_aligned, partition_info) h_indices = HilbertIndex(coords) F_pos = F_local + PosEmbedding(coords) F_global = BidirectionalMamba(F_pos, h_indices) return F_global + F_local + F_aligned

The block can be denoted:

$$\mathrm{HMB}(\mathbf{F}, \mathcal{C}) = \mathrm{GlobalMamba} ( \mathrm{LocalMamba}(\mathbf{F}, \mathcal{P}_r) + \mathbf{F}, \mathcal{P}_h )$$

ensuring every token undergoes both local and global contextualization, with cross-modal and spatial dependencies handled in a unified streaming fashion.

4. Dense Global Fusion and Complexity Analysis

Chaining local and global Mamba modules yields dense, global, all-to-all fusion in linear time and space. The total cost is $O(N C^2)$ for $N$ tokens and feature dimension $C$—a dramatic improvement over O($N^2 C$) for quadratic self-attention approaches. There is no windowing, no arbitrary sampling, and no loss of spatial contiguity. Empirical evaluation confirms the method's efficiency: MambaFusion-Base achieves 4.7 FPS on $704 \times 256$ nuScenes input, outpacing previous Transformer-based and hybrid baselines (Wang et al., 6 Jul 2025).

5. Empirical Evaluation and Benchmarking

Lag Fusion Mamba demonstrates state-of-the-art performance on the nuScenes 3D detection benchmark. Notable validation results include:

Method	Resolution	NDS (val)	mAP (val)	FPS
UniTR (SOTA)	704×256	73.3	70.5	4.9
IS-FUSION	1056×384	74.0	72.8	3.2
SparseLIF	1600×640	74.6	71.2	2.9
MambaFusion-Lite	704×256	74.0	71.6	5.4
MambaFusion-Base	704×256	75.0	72.7	4.7

This architecture attains new single-frame record NDS and mAP at a fraction of the computation—setting benchmarks for both accuracy and speed (Wang et al., 6 Jul 2025).

6. Analysis of Alignment and Failure Modes

Purely replacing window-based Transformers with generic linear SSMs does not suffice. Without height-fidelity encoding, sequence misalignment and quantization error cause improper mixing of 3D tokens, degrading downstream performance (mAP drops by up to 4 points). The explicit preservation of $z$-coordinates via continuous-space voxel compression and Hilbert-based serialization is necessary to attain optimal fusion. Ablative studies confirm that height-fidelity encoding alone recovers ~1.7 mAP and 1.0 NDS, while the hybrid block further boosts detection.

Potential limitations include the fixed nature of Hilbert sequence mapping—which may degrade under severe occlusion—and the memory overhead of raw point associations for continuous-space encoding.

7. Comparative Insights and Broader Significance

Lag Fusion Mamba provides a pure SSM alternative to other fusion design patterns:

• Dense Local windows (e.g., BEVFusion) are context-limited and yield elevated false-positive rates on large/distant objects.
• Sparse Global approaches (e.g., SparseFusion) discard background information, leading to misclassification in cluttered scenes.
• Quasi Dense (e.g., UniTR) architectures can fail for objects crossing spatial windows.
• Lag Fusion Mamba delivers full, omniscient global fusion—every token interacts with every other—while maintaining linear efficiency through joint spatial and modal alignment.

This approach demonstrates the viability of dense, all-to-all, globally contextual fusion in large-scale multi-modal perception, positioning SSM-based models as strong candidates for future real-time and embedded 3D scene understanding (Wang et al., 6 Jul 2025).