---
title: 'InsFusion: Multi-Source LiDAR-Camera Fusion'
url: https://www.emergentmind.com/topics/insfusion
type: topic
---

# InsFusion: Multi-Source LiDAR-Camera Fusion

InsFusion is a LiDAR-camera fusion method for three-dimensional object detection that targets cumulative noise arising across raw feature extraction, perspective transformation, and feature fusion in Bird’s-Eye View (BEV) pipelines. Its central design principle is to extract instance proposals not only from fused features but also from modality-specific raw features, and then to use those proposals to query raw image features, raw LiDAR BEV features, and fused BEV features through attention-based refinement. In the reported formulation, this rethinks instance-level fusion as a mechanism for mitigating error accumulation rather than relying exclusively on already fused representations, and it is evaluated on the nuScenes benchmark with improvements over strong BEV-based baselines [2509.08374].

## 1. Problem setting and motivation

InsFusion is motivated by a failure mode in multimodal 3D detection pipelines for autonomous driving: errors introduced in early stages are not isolated, but accumulate through successive operations. The paper identifies three stages at which such degradation emerges—raw feature extraction from camera and LiDAR, transformation to a uniform coordinate space such as BEV, and subsequent feature fusion. Examples named in the description include depth estimation errors in camera features, imperfect coordinate transformation, and fusion mismatches. By the time instance-level proposals are generated, the underlying representation may already be corrupted, which degrades detection quality on downstream heads [2509.08374].

The method addresses this by preserving access to less corrupted sources during instance refinement. Rather than treating the fused BEV tensor as the sole substrate for proposal generation and decoding, InsFusion introduces a multi-source query design: proposals are extracted from raw camera features, raw LiDAR BEV features, and fused features, then jointly refined. This shifts the role of instance-level fusion from late aggregation to error-aware correction. In that sense, the architecture is explicitly organized around robustness to accumulated noise rather than only around stronger multimodal interaction.

## 2. Proposal extraction from raw and fused features

The architecture is described as having three main stages, beginning with proposal extraction from raw and fused representations. For the camera branch, given image features $F_{\text{img}} \in \mathbb{R}^{h \times w \times C_{\text{img}}}$, InsFusion defines $K$ learnable camera queries, drawn from a random Gaussian and learned, each of dimension $D_q$. Instance features
\[
Q_{\text{img}}^{(0)} \in \mathbb{R}^{K \times D_q}
\]
are then extracted via adaptive sampling and mixing from the 2D feature maps, with SparseBEV cited as the reference mechanism for this step [2509.08374].

For the LiDAR branch, the input is BEV LiDAR features
\[
F_{\text{lidar\_bev}} \in \mathbb{R}^{X \times Y \times C_{\text{lidar\_bev}}}.
\]
InsFusion uses peak detection on a predicted heatmap $H_{\text{lidar}}$ and selects the top $K$ instance proposals. These proposals are encoded into query embeddings
\[
Q_{\text{lidar}}^{(0)} \in \mathbb{R}^{K \times D_q}.
\]
The fused branch is intentionally generic: InsFusion can plug into any feature-fusion strategy, including heatmap-based or transformer-based fusion, to produce fused instance features $Q_{\text{fusion}}^{(0)}$ [2509.08374].

A concise summary of the three proposal sources is given below.

| Branch | Source features | Proposal mechanism |
|---|---|---|
| Camera | Raw image features $F_{\text{img}}$ | $K$ learnable queries with adaptive sampling and mixing |
| LiDAR | Raw LiDAR BEV features $F_{\text{lidar\_bev}}$ | Top-$K$ peaks from predicted heatmap $H_{\text{lidar}}$ |
| Fusion | Fused features | Baseline-specific fusion branch yielding $Q_{\text{fusion}}^{(0)}$ |

This design is the main departure from pipelines that rely only on fused representations for proposal generation and refinement. By retaining modality-specific proposals, InsFusion preserves access to features that have undergone less corruption than the final fused tensor.

## 3. Query alignment and multi-source deformable refinement

After proposal extraction, the queries from the three branches are aligned into a shared latent space through modality-specific linear projections:
\[
\hat{Q}_{\text{img}} = W_{\text{img}} Q_{\text{img}} + b_{\text{img}},
\]
\[
\hat{Q}_{\text{lidar}} = W_{\text{lidar}} Q_{\text{lidar}} + b_{\text{lidar}},
\]
\[
\hat{Q}_{\text{fusion}} = W_{\text{fusion}} Q_{\text{fusion}} + b_{\text{fusion}},
\]
where $W_{\text{img}}, W_{\text{lidar}}, W_{\text{fusion}} \in \mathbb{R}^{D_q \times D_q}$ and the corresponding biases are learnable. The stated purpose of these projections is to resolve distribution alignment while maintaining modality-specific strengths [2509.08374].

The aligned queries are concatenated as
\[
Q^{(0)} = [\hat{Q}_{\text{img}},\ \hat{Q}_{\text{lidar}},\ \hat{Q}_{\text{fusion}}] \in \mathbb{R}^{3K \times D_q}.
\]
Refinement then proceeds through a stack of $L$ deformable Transformer decoder layers. Each layer performs cross-attention in parallel to three Key-Value sources: raw image features $F_{\text{img}}$, raw LiDAR BEV features $F_{\text{lidar\_bev}}$, and fused BEV features $F_{\text{fusion\_bev}}$. The outputs from these three sources are aggregated via element-wise addition, and the iterative update is written as
\[
Q^{(l)} = \mathrm{DeformableTransformerLayer}\left(
Q^{(l-1)},
\mathrm{Flatten}(F_{\text{img}}),
\mathrm{Flatten}(F_{\text{lidar\_bev}}),
\mathrm{Flatten}(F_{\text{fusion\_bev}})
\right).
\]
The process is repeated for $L$ iterations, with the reported ablation indicating that $L=2$ is optimal [2509.08374].

The key architectural claim is that the refinement remains anchored to both raw modality features and fused features at every decoder stage. This is not a simple late fusion block; it is a recurrent querying mechanism in which modality-specific and fused contexts are jointly revisited during instance refinement. The paper’s stated interpretation is that this actively counteracts error accumulation typical of cascaded fusion pipelines.

## 4. Position within BEV-based multimodal detection

InsFusion is described as compatible with standard BEV-based LiDAR-camera fusion pipelines with minimal modifications. Traditional pipelines in this family typically fuse features in BEV after projection, then rely on the fused representation for proposal generation and refinement. InsFusion changes this assumption by explicitly extracting and maintaining instance proposals from raw camera features and raw LiDAR features in addition to fused BEV features, followed by multi-source refinement through deformable attention [2509.08374].

This places the method in a distinct niche relative to earlier BEV-centered multimodal detectors. IS-Fusion, for example, jointly models scene-level and instance-level context through a Hierarchical Scene Fusion module and an Instance-Guided Fusion module, with instance features mined from BEV scene features and then propagated back to the scene representation [2403.15241]. InsFusion addresses a different bottleneck: it focuses on noise and error accumulation before and during fusion, and therefore treats raw-feature querying as a first-class component rather than as an auxiliary detail. The two approaches are related by their shared emphasis on instance-level reasoning, but they intervene at different points in the multimodal pipeline.

A common point of confusion is to interpret InsFusion as a replacement for a BEV fusion backbone. In the reported design, it is instead a plug-and-play instance-level fusion paradigm that can be inserted into existing BEV-based systems. The paper explicitly states compatibility with various advanced baseline methods and reports results when integrated into FocalFormer3D and IS-Fusion [2509.08374].

## 5. Experimental evaluation and efficiency

The reported experiments use the nuScenes dataset, with mean Average Precision (mAP) and NuScenes Detection Score (NDS) as the principal metrics. InsFusion is evaluated by plugging it into two recent strong BEV fusion baselines. For FocalFormer3D, the integration improves mAP from $70.5$ to $71.5$ and NDS from $73.1$ to $74.2$. For IS-Fusion, it improves mAP from $72.3$ to $73.4$ and NDS from $73.7$ to $74.3$ [2509.08374].

| Integration setting | mAP | NDS |
|---|---:|---:|
| FocalFormer3D | $70.5 \rightarrow 71.5$ | $73.1 \rightarrow 74.2$ |
| IS-Fusion | $72.3 \rightarrow 73.4$ | $73.7 \rightarrow 74.3$ |

The paper characterizes these results as new state-of-the-art performance for 3D object detection on the reported benchmark setting. The ablation study further states that $L=2$ deformable Transformer layers is optimal: too few layers limit refinement capacity, whereas too many layers induce overfitting to noise. This is consistent with the method’s stated objective, since its decoder is intended to suppress accumulated corruption rather than to deepen feature transformation indiscriminately [2509.08374].

The additional cost is reported as minor. One concrete example is a $9\%$ reduction in inference speed for FocalFormer3D with InsFusion, described as negligible in practical deployment. Training is also described as efficient because baseline weights are maintained and only the newly introduced modules require learning. These details reinforce the paper’s claim that the method is plug-and-play rather than a wholesale retraining strategy.

## 6. Nomenclature, scope, and adjacent lines of work

InsFusion belongs specifically to multimodal 3D object detection with LiDAR-camera fusion. It should not be conflated with similarly named methods in unrelated domains, including “InfiFusion: A Unified Framework for Enhanced Cross-Model Reasoning via LLM Fusion” [2501.02795], “SinFusion: Training Diffusion Models on a Single Image or Video” [2211.11743], “InFusion: Inject and Attention Fusion for Multi Concept Zero-Shot Text-based Video Editing” [2308.00135], or “INFusion: Diffusion Regularized Implicit Neural Representations for 2D and 3D accelerated MRI reconstruction” [2406.13895]. The naming overlap reflects a broader use of “fusion” terminology across machine learning, but these methods address fundamentally different data modalities, objectives, and optimization regimes.

Within autonomous driving, the more relevant comparison class is BEV-based multimodal detection. Here, the contribution of InsFusion is not a new sensor encoder or a new global fusion operator, but an instance-level querying and refinement mechanism that revisits raw modality-specific features after fusion has already occurred. This suggests a specific interpretation of its significance: the method treats error accumulation as an architectural problem in multimodal perception, and proposes proposal extraction plus multi-source attention as the corrective interface between raw and fused representations.

In that sense, InsFusion can be understood as a reformulation of instance-level LiDAR-camera fusion around proposal provenance. The method’s proposals are deliberately drawn from raw image features, raw LiDAR BEV features, and fused features, then aligned and refined in a shared latent space. The reported gains on nuScenes indicate that this provenance-aware design is effective when attached to existing BEV detectors, especially in settings where cumulative transformation and fusion errors would otherwise degrade final 3D detection quality [2509.08374].

Source: https://www.emergentmind.com/topics/insfusion