---
title: 'LDRFusion: Multimodal Fusion in 3D Detection'
url: https://www.emergentmind.com/topics/ldrfusion
type: topic
---

# LDRFusion: Multimodal Fusion in 3D Detection

Searching arXiv for papers using the term “LDRFusion” and close variants to ground the article.
LDRFusion denotes multiple technically distinct fusion methods in recent arXiv literature. In its most specific and title-level usage, it refers to a LiDAR-dominant multimodal refinement framework for 3D object detection that stages LiDAR-only proposal generation ahead of LiDAR–image refinement with pseudo point clouds [2507.16224]. In adjacent literature, the same label or near-homonymous shorthand has also been used for an anchor-box-free LiDAR–radar BEV detector derived from RaLiBEV [2211.06108], for minimum-information-loss fusion of labeled random finite set densities in distributed multitarget tracking [1911.01083], and as an informal shorthand in some LiDAR–radar place-recognition and detection summaries. The term therefore spans both embodied perception and Bayesian multi-object fusion, with the 2025 LiDAR-dominant detector providing the most explicit standalone use of the name in the cited corpus [2507.16224].

## 1. LiDAR-dominant two-stage detector

In "LDRFusion: A LiDAR-Dominant multimodal refinement framework for 3D object detection" [2507.16224], the framework is organized as a two-stage pipeline. The first stage relies solely on raw 3D point clouds. Points are voxelized and processed by a Voxel Feature Encoder followed by a sparse-conv 3D backbone to produce a feature volume $S^L\in\mathbb{R}^{C\times X\times Y\times Z}$. A Region Proposal Network predicts per-anchor objectness and box offsets, yielding $N$ initial RoIs $\{o_i^L\}_{i=1}^N$. RoI pooling extracts fixed-size features $f_i^L$, and two MLP heads produce a box residual $r_i^L=R^L(f_i^L)$ and a classification score $c_i^L=C^L(f_i^L)$. A Part-Sensitive Warping module refines scores via an auxiliary classification map $\mathcal X$ [2507.16224].

The stage-one formulation is given as
$$
f^{L}_i = \mathrm{Pooling}(S^L,\,o^{L}_i),\quad
r^L_i = R^L\bigl(f^L_i\bigr),\quad
c^L_i = C^L\bigl(f^L_i\bigr)+\mathrm{PSW}\bigl(\mathcal X,b^L_i\bigr),
$$
with detections $d^L_i = (r^L_i,c^L_i)$ for $i=1\ldots N$ [2507.16224]. The first-stage losses are Focal Loss for classification and Smooth $L_1$ for regression.

The second stage introduces RGB-derived pseudo point clouds only after LiDAR has already produced accurately localized proposals. Given an image $I$, a depth-completion network $\varphi$ estimates a dense depth map $\hat D$, which is back-projected through camera intrinsics and extrinsics into a dense pseudo point cloud $Q$. The first-stage detections $\{d_i^L\}$ are reused as cascaded RoIs $\{o_i^M=d_i^L\}$ [2507.16224].

For each cascaded RoI, pseudo points are cropped, encoded by a hierarchical pseudo point residual module, pooled jointly with LiDAR features, and fused at RoI level:
$$
f^M_i = \mathrm{Fusion}\bigl(f_i^{L'},\,f_i^C\bigr).
$$
Final heads then produce
$$
r^M_i = R^M\bigl(f^M_i\bigr),\quad
c^M_i = C^M\bigl(f^M_i\bigr),\quad
d^M_i=(r^M_i,c^M_i).
$$
At inference, stage-one and stage-two outputs are merged by a weighted sum
$$
D = \alpha\,D^L + (1-\alpha)\,D^M,\qquad \alpha=0.5,
$$
followed by standard 3D NMS [2507.16224].

This design assigns distinct functional roles to the two modalities. LiDAR is used first for accurate localization, while pseudo point clouds are deferred to a refinement stage aimed at detecting challenging instances. The paper explicitly frames this as a response to the noise introduced by pseudo points when they are used too early in proposal-refinement pipelines [2507.16224].

## 2. Hierarchical pseudo point residual encoding

A central module in the 2025 detector is the Hierarchical Pseudo Point Residual Encoding module, abbreviated HPR [2507.16224]. Each pseudo point $q_{i,j}$ carries 9-dimensional attributes $(x,y,z,r,g,b,u,v,d)$. Around each point, a neighborhood $Q_{i,j}=\{q_{i,j,k}\}_{k=0}^K$ is built in image space by offsets $\delta u,\delta v\in\{\pm n\}$, with $q_{i,j,0}=q_{i,j}$ [2507.16224].

Initialization is performed by a small MLP:
$$
s^0_{i,j,k}
= \mathrm{MLP}_\mathrm{init}\bigl(x_{i,j,k},\,y_{i,j,k},\,z_{i,j,k},\,u_{i,j,k},\,v_{i,j,k},\,r_{i,j,k}\bigr).
$$
At each iteration $t\to t+1$, the module computes both feature and positional residuals:
$$
\Delta s^t_{i,j,k}=s^t_{i,j,k}-s^t_{i,j,0},
\qquad
\Delta p_{i,j,k}=p_{i,j,k}-p_{i,j,0},\;p=(x,y,z,u,v).
$$
A learned weighting $\omega^t_{i,j,k}=\mathcal M_\theta^t(\Delta p_{i,j,k})$ is then applied, and neighbor features are updated as
$$
s^{t+1}_{i,j,k}
= \omega^t_{i,j,k}\odot
\mathrm{Concat}\bigl(\Delta s^t_{i,j,k},\,s^t_{i,j,k}\bigr).
$$
Aggregation across the neighborhood uses another MLP:
$$
s^{t+1}_{i,j}
= \mathcal M_\gamma^t\bigl(s^{t+1}_{i,j,0},\,s^{t+1}_{i,j,1},\dots,s^{t+1}_{i,j,K}\bigr).
$$
The final pseudo-point representation concatenates the outputs of all iterations:
$$
s_{i,j}=\mathrm{Concat}\bigl(s^1_{i,j},\,s^2_{i,j},\dots,s^{T}_{i,j}\bigr).
$$

The paper states that HPR is intended to enhance the representation of local structures in pseudo point clouds by encoding neighborhood sets using both feature and positional residuals [2507.16224]. A plausible implication is that this residualized neighborhood modeling is designed to compensate for the structural brittleness of depth-completed pseudo geometry relative to native LiDAR measurements.

## 3. Optimization, losses, and proposal fusion

The second-stage training objective in the 2025 framework includes auxiliary LiDAR and pseudo-point losses together with the multimodal fusion loss. The auxiliary losses $\mathcal L^L_{\mathrm aux}$ and $\mathcal L^C_{\mathrm aux}$ each use Focal Loss for classification and Smooth $L_1$ for regression. The multimodal fusion loss $\mathcal L^M$ uses Focal for classification and Smooth $L_1$ plus GIoU for regression [2507.16224].

The full multi-stage loss is
$$
\mathcal{L} = \mathcal L_{\mathrm{RPN}}
+\lambda_1\,\mathcal L^L
+\lambda_2\,\bigl(\mathcal L^L_{\mathrm aux}+\mathcal L^C_{\mathrm aux}\bigr)
+\lambda_3\,\mathcal L^M,
$$
with $\lambda_1=1.0,\;\lambda_2=0.5,\;\lambda_3=1.0$ [2507.16224].

Proposal fusion is deliberately instance-level rather than an earlier dense cross-modal blending. The paper gives the final merging rule as
$$
D_{\mathrm final}
= \alpha\,\{d^L_i\}_{i=1}^N + (1-\alpha)\,\{d^M_i\}_{i=1}^N,\quad
\alpha=0.5,
$$
followed by standard 3D non-maximum suppression [2507.16224]. This makes the framework “LiDAR-dominant” not only in nomenclature but in optimization topology: proposals are anchored by the first stage, and multimodal refinement operates downstream of that anchor.

The authors contrast this strategy with prior LiDAR–camera fusion methods that construct spatial pseudo point clouds as auxiliary input and adopt a proposal-refinement framework throughout. Their stated concern is that introducing pseudo points inevitably brings noise, potentially resulting in inaccurate predictions [2507.16224]. The architecture responds by restricting the role of pseudo points to second-stage refinement rather than first-stage proposal genesis.

## 4. Empirical results on KITTI

The 2025 LDRFusion paper reports results on KITTI across single-class and multi-class settings [2507.16224]. For single-class car detection on the test set using 3D AP at $R_{40}$, the reported numbers are:

| Method | Easy | Mod. | Hard |
|---|---:|---:|---:|
| Voxel R-CNN | 90.90 | 81.62 | 77.06 |
| SFD (fusion) | 91.73 | 84.76 | 77.92 |
| LDRFusion (Ours) | 91.92 | 85.47 | 80.43 |

On the validation set, the paper reports both $R_{40}$ and $R_{11}$ car AP:

| Metric | Easy | Mod. | Hard |
|---|---:|---:|---:|
| 3D-AP ($R_{40}$) | 96.00 | 89.06 | 86.47 |
| 3D-AP ($R_{11}$) | 90.24 | 87.72 | 86.21 |

For multi-class validation at $R_{40}$, LDRFusion reports car AP of **95.86**, **88.77**, and **86.39** for Easy, Moderate, and Hard; pedestrian AP of 73.67, 66.12, and 60.06; cyclist AP of 91.32, **75.42**, and **70.75**; and an overall **mAP** of **78.71** [2507.16224].

The paper also reports ablations showing that a cascade refinement added to the baseline raises 3D-AP mAP from 89.92 to 90.35, and that adding HPR alone yields 90.18, while “+ Cascade + HPR (Ours)” reaches **90.51** [2507.16224]. It further states that on “Hard” car instances, LDRFusion improves over LiDAR-only by approximately 3.3 percentage points, from 77.06 to 80.43 on the test set.

Inference throughput is reported as approximately 10 FPS on a pair of 3090 GPUs [2507.16224]. This positions the method as a refinement-oriented detector that attempts to preserve real-time viability while retaining a two-stage design.

## 5. Other uses of the name in LiDAR–radar perception

The label “LDRFusion” is also used in a different sense in the expanded description of "RaLiBEV: Radar and LiDAR BEV Fusion Learning for Anchor Box Free Object Detection Systems" [2211.06108]. There, “LDRFusion” refers to an anchor-box-free BEV fusion detector using compact BEV-raster inputs, anchor-free single-point label assignment via GACHIPS, and a symmetric interactive transformer denoted DQMITBF [2211.06108].

In that formulation, radar range–azimuth heatmaps are reprojected into Cartesian BEV on a fixed $320\times320$ grid, forming $R_0\in\mathbb{R}^{320\times320\times1}$, while LiDAR point clouds are pillarized into a $320\times320$ grid with $xy$-resolution $0.2\,\mathrm{m}$, yielding $L_0\in\mathbb{R}^{320\times320\times9}$ [2211.06108]. The detector attaches a multi-scale head in the style of YOLOv4/Centripoint and predicts a 1-channel center heatmap and a 6-channel regression output encoding $\Delta x,\Delta y,w,\ell,\theta_{cls},\Delta\theta$ [2211.06108].

The same source describes four label-assignment strategies—DIPS, GAHPS, GAHIPS, and GACHIPS—of which GACHIPS is presented as the consistent-cost formulation that forces the same single grid cell to own both classification and box-regression loss [2211.06108]. Under the hardest setting, train = Clear+Fog and test = Fog at IoU = 0.8, “LDRFusion with GACHIPS alone achieves AP = 93.7%, and with the full DQMITBF fusion it reaches AP = 94.1%,” while the cited prior art ST-MVDNet++ scores 75.1% at the same operating point [2211.06108].

This usage differs categorically from the 2025 LiDAR–camera LDRFusion. It is single-stage rather than two-stage, LiDAR–radar rather than LiDAR–image, and center-based anchor-free rather than proposal-refinement-based. The shared name therefore does not denote a single canonical architecture across object-detection literature.

## 6. Minimum-information-loss LDRFusion in labeled RFS tracking

A further and conceptually separate use appears in "Fusion of labeled RFS densities with minimum information loss" [1911.01083]. Here LDRFusion denotes a minimum-information-loss fusion method for labeled random finite set densities in distributed multi-sensor multitarget tracking rather than deep neural perception.

The method seeks a fused density $\bar\pi$ minimizing weighted information loss:
$$
\bar\pi_{\mathrm{mil}} \coloneqq \arg\min_{\pi}\sum_{i\in\mathbb N}\omega^i\cdot D_{KL}(\pi^i\|\pi),
$$
with $\omega^i\ge 0$, $\sum \omega^i=1$, and
$$
D_{KL}(p\|q)=\int p(X)\log\!\left[\frac{p(X)}{q(X)}\right]\delta X.
$$
The unconstrained solution is the linear opinion pool
$$
\bar\pi(X)=\sum_{i\in\mathbb N}\omega^i\pi^i(X)
$$
[1911.01083].

To preserve conjugacy, the paper constrains the fused density to remain in the same family as the local densities. For M$\delta$-GLMB densities, the fused JEP is
$$
\bar p_M(L)=\sum_i\omega^i p_M^i(L),
$$
with track-wise mixture weights
$$
\tilde\omega^i(L)=\frac{\omega^i p_M^i(L)}{\sum_j\omega^j p_M^j(L)},
$$
and fused per-label densities
$$
\bar f_{l|L}(x)=\sum_i\tilde\omega^i(L)\,f_{l|L}^i(x).
$$
For LMB densities, the fused parameters are
$$
\bar r_l=\sum_i\omega^i r_l^i,\qquad
\bar f_l(x)=\sum_i\tilde\omega_l^i f_l^i(x),
$$
with
$$
\tilde\omega_l^i=\frac{\omega^i r_l^i}{\sum_j\omega^j r_l^j}
$$
[1911.01083].

The same framework handles differing fields of view through subspace decomposition of the global label space and addresses label mismatching via rank assignment optimization. For two LMBs, a cost matrix is constructed from track-to-track divergences such as Jensen–Shannon divergence or Cauchy–Schwarz divergence, and the linear assignment is solved by the Hungarian algorithm [1911.01083].

Simulation results reported in that source indicate that, in same-FoV settings, MIL and GCI are comparable at high $P_d=0.98$, while at moderate or low $P_d$ of $0.5$–$0.7$, MIL outperforms GCI in localization and cardinality accuracy. In different-FoV settings, GCI “zeros out” tracks outside the common FoV, whereas MIL preserves exclusive-FoV tracks via subspace decomposition and RAO [1911.01083].

This probabilistic LDRFusion is unrelated to the 2025 LiDAR-dominant detector except by name. Its domain is consensus multitarget tracking, its objects are labeled-RFS densities, and its operators are KLD minimization, family-preserving pooling, and divergence-based label assignment.

## 7. Scope, ambiguity, and relation to adjacent fusion research

The term’s ambiguity is amplified by nearby literature that does not always use “LDRFusion” as an official paper title but employs closely related LiDAR–radar fusion formulations. "Bi-LRFusion: Bi-Directional LiDAR-Radar Fusion for 3D Dynamic Object Detection" introduces bidirectional interaction through LiDAR→Radar query-based height features and Radar→LiDAR query-based BEV fusion, then merges enriched BEV maps before a CenterPoint-style head [2306.01438]. On nuScenes val, it reports 67.5% mAP and 69.8% NDS versus 62.0% mAP and 66.1% NDS for CenterPoint, with the largest gains on dynamic classes [2306.01438]. "L4DR: LiDAR-4DRadar Fusion for Weather-Robust 3D Object Detection" instead emphasizes Multi-Modal Encoding, Foreground-Aware Denoising, an IM$^2$ backbone, and Multi-Scale Gated Fusion, reporting up to 20.0% 3D mAP improvement over LiDAR-only under simulated fog on VoD [2408.03677].

Likewise, "DeepFusion: A Robust and Modular 3D Object Detector for Lidars, Cameras and Radars" adopts a modular BEV alignment-and-addition scheme for LiDAR, camera, and radar feature maps [2209.12729], while "LRC-WeatherNet: LiDAR, RADAR, and Camera Fusion Network for Real-time Weather-type Classification in Autonomous Driving" uses early BEV fusion and mid-level gated fusion for weather-type recognition over nine classes on MSU-4S [2603.21987]. These works form the broader technical context in which the 2025 LDRFusion detector situates its LiDAR-first, pseudo-point-later design.

A common misconception would be to treat LDRFusion as a single standardized architecture across autonomous-driving perception and multitarget tracking. The cited literature does not support that reading. Instead, the label is polysemous: it names at least one LiDAR-dominant two-stage detector [2507.16224], one minimum-information-loss labeled-RFS fusion rule [1911.01083], and a LiDAR–radar anchor-free BEV detector in an expanded RaLiBEV description [2211.06108]. This suggests that, in technical discussion, “LDRFusion” should be disambiguated by domain and citation rather than assumed to designate a unique method.

Source: https://www.emergentmind.com/topics/ldrfusion