DVLO4D: 4D Visual–LiDAR Odometry

• DVLO4D is a learning-based visual–LiDAR odometry framework that integrates RGB images and LiDAR data with explicit temporal memory for robust 6-DoF pose estimation.
• It employs sparse query fusion via deformable cross-attention to achieve efficient geometric alignment, reducing drift compared to dense or hand-engineered approaches.
• Clip-level supervision and multi-frame temporal aggregation enable DVLO4D to deliver state-of-the-art accuracy with real-time performance on challenging datasets like KITTI and Argoverse.

DVLO4D is a learning-based visual–LiDAR odometry framework for estimating the relative pose $T_{S\to T}\in SE(3)$ between consecutive frames from an RGB image pair $(I_S, I_T)$, a LiDAR point cloud pair $(PC_S, PC_T)$, and temporal history over several seconds (Liu et al., 7 Sep 2025). It is designed to address three specific deficiencies identified in earlier VO, LO, and VLO systems: dense or hand-engineered multi-sensor fusion that is expensive or configuration-specific, limited use of temporal information with frame-by-frame losses that induce scale drift, and restricted robustness to temporal misalignment, LiDAR sparsity, noise, dynamics, and occlusions (Liu et al., 7 Sep 2025). Relative to the earlier DVLO framework, which introduced local-to-global feature fusion and bi-directional structure alignment for synchronized image–LiDAR pairs, DVLO4D extends the formulation with query-based fusion, explicit temporal memory banks, and clip-level optimization (Liu et al., 2024, Liu et al., 7 Sep 2025).

1. Problem formulation and architectural position

DVLO4D operates on monocular RGB images $I_S,I_T \in \mathbb{R}^{H\times W\times 3}$ from the left camera and LiDAR point clouds $PC_S,PC_T \in \mathbb{R}^{N\times 3}$ (Liu et al., 7 Sep 2025). The target is relative 6-DoF pose estimation, with rotation represented by a unit quaternion $q\in\mathbb{R}^4$ and translation by $t\in\mathbb{R}^3$. The framework preserves the coarse-to-fine, iterative odometry structure used in earlier deep LiDAR odometry systems, but replaces pairwise-only fusion and supervision with a sparse spatial-temporal design (Liu et al., 7 Sep 2025).

The LiDAR branch begins with cylindrical projection. For each point $(x,y,z)$, DVLO4D computes

$$u = \frac{\arctan2(y,x)}{\Delta\theta}, \quad v = \frac{\arcsin\big(z / \sqrt{x^2+y^2+z^2}\big)}{\Delta\phi},$$

where $\Delta\theta,\Delta\phi$ are LiDAR horizontal and vertical angular resolutions (Liu et al., 7 Sep 2025). This produces a pseudo-image of size $(I_S, I_T)$0 with channels containing the original 3D coordinates, thereby providing a dense grid for CNN-like processing and alignment with image features.

The modality-specific encoders follow the two-stream structure already familiar from DVLO: a hierarchical CNN-like backbone for the point pseudo-image, and a convolutional feature pyramid for RGB images (Liu et al., 2024, Liu et al., 7 Sep 2025). In DVLO4D, the point encoder yields multi-scale LiDAR feature maps

$(I_S, I_T)$1

and the image encoder yields multi-scale image features

$(I_S, I_T)$2

A key implementation detail is that sparse LiDAR queries are sampled at each pyramid level, with examples given as 116, 228, 904, and 3600 queries for the four levels (Liu et al., 7 Sep 2025).

This placement is important in relation to DVLO. The earlier system reconciled the structural mismatch between image grids and sparse 3D points through image-to-point clustering and point-to-image cylindrical projection, then fused locally and globally (Liu et al., 2024). DVLO4D preserves cylindrical projection and hierarchical odometry estimation, but moves the heavy cross-modal operation from dense or cluster-based fusion to sparse query fusion and augments the pairwise architecture with explicit temporal modeling (Liu et al., 7 Sep 2025). This suggests a shift from structural alignment as the primary organizing principle to sparse, geometry-aware query selection plus temporal memory.

2. Sparse Query Fusion and geometry-aware multimodal alignment

The first defining component of DVLO4D is Sparse Query Fusion (SQF), which treats selected LiDAR points as sparse queries with 3D positional embeddings and fuses them with image features using a deformable-DETR-style cross-attention mechanism (Liu et al., 7 Sep 2025). The stated purpose is to obtain efficient, geometry-aware, fine-grained visual–LiDAR fusion that adapts to different sensor setups and misalignments.

Let $(I_S, I_T)$3 denote sparse LiDAR reference points, with $(I_S, I_T)$4. For each point and camera $(I_S, I_T)$5, DVLO4D projects $(I_S, I_T)$6 into the image as $(I_S, I_T)$7 using known camera intrinsics and extrinsics (Liu et al., 7 Sep 2025). Rather than performing dense alignment on the entire image feature map, the network predicts, for each query, small sets of offsets $(I_S, I_T)$8 and corresponding attention weights $(I_S, I_T)$9. The sampled camera feature is

$(PC_S, PC_T)$0

with bilinear interpolation on the image feature map (Liu et al., 7 Sep 2025). The effect is that each LiDAR query only “looks around” a projected location through a few learned sampling offsets.

DVLO4D then applies multi-head cross-attention with LiDAR features as queries and sampled visual features as keys and values: $(PC_S, PC_T)$1 Conceptually, for each attention head $(PC_S, PC_T)$2,

$(PC_S, PC_T)$3

The output is a set of fused LiDAR–image features at sparse LiDAR query locations, and the operation is performed hierarchically across scales (Liu et al., 7 Sep 2025).

The paper explicitly contrasts this with the earlier DVLO local fusion scheme. DVLO used a clustering-based Local Fuser in which projected LiDAR points served as cluster centers for similarity-weighted aggregation over image pseudo-points, followed by a Global Fuser on the cylindrical pseudo-image grid (Liu et al., 2024). DVLO4D still supplements SQF with a global adaptive fusion scheme similar to Delflow and DVLO to compensate for the local receptive field of deformable sampling, but its central change is that “the heavy cross-modal fusion is done on sparse queries, not dense grids” (Liu et al., 7 Sep 2025).

The empirical significance of this redesign is explicit. In local fusion ablations on KITTI 07–10, attention-based fusion yields $(PC_S, PC_T)$4, $(PC_S, PC_T)$5 at 180 ms; CNN-based fusion yields $(PC_S, PC_T)$6, $(PC_S, PC_T)$7 at 85 ms; cluster-based DVLO-style fusion yields $(PC_S, PC_T)$8, $(PC_S, PC_T)$9 at 94 ms; and query-based fusion yields $I_S,I_T \in \mathbb{R}^{H\times W\times 3}$0, $I_S,I_T \in \mathbb{R}^{H\times W\times 3}$1 at 82 ms (Liu et al., 7 Sep 2025). The same section reports that removing SQF increases $I_S,I_T \in \mathbb{R}^{H\times W\times 3}$2 from 0.73 to 0.98 and $I_S,I_T \in \mathbb{R}^{H\times W\times 3}$3 from 0.37 to 0.57 (Liu et al., 7 Sep 2025). Within the terms of the paper, SQF is therefore both the fastest and the most accurate among the tested fusion variants.

3. Pose estimation and temporal memory in four-dimensional context

After spatial fusion, DVLO4D computes an attentive cost volume at the coarsest scale using fused source and target features, following PWCLO and EfficientLO (Liu et al., 7 Sep 2025). This produces a per-query embedding

$I_S,I_T \in \mathbb{R}^{H\times W\times 3}$4

which is reweighted by a learnable mask $I_S,I_T \in \mathbb{R}^{H\times W\times 3}$5 and mapped by fully connected layers to initial pose parameters. The coarse quaternion and translation are

$I_S,I_T \in \mathbb{R}^{H\times W\times 3}$6

DVLO4D then performs iterative refinement over layers $I_S,I_T \in \mathbb{R}^{H\times W\times 3}$7, predicting residuals $I_S,I_T \in \mathbb{R}^{H\times W\times 3}$8 and composing them through quaternion algebra: $I_S,I_T \in \mathbb{R}^{H\times W\times 3}$9

$PC_S,PC_T \in \mathbb{R}^{N\times 3}$0

This keeps the update rule in a consistent $PC_S,PC_T \in \mathbb{R}^{N\times 3}$1 form across scales (Liu et al., 7 Sep 2025).

The explicitly temporal part of the system is the Temporal Interaction and Update module (TIU). TIU maintains two memory structures across a sliding history window: a Memory Feature Bank (MFB) storing historical fused cost-volume features, and a Memory Pose Bank (MPB) storing recent refined pose estimates $PC_S,PC_T \in \mathbb{R}^{N\times 3}$2 (Liu et al., 7 Sep 2025). At time $PC_S,PC_T \in \mathbb{R}^{N\times 3}$3, both banks are zero-initialized. Thereafter, each frame contributes its feature and refined pose, and only the last $PC_S,PC_T \in \mathbb{R}^{N\times 3}$4 frames are retained. A typical setting is $PC_S,PC_T \in \mathbb{R}^{N\times 3}$5 frames, corresponding to approximately 3 seconds at 10 Hz (Liu et al., 7 Sep 2025).

The temporal interaction stage proceeds in four steps (Liu et al., 7 Sep 2025). First, an ego-instance feature $PC_S,PC_T \in \mathbb{R}^{N\times 3}$6 is extracted from the current frame’s coarsest cost volume. Second, the pose history from MPB is encoded: sequences $PC_S,PC_T \in \mathbb{R}^{N\times 3}$7 and $PC_S,PC_T \in \mathbb{R}^{N\times 3}$8 are processed by a transformer-based Multi-Head Self-Attention module followed by an LSTM, yielding encoded histories $PC_S,PC_T \in \mathbb{R}^{N\times 3}$9 and $q\in\mathbb{R}^4$0. Third, the current ego feature is refined by cross-attention over historical ego features stored in MFB: $q\in\mathbb{R}^4$1 Fourth, the temporally refined ego feature is concatenated with encoded pose histories and passed through MLPs to predict the initial pose at the coarsest layer: $q\in\mathbb{R}^4$2

$q\in\mathbb{R}^4$3

The paper states that this replaces purely frame-pair-based initialization and provides a much better starting point for iterative refinement (Liu et al., 7 Sep 2025).

The Temporal Update Module then refines the final layer-0 estimate $q\in\mathbb{R}^4$4 using historical pose context once more. After MLP encodings of $q\in\mathbb{R}^4$5 and $q\in\mathbb{R}^4$6, the network concatenates these representations with $q\in\mathbb{R}^4$7 and $q\in\mathbb{R}^4$8 to predict the refined pose $q\in\mathbb{R}^4$9, which is used at test time and supervised during training (Liu et al., 7 Sep 2025).

The role of TIU is quantitatively isolated in ablation. Removing TIU changes mean $t\in\mathbb{R}^3$0 from 0.73 to 0.92 and $t\in\mathbb{R}^3$1 from 0.37 to 0.54 (Liu et al., 7 Sep 2025). The paper interprets this as evidence that TIU improves initialization and reduces drift. A plausible implication is that DVLO4D’s “4D” character is not merely a multi-frame aggregation heuristic; it is implemented through explicit memory, sequence encoding, and temporally informed pose priors.

4. Clip-level supervision and the Collective Average Loss

A second major departure from frame-pair training is DVLO4D’s clip-level optimization. Instead of training on independent frame pairs, the method splits each KITTI training sequence 00–06 into clips of length $t\in\mathbb{R}^3$2 frames, approximately 6 seconds at 10 Hz, and further divides each clip into sub-clips of length $t\in\mathbb{R}^3$3 frames, approximately 0.3 s (Liu et al., 7 Sep 2025). During training, TIU can consume temporal context up to a history length $t\in\mathbb{R}^3$4, while the loss is averaged over frames in the sub-clip.

For each layer $t\in\mathbb{R}^3$5 and frame $t\in\mathbb{R}^3$6, the per-frame pose loss is

$t\in\mathbb{R}^3$7

where $t\in\mathbb{R}^3$8 and $t\in\mathbb{R}^3$9 are learnable scalars initialized as $(x,y,z)$0 and $(x,y,z)$1 (Liu et al., 7 Sep 2025). For the refined pose,

$(x,y,z)$2

These learnable scalars function as balancing parameters between translation and rotation magnitudes; the paper notes that they effectively act like log-variance terms in a heteroscedastic regression model (Liu et al., 7 Sep 2025).

The Collective Average Loss (CAL) for a sub-clip of length $(x,y,z)$3 is

$(x,y,z)$4

with $(x,y,z)$5 for the four layers and $(x,y,z)$6 for the refined loss (Liu et al., 7 Sep 2025). The formal point of CAL is that gradients are driven by average trajectory behavior over several frames rather than by a single frame. The paper explicitly states that this forces the network to learn globally consistent trajectories and significantly reduces drift and improves long-term scale stability (Liu et al., 7 Sep 2025).

The ablation evidence is direct. Without CAL, that is, with frame-wise training, mean $(x,y,z)$7 changes from 0.73 to 0.83 and $(x,y,z)$8 from 0.37 to 0.49 (Liu et al., 7 Sep 2025). For $(x,y,z)$9, increasing the sub-clip length from $$u = \frac{\arctan2(y,x)}{\Delta\theta}, \quad v = \frac{\arcsin\big(z / \sqrt{x^2+y^2+z^2}\big)}{\Delta\phi},$$0 to $$u = \frac{\arctan2(y,x)}{\Delta\theta}, \quad v = \frac{\arcsin\big(z / \sqrt{x^2+y^2+z^2}\big)}{\Delta\phi},$$1 improves performance from $$u = \frac{\arctan2(y,x)}{\Delta\theta}, \quad v = \frac{\arcsin\big(z / \sqrt{x^2+y^2+z^2}\big)}{\Delta\phi},$$2 to $$u = \frac{\arctan2(y,x)}{\Delta\theta}, \quad v = \frac{\arcsin\big(z / \sqrt{x^2+y^2+z^2}\big)}{\Delta\phi},$$3 (Liu et al., 7 Sep 2025). In the paper’s interpretation, CAL and longer sub-clips improve the network’s ability to learn temporal coherence, directly reducing drift.

This training strategy marks a substantive methodological contrast with the earlier DVLO formulation. DVLO used a multi-level supervised loss over iterative pose estimates but remained fundamentally pairwise in training and inference organization (Liu et al., 2024). DVLO4D retains the same broad loss-balancing philosophy while moving the unit of supervision from individual frame pairs to temporally coherent sub-clips (Liu et al., 7 Sep 2025).

5. Quantitative performance, runtime, and component effects

The principal evaluation on KITTI odometry uses translational RMSE $$u = \frac{\arctan2(y,x)}{\Delta\theta}, \quad v = \frac{\arcsin\big(z / \sqrt{x^2+y^2+z^2}\big)}{\Delta\phi},$$4 and rotational RMSE $$u = \frac{\arctan2(y,x)}{\Delta\theta}, \quad v = \frac{\arcsin\big(z / \sqrt{x^2+y^2+z^2}\big)}{\Delta\phi},$$5, averaged over path segments of lengths 100–800 m (Liu et al., 7 Sep 2025). On test sequences 07–10, DVLO4D reports mean $$u = \frac{\arctan2(y,x)}{\Delta\theta}, \quad v = \frac{\arcsin\big(z / \sqrt{x^2+y^2+z^2}\big)}{\Delta\phi},$$6 and $$u = \frac{\arctan2(y,x)}{\Delta\theta}, \quad v = \frac{\arcsin\big(z / \sqrt{x^2+y^2+z^2}\big)}{\Delta\phi},$$7, compared with EfficientLO at $$u = \frac{\arctan2(y,x)}{\Delta\theta}, \quad v = \frac{\arcsin\big(z / \sqrt{x^2+y^2+z^2}\big)}{\Delta\phi},$$8 and $$u = \frac{\arctan2(y,x)}{\Delta\theta}, \quad v = \frac{\arcsin\big(z / \sqrt{x^2+y^2+z^2}\big)}{\Delta\phi},$$9, and the earlier multi-modal DVLO at $\Delta\theta,\Delta\phi$0 and $\Delta\theta,\Delta\phi$1 (Liu et al., 7 Sep 2025). The paper summarizes this as an approximately 18% translation improvement over EfficientLO and approximately 12% over DVLO, with slight rotational improvement as well (Liu et al., 7 Sep 2025).

Method	KITTI 07–10 mean
DVLO4D	$\Delta\theta,\Delta\phi$2, $\Delta\theta,\Delta\phi$3
EfficientLO	$\Delta\theta,\Delta\phi$4, $\Delta\theta,\Delta\phi$5
DVLO	$\Delta\theta,\Delta\phi$6, $\Delta\theta,\Delta\phi$7

A second comparison on KITTI sequences 09–10 places DVLO4D against learning-based multimodal odometry methods trained on sequences 00–08, whereas DVLO4D is trained only on 00–06 (Liu et al., 7 Sep 2025). Mean performance is reported as $\Delta\theta,\Delta\phi$8, $\Delta\theta,\Delta\phi$9 for DVLO4D and $(I_S, I_T)$00, $(I_S, I_T)$01 for the next best method, H-VLO (Liu et al., 7 Sep 2025). The paper characterizes this as roughly halving the translation error.

On Argoverse odometry, where evaluation uses Absolute Trajectory Error (ATE) and Relative Pose Error (RPE), DVLO4D reports mean ATE 0.089 and RPE 0.025 over sequences 00–23 (Liu et al., 7 Sep 2025). The listed baselines are PyLiDAR with ATE 6.900 and RPE 0.109, A-LOAM with ATE 4.138 and RPE 0.066, and DSLO with ATE 0.111 and RPE 0.027 (Liu et al., 7 Sep 2025). The stated conclusion is that DVLO4D improves ATE by approximately 20% over DSLO and demonstrates strong generalization beyond KITTI.

Runtime is another central claim. Average per-frame inference time on KITTI 07–10 is reported as 82 ms for DVLO4D, compared with 167 ms for DV-LOAM, 200 ms for PL-LOAM, 143 ms for OKVIS-S, 100 ms for Shu et al., and 98.5 ms for DVLO (Liu et al., 7 Sep 2025). Since KITTI LiDAR operates at 10 Hz, corresponding to a 100 ms period, the paper describes DVLO4D as real-time capable (Liu et al., 7 Sep 2025).

Method	Inference time
DVLO4D	82 ms
DVLO	98.5 ms
DV-LOAM	167 ms
PL-LOAM	200 ms

The ablations tie these results to individual components. Removing SQF raises $(I_S, I_T)$02 from 0.73 to 0.98 and $(I_S, I_T)$03 from 0.37 to 0.57; removing TIU raises them to 0.92 and 0.54; removing CAL raises them to 0.83 and 0.49 (Liu et al., 7 Sep 2025). The history length study further indicates the best results at $(I_S, I_T)$04, with shorter or longer histories performing worse (Liu et al., 7 Sep 2025). Within the evidence presented, the overall performance is not attributable to any single addition alone but to the cumulative effect of sparse query fusion, temporal memory, and clip-level supervision.

6. Robustness, operational assumptions, and research directions

DVLO4D assumes known LiDAR–camera calibration for the projection $(I_S, I_T)$05, but the paper argues that it is relatively robust to calibration imperfections because of deformable sampling around each projected point and because sparse query-based fusion can shift sampling points to compensate for small misalignments (Liu et al., 7 Sep 2025). The framework is described as not tightly bound to a specific LiDAR resolution or camera field of view and, in principle, adaptable to different sensor setups by retraining (Liu et al., 7 Sep 2025).

Robustness is quantified under temporal downsampling and additive LiDAR noise. Under temporal downsampling from 10 Hz to 5 Hz LiDAR, EfficientLO’s mean $(I_S, I_T)$06 changes from 0.86 to 0.99, a 15% increase, whereas DVLO4D changes from 0.73 to 0.75, approximately a 3% increase (Liu et al., 7 Sep 2025). With Gaussian noise added to LiDAR points, EfficientLO changes from 0.86 to 0.97, approximately a 13% increase, whereas DVLO4D again changes from 0.73 to 0.75, approximately a 3% increase (Liu et al., 7 Sep 2025). The paper attributes this to temporal aggregation in TIU, the ability of each LiDAR query to attend to image regions even under sparsity or noise, and the temporal consistency induced by CAL (Liu et al., 7 Sep 2025).

Dynamic objects and occlusions are described as being partially mitigated by three mechanisms: large-scale, coarsest-level ego features for global scene context; historical features in MFB, which can contribute when a region is temporarily occluded; and historical pose dynamics in MPB, which provide a prior on plausible motion and reduce the influence of transient outliers (Liu et al., 7 Sep 2025). This is consistent with the broader architecture, in which temporal memory supplements instantaneous geometric correspondence.

The implementation details reported are specific: training uses an NVIDIA RTX 4090, Adam with $(I_S, I_T)$07, an initial learning rate of $(I_S, I_T)$08 with exponential decay every 13 epochs to $(I_S, I_T)$09, 300 epochs, and batch size 8 (Liu et al., 7 Sep 2025). KITTI sequences 00–06 are used for training and 07–10 for testing. LiDAR preprocessing uses sparse sampling and augmentation strategies as in EfficientLO, including random downsampling and jittering, and a fusion mask indicates which LiDAR queries can be meaningfully fused with imagery, for example when points fall outside the camera field of view (Liu et al., 7 Sep 2025).

The paper also delineates limitations. DVLO4D requires supervised training with ground-truth poses, remains odometry-only without explicit mapping or loop closure, still depends on approximate LiDAR–camera extrinsics despite some robustness to miscalibration, and incurs increased memory usage because of MFB, MPB, transformers, LSTM, and clip-based training (Liu et al., 7 Sep 2025). Suggested extensions include multi-view or multi-LiDAR variants, integrating DVLO4D as a front-end in a full SLAM system with mapping and loop closure, self- or semi-supervised training with geometric self-supervision, domain adaptation to different weather, lighting, and sensor types, and uncertainty modeling based on temporal pose history and loss weights (Liu et al., 7 Sep 2025).

Taken together, DVLO4D occupies a specific position in visual–LiDAR odometry research. It inherits the cross-modal motivation and cylindrical projection lineage of DVLO (Liu et al., 2024), but redefines the core computational strategy around sparse query-based fusion, temporal feature and pose memory, and collective clip-level optimization (Liu et al., 7 Sep 2025). The documented outcome is a system that, within the reported KITTI and Argoverse evaluations, combines state-of-the-art accuracy, robustness under rate changes and noise, and 82 ms real-time-capable inference (Liu et al., 7 Sep 2025).