Spatiotemporal Point Cloud Registration

• Spatiotemporal point cloud registration is the process of aligning 3D scans captured at different times, overcoming challenges such as evolving geometry, low overlap, and scale uncertainty.
• It leverages diverse methods, from pairwise rigid alignment and pose-graph optimization to learned denoising and autoregressive refinement, to ensure global spatial consistency.
• Applications span construction-scale indoor mapping, automotive LiDAR, and SLAM, where robust registration enhances mapping accuracy and facilitates longitudinal scene monitoring.

Searching arXiv for recent and relevant papers on spatiotemporal point cloud registration and related registration under temporal change, low overlap, and sequence consistency. Spatiotemporal point cloud registration (PCR) is the estimation of transformations that align point cloud observations acquired at different times into a coherent spatial or spatiotemporal representation. In current literature, the term spans pairwise rigid registration between temporally separated fragments, multi-way synchronization of many fragments into a global frame, and sequence-level global registration that explicitly exploits temporal continuity. The problem becomes materially different from conventional static registration when geometry and topology evolve, overlap is small or uneven, scale is uncertain, or the scene contains dynamic and transient structure. Those conditions arise in construction-scale indoor mapping, automotive LiDAR, monocular SLAM, remote sensing, and distributed swarm formation (Sun et al., 2023, Drory et al., 2022).

1. Problem definition and mathematical structure

A common formulation treats pairwise PCR as rigid alignment between a source point cloud and a target point cloud. In the Nothing Stands Still (NSS) benchmark, pairwise spatiotemporal registration estimates the rigid transform between two partial point clouds that may come from the same temporal stage or from different stages separated by weeks to months, while multi-way spatiotemporal registration integrates many fragments spanning multiple stages into a single global coordinate frame via pose-graph optimization (Sun et al., 2023). The basic robust pairwise objective is

$$\min_{R\in SO(3),\, t\in\mathbb{R}^3} \sum_i w_i\, \rho\!\left(\|R x_i + t - y_{\pi(i)}\|_2^2\right),$$

where $\rho$ is a robust loss and $w_i$ are weights. For partial overlap, NSS also formalizes trimmed ICP as

$$\min_{R,t,\mathcal{I}\subset\{1,\dots,N\},\, |\mathcal{I}|=\tau N} \sum_{i\in\mathcal{I}} \|R x_i + t - y_{\pi(i)}\|_2^2.$$

Multi-way registration is posed on $SE(3)$ as

$\min_{\{T_i\}} \sum_{(i,j)\in\mathcal{E}} \rho\!\left(\|\Log(T_i^{-1} T_{ij} T_j)\|_\Sigma^2\right),$

with edges induced by successful pairwise registrations (Sun et al., 2023).

Sequence-oriented work broadens this view. STREP formulates global registration of a temporal sequence as joint optimization over a sequence of latent variables and network parameters, with temporal fusion

$z_1=\tilde z_1,\qquad z_k=\tilde z_k + w z_{k-1},$

and a combined local Chamfer objective plus global occupancy loss, so that the entire sequence is aligned to a single global canonical frame rather than treated as isolated pairwise problems (Wang et al., 2020). FLIP adopts a different formalization: at each time index $m$, each agent solves a per-frame similarity registration problem,

$$\min_{\mathbf{R}_m,\mathbf{t}_m,s_m} \sum_{j\in N_a\setminus\{i\}} \|p_j^{des} - (s_m\mathbf{R}_m p_j(m)+\mathbf{t}_m)\|_2,$$

then maps its own desired formation position through the estimated $(s_m,\mathbf R_m,\mathbf t_m)$ to obtain an Optimal Formation Position Sequence (OFPS). The paper states that there is no extra temporal consistency term inside the PCR objective; temporal coupling is introduced by repeating PCR across the horizon and using the resulting OFPS as trajectory constraints (Zhou et al., 28 May 2026).

When scale is not fixed, PCR-Pro models alignment as a similarity transform $\rho$0 with $\rho$1, and uses keyframe-based scale estimation before rigid ICP and pose-graph insertion. This places spatiotemporal PCR in a broader family that includes both $\rho$2 and $\rho$3 estimation, depending on sensing and reconstruction conditions (Bhutta et al., 2018).

2. Benchmarks, datasets, and evaluation protocols

The NSS benchmark is the most explicit benchmark for large temporal and structural change. It defines spatiotemporal PCR as registering two or more 3D point cloud fragments of the same scene captured at different times into a single coherent spatiotemporal map, and evaluates both pairwise and multi-way settings (Sun et al., 2023). NSS contains 6 large-scale indoor areas at building-floor scale, with average $\rho$4 per area, captured over 2–6 temporal stages per area with weeks–months between stages. Five areas are under active construction; one area is pre-/post-renovation with significant structural and functional changes. The benchmark defines three scenarios: Original, Cross-area, and Cross-stage. It uses Relative Rotation Error (RRE), Relative Translation Error (RTE), registration recall, Overlap Ratio (OR), Temporal Change Ratio (TCR), and curvature-based geometric complexity. The pairwise success criterion is $\rho$5 and $\rho$6, and OR/TCR use a distance threshold $\rho$7 (Sun et al., 2023).

NSS is designed precisely because earlier benchmarks emphasize much smaller scene evolution. The paper states that 3DMatch and 3DLoMatch are substantially easier, and that RIO10 focuses on object relocation/addition/removal in lived-in rooms, with a much smaller same-to-different stage gap than NSS. Under NSS protocol, methods that achieve approximately $\rho$8–$\rho$9 recall on 3DMatch fall to approximately $w_i$0–$w_i$1 recall on NSS for all pairs, with same-stage approximately $w_i$2–$w_i$3 and different-stage approximately $w_i$4–$w_i$5 (Sun et al., 2023).

Automotive LiDAR evaluation emphasizes a different stress regime: large $w_i$6 motion, dynamic objects, low overlap, and domain shift across cities. The automotive stress-test paper constructs balanced registration sets in 6D motion space and reports that KITTI-10m is saturated, motivating harder balanced splits for NuScenes-Boston, NuScenes-Singapore, and Apollo-Southbay. It uses success thresholds $w_i$7 and $w_i$8, reports Recall and wall-clock runtime, and shows that recall begins degrading measurably below overlap $w_i$9, though even at $$\min_{R,t,\mathcal{I}\subset\{1,\dots,N\},\, |\mathcal{I}|=\tau N} \sum_{i\in\mathcal{I}} \|R x_i + t - y_{\pi(i)}\|_2^2.$$0 most registrations still succeed with the best pipeline (Drory et al., 2022).

Sequence-level unsupervised evaluation also appears in STREP, which uses 2D simulated LIDAR-like trajectories and 3D Active Vision Dataset (AVD) sequences, reporting Absolute Trajectory Error (ATE) and point-wise $$\min_{R,t,\mathcal{I}\subset\{1,\dots,N\},\, |\mathcal{I}|=\tau N} \sum_{i\in\mathcal{I}} \|R x_i + t - y_{\pi(i)}\|_2^2.$$1 distance between the ground-truth point cloud and the registered result (Wang et al., 2020). Low-overlap pairwise evaluation remains central in diffusion-based work: DiffusionPCR reports 3DMatch and 3DLoMatch for indoor registration and KITTI Odometry for outdoor sequential scans, with 3DLoMatch explicitly targeting 10–30% overlap and KITTI evaluating frame pairs at least 10 m apart (Chen et al., 2023). EADReg uses KITTI, NuScenes, and Apollo-SouthBay with frame interval $$\min_{R,t,\mathcal{I}\subset\{1,\dots,N\},\, |\mathcal{I}|=\tau N} \sum_{i\in\mathcal{I}} \|R x_i + t - y_{\pi(i)}\|_2^2.$$2, success thresholds $$\min_{R,t,\mathcal{I}\subset\{1,\dots,N\},\, |\mathcal{I}|=\tau N} \sum_{i\in\mathcal{I}} \|R x_i + t - y_{\pi(i)}\|_2^2.$$3 and $$\min_{R,t,\mathcal{I}\subset\{1,\dots,N\},\, |\mathcal{I}|=\tau N} \sum_{i\in\mathcal{I}} \|R x_i + t - y_{\pi(i)}\|_2^2.$$4, and downsampled LiDAR frames tailored to outdoor sparsity and scale (Gong et al., 2024).

3. Methodological families

Current spatiotemporal PCR methods divide into robust correspondence-based estimators, learned pairwise regressors or refiners, and sequence- or graph-level global methods. Classical and benchmarked pairwise baselines in NSS include FPFH+RANSAC, FCGF, D3Feat, PREDATOR, and GeoTransformer. The quantitative pattern is sharp: on the Original split, same-stage recall is 92.99% for PREDATOR, 55.59% for GeoTransformer, 36.51% for D3Feat, and 30.82% for FPFH, whereas different-stage recall falls to 28.42%, 17.51%, 4.76%, and 0.42%, respectively (Sun et al., 2023). This establishes overlap-aware attention and coarse-to-fine attention-based matching as the strongest evaluated pairwise baselines under large temporal change.

PCRNet represents a correspondence-free regression line of work. It uses a Siamese PointNet encoder, concatenated global descriptors, and a regression head that outputs translation and a unit quaternion. The single-shot model predicts the transform in one forward pass, while iterative PCRNet composes successive transforms according to

$$\min_{R,t,\mathcal{I}\subset\{1,\dots,N\},\, |\mathcal{I}|=\tau N} \sum_{i\in\mathcal{I}} \|R x_i + t - y_{\pi(i)}\|_2^2.$$5

PCRNet is trained with Earth Mover’s Distance and can terminate early when $$\min_{R,t,\mathcal{I}\subset\{1,\dots,N\},\, |\mathcal{I}|=\tau N} \sum_{i\in\mathcal{I}} \|R x_i + t - y_{\pi(i)}\|_2^2.$$6. On a noisy car dataset, the single-shot model reports mean rotation error $$\min_{R,t,\mathcal{I}\subset\{1,\dots,N\},\, |\mathcal{I}|=\tau N} \sum_{i\in\mathcal{I}} \|R x_i + t - y_{\pi(i)}\|_2^2.$$7, mean translation error $$\min_{R,t,\mathcal{I}\subset\{1,\dots,N\},\, |\mathcal{I}|=\tau N} \sum_{i\in\mathcal{I}} \|R x_i + t - y_{\pi(i)}\|_2^2.$$8, time $$\min_{R,t,\mathcal{I}\subset\{1,\dots,N\},\, |\mathcal{I}|=\tau N} \sum_{i\in\mathcal{I}} \|R x_i + t - y_{\pi(i)}\|_2^2.$$9, and AUC $SE(3)$0, while iterative PCRNet reports mean rotation error $SE(3)$1, mean translation error $SE(3)$2, time $SE(3)$3, and AUC $SE(3)$4 (Sarode et al., 2019). The paper explicitly positions these properties as suitable for frame-to-frame alignment in tracking, mapping, and reconstruction.

DiffusionPCR reframes registration as a multi-step denoising process on $SE(3)$5, interpolating from a prior transform toward the ground truth with Slerp for rotations and linear interpolation for translations. Its denoising backbone is GeoTransformer augmented with overlap-aware one-way self- and cross-attention conditioned on the previous pose estimate. The paper reports 3DLoMatch recall up to 80.4%, 3DMatch recall 95.3% with LGR estimator and 2D priors, 3DLoMatch recall 81.6% under the same setting, and KITTI performance of $SE(3)$6, $SE(3)$7, and $SE(3)$8 with only 2 iterative steps (Chen et al., 2023). The central methodological claim is that training on transformations with varying registration qualities, rather than only random noise or a fixed pretrained model output, improves adaptiveness under small overlap.

Outdoor LiDAR has motivated additional coarse-to-fine designs. EADReg uses a hierarchical detector-descriptor backbone, a Bi-directional Gaussian Mixture Model (BGMM) coarse stage for outlier rejection, and an efficient autoregressive diffusion fine stage over top-$SE(3)$9 local candidate neighborhoods rather than dense $\min_{\{T_i\}} \sum_{(i,j)\in\mathcal{E}} \rho\!\left(\|\Log(T_i^{-1} T_{ij} T_j)\|_\Sigma^2\right),$0 correspondences. It reports on KITTI $\min_{\{T_i\}} \sum_{(i,j)\in\mathcal{E}} \rho\!\left(\|\Log(T_i^{-1} T_{ij} T_j)\|_\Sigma^2\right),$1, $\min_{\{T_i\}} \sum_{(i,j)\in\mathcal{E}} \rho\!\left(\|\Log(T_i^{-1} T_{ij} T_j)\|_\Sigma^2\right),$2, Recall 100%, and $\min_{\{T_i\}} \sum_{(i,j)\in\mathcal{E}} \rho\!\left(\|\Log(T_i^{-1} T_{ij} T_j)\|_\Sigma^2\right),$3 runtime; on NuScenes $\min_{\{T_i\}} \sum_{(i,j)\in\mathcal{E}} \rho\!\left(\|\Log(T_i^{-1} T_{ij} T_j)\|_\Sigma^2\right),$4, $\min_{\{T_i\}} \sum_{(i,j)\in\mathcal{E}} \rho\!\left(\|\Log(T_i^{-1} T_{ij} T_j)\|_\Sigma^2\right),$5, Recall 100%, and $\min_{\{T_i\}} \sum_{(i,j)\in\mathcal{E}} \rho\!\left(\|\Log(T_i^{-1} T_{ij} T_j)\|_\Sigma^2\right),$6; and on Apollo-SouthBay $\min_{\{T_i\}} \sum_{(i,j)\in\mathcal{E}} \rho\!\left(\|\Log(T_i^{-1} T_{ij} T_j)\|_\Sigma^2\right),$7, $\min_{\{T_i\}} \sum_{(i,j)\in\mathcal{E}} \rho\!\left(\|\Log(T_i^{-1} T_{ij} T_j)\|_\Sigma^2\right),$8, Recall 99.92%, and $\min_{\{T_i\}} \sum_{(i,j)\in\mathcal{E}} \rho\!\left(\|\Log(T_i^{-1} T_{ij} T_j)\|_\Sigma^2\right),$9 (Gong et al., 2024).

Robust minimal-structure estimation remains important under extreme outlier rates. PCR-99 uses deterministic 3-point sampling, pairwise log-scale consistency scoring, and $z_1=\tilde z_1,\qquad z_k=\tilde z_k + w z_{k-1},$0 triplet log-scale prescreening to handle outlier ratios up to 99% for both $z_1=\tilde z_1,\qquad z_k=\tilde z_k + w z_{k-1},$1 and $z_1=\tilde z_1,\qquad z_k=\tilde z_k + w z_{k-1},$2 problems. At 99% outlier ratio, it outperforms the state of the art for both known-scale and unknown-scale problems, and the paper reports large median speedups relative to prior methods (Lee et al., 2024). TurboReg similarly focuses on correspondence robustness, but replaces exponential-time maximal clique search with a linear-time Pivot-Guided Search over a highly-constrained compatibility graph. On the 3DMatch+FCGF dataset, TurboReg (1K) operates $z_1=\tilde z_1,\qquad z_k=\tilde z_k + w z_{k-1},$3 faster than 3DMAC while also achieving higher recall, and on KITTI it reports Recall 98.56% with FPFH and 98.38% with FCGF at approximately 61 FPS on GPU (Yan et al., 2 Jul 2025).

A different robustification strategy appears in the probabilistic self-update framework based on local correspondences and line vector sets. It combines global RANSAC and local RANSAC, constructs local sets by angle histogram statistics and line-vector length preservation, and refines them with a probabilistic self-updating strategy before weighted SVD. Across 3DMatch, KITTI, and WHU-TLS, the paper reports at least a 10% RMSE improvement over state-of-the-art methods on average and an average runtime of approximately $z_1=\tilde z_1,\qquad z_k=\tilde z_k + w z_{k-1},$4 (Chung et al., 29 Apr 2026).

4. Core difficulties: temporal change, low overlap, repetition, and transfer

The central empirical finding of NSS is that large temporal changes devastate pairwise recall across all evaluated methods. On the Cross-area split, same-stage versus different-stage recall drops by approximately 40.8 percentage points on average, and Cross-area is the most challenging scenario while Cross-stage is the easiest because it benefits from building-specific training (Sun et al., 2023). The benchmark attributes this to evolving geometry and topology, small and uneven overlap, violated assumptions about spatial continuity and stable local descriptors, and the concentration of correspondences on planar or repetitive structures such as studs and corridors.

NSS explicitly characterizes difficulty with OR and TCR. Recall increases with OR for all splits, but Cross-area remains the hardest even at high OR. Recall decreases monotonically with TCR, and low OR with high TCR produces the regime of insufficient static anchors and false matches concentrated on smooth planes. The benchmark also documents repetitive structures, cross-area domain shift, and multi-way drift or fragment pulling when many pairwise edges are wrong (Sun et al., 2023). This suggests that temporal persistence, rather than geometric saliency alone, becomes the critical latent variable under strong scene evolution.

Automotive LiDAR stress tests expose a related but not identical failure profile. In that regime, overlap is again the main bottleneck: failure analysis shows recall begins degrading measurably below overlap approximately 0.35, while dynamic objects, long-range motion, vegetation, and repetitive outdoor structure intensify outlier rates. The same paper reports that modern pipelines built on FCGF and a strong RANSAC variant are “extremely rotation invariant,” with negligible correlation between rotation magnitude and failure rate except at very extreme rotations. Transferability is limited: cross-domain tests show an average drop of approximately 16 percentage points when training and testing across different cities or datasets (Drory et al., 2022).

Low-overlap indoor registration and sparse outdoor sequential scans motivate overlap-aware refinement mechanisms. DiffusionPCR attributes its robustness under small overlap to one-way self- and cross-attention driven by anchors derived from the previous transform, plus training across a continuum from inaccurate to accurate priors (Chen et al., 2023). EADReg frames the outdoor difficulty as sparsity, irregularity, dynamic objects, and huge point scale, and addresses it by BGMM purification and top-$z_1=\tilde z_1,\qquad z_k=\tilde z_k + w z_{k-1},$5 regional correspondence prediction rather than global dense correspondence generation (Gong et al., 2024). Both lines are consistent with the broader observation that partial matching and explicit rejection of non-overlapping structure are not peripheral implementation choices but primary design requirements in spatiotemporal PCR.

5. Global consistency, synchronization, and sequence integration

Pairwise registration alone is usually insufficient for long temporal horizons. NSS therefore evaluates multi-way PCR by building dense pose graphs from successful pairwise registrations and optimizing absolute poses over all fragments. It compares a robust pose-graph solver based on Choi et al. and a learned synchronization GNN by Yew and Lee. The learned synchronization model is consistently stronger under heavy outliers: on Cross-area with PREDATOR edges it improves by +20.96 percentage points to 77.28%, on Cross-stage with GeoTransformer edges by +11.71 percentage points, and on Original with GeoTransformer edges by +9.65 percentage points to 65.35% (Sun et al., 2023). The reported valid-edge ratios after pruning are only approximately 26–32% for the Choi-style solver, which underscores the severity of temporal outliers in spatiotemporal pose graphs.

STREP tackles global consistency from a different direction. Rather than optimizing a pose graph, it jointly optimizes a temporal sequence of latent features and decodes them into a temporally and spatially continuous pose sequence. Its local objective is the Chamfer distance over neighboring frames,

$z_1=\tilde z_1,\qquad z_k=\tilde z_k + w z_{k-1},$6

combined with a global occupancy loss that encourages the transformed frames to form a consistent map. The paper states that temporal fusion of latents is the main contributor to improved global registration beyond latent-space search alone, and reports that the proposed method beats DeepMapping on both simulated 2D and real 3D datasets (Wang et al., 2020).

In SLAM-oriented settings, global integration also requires uncertainty. PCR-Pro estimates scale and rigid alignment between sparse keyframe point clouds, then derives a closed-form covariance and an information matrix for pose-graph SLAM. After ICP converges, the edge information matrix is set as $z_1=\tilde z_1,\qquad z_k=\tilde z_k + w z_{k-1},$7 and incorporated into a graph cost of the form

$z_1=\tilde z_1,\qquad z_k=\tilde z_k + w z_{k-1},$8

with $z_1=\tilde z_1,\qquad z_k=\tilde z_k + w z_{k-1},$9. This links spatiotemporal PCR to uncertainty-aware back-end optimization rather than only front-end pairwise alignment (Bhutta et al., 2018).

FLIP shows that similar principles can be deployed in a non-SLAM distributed setting. Each agent performs per-frame RANSAC-based $m$0 registration against the desired formation positions of all other agents, extracts an OFPS, and then solves a trajectory optimization problem constrained by that sequence. The paper reports end-to-end planning times below $m$1 up to 100 agents, OFPS-only maximum computation time below $m$2 up to 1000 agents, and resilience to approximately 12% abnormal agents while keeping the remaining agents’ formation error below 0.5 (Zhou et al., 28 May 2026). A plausible implication is that spatiotemporal PCR is becoming a generic mechanism for sequence-consistent spatial assignment wherever rigid or similarity structure is more informative than combinatorial matching.

6. Applications, practical pipelines, and open directions

Construction-scale indoor mapping is the most explicit application domain in NSS. The benchmark states that spatiotemporal maps enable quantitative progress monitoring and quality control, reducing costly rework, with a reported 52% share in out-of-estimate costs, and that detailed historical geometry supports material inventories for reuse, noting that up to 95% of non-hazardous construction and demolition waste is reusable or recyclable but often undocumented (Sun et al., 2023). In this setting, spatiotemporal PCR is not merely a localization primitive; it is part of longitudinal geometry management under structural change.

Automotive and outdoor LiDAR applications emphasize real-time pairwise robustness within larger odometry or mapping pipelines. The automotive stress-test recommends FCGF + GPF + RANSAC with PROSAC, ELC, and LO-RANSAC, followed by ICP for high accuracy, or MNN-filtered RANSAC when speed is critical. On NuScenes-Boston-Balanced with ICP, RANSAC (GPF) reports 91.90% recall at 0.257 s, while RANSAC (MNN) reports 89.01% recall at 0.099 s (Drory et al., 2022). DiffusionPCR and EADReg extend this regime toward learned multi-step refinement, with the former emphasizing overlap-aware denoising on $m$3 and the latter emphasizing efficient local autoregressive correspondence generation for large LiDAR scans (Chen et al., 2023, Gong et al., 2024).

For practical spatiotemporal PCR in construction-scale indoor environments, NSS recommends a pipeline with overlap screening, multi-scale attention-based descriptors, robust losses such as Huber or Geman–McClure, trimmed objectives for partial overlap, ICP variants restricted to high-confidence overlaps, and learned synchronization for multi-way optimization. The benchmark further recommends evaluating across OR/TCR bins, rejecting non-overlapping pairs early, emphasizing semantic or structural anchors, and reweighting edges by persistence and uncertainty (Sun et al., 2023). This suggests that a modern spatiotemporal pipeline is inherently hierarchical: pair rejection, overlap localization, robust pairwise estimation, and global synchronization are all first-class stages.

Open research directions are explicitly identified across the cited literature. NSS calls for time-aware descriptors robust to topology change, principled partial matching under extreme outliers and low overlap, multi-temporal SLAM back-ends with temporal edges and uncertainty-aware synchronization, semantics-driven registration, and hybrid rigid/non-rigid models (Sun et al., 2023). DiffusionPCR proposes temporal consistency losses, recurrent refinement across frames, motion models and inertial priors, and pose-graph integration as natural extensions of its multi-step denoising framework (Chen et al., 2023). PCR-Pro indicates the continuing relevance of $m$4 estimation and uncertainty propagation when scale drift or monocular reconstruction is present (Bhutta et al., 2018). Taken together, these trajectories indicate that spatiotemporal PCR is moving away from the assumption that time merely perturbs a static scene, and toward the view that persistence, uncertainty, and sequence structure are primary registration variables.