Temporal Point Clouds (TPCs)

• Temporal Point Clouds (TPCs) are time-indexed collections of 3D point sets that evolve to capture spatial-temporal dynamics for tasks such as forecasting and interpolation.
• Methodologies for TPCs range from LiDAR forecasting with range images to point-native and sparse voxel approaches, each balancing efficiency and geometric fidelity.
• TPCs enable applications in LiDAR prediction, biological alignment, greenhouse monitoring, and motion forecasting, demonstrating improved accuracy and compression rates.

Temporal Point Clouds (TPCs) are time-indexed geometric representations in which each time step is a 3D point set, or more generally a set of spatio-temporal samples with optional attributes, evolving over time. In the LiDAR forecasting formulation, a TPC over a temporal window of length $N$ and horizon $F$ is the ordered sequence $S(t)=[P_{t-N+1},\dots,P_t]$ with future target $$[P_{t+1},\dots,P_{t+F}] = f_\theta(P_{t-N+1},\dots,P_t)$$, where each scan is an unordered set $P_t=\{p^{(n,t)}\}$ and $p^{(n,t)} \in \mathbb{R}^3$ (Mersch et al., 2021). Other formulations generalize the same idea to irregular spatio-temporal samples $S=\{(x_i,t_i,f_i,e_i)\}$ with arbitrary query points, colored greenhouse scans with cross-session associations, minute-resolved biological 3D+t sequences, and skeleton-agnostic motion representations with persistent point identities (Merrill et al., 2021, Fusaro et al., 2024, Chen et al., 10 Feb 2025, Mo et al., 27 Jul 2025). This breadth indicates that TPCs are not a single data structure tied to one sensor, but a family of temporal geometric representations whose exact semantics depend on the task.

1. Formal scope and representational variants

The most common TPC definition treats each frame as an unordered point set in $\mathbb{R}^3$, with temporal order carried by the sequence rather than by intra-frame indexing. This is explicit in LiDAR prediction, scene flow, completion, interpolation, and compression, where the object of interest is a sequence $\{P_t\}$ of scans or partial shapes (Mersch et al., 2021, Wang et al., 2021, Shi et al., 2022, Jiang et al., 2024, Ruan et al., 2024). In this setting, temporal learning is usually expressed as forecasting, interpolation, warping, or compression across frames.

A broader formalization appears in irregular spatio-temporal learning, where a TPC is a set of tuples $(x_i,t_i,f_i,e_i)$ comprising spatial location, timestamp, measured features, and an optional entity identifier. Under this view, the central problem is not necessarily predicting the next point cloud, but answering arbitrary queries $F$0 by learning continuous functions over irregularly sampled space-time without voxelization (Merrill et al., 2021). This formulation shifts emphasis from sequence-to-sequence forecasting toward continuous spatio-temporal regression.

Session-based formulations introduce another layer: point clouds may be temporally sparse rather than densely sampled. In horticultural monitoring, temporally indexed greenhouse scans are colored point clouds $F$1 collected about one week apart, with outputs consisting of per-session fruit instances $F$2 and cross-session associations $F$3 (Fusaro et al., 2024). In zebrafish embryogenesis, the sequence is minute-resolved 3D+t nuclei positions over $F$4 frames from 4.7 to 10.0 hpf, used for developmental alignment rather than forecasting (Chen et al., 10 Feb 2025).

Human-motion TPCs complicate the standard “unordered set” assumption. PUMPS represents motion as frame-wise point clouds with fixed $F$5 points per frame, but crucially maintains consistent point identities across time by conditioning each trajectory on a fixed latent Gaussian noise vector (Mo et al., 27 Jul 2025). A common misconception is therefore that TPCs are always fully anonymous sets; in most work they are unordered within each frame, but some formulations deliberately preserve temporal point identity when the downstream task requires coherent trajectories.

Formulation	Per-time representation	Primary target
LiDAR forecasting	$F$6	Future scans $F$7
Irregular spatio-temporal regression	$F$8 samples	Arbitrary query prediction
Greenhouse monitoring	Colored point cloud $F$9	Instances $S(t)=[P_{t-N+1},\dots,P_t]$0 and associations $S(t)=[P_{t-N+1},\dots,P_t]$1
Biological 3D+t alignment	Nuclei point cloud $S(t)=[P_{t-N+1},\dots,P_t]$2	Reference-time alignment
Skeleton-agnostic motion	Fixed-size point cloud $S(t)=[P_{t-N+1},\dots,P_t]$3	Motion synthesis in latent TPC space

2. Operationalizing TPCs for learning

A major divide in the literature concerns how temporal point sets are converted into learnable tensors or processed directly. One widely used operationalization is the LiDAR range image. In the sensor-level forecasting framework, each 3D scan is projected with spherical coordinates $S(t)=[P_{t-N+1},\dots,P_t]$4, $S(t)=[P_{t-N+1},\dots,P_t]$5, $S(t)=[P_{t-N+1},\dots,P_t]$6 into a 2D range image via $S(t)=[P_{t-N+1},\dots,P_t]$7, then $S(t)=[P_{t-N+1},\dots,P_t]$8 past range images are stacked into a spatio-temporal tensor $S(t)=[P_{t-N+1},\dots,P_t]$9; inverse spherical mapping reconstructs predicted points from range and validity masks (Mersch et al., 2021). ATPPNet adopts the same range-image premise, operating on $$[P_{t+1},\dots,P_{t+F}] = f_\theta(P_{t-N+1},\dots,P_t)$$0 KITTI scans and $$[P_{t+1},\dots,P_{t+F}] = f_\theta(P_{t-N+1},\dots,P_t)$$1 nuScenes scans, and predicts both future range images and re-projection masks (Pal et al., 2024). This representation preserves LiDAR topology and makes 3D temporal prediction amenable to convolutional architectures.

Other work avoids rasterization and retains the irregular point structure. TemporalPointConv defines neighborhoods directly in continuous space-time: spatial aggregation occurs within a narrow temporal window, and temporal aggregation occurs within a narrow spatial band, enabling interpolation, extrapolation, backcasting, and nowcasting over irregular samples (Merrill et al., 2021). FESTA likewise operates natively on point clouds for scene flow, with attention-based spatial abstraction and motion-rectified temporal abstraction over paired frames (Wang et al., 2021). These approaches keep exact point coordinates in the computational graph and avoid the quantization trade-offs of voxels or images.

Sparse voxelization remains important when local context and memory efficiency matter. The horticultural monitoring pipeline voxelizes colored greenhouse scans with fixed voxel size $$[P_{t+1},\dots,P_{t+F}] = f_\theta(P_{t-N+1},\dots,P_t)$$2 for Minkowski-style sparse convolutions and defines fixed-radius support neighborhoods $$[P_{t+1},\dots,P_{t+F}] = f_\theta(P_{t-N+1},\dots,P_t)$$3 with $$[P_{t+1},\dots,P_{t+F}] = f_\theta(P_{t-N+1},\dots,P_t)$$4 (Fusaro et al., 2024). For dynamic compression, NeRC$$[P_{t+1},\dots,P_{t+F}] = f_\theta(P_{t-N+1},\dots,P_t)$$5 and 4D-NeRC$$[P_{t+1},\dots,P_{t+F}] = f_\theta(P_{t-N+1},\dots,P_t)$$6 voxelize point clouds at $$[P_{t+1},\dots,P_{t+F}] = f_\theta(P_{t-N+1},\dots,P_t)$$7-bit resolution, partition space into cubes, and learn coordinate-based occupancy and attribute fields over 3D or 4D voxel coordinates (Ruan et al., 2024). These representations are not point-native in the strict sense, but they still model temporal point clouds as temporally evolving occupancy structures.

A third class of operationalizations imposes sequence structure without dense gridding. Point2Point maps each frame to a locality-preserving 1D ordering using the Hilbert space-filling curve and then applies 1D convolutions to Hilbert-sorted sequences for spatio-temporal occupancy prediction (Pandhare, 2023). WiFi-CSI reconstruction takes a different route again: a transformer consumes a CSI tensor $$[P_{t+1},\dots,P_{t+F}] = f_\theta(P_{t-N+1},\dots,P_t)$$8 and outputs $$[P_{t+1},\dots,P_{t+F}] = f_\theta(P_{t-N+1},\dots,P_t)$$9 3D points per window, which can be slid across time to form a TPC $P_t=\{p^{(n,t)}\}$0 (Määttä et al., 2024). This suggests that TPCs can be either native observations or geometric outputs synthesized from non-geometric temporal signals.

3. Modeling strategies across the literature

Architecturally, TPC research is notably heterogeneous. In future LiDAR prediction, one influential design is a U-Net-like encoder-decoder with 3D convolutions over temporal and spatial axes. The encoder compresses spatial resolution while reducing temporal extent, the decoder mirrors this process with transposed 3D convolutions, skip connections preserve detail, and horizontal circular padding stabilizes panoramic continuity across azimuth wrap-around (Mersch et al., 2021). ATPPNet extends the range-image family by combining $P_t=\{p^{(n,t)}\}$1 Conv-LSTM branches, dual channel-wise and spatial attention, and a complementary 3D-CNN branch at the highest scale, then decoding future range-plus-mask outputs (Pal et al., 2024). These tensorized models treat a TPC as a structured temporal image volume.

Point-native models instead build temporal reasoning from local geometric operators. TemporalPointConv stacks sequential spatial PointConv and temporal PointConv layers, with residual concatenation and a final query PointConv that answers arbitrary space-time queries (Merrill et al., 2021). FESTA addresses the instability of point abstraction under irregular sampling with Spatial Abstraction with Attention (SA$P_t=\{p^{(n,t)}\}$2) and Temporal Abstraction with Attention (TA$P_t=\{p^{(n,t)}\}$3), yielding stable representatives and motion-rectified cross-frame aggregation for scene flow (Wang et al., 2021). TPCN, for motion forecasting, combines PointNet++-style point-set learning, sparse voxel convolutions, multi-interval temporal learning, and instance pooling over agent trajectories and lane polylines (Ye et al., 2021).

Association-heavy TPC problems add explicit matching modules. In temporal fruit monitoring, MinkPanoptic performs point-cloud panoptic segmentation and mean-shift clustering, then a MinkowskiNet descriptor encoder and attention-based matching network associate fruit instances across sessions with a learned “no-match” option and an injectivity regularizer (Fusaro et al., 2024). In unsupervised global registration, STREP dispenses with an encoder in the usual sense and instead optimizes a temporally fused latent sequence $P_t=\{p^{(n,t)}\}$4, which is decoded into a temporally and spatially continuous sequence of rigid transforms for global alignment (Wang et al., 2020).

Latent and implicit models handle TPCs differently again. The zebrafish alignment pipeline uses a FoldingNet-style autoencoder with a spherical template and a Modified Chamfer Distance that averages $P_t=\{p^{(n,t)}\}$5-nearest-neighbor discrepancies, then regresses latent features to developmental time indices with a separate MLP and monotone postprocessing (Chen et al., 10 Feb 2025). NeuroGauss4D-PCI combines iterative Gaussian soft clustering, temporal radial basis function Gaussian residuals, a 4D Gaussian deformation field, and a 4D neural field for continuous point cloud interpolation (Jiang et al., 2024). PUMPS uses a PointTransformer encoder, time-only RoPE self-attention, and per-point latent Gaussian noise identifiers to decode temporally consistent point trajectories while avoiding point-wise spatio-temporal attention (Mo et al., 27 Jul 2025). These lines of work emphasize that temporal structure can be imposed in latent space, parameter space, or coordinate space rather than only by explicit frame-to-frame convolutions.

4. Task families and objective functions

TPCs serve a wide range of temporal geometry tasks. Forecasting is the most explicit instance: LiDAR works predict either the next scan or multiple future scans from past scans, with outputs at sensor level rather than at the level of tracks or boxes (Mersch et al., 2021, Pal et al., 2024). Interpolation asks for intermediate frames at arbitrary timestamps; NeuroGauss4D-PCI predicts $P_t=\{p^{(n,t)}\}$6 from sparse training frames using continuous spatiotemporal fields (Jiang et al., 2024). Completion uses temporal evidence to reconstruct aligned, complete point clouds from sparse, pose-disturbed partial inputs, operating online with a refined output at each frame (Shi et al., 2022).

Other tasks do not predict future geometry directly but still rely on TPC structure. TemporalPointConv answers arbitrary spatio-temporal queries for nowcasting, forecasting, interpolation, and backcasting in irregular domains (Merrill et al., 2021). FESTA estimates dense scene flow between paired point clouds, and its outputs can be interpreted as a motion field over a minimal TPC of length two (Wang et al., 2021). Horticultural monitoring infers both per-session instances and temporal associations under occlusions, growth-induced geometry and color changes, and births or deaths of fruit (Fusaro et al., 2024). STREP solves global registration of temporal sequences, while TPCN uses temporal point sets of agents and maps to forecast trajectories in motion forecasting (Wang et al., 2020, Ye et al., 2021).

The objective landscape is equally varied. Sensor-level LiDAR predictors combine per-pixel range regression, binary cross-entropy for validity masks, and optional Chamfer Distance fine-tuning in point space (Mersch et al., 2021, Pal et al., 2024). Scene flow models optimize flow losses and mask losses; interpolation models employ Chamfer Distance, Earth Mover’s Distance, and smoothness constraints; Point2Point uses Sinkhorn distance for point-set prediction; biological alignment replaces standard Chamfer with Modified Chamfer Distance to preserve density evolution; PUMPS replaces Chamfer with frame-wise linear assignment followed by paired $P_t=\{p^{(n,t)}\}$7 reconstruction (Wang et al., 2021, Jiang et al., 2024, Pandhare, 2023, Chen et al., 10 Feb 2025, Mo et al., 27 Jul 2025). Compression adds D1 PSNR, BD-BR, and bits-per-point criteria, either in explicit spatio-temporal codecs or 4D implicit fields (Feng et al., 2020, Ruan et al., 2024).

A recurring misconception is that temporal learning on point clouds must be expressed through recurrent prediction of future frames. The literature shows otherwise. Some methods regress time indices from latent morphology rather than geometry at future timestamps, some solve cross-session re-identification, some compress sequences by exploiting temporal redundancy, and some use TPCs as a skeleton-agnostic medium for universal motion pre-training (Chen et al., 10 Feb 2025, Fusaro et al., 2024, Ruan et al., 2024, Mo et al., 27 Jul 2025). This suggests that “TPC” names a representational paradigm more than a single canonical benchmark.

5. Empirical behavior across domains

Empirical performance confirms that TPC formulations can be effective across highly dissimilar domains. In future LiDAR prediction on KITTI Odometry, the 3D spatio-temporal convolutional model reports mean full-scale Chamfer Distance $P_t=\{p^{(n,t)}\}$8 compared with Identity $P_t=\{p^{(n,t)}\}$9, Constant Velocity $p^{(n,t)} \in \mathbb{R}^3$0, and Ray Tracing $p^{(n,t)} \in \mathbb{R}^3$1, and predicts 5 future point clouds in $p^{(n,t)} \in \mathbb{R}^3$2 ms ($p^{(n,t)} \in \mathbb{R}^3$3 Hz), faster than a typical 10 Hz LiDAR rate. On Apollo-SouthBay, a KITTI-trained model reaches $p^{(n,t)} \in \mathbb{R}^3$4 mean CD and improves to $p^{(n,t)} \in \mathbb{R}^3$5 after a single epoch of self-supervised fine-tuning (Mersch et al., 2021). ATPPNet reports on KITTI a full-scale mean Chamfer Distance of $p^{(n,t)} \in \mathbb{R}^3$6 and mean Range Loss of $p^{(n,t)} \in \mathbb{R}^3$7, and on nuScenes a mean Chamfer Distance of $p^{(n,t)} \in \mathbb{R}^3$8 and mean Range Loss of $p^{(n,t)} \in \mathbb{R}^3$9; its inference latency is $S=\{(x_i,t_i,f_i,e_i)\}$0 ms on KITTI and $S=\{(x_i,t_i,f_i,e_i)\}$1 ms on nuScenes (Pal et al., 2024).

In irregular spatio-temporal regression, TemporalPointConv outperforms set-function and sparse-convolution baselines on both weather and StarCraft II. On weather nowcasting it achieves normalized loss $S=\{(x_i,t_i,f_i,e_i)\}$2 versus SeFT $S=\{(x_i,t_i,f_i,e_i)\}$3, DeepSets $S=\{(x_i,t_i,f_i,e_i)\}$4, and Minkowski $S=\{(x_i,t_i,f_i,e_i)\}$5; on StarCraft II it reports total error $S=\{(x_i,t_i,f_i,e_i)\}$6 versus SeFT $S=\{(x_i,t_i,f_i,e_i)\}$7, DeepSets $S=\{(x_i,t_i,f_i,e_i)\}$8, and Minkowski $S=\{(x_i,t_i,f_i,e_i)\}$9 (Merrill et al., 2021). For scene flow, FESTA reaches KITTI EPE $\mathbb{R}^3$0, Acc Strict $\mathbb{R}^3$1, and Acc Relax $\mathbb{R}^3$2, while on FlyingThings3D it reports EPE $\mathbb{R}^3$3 for geometry-only and $\mathbb{R}^3$4 for geometry+RGB (Wang et al., 2021).

Sessional and biological TPCs show similar gains from explicit temporal structure. In greenhouse strawberry monitoring, MinkPanoptic yields RQ $\mathbb{R}^3$5, SQ $\mathbb{R}^3$6, PQ $\mathbb{R}^3$7, and IoU $\mathbb{R}^3$8 on $\mathbb{R}^3$9, while the attentive matcher reaches mean $\{P_t\}$0 $\{P_t\}$1 compared with $\{P_t\}$2 for nearest neighbor and $\{P_t\}$3 for the Riccardi descriptor baseline (Fusaro et al., 2024). In zebrafish embryo alignment, the unsupervised 3D+t pipeline achieves an average mismatch of only $\{P_t\}$4 minutes over a $\{P_t\}$5-hour window; with scattering plus rotations the average is $\{P_t\}$6 minutes, whereas centering to the origin degrades performance to $\{P_t\}$7 minutes (Chen et al., 10 Feb 2025). The latter result is especially instructive because it shows that enforcing translation invariance can remove biologically meaningful centroid shifts.

Compression and representation learning extend the empirical landscape further. The real-time spatio-temporal LiDAR codec reports $\{P_t\}$8 to $\{P_t\}$9 compression rate, up to $(x_i,t_i,f_i,e_i)$0 FPS on a desktop CPU and $(x_i,t_i,f_i,e_i)$1 FPS on Jetson TX2, while retaining high downstream registration, detection, and segmentation accuracies (Feng et al., 2020). In dynamic implicit compression, 4D-NeRC$(x_i,t_i,f_i,e_i)$2 achieves internal gains of $(x_i,t_i,f_i,e_i)$3 BD-BR for geometry and $(x_i,t_i,f_i,e_i)$4 for joint geometry+attributes relative to intra-frame NeRC$(x_i,t_i,f_i,e_i)$5, and against baselines reports $(x_i,t_i,f_i,e_i)$6 BD-BR versus G-PCC octree for geometry-only compression (Ruan et al., 2024). PUMPS, operating in a skeleton-agnostic TPC space, reports reconstruction loss $(x_i,t_i,f_i,e_i)$7 for its autoencoder and matches state-of-the-art interpolation and transition performance while also improving motion estimation and denoising after fine-tuning (Mo et al., 27 Jul 2025).

6. Limitations, misconceptions, and open directions

Several limitations recur across the literature. Domain shift remains central in LiDAR forecasting: models trained with a fixed maximum range such as $(x_i,t_i,f_i,e_i)$8 on KITTI underperform when test data extend to $(x_i,t_i,f_i,e_i)$9 on Apollo-SouthBay until fine-tuned, and different sensor mounts change invalid-mask structure (Mersch et al., 2021). Cross-session monitoring depends on stable geometry and registration assumptions; the greenhouse study explicitly notes that its scans share a greenhouse coordinate frame, and that general TPCs may require rigid alignment such as ICP or KISS-ICP before temporal association is reliable (Fusaro et al., 2024). NeuroGauss4D-PCI identifies runtime scene optimization as its main efficiency bottleneck, with about $F$00 of time spent in optimization and about $F$01 in inference (Jiang et al., 2024).

The literature also documents important modeling caveats. TemporalPointConv is sensitive to the choice of $F$02, $F$03, neighborhood radii, and Max Neighbors, and its joint space-time distance variant underperformed the sequential space-then-time design (Merrill et al., 2021). FESTA is limited to two-frame pairs and does not explicitly model longer-term temporal consistency (Wang et al., 2021). ATPPNet’s own ablations show that increasing the window from $F$04 to $F$05 degrades performance, indicating that simply lengthening temporal context is not guaranteed to help (Pal et al., 2024). PUMPS fixes the point count per frame at $F$06 and depends on PC-MRL-style TPC-to-skeleton reconstruction with at least one known keyframe for temporal offset correction (Mo et al., 27 Jul 2025).

Some apparent “best practices” are not universally valid. Translation invariance is often assumed desirable in geometric learning, yet zebrafish alignment found that centering point clouds at the origin before alignment degrades performance because centroid displacement carries developmental information (Chen et al., 10 Feb 2025). Likewise, unordered-set losses such as Chamfer Distance can be inappropriate when the task requires persistent temporal point identities: PUMPS replaces Chamfer with linear assignment-based pairing precisely to avoid point collapse toward cluster centroids (Mo et al., 27 Jul 2025). These cases show that TPC design choices are strongly task-dependent.

The open directions proposed across papers are correspondingly diverse. They include adaptive normalization or domain adaptation for variable LiDAR ranges and sensor mounts, uncertainty-aware multi-step forecasting, semantic awareness in geometry-only predictors, temporal graph neural networks and spatio-temporal transformers for cross-session monitoring, cycle consistency and isotonic-style sequence constraints for biological alignment, learned inter-session registration, 4D Hilbert curves for longer-horizon occupancy prediction, and explicit temporal scene flow or semantic heads for CSI-derived point clouds (Mersch et al., 2021, Fusaro et al., 2024, Chen et al., 10 Feb 2025, Pandhare, 2023, Määttä et al., 2024). Taken together, these directions indicate that TPC research is converging less toward a single canonical model than toward a common question: how to preserve geometric fidelity, temporal coherence, and task-specific structure when the underlying data are sparse, irregular, and time-evolving.