Velocity-guided PointTrack (VGPT)

• Velocity-guided PointTrack (VGPT) is a deformation-field tracker that integrates 4D radar-derived radial velocity with optical flow to enable precise dynamic scene reconstruction.
• The method learns an invertible spatio-temporal warp that maps dynamic Gaussians into a canonical space for artifact-free reconstructions and novel view synthesis.
• Dual supervision from lifted scene flow and radar constraints enhances sharpness and temporal consistency, reducing motion blur and ghosting in dynamic driving scenes.

Searching arXiv for the specified paper to ground the article and citation. {"query":"arXiv (Tang et al., 16 Sep 2025) 4DRadar-GS Self-Supervised Dynamic Driving Scene Reconstruction with 4D Radar", "max_results": 5} Velocity-guided PointTrack (VGPT) is a velocity-guided deformation-field tracker introduced within the 4DRadar-GS framework for self-supervised dynamic driving scene reconstruction. It is designed to address two principal difficulties in dynamic scenes: inaccurate motion estimation and weak temporal consistency. In the formulation reported for 4DRadar-GS, VGPT injects a direct physical prior—4D radar radial velocity—into dynamic tracking and deformation learning, and is jointly supervised by pseudo 3D scene flow lifted from 2D optical flow and measured radial relative velocity from 4D radar. The model learns a deformation field that maps dynamic points or Gaussians into a canonical space and back, enabling temporally coherent dynamic representations for reconstruction and novel view synthesis without manual 3D bounding box supervision (Tang et al., 16 Sep 2025).

1. Functional role in dynamic scene reconstruction

VGPT is motivated by the observation that prior self-supervised approaches for dynamic driving scenes, particularly those based on frequency-domain decoupling or optical flow-only supervision, often fail to establish reliable inter-frame correspondence for fast-moving actors and under rapid ego-motion. The reported consequences are motion blur, ghosting, and incomplete dynamic reconstructions. VGPT is introduced specifically to remedy these deficits by coupling geometric correspondence supervision with radar-derived motion constraints.

Within the overall 4DRadar-GS pipeline, initialization uses 4D radar to aid dynamic segmentation and monocular depth scale recovery, after which dynamic Gaussians are created from scaled monocular depth in dynamic regions. VGPT then maps these dynamic Gaussians into a canonical space and back to time $t$ through a learned deformation field $\mathcal{D}_t$, thereby constructing temporally consistent dynamic reconstructions. The reported effect is sharper, artifact-free reconstructions and novel views of moving vehicles and pedestrians.

A central conceptual point is that VGPT is not presented as a standalone multi-object tracker in the conventional sense. Tracking is achieved through learned spatio-temporal warping of dynamic Gaussian primitives rather than through explicit track state management, Kalman filtering, or combinatorial assignment.

2. Inputs, outputs, and dynamic state representation

The inputs to VGPT are dynamic point or Gaussian primitives obtained through 4D radar-assisted initialization, 4D radar dynamic measurements per frame, and image-derived optical flow. The initialization stage provides monocular depth whose scale is recovered using 4D radar static points, together with a dynamic segmentation mask guided by 4D radar velocity. The radar stream contributes per-radar-point spatial coordinates in the camera or radar frame and per-radar-point radial velocity. The image stream contributes optical flow between pairs of frames via RAFT, which is then lifted into sparse 3D scene flow using the scale-recovered depth.

The outputs are tracked temporal correspondences for dynamic points via an invertible spatio-temporal deformation field to and from canonical space, scene flow fields implicitly induced by forward and backward warps, and associations between dynamic Gaussians and 4D radar points via KNN. Dynamic actors are represented as sets of 3D Gaussians, and VGPT learns a time-dependent deformation field that aligns per-frame dynamic Gaussians to canonical space and warps them to any target time for rendering.

For exposition, the per-point state at time $t$ is denoted as

$$\mathbf{s}_i^t = [\mathbf{x}_i^t, v_i^{r,t}, \mathbf{f}_i^t, \Sigma_i^t],$$

where $\mathbf{x}_i^t \in \mathbb{R}^3$ is the 3D center of the $i$-th dynamic Gaussian, $v_i^{r,t} \in \mathbb{R}$ is the inherited radar radial velocity from the nearest radar point, $\mathbf{f}_i^t$ denotes appearance or radiance features implicitly optimized by 3DGS, and $\Sigma_i^t$ denotes Gaussian spatial covariance or scale parameters optimized by the 3DGS renderer. The paper explicitly notes that this state is a notational convenience: VGPT optimizes these quantities implicitly through joint rendering and flow losses rather than maintaining them as an explicit tracking state (Tang et al., 16 Sep 2025).

3. Architecture and deformation-field formulation

VGPT is realized as a time-conditioned, invertible deformation field implemented by a single MLP inspired by RealNVP-style invertible networks. Its inputs are a 3D coordinate $\mathbf{x} \in \mathbb{R}^3$ and a normalized timestamp $\mathcal{D}_t$0, and its output is the warped 3D coordinate $\mathcal{D}_t$1. The deformation field is invertible by construction, which allows both forward $\mathcal{D}_t$2 and inverse $\mathcal{D}_t$3 mappings to be derived from a single forward pass, and its weights are shared across time to ensure temporal continuity.

The canonical-space formulation is central to the model’s operation. Given a point observed at time $\mathcal{D}_t$4, its predicted location at time $\mathcal{D}_t$5 is written as

$\mathcal{D}_t$6

For Gaussian means, the canonical-to-time mapping is

$\mathcal{D}_t$7

where $\mathcal{D}_t$8 denotes the canonical center optimized jointly with the renderer. Static Gaussians are optimized separately, while dynamic Gaussians are warped by VGPT for rendering at each time step.

The paper is explicit about what VGPT does not contain. There is no graph-attention, RNN, or Kalman-style module, and there is no Hungarian algorithm or differentiable assignment. Radar association is instead performed by simple KNN with $\mathcal{D}_t$9 at each timestamp, attaching a measured radial velocity $t$0 to each dynamic Gaussian. This design makes the deformation field itself the tracking mechanism.

4. Dual supervision: optical-flow-lifted scene flow and radar radial velocity

VGPT is jointly supervised by pseudo 3D scene flow lifted from 2D optical flow and by radial relative velocity measured by 4D radar. The geometric part of the supervision begins from sparse 3D correspondences $t$1 formed in dynamic regions using optical flow and scale-recovered depth. The deformation field is required to satisfy

$t$2

leading to the geometric consistency loss

$t$3

The velocity-guidance mechanism uses radar radial velocity as a direct physical constraint on the learned scene flow. Let $t$4 be the radar center at time $t$5, let $t$6 denote the measured radial velocity, let $t$7, and let $t$8 denote the coordinate transform from frame $t$9 to $$\mathbf{s}_i^t = [\mathbf{x}_i^t, v_i^{r,t}, \mathbf{f}_i^t, \Sigma_i^t],$$0. The predicted scene flow vector in the coordinates of $$\mathbf{s}_i^t = [\mathbf{x}_i^t, v_i^{r,t}, \mathbf{f}_i^t, \Sigma_i^t],$$1 is defined as

$$\mathbf{s}_i^t = [\mathbf{x}_i^t, v_i^{r,t}, \mathbf{f}_i^t, \Sigma_i^t],$$2

The physical constraint projects the true flow onto the radar line-of-sight:

$$\mathbf{s}_i^t = [\mathbf{x}_i^t, v_i^{r,t}, \mathbf{f}_i^t, \Sigma_i^t],$$3

and the VGPT supervision enforces agreement between the predicted radial component and the measured radial displacement:

$$\mathbf{s}_i^t = [\mathbf{x}_i^t, v_i^{r,t}, \mathbf{f}_i^t, \Sigma_i^t],$$4

The paper characterizes this as a constant-velocity constraint on the radial component across $$\mathbf{s}_i^t = [\mathbf{x}_i^t, v_i^{r,t}, \mathbf{f}_i^t, \Sigma_i^t],$$5. It also states that VGPT does not use an explicit gating mechanism for inter-frame tracking. Instead, it learns the warp that minimizes optical-flow-lifted geometric consistency and radar radial consistency. A conceptual constant-velocity predictor,

$$\mathbf{s}_i^t = [\mathbf{x}_i^t, v_i^{r,t}, \mathbf{f}_i^t, \Sigma_i^t],$$6

is given only as a reference; in practice, VGPT learns $$\mathbf{s}_i^t = [\mathbf{x}_i^t, v_i^{r,t}, \mathbf{f}_i^t, \Sigma_i^t],$$7 directly and does not maintain explicit per-point velocity states.

5. Association strategy, optimization, and training protocol

The association mechanism is deliberately simple. At each timestamp, each dynamic Gaussian is associated to its nearest 4D radar point by KNN with $$\mathbf{s}_i^t = [\mathbf{x}_i^t, v_i^{r,t}, \mathbf{f}_i^t, \Sigma_i^t],$$8, inheriting that point’s radial velocity $$\mathbf{s}_i^t = [\mathbf{x}_i^t, v_i^{r,t}, \mathbf{f}_i^t, \Sigma_i^t],$$9. Inter-frame correspondences are obtained by lifting optical-flow pixel correspondences into 3D using depth, thereby forming the pairs used in $\mathbf{x}_i^t \in \mathbb{R}^3$0. For each training pair $\mathbf{x}_i^t \in \mathbb{R}^3$1), dynamic regions are identified via the 4D radar-guided segmentation mask, optical-flow correspondences are lifted to 3D, the nearest radar point is assigned to each $\mathbf{x}_i^t \in \mathbb{R}^3$2, $\mathbf{x}_i^t \in \mathbb{R}^3$3 is computed through the deformation field, and $\mathbf{x}_i^t \in \mathbb{R}^3$4 together with $\mathbf{x}_i^t \in \mathbb{R}^3$5 is accumulated and backpropagated to the deformation field and Gaussian parameters.

Occlusions and view sparsity are mitigated through Gaussian Dropout Regularization, described as randomly zeroing the opacity of a subset of Gaussians to expose occluded points to supervision. Track birth and death are handled implicitly: dynamic Gaussians exist where the dynamic mask detects motion, and no explicit kill or merge module is used. Robustness to temporary occlusions is attributed to dropout and global rendering supervision.

Training is reported on a single NVIDIA RTX 4090 GPU. The segmentation model is trained for 50 epochs with Adam at learning rate $\mathbf{x}_i^t \in \mathbb{R}^3$6, batch size 6, input resolution $\mathbf{x}_i^t \in \mathbb{R}^3$7, and patch size $\mathbf{x}_i^t \in \mathbb{R}^3$8, requiring approximately 12 hours and using Grounded-SAM prompts. The Gaussian model is trained for 30,000 iterations per sequence, with stage-2 starting at 15k iterations and stage-3 at 20k, requiring approximately 1 hour per sequence. The three-stage schedule is: Stage-1, train static Gaussians and dynamic deformation field; Stage-2, freeze static Gaussians and jointly train dynamic Gaussians and the deformation field; Stage-3, unfreeze all components for end-to-end fine-tuning. The rendering stack uses loss weights $\mathbf{x}_i^t \in \mathbb{R}^3$9, $i$0, $i$1, and $i$2, with the mask loss enabled in stage-3. Optical flow is obtained via RAFT, and the implementation is in PyTorch (Tang et al., 16 Sep 2025).

6. Empirical performance, comparisons, and limitations

On the OmniHD-Scenes dataset, the full 4DRadar-GS method with VGPT and dual supervision is reported to achieve state-of-the-art performance among self-supervised dynamic reconstruction methods. For reconstruction over all regions, the reported metrics are PSNR 34.96, SSIM 0.958, LPIPS 0.119, and dynamic PSNR* 29.81. For novel view synthesis, the reported metrics are PSNR 26.68, SSIM 0.790, LPIPS 0.265, and dynamic PSNR* 23.33.

The VGPT ablation for novel view synthesis isolates the contribution of the supervision signals. Without VGPT, the baseline reports PSNR 25.02, SSIM 0.762, LPIPS 0.305, and PSNR* 18.38. With VGPT and optical flow only, the metrics are PSNR 25.97, SSIM 0.775, LPIPS 0.273, and PSNR* 22.29. With VGPT and radar only, the metrics are PSNR 25.58, SSIM 0.770, LPIPS 0.285, and PSNR* 20.93. With both signals, the metrics rise to PSNR 26.68, SSIM 0.790, LPIPS 0.265, and PSNR* 23.33. The paper interprets these results as showing that optical flow supervision substantially reduces blur but leaves ghosting, while radar-only supervision constrains radial motion but is insufficient for full 3D dynamics; the combination yields the sharpest and most coherent dynamic reconstructions.

The comparative discussion positions VGPT against two broad alternatives. Relative to optical-flow-only tracking, radar guidance adds a physical measurement of motion through radial relative velocity, directly constraining the radial component of scene flow and improving temporal consistency. Relative to frequency-domain decoupling, which can mis-associate or over-smooth dynamics in non-periodic driving motion, VGPT learns point-wise spatio-temporal warps guided by radar velocities and avoids periodicity assumptions. The paper also states that baseline association methods such as PVG and AD-GS either suffer dynamic artifacts due to correspondence errors or lose fine detail due to strong smoothing priors, whereas VGPT achieves higher PSNR and SSIM and lower LPIPS on multiple sequences, with clearer dynamic actors.

The reported limitations define the practical envelope of the method. Radar is sparser and noisier than LiDAR, and radial relative velocity measurements can be noisy. Multipath and ghost returns may misguide the radial constraint when associations are incorrect, although the paper states that KNN association and multi-view image losses mitigate this. Because radial relative velocity constrains only line-of-sight motion, complex lateral motion still depends on optical-flow supervision. Low-speed or near-stationary objects provide weak radial signals, fast accelerations can violate the constant-velocity assumption across $i$3, and extreme occlusions remain difficult despite Gaussian dropout. The paper further notes that radar-camera calibration and sensor placement variations can affect scale recovery and association quality. Taken together, these points indicate that VGPT’s strongest regime is one in which radar radial motion and image-based correspondence are both informative, with each supervision source compensating for the other’s failure modes (Tang et al., 16 Sep 2025).