TrackerSplat: Dynamic 3D Reconstruction

• TrackerSplat is a dynamic 3D reconstruction method that integrates point tracking with 3D Gaussian Splatting to pre-align Gaussians for fast-motion scenes.
• It employs dense optical tracking and per-Gaussian affine motion fitting to reposition and update Gaussians, enabling fully parallel frame processing across GPUs.
• Empirical results demonstrate improved PSNR/SSIM metrics with reduced fading and drift compared to baseline methods on both short and long video sequences.

TrackerSplat is the name of a method for dynamic scene reconstruction with 3D Gaussian Splatting (3DGS) that integrates point tracking to enhance robustness to large inter-frame displacements by relocating, rotating, and scaling Gaussians before gradient-based refinement (Yin et al., 2 Apr 2026). In the supplied arXiv literature, the same name is also attached to an object-centric RGB-D tracking-and-reconstruction system, a particle multi-target tracker for superpositional measurements, and a spectral-moment-based 3DGS tracking recipe (Ikeda et al., 17 May 2025, Papi et al., 2014, Rimon et al., 25 Mar 2026). The most detailed and explicit use associates TrackerSplat with synchronized multi-view dynamic reconstruction from short video clips, where off-the-shelf point tracking, per-view affine motion fitting, multi-view triangulation, and standard 3DGS refinement are organized into a pipeline that can be fully parallelized over frames and GPUs (Yin et al., 2 Apr 2026).

1. Problem setting and representation

TrackerSplat, in the dynamic 3DGS formulation, takes as input a short clip of synchronized multi-view video frames $\{I_v^t\}$ for views $v=1\ldots V$ and timesteps $t=1\ldots T$. The pipeline is organized into four stages: initialization, point tracking, motion compensation, and refinement. Initialization reconstructs a static 3D Gaussian splatting model on frame $t=1$ via any static 3DGS method, with InstantSplat given as an example. Each Gaussian $i$ is parameterized by weight $w_i$, 3D mean $\mu_i \in \mathbb{R}^3$, and covariance

$\Sigma_i = R_i S_i^2 R_i^\top,$

where $R_i \in SO(3)$ and $S_i$ is diagonal (Yin et al., 2 Apr 2026).

This representation is central to TrackerSplat’s operating logic. The static model built at the first frame serves as the reference state from which subsequent framewise Gaussian states are inferred. The method does not propagate refinements recursively from frame to frame; instead, each target frame is reconstructed from the frame-$v=1\ldots V$0 initialization plus tracked trajectories. This design is what permits full frame-level parallelization across multiple devices.

A common misunderstanding is to view TrackerSplat as a purely photometric optimizer. In the supplied formulation, the essential intervention occurs before photometric training: Gaussians are explicitly repositioned from tracked image trajectories and triangulation, and only then refined by gradient descent. This distinction matters because the reported robustness to fast motion is attributed to accurate pre-positioning of Gaussians prior to standard 3DGS optimization rather than to a modified renderer or loss alone.

2. Point tracking and per-Gaussian 2D motion fitting

For each view $v=1\ldots V$1, TrackerSplat runs Dense Optical Tracking (DOT) to obtain pixel-level trajectories $v=1\ldots V$2 for every pixel $v=1\ldots V$3 in frame $v=1\ldots V$4 across all subsequent frames. For a given Gaussian and view, the method collects the pixels covered by that Gaussian in frame $v=1\ldots V$5 and their tracked positions in frame $v=1\ldots V$6, then fits an affine 2D Gauss motion $v=1\ldots V$7 by weighted least squares. The stated objective is

$v=1\ldots V$8

In matrix form,

$v=1\ldots V$9

with

$t=1\ldots T$0

The implementation is given as Parallel Weighted Incremental Least Squares (PWI-LS), which accumulates

$t=1\ldots T$1

via one pass of the rasterization, and then computes

$t=1\ldots T$2

Gaussians are discarded if $t=1\ldots T$3, total opacity $t=1\ldots T$4, or fewer than $t=1\ldots T$5 covered pixels are available (Yin et al., 2 Apr 2026).

The PWI-LS formulation is not merely a numerical convenience. It is the mechanism by which TrackerSplat turns dense pixel trajectories into per-Gaussian motion hypotheses without introducing a per-Gaussian iterative solver external to the rendering pipeline. The reliance on one rasterization pass and accumulated sufficient statistics is consistent with the throughput claims later reported for multi-GPU settings.

The paper also specifies trajectory-linking thresholds and a static-Gaussian detection rule. Pixels moving less than $t=1\ldots T$6 px are counted as static; if more than $t=1\ldots T$7 of a Gaussian’s hits in at least two views are static, the Gaussian is classified as static and neighbor motion is propagated instead of using its own. This is paired with median filtering among each Gaussian’s $t=1\ldots T$8 nearest neighbors, where the implementation details later fix $t=1\ldots T$9.

3. Multi-view lifting to 3D and framewise refinement

A single 3D Gaussian $t=1$0 is represented as $t=1$1 with

$t=1$2

Projection to view $t=1$3 yields a 2D Gaussian with

$t=1$4

After applying the fitted affine transform,

$t=1$5

TrackerSplat then triangulates the updated 2D means and covariances from multiple views into a new 3D mean $t=1$6 and covariance $t=1$7. The multi-view 3D mean is obtained by standard triangulation by SVD on the stacked depth-weighted rays from $t=1$8 in at least two views. The multi-view 3D covariance is recovered by solving the linear system for $t=1$9 from at least two equations of the form

$i$0

The reconstructed covariance is decomposed by eigen-decomposition,

$i$1

with negative eigenvalues discarded and eigenpairs re-ordered to match frame $i$2 (Yin et al., 2 Apr 2026).

Once motion compensation has produced $i$3, all Gaussians are optimized over all views at frame $i$4 with the standard 3DGS photometric reconstruction loss and alpha-blending renderer for $i$5 iterations:

$i$6

plus any 3DGS regularizers, with opacity sparsity given as an example. Gradients with respect to $i$7, $i$8, and $i$9 are computed by autodiff through rasterization and blending.

The treatment of large displacements is explicit. Prior Gaussian-based dynamic methods are described as suffering from “fading” or color drift when object regions move outside the local gradient neighborhood. TrackerSplat addresses this by pre-relocating Gaussians through multi-view tracking before any gradient update so that Gaussians remain within the correct image region. This suggests that the method’s robustness derives less from changing the local optimization landscape than from ensuring that the initial state for each frame lies inside a usable basin for standard photometric refinement.

4. Parallel execution model and empirical behavior

Because each frame’s motion compensation and refinement depend only on frame-$w_i$0 initialization and tracked trajectories, rather than on previous refinements, stages 2–4 can be fully parallelized over frames and GPUs. The implementation states that each GPU handles one or more frames completely independently once tracking data are available (Yin et al., 2 Apr 2026).

The reported quantitative results cover short clips, long-video sequences, and per-frame throughput. On short $w_i$1–$w_i$2 frame clips at $w_i$3 GPUs, TrackerSplat reports average PSNR/SSIM/LPIPS of approximately $w_i$4 dB / $w_i$5 / $w_i$6, whereas the best baseline, HiCoM, is reported at approximately $w_i$7 dB / $w_i$8 / $w_i$9. On long-video sequences of more than $\mu_i \in \mathbb{R}^3$0 frames with $\mu_i \in \mathbb{R}^3$1 GPUs, TrackerSplat is reported to maintain stable PSNR of approximately $\mu_i \in \mathbb{R}^3$2 dB over time, while baselines drift downward or fail with NaNs. For the Meeting Room dataset with $\mu_i \in \mathbb{R}^3$3 views at $\mu_i \in \mathbb{R}^3$4, the stated throughput at $\mu_i \in \mathbb{R}^3$5 GPUs is $\mu_i \in \mathbb{R}^3$6 s/frame total, decomposed into $\mu_i \in \mathbb{R}^3$7 s/frame for tracking and $\mu_i \in \mathbb{R}^3$8 s/frame for refinement; the baselines are HiCoM at $\mu_i \in \mathbb{R}^3$9 s/frame, Dyn3DGS at $\Sigma_i = R_i S_i^2 R_i^\top,$0 s/frame, and ST-4DGS at $\Sigma_i = R_i S_i^2 R_i^\top,$1 s/frame (Yin et al., 2 Apr 2026).

Setting	TrackerSplat	Baselines
Short clips, 8 GPUs, PSNR/SSIM/LPIPS	$\Sigma_i = R_i S_i^2 R_i^\top,$2 dB / $\Sigma_i = R_i S_i^2 R_i^\top,$3 / $\Sigma_i = R_i S_i^2 R_i^\top,$4	HiCoM: $\Sigma_i = R_i S_i^2 R_i^\top,$5 dB / $\Sigma_i = R_i S_i^2 R_i^\top,$6 / $\Sigma_i = R_i S_i^2 R_i^\top,$7
Long videos, 100+ frames, 8 GPUs	stable PSNR $\Sigma_i = R_i S_i^2 R_i^\top,$8 dB	drift downward or NaNs
Meeting Room, 13 views@1280×720, 8 GPUs	$\Sigma_i = R_i S_i^2 R_i^\top,$9 s/frame = $R_i \in SO(3)$0	HiCoM $R_i \in SO(3)$1, Dyn3DGS $R_i \in SO(3)$2, ST-4DGS $R_i \in SO(3)$3

The qualitative results are summarized as reduced fading and drift artifacts and sharper renderings in fast-motion regions. Read together with the parallel execution model, these observations define TrackerSplat’s intended operating niche: dynamic multi-view reconstruction where motion is too large for purely local gradient-based updates to remain stable, but where framewise independence can be exploited for throughput.

5. Implementation profile and operational constraints

The implementation details specify hardware, software, tracking resolutions, optimization schedules, and kernel-level choices. The reported hardware is $R_i \in SO(3)$4 NVIDIA A100 SXM4 40 GB. The software stack is PyTorch, Taichi, and custom CUDA kernels for PWI-LS. Point tracking uses DOT on resized images, with the following mappings: Meeting Room $R_i \in SO(3)$5, N3DV $R_i \in SO(3)$6, and st-nerf $R_i \in SO(3)$7. Initialization uses $R_i \in SO(3)$8 K iterations on frame $R_i \in SO(3)$9; refinement uses $S_i$0 K iterations per frame with no density growth. PWI-LS discards Gaussians if $S_i$1, $S_i$2, or fewer than $S_i$3 pixels are covered. Multi-view reconstruction discards Gaussians if visibility is fewer than $S_i$4 views or $S_i$5. Median filtering uses $S_i$6 nearest neighbors. The PWI-LS kernel accumulates per-pixel contributions into $S_i$7 and $S_i$8 via CUDA atomics; triangulation uses batched SVD from cuSOLVER; eigen-decomposition also uses cuSOLVER. The code is available at https://github.com/yindaheng98/TrackerSplat (Yin et al., 2 Apr 2026).

These details delimit the method’s practical assumptions. TrackerSplat presumes synchronized multi-view video, an initial static 3DGS on frame $S_i$9, and access to point trajectories generated by an off-the-shelf tracker. It also assumes that sufficient per-Gaussian support survives thresholding so that affine motion fitting and multi-view lifting are solvable. The paper’s own discard rules, static-Gaussian classification, and neighbor-motion propagation indicate that a nontrivial fraction of Gaussians may require regularization or fallback handling in difficult regions.

A further misconception is that parallelization alone explains the reported quality. The supplied workflow indicates otherwise: the argument is that parallelization becomes viable because motion compensation prevents the quality degradation associated with large frame gaps when processing multiple adjacent frames in parallel across multiple devices. In other words, the throughput claim is inseparable from the pre-alignment mechanism.

6. Other arXiv usages of the name

The supplied literature uses the name TrackerSplat for multiple, technically distinct systems.

In the paper titled "GTR: Gaussian Splatting Tracking and Reconstruction of Unknown Objects Based on Appearance and Geometric Complexity" (Ikeda et al., 17 May 2025), TrackerSplat is described as an object-centric tracking-and-reconstruction system for an unknown rigid object observed by a single, static RGB-D camera. The object is represented by $v=1\ldots V$00 three-dimensional Gaussians, each with mean $v=1\ldots V$01, covariance $v=1\ldots V$02, and RGB color $v=1\ldots V$03. The system alternates between coarse relative pose estimation via tracked 2D–2D keypoint correspondences and keyframe-triggered joint pose-and-shape refinement under a hybrid energy

$v=1\ldots V$04

Its defining features are hybrid geometry/appearance tracking, adaptive keyframe selection based on rotational geodesic distance and keypoint-tracking visibility rate, and final TSDF fusion. This TrackerSplat addresses 6-DoF rigid-object tracking and reconstruction rather than dynamic multi-view scene reconstruction.

In "A Particle Multi-Target Tracker for Superpositional Measurements using Labeled Random Finite Sets" (Papi et al., 2014), the supplied details use TrackerSplat to denote a particle-based multi-target tracker for superpositional measurements grounded in labeled random finite sets. The multi-target state is $v=1\ldots V$05, and the measurement model is

$v=1\ldots V$06

with white Gaussian noise. The method combines a labeled-RFS Bayes filter, an SA-CPHD-driven proposal, and Sequential Monte Carlo sampling, with LMB or Vo-Vo proposal distributions and an importance weight

$v=1\ldots V$07

This usage belongs to multi-target tracking under radar-like superpositional sensing and is unrelated to Gaussian splatting in computer vision.

In "SpectralSplats: Robust Differentiable Tracking via Spectral Moment Supervision" (Rimon et al., 25 Mar 2026), the supplied details describe how to turn SpectralSplats into a full TrackerSplat system by replacing fragile spatial photometric objectives with a global spectral-moment loss. The core construction defines spectral moments

$v=1\ldots V$08

and optimizes a spectral image loss over an annealed band of frequencies before switching to a conventional pixel-loss phase. The motivation is the vanishing-gradient bottleneck of spatial losses in 3DGS tracking when rendered Gaussians have no overlap with their target image support. This TrackerSplat is therefore a robust differentiable tracking recipe built on frequency-domain supervision rather than on point tracking and multi-view triangulation.

This suggests that TrackerSplat is better treated as a polysemous label than as a single established term. In current arXiv usage, its meaning depends on the surrounding paper: dynamic 3DGS reconstruction from synchronized multi-view video (Yin et al., 2 Apr 2026), rigid-object RGB-D tracking and reconstruction (Ikeda et al., 17 May 2025), labeled-RFS particle tracking for superpositional measurements (Papi et al., 2014), or a spectral-moment tracking framework layered on top of 3DGS (Rimon et al., 25 Mar 2026).