ATGS: Anchored Temporal Gaussian Splatting for Long Volumetric Video Representation

Abstract: Volumetric video enables immersive free viewpoint rendering of dynamic real world scenes, yet existing methods struggle with long sequences and complex motions, often leading to temporal instability and visual artifacts. To address these challenges, we propose \ourname, a Gaussian splatting based framework for volumetric video reconstruction. Our key insight is that explicitly tracking long term complex motion with individual Gaussian primitives is inherently unstable. Instead, we organize Gaussians around time conditioned anchors that localize their spatial and temporal support, thereby reducing long range motion complexity. We further introduce a temporal windowing strategy to activate only anchors relevant to the queried time, which improves scalability and temporal coherence. In addition, to ensure spatial and temporal stability, we design a compact set of multi level anchor features that encode global features, local spatial features, and local temporal features, jointly constraining Gaussian generation. Extensive experiments demonstrate that \ourname \ consistently outperforms prior methods on long sequence volumetric videos with complex motions. Project page: https://github.com/WuJH2001/ATGS.

• The paper presents Anchored Temporal Gaussian Splatting (ATGS), a method that stabilizes long volumetric video reconstruction
• ATGS uses a combination of time-conditioned anchors, temporal windowing, and hierarchical features
• ATGS achieves consistent quality improvements over dynamic Gaussian and neural-field baselines.

Problem setting and contribution

“ATGS: Anchored Temporal Gaussian Splatting for Long Volumetric Video Representation” (2608.30184) addresses a specific scalability failure in dynamic neural rendering: methods that perform well on short clips often become unstable when required to represent minute-level multi-view video containing large inter-frame motion. The difficulty is not merely the temporal dimensionality. Long sequences also increase the optimization burden associated with tracking individual Gaussian primitives, producing motion drift, temporal jitter, blur, missing geometry, and flickering when primitives are repeatedly deformed or updated over extended intervals.

The paper’s central claim is that long-range motion should not be represented by explicitly tracking every Gaussian primitive through the entire sequence. ATGS instead organizes Gaussian primitives around time-conditioned anchors. Each anchor is localized spatially and associated with a keyframe timestamp; its features are decoded into Gaussians only within a local temporal neighborhood. This converts a difficult global trajectory-estimation problem into a collection of temporally localized generation problems.

The method combines three design elements:

1. Time-conditioned anchors initialized from periodically sampled keyframes.
2. Temporal windowing that activates only anchors near the queried timestamp.
3. Hierarchical feature decomposition into independent anchor, static spatial, and local temporal features.

The resulting representation is intended to support both long temporal coverage and complex motion while retaining the rasterization efficiency of 3D Gaussian Splatting [kerbl20233d]. The paper evaluates ATGS on N3DV, VRU Basketball, MeetRoom, SelfCap, and broader human-motion datasets, reporting consistent gains in reconstruction quality over representative dynamic Gaussian and neural-field baselines.

Representation and temporal localization

ATGS initializes a sparse set of anchors from periodically sampled keyframes. An anchor contains a 3D position $\mu$, a spatial scale $v$, a discrete temporal index $k$, and a learnable 64-dimensional feature $f_a$. The temporal index does not represent a persistent trajectory. Rather, it identifies the temporal context in which the anchor was initialized and determines when the anchor participates in Gaussian generation.

This distinction is important. Deformation-based dynamic representations generally attempt to map canonical primitives to their time-dependent locations. Such mappings become increasingly difficult to optimize when topology, occlusion, visibility, and motion vary over long intervals. ATGS avoids requiring a Gaussian to remain identifiable throughout the entire sequence. Anchors instead provide local support for Gaussian generation, and motion can be represented through the time-dependent appearance, disappearance, and configuration of generated primitives.

Given a queried time $t$, ATGS identifies the nearest keyframe and selects anchors whose temporal indices lie within a window of width $W$. The default is $W=7$, corresponding to a half-width of three keyframes except near sequence boundaries. Only the selected anchors are decoded and optimized for that timestamp. The strategy reduces interference from distant temporal regions and makes the active representation evolve gradually between adjacent frames.

The system architecture is summarized below.

Figure 1: ATGS extracts keyframes, initializes time-conditioned anchors, queries spatial and temporal feature structures, and decodes the resulting features into temporally varying Gaussian primitives.

Temporal windowing provides two forms of sparsity. It limits the number of active anchors for each query and restricts optimization to the temporal structures relevant to the current training sample. This is particularly consequential for long sequences: the model does not need to backpropagate through a globally active set of primitives whose behavior is only weakly related to the current frame. The ablation results support the role of this mechanism. On VRU Long, removing the temporal window reduces PSNR from 24.42 to 24.02 and worsens LPIPS from 0.147 to 0.153, whereas window sizes from 1 to 7 produce relatively similar results. Thus, the principal benefit comes from localization itself rather than from a narrowly tuned window width.

Hierarchical spatio-temporal features

The anchor representation is augmented with two shared feature fields. A spatial grid $F_S$ provides a time-invariant local feature $f_s$ through trilinear interpolation at the anchor position. A collection of temporal hash grids $F_{T,m}$ provides a local temporal feature $v$0 through quadrilinear interpolation in position and normalized time. The temporal domain is partitioned into $v$1 segments, with $v$2 for N3DV and MeetRoom and $v$3 for VRU.

The three components have deliberately different roles:

• $v$4 stores high-capacity anchor-specific scene information and supports fine-detail control.
• $v$5 imposes spatial locality and consistency across independently optimized anchors.
• $v$6 models temporal variation within a short temporal segment without requiring a single representation to encode the full sequence duration.

The paper’s feature ablation yields an informative, somewhat asymmetric result: the anchor features encode most of the scene content, while the temporal features primarily control dynamic changes rather than reconstructing the scene itself. The visualization indicates that $v$7 alone captures substantially more information than $v$8 or $v$9, while $k$0 accounts for nearly all static scene content. The temporal component therefore functions less as a complete dynamic scene representation than as a control signal for time-dependent Gaussian attributes and visibility.

Figure 2: Feature decomposition shows that anchor features carry most scene information, static features stabilize spatial content, and temporal features primarily modulate dynamic appearance and disappearance.

The decoder concatenates the static combination $k$1 with the temporal feature $k$2, rather than summing all three components. This design avoids forcing the decoder to implicitly disentangle static and dynamic information. The paper reports that the alternative summation formulation incurs an approximately 10% training slowdown. The adopted formulation is consequently both an architectural decomposition and an optimization choice.

For each active anchor, a lightweight two-layer MLP generates a fixed number of temporal Gaussians. Their positions are offsets from the anchor center, scaled by the anchor’s spatial extent:

$k$3

where $k$4 denotes the fused feature representation. Orientation, scale, opacity, and color are decoded analogously. The Gaussian parameters are therefore functions of the queried time, but their spatial support remains constrained by the anchor. A volume regularizer penalizes the product of Gaussian scales, encouraging compact primitives and discouraging uncontrolled expansion away from their anchors.

The ablation of feature components confirms that both anchor and static features are necessary. On VRU Long, using $k$5 gives PSNR 24.30 and LPIPS 0.151, compared with PSNR 24.42 and LPIPS 0.147 for the full model. Replacing the anchor feature with $k$6 is substantially worse, yielding PSNR 22.58 and LPIPS 0.240. This result establishes that the shared spatial grid is not an adequate substitute for anchor-specific capacity; its role is complementary, providing local consistency rather than complete scene encoding.

Experimental design and efficiency

The experiments cover different combinations of temporal duration, motion complexity, and camera coverage. N3DV provides a conventional multi-view benchmark with one 1,200-frame sequence. VRU is the most demanding setting, containing fast basketball motion and a 1,400-frame, approximately one-minute sequence. MeetRoom tests sparse-view reconstruction using only 11–12 cameras. SelfCap evaluates longer sequences, including a 2,000-frame test segment, while PKU-DyMVHumans provides broader human-motion variation.

The implementation uses 64-dimensional features for all three feature types, a $k$7 spatial grid with 64-dimensional entries, and temporal hash grids with table size $k$8. All experiments use an NVIDIA A100 with 80 GB of memory. The number of keyframes and temporal grids is adapted to motion magnitude: $k$9, $f_a$0 for N3DV and MeetRoom, and $f_a$1, $f_a$2 for VRU. This adaptation is effective but introduces scene-dependent hyperparameters; the representation is not entirely parameter-free with respect to temporal complexity.

Quantitative reconstruction results

On N3DV, ATGS obtains PSNR 32.56, DSSIM$f_a$3 0.027, DSSIM$f_a$4 0.013, and LPIPS 0.043. It improves over LocalDyGS, whose PSNR is 32.28 and LPIPS is 0.044, and over SpaceTimeGS, whose PSNR is 32.05 and LPIPS is 0.044. ATGS renders at 70 FPS, below the 105 FPS reported for LocalDyGS but still within the paper’s real-time rendering criterion. Its training time is 0.9 hours, compared with 0.58 hours for LocalDyGS. The result therefore reflects a quality–throughput trade-off rather than uniformly improved efficiency.

MeetRoom provides a stronger quality and compactness comparison. ATGS reaches PSNR 32.79, compared with 30.79 for 3DGStream and 30.27 for 4DGaussian. It requires 0.45 hours and 110 MB, while 3DGStream requires 1,230 MB and StreamRF requires 2,700 MB. ATGS consequently achieves the best reported quality with a substantially smaller model than the streaming baseline and a modest storage requirement relative to the competing Gaussian approaches.

Dataset or setting	Method	PSNR	Additional result
N3DV	ATGS	32.56	70 FPS; LPIPS 0.043
N3DV	LocalDyGS	32.28	105 FPS; LPIPS 0.044
MeetRoom	ATGS	32.79	0.45 h; 110 MB
MeetRoom	3DGStream	30.79	0.60 h; 1,230 MB
SelfCap, 2,000 frames	ATGS	29.13	LPIPS 0.110
SelfCap, 2,000 frames	4DGaussian	27.86	LPIPS 0.145

On SelfCap, ATGS improves PSNR from 27.86 to 29.13 over 4DGaussian and reduces LPIPS from 0.145 to 0.110. This is a meaningful improvement on a 2,000-frame sequence, although the comparison includes only one baseline in the reported quantitative result and therefore provides weaker evidence than the broader N3DV, VRU, and MeetRoom comparisons.

The VRU results most directly test the paper’s principal claim. On the 250-frame GZ sequence, ATGS achieves PSNR 30.61 and SSIM 0.948, compared with 29.23 and 0.939 for LocalDyGS. On DG, it reaches 30.37 PSNR and 0.939 SSIM, compared with 29.01 and 0.931. On VRU Long, ATGS obtains 24.78 PSNR and 0.881 SSIM, exceeding LocalDyGS at 23.21 and 0.875.

VRU sequence	LocalDyGS PSNR / SSIM	ATGS PSNR / SSIM
GZ, 250 frames	29.23 / 0.939	30.61 / 0.948
DG, 250 frames	29.01 / 0.931	30.37 / 0.939
Long, 1,400 frames	23.21 / 0.875	24.78 / 0.881

The strongest systems-level claim is that ATGS reconstructs the entire 1,400-frame VRU sequence in one run, whereas the cited prior local methods generally operate on approximately 20-frame segments. This corresponds to roughly a 70-fold increase in temporal coverage. The comparison is important, but it must be interpreted with care: the baselines are not always evaluated under identical temporal-training protocols, and the paper does not report a complete memory, training-time, and parameter-count comparison for every method at the full 1,400-frame scale.

Qualitative results are consistent with the numerical measurements. On N3DV, ATGS preserves fine details over 1,200 frames, whereas methods trained on shorter segments show degradation when temporal coverage is extended.

Figure 3: On N3DV, ATGS reconstructs a 1,200-frame sequence in one run while retaining sharper details than methods operating on shorter temporal segments.

On VRU, the advantage is most visible around rapidly moving athletes. Competing methods exhibit missing body parts, blur, and accumulated geometry errors, while ATGS maintains more coherent subject structure across the long sequence.

Figure 4: On the 1,400-frame VRU sequence, ATGS preserves dynamic subject geometry more effectively than competing methods.

MeetRoom tests a different failure mode: sparse camera coverage. ATGS retains fine details and clarity in regions undergoing large motion, suggesting that its anchor-localized representation is useful not only for long sequences but also when view-dependent supervision is limited.

Figure 5: Under sparse-view MeetRoom capture, ATGS preserves detail and reduces blur in high-motion regions.

Ablation of temporal capacity and coverage

The temporal-grid ablation provides direct evidence for the proposed segmented temporal representation. On VRU GZ, increasing $f_a$5 from 1 to 12 improves PSNR from 29.10 to 30.61, SSIM from 0.933 to 0.948, and reduces LPIPS from 0.086 to 0.053.

Number of temporal grids $f_a$6	PSNR	SSIM	LPIPS
1	29.10	0.933	0.086
4	29.31	0.939	0.077
8	30.11	0.942	0.062
12	30.61	0.948	0.053

The monotonic trend supports the claim that local temporal structures provide useful capacity for complex motion. It also exposes a practical limitation: performance depends on allocating sufficient temporal capacity, and the paper selects $f_a$7 according to the dataset. The model’s scalability therefore arises from localized temporal modeling, but not from a fixed representation whose capacity is independent of motion complexity.

The keyframe ablation shows a similar dependence on temporal sampling. On VRU Long with a 250-frame evaluation setting, increasing $f_a$8 from 12 to 250 improves PSNR from 23.76 to 24.42 and reduces LPIPS from 0.165 to 0.147. This indicates that anchor coverage is not merely an initialization detail. It materially determines the temporal support available to the decoder.

The qualitative ablation further illustrates the effect of removing these components.

Figure 6: Ablation visualizations on VRU Long show degradation when temporal coverage or feature components are reduced.

Inference profiling on VRU GZ reports 15.6 ms per query frame: 0.7 ms for feature extraction, 5.9 ms for MLP decoding, 4.4 ms for rendering, and 4.6 ms for other operations. The resulting throughput is approximately 64 frames per second under the reported breakdown, consistent with the broader claim of real-time rendering. However, this figure concerns inference after offline reconstruction and should not be conflated with real-time capture or online training.

Figure 7: Additional qualitative results across datasets demonstrate the method’s applicability to varied dynamic scenes and capture configurations.

Limitations and open questions

ATGS remains an offline reconstruction method and does not support real-time processing during capture. Its rendering stage is efficient, but the training procedure and preprocessing pipeline are not. In particular, the method depends on camera poses and sparse point clouds estimated by COLMAP. This introduces a potentially substantial computational bottleneck and makes the reported system dependent on reliable structure-from-motion initialization. Although COLMAP-free Gaussian reconstruction methods exist, the paper does not evaluate whether ATGS retains its performance with estimated rather than COLMAP-derived geometry.

The method also assumes sufficient multi-view coverage. The authors report mild temporal jitter in distant audience regions of VRU, where observations are sparse and subjects occupy few pixels. This limitation is structurally consistent with the representation: temporal windowing can restrict optimization, but it cannot recover geometric constraints absent from the input views. The reported gains therefore apply most strongly to well-observed foreground content and should not be generalized to severely under-constrained regions.

A second open issue concerns hyperparameter scaling. The experiments use different values of $f_a$9 and $t$0 for low- and high-motion datasets. The ablations show that both keyframe density and temporal-grid capacity influence quality, but the paper does not provide an automatic rule for selecting them from motion statistics, scene extent, or camera configuration. It also does not establish how memory and training time scale with sequence duration beyond the evaluated settings.

Finally, the feature analysis raises a representational question. Since $t$1 and $t$2 encode nearly all scene content and $t$3 mainly modulates visibility and temporal variation, the temporal feature may be functioning primarily as a dynamic gating mechanism rather than as an explicit motion representation. This is sufficient for perceptual reconstruction in the presented benchmarks, but it leaves open whether ATGS can preserve physically meaningful correspondences, support temporally consistent geometry editing, or maintain identity-level primitive tracking under topology changes.

Conclusion

ATGS presents a coherent strategy for long-sequence volumetric video reconstruction: replace globally tracked Gaussian trajectories with temporally localized, time-conditioned anchors; restrict computation through temporal windows; and stabilize Gaussian generation with separate anchor, static spatial, and local temporal features. The empirical results support the design, particularly on VRU, where ATGS reconstructs 1,400 frames in a single run and improves PSNR over LocalDyGS from 23.21 to 24.78 on the long sequence. It also achieves strong quality–storage trade-offs on MeetRoom and improves perceptual metrics on SelfCap.

The paper’s main contribution is therefore not a new Gaussian primitive but a different allocation of temporal responsibility. Anchors provide localized support, shared spatial features impose consistency, and temporal grids encode short-range variation. The resulting system substantially extends the temporal coverage of dynamic Gaussian reconstruction, while remaining dependent on offline preprocessing, adequate multi-view supervision, and motion-dependent capacity selection.