FreeStreamGS: Online 3D Gaussian Splatting

• FreeStreamGS is an online, feed-forward framework for novel view synthesis that reconstructs a 3D Gaussian scene from unposed streaming image inputs.
• It introduces a Decoupled Intrinsic Recovery Head and Dynamic Point Refinement Offset to counter cumulative intrinsic bias and pose-depth drift, ensuring multi-view consistency.
• The method achieves low latency (250 ms per frame) and competitive benchmark performance by integrating causal feature extraction and recursive Gaussian fusion for stable, real-time scene rendering.

FreeStreamGS is an online, feed-forward framework for novel view synthesis from unposed streaming image inputs. It reconstructs and maintains a 3D Gaussian Splatting scene representation causally, without access to future frames, so that each incoming frame can be processed immediately and incorporated into a global scene while preserving multi-view consistency over long sequences (Chen et al., 2 Jun 2026). In the Gaussian-splatting literature, the name is also adjacent to a broader family of streaming systems for free-viewpoint video and volumetric scene delivery; however, the specific method titled “FreeStreamGS: Online Feed-forward 3D Gaussian Splatting from Unposed Streaming Inputs” denotes the unposed, online novel-view-synthesis framework rather than a generic label for all streaming Gaussian methods (Chen et al., 2 Jun 2026).

1. Problem formulation and motivation

FreeStreamGS addresses online novel view synthesis from a stream of unposed RGB frames, where the system must estimate camera parameters and incrementally build a 3D scene representation that can render novel viewpoints on-the-fly (Chen et al., 2 Jun 2026). The central difficulty is that the online setting deprives the model of future frames and long-range non-causal context, weakening implicit multi-view regularization and making geometry and pose estimation progressively unstable.

The method is motivated by the observation that feed-forward 3D Gaussian Splatting is attractive for low-latency operation, because it amortizes scene reconstruction into a single forward pass per frame and avoids iterative optimization. At the same time, the paper argues that novel view synthesis is substantially more sensitive to geometric inconsistency than pure geometry recovery: even small errors in camera intrinsics and pose-depth coupling accumulate along viewing rays, perturb Gaussian placement and scales, and manifest as visible rendering artifacts when composited across views (Chen et al., 2 Jun 2026).

Two failure modes are emphasized. The first is cumulative intrinsic bias, in which per-frame intrinsic prediction drifts over time, changing projection scale and producing global scene scale jitter. The second is pose-depth drift coupled through rigid unprojection, where small pose and depth errors accumulate under standard per-pixel lifting and distort Gaussian placement. FreeStreamGS is designed around these two failure modes and introduces a Decoupled Intrinsic Recovery Head and a Dynamic Point Refinement Offset strategy to address them directly (Chen et al., 2 Jun 2026).

2. Causal architecture and Gaussian representation

FreeStreamGS processes each frame causally and maintains a global Gaussian scene through the mapping

$$(\mathcal{G}_t,\mathcal{P}_t,\mathcal{H}_t)=f_\theta(I_t,\mathcal{H}_{t-1}),$$

where $\mathcal{G}_t$ is the set of per-pixel Gaussian attributes, $\mathcal{P}_t=\{\hat{K},\hat{R}_t,\hat{t}_t\}$ are camera intrinsics and extrinsics, and $\mathcal{H}_t$ is the updated global state cache (Chen et al., 2 Jun 2026).

Its per-frame data flow comprises four stages. First, causal feature extraction with full-history attention and KV caching produces frame features $F_t$. Second, decoupled camera recovery predicts relative extrinsics with a causal extrinsic head and predicts a single normalized focal ratio with the intrinsic head. Third, a Gaussian decoder maps $F_t$ to per-pixel attributes

$A_t=\{D_t,q_t,s_t,\alpha_t,c_t\}$

and a confidence $w_t$, while Dynamic Point Refinement Offsets predict per-pixel $\Delta\mu_t$ to refine lifted 3D centers. Fourth, Online Recursive Gaussian Fusion integrates the current frame’s Gaussians into the global voxelized state cache via recursive confidence-weighted averaging (Chen et al., 2 Jun 2026).

The underlying 3D Gaussian Splatting representation follows the standard explicit parameterization. For each primitive, FreeStreamGS uses a world-space center $\mu_u\in\mathbb{R}^3$, an orientation $\mathcal{G}_t$0 producing a rotation in $\mathcal{G}_t$1, anisotropic scale $\mathcal{G}_t$2, opacity $\mathcal{G}_t$3, and color $\mathcal{G}_t$4. The covariance is represented through scale and rotation, and rendering uses standard front-to-back alpha compositing after projection of each 3D Gaussian to a 2D elliptical footprint through the projection Jacobian (Chen et al., 2 Jun 2026).

This architecture places FreeStreamGS within the feed-forward branch of online Gaussian methods rather than the optimization-heavy branch. A plausible implication is that its design goal is not merely low latency, but causal maintenance of a globally coherent Gaussian scene under unposed inputs, which distinguishes it from streaming free-viewpoint video systems that assume known multi-view calibration or transmit explicitly reconstructed dynamic content (Chen et al., 2 Jun 2026).

3. Decoupled Intrinsic Recovery Head and Dynamic Point Refinement Offset

The Decoupled Intrinsic Recovery Head, or DIR-Head, is introduced to eliminate recursive intrinsic drift. Instead of predicting frame-wise intrinsics, a lightweight MLP $\mathcal{G}_t$5 takes only the first-frame features $\mathcal{G}_t$6 and predicts a single normalized focal ratio $\mathcal{G}_t$7, yielding a fixed focal length

$\mathcal{G}_t$8

where $\mathcal{G}_t$9 is the image width. The resulting intrinsic matrix is fixed per sequence:

$\mathcal{P}_t=\{\hat{K},\hat{R}_t,\hat{t}_t\}$0

This temporally decoupled design is intended to remove cumulative intrinsic bias and stabilize global scale across long streams (Chen et al., 2 Jun 2026).

During training, DIR-Head uses teacher-forcing for stability. A frozen teacher, VGGT, provides $\mathcal{P}_t=\{\hat{K},\hat{R}_t,\hat{t}_t\}$1, and the student is supervised by

$\mathcal{P}_t=\{\hat{K},\hat{R}_t,\hat{t}_t\}$2

with the teacher focal length also used in unprojection during training. At test time, the teacher is removed and DIR-Head alone supplies $\mathcal{P}_t=\{\hat{K},\hat{R}_t,\hat{t}_t\}$3 (Chen et al., 2 Jun 2026).

The second key mechanism is Dynamic Point Refinement Offset, abbreviated in the paper as DPRO or DPR-Offsets. Under rigid unprojection, a pixel $\mathcal{P}_t=\{\hat{K},\hat{R}_t,\hat{t}_t\}$4 with predicted depth $\mathcal{P}_t=\{\hat{K},\hat{R}_t,\hat{t}_t\}$5 is lifted as

$\mathcal{P}_t=\{\hat{K},\hat{R}_t,\hat{t}_t\}$6

FreeStreamGS then predicts a learned residual $\mathcal{P}_t=\{\hat{K},\hat{R}_t,\hat{t}_t\}$7 and forms the final center

$\mathcal{P}_t=\{\hat{K},\hat{R}_t,\hat{t}_t\}$8

This relaxes the rigid viewing-ray constraint so that coupled pose-depth drift can be corrected directly in 3D (Chen et al., 2 Jun 2026).

To stabilize the baseline depth used by DPRO, the method introduces a pseudo-depth prior from Depth Anything V3 and supervises the predicted depth through a scale-shift invariant alignment. With per-frame scale $\mathcal{P}_t=\{\hat{K},\hat{R}_t,\hat{t}_t\}$9 and shift $\mathcal{H}_t$0, the geometric loss is

$\mathcal{H}_t$1

The paper’s interpretation is that this stabilizes relative structure while DPRO learns spatial residuals through photometric supervision (Chen et al., 2 Jun 2026).

4. Online Recursive Gaussian Fusion and supervision

FreeStreamGS maintains a global voxelized cache to avoid redundancy and noise during accumulation. For each voxel $\mathcal{H}_t$2, it stores attributes $\mathcal{H}_t$3 and weight $\mathcal{H}_t$4. If $\mathcal{H}_t$5 denotes the pixels that project into voxel $\mathcal{H}_t$6 at time $\mathcal{H}_t$7, the update is defined by

$\mathcal{H}_t$8

and

$\mathcal{H}_t$9

where $F_t$0. This is a causal, confidence-weighted fusion rule that allows recursive integration of frame-wise Gaussian predictions into a persistent scene representation (Chen et al., 2 Jun 2026).

The total objective is

$F_t$1

Photometric reconstruction uses

$F_t$2

while Novel-View-Weighted Supervision augments input-view supervision with unseen targets:

$F_t$3

The paper states that this encourages geometry-driven generalization rather than view-specific appearance encoding (Chen et al., 2 Jun 2026).

Implementation-wise, FreeStreamGS initializes the causal feature extractor and extrinsic head from a Streaming 4D Visual Geometry Transformer with full-history causal attention and KV caching, keeps them frozen to avoid instability in unposed training, and uses a DPT-like convolutional decoder to predict depth, Gaussian attributes, confidence, and DPRO offsets. Optimization uses AdamW with batch size 4 across 4 RTX 5880 Ada GPUs for 5 epochs, with loss weights $F_t$4, $F_t$5, $F_t$6, $F_t$7, and $F_t$8 (Chen et al., 2 Jun 2026).

5. Empirical results, ablations, and applications

FreeStreamGS is trained on 10,000 DL3DV-10K sequences and 10,000 stereo trajectories, evaluated on the DL3DV official benchmark and 200 stereo test scenes, and also tested zero-shot on 100 NYUv2 scenes. All inputs are resized to width 518 pixels with proportional height (Chen et al., 2 Jun 2026).

In the sparse 5-view setting, the paper reports that FreeStreamGS achieves PSNR 21.884, SSIM 0.688, LPIPS 0.273 on DL3DV and PSNR 25.797, SSIM 0.833, LPIPS 0.200 on Stereo data. It remains competitive in the 10-view setting and, in the 64-view setting, handles dense inputs through Online Recursive Gaussian Fusion while offline feed-forward baselines run out of memory (Chen et al., 2 Jun 2026). For latency, the reported runtime is 250 ms per frame for the feed-forward online method, compared with 450 ms per frame for an optimization-based online baseline (Chen et al., 2 Jun 2026).

Ablation results isolate the contribution of the principal components. On DL3DV 5-view PSNR, removing DIR-Head reduces performance to 20.546, removing DPRO yields 21.348, removing Novel-View-Weighted Supervision yields 21.188, and the full model reaches 21.884 (Chen et al., 2 Jun 2026). The paper states that removing DIR-Head causes the largest drop, while removing DPRO also degrades quality, indicating that both mechanisms are essential for stabilizing geometry and improving rendering quality.

Cross-dataset generalization on NYUv2 is also reported. In the 5-view setting, FreeStreamGS achieves PSNR 24.875, SSIM 0.717, LPIPS 0.291, surpassing offline state-of-the-art methods in the sparse setting while maintaining competitive performance in denser regimes (Chen et al., 2 Jun 2026). Qualitatively, the method is reported to preserve fine details and sharp boundaries under causal streaming, whereas the optimization baseline often over-smooths high-frequency structure (Chen et al., 2 Jun 2026).

Beyond novel view synthesis, the paper demonstrates online full-frame video stabilization by smoothing the estimated extrinsic trajectory with a Savitzky–Golay filter and rendering along the stabilized path from the maintained 3DGS over 150-frame sequences. This application is presented as evidence that the maintained online Gaussian scene can support downstream camera-path manipulation, not only direct rendering (Chen et al., 2 Jun 2026).

6. Relation to adjacent streaming Gaussian systems and naming ambiguity

Within the broader literature, FreeStreamGS occupies the unposed, online novel-view-synthesis corner of streaming Gaussian research. Other systems address different problem formulations. “3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos” focuses on multi-view dynamic scenes with known cameras, using a Neural Transformation Cache and adaptive 3DG addition for efficient free-viewpoint video streaming (Sun et al., 2024). “AirGS: Real-Time 4D Gaussian Streaming for Free-Viewpoint Video Experiences” rearchitects training and delivery for 4DGS streaming, converting Gaussian streams into multi-channel 2D formats, selecting quality-driven keyframes, and transmitting adaptively pruned updates under bandwidth constraints (Wang et al., 24 Dec 2025).

A separate line uses compact temporal encodings for real-time FVV delivery. “StreamSTGS: Streaming Spatial and Temporal Gaussian Grids for Real-Time Free-Viewpoint Video” represents canonical Gaussian attributes as images and temporal features as a video, reporting an average frame size of about 170 KB while supporting adaptive bitrate control without retraining (Ke et al., 8 Nov 2025). “Motion Matters: Compact Gaussian Streaming for Free-Viewpoint Video Reconstruction” instead transmits sparse motion-sensitive keypoints and influence fields, reducing storage by over 159 X compared to 3DGStream and 14 X compared to the SOTA method QUEEN according to the paper (Chen et al., 22 May 2025). “Instant Gaussian Stream: Fast and Generalizable Streaming of Dynamic Scene Reconstruction via Gaussian Splatting” uses a generalized Anchor-driven Gaussian Motion Network and key-frame-guided streaming to reduce average per-frame reconstruction time to 2.67 s while maintaining 200+ FPS rendering (Yan et al., 21 Mar 2025).

These systems are related but not equivalent. FreeStreamGS does not assume the multi-view, calibrated free-viewpoint-video setup used by AirGS, StreamSTGS, ComGS, or 3DGStream. Instead, it starts from unposed streaming inputs and reconstructs a consistent 3DGS scene online (Chen et al., 2 Jun 2026). This suggests that “FreeStreamGS” should be understood primarily as the name of the 2026 unposed-streaming framework, while in surrounding discussion the term may also appear as an informal shorthand for streaming Gaussian splatting more broadly.

A further distinction concerns delivery-centric systems. “GSStream: 3D Gaussian Splatting based Volumetric Scene Streaming System” addresses volumetric scene distribution through collaborative viewport prediction and DRL-based bitrate adaptation rather than online reconstruction from images (Tang et al., 10 Mar 2026). Likewise, “PRoGS: Progressive Rendering of Gaussian Splats” prioritizes splats for progressive loading and rendering of already trained scenes rather than causal reconstruction (Zoomers et al., 2024). These works are complementary to FreeStreamGS rather than direct alternatives: they address transport, prioritization, or streaming QoE, whereas FreeStreamGS addresses online scene formation from unposed inputs (Chen et al., 2 Jun 2026).

7. Limitations and prospective directions

The paper identifies several limitations. Transparent or reflective surfaces and thin or high-frequency structures, such as dense grass, remain difficult because of depth ambiguity and insufficient fusion resolution. Highly dynamic scenes and ultra large-scale environments are also challenging, since the method lacks explicit dynamic modeling and 3DGS storage overhead limits scalability (Chen et al., 2 Jun 2026).

These constraints place FreeStreamGS in a specific design regime: it is optimized for causal, feed-forward, geometry-consistent online reconstruction rather than dynamic-scene factorization or compact network delivery. A plausible implication is that future extensions would need to combine its causal camera-and-geometry stabilization mechanisms with explicit dynamic modeling, scene compression, or delivery-aware scheduling if the goal is end-to-end free-viewpoint streaming at scale.

In the broader trajectory of Gaussian-splatting research, FreeStreamGS establishes that carefully decoupling intrinsics from temporal dynamics and relaxing rigid lifting with learned spatial offsets is sufficient to restore the multi-view geometric consistency required for high-fidelity online novel view synthesis without future frames (Chen et al., 2 Jun 2026). Its significance lies less in bandwidth minimization than in demonstrating that unposed, streaming inputs can be converted online into a stable Gaussian scene suitable for immediate rendering, long-horizon accumulation, and downstream view manipulation.