ReCoSplat: Online Novel View Synthesis

• ReCoSplat is an autoregressive feed-forward Gaussian Splatting system that synthesizes novel views online by integrating sequential image chunks into a persistent scene.
• It leverages a Render-and-Compare module to condition Gaussian predictions on discrepancies between observed and rendered outputs, mitigating pose error.
• Hybrid KV cache compression reduces memory usage by over 90% for long sequences, enabling efficient and robust real-time online reconstruction.

ReCoSplat is an autoregressive feed-forward Gaussian Splatting system for online novel view synthesis from a stream of images arriving sequentially. It is designed for settings in which observations may be posed or unposed, and in which camera intrinsics may or may not be available. The method addresses the central difficulties of online reconstruction—sequential arrivals, pose uncertainty, error accumulation, and long-sequence memory growth—by predicting local Gaussians per chunk, assembling them into a persistent scene, conditioning Gaussian prediction on a Render-and-Compare signal, and compressing the transformer KV cache for long histories (Cheng et al., 10 Mar 2026).

1. Problem formulation and representational strategy

ReCoSplat studies online novel view synthesis from a continuous stream of images processed in chunks. At each time step, the model receives a chunk of images and must estimate camera parameters if needed, predict 3D Gaussian primitives for the current chunk, merge them into a persistent scene representation, and preserve enough history to process future frames. The setting is harder than offline reconstruction because the model does not see all views at once, cannot jointly optimize over the full trajectory, and must remain robust when future views are unavailable (Cheng et al., 10 Mar 2026).

The paper distinguishes three input regimes: posed + calibrated, posed but uncalibrated, and unposed. This formulation makes camera extrinsics optional and allows intrinsics to be either provided or inferred. A practical consequence is that reconstruction quality depends not only on Gaussian prediction but also on how well the system can infer and use camera parameters in an online loop.

The core representational choice is to predict local Gaussians in each image’s camera coordinate frame and then assemble them into world coordinates using an assembly pose. The scene update is written as

$$S_t = S_{t-1} \cup \mathrm{Transform}(\mathbf{G}_t, \mathbf{A}_t).$$

This “predict locally, assemble globally” strategy is contrasted with canonical-space prediction. The paper states that local prediction scales better, avoids forcing everything into a single global latent coordinate system early, and supports chunk-by-chunk integration of newly observed views. This suggests that ReCoSplat treats geometric persistence as an assembly problem rather than as direct canonical reconstruction (Cheng et al., 10 Mar 2026).

2. Autoregressive architecture

ReCoSplat is an autoregressive feed-forward Gaussian Splatting model that maintains a KV cache for transformer history, an accumulated scene $S_{t-1}$, and camera pose estimates for the current chunk. Its chunk-level update is expressed as

$$(\mathbf{G}_t, \hat{\mathbf{K}}_t, \hat{\mathbf{P}}_t, M_t) = f_\theta(\mathbf{I}_t, \mathbf{K}_t, \mathbf{P}_t, M_{t-1}, S_{t-1}).$$

Here the model predicts local Gaussians $\mathbf{G}_t$, camera intrinsics $\hat{\mathbf{K}}_t$, camera extrinsics $\hat{\mathbf{P}}_t$, and an updated memory state $M_t$ from the current image chunk and prior state (Cheng et al., 10 Mar 2026).

The backbone is based on a YoNoSplat-style architecture with a DINOv2 ViT encoder and an alternating-attention transformer decoder. For each image $I_t^k$, the encoder produces a global camera token and patch tokens,

$[c_t^k, E_t^k] = \mathrm{Encoder}(I_t^k).$

An intrinsics head predicts camera intrinsics from the camera token, the decoder processes patch tokens with an autoregressive KV cache, an extrinsics head predicts camera poses from decoded features, and Gaussian heads predict local Gaussian parameters. The Gaussian prediction stage is conditioned on Render-and-Compare tokens $Z_t^k$, which are introduced later:

$S_{t-1}$0

The implementation details reported in the paper specify a DINOv2 ViT encoder, a 36-layer alternating-attention transformer decoder, three training stages, initialization from a YoNoSplat DL3DV-10K checkpoint, and training on 8 GPUs with batch size 1. The first chunk is fixed at 8 images, while later chunks are sampled in $S_{t-1}$1 during curriculum training. These details matter because the method is explicitly optimized for autoregressive deployment rather than per-scene optimization (Cheng et al., 10 Mar 2026).

3. Render-and-Compare and the training–inference mismatch

A central problem addressed by ReCoSplat is the mismatch between training with ground-truth assembly poses and inference with predicted poses. Ground-truth poses stabilize Gaussian placement during training, but at inference time predicted poses can be noisy, producing misalignment, blur, artifacts, and compounding scene errors. The paper further states that a naïve curriculum mixing predicted and ground-truth poses during training does not solve this well in the autoregressive setting, because predicted online poses are often too noisy to provide strong supervision (Cheng et al., 10 Mar 2026).

The principal mechanism introduced to address this issue is the Render-and-Compare (ReCo) module. For each incoming image $S_{t-1}$2, the model renders the current accumulated scene $S_{t-1}$3 from the current assembly pose $S_{t-1}$4:

$S_{t-1}$5

The rendered output has shape

$S_{t-1}$6

comprising 3 RGB channels and 9 learned feature channels. The paper states that these extra feature channels provide richer scene context and that removing them degrades performance when predicted assembly poses are used.

The rendered output is concatenated with the incoming observation and patchified,

$S_{t-1}$7

and the resulting tokens are used as a conditioning signal for the Gaussian heads via cross-attention. The comparison is therefore between the observed incoming view and the rendered current reconstruction at the same pose. The paper characterizes this as “analysis by synthesis”: the model renders its current hypothesis, compares it to the next observation, and uses the discrepancy to guide Gaussian prediction. Small pose errors alter the rendered image directly, exposing where the current scene already explains the observation and where corrections are needed. A plausible implication is that ReCoSplat does not attempt to eliminate pose error directly; rather, it learns Gaussian prediction conditioned on pose-perturbed rendered evidence (Cheng et al., 10 Mar 2026).

Ablations reported in the paper state that using predicted assembly poses directly hurts performance, that mixed ground-truth/predicted pose training gives little benefit, and that geometric conditioning alone via Plücker raymaps is weaker than ReCo. The appendix also shows that the extra 9 Gaussian feature dimensions improve robustness, especially when predicted assembly poses are used at inference.

4. Objectives, pose handling, and sparsity constraints

ReCoSplat is trained with a multi-task objective combining photometric, camera, and sparsity terms:

$S_{t-1}$8

The reported weights are $S_{t-1}$9, $$(\mathbf{G}_t, \hat{\mathbf{K}}_t, \hat{\mathbf{P}}_t, M_t) = f_\theta(\mathbf{I}_t, \mathbf{K}_t, \mathbf{P}_t, M_{t-1}, S_{t-1}).$$0, $$(\mathbf{G}_t, \hat{\mathbf{K}}_t, \hat{\mathbf{P}}_t, M_t) = f_\theta(\mathbf{I}_t, \mathbf{K}_t, \mathbf{P}_t, M_{t-1}, S_{t-1}).$$1, $$(\mathbf{G}_t, \hat{\mathbf{K}}_t, \hat{\mathbf{P}}_t, M_t) = f_\theta(\mathbf{I}_t, \mathbf{K}_t, \mathbf{P}_t, M_{t-1}, S_{t-1}).$$2, and $$(\mathbf{G}_t, \hat{\mathbf{K}}_t, \hat{\mathbf{P}}_t, M_t) = f_\theta(\mathbf{I}_t, \mathbf{K}_t, \mathbf{P}_t, M_{t-1}, S_{t-1}).$$3 (Cheng et al., 10 Mar 2026).

Photometric supervision uses pixel MSE and LPIPS between rendered novel views and held-out ground-truth images. The intrinsic loss penalizes focal-length error between predicted and ground-truth intrinsics. For extrinsics, the paper gives a pairwise relative pose loss. With predicted poses $$(\mathbf{G}_t, \hat{\mathbf{K}}_t, \hat{\mathbf{P}}_t, M_t) = f_\theta(\mathbf{I}_t, \mathbf{K}_t, \mathbf{P}_t, M_{t-1}, S_{t-1}).$$4, $$(\mathbf{G}_t, \hat{\mathbf{K}}_t, \hat{\mathbf{P}}_t, M_t) = f_\theta(\mathbf{I}_t, \mathbf{K}_t, \mathbf{P}_t, M_{t-1}, S_{t-1}).$$5 and ground-truth poses $$(\mathbf{G}_t, \hat{\mathbf{K}}_t, \hat{\mathbf{P}}_t, M_t) = f_\theta(\mathbf{I}_t, \mathbf{K}_t, \mathbf{P}_t, M_{t-1}, S_{t-1}).$$6, $$(\mathbf{G}_t, \hat{\mathbf{K}}_t, \hat{\mathbf{P}}_t, M_t) = f_\theta(\mathbf{I}_t, \mathbf{K}_t, \mathbf{P}_t, M_{t-1}, S_{t-1}).$$7, the relative poses are

$$(\mathbf{G}_t, \hat{\mathbf{K}}_t, \hat{\mathbf{P}}_t, M_t) = f_\theta(\mathbf{I}_t, \mathbf{K}_t, \mathbf{P}_t, M_{t-1}, S_{t-1}).$$8

The loss is

$$(\mathbf{G}_t, \hat{\mathbf{K}}_t, \hat{\mathbf{P}}_t, M_t) = f_\theta(\mathbf{I}_t, \mathbf{K}_t, \mathbf{P}_t, M_{t-1}, S_{t-1}).$$9

with

$\mathbf{G}_t$0

and

$\mathbf{G}_t$1

The opacity term is an $\mathbf{G}_t$2 penalty encouraging sparse Gaussians, and Gaussians with opacity below $\mathbf{G}_t$3 are pruned. During inference, if ground-truth extrinsics are available, they are not required to be metric. Instead, their scale is aligned to predicted poses using only the first chunk:

$\mathbf{G}_t$4

The paper notes that this first-chunk normalization is necessary because an autoregressive model must commit to a scale immediately, unlike offline methods that can normalize across the whole trajectory (Cheng et al., 10 Mar 2026).

5. Long-sequence memory compression

A defining systems contribution of ReCoSplat is hybrid KV cache compression for long sequences. Because the transformer backbone is autoregressive, the KV cache grows with sequence length and can become memory-bound. ReCoSplat introduces a two-part strategy combining early-layer truncation with chunk-level selective retention (Cheng et al., 10 Mar 2026).

In the first component, the first 10 of 18 global attention layers are truncated so that no historical KV cache is stored for those layers; they attend only to the current chunk. The paper justifies this by stating that early layers mostly extract local features and contribute less to multi-view correspondence.

In the second component, the remaining 8 global attention layers retain only a compressed summary of past chunks. The first chunk is kept fully to establish a strong initial context. For later chunks, only one representative view is kept, specifically the last frame of the chunk, and a trainable register token marks retained views. The paper also reports variable chunk-size training over chunk sizes $\mathbf{G}_t$5–$\mathbf{G}_t$6, allowing flexible test-time chunk sizes with little or no performance loss.

For $\mathbf{G}_t$7 images and chunk size $\mathbf{G}_t$8, a standard cache stores 4608 token sets, whereas ReCoSplat stores 312 token sets, corresponding to a 93% reduction in KV-cache memory. The abstract summarizes this result as a reduction of over 90% for sequences of 100+ frames. The paper further states that compression causes minimal performance degradation while substantially lowering memory usage. This suggests that ReCoSplat’s scalability is not only architectural but also explicitly memory-optimized for long online streams (Cheng et al., 10 Mar 2026).

6. Empirical performance, scope, and limitations

The reported training datasets are DL3DV and ScanNet++, and the reported test datasets are DL3DV, ScanNet++, RealEstate10K, ACID, and ScanNet. Across DL3DV, ScanNet, RealEstate10K, and ScanNet++, ReCoSplat is reported to consistently outperform autoregressive baselines such as KV Cache and GIR, and to improve over S3PO-GS in the settings reported. In fully posed settings it remains competitive with, and sometimes surpasses, the offline baseline YoNoSplat in PSNR (Cheng et al., 10 Mar 2026).

The paper reports several qualitative patterns: on DL3DV, ReCoSplat is best across 32/64/128/256-view settings; on out-of-distribution ScanNet and RealEstate10K it also leads; and on ScanNet++ with 256 and 512 views it remains the best autoregressive method. For camera pose estimation, it performs strongly on ACID, RealEstate10K, and DL3DV, and outperforms online baselines with recurrent memory, token pools, or uncompressed KV caches. The qualitative conclusions reported are sharper geometry, fewer artifacts, better long-range consistency, and more robust reconstruction in unposed settings.

The paper’s stated main contributions are fourfold: Render-and-Compare, hybrid KV cache compression, autoregressive feed-forward Gaussian Splatting for online NVS under posed or unposed inputs and with or without intrinsics, and strong empirical performance among autoregressive methods. The practical applications named are embodied AI, AR/VR, live scene capture, online telepresence, and video-to-3D reconstruction.

The stated limitations are equally explicit. ReCoSplat improves robustness but cannot fully decouple reconstruction quality from pose estimation accuracy. Large pose errors can still propagate into Gaussian assembly and degrade rendering, and better online pose estimation would likely further improve results. A common source of confusion is nomenclature: ReCoSplat is distinct from ReferSplat, which addresses Referring 3D Gaussian Splatting Segmentation rather than online novel view synthesis (He et al., 11 Aug 2025). The two works both operate in 3D Gaussian Splatting, but they target different tasks and introduce different conditioning mechanisms.