RayDer: Unified Transformer for NVS

• RayDer is a unified self-supervised transformer that integrates per-frame camera estimation and a minimal dynamic state to enable novel view synthesis from unposed monocular videos.
• It treats dynamic content as a nuisance factor, using a dedicated state dropout mechanism to improve static-scene rendering and ensure robust pose prediction.
• Empirical results demonstrate clean power-law scaling with increased data and compute, achieving near state-of-the-art performance across multiple benchmarks.

Searching arXiv for the cited RayDer paper and closely related ray-centric work for citation grounding. RayDer is a unified, feed-forward transformer for self-supervised novel view synthesis (NVS) from unposed, real-world video, introduced as a model that consolidates camera estimation, scene reconstruction, and rendering into a single backbone (Prestel et al., 29 May 2026). Its target problem is static-scene NVS under monocular video supervision without camera intrinsics or extrinsics labels, while dynamic content is treated as a nuisance factor rather than as a reconstruction target. In this formulation, the model predicts per-frame camera parameters and a minimal dynamic state, and then synthesizes target views in a single forward pass. The central claim of the method is that this unification turns self-supervised NVS into a single-model scaling problem with clean power-law behavior in data and compute, while enabling stable training on unconstrained real-world video (Prestel et al., 29 May 2026).

1. Problem formulation and scope

RayDer addresses self-supervised NVS for mostly static scenes from unposed, real-world videos (Prestel et al., 29 May 2026). A training sample is a short monocular video clip

$$\mathcal{I} = \{I_i \in \mathbb{R}^{H\times W\times 3} \mid i=1,\dots,K\},$$

with no camera intrinsics or extrinsics labels. The model is required to learn per-frame intrinsics $K_i$, extrinsics $T_i=(R_i,t_i)\in SE(3)$, a latent scene representation, and a rendering function such that, for an arbitrary target camera $p_j=(K_j,R_j,t_j)$ and context views, it can synthesize a novel view $\hat I_j$ close to the true $I_j$ (Prestel et al., 29 May 2026).

The method is explicitly positioned against several neighboring paradigms. In contrast to supervised NVS, it learns poses from scratch and uses its own predicted poses at test time. In contrast to NeRF-style per-scene optimization and 3D Gaussian methods, it is a single global model trained once and applied zero-shot to new scenes. In contrast to dynamic or 4D NVS, it does not attempt to reconstruct dynamics as a first-class target; instead, it uses dynamic content only as scalable supervision for static-scene NVS (Prestel et al., 29 May 2026).

Three obstacles motivate this design. First, dynamic content in realistic video can entangle scene changes with camera motion, causing pose drift, inconsistency, and training instability when the model is only allowed to encode camera variables. Second, prior self-supervised NVS systems based on multiple separate networks exhibit brittle scaling because parameter allocation and compute balance across components are difficult to tune. Third, prior methods trained on curated static datasets displayed opaque or saturating scaling behavior. RayDer’s formulation is intended to remove these constraints by making the entire problem a single end-to-end photometric learning system (Prestel et al., 29 May 2026).

A plausible implication is that RayDer should be understood less as a ray-based geometric detector and more as a ray-conditioned generative geometry model: its “ray” component arises through camera representation and rendering geometry, not through explicit object-centric ray queries of the type used in multi-camera 3D detection systems such as RayFormer (Chu et al., 2024).

2. Unified transformer architecture

RayDer replaces the multi-network decomposition of earlier systems with a single transformer backbone $\mathcal{M}$ that operates in two modes (Prestel et al., 29 May 2026). In camera mode,

$\mathcal{M} : \{I_i\} \mapsto \{(p_i, s_i)\},$

it predicts per-view pose $p_i$ and a dynamic state $s_i$. In NVS mode,

$K_i$0

it synthesizes a target image from input views and a target camera-plus-state specification. The same backbone is reused across these functions, with role-conditioned blocks distinguishing camera tokens, state tokens, and view tokens (Prestel et al., 29 May 2026).

Image tokenization uses patch embeddings, typically at $K_i$1 resolution in scaling experiments and up to $K_i$2 for the final large model, with patch size usually $K_i$3. Local high-resolution layers use Neighborhood Attention to improve fine detail without incurring the global cost that would follow from reducing patch size throughout the full network (Prestel et al., 29 May 2026).

Camera information is represented through predicted intrinsics and extrinsics and then converted into Plücker ray maps, providing per-pixel ray origin and direction

$K_i$4

which are concatenated with RGB channels. Extrinsics are parameterized by a twist $K_i$5 mapped to $K_i$6 via the exponential map,

$K_i$7

and intrinsics use a scalar focal length in normalized coordinates,

$K_i$8

The principal point is implicitly fixed at image center (Prestel et al., 29 May 2026).

The transformer uses axial RoPE for intra-frame spatial attention and no positional encoding for global attention across frames. Token roles are explicitly defined along the axes

$K_i$9

with role-specific adaptive RMSNorm. Each block follows a LLaMA2-style structure, and each layer contains two attention stages: intra-view attention within each frame and global attention across view tokens (Prestel et al., 29 May 2026).

The paper reports four principal model scales.

Model	Layers $T_i=(R_i,t_i)\in SE(3)$0	Params
RayDer-XS	12	59M
RayDer-S	18	145M
RayDer-B	24	422M
RayDer-L	24	743M

A further systems contribution is the use of factorized attention masking, parallel-target attention, and KV caching. This allows input-view tokens to be encoded once and reused across multiple target predictions, yielding an approximately $T_i=(R_i,t_i)\in SE(3)$1 reduction in per-novel-view compute with minor quality tradeoff (Prestel et al., 29 May 2026).

3. Dynamic state and static-scene factorization

A defining feature of RayDer is the introduction of a minimal dynamic state $T_i=(R_i,t_i)\in SE(3)$2, typically 256-dimensional, predicted for each frame alongside camera parameters (Prestel et al., 29 May 2026). The purpose of this latent is not to represent a physically coherent 4D scene but to absorb time-varying content that would otherwise corrupt pose estimation and destabilize training.

The method assumes that frame-to-frame differences in realistic video reflect both camera motion and scene state change. If only camera variables are available to the model, dynamic phenomena such as moving people or cars can be encoded spuriously into pose tokens. RayDer therefore augments the camera estimate with a state embedding:

$T_i=(R_i,t_i)\in SE(3)$3

and conditions rendering on both pose and state:

$T_i=(R_i,t_i)\in SE(3)$4

This state has no explicit flow, temporal field, or object decomposition; it is deliberately a nuisance latent (Prestel et al., 29 May 2026).

Because a target-state embedding is unavailable for arbitrary novel views at test time, RayDer introduces state dropout during training. With some probability, exemplified in the paper by $T_i=(R_i,t_i)\in SE(3)$5, the target state is zeroed:

$T_i=(R_i,t_i)\in SE(3)$6

This forces the network to learn rendering both with and without access to dynamic information and enables static-scene inference by setting $T_i=(R_i,t_i)\in SE(3)$7 (Prestel et al., 29 May 2026).

The paper states that adding this state eliminates the catastrophic divergences that RayZer exhibits on dynamic video and improves pose metrics such as $T_i=(R_i,t_i)\in SE(3)$8 and $T_i=(R_i,t_i)\in SE(3)$9 relative to a baseline without state. At inference on standard static benchmarks, the model predicts poses and states from input images but discards the state during novel-view synthesis by zeroing it. On dynamic scenes, the result is that static background is reconstructed coherently while moving objects tend to become blurred or averaged (Prestel et al., 29 May 2026).

This suggests that RayDer’s factorization is intentionally asymmetric: it aims to protect static 3D geometry learning from dynamic contamination, not to solve a disentangled static-plus-dynamic decomposition problem. That point distinguishes it sharply from explicit 4D NVS systems.

4. Self-supervised objective and training regime

RayDer is trained purely through photometric reconstruction from large-scale real video. The principal training sources are SpatialVid, comprising about $p_j=(K_j,R_j,t_j)$0 million videos, and SA-V, with controlled ablations additionally using SpatialVid-HQ and static mixtures such as RE10K, DL3DV-10K, and uCO3D (Prestel et al., 29 May 2026). From each video, eight frames are extracted at about $p_j=(K_j,R_j,t_j)$1 fps after pre-downsampling to $p_j=(K_j,R_j,t_j)$2 fps for I/O efficiency, with slight random jitter in timestamps; videos shorter than $p_j=(K_j,R_j,t_j)$3 seconds are skipped (Prestel et al., 29 May 2026).

Training uses an autoregressive view regime with random frame order. For a sequence $p_j=(K_j,R_j,t_j)$4, earlier frames in a random permutation serve as inputs and later frames serve as targets. This random-order autoregression is designed to prevent the model from exploiting real temporal order as a shortcut and instead to force geometry-based pose learning (Prestel et al., 29 May 2026). A view-level causal mask in global attention enforces this autoregressive structure.

The training loss is

$p_j=(K_j,R_j,t_j)$5

Here $p_j=(K_j,R_j,t_j)$6 is per-pixel mean squared error, and $p_j=(K_j,R_j,t_j)$7 is an optional LPIPS-style perceptual loss. For almost all large-scale RayDer training, the perceptual weight is set to zero; only a small supervised “RayDer-B on DL3DV” setup uses perceptual loss (Prestel et al., 29 May 2026).

The training objective contains no explicit pose supervision, no depth supervision, no geometry regularizers, no multiview consistency term, and no explicit loss for the dynamic state. The paper characterizes the total objective as photometric reconstruction alone, with camera and rendering learned jointly end-to-end in a single backbone (Prestel et al., 29 May 2026).

This training design is central to the model’s scientific positioning. The reported claim is not merely that self-supervised NVS can work without labels, but that when the architecture is unified and dynamics are treated as nuisance variation, photometric supervision alone can produce both competitive rendering quality and transferable camera geometry at scale (Prestel et al., 29 May 2026).

5. Inference, zero-shot operation, and evaluation setting

At test time, RayDer is fully amortized: it requires no per-scene optimization, no finetuning, and no external pose solver (Prestel et al., 29 May 2026). The typical procedure is to estimate camera parameters from one or more input images, define one or more target cameras, and synthesize the corresponding novel views in a single forward pass. The rendering step is

$p_j=(K_j,R_j,t_j)$8

with target state usually zeroed to obtain static-scene rendering (Prestel et al., 29 May 2026).

The model can operate in few-view settings with one to three inputs, in denser multi-view settings such as 24 views, and along interpolated camera trajectories. For benchmarks, target cameras may be specified by ground-truth poses in order to isolate rendering performance from pose estimation error, or by the model’s own predicted poses when pose estimation itself is under evaluation (Prestel et al., 29 May 2026).

The resulting behavior on dynamic scenes is consistent with the training objective: static geometry is rendered with strong geometric coherence, while moving objects appear as blurred or averaged components when target state is zeroed (Prestel et al., 29 May 2026). This is not treated as a defect relative to the target task, because the method is explicitly optimized for static-scene NVS rather than for temporally faithful dynamic rendering.

A common misconception is that any model trained on video should reconstruct motion explicitly. RayDer rejects that premise. Its stated target task remains static-scene NVS, and dynamic content is leveraged purely as supervision to expose the network to orders-of-magnitude more data than curated static datasets provide (Prestel et al., 29 May 2026).

6. Scaling behavior and empirical performance

A major contribution of RayDer is the claim that self-supervised NVS exhibits clean power-law scaling when posed as a unified single-model problem (Prestel et al., 29 May 2026). Scaling experiments train RayDer-XS, S, B, and L on fractions of SpatialVid: 1% (about 27k videos), 10% (about 270k), and 100% (about 2.7M), evaluated zero-shot on RE10K under the PixelSplat protocol (Prestel et al., 29 May 2026).

The paper reports that increasing data consistently improves PSNR and SSIM while lowering LPIPS and MSE, provided compute is sufficient. Increasing model size improves performance until data becomes the bottleneck; at 1% data, large models can overfit and perform worse than smaller ones, while at 10% scaling still helps but begins to saturate for the largest models (Prestel et al., 29 May 2026).

For compute-optimal runs, the paper fits a joint power-law scaling form

$p_j=(K_j,R_j,t_j)$9

where $\hat I_j$0 is a lower-is-better metric, $\hat I_j$1 is training compute in GFLOPs, and $\hat I_j$2 is dataset size in number of videos (Prestel et al., 29 May 2026). The fitted laws on RE10K are reported as

$\hat I_j$3

$\hat I_j$4

$\hat I_j$5

with $\hat I_j$6 in all cases (Prestel et al., 29 May 2026). The paper interprets the nonzero $\hat I_j$7 terms as irreducible error associated with inherently unobserved or occluded content, and emphasizes that scaling compute without scaling data yields diminishing returns (Prestel et al., 29 May 2026).

The comparison between static-only and dynamic-video training is also central. On RE10K zero-shot evaluation, RayDer-L trained on a static mix of about 247k videos achieves 28.68 dB PSNR, training on SpatialVid only achieves 29.38 dB, and training on SpatialVid plus the static mix achieves 29.42 dB (Prestel et al., 29 May 2026). The paper’s conclusion is that large-scale dynamic video is more valuable than domain-matched static mixtures for scaling this class of models.

Across open-set NVS benchmarks including LLFF, DTU, CO3D, WildRGBD, Mip-NeRF 360, and Tanks & Temples, the paper reports that RayDer-L-$\hat I_j$8 is state-of-the-art or near-SOTA in many settings, significantly outperforming E-RayZer in almost all cases and rivaling or exceeding several diffusion-based or pose-supervised methods despite no pose supervision, no 2D or video diffusion pretraining, and smaller training compute (Prestel et al., 29 May 2026). One notable exception is DTU, where the model underperforms supervised methods; the authors attribute this to distribution mismatch and unreliable pose estimation under turntable scenes with blank backgrounds (Prestel et al., 29 May 2026).

On DL3DV-10K, RayDer-B trained only on DL3DV slightly outperforms RayZer and LVSM, with PSNR 24.51 versus 24.36 and 23.69, respectively (Prestel et al., 29 May 2026). On SpatialVid, a purely self-supervised RayDer-B achieves 28.35 dB on RE10K, compared with 25.44 dB for LVSM trained on the same data using MegaSaM pseudo-ground-truth poses, while requiring much less overall compute than the pseudo-pose generation pipeline (Prestel et al., 29 May 2026).

7. Camera geometry, relation to other ray-based methods, and limitations

RayDer’s learned camera representation is evaluated not only by rendering quality but also by transferability. A pose probe trained on frozen camera tokens regresses ground-truth poses on DL3DV-10K, and a pose-transferability evaluation measures how estimated poses from one scene drive reconstruction in another (Prestel et al., 29 May 2026). The reported transferability table shows:

Model	R@10°	T@10°
RayZer	0.48	0.12
XFactor	0.93	0.55
RayDer-L	0.92	0.44

RayDer-L also reaches $\hat I_j$9, $I_j$0, $I_j$1, and $I_j$2, matching XFactor on the latter two thresholds and surpassing RayZer by a large margin (Prestel et al., 29 May 2026). The paper argues that the simultaneous improvement of probe and transfer metrics with scale indicates genuine 3D geometry learning rather than shortcut exploitation.

In the broader taxonomy of ray-centric methods, RayDer differs fundamentally from ray-based 3D detection models. RayFormer, for example, uses rays as an object-query parameterization in multi-camera 3D object detection, with polar BEV queries, ray-segment sampling, and a Hungarian radian cost for detection refinement (Chu et al., 2024). RayDer instead uses predicted camera geometry to construct per-pixel Plücker ray maps for feed-forward NVS (Prestel et al., 29 May 2026). The shared terminology of “ray” therefore masks two distinct methodological roles: in RayFormer, rays index detection hypotheses; in RayDer, rays instantiate the camera-conditioned rendering geometry.

The principal limitations reported for RayDer are fourfold. First, unobserved regions produce blurry mean predictions because the model uses regression rather than an uncertainty-aware or generative decoder (Prestel et al., 29 May 2026). Second, dynamic content is not faithfully reconstructed; moving objects become temporally averaged and spatially blurred (Prestel et al., 29 May 2026). Third, distribution mismatch causes failures on datasets such as DTU, where pose estimation can degrade under blank backgrounds and turntable capture (Prestel et al., 29 May 2026). Fourth, the largest models remain compute- and memory-intensive, although the paper notes that they are still much cheaper than very large diffusion-based NVS models with video pretraining (Prestel et al., 29 May 2026).

The future directions proposed by the authors include generative decoders or uncertainty-aware outputs for unobserved regions, extending the minimal dynamic state into an explicit 4D representation while preserving training on generic monocular video, incorporating partial pose or depth supervision where available, and further scaling toward a more general 3D world model (Prestel et al., 29 May 2026). This suggests that RayDer is best interpreted as a transitional architecture: it demonstrates that large-scale self-supervised NVS from unconstrained video is feasible and predictable under scaling, while leaving open the question of how much explicit dynamics, uncertainty, and hybrid supervision can be incorporated without losing that scaling regularity.