Scene Representation Transformer Overview

• Scene Representation Transformer is a transformer-based approach that aggregates multi-view image tokens into a reusable, geometry-free scene representation for novel view synthesis.
• It employs an encoder–decoder architecture where the encoder collects unordered image patches and the decoder cross-attends with ray-conditioned queries to predict RGB output.
• Variants expand on SRT by integrating object-centric slots, latent pose estimations, dynamic controls, and explicit geometry representations for diverse scene reconstruction tasks.

Scene Representation Transformer (SRT) denotes a transformer-based approach to inferring a scene representation from a set of observations and querying that representation for downstream outputs, most prominently novel view synthesis. In the canonical formulation, SRT encodes a small set of posed or unposed RGB images into a set-latent scene representation and synthesises novel views in a single feed-forward pass, using attention rather than per-scene optimization or volumetric ray marching (Sajjadi et al., 2021). Subsequent work preserved this encoder–decoder premise while changing the representation type, conditioning signal, or output domain, yielding object-centric slot representations, latent pose spaces, dynamic control latents, continuous signed distance fields, 3D Gaussian splat scene representations, and topological scene graphs (Sajjadi et al., 2022, Sajjadi et al., 2022, Seitzer et al., 2023, Wu et al., 15 May 2026, Imtiaz et al., 29 Sep 2025, Zhang et al., 2024). This suggests that “Scene Representation Transformer” now refers both to the original geometry-free model and to a broader architectural family defined by multi-view token aggregation and query-based scene access.

1. Origins and conceptual scope

The original SRT was introduced for the classical problem of inferring a 3D scene representation from few images that can be used to render novel views at interactive rates (Sajjadi et al., 2021). Its central claim was that explicit geometric operators such as reprojection, volumetric integration, or per-scene latent optimization are not necessary for effective multi-view scene reasoning if a transformer is allowed to integrate information globally over an unordered set of image patches. The representation is “geometry-free” in the specific sense used by the paper: the model does not reconstruct a mesh, voxel grid, or radiance field, and it parameterizes a light field directly through attention over latent scene tokens rather than through density accumulation along rays (Sajjadi et al., 2021).

This formulation differed from radiance-field methods, which typically require accurate camera poses and expensive per-scene optimization, and from latent light field methods that rely on a single global latent (Sajjadi et al., 2021). At the same time, it already established several themes that remained stable across later work: amortization across scenes, permutation handling over input views, attention-based aggregation of multi-view evidence, and query-conditioned decoding. These themes reappear in object-centric rendering in OSRT, pose-free novel view synthesis in RUST, dynamic-video factorization in DyST, and continuous geometric modeling in IVGT (Sajjadi et al., 2022, Sajjadi et al., 2022, Seitzer et al., 2023, Wu et al., 15 May 2026).

A broader reading of the term emerged as transformers were used to produce scene-level outputs that were not limited to RGB rendering. Multiview Scene Graph builds a graph of place and object nodes from unposed images, while decision-making systems such as Scene-Rep Transformer and GITSR use transformer-derived scene latents as inputs to reinforcement-learning policies (Zhang et al., 2024, Liu et al., 2022, Hu et al., 2024). In that broader sense, a scene representation transformer is any transformer whose primary product is a reusable scene-level state rather than a purely image-level descriptor.

2. Canonical SRT architecture

In SRT, the input is an unordered collection of optionally posed images of a single scene,

$$\{I_i \in \mathbb{R}^{H\times W\times 3},\; c_i \in SE(3),\; K_i \in \mathbb{R}^{3\times 3}\}.$$

At train and test time, the input set is randomly shuffled and the model treats $I_0$ as the canonical reference frame; if poses are unavailable, $c_0$ is the identity (Sajjadi et al., 2021). For posed inputs, each ray $r=(o,d)$ is encoded relative to the canonical frame by

$$\mathrm{posenc}(r)=\mathrm{posenc}(c_0^{-1}\!\cdot o,\nu_o)\;\mathrm{concat}\;\mathrm{posenc}(c_0^{-1}\!\cdot d,\nu_d),$$

and these pose channels are concatenated to RGB before patch tokenization (Sajjadi et al., 2021).

A shared CNN with patch size $p=16$ converts each image into a grid of patch features. SRT adds a globally learned 2D position embedding to all patches and camera-identity embeddings that distinguish the canonical image from the remaining images. All image patch tokens are then gathered into an unordered set and processed by a transformer encoder:

$$\{z_z \in \mathbb{R}^d\}=E_{\theta_E}\!\left(CNN_{\theta_{CNN}}(\{I_i,c_i,K_i\})\right).$$

The result is the set-latent scene representation, or SLSR, whose size scales with the amount of input information rather than collapsing the scene into a single vector (Sajjadi et al., 2021).

Novel-view synthesis is performed by a lightweight decoder transformer that parameterizes the light field. Given a query ray, the decoder cross-attends into the SLSR and predicts color directly,

$C(r)=D_{\theta_D}(r\mid\{z_z\}).$

The decoder uses the ray embedding as query, computes keys and values from the scene tokens, and sends the resulting feature through a 2-layer MLP with a final sigmoid to produce RGB (Sajjadi et al., 2021). The paper reports that two decoder layers are sufficient and effective, whereas one layer performs poorly and deeper decoders give only marginal gains (Sajjadi et al., 2021).

Training is end-to-end with a novel-view reconstruction objective,

$$\arg\min_\theta \sum_s \mathbb{E}_{r\sim I_{s,i}^{gt}} \|C(r)-I_{s,i}^{gt}(r)\|_2^2,$$

using Adam with an initial learning rate $1\mathrm{e}{-4}$, decay to $I_0$0 over $I_0$1M steps, warmup for the first $I_0$2k steps, batch size $I_0$3, and $I_0$4 rays sampled per data point (Sajjadi et al., 2021). No explicit 3D supervision is used; geometry is learned implicitly through the combination of multi-view attention and ray-conditioned decoding.

3. Camera conditioning, pose freedom, and reference-frame invariance

The original SRT uses camera poses with respect to an arbitrarily chosen reference camera. RePAST identified that this makes the model not invariant to the order of the input views, because changing the reference camera changes the coordinate representation fed to both encoder and decoder (Safin et al., 2023). RePAST resolves this by injecting pairwise relative camera pose information directly into the attention mechanism. For camera poses $I_0$5, the relative pose is

$I_0$6

and query and key embeddings are formed from ray origin and direction expressed in the key token’s camera frame before the linear projections for $I_0$7 and $I_0$8 (Safin et al., 2023). Because these features depend only on pairwise relative pose, the attention logits are invariant to any global transform $I_0$9 applied to all cameras. On MultiShapeNet-Hard, RePAST reports PSNR $c_0$0, SSIM $c_0$1, and LPIPS $c_0$2, compared with SRT at PSNR $c_0$3, SSIM $c_0$4, and LPIPS $c_0$5; the decoder ablation RePAST-B reaches PSNR $c_0$6, SSIM $c_0$7, and LPIPS $c_0$8 (Safin et al., 2023).

RUST removes ground-truth cameras altogether while retaining the encoder–decoder paradigm of SRT (Sajjadi et al., 2022). Its key modification is a Pose Estimator that peeks at the target image during training and produces a low-dimensional latent pose embedding. A randomly chosen half of the target image is patchified, cross-attends into the subset of scene tokens corresponding to the first input view, and is linearly projected to an 8-dimensional latent pose

$c_0$9

For each target pixel at integer coordinates $r=(o,d)$0, RUST forms the decoder query as

$r=(o,d)$1

and predicts RGB through cross-attention into the SLSR:

$r=(o,d)$2

The model is trained end-to-end from unordered RGB images only, with no camera parameters for inputs or targets, using 5 input views, 3 target views, an $r=(o,d)$3 reconstruction loss, and gradient scaling $r=(o,d)$4 for the Pose Estimator (Sajjadi et al., 2022).

Empirically, RUST achieves PSNR $r=(o,d)$5 on MSN without any poses, compared with improved SRT at $r=(o,d)$6 with perfect poses and improved UpSRT at $r=(o,d)$7 with perfect target poses; when both input and target poses are noisy, SRT and UpSRT collapse to $r=(o,d)$8 dB, whereas RUST outperforms all baselines when target poses are noisy (Sajjadi et al., 2022). The learned latent pose space is structured: on MSN, the first three principal components form a cylinder, PC1 correlates strongly with camera height with Pearson $r=(o,d)$9, and PC4 captures distance from the scene center with $$\mathrm{posenc}(r)=\mathrm{posenc}(c_0^{-1}\!\cdot o,\nu_o)\;\mathrm{concat}\;\mathrm{posenc}(c_0^{-1}\!\cdot d,\nu_d),$$0 (Sajjadi et al., 2022). An auxiliary Explicit Pose Estimation head trained on top of a frozen RUST achieves MSE $$\mathrm{posenc}(r)=\mathrm{posenc}(c_0^{-1}\!\cdot o,\nu_o)\;\mathrm{concat}\;\mathrm{posenc}(c_0^{-1}\!\cdot d,\nu_d),$$1 and $$\mathrm{posenc}(r)=\mathrm{posenc}(c_0^{-1}\!\cdot o,\nu_o)\;\mathrm{concat}\;\mathrm{posenc}(c_0^{-1}\!\cdot d,\nu_d),$$2 over 95 MSN test scenes using 7 total views, with success rate $$\mathrm{posenc}(r)=\mathrm{posenc}(c_0^{-1}\!\cdot o,\nu_o)\;\mathrm{concat}\;\mathrm{posenc}(c_0^{-1}\!\cdot d,\nu_d),$$3 (Sajjadi et al., 2022).

A common misconception is that “unposed” scene representation transformers do not learn camera structure. RUST and DyST show the opposite: they remove explicit camera supervision from the rendering path, but they still learn latent camera control variables with measurable geometric structure (Sajjadi et al., 2022, Seitzer et al., 2023).

4. Object-centric and dynamic factorizations

OSRT extends SRT toward unsupervised, object-centric scene decomposition by inserting Slot Attention between the SLSR and the decoder (Sajjadi et al., 2022). The SLSR

$$\mathrm{posenc}(r)=\mathrm{posenc}(c_0^{-1}\!\cdot o,\nu_o)\;\mathrm{concat}\;\mathrm{posenc}(c_0^{-1}\!\cdot d,\nu_d),$$4

is converted into a fixed set of slots $$\mathrm{posenc}(r)=\mathrm{posenc}(c_0^{-1}\!\cdot o,\nu_o)\;\mathrm{concat}\;\mathrm{posenc}(c_0^{-1}\!\cdot d,\nu_d),$$5 through slot-normalized attention,

$$\mathrm{posenc}(r)=\mathrm{posenc}(c_0^{-1}\!\cdot o,\nu_o)\;\mathrm{concat}\;\mathrm{posenc}(c_0^{-1}\!\cdot d,\nu_d),$$6

followed by a GRU-based update (Sajjadi et al., 2022). Rather than decoding each slot independently, OSRT introduces the Slot Mixer. A decoder transformer produces a per-ray feature $$\mathrm{posenc}(r)=\mathrm{posenc}(c_0^{-1}\!\cdot o,\nu_o)\;\mathrm{concat}\;\mathrm{posenc}(c_0^{-1}\!\cdot d,\nu_d),$$7, scalar slot weights are computed by normalized dot-product similarity, and a single mixed slot representation $$\mathrm{posenc}(r)=\mathrm{posenc}(c_0^{-1}\!\cdot o,\nu_o)\;\mathrm{concat}\;\mathrm{posenc}(c_0^{-1}\!\cdot d,\nu_d),$$8 is passed to an MLP:

$$\mathrm{posenc}(r)=\mathrm{posenc}(c_0^{-1}\!\cdot o,\nu_o)\;\mathrm{concat}\;\mathrm{posenc}(c_0^{-1}\!\cdot d,\nu_d),$$9

This keeps the light-field parameterization of SRT—direct mapping from ray $p=16$0 to color—while making the representation object-aware (Sajjadi et al., 2022).

OSRT is trained solely with the same kind of $p=16$1 novel-view loss used in SRT. On MSN-Hard, ObSuRF reports PSNR $p=16$2 and FG-ARI $p=16$3, whereas OSRT(1) reaches PSNR $p=16$4 and FG-ARI $p=16$5, and OSRT(5) reaches PSNR $p=16$6 and FG-ARI $p=16$7 (Sajjadi et al., 2022). In the decoder ablation, the non-object-centric SRT decoder gives PSNR $p=16$8 and FG-ARI $p=16$9, the Spatial Broadcast decoder gives PSNR $$\{z_z \in \mathbb{R}^d\}=E_{\theta_E}\!\left(CNN_{\theta_{CNN}}(\{I_i,c_i,K_i\})\right).$$0 and FG-ARI $$\{z_z \in \mathbb{R}^d\}=E_{\theta_E}\!\left(CNN_{\theta_{CNN}}(\{I_i,c_i,K_i\})\right).$$1, and Slot Mixer gives PSNR $$\{z_z \in \mathbb{R}^d\}=E_{\theta_E}\!\left(CNN_{\theta_{CNN}}(\{I_i,c_i,K_i\})\right).$$2 and FG-ARI $$\{z_z \in \mathbb{R}^d\}=E_{\theta_E}\!\left(CNN_{\theta_{CNN}}(\{I_i,c_i,K_i\})\right).$$3 at $$\{z_z \in \mathbb{R}^d\}=E_{\theta_E}\!\left(CNN_{\theta_{CNN}}(\{I_i,c_i,K_i\})\right).$$4 fps (Sajjadi et al., 2022). The paper reports $$\{z_z \in \mathbb{R}^d\}=E_{\theta_E}\!\left(CNN_{\theta_{CNN}}(\{I_i,c_i,K_i\})\right).$$5 fps for OSRT on a V100 GPU versus $$\{z_z \in \mathbb{R}^d\}=E_{\theta_E}\!\left(CNN_{\theta_{CNN}}(\{I_i,c_i,K_i\})\right).$$6 fps for ObSuRF, attributing the speedup to the light-field formulation and to Slot Mixer (Sajjadi et al., 2022).

DyST generalizes the SRT pattern to dynamic monocular videos by separating scene content, camera, and dynamics (Seitzer et al., 2023). Given a dynamic scene $$\{z_z \in \mathbb{R}^d\}=E_{\theta_E}\!\left(CNN_{\theta_{CNN}}(\{I_i,c_i,K_i\})\right).$$7 and input subset $$\{z_z \in \mathbb{R}^d\}=E_{\theta_E}\!\left(CNN_{\theta_{CNN}}(\{I_i,c_i,K_i\})\right).$$8, DyST forms a set-based scene representation

$$\{z_z \in \mathbb{R}^d\}=E_{\theta_E}\!\left(CNN_{\theta_{CNN}}(\{I_i,c_i,K_i\})\right).$$9

then infers a camera control latent and a dynamics control latent from the target:

$C(r)=D_{\theta_D}(r\mid\{z_z\}).$0

and synthesizes

$C(r)=D_{\theta_D}(r\mid\{z_z\}).$1

Disentanglement is induced by a latent control swap scheme on the synthetic DySO dataset, where camera information for one target is taken from another frame with the same camera but different dynamics, and dynamics information is taken from a frame with the same dynamics but different camera (Seitzer et al., 2023). On DySO, the full model reports PSNR $C(r)=D_{\theta_D}(r\mid\{z_z\}).$2, LPIPS $C(r)=D_{\theta_D}(r\mid\{z_z\}).$3, $C(r)=D_{\theta_D}(r\mid\{z_z\}).$4, and $C(r)=D_{\theta_D}(r\mid\{z_z\}).$5, while the “No swap” ablation gives PSNR $C(r)=D_{\theta_D}(r\mid\{z_z\}).$6, LPIPS $C(r)=D_{\theta_D}(r\mid\{z_z\}).$7, $C(r)=D_{\theta_D}(r\mid\{z_z\}).$8, and $C(r)=D_{\theta_D}(r\mid\{z_z\}).$9 (Seitzer et al., 2023).

A specialized but structurally related variant is FSRT for face reenactment, which learns a set-latent representation of the source identity and conditions the decoder on driving keypoints and a learned facial expression vector rather than on camera pose (Rochow et al., 2024). It preserves the per-pixel cross-attentive decoding pattern of SRT, naturally supports multiple source images, and reports multi-source self-reenactment metrics of SSIM $$\arg\min_\theta \sum_s \mathbb{E}_{r\sim I_{s,i}^{gt}} \|C(r)-I_{s,i}^{gt}(r)\|_2^2,$$0, PSNR $$\arg\min_\theta \sum_s \mathbb{E}_{r\sim I_{s,i}^{gt}} \|C(r)-I_{s,i}^{gt}(r)\|_2^2,$$1, L1 $$\arg\min_\theta \sum_s \mathbb{E}_{r\sim I_{s,i}^{gt}} \|C(r)-I_{s,i}^{gt}(r)\|_2^2,$$2, and AKD $$\arg\min_\theta \sum_s \mathbb{E}_{r\sim I_{s,i}^{gt}} \|C(r)-I_{s,i}^{gt}(r)\|_2^2,$$3; its small-decoder variant reaches approximately $$\arg\min_\theta \sum_s \mathbb{E}_{r\sim I_{s,i}^{gt}} \|C(r)-I_{s,i}^{gt}(r)\|_2^2,$$4 fps on a single NVIDIA RTX 4090 (Rochow et al., 2024).

5. Explicit geometry, local attention, and broader scene representations

The SRT family does not remain confined to geometry-free light fields. Several later models preserve multi-view transformer aggregation while replacing the latent scene type and rendering mechanism.

Model	Scene representation	Output
TransNeRF	coordinate-based radiance field $$\arg\min_\theta \sum_s \mathbb{E}_{r\sim I_{s,i}^{gt}} \|C(r)-I_{s,i}^{gt}(r)\|_2^2,$$5	NeRF-style novel view synthesis
IVGT	continuous signed distance field $$\arg\min_\theta \sum_s \mathbb{E}_{r\sim I_{s,i}^{gt}} \|C(r)-I_{s,i}^{gt}(r)\|_2^2,$$6 and color field $$\arg\min_\theta \sum_s \mathbb{E}_{r\sim I_{s,i}^{gt}} \|C(r)-I_{s,i}^{gt}(r)\|_2^2,$$7 in canonical space	RGB, depth, normals, meshes, camera poses
LVT	local-view transformer tokens decoded to a 3D Gaussian Splat scene representation	large-scale scene reconstruction and novel view synthesis
MSG	Multiview Scene Graph with place and object nodes	topological scene graph

TransNeRF is transformer-based NeRF rather than direct light-field rendering (Wang et al., 2022). It processes multi-view tokens in a surrounding-view space and local ray windows in a ray-cast space, predicting per-sample density and color for volume rendering:

$$\arg\min_\theta \sum_s \mathbb{E}_{r\sim I_{s,i}^{gt}} \|C(r)-I_{s,i}^{gt}(r)\|_2^2,$$8

On the synthetic scene-agnostic benchmark, average PSNR is $$\arg\min_\theta \sum_s \mathbb{E}_{r\sim I_{s,i}^{gt}} \|C(r)-I_{s,i}^{gt}(r)\|_2^2,$$9 for $1\mathrm{e}{-4}$0, $1\mathrm{e}{-4}$1 for $1\mathrm{e}{-4}$2, $1\mathrm{e}{-4}$3 for $1\mathrm{e}{-4}$4, and $1\mathrm{e}{-4}$5 for $1\mathrm{e}{-4}$6, versus IBRNet at $1\mathrm{e}{-4}$7, $1\mathrm{e}{-4}$8, $1\mathrm{e}{-4}$9, and $I_0$00 (Wang et al., 2022). This is a scene representation transformer in the sense that attention fuses multi-view evidence into a scene-conditioned radiance field, but it departs from SRT’s direct-pixel light field.

IVGT goes further by making geometry explicit through a canonical implicit signed distance field and a view-dependent color field (Wu et al., 15 May 2026). Given unposed images, a transformer backbone initialized from VGGT produces per-view features, per-view depth maps, and camera parameters while building a unified global scene representation in the canonical coordinates of the first frame. A 3D query point $I_0$01 retrieves projected multi-view features, a geometry decoder predicts SDF and an intermediate appearance feature, normals are obtained by $I_0$02, and a color decoder predicts view-dependent color (Wu et al., 15 May 2026). Rendering is performed with VolSDF-style density mapping and NeRF transmittance integration. On ScanNet mesh reconstruction, IVGT reports Chamfer $I_0$03, Comp $I_0$04, and F-score $I_0$05 (Wu et al., 15 May 2026). The paper is explicit that this differs from earlier SRT-like variants, which typically rely on known cameras and focus on appearance synthesis rather than explicit geometry or SDF-based surfaces (Wu et al., 15 May 2026).

LVT addresses scaling by replacing global self-attention with attention restricted to local neighborhoods of nearby views (Imtiaz et al., 29 Sep 2025). For $I_0$06 views with $I_0$07 tokens each, global attention has complexity $I_0$08, whereas LVT uses local-view attention with complexity $I_0$09 for neighborhood size $I_0$10 (Imtiaz et al., 29 Sep 2025). Relative pose, encoded as quaternion plus translation, is injected into keys and values, and final-layer tokens are decoded into per-pixel Gaussian parameters for a 3D Gaussian Splat scene representation (Imtiaz et al., 29 Sep 2025). On DL3DV-140 at $I_0$11, LVT_SH-rgba reports PSNR $I_0$12, SSIM $I_0$13, and LPIPS $I_0$14, compared with Long-LRM at PSNR $I_0$15, SSIM $I_0$16, and LPIPS $I_0$17 (Imtiaz et al., 29 Sep 2025).

MSG broadens the phrase “scene representation transformer” beyond metric reconstruction (Zhang et al., 2024). It defines a Multiview Scene Graph

$I_0$18

with place nodes, object nodes, place-place edges, and place-object edges, all inferred from unposed RGB images (Zhang et al., 2024). AoMSG uses a DETR-like decoder over image tokens to produce embeddings for places and object detections, and graph quality is evaluated with adjacency IoU. AoMSG-4 reports Recall@1 $I_0$19, PP IoU $I_0$20, and PO IoU $I_0$21 with GT detections (Zhang et al., 2024). This does not synthesize images, but it fits the broader transformer-to-scene-state template.

6. Empirical profile, downstream uses, and recurrent limitations

SRT established the empirical profile that made the architecture notable: fast scene onboarding, interactive rendering, and generalization across scenes (Sajjadi et al., 2021). On NMR it reports PSNR $I_0$22, SSIM $I_0$23, and LPIPS $I_0$24, compared with PixelNeRF at PSNR $I_0$25, SSIM $I_0$26, and LPIPS $I_0$27; on MultiShapeNet it reports PSNR $I_0$28, SSIM $I_0$29, and LPIPS $I_0$30, compared with PixelNeRF at PSNR $I_0$31, SSIM $I_0$32, and LPIPS $I_0$33 (Sajjadi et al., 2021). Measured on a single NVIDIA V100, SRT encodes a scene in $I_0$34 s, renders at $I_0$35 fps, and produces a 100-frame new-scene video in $I_0$36 s with batching (Sajjadi et al., 2021). The same paper also shows that the frozen scene representation can support downstream semantic segmentation, with a separate decoder trained to predict 46-class semantic masks, and that a 4-D appearance encoder helps on Street View imagery with exposure and white-balance variation (Sajjadi et al., 2021).

Across the literature, several misconceptions recur. One is that scene representation transformers are necessarily geometry-free. This is true of the original SRT and of OSRT’s light-field rendering, but not of TransNeRF’s volumetric radiance field, IVGT’s canonical SDF, or LVT’s 3D Gaussian Splat scene representation (Sajjadi et al., 2021, Wang et al., 2022, Wu et al., 15 May 2026, Imtiaz et al., 29 Sep 2025). Another is that global all-to-all attention is intrinsic to the paradigm; LVT shows that local neighborhoods can replace quadratic attention without abandoning the scene-representation formulation (Imtiaz et al., 29 Sep 2025). A third is that removing camera supervision discards camera structure; RUST and DyST instead replace explicit pose with learned low-dimensional camera latents (Sajjadi et al., 2022, Seitzer et al., 2023).

The limitations are similarly consistent. The original SRT exhibits blurriness under $I_0$37 loss in regions of uncertainty and may underperform explicit-geometry methods on very small datasets or when target views are very close to the inputs and poses are perfect (Sajjadi et al., 2021). RePAST remains sensitive to pose noise and very sparse views because it injects relative pose directly into attention (Safin et al., 2023). RUST learns dataset camera distributions and may struggle with out-of-distribution poses; it also shows higher PSNR variance across seeds, with standard error approximately $I_0$38 over 3 seeds (Sajjadi et al., 2022). IVGT notes that appearance fidelity can lag behind dedicated splatting or graphics methods, that Eikonal smoothness may damp thin structures and sharp edges, and that static, bounded scenes are assumed (Wu et al., 15 May 2026). LVT can miss long-range dependencies when many redundant overlapping views are present, and rasterization can become a bottleneck for very large high-resolution scenes (Imtiaz et al., 29 Sep 2025). DyST identifies multiple independently moving objects, longer trajectories, and lighting changes as remaining challenges (Seitzer et al., 2023).

Taken together, the literature defines Scene Representation Transformer less as a single network and more as a research program. The stable core is a transformer that aggregates multi-view or multi-entity evidence into a reusable scene-level state. What varies is the ontology of that state: a set-latent light field in SRT, slots in OSRT, latent pose in RUST, camera and dynamics controls in DyST, a radiance field in TransNeRF, a canonical SDF in IVGT, Gaussian splats in LVT, or a topological graph in MSG (Sajjadi et al., 2021, Sajjadi et al., 2022, Sajjadi et al., 2022, Seitzer et al., 2023, Wang et al., 2022, Wu et al., 15 May 2026, Imtiaz et al., 29 Sep 2025, Zhang et al., 2024). The technical evolution of the area has therefore proceeded not by discarding the SRT paradigm, but by reinterpreting what the scene representation should be and what forms of query-conditioned reasoning it should support.