PanoSplatt3R: Unposed Panorama Reconstruction

• PANORAMA refers to full 360° scene imagery used for immersive reconstruction, combining perspective priors with spherical adaptations for pose-free synthesis.
• PanoSplatt3R employs a Vision Transformer with cross-attention and RoPE rolling to extract and correlate features from two unposed wide-baseline panoramas.
• Quantitative results demonstrate improved PSNR and SSIM, with progressive training and perspective transfer yielding approximately 1 dB gains over pose-dependent methods.

Searching arXiv for the main paper and a few directly related works mentioned in the provided material. I’m going to look up the cited panorama-reconstruction line and the perspective pretraining papers on arXiv so the article can ground the discussion with current citations. PanoSplatt3R is an unposed wide-baseline panorama reconstruction method that reconstructs a full $360^\circ$ scene geometry and synthesizes photorealistic novel views from two unposed wide-baseline panoramas. Its central premise is that panorama reconstruction need not remain tied to accurate relative pose input: instead, foundational reconstruction pretrainings from the perspective domain can be extended and adapted to the panoramic domain, allowing geometric correspondences to emerge through attention rather than being imposed by an external pose estimate (Ren et al., 29 Jul 2025).

1. Problem setting and motivation

Wide-baseline panorama reconstruction has emerged as a highly effective approach for both geometric reconstruction of the surrounding 3D environment and generation of highly realistic and immersive novel views. In the formulation addressed by PanoSplatt3R, the input consists of two panoramas, the baseline is wide, and no camera pose is provided to the model at inference time (Ren et al., 29 Jul 2025).

The difficulty of this setting is twofold. First, existing panorama reconstruction methods, including PanoGRF, Splatter-360, and PanSplat, typically build a cost volume over depth hypotheses using known relative poses. In practical settings, however, accurate pose acquisition is expensive, noise-sensitive, and often unavailable. Second, the panoramic domain differs radically from the perspective-image priors on which most neural stereo and splatting methods have been trained: the representation is equirectangular, the image has a horizontal loop, and the ray density varies across the image (Ren et al., 29 Jul 2025).

This framing positions PanoSplatt3R at the intersection of two research directions. One is pose-dependent panoramic reconstruction; the other is transfer of large-scale perspective pretraining into spherical imagery. A plausible implication is that the method is less an isolated panorama-specific architecture than a transfer mechanism for bringing generalized stereo priors into the $360^\circ$ domain.

2. Architectural formulation

PanoSplatt3R leverages the strong generalization of foundational perspective stereo models such as Dust3R and Mast3R by transferring their weights into a panoramic pipeline with only minimal structural changes (Ren et al., 29 Jul 2025). The architecture consists of three principal stages.

A single Vision Transformer encoder extracts tokenized features from each input panorama. A decoder composed of cross-attention layers then learns to correlate features from view 1 and view 2 without any explicit pose input. The paper characterizes this mechanism directly: geometric correspondences emerge through attention (Ren et al., 29 Jul 2025). This is the core departure from cost-volume pipelines whose correlation structure is organized by known camera geometry.

The output representation is a shared set of 3D Gaussians. A pair of Dense Prediction Transformer heads estimate per-pixel 3D Gaussians for each view, parameterized by center location $x\in\mathbb{R}^3$, opacity $\alpha$, orientation $r\in SO(3)$, scaling $s\in\mathbb{R}^3$, and spherical-harmonic color coefficients $c$. The per-view Gaussian sets are fused by simple concatenation into a shared 3D Gaussian set, which can then be rendered at any novel camera pose via Gaussian splatting (Ren et al., 29 Jul 2025).

This representation choice links reconstruction and rendering tightly. Rather than predicting depth alone and relying on a separate rendering stage, the model directly predicts a renderable scene representation. A plausible implication is that the novel-view synthesis objective and the geometric objective operate on the same latent scene parameterization rather than on loosely coupled outputs.

3. Panoramic transfer and RoPE rolling

A central technical obstacle in transferring perspective pretraining to panoramas is the mismatch between standard rotary positional embeddings and equirectangular periodicity. RoPE normally splits each token’s $d$-dimensional feature into two halves, rotating one half by $\theta_t\cdot p_n^x$ and the other by $\theta_t\cdot p_n^y$. Under equirectangular projection, pixels at left $360^\circ$0 and right $360^\circ$1 should be adjacent, but RoPE treats them as maximally distant (Ren et al., 29 Jul 2025).

PanoSplatt3R addresses this with RoPE rolling across attention heads. For head $360^\circ$2 out of $360^\circ$3, the horizontal coordinate is offset by $360^\circ$4 before computing RoPE:

$360^\circ$5

By slicing the panoramic width into $360^\circ$6 circular phase shifts, every head sees the panorama with a different wrap-around offset, ensuring that left/right continuity is represented somewhere in the multi-head aggregation (Ren et al., 29 Jul 2025).

The method is intentionally conservative in how it alters RoPE. Rather than re-tuning $360^\circ$7 to $360^\circ$8, which still leaves higher-frequency channels misaligned, RoPE rolling maintains a minimal modification to RoPE’s mechanism while modeling the horizontal periodicity of panorama images (Ren et al., 29 Jul 2025). The ablations reported in the paper support this design choice: omitting RoPE rolling drops PSNR by about $360^\circ$9 dB, and purely increasing $x\in\mathbb{R}^3$0 to $x\in\mathbb{R}^3$1 fares worse (Ren et al., 29 Jul 2025).

This design belongs to a broader panorama-specific positional-encoding literature. HORIZON introduces Spherical Relative Embedding as an extension of RoPE for panoramic synthesis, while PanoWorld uses Circular PRoPE so that $x\in\mathbb{R}^3$2 and $x\in\mathbb{R}^3$3 share identical sinusoidal features (Yan et al., 2022, Jia et al., 18 May 2026). PanoSplatt3R’s contribution within this landscape is to preserve the pretrained RoPE structure and distribute periodic continuity across heads rather than redefine the embedding globally.

4. Training procedure and scale recovery

Training proceeds in two stages without posing. Stage 1 is a geometry warm-up in which all Gaussian parameters except centers are frozen. The model supervises the predicted centers $x\in\mathbb{R}^3$4 against ground-truth 3D points $x\in\mathbb{R}^3$5 obtained by projecting each pixel with its known depth. For pixels in view 1, $x\in\mathbb{R}^3$6 is directly known; for view 2, the point is reprojected with $x\in\mathbb{R}^3$7 and transformed by the known relative pose, which is used only for supervision. The stage-1 loss is

$x\in\mathbb{R}^3$8

Stage 2 adds the remaining Gaussian parameters and supervises 6 cube-map renderings $x\in\mathbb{R}^3$9 at a novel viewpoint with

$\alpha$0

The total loss is $\alpha$1 in stage 1 and $\alpha$2 in stage 2 (Ren et al., 29 Jul 2025).

The curriculum is progressive: first $\alpha$3 only, then $\alpha$4 is added. The stated reason is stabilization. Early random Gaussian opacities yield weak photometric gradients if attended before centers are accurate, so the geometry warm-up is not merely a convenience but a condition for stable optimization (Ren et al., 29 Jul 2025). The ablation confirms that skipping the progressive $\alpha$5 schedule hurts final PSNR by about $\alpha$6 dB (Ren et al., 29 Jul 2025).

Because reconstruction from two unposed panoramas has an inherent scale ambiguity, the method recovers a global scale after prediction. It estimates the relative pose between views via Perspective-n-Point using the predicted point maps, computes

$\alpha$7

and scales all 3D Gaussians’ centers and scales by $\alpha$8. The paper emphasizes that this step does not affect geometry quality; it simply aligns units so that depth metrics are meaningful (Ren et al., 29 Jul 2025).

An important clarification follows from this design. A common misconception is that any use of pose in supervision invalidates the claim of unposed reconstruction. In PanoSplatt3R, relative pose is not an input to the reconstruction pipeline; it is used only for supervision in stage 1 and for post hoc scale alignment. The forward model itself remains pose-free at inference.

5. Quantitative behavior and ablations

The reported experiments cover HM3D and Replica two-view tests, fixed-baseline comparisons, extrapolation tests, and component ablations. The headline claim is that PanoSplatt3R, even in the absence of pose information, significantly outperforms current state-of-the-art methods in both the generation of high-quality novel views and the accuracy of depth estimation (Ren et al., 29 Jul 2025).

Dataset	PSNR	SSIM	LPIPS	AbsRel	RMSE	$\alpha$9
HM3D	28.94	0.869	0.147	0.044	0.147	97.65%
Replica	31.52	0.935	0.082	0.034	0.126	98.12%

On these two-view tests, the method substantially exceeds posed SOTA such as Splatter-360 and PanoGRF, despite those methods requiring accurate poses (Ren et al., 29 Jul 2025). The paper further reports that when existing methods are given estimated poses from SIFT+8-point, their performance collapses, which is presented as evidence for the fragility of pose-dependent pipelines (Ren et al., 29 Jul 2025).

The fixed-baseline comparisons are also notable. Although trained on HM3D, PanoSplatt3R matches or outperforms methods trained on Matterport3D at $r\in SO(3)$0 m, $r\in SO(3)$1 m, and $r\in SO(3)$2 m baselines on Replica and Residential test sets, and degrades gracefully as baseline widens (Ren et al., 29 Jul 2025). This suggests that the transferred perspective priors are not narrowly tuned to a single baseline regime.

The ablation study identifies four especially consequential components. Omitting RoPE rolling reduces PSNR and increases LPIPS; removing perspective pretraining causes a loss greater than $r\in SO(3)$3 dB; skipping the progressive training schedule hurts the final PSNR; and 2DGS splatting outperforms 3DGS by about $r\in SO(3)$4 dB (Ren et al., 29 Jul 2025). The largest of these effects is the removal of perspective pretraining, which indicates that the method’s generalization is fundamentally inherited rather than learned from scratch in the panoramic domain.

Extrapolation tests reported in Table A1 show higher PSNR and LPIPS and lower depth errors beyond the training span, which the paper interprets as revealing robust spatial priors transferred from perspective data (Ren et al., 29 Jul 2025). This suggests that the model is not merely interpolating between two observed panoramas but using pretrained geometric structure to support out-of-span synthesis.

6. Significance, limitations, and research context

PanoSplatt3R’s practical significance lies in removing pose input from a task that has traditionally depended on it. The paper explicitly identifies VR/AR, real estate walkthroughs, robotics, and autonomous navigation as target settings where panoramas are easily captured but poses are noisy or unavailable. Because the output is a Gaussian-splatting representation, rendering is real-time and integration with downstream graphics engines is straightforward (Ren et al., 29 Jul 2025).

Methodologically, the work is part of a broader shift in panoramic vision away from treating $r\in SO(3)$5 imagery as an isolated modality. HORIZON addressed spherical distortion and left-right continuity for panorama synthesis through Spherical Relative Embedding and spherical conditioning (Yan et al., 2022). PanoVGGT introduced spherical-aware positional embeddings, panorama-specific $r\in SO(3)$6 rotation augmentation, and stochastic anchoring for feed-forward 3D reconstruction from panoramic imagery (Guo et al., 18 Mar 2026). PanoSplatt3R differs from both by focusing specifically on generalized unposed wide-baseline panorama reconstruction and by making transfer from perspective stereo pretraining the primary mechanism.

The limitations stated in the paper are also specific. The method relies on wide-baseline overlap for cross-attention, and it can exhibit occasional artifacts in extreme lighting (Ren et al., 29 Jul 2025). These are not incidental shortcomings. The first follows directly from the attention-based correspondence mechanism; without sufficient overlap, cross-view correlation becomes ill-posed. The second indicates that domain transfer from perspective imagery and synthetic supervision does not fully resolve photometric edge cases in panoramic environments.

Future work proposed by the paper includes multi-view extensions, dynamic scene handling, and self-supervised geometric consistency losses to improve robustness and reduce dependence on synthetic depth for supervision (Ren et al., 29 Jul 2025). A plausible implication is that PanoSplatt3R can be read as a two-view proof of principle for a broader research program: panorama reconstruction without pose input, using minimal architectural deviation from strong perspective foundations while introducing just enough spherical inductive bias to make transfer effective.