UniSHARP: Universal Sharp Monocular View Synthesis

Abstract: In this work, we focus on extending SHARP, the popular photorealistic view synthesis method, for universal monocular rendering across a continuum of camera systems, from conventional perspective cameras to wide-field-of-view, fisheye and omnidirectional panoramic settings. To overcome the pinhole-specific assumptions of SHARP, our key idea is to align various images in a unified omnidirectional latent space. Thus, we propose UniSHARP, which performs implicit alignment in both feature and Gaussian spaces. Specifically, Gaussian primitives are arranged along rays and radial distances in a ray-based universal representation, while 2D semantic and 3D spatial features extracted from UniK3D-inspired encoders are jointly decoded to generate the complete Gaussian cloud. To comprehensively evaluate our method, we construct a benchmark covering diverse imaging systems across various scenes. The benchmark is further stratified by field of view (FoV) to enable fine-grained assessment of the universal monocular rendering task. Extensive experiments on the proposed benchmark demonstrate the effectiveness of UniSHARP, outperforming alternative methods by a large margin. The project page can be found at: https://insta360-research-team.github.io/Unisharp-website/

• The paper introduces a ray-distance-based Gaussian representation that unifies novel view synthesis across diverse camera projections.
• It demonstrates superior performance with state-of-the-art metrics on perspective, wide-FoV, fisheye, and panoramic inputs, reducing artifacts seen in traditional approaches.
• The method supports pose-free inference by inferring camera parameters from predicted ray fields, expediting rendering and expanding real-world applicability.

Universal Monocular View Synthesis Across Camera Projections with UniSHARP

Introduction

The monocular novel view synthesis (NVS) problem remains underconstrained due to the loss of spatial information in single images, particularly when extending beyond the perspective pinhole camera model. Conventional 3D Gaussian Splatting (3DGS) and feedforward approaches such as SHARP [mescheder2026sharp] are typically specialized to perspective views, exhibiting degraded performance when deployed on images captured using wide-field-of-view (FoV), fisheye, or panoramic cameras. The capacity to generalize single-image NVS to arbitrary camera projections is critical for advancing embodied visual intelligence, AR/VR systems, robotics, and scalable 3D content generation pipelines constrained by heterogeneous real-world imaging hardware.

UniSHARP presents a unification of single-image 3DGS by reformulating Gaussian prediction and scene representation in a projection-agnostic, ray-distance-based space. It enables robust, photorealistic view synthesis from a single image across perspective, wide-FoV, fisheye, and panoramic cameras, removing projective constraints and specialized re-projection heuristics. Robust evaluation on a stratified benchmark—comprising both real and simulated environments—demonstrates not only state-of-the-art numerical performance but also strong qualitative results that generalize seamlessly to unseen projection geometries.

Figure 1: UniSHARP enables novel view synthesis with diverse camera types (perspective, wide-FoV, fisheye, panoramic) from a single input image by predicting a 3D Gaussian cloud.

Unified Ray-Distance-Based Gaussian Representation

UniSHARP initiates a departure from image-plane-centric approaches by structuring all Gaussian primitives along rays parameterized by their direction and radial distance from the camera origin, as opposed to fixed image-grid coordinates. This universal representation directly generalizes angular sampling and spatial footprint across arbitrary projection models, e.g., equirectangular, fisheye, and perspective. The architecture extends UniK3D-style feature extractors to both predict per-pixel ray fields (in $\mathbb{S}^2$) and multi-scale features.

Scene reconstruction is performed by first initializing two-layer "Geometry Anchored Gaussians" along these rays to capture both immediate surfaces and potential disocclusions. Feature-Conditioned Gaussian residuals, synthesized by fusing 2D semantic encodings with 3D spatial features, are subsequently decoded and composed with the geometry anchors. The resulting Gaussians collectively form a point cloud that is rendered using a unified 3DGS rasterizer capable of accommodating any camera model.

Figure 2: UniSHARP architecture leverages a pipeline wherein a ray-distance representation standardizes geometry, supporting all camera projection models within a single framework.

Camera and Pose Flexibility

A salient feature of UniSHARP is its ability to operate without explicit camera intrinsics. The model infers camera geometry and projection parameters directly from the learned ray field, supporting pose-free monocular inference. For perspective and fisheye images, canonical camera parameters are fitted to the predicted rays. For panoramas, spherical models are adopted. This flexibility is critical for real-world deployment in unconstrained environments.

Training, Losses, and Panoramic Regularization

UniSHARP is trained jointly on perspective, wide-FoV, fisheye, and panoramic images using a mixture of datasets and simulated data (e.g., the OmniRooms benchmark). The loss function comprises photometric appearance, depth, and regularization terms for the Gaussian fields. Panoramic inputs, with equirectangular over-sampling at the poles, require further regularization: a spherical latitude-dependent probabilistic dropout is applied to Gaussians in the second layer, regularizing opacity and mitigating polar distortion artifacts.

Experimental Results

Quantitative analysis on the stratified benchmark shows UniSHARP outperforming prior single-image 3DGS baselines (e.g., SHARP, Flash3D) and multi-plane/MLP alternatives across all camera groups.

Perspective Cameras: On datasets such as RealEstate10K, DL3DV, and WildRGB-D, UniSHARP achieves top PSNR, SSIM, and LPIPS values, demonstrating no compromise in perspective rendering quality despite universal training. Out-of-domain generalization on Tanks and Temples further validates improved cross-dataset fidelity.

Figure 3: On perspective images, UniSHARP generates sharper, structurally consistent target views with fewer artifacts than monocular baselines.

Wide-FoV and Fisheye: On ScanNet++ Fisheye and projected OmniRooms-Wide, UniSHARP achieves improvements of up to 4–6 dB PSNR and 2× better LPIPS compared to Matrix3D and PanoDreamer, demonstrating generalization to severe radial distortions.

Figure 4: Validation data visualizations for native fisheye inputs emphasize the angular distortion captured and handled by UniSHARP’s universal representation.

Panoramic Synthesis: On HM3D, OmniRooms, and Replica panoramas, UniSHARP attains best-in-class metrics—up to 5–7 dB above nearest competitors—highlighting the efficacy of native panoramic training and rendering without requiring face-wise cubemap decomposition.

Figure 5: Given a panoramic input, UniSHARP reconstructs coherent 3D geometry and renders sharp, distortion-free target views.

Architectural Ablations: Removing native resolution Gaussian allocation, the second Gaussian layer, or panoramic-specific regularization, each yields statistically significant drops in evaluation metrics, confirming the necessity of each design facet.

Cubemap-Baseline Comparison: Standard SHARP cannot directly process panoramic inputs and must employ cubemap decomposition, resulting in pronounced stitching artifacts at face boundaries. UniSHARP’s native approach yields seamless, artifact-free outputs.

Figure 6: Unlike the stitched cubemap strategy, UniSHARP's direct panoramic rendering prevents seam artifacts, maintaining geometric and appearance consistency.

Inference Efficiency: UniSHARP demonstrates an order of magnitude faster inference (3.1s per sample) versus diffusion-based or optimization-centric models, owing to its feedforward Gaussian decoding pipeline.

Implications and Outlook

UniSHARP’s unified ray-distance space formulation demonstrates that explicit, projection-agnostic representations can achieve both state-of-the-art performance and strong generalization across arbitrary camera geometries in monocular NVS. This contrasts with models that rely on projection-specific heuristics or re-projection pipelines. Practical applications are extensive—ranging from enhanced AR/VR content creation on consumer devices using non-pinhole lenses, to improved spatial perception for autonomous robots and embodied agents.

On the theoretical front, UniSHARP’s design establishes universal ray-based coordinates as a viable backbone for single-image NVS, subsuming a range of projective setups and distortion models without increasing architectural or computational complexity. The inclusion of pose-free rendering mechanisms makes this paradigm robust to scenarios with unknown camera calibration.

Future work should address the extrapolation limits when unseen content comprises most of the novel view, as well as extending ray-based representations to temporally consistent video generation and dynamic scene modeling.

Conclusion

UniSHARP offers a unified, efficient, and highly generalizable solution for monocular novel view synthesis across camera models—enabling reliable 3D Gaussian rendering from a single image regardless of the underlying projection. Its ray-distance representation, feature-conditioned Gaussian field design, and joint training with panoramic-specific regularization establish a new reference for universal-camera NVS frameworks. This foundational advance paves the way for delivering robust 3D understanding and immersive visual synthesis in unconstrained, real-world imaging systems.

(2606.07514)