UniSHARP: Monocular Synthesis & Medical Segmentation

• UniSHARP is a unified paradigm that integrates monocular novel view synthesis via a ray-distance space with advanced UNet# segmentation innovations.
• It employs layered Gaussian primitives and a hybrid 2D+3D feature fusion to support diverse camera models from pinhole to panoramic, yielding superior performance metrics.
• The UNet# variant redesigns skip connections by merging dense and full-scale links with deep supervision, enhancing accuracy and enabling efficient, scalable medical image segmentation.

UniSHARP and its variants represent a new paradigm for both monocular view synthesis in computer vision and medical image segmentation, targeting universal applicability through architectural innovations. Two unrelated neural models share the designation "UniSHARP" or "UNet#" (pronounced "Uni-sharp"): (1) UniSHARP for universal sharp monocular view synthesis (Song et al., 5 Jun 2026), and (2) UNet# for UNet-like segmentation networks with redesigned skip connections (Qian et al., 2022). Both systems are characterized by an overview of legacy methods and principled architectural enhancements, achieving superior generalization and performance across challenging domains.

1. Universal Sharp Monocular View Synthesis

UniSHARP (Song et al., 5 Jun 2026) generalizes monocular novel view synthesis (NVS) to accommodate perspective, wide-field-of-view (FoV), fisheye, and full 360° panoramic inputs with a unified ray-based latent space. Standard feed-forward Gaussian Splatting methods such as SHARP and Flash3D assume a pinhole camera model, limiting applicability for wide-FoV projections. UniSHARP resolves this by aligning feature and geometry priors in a ray-distance space, defined so that all camera models share a universal 3D parameterization.

Key representation advances include:

• Ray-Distance Space: Each pixel $p$ in source image $I_s\in\mathbb{R}^{3\times H\times W}$ is mapped to a unit ray $r_p\in S^2$ and radial distance $d_p>0$, with the 3D location $x_p=d_p r_p$. This obviates reliance on image-plane coordinates.
• Layered Gaussian Primitives: A two-layer grid per ray, with geometry-anchored Gaussians $$B_{p,\ell} = \{r_p, \rho_{p,\ell}, s^0_{p,\ell}, q^0, c^0_p, \alpha^0_\ell\}$$, where $\ell\in\{1,2\}$. The first layer aligns with the primary surface, the second with possibly occluded surfaces, and each primitive admits 14-dimensional learned residuals.
• Omnidirectional Rendering: By representing all Gaussians in the ray-distance frame, UniSHARP directly supports pinhole, fisheye, and panoramic (equirectangular) projections without per-pixel warping or camera-specific branches.

2. Implicit Feature and Gaussian Space Alignment

Feature extraction and spatial reasoning leverage a hybrid architecture:

• ViT-L Backbone: Multi-scale 2D activation maps are extracted at Gaussian-grid resolution ($D\approx768$ per location). Concurrently, a shallow Transformer-based "ray head" regresses ray directions and distances supervised by ground-truth depth.
• 2D+3D Feature Fusion: For each anchor, 2D semantic features $F^2(p)$ and 3D geometric features $F^3(p,\ell)$ are concatenated and decoded with a shallow MLP (3–4 layers), using FiLM-style dynamic conditioning.
• Shared Weights: The network handles all camera intrinsic variations without model branching, directly fusing geometric and semantic signals in the latent space.

3. Training Protocol and Objective Function

UniSHARP is optimized end-to-end on a mixed-camera batch regime:

• Composite Loss: The total loss $I_s\in\mathbb{R}^{3\times H\times W}$0, combining photometric/perceptual fidelity, inverse-depth supervision, and Gaussian regularization.
  • Photometric/Perceptual Loss: $I_s\in\mathbb{R}^{3\times H\times W}$1 incorporates $I_s\in\mathbb{R}^{3\times H\times W}$2 color error, binary cross-entropy opacity matching, and a perceptual + Gram loss on VGG features.
  • Depth Loss: $I_s\in\mathbb{R}^{3\times H\times W}$3 encourages both anchor and rendered depths to match ground truth.
  • Regularization: $I_s\in\mathbb{R}^{3\times H\times W}$4 includes TV smoothing, "floater" suppression at depth discontinuities, and Sobel-edge log-depth gradient alignment.
• Panoramic Adaptation: For equirectangular images, the second Gaussian layer is probabilistically dropped near the poles (proportional to $I_s\in\mathbb{R}^{3\times H\times W}$5), mitigating polar over-sampling.
• Training Procedure: Curriculum phases warm up the ray prediction head, followed by full-objective minimization with cosine learning-rate decay. Batches are sampled evenly from perspective, wide, fisheye, and panorama datasets, with shared network weights.

4. Benchmarking and Experimental Evaluation

UniSHARP is evaluated on a comprehensive, FoV-stratified benchmark:

Camera Group	Datasets	Pairs (k)
Pinhole (60–90°)	RealEstate10K, DL3DV, WildRGB-D, Tanks&Temples	~36.9
Wide-FoV (90–140°)	OmniRooms-Wide	~10.7
Fisheye (140–180°)	ScanNet++ Fisheye	~14.2
Panorama (360°)	HM3D, Replica, OmniRooms	~42.8

Tests employ overlap $I_s\in\mathbb{R}^{3\times H\times W}$660%, center distance $I_s\in\mathbb{R}^{3\times H\times W}$70.5 m, and maximum frame gap of 10, measuring PSNR (↑), SSIM (↑), and LPIPS (↓), averaged by target, dataset, and camera group.

Major findings:

• In-domain Perspective: Outperforms SHARP/Flash3D by 3–4 dB PSNR and ∼0.1 SSIM.
• Generalization: On Tanks & Temples, UniSHARP scores 16.32 PSNR vs. 15.99 (Flash3D) and 15.96 (SHARP).
• Wide/Fisheye: Achieves 25.24/20.66 PSNR on OmniRooms-Wide/ScanNet++ Fisheye, compared to Matrix3D’s 18.88/16.38.
• Panorama: Scores 29.24 PSNR, 0.895 SSIM on HM3D, cf. Matrix3D 23.40, 0.793.

Ablation studies show that 2D+3D feature fusion, the dual-layer anchor, panoramic dropout, and target-depth loss all contribute materially to final accuracy (ΔPSNR up to –1.2 dB if removed).

5. Limitations and Prospective Directions

Current limitations include:

• Feed-Forward Limitation: UniSHARP acts as a regressor, not a generative completion model, and cannot hallucinate large unseen areas, resulting in visible holes under extreme extrapolation.
• Disocclusion Handling: Moderate disocclusion can be addressed, but not extensive novel content.
• Dynamic Scenes: Extension to dynamic objects or temporally-varying lighting remains unaddressed.

Future research is necessary to incorporate generative priors (diffusion or retrieval-based) for content completion within the 3DGS real-time suite, and to support non-static environments (Song et al., 5 Jun 2026).

6. UNet#: Unified Skip Connection Redesign for Medical Image Segmentation

UNet# (Qian et al., 2022) is a network architecture for medical image segmentation that unifies dense and full-scale skip connections from UNet2+ and UNet3+, constructing a 5×5 “#”-shaped computational grid. Each node $I_s\in\mathbb{R}^{3\times H\times W}$8 aggregates features from both encoder and decoder paths with lightweight convolution $I_s\in\mathbb{R}^{3\times H\times W}$9. This design balances fine-grained localization and multi-scale semantic context, interleaving dense horizontal and vertical (full-scale) connections at every node.

Distinctive architectural features include:

• Skip Connection Unification: Dense (in-row) and full-scale (vertical) skip-links at every decoder node, creating a “#” pattern.
• Deep Supervision and Pruning: Eight auxiliary outputs at different grid nodes enable hybrid multi-branch loss, supporting pruning to four sub-network levels ($r_p\in S^2$0–$r_p\in S^2$1) for mobile deployment. Each pruned model inherits supervised weights, ensuring calibration and speedup with marginal accuracy loss (from 9.71M to 0.10M params; 45FPS to 200FPS on 512×512 inputs).
• Classification-Guided Module (CGM): Small classification heads—large- or small-channel depending on location—suppress false positives at deep supervision branches, typically yielding 0.4–0.5% IoU improvement by trimming segmentation overreach.

7. Comparative Results and Applications in Medical Segmentation

UNet# demonstrates state-of-the-art or near-state-of-the-art performance across a range of 2D and 3D medical imaging tasks and biomedical datasets:

Dataset	UNet# +DS IoU	Competing SOTA
Dsb2018	92.67	UNet3+ +DS 92.15
BraTS19	92.38	UNet3+ +DS 91.69
LiTS17	95.36	UNet3+ +DS 92.14
LUNA16 (3D)	79.45	V-Net3+ 78.23

For instance segmentation on Dsb2018, replacing Mask R-CNN’s FPN skip-connections with the “#”-lattice yields IoU/Dice of 95.87/92.25 (ResNet101 backbone), outperforming all competitor modifications.

The architecture exhibits rapid convergence and narrow variance over epochs, and supports real-world deployment scenarios via aggressive pruning. On challenging organ and lesion segmentation tasks (BraTS19, LiTS17), the model outperforms DeepLabV3+, nnU-Net, Attn-UNet, and TransUNet.

---

For comprehensive technical detail, implementation resources, and further experimental evidence, see (Song et al., 5 Jun 2026) and (Qian et al., 2022).