--- title: 'FisheyeGaussianLift: 3D Reconstruction from Fisheye' url: https://www.emergentmind.com/topics/fisheyegaussianlift type: topic --- # FisheyeGaussianLift: 3D Reconstruction from Fisheye FisheyeGaussianLift refers to a class of algorithms and frameworks for lifting distorted fisheye imagery into explicit 3D or BEV (bird’s-eye view) Gaussian representations, enabling high-fidelity reconstruction, novel view synthesis, and semantic understanding directly from wide-FOV, distortion-heavy inputs. These methods address the incompatibilities between traditional 3D Gaussian Splatting (3DGS), which assumes a pinhole camera model, and the unique projections of fisheye or omnidirectional sensors. FisheyeGaussianLift encompasses approaches that adapt both the geometric lifting (“unprojection”) and the differentiable rendering pipeline to wide-angle and non-pinhole lenses, supporting advances in autonomous driving, robotics, 360° scene rendering, and scene segmentation. ## 1. Mathematical Foundations and Gaussian Parameterization FisheyeGaussianLift algorithms jointly model scene geometry and uncertainty using explicit anisotropic 3D Gaussian primitives. Each primitive is defined by a center $\mu_i\in\mathbb{R}^3$, a symmetric positive-definite covariance $\Sigma_i \in \mathbb{R}^{3\times3}$, view-dependent color SH coefficients $b_i$ (or simply RGB $c_i$), and an opacity parameter $o_i$ or $\alpha_i$. The density is given by \[ G_i(x) = (2\pi)^{-3/2} |\Sigma_i|^{-1/2} \exp\left(-\frac{1}{2}(x-\mu_i)^T \Sigma_i^{-1}(x-\mu_i)\right) \] Color is typically modeled via spherical harmonics or per-ray direction mappings. For semantic lifting (e.g., BEV), per-pixel feature vectors $f_i$ and associated uncertainty-aware $\Sigma_i$ are also predicted [2511.17210]. This parameterization is propagated through the entire rendering or segmentation pipeline in a fully differentiable fashion, allowing optimization via gradient descent. Covariance updates are managed by eigendecomposition or affine pushforward under the projection's Jacobian [2411.15355, 2409.04751]. ## 2. Fisheye Camera Models and Geometric Lifting Standard projection schemes (pinhole, perspective) become inaccurate with wide-FOV or fisheye lenses, necessitating explicit modeling of lens distortions. FisheyeGaussianLift relies on calibrated or learnable models such as: - **Equidistant Model:** $r = f\,\theta$ for incidence angle $\theta$; linear mapping of angle to radial distance [2409.04751, 2508.06968]. - **Polynomial/Radial Distortion (e.g., Kannala-Brandt):** $r = f\,\theta + \sum k_i \theta^{2i+1}$, supporting parameterized nonlinearities [2411.15355, 2511.17210]. - **Dual-fisheye and ERP (Equirectangular Projection):** Mapping between panorama and spherical coordinates with per-pixel sampled rays, and learnable angular distortions for lens stitching [2508.20080]. Geometric lifting proceeds by inverting the projection per-pixel via LUTs or analytically, assigning each pixel a calibrated 3D ray and predicted depth, then computing the corresponding 3D Gaussian mean/covariance [2511.17210]: \[ \mu_i = R_i (u_i \mu_{z,i}) + t_i \] \[ \Sigma_i = \sigma_i^2 I_3 + \Delta_d \Delta_d^\top \] This fidelity in geometric unprojection is central to avoiding undistortion artifacts and utilizing the full field of view. ## 3. Differentiable Rendering and Splatting Differentiable rendering under FisheyeGaussianLift involves the projection of 3D Gaussians into 2D (or BEV) under the distortion-aware model, rasterizing their effect as anisotropic elliptical "splats". The core steps are: - **Affine Warping:** Apply rotation and stretching to Gaussian center and covariance to account for ray bending and field-of-view distortions (polar/tangential stretching) [2411.15355]. - **Projection and Splatting:** Calculate 2D location and image-plane covariance via the model Jacobian. Splat using elliptical weighted averages or kernel integration [2409.04751, 2505.24053]. - **Alpha-Blending/Compositing:** Front-to-back (transmittance-aware) compositing for color accumulation [2411.15355], or kernel-weighted summation for features in BEV contexts [2511.17210]. - **Gradient Backpropagation:** All transformation and rendering steps expose gradients (through quaternion operations, stretching, and Jacobian evaluation), enabling end-to-end training or optimization. - **Fast Association (PBF/BEAP):** Efficient mapping of Gaussians to rays in large-FOV settings by bounding frusta in angular domains rather than image-plane AABBs (Particle Bounding Frustum, Bipolar Equiangular Projection) [2505.24053]. ## 4. Multi-View and Multi-Modal Scene Unification FisheyeGaussianLift enables unification of multiple sensor types by sharing a common set of 3D Gaussians across all camera models (pinhole and fisheye), adapting their projection at render time via analytic or learned warps [2411.15355, 2508.20080]. Supervision can be constructed from: - Photometric loss on rendered pixel colors - Similarity terms (e.g., SSIM) for image-level quality - Cross-entropy/IoU-based segmentation loss in BEV applications - Depth and geometric regularization, enforcing consistency between modality-specific views (LiDAR, depth, normal, semantics) - Calibration variable regularization when jointly optimizing lens translation and stitching corrections [2508.20080] Consequently, the learned Gaussian cloud encodes a multi-view, multi-modal scene, supporting both reconstruction and semantic segmentation. ## 5. Practical Implementations and Computational Performance Frameworks such as UniGaussian [2411.15355], Fisheye-GS [2409.04751], FisheyeGaussianLift-BEV [2511.17210], and Seam360GS [2508.20080] implement FisheyeGaussianLift on top of established 3DGS pipelines (often in PyTorch, CUDA). Key implementation points: - Lightweight kernel replacement: Only the projection and its derivatives need adaptation for fisheye, leaving the tile-binning, sorting, and blending intact (Fisheye-GS: +60 lines of CUDA) [2409.04751]. - Efficient handling of peripheral distortion: Affine approximation suffices for small Gaussians, but methods like 3DGEER utilize closed-form integrals for exactness and real-time rates even on extreme FOVs [2505.24053]. - Real-time performance: FlashGS + Fisheye-GS yields ~405 FPS on an A100 GPU at 1752×1168 [2409.04751]; 3DGEER achieves ~251 FPS at state-of-the-art quality [2505.24053]. - Memory/compute: Asymptotic complexity matches pinhole 3DGS, with marginal impact (~10–15% intersects increase) in practice. ## 6. Quantitative Results and Empirical Validation FisheyeGaussianLift approaches consistently outperform undistort-then-3DGS and pinhole-based baselines, especially in peripheral fidelity and semantic robustness. Selected results: | Dataset / Metric | FisheyeGaussianLift Variant | PSNR (dB) | SSIM | LPIPS | |---------------------------------------|----------------------------------|-----------|--------|--------| | KITTI-360 fisheye (UniGaussian) | Warped affine + stretch [2411.15355] | 26.19 | 0.897 | 0.185 | | KITTI-360 pinhole+fisheye (HUGS++) | [2411.15355] | 25.2–26.2 | — | — | | ScanNet++ full FoV (3DGEER) | [2505.24053] | 31.50 | 0.953 | 0.126 | | ScanNet++ peripheral (3DGEER) | [2505.24053] | 28.94 | 0.945 | — | | Parking BEV IoU (FisheyeGaussianLift) | Drivable [2511.17210] | 87.75% | — | — | | Parking BEV IoU (FisheyeGaussianLift) | Vehicle [2511.17210] | 57.26% | — | — | PSNR and SSIM improvements are robust to FOV and initialization method (SfM- vs. depth-based); “sweet spot” FOV for Fisheye-GS is ~160° [2508.06968]. Methods that properly model distortion avoid artifacts such as halo-rings and peripheral clipping. ## 7. Extensions, Limitations, and Outlook FisheyeGaussianLift is extensible to any central projection model by supplying analytic projection and derivatives; frameworks are camera-model-agnostic aside from the initial “lift” mapping [2409.04751]. Dual-fisheye/ERP approaches extend to 360° content, with learnable calibration for lens gap and stitching [2508.20080]. BEV segmentation instantiations explicitly propagate depth and lift-induced uncertainty through anisotropic Gaussian covariances [2511.17210]. Challenges remain in initialization for scenes with strong distortion/sfM failure; depth-based approaches (e.g., UniK3D) offer high-density alternatives. Performance remains real-time under GPU implementations for both 3D reconstruction and segmentation settings. FisheyeGaussianLift tightly integrates probabilistic geometric modeling, differentiable rendering, and explicit handling of distortion, enabling state-of-the-art novel view synthesis, scene reconstruction, and semantic segmentation across wide-angle, multi-view, and omnidirectional camera systems [2511.17210, 2411.15355, 2409.04751, 2505.24053, 2508.20080, 2508.06968]. Source: https://www.emergentmind.com/topics/fisheyegaussianlift