Distortion-Aware BEV Segmentation

Updated 28 November 2025

The paper presents a framework that explicitly models fisheye distortion, enabling sub-pixel accurate BEV maps without preprocessing undistortion.
It leverages novel projection techniques such as anchor-based attention and Gaussian lifting to robustly fuse multi-view image features into a coherent BEV representation.
Temporal self-attention and occlusion-aware pooling enhance real-time scene understanding, significantly outperforming traditional rectification-based methods.

Distortion-aware BEV segmentation frameworks are specialized perception architectures designed to generate accurate Bird's-Eye View (BEV) semantic or height maps directly from surround-view fisheye camera arrays, accounting for the severe geometric distortion intrinsic to wide-angle imaging. Unlike pinhole-based approaches, these systems integrate explicit fisheye projection models and camera calibration into the spatial fusion and semantic heads, thereby maintaining metric accuracy, high fidelity at object boundaries, and full field of view without relying on undistortion preprocessing. Recent advances span transformer-based attention pipelines, differentiable Gaussian-based feature lifting, uncertainty modeling, and occlusion-aware BEV aggregation.

1. Mathematical and Geometric Modeling of Fisheye Distortion

All distortion-aware BEV segmentation methods incorporate a calibrated mapping between 3D world coordinates and 2D fisheye image locations, typically parameterized using the polynomial model of Kannala–Brandt (Yogamani et al., 9 Apr 2024), the unified omnidirectional camera model (Samani et al., 2023), or similar. For a 3D point $\mathbf{P} = [X, Y, Z]^\top$ in camera frame, the standard mapping proceeds as:

Normalize to ray $(x, y) = (X/Z, Y/Z)$ , with radius $r = \sqrt{x^2 + y^2}$ .
The "distortion angle" $\theta = p_1 r + p_2 r^2 + \ldots + p_9 r^9$ .
Projection to image:

$u = c_x + f\theta \cos\varphi , \quad v = c_y + f\theta \sin\varphi$

with $(c_x, c_y)$ the principal point, $f$ the focal length, and $\varphi = \arctan(y/x)$ .

The inverse mapping (from $(u, v)$ to 3D) requires inverting $\theta(r)$ (often via LUTs or fast root-finding (Wu et al., 2022, Sonarghare et al., 21 Nov 2025)). These models capture not only the radial stretch of fisheye images, but also tangential distortions and principal-axis offsets, facilitating sub-pixel-accurate BEV grid alignment.

2. Distortion-aware Feature Lifting and Fusion

The core of distortion-aware BEV segmentation is a feature lifting or projection mechanism that guides learned image-space features into a top-down BEV volume with explicit respect to camera distortion.

Approaches:

Anchor-based attention: F2BEV (Samani et al., 2023) and FishBEV (Li et al., 17 Sep 2025) use transformer queries for BEV cells, projecting fixed anchor heights above each cell via the fisheye model, then attending to corresponding locations in image-space features ("distortion-aware spatial cross-attention" or DA-SCA).
Gaussian lifting: FisheyeGaussianLift (Sonarghare et al., 21 Nov 2025) models each pixel and discretized depth bin as a full 3D Gaussian, parameterized by predicted mean and covariance, projecting these into BEV via analytic marginalization and differentiable splatting. This incorporates per-pixel depth distribution uncertainty and enables sub-grid smoothness.
Learnable BEV pooling: DaF-BEVSeg (Yogamani et al., 9 Apr 2024) generalizes pooling strategies by conditioning BEV fusion on per-camera intrinsics and frustum geometry, using embeddings that capture individual distortion models for robust overlapping-feature aggregation.

In all approaches, learning proceeds on the raw fisheye data, with BEV fusion, cross-attention, and splatting weights explicitly computed with reference to geometric calibration.

3. Temporal and Occlusion Reasoning Mechanisms

Temporal fusion and occlusion awareness are central to robust BEV mapping from multi-view fisheye rigs, due to the dynamic and partially occluded nature of automotive perception scenes.

Temporal self-attention: Frameworks such as F2BEV (Samani et al., 2023) and FishBEV (Li et al., 17 Sep 2025) incorporate transformer-based temporal blocks. FishBEV introduces a "distance-aware temporal self-attention" (D-TSA) mechanism that differentially weights recent and historical features depending on spatial proximity to the vehicle, stabilizing far-field features while emphasizing freshness in the near field.
Occlusion modeling: DaF-BEVSeg (Yogamani et al., 9 Apr 2024) computes a visibility/occlusion probability per BEV cell, using geometric ray-casting from all fisheye cameras. A secondary network head predicts per-cell occupancy probabilities, regularized by binary cross-entropy, and the main semantic loss is masked during training on occluded regions. This approach prevents the hallucination of non-visible content and encourages scene consistency.

4. Supervision Strategies and Loss Functions

Distortion-aware BEV frameworks utilize domain-adapted loss structures to account for discretized or continuous semantic and height outputs, and in some cases, uncertainty modeling.

Semantic segmentation: Weighted cross-entropy over BEV grid classes (Sonarghare et al., 21 Nov 2025, Yogamani et al., 9 Apr 2024, Samani et al., 2023), with class-balancing weights.
Height/vertical discretization: F2BEV (Samani et al., 2023) discretizes height into three bins ("below," "at," "above" car), using categorical cross-entropy or focal loss.
Uncertainty and regularization: GaussianLift (Sonarghare et al., 21 Nov 2025) and FishBEV (Li et al., 17 Sep 2025) both incorporate learned or inferred per-pixel depth variances. FishBEV penalizes excessive variance or collapse using a KL-divergence term; GaussianLift ties the splatting kernel in BEV directly to the covariance.
Occlusion loss: DaF-BEVSeg (Yogamani et al., 9 Apr 2024) uses binary cross-entropy for occupancy with an explicit $\lambda$ -weighted sum in the total loss.

5. Experimental Results, Ablations, and Practical Observations

Distortion-aware BEV frameworks consistently outperform undistortion-then-BEV or pinhole-centric baselines on both synthetic and real (where available) evaluation sets.

Framework	Key Result Metrics (IoU/mIoU/FW-IoU)	Distortion Handling Mechanism
F2BEV (Samani et al., 2023)	Height FW-IoU 86.7%, Segm FW-IoU 86.2%, UP $\sim$ 8 pts	DA-SCA attention
FisheyeGaussianLift (Sonarghare et al., 21 Nov 2025)	Drivable IoU 87.75%, Vehicle IoU 57.26%	3D Gaussian splatting
FishBEV (Li et al., 17 Sep 2025)	(Synwoodscapes) mIoU up to 64.2%	Uncertainty-aware attention
DaF-BEVSeg (Yogamani et al., 9 Apr 2024)	(Cognata) mIoU up to 0.796 (Easy), 0.690 (Medium)	Intrinsics-aware BEV pooling, occlusion
FPNet (Wu et al., 2022)	AP $_\mathrm{BEV}$ = 80.74 (FPD, detection-to-BEV)	Fisheye in backbone

Ablation studies show that removing explicitly distortion-aware modules results in significant drops in IoU (typically 3–7 points), especially for near-field objects and under occlusion (Sonarghare et al., 21 Nov 2025, Samani et al., 2023, Yogamani et al., 9 Apr 2024). Incorporating learned uncertainty (covariance) further sharpens BEV segmentation masks and improves IoU.

Performance gains over baselines are robust to camera model choices, as these frameworks generalize across various fisheye and omnidirectional parameterizations without requiring re-training.

6. Deployment, Dataset Characteristics, and Generalization

All recent distortion-aware BEV segmentation frameworks are compatible with real-time or near-real-time inference on automotive-grade or data center hardware (Wu et al., 2022), obviate the need for costly undistortion, and preserve the wide angular field-of-view of fisheye arrays.

Datasets: Synthetic datasets (FB-SSEM (Samani et al., 2023), Cognata (Yogamani et al., 9 Apr 2024), proprietary sets (Sonarghare et al., 21 Nov 2025)) remain predominant, featuring synchronized multi-fisheye captures, simulated or measured ego-motion, and per-cell semantic or height labels. FPNet (Wu et al., 2022) additionally exposes the Fisheye Parking Dataset (FPD), facilitating cross-domain transfer.
Augmentation and style transfer: Synthetic–real domain gap is mitigated by style transfer (F2BEV, Gatys et al.) or extensive data augmentation pipelines (Sonarghare et al., 21 Nov 2025).
Flexibility: Frameworks are extensible to any central projection system (pinhole, double sphere, UCM, Kannala–Brandt) with parameter adjustment alone (Yogamani et al., 9 Apr 2024).
Field of view: Full fisheye lens FOV is leveraged, maximizing coverage and minimizing blind spots, a substantial advantage over rectified pinhole input.

A plausible implication is that eliminating the pre-processing undistortion stage not only improves accuracy but also simplifies calibration workflows, supports hardware diversity, and reduces latency.

7. Trends and Comparative Analysis

In summary, distortion-aware BEV segmentation frameworks now constitute the dominant approach for surround-view, near-field scene understanding in autonomous driving when using fisheye cameras. The principal innovations include:

Tightly coupled geometric modeling of distortion in feature lifting, BEV projection, and attention mechanisms;
Explicit modeling—and, in some cases, supervision—of depth and occlusion uncertainty for robust aggregation;
Transformer-based fusion pipelines enabling multi-view, multi-scale, temporal self-attention that treat distorted image-space as a first-class computational domain, rather than aiming to compensate distortion via preprocessing.

Quantitative and qualitative results across multiple works confirm the superiority of distortion-aware methods over both naive BEV segmentation and standard rectification-based pipelines, with improvements most pronounced in object boundary localization, occlusion handling, and overall segmentation IoU (Samani et al., 2023, Sonarghare et al., 21 Nov 2025, Li et al., 17 Sep 2025, Yogamani et al., 9 Apr 2024).