FishBEV: Fisheye BEV Segmentation

• FishBEV is a BEV segmentation framework designed for fisheye cameras, addressing severe non-linear distortion, ambiguous multi-view correspondences, and unstable temporal dynamics.
• It employs a DINOv2-based multi-scale extraction (DRME) along with uncertainty-aware spatial and distance-aware temporal attention to effectively fuse multi-view fisheye data.
• Experimental results on SynWoodscapes show mIoU improvements from 54.39% to 64.22%, though real-world adaptation and computational costs remain challenging.

FishBEV denotes a line of fisheye-aware Bird’s-Eye View perception research centered on surround-view wide-angle cameras. In its primary usage, it refers to the BEV segmentation framework introduced in "FishBEV: Distortion-Resilient Bird's Eye View Segmentation with Surround-View Fisheye Cameras," which targets BEV segmentation under severe geometric distortion, ambiguous multi-view correspondences, and unstable temporal dynamics (Li et al., 17 Sep 2025). The name is also used informally in discussion of the FishBEVOD benchmark, which addresses multi-view BEV 3D object detection with mixed pinhole and fisheye cameras (Liu et al., 29 Mar 2026). Within the segmentation literature, FishBEV is characterized by a DINOv2-based multi-scale backbone, uncertainty-aware cross-view fusion, and distance-aware temporal aggregation tailored to fisheye geometry.

1. Nomenclature and problem domain

FishBEV addresses BEV segmentation from surround-view fisheye imagery rather than pinhole imagery. The underlying motivation is that existing BEV methods had achieved strong results with pinhole cameras, but extending them to fisheye cameras is non-trivial because the projection is highly non-linear, cross-view correspondences are ambiguous, and temporal fusion becomes unstable under distortion (Li et al., 17 Sep 2025).

The framework is evaluated on SynWoodscapes, a simulated surround-view fisheye dataset with 4 cameras of $190^\circ$ field of view and image resolution $1280\times 966$. BEV ground truth is generated at $1024\times 1024$, and 25 original classes are remapped to 6 classes, with void excluded in mIoU computation. The evaluation uses per-class IoU,

$\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c},$

and

$$\mathrm{mIoU}=\frac1C\sum_{c=1}^C \mathrm{IoU}_c,\quad C=6.$$

These design choices place FishBEV within the semantic BEV mapping regime rather than 3D detection or occupancy prediction (Li et al., 17 Sep 2025).

A potential source of confusion is terminological overlap. The 2026 mixed-camera 3D detection benchmark is formally named FishBEVOD, although its summary explicitly labels the benchmark “FishBEV.” A plausible implication is that the term now functions as a broader label for fisheye-aware BEV perception, but the 2025 segmentation framework remains the clearest primary referent in the literature (Liu et al., 29 Mar 2026).

2. Distortion model and input representation

FishBEV models fisheye image formation through the SynWoodscapes camera generation pipeline. Each 3D ray with incident angle $\theta$ to the optical axis is mapped to image radius by the polynomial

$$r(\theta)=a_1\theta+a_2\theta^2+a_3\theta^3+a_4\theta^4.$$

The coefficients $\{a_i\}$ control radial distortion; after computing $r$, cubemap projection and ray tracing are used to form the final fisheye image (Li et al., 17 Sep 2025).

The image input tensor is

$X\in\mathbb R^{B\times C\times H\times W},$

and the images are first resized, for example to $1280\times 966$0, then partitioned into non-overlapping $1280\times 966$1 patches. These patches are passed to a pretrained DINOv2 transformer, which produces layer-wise token features

$1280\times 966$2

This formulation is significant because FishBEV does not rely on perspective rectification as its primary representation strategy. Instead, it retains fisheye geometry through the feature-extraction and BEV-encoding stages. This suggests a design commitment to distortion-aware modeling rather than distortion removal (Li et al., 17 Sep 2025).

3. Distortion-Resilient Multi-scale Extraction

The Distortion-Resilient Multi-scale Extraction module, or DRME, is the backbone component of FishBEV. It uses DINOv2 in small, base, or large variants pretrained on large-scale data, and extracts the last four transformer layers $1280\times 966$3 (Li et al., 17 Sep 2025).

The token features, excluding the CLS token, are reshaped to spatial feature maps: $1280\times 966$4 They are then channel-aligned through $1280\times 966$5 convolution and interpolation to a common resolution,

$1280\times 966$6

followed by FPN fusion: $1280\times 966$7

The stated rationale is scale consistency preservation. Deep layers contribute global semantics, shallow layers preserve local detail, and interpolation in the FPN aligns feature map sizes so that a point in real space projects consistently across scales. In the paper’s ablation, the baseline without fisheye-specific modules attains 54.39 mIoU, while adding DRME raises performance to 59.65 mIoU, indicating that multi-scale fisheye feature extraction is not merely auxiliary but a major contributor to overall accuracy (Li et al., 17 Sep 2025).

4. Uncertainty-aware spatial fusion and distance-aware temporal aggregation

FishBEV’s BEV encoder combines two specialized attention modules: Uncertainty-aware Spatial Cross-Attention (U-SCA) and Distance-aware Temporal Self-Attention (D-TSA). The encoder maintains $1280\times 966$8 BEV queries at positions $1280\times 966$9. For each camera $1024\times 1024$0, a BEV query is projected through the fisheye intrinsics, extrinsics, and distortion model to a 2D feature-map location $1024\times 1024$1. Deformable cross-attention then samples $1024\times 1024$2 offsets per query: $1024\times 1024$3 where $1024\times 1024$4 is the feature map of camera $1024\times 1024$5 (Li et al., 17 Sep 2025).

U-SCA augments this with uncertainty estimation. Two MLP heads predict

$1024\times 1024$6

with

$1024\times 1024$7

Monte Carlo samples are generated through

$1024\times 1024$8

and camera contributions are fused using precision weights

$1024\times 1024$9

together with a visibility mask $\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c},$0. Uncertainty is regularized by KL divergence to a prior $\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c},$1 with $\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c},$2. The empirical role of this module is to weight multi-view evidence by confidence rather than treating all views as equally reliable (Li et al., 17 Sep 2025).

D-TSA addresses temporal instability. FishBEV keeps a short buffer

$\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c},$3

where each $\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c},$4. For query $\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c},$5 at location $\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c},$6, offsets are sampled in both frames. A normalized distance to the BEV center $\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c},$7 is defined by

$\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c},$8

and a sigmoid gate

$\mathrm{IoU}_c=\frac{TP_c}{TP_c+FP_c+FN_c},$9

controls near-far emphasis. Near-field queries satisfy $$\mathrm{mIoU}=\frac1C\sum_{c=1}^C \mathrm{IoU}_c,\quad C=6.$$0, implying $$\mathrm{mIoU}=\frac1C\sum_{c=1}^C \mathrm{IoU}_c,\quad C=6.$$1; far-field queries imply $$\mathrm{mIoU}=\frac1C\sum_{c=1}^C \mathrm{IoU}_c,\quad C=6.$$2. Temporal weights are then modulated as

$$\mathrm{mIoU}=\frac1C\sum_{c=1}^C \mathrm{IoU}_c,\quad C=6.$$3

This gives current-frame detail greater influence in the near field and shifts weight toward temporal context in the far field. The ablation sequence $$\mathrm{mIoU}=\frac1C\sum_{c=1}^C \mathrm{IoU}_c,\quad C=6.$$4 for baseline, $$\mathrm{mIoU}=\frac1C\sum_{c=1}^C \mathrm{IoU}_c,\quad C=6.$$5DRME, $$\mathrm{mIoU}=\frac1C\sum_{c=1}^C \mathrm{IoU}_c,\quad C=6.$$6DRME$$\mathrm{mIoU}=\frac1C\sum_{c=1}^C \mathrm{IoU}_c,\quad C=6.$$7U-SCA, and $$\mathrm{mIoU}=\frac1C\sum_{c=1}^C \mathrm{IoU}_c,\quad C=6.$$8DRME$$\mathrm{mIoU}=\frac1C\sum_{c=1}^C \mathrm{IoU}_c,\quad C=6.$$9U-SCA$\theta$0D-TSA suggests that FishBEV’s gains are distributed across feature extraction, cross-view uncertainty handling, and temporal modeling rather than concentrated in a single module (Li et al., 17 Sep 2025).

5. End-to-end pipeline, optimization, and empirical performance

The integrated FishBEV pipeline begins with four surround-view fisheye images per timestep. DRME produces multi-scale fisheye features $\theta$1. A fisheye BEV encoder then stacks $\theta$2 blocks, each containing U-SCA, D-TSA, and an FFN implemented as a two-layer MLP with ReLU and dropout. A BEV decoder, denoted MaskHead, produces per-class BEV masks

$\theta$3

followed by upsampling and refinement to obtain the final BEV segmentation (Li et al., 17 Sep 2025).

Training uses focal loss for segmentation together with KL regularization: $\theta$4 Optimization uses AdamW with initial learning rate $\theta$5, decay $\theta$6 per epoch, 50 epochs, and batch size 2/GPU. Depending on the DINOv2 model size, 60–80% of layers are frozen (Li et al., 17 Sep 2025).

On SynWoodscapes, FishBEV is reported against several baselines:

Method	mIoU (%)
IPM	40.89
BEVFormer	49.37
SimpleBEV	49.49
F2BEV	53.39
FishBEV (D-Small)	55.41
FishBEV (D-Base)	62.06
FishBEV (D-Large)	64.22

The module-level ablation on FishBEV (D-Large) is likewise explicit:

Configuration	mIoU (%)
Baseline (no fisheye modules)	54.39
+ DRME	59.65
+ DRME + U-SCA	62.07
+ DRME + U-SCA + D-TSA	64.22

The paper further states that qualitative results show sharper BEV masks, especially at near-far boundaries and small vehicles. This is consistent with the intended roles of DRME, uncertainty-weighted fusion, and distance-aware temporal aggregation (Li et al., 17 Sep 2025).

6. Relation to adjacent fisheye BEV research

FishBEV occupies a broader research trajectory in which fisheye BEV perception is treated as a first-class geometric problem rather than a pinhole approximation. An early reference point is F2BEV, described as the first end-to-end method that lifts surround-view fisheye image features directly into a BEV representation. F2BEV uses a distortion-aware spatial cross-attention module built on the unified omnidirectional model with radial-tangential distortion, and on the synthetic FB-SSEM dataset it reports a best height-map mean IoU of 0.9106 and a best semantic segmentation frequency-weighted IoU of 0.8624 (Samani et al., 2023).

A second line is FisheyeGaussianLift, which addresses BEV semantic segmentation through calibrated geometric unprojection, per-pixel depth-distribution estimation, anisotropic Gaussian uncertainty, and differentiable splatting into BEV. It uses a unified fisheye projection model, a lookup table of direction vectors for backprojection, and a weighted cross-entropy segmentation loss without auxiliary depth or uncertainty losses. On a proprietary 4-camera real-world fisheye dataset, it reports IoU scores of 87.75% for drivable regions and 57.26% for vehicles (Sonarghare et al., 21 Nov 2025).

The FishBEVOD benchmark extends the discussion from segmentation to 3D object detection with mixed pinhole and fisheye cameras. It converts KITTI-360 to nuScenes format, studies rectification, distortion-aware view transformation modules via the MEI model of Mei and Rives, and polar BEV representations, and evaluates BEVFormer, BEVDet, and PETR. Reported results include, for example, PETR at 0.266/0.280 mAP/NDS in the baseline, 0.280/0.288 with DA$\theta$7Polar, and 0.295/0.337 with rectification, while the authors conclude that projection-free architectures are inherently more robust and effective against fisheye distortion than other VTMs (Liu et al., 29 Mar 2026).

Taken together, these works indicate three major design families: distortion-aware attention over projected fisheye features, uncertainty-aware geometric lifting, and mixed-camera adaptation with explicit camera models. FishBEV belongs primarily to the first family, but incorporates uncertainty and temporal mechanisms that partially overlap with concerns emphasized by the other two.

7. Limitations, misconceptions, and open directions

A recurring misconception in fisheye BEV perception is that fisheye images can be treated as ordinary pinhole views after undistortion. The literature summarized here does not support that simplification. F2BEV argues that undistortion loses field of view or introduces severe resampling artifacts, and FishBEVOD states that rectification is easy to deploy but sacrifices FoV and near-field resolution, especially when fisheye is the primary sensing modality (Samani et al., 2023). FishBEV’s architecture, which keeps distortion-aware processing within the backbone and encoder, is aligned with that critique (Li et al., 17 Sep 2025).

The main limitation explicitly stated for FishBEV is data realism. Its experiments rely on SynWoodscapes, and the paper notes that real-world fisheye sensors may exhibit different lens artifacts and lighting. It also identifies computational cost as a constraint: DINOv2-Large, deformable attentions, and Monte Carlo sampling add overhead. In addition, far-field segmentation remains relatively noisy because of low effective resolution and heavy distortion (Li et al., 17 Sep 2025).

The open directions identified in the same source are domain adaptation to real fisheye cameras, lightweight uncertainty sampling, and extension to 3D detection and occupancy prediction. A plausible implication is that later detection-oriented work such as FishBEVOD can be read not as a replacement for FishBEV, but as an adjacent response to one of those open directions—namely, extending fisheye-aware BEV reasoning beyond segmentation into real-data 3D detection under mixed camera configurations (Liu et al., 29 Mar 2026).