Distortion-Resilient Multi-scale Extraction (DRME)

• DRME is a DINOv2-based multi-scale fisheye backbone that extracts robust, semantically consistent features from distortion-prone fisheye images.
• It combines multi-layer extraction from the last four transformer layers with FPN fusion to preserve spatial details and scale consistency.
• Empirical results show that DRME improves mIoU by 5.26 points in BEV segmentation, enhancing detection of roads, sidewalks, vegetation, and vehicles.

Searching arXiv for papers directly relevant to Distortion-Resilient Multi-scale Extraction and closely related multi-scale distortion-robust feature extraction. Distortion-Resilient Multi-scale Extraction (DRME) is the image-backbone component introduced in FishBEV for surround-view fisheye Bird’s Eye View (BEV) segmentation. In that formulation, DRME converts surround-view fisheye images into robust multi-scale features before BEV lifting and fusion. Its stated purpose is to learn features that remain semantically strong and spatially informative under severe fisheye distortion while preserving scale consistency, so that downstream cross-view and temporal BEV modules operate on less brittle representations (Li et al., 17 Sep 2025).

1. Origin and problem setting

DRME was proposed in the context of fisheye-camera BEV segmentation, where three coupled difficulties are emphasized. The first is severe geometric distortion, including radial distortion, nonlinear projection, non-uniform magnification, and anisotropy, especially near image peripheries. The second is scale inconsistency: in fisheye imagery, apparent object scale is not only depth-dependent but also strongly position-dependent. The third is a multi-scale extraction problem: BEV segmentation simultaneously requires broad semantic context for classes such as roads and sidewalks and fine local detail for vehicles and thin boundaries. FishBEV argues that conventional backbones such as ResNet and VoVNet, and by implication standard CNN/FPN pipelines designed around pinhole-image statistics, “fail to capture distortion-resilient and semantically consistent features from fisheye images” (Li et al., 17 Sep 2025).

Within that framing, DRME is not an auxiliary refinement module but the front-end representation substrate of the full system. Its output precedes the Fisheye BEV Encoder and therefore conditions subsequent spatial cross-view interaction and temporal aggregation. The paper’s formulation makes the backbone problem representational rather than purely geometric: the aim is not explicit undistortion, but feature extraction that remains semantically robust despite projection-induced deformation.

A recurrent misconception is to treat DRME as a new fisheye-specific convolutional operator family. The FishBEV description does not support that reading. DRME is introduced as a modified backbone built on pre-trained DINOv2 with a multi-scale feature pyramid network design, not as a backbone centered on custom fisheye convolutions or a dedicated analytical distortion model. Its “distortion-resilient” character is attributed to large-scale self-supervised pretraining and multi-layer fusion rather than to explicit calibration terms (Li et al., 17 Sep 2025).

2. Architectural composition

Architecturally, DRME is a hybrid backbone: a pre-trained DINOv2 Vision Transformer followed by FPN-style multi-scale fusion. The processing sequence is explicit. Surround-view fisheye images are patchified, fed into pre-trained DINOv2, hidden representations are extracted from the last four transformer layers, token sequences are reshaped into 2D feature maps, those maps are projected to a uniform channel space using convolutions, resized through interpolation, fused with an FPN, and emitted as the final representation $F_{\mathrm{DRME}}$ (Li et al., 17 Sep 2025).

The input image tensor is defined as

$X \in \mathbb{R}^{B\times C\times H\times W},$

with $B$ batch size, $C$ channels, and $H,W$ spatial dimensions. Patch embedding partitions each fisheye image into non-overlapping patches of size $P\times P$, linearly projects them into token embeddings, and applies DINOv2: $F = \mathrm{DINOv2}(\mathrm{PatchEmbed}(X)).$ The hidden-state set is

$$F=\left \{F_{i} \,|\,i \in [1,\cdots,L],\; F_{i}\in \mathbb{R}^{B\times (N+1) \times D} \right \},$$

where $N=\frac{H\cdot W}{P^2}$ is the number of patches and $D$ is the hidden dimension.

DRME does not use only the final transformer output. Instead, it selects the last four layers,

$X \in \mathbb{R}^{B\times C\times H\times W},$0

on the stated rationale that they jointly provide strong global semantic consistency from deep layers while preserving local details through multi-layer extraction. Strictly speaking, the multi-scale behavior is therefore not native hierarchical downsampling of the CNN type. All ViT layers have the same token resolution before FPN processing, and the scale pyramid is constructed afterward through convolutional projection and interpolation. This detail is central to the module’s identity: DRME is a DINOv2-FPN fisheye backbone rather than a hierarchical pyramid transformer in the usual sense (Li et al., 17 Sep 2025).

3. Mathematical formulation and operational semantics

For each selected layer, DRME removes the class token and reshapes the patch tokens into a spatial map: $X \in \mathbb{R}^{B\times C\times H\times W},$1 The reshaped tensor satisfies

$X \in \mathbb{R}^{B\times C\times H\times W},$2

This restores 2D spatial organization so that convolutional channel mapping and feature-pyramid fusion become possible.

Each selected map is then channel-adjusted by convolution and resized by interpolation: $X \in \mathbb{R}^{B\times C\times H\times W},$3 The final DRME output is produced by FPN fusion: $X \in \mathbb{R}^{B\times C\times H\times W},$4 The paper’s notation contains a typesetting issue, but this corrected reading is the intended one. The interpretation is straightforward: selected transformer features are transformed into a normalized multi-scale set and fused into a representation suited for downstream BEV interaction (Li et al., 17 Sep 2025).

The explicit operations attributed to DRME are limited and important to enumerate precisely: DINOv2 backbone, patch embedding, multi-layer extraction from the final four transformer layers, reshaping tokens into 2D maps, convolution for channel mapping, bilinear upsampling or downsampling, and FPN fusion. Equally important are the operations that are not specified. DRME itself does not include a special fisheye convolution, deformable convolution, distortion-aware normalization, an explicit analytical distortion calibration term, or a dedicated scale-consistency loss. The backbone’s resilience is therefore representational rather than hard-coded.

This operational boundary distinguishes DRME from other distortion-robust designs in the literature. SphereDiffusion, for example, addresses spherical distortion using semantic conditioning and a deformable distortion-aware block, but it does not introduce an explicit multi-scale extraction module of the DRME type; its multi-scale behavior largely comes from inherited ControlNet feature injection (Wu et al., 2024). FishBEV’s DRME is, by contrast, explicitly a multi-layer multi-scale backbone, but without custom distortion-aware operators.

4. Functional role inside FishBEV

Within the full FishBEV pipeline, DRME sits at the front: surround-view fisheye images $X \in \mathbb{R}^{B\times C\times H\times W},$5 DRME $X \in \mathbb{R}^{B\times C\times H\times W},$6 Fisheye BEV Encoder (U-SCA + D-TSA + FFN) $X \in \mathbb{R}^{B\times C\times H\times W},$7 BEV decoder (Li et al., 17 Sep 2025).

Its first immediate consumer is the Uncertainty-aware Spatial Cross-Attention (U-SCA) module. U-SCA projects BEV queries into each fisheye camera feature map and samples image features by deformable attention. The feature maps sampled there are exactly the multi-scale fisheye representations produced by DRME. The paper’s interpretation is that DRME improves cross-view alignment indirectly by providing semantically richer and more stable per-view features before any projection, and by preserving local details and semantic consistency across layers so that projected BEV-query sampling becomes less ambiguous.

The Distance-aware Temporal Self-Attention (D-TSA) module operates in BEV space after spatial fusion rather than directly on DRME outputs. Even so, DRME affects temporal modeling indirectly because better image features yield a better initial BEV representation after U-SCA. That improved BEV state is then temporally aggregated by D-TSA. A plausible implication is that DRME is necessary but insufficient: it stabilizes the image representation stage, but does not by itself resolve ambiguous multi-view correspondences or temporal instability. FishBEV’s modular structure strongly implies this division of labor, although the paper does not present it as an explicit failure theorem (Li et al., 17 Sep 2025).

This decomposition also clarifies what DRME does not claim. It is not the mechanism for uncertainty estimation, not the temporal coherence operator, and not the BEV-space fusion module. Its scope is upstream: robust multi-scale fisheye feature extraction with preserved semantic consistency.

5. Implementation profile and empirical support

FishBEV provides several concrete implementation details specific to DRME. The original SynWoodscapes images are $X \in \mathbb{R}^{B\times C\times H\times W},$8, while the input surround-view fisheye images are resized to $X \in \mathbb{R}^{B\times C\times H\times W},$9. The system uses surround-view fisheye images from four cameras in the overview discussion, each with about $B$0 field of view in SynWoodscapes, and DRME is used symmetrically as the common backbone for all views. Three DINOv2-FPN instantiations are reported: DINOv2-Small-FPN (D-S), DINOv2-Base-FPN (D-B), and DINOv2-Large-FPN (D-L) (Li et al., 17 Sep 2025).

Backbone	Trainable parameters	Freeze ratio / selected layers
ResNet34-FPN baseline	23.89M	—
DINOv2-Small-FPN	12.60M	60%, $B$1
DINOv2-Base-FPN	33.21M	70%, $B$2
DINOv2-Large-FPN	68.59M	80%, $B$3

Training context affecting DRME includes AdamW, an initial learning rate of $B$4, learning-rate decay of $B$5 per epoch, 50 epochs, and batch size 2 per GPU on two A6000 GPUs. Training samples use sequences of three consecutive frames. Several implementation details remain absent: the patch size $B$6, the common FPN channel dimension, exact output pyramid scales, exact convolution kernel sizes, exact number of FPN levels, and FLOPs or inference latency of DRME alone are not specified and cannot be reconstructed faithfully from the paper.

The clearest isolated empirical evidence comes from the ablation study. Starting from the baseline, adding DRME raises mIoU from 54.39 to 59.65, a gain of $B$7 mIoU. Class-wise gains are reported as Road $B$8, Sidewalk $B$9, Vegetation $C$0, Four-wheeler vehicle $C$1, and Ego-vehicle $C$2. The paper explicitly interprets these gains as demonstrating “the importance of distortion-resilient multi-scale extraction.” In full-model comparisons, FishBEV with DRME also surpasses F2BEV, with reported mIoU values of 53.39 for F2BEV, 55.41 for FishBEV (D-S), 62.06 for FishBEV (D-B), and 64.22 for FishBEV (D-L) (Li et al., 17 Sep 2025).

The class trends are especially revealing. Large improvements on sidewalk, vegetation, and four-wheeler vehicle suggest that the backbone is particularly helpful for elongated drivable structures, semantically difficult background classes, and smaller or scale-sensitive objects. The paper does not provide distance-stratified metrics specifically for DRME, so no direct near/far interpretation should be attached to the backbone alone.

6. Relation to adjacent research programs

Although the name “Distortion-Resilient Multi-scale Extraction” is specific to FishBEV, several neighboring works instantiate closely related technical ideas. This suggests that DRME can also be read as a broader design pattern rather than only a single named module (Li et al., 17 Sep 2025).

In accelerated MRI reconstruction, “Multi-scale MRI reconstruction via dilated ensemble networks” treats aliasing artifacts as structured and non-local distortions and responds by enlarging receptive fields through parallel dilated branches without introducing a pooling bottleneck. The core principle is resolution-preserving multi-scale extraction under measurement constraints, with large-context branches addressing structured artifacts and small-context branches protecting fine detail (Ma et al., 2023). In visible-light OCT retinal segmentation, BreakNet addresses blood-vessel-shadow-induced boundary discontinuities through dual patch scales, parallel convolutional and transformer paths, and hierarchical decoding, making robustness depend on the fusion of local evidence and long-range continuity priors rather than local edge cues alone (Ganjee et al., 2024). In full-reference image quality assessment, “Deep Multi-Scale Features Learning for Distorted Image Quality Assessment” combines distortion-explicit residual maps with spatial pyramid pooling and feature-pyramid aggregation, again coupling explicit distortion cues to multi-scale representation learning (Zhou et al., 2020).

These analogues clarify both the scope and the limits of DRME. A common thread across them is that distortion resilience usually emerges when multi-scale processing is paired with a representation that preserves or restores the information most vulnerable to the distortion at hand: phase information in MRI, global anatomical continuity in retinal OCT, residual distortion maps in IQA, or semantically invariant transformer features in fisheye BEV. Conversely, the mechanisms differ markedly by domain. FishBEV’s DRME does not use dilated same-resolution branches as in MRI, dual local/global patch-scale paths as in BreakNet, or residual-map guidance as in IQA. Its distinctive choice is to anchor distortion resilience in DINOv2 pretraining plus FPN fusion over the last four transformer layers.

A second misconception follows from that comparison. DRME is not a universal solution to all fisheye failures. FishBEV itself indicates that DRME handles robust feature extraction, U-SCA handles cross-view uncertainty, and D-TSA handles temporal near/far imbalance. The broader literature reinforces the same point: distortion resilience is typically modular. SphereDiffusion adds deformable latent adaptation and spherical rotation-aware training for panorama generation (Wu et al., 2024), while other fields use structured redundancy for local synchronization under geometric warp (Ma et al., 2020) or scale-space cleaning plus iterative deblending under cluttered astronomical backgrounds (Men'shchikov et al., 2012). In that sense, DRME is best understood as one member of a wider class of distortion-aware multi-scale extractors whose exact form remains task-dependent.

7. Interpretive boundaries and significance

The strongest concise characterization of DRME is therefore: a DINOv2-based multi-scale fisheye backbone with FPN fusion, introduced to preserve semantic consistency and local detail under fisheye distortion before BEV lifting. Its mathematical definition is compact,

$C$3

$C$4

but its practical role is broader: it regularizes the entire downstream BEV pipeline by making the initial per-view image features less sensitive to nonlinear projection and scale inconsistency (Li et al., 17 Sep 2025).

Its significance lies less in inventing a new primitive than in identifying a precise representational compromise for fisheye BEV segmentation. The module preserves the semantic strength of a large-scale self-supervised ViT, constructs a multi-scale representation from the last four layers rather than from native CNN downsampling, and delegates remaining uncertainty and temporal instability to dedicated downstream modules. This suggests a general principle: when distortion mainly disrupts semantic consistency rather than only geometry, a distortion-resilient extractor may be more effectively built by combining strong pretrained semantics with carefully constructed multi-scale fusion than by introducing a narrowly specialized low-level operator.

Within FishBEV, that principle is supported directly by the DRME ablation. Beyond FishBEV, the surrounding literature suggests that the same general design logic recurs across modalities whenever the dominant corruption is structured, spatially nonuniform, and not adequately handled by single-scale or purely local processing.