FastViDAR: Real-Time 360° Depth Estimation

• FastViDAR is a real-time omnidirectional depth estimation framework that fuses four fisheye inputs into a unified equirectangular projection for complete 360° coverage.
• Its innovative Alternative Hierarchical Attention mechanism efficiently mixes local and global features, significantly reducing computational overhead for embedded systems.
• The method achieves competitive accuracy and speed, outperforming traditional LiDAR and stereo approaches with fast inference on embedded robotics hardware.

Searching arXiv for FastViDAR and related ViDAR papers to ground the article with current references. FastViDAR is a real-time omnidirectional depth estimation framework for rigs with multiple fisheye cameras. It takes four fisheye camera inputs and produces a full $360^\circ$ depth map along with per-camera depth, fusion depth, and confidence estimates. The method is designed for dense depth estimation over the full spherical field of view from a compact multi-camera setup, while remaining fast enough for embedded robotics hardware. Its core design combines a common equirectangular projection representation, an Alternative Hierarchical Attention mechanism for cross-view feature mixing with reduced overhead, and an ERP fusion procedure that reprojects per-view predictions into a shared spherical coordinate system (Zhao et al., 28 Sep 2025).

1. Definition and scope

FastViDAR addresses the problem of accurate, consistent, real-time $360^\circ$ depth estimation from a multi-fisheye camera system. The paper positions this against several alternatives. Traditional LiDAR is described as accurate but costly and power-hungry. Stereo and multi-view methods for omnidirectional cameras often rely on spherical cost volumes or plane sweeping, which are computationally heavy and difficult to deploy in real time. Monocular omnidirectional depth methods can generalize across camera types, but they are fundamentally scale ambiguous and cannot fully exploit multi-view constraints (Zhao et al., 28 Sep 2025).

The framework assumes $S$ fisheye images as input, typically $S=4$ in experiments, and supports arbitrary camera intrinsics and arbitrary camera poses and orientations, as well as any central fisheye model, such as KB, equidistant, equisolid, OCamCalib, Double Sphere, and unified model. The output comprises three prediction families: per-camera depth maps, a fusion depth map over the shared ERP space, and per-pixel confidence estimates used in the loss and available for fusion variants (Zhao et al., 28 Sep 2025).

The emphasis on four fisheye cameras reflects a concrete systems assumption: four ultra-wide fisheye cameras can cover the entire sphere with overlap, yielding complete $360^\circ$ coverage, better geometric constraints than monocular input, and a realistic, low-cost alternative to spinning LiDAR. The central challenge, as formulated in the paper, is to exploit these advantages without the overhead of full attention or cost-volume methods (Zhao et al., 28 Sep 2025).

2. Equirectangular projection as the common representation

A foundational design choice in FastViDAR is the conversion of all fisheye images into a common equirectangular projection lattice. For a unit ray $\mathbf d=(d_x,d_y,d_z)$, the ERP coordinates are defined as

$$\lambda=\operatorname{atan2}(d_x,d_z), \qquad \phi=\arcsin(d_y)$$

and

$$x=\Big(\tfrac{\lambda}{2\pi}+\tfrac{1}{2}\Big)W, \qquad y=\Big(\tfrac{1}{2}-\tfrac{\phi}{\pi}\Big)H.$$

The reverse mapping is also specified: from ERP $(x,y)$, recover $(\lambda,\phi)$, form the ray

$360^\circ$0

and project through the camera model if needed (Zhao et al., 28 Sep 2025).

This representation is explicitly camera-agnostic. Once the fisheye image is projected to ERP, the network operates on a stable spherical grid rather than lens-specific image distortions. The paper further motivates ERP by noting that it keeps the full spherical field of view, unlike pinhole views that crop away large portions of the scene (Zhao et al., 28 Sep 2025).

A plausible implication is that ERP serves two distinct roles simultaneously: it standardizes heterogeneous fisheye optics into a shared coordinate system and provides the geometric substrate for later multi-view fusion. This interpretation is consistent with the paper’s repeated use of ERP both before feature extraction and after per-view depth prediction.

3. Alternative Hierarchical Attention

The central architectural contribution of FastViDAR is Alternative Hierarchical Attention, abbreviated AHA. It is introduced as an efficient fusion mechanism that mixes features across views through separate intra-frame and inter-frame windowed self-attention, thereby achieving cross-view feature mixing with reduced overhead (Zhao et al., 28 Sep 2025).

The motivation is stated in computational terms. If all tokens from $360^\circ$1 frames are concatenated, with each frame containing $360^\circ$2 tokens, then full self-attention has cost

$360^\circ$3

in both compute and memory, which the paper regards as too expensive for embedded inference (Zhao et al., 28 Sep 2025).

FastViDAR instead uses a hierarchical, alternating attention structure composed of three levels:

1. window attention over local tokens inside each frame;
2. frame attention over summary tokens within each frame;
3. global attention over summary tokens across all frames (Zhao et al., 28 Sep 2025).

The stage-2 feature map is written as

$360^\circ$4

with batch size $360^\circ$5, frames $360^\circ$6, channels $360^\circ$7, and spatial size $360^\circ$8. The image is partitioned into non-overlapping windows of size $360^\circ$9. After adaptive padding, each frame has $S$0 windows and $S$1 tokens per window. Two token sets are formed:

$S$2

and

$S$3

where each summary token is average pooled over its window:

$S$4

Window attention is applied independently inside each local window; frame attention mixes summary tokens from the same frame; and global attention is applied after concatenating the frame summaries from all views. A learnable frame or camera embedding $S$5 is added so that the model knows which view the summaries come from (Zhao et al., 28 Sep 2025).

The paper defines the leading-order cost of AHA as

$S$6

which becomes

$S$7

since $S$8. The ratio to full attention is given as

$S$9

For the example $S=4$0, $S=4$1, and $S=4$2 windows with $S=4$3, the complexity ratio is about $S=4$4, described as around a $S=4$5 theoretical reduction relative to full attention (Zhao et al., 28 Sep 2025).

After the AHA blocks, FastViDAR applies a stage-4 local refinement module using stacked window self-attention only:

$S=4$6

and

$S=4$7

This refinement stage is intended to sharpen local detail after global cross-view mixing has injected contextual information (Zhao et al., 28 Sep 2025).

4. ERP fusion and confidence-aware supervision

FastViDAR predicts per-view depth in ERP space and then fuses those predictions into a final omnidirectional depth map. For each frame $S=4$8, the predicted ERP depth is $S=4$9. Using the corresponding unit ray $360^\circ$0, the point in the rig or world frame is

$360^\circ$1

These points are then projected back into the shared ERP lattice (Zhao et al., 28 Sep 2025).

The reprojection stage uses distance-aware splatting. Each sample is splatted into a $360^\circ$2 footprint centered at the reprojected ERP coordinate, where $360^\circ$3 grows for nearer points and shrinks for farther ones. The paper describes the use of amin or z-buffer-like logic for depth, sum and count accumulation for color, and light hole filling (Zhao et al., 28 Sep 2025).

The simplest and best-performing fusion strategy in the reported ablation is masked mean fusion:

$360^\circ$4

and, similarly, for confidence,

$360^\circ$5

Here $360^\circ$6 is the field-of-view mask for frame $360^\circ$7. The paper notes that predictions can extend slightly beyond nominal field-of-view boundaries, so masked averaging gives complementary coverage and reduces variance (Zhao et al., 28 Sep 2025).

Training uses an ERP-weighted loss to compensate for spherical distortion. Rows in ERP are weighted by

$360^\circ$8

This weighting is intended to make supervision approximately equal-solid-angle. The per-view data term is

$360^\circ$9

with $\mathbf d=(d_x,d_y,d_z)$0 the Huber loss, and the multi-scale gradient term is

$\mathbf d=(d_x,d_y,d_z)$1

The final depth loss is

$\mathbf d=(d_x,d_y,d_z)$2

Confidence enters the loss as an optional weighting factor (Zhao et al., 28 Sep 2025).

This suggests that FastViDAR treats confidence not merely as an auxiliary output but as a control signal for supervision and for downstream fusion variants.

5. Datasets, evaluation protocol, and reported performance

FastViDAR is trained and evaluated using ERP image-depth pairs generated from HM3D and 2D-3D-S. HM3D is used for training and ablations, with 800 training scenes, 200 test scenes, 421,127 training groups, and 52,484 test groups. Each group contains 4 ERP views from a 4-camera rigid rig, with randomized relative poses and field of view in $\mathbf d=(d_x,d_y,d_z)$3 (Zhao et al., 28 Sep 2025).

The 2D-3D-S dataset is used for zero-shot evaluation. It contains 6 large scenes and 6,000 groups total, with a fixed rigid ring of 4 fisheye cameras, field of view $\mathbf d=(d_x,d_y,d_z)$4, baseline $\mathbf d=(d_x,d_y,d_z)$5, and adjacent cameras separated by $\mathbf d=(d_x,d_y,d_z)$6. The model is not fine-tuned on 2D-3D-S (Zhao et al., 28 Sep 2025).

Evaluation is performed on the ERP grid under the intersection mask

$\mathbf d=(d_x,d_y,d_z)$7

so only pixels valid in both ground truth and the method’s field of view are scored. The reported metrics are AbsRel,

$\mathbf d=(d_x,d_y,d_z)$8

RMSE, Log10, and $\mathbf d=(d_x,d_y,d_z)$9 accuracy, with threshold accuracy defined by

$$\lambda=\operatorname{atan2}(d_x,d_z), \qquad \phi=\arcsin(d_y)$$0

These metrics are standard dense depth metrics, but here they are explicitly evaluated on a spherical grid with validity masking (Zhao et al., 28 Sep 2025).

The paper’s headline zero-shot comparison on 2D-3D-S reports the following values:

Method	AbsRel	RMSE	Log10
VGGT	0.557	1.934	0.396
OmniStereo	0.619	1.450	0.154
LightStereo	0.125	0.667	0.050
FastViDAR	0.119	0.433	0.046

In the same comparison, the reported $$\lambda=\operatorname{atan2}(d_x,d_z), \qquad \phi=\arcsin(d_y)$$1 accuracy is 0.043 for VGGT, 0.554 for OmniStereo, 0.851 for LightStereo, and 0.929 for FastViDAR, with runtime 120 ms, 66 ms, 33 ms, and 36 ms respectively (Zhao et al., 28 Sep 2025).

The embedded deployment claim is that FastViDAR achieves up to 20 FPS on NVIDIA Orin NX with TensorRT fp16 optimization. At $$\lambda=\operatorname{atan2}(d_x,d_z), \qquad \phi=\arcsin(d_y)$$2 with 4 frames, it is also reported to be 3.3× faster than VGGT in practice, with the advantage increasing with input resolution and/or frame count (Zhao et al., 28 Sep 2025).

6. Ablations, interpretation, and relation to other ViDAR usages

The paper includes two central ablation studies. The first isolates the effect of global summary attention by comparing a No-Global variant, which uses window and frame attention only, to the full AHA configuration. No-Global reports AbsRel 0.135, RMSE 0.454, Log10 0.181, and $$\lambda=\operatorname{atan2}(d_x,d_z), \qquad \phi=\arcsin(d_y)$$3 0.892 at 34 ms, whereas AHA reports AbsRel 0.111, RMSE 0.384, Log10 0.163, and $$\lambda=\operatorname{atan2}(d_x,d_z), \qquad \phi=\arcsin(d_y)$$4 0.904 at 36 ms. The result is presented as evidence that adding global summary attention improves accuracy substantially with only a small runtime increase (Zhao et al., 28 Sep 2025).

The second ablation evaluates fusion strategies. The reported values are No-fusion: AbsRel 0.109, RMSE 0.369, Log10 0.149, $$\lambda=\operatorname{atan2}(d_x,d_z), \qquad \phi=\arcsin(d_y)$$5 0.897; Nearest: 0.113, 0.384, 0.153, 0.898; Weighted: 0.108, 0.365, 0.146, 0.901; and Mean: 0.108, 0.364, 0.146, 0.901. Mean fusion is therefore described as best or tied-best (Zhao et al., 28 Sep 2025).

Taken together, the reported design rationale is threefold: ERP normalization unifies different fisheye lenses into a shared spherical domain; AHA captures local structure efficiently while enabling cross-view feature mixing without full attention; and ERP fusion aggregates per-camera depth in a common coordinate system to improve coverage and stability (Zhao et al., 28 Sep 2025).

The term “ViDAR” is used in other contemporary arXiv work with different meanings. “ViDAR: Video Diffusion-Aware 4D Reconstruction From Monocular Inputs” denotes a monocular dynamic novel view synthesis and 4D reconstruction framework based on diffusion-generated pseudo multi-view supervision and Gaussian splatting (Nazarczuk et al., 23 Jun 2025). “A Novel ViDAR Device With Visual Inertial Encoder Odometry and Reinforcement Learning-Based Active SLAM Method” uses ViDAR to refer to a Video Detection and Ranging device with a rotating camera, encoder, and tightly coupled visual-inertial-encoder odometry for active SLAM (Xin et al., 16 Jun 2025). FastViDAR is distinct from both usages: it is neither a monocular 4D reconstruction framework nor an active-vision SLAM platform, but a multi-fisheye omnidirectional depth estimator centered on ERP unification and hierarchical cross-view attention (Zhao et al., 28 Sep 2025).

This distinction matters because the shared acronym can obscure fundamentally different problem formulations. In the FastViDAR paper, the primary contribution is real-time omnidirectional dense depth estimation from four fisheye cameras; in the other ViDAR papers, the central objects are diffusion-aware 4D reconstruction and visual-inertial active SLAM, respectively (Zhao et al., 28 Sep 2025, Nazarczuk et al., 23 Jun 2025, Xin et al., 16 Jun 2025).