---
title: 'FUSE: Label-Free Image–Event Depth Estimation'
url: https://www.emergentmind.com/topics/fuse
type: topic
---

# FUSE: Label-Free Image–Event Depth Estimation

Searching arXiv for the specified paper and the cited foundation model to ground the article.
arxiv_search: {"query":"id:2503.19739 OR title:\"FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"}
FUSE is a label-free image–event joint monocular depth estimation framework that addresses two coupled obstacles in multimodal depth prediction: the scarcity of annotated image–event–depth datasets and the intrinsic frequency mismatch between static images and dynamic event streams. Its central design combines a Frequency-decoupled Unified Self-supervised Encoder with two components—Parameter-efficient Self-supervised Transfer (PST) and the Frequency-Decoupled Fusion module (FreDFuse)—to distill geometric priors from an image-only depth foundation model and to fuse image and event features in frequency-separated branches where each modality dominates the band it is physically best suited to represent. The resulting system is intended as a universal image–event encoder that can be transferred to target datasets with only lightweight decoder adaptation [2503.19739].

## 1. Problem formulation and sensor complementarity

Image–event depth estimation exploits complementary sensing modalities. In the formulation used by FUSE, conventional images are treated as fixed-exposure measurements that emphasize low-frequency scene structure but suppress rapid changes, whereas event streams are asynchronous brightness-change measurements that emphasize high-frequency temporal edges but are weak in static low-gradient regions [2503.19739].

Two failure modes motivate the framework. First, annotated image–event–depth datasets are limited in diversity and scale relative to image–depth corpora, which restricts the generalization of fully supervised fusion methods. Second, naïve cross-modal fusion can induce destructive interference: event high-frequency signals can fragment image structure, while image low-frequency signals can drown event motion cues. FUSE addresses the first issue through label-free cross-modal transfer from an image-only depth foundation model and the second through explicit frequency decoupling and physics-aware fusion.

The event stream is represented as
\[
\mathbf{S} = \{e_n\}_{n=0}^{N-1}, \qquad e_n = (x_n, y_n, t_n, p_n),
\]
and is converted into a voxel grid
\[
\mathit{V} \in \mathbb{R}^{H \times W \times B}
\]
with a triangular temporal kernel over \(B = 3\) time bins within a window \(\Delta T = t_{N-1} - t_0\):
\[
\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\},
\qquad
t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).
\]
This yields a frame-like event volume compatible with ViT-style encoders. Images are used directly. For MVSEC alignment, FUSE pairs the most recent image with events from the 50 ms preceding each LiDAR depth timestamp.

## 2. System architecture and transfer workflow

FUSE comprises an image encoder, an event encoder, FreDFuse, and a depth decoder. The image encoder and decoder are initialized from Depth Anything V2, and the full training procedure is divided into two PST stages followed by target-dataset decoder adaptation [2503.19739].

The overarching workflow is as follows. PST Stage I aligns the event branch to the image latent space by initializing the event encoder from the image encoder and tuning only LoRA matrices and PatchEmbed parameters, which account for about \(1.8\%\) of encoder parameters. PST Stage II then trains only FreDFuse on randomly degraded image–event pairs so that fusion becomes robust to sensor-specific degradations. After these two stages, the image–event joint encoder is frozen. Transfer to a new labeled dataset requires training only a small depth decoder with a metric-depth objective.

This organization is notable because the framework does not rely on photometric, smoothness, or geometry-consistency losses. Supervision during PST is distilled from the image foundation model, and only at target adaptation time is this replaced by a metric-depth loss. A plausible implication is that the method treats cross-modal representation transfer, rather than scene-reconstruction consistency, as the principal route to generalization.

## 3. Parameter-efficient Self-supervised Transfer

PST is the self-supervised core of FUSE. In Stage I, the clean image \(I\) is passed through the image foundation model to produce pseudo-depth \(d\) and latent features \(F\), while the event branch produces \(d^*\) and \(F^*\). The alignment loss is
\[
\mathcal{L}_{align} = \mathcal{L}_1 + \mathcal{L}_{\cos},
\]
with
\[
\mathcal{L}_1 = \lVert d - d^* \rVert_1,
\]
and
\[
\mathcal{L}_{\cos} =
\left[ 1 - \cos(F, F^*) \right] \cdot \mathbb{I}(\alpha \le \cos(F, F^*) \le \beta),
\]
where \(\alpha = 0.2\) and \(\beta = 0.85\). The gated cosine term avoids over-constraint when the representations are already well aligned or remain too dissimilar [2503.19739].

The design transfers both prediction priors and latent geometric priors from the image model into the event branch without any depth ground truth. PST is compatible with ViT-style image encoders; the paper specifies that CLIP and DINO are not used, and that the transfer originates from an image-based depth foundation model.

Stage II keeps the inherited image encoder and decoder fixed, retains the event encoder obtained from Stage I, and trains only FreDFuse on randomly degraded image–event pairs. The degradations are brightness shifts, overexposure, motion blur, and occlusion masks. The loss remains \(\mathcal{L}_{align}\), still referenced to clean-image pseudo-supervision, forcing fusion to become degradation-robust.

For target datasets with metric depth labels, only the decoder is trained, using the scale-invariant logarithmic loss
\[
\mathcal{L}_{SiLog} =
\sqrt{
\frac{1}{N} \sum_i e_i^2
-
\lambda \left( \frac{1}{N} \sum_i e_i \right)^2
},
\qquad
e_i = \ln d_i - \ln d_i^*.
\]
The encoder remains frozen throughout this phase. This separation between universal encoder learning and lightweight decoder adaptation is one of the defining properties of the framework.

## 4. Frequency-Decoupled Fusion and physics-aware cross-attention

FreDFuse explicitly separates low-frequency structure from high-frequency detail before fusion. Given image features \(\mathbf{F}_I \in \mathbb{R}^{B \times N \times C}\) and event features \(\mathbf{F}_E \in \mathbb{R}^{B \times N \times C}\), FUSE reshapes tokens into 2D maps and constructs a Gaussian–Laplacian pyramid with \(L = 3\) levels for each modality [2503.19739]:
\[
\mathbf{G}_m^{(l)} = \text{Down} \big( \text{GaussianBlur}(\mathbf{G}_m^{(l-1)}) \big),
\]
\[
\mathbf{L}_m^{(l)} = \mathbf{G}_m^{(l)} - \text{Up}(\mathbf{G}_m^{(l+1)}),
\qquad m \in \{I, E\}.
\]
The Gaussian pyramid captures low-frequency structure, while the Laplacian pyramid captures high-frequency edges. Conceptually, the decomposition is
\[
X_{low} = G_\sigma * X, \qquad X_{high} = X - X_{low}.
\]

Each pyramid then undergoes multi-scale top-down fusion: features are upsampled and concatenated, compressed with \(1 \times 1\) grouped convolutions, followed by channel shuffle and another grouped projection. This stage promotes cross-scale and cross-channel interaction while preserving the separation between spectral bands.

Cross-modal attention is asymmetric and explicitly aligned with sensor physics. The high-frequency branch is event-driven:
\[
\mathbf{Q} = \mathbf{F}_E^{high}, \qquad \mathbf{K}, \mathbf{V} = \mathbf{F}_I^{high},
\]
whereas the low-frequency branch is image-driven:
\[
\mathbf{Q} = \mathbf{F}_I^{low}, \qquad \mathbf{K}, \mathbf{V} = \mathbf{F}_E^{low}.
\]
The attention operator is
\[
\text{Attn}(\mathbf{Q}, \mathbf{K}, \mathbf{V}) =
\text{Softmax}\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d}}\right)\mathbf{V}.
\]
The fused branches are recombined by
\[
\mathbf{F}_{fused} =
\text{LayerNorm}\big(
\text{Linear}(\mathbf{F}_{low} + \mathbf{F}_{high})
\big).
\]

The framework also summarizes the decomposition as
\[
I = I_{low} + I_{high}, \qquad E = E_{low} + E_{high},
\]
with
\[
I_{low} = G_\sigma * I, \quad I_{high} = I - I_{low};
\qquad
E_{low} = G_\sigma * E, \quad E_{high} = E - E_{low}.
\]
A plausible interpretation is that FreDFuse converts an otherwise generic attention block into a modality-prioritized spectral mixer: events inject crisp edges and motion, whereas images stabilize global layout.

## 5. Training protocol, benchmarks, and empirical results

PST uses EventScape image–event pairs without depth labels. Evaluation is conducted on MVSEC and DENSE. For MVSEC, sequences outdoor_day2, outdoor_night2, and outdoor_night3 are used for training, with outdoor_day1 and outdoor_night1 for testing; the depth range is \(1.97\)–\(80\) m. DENSE uses the provided train/test splits with a \(3.34\)–\(1000\) m depth range. Optimization uses Adam with learning rate \(5 \times 10^{-5}\), batch size \(48\), PyTorch, and two RTX 3090 Ti GPUs. PST on EventScape runs for 10 epochs, and decoder fine-tuning on MVSEC and DENSE runs for 20 epochs [2503.19739].

Metrics follow prior work: Abs.Rel, RMSE, RMSELog, \(\delta\) thresholds \((\delta < 1.25^n, n=1,2,3)\), and truncated average depth errors at 10, 20, and 30 m.

| Benchmark | FUSE result | Comparative note |
|---|---:|---|
| MVSEC outdoor_day1 | \(\delta_1 = 0.745\), \(\delta_2 = 0.892\), \(\delta_3 = 0.957\), Abs.Rel \(= 0.196\), RMSE \(= 6.004\), RMSELog \(= 0.270\) | Better than prior image–event fusion baselines |
| MVSEC outdoor_day1 vs. PCDepth | Abs.Rel \(0.196\) vs. \(0.228\); RMSELog \(0.270\) vs. \(0.301\) | \(14\%\) Abs.Rel and \(10.2\%\) RMSELog improvement |
| MVSEC outdoor_night1 | Abs.Rel \(= 0.261\) | PCDepth \(= 0.271\) |
| DENSE | Abs.Rel \(= 0.385\), RMSELog \(= 0.457\) | Better than SRFNet |
| DENSE vs. SRFNet | Abs.Rel \(0.385\) vs. \(0.513\); RMSELog \(0.457\) vs. \(0.687\) | \(24.9\%\) Abs.Rel and \(33.4\%\) RMSELog improvement |

Qualitative results are reported to show robustness under extreme lighting and motion blur, including low light, overexposure, and occlusions. The paper attributes this behavior to event-driven high-frequency fusion preserving structural consistency when images are severely degraded. It also describes the framework as exhibiting zero-shot adaptability in such scenarios.

## 6. Ablations, efficiency profile, and limitations

Ablation studies isolate both of the framework’s principal components. With ViT-Small backbones on MVSEC, replacing standard cross-attention fusion by FreDFuse yields an \(8\%\) Abs.Rel improvement. Incorporating PST reduces trainable parameters by \(82.2\%\) relative to training from scratch and improves performance by an average of \(19.7\%\) across metrics and scenarios. A one-stage transfer baseline that tunes the event encoder and FreDFuse simultaneously performs worse than the two-stage PST, indicating that latent alignment and degradation-robust fusion benefit from staged optimization [2503.19739].

The parameter counts reported in the ablations further clarify the efficiency regime. Baseline-1, which uses scratch training with cross-attention, has \(49.2\)M trainable parameters. Baseline-2, scratch training with FreDFuse, has \(51.6\)M. Baseline-3, one-stage transfer with FreDFuse, uses \(6\)M + \(2.7\)M. FUSE itself uses \(1.4\)M + \(4.7\)M + \(2.7\)M across PST Stage I, PST Stage II, and decoder adaptation. FLOPs and runtime are not explicitly reported, but the frozen-encoder transfer protocol is described as reducing memory and computation during adaptation.

The paper also specifies several limitations. Events are represented as fixed-interval voxel grids, which may underutilize microsecond temporal resolution. The method assumes synchronized image–event pairing, so misalignment can degrade quality. Performance can be sensitive to the pyramid depth \(L\), the Gaussian blur scale, and the gating parameters \((\alpha, \beta)\) in \(\mathcal{L}_{\cos}\). Domain gaps between synthetic and real scenes may require careful decoder adaptation. These caveats delimit the current operating regime of the universal encoder claim: the representation is reusable, but not independent of sensor synchronization, frequency-hyperparameter choice, or domain-specific decoder tuning.

Source: https://www.emergentmind.com/topics/fuse