FUSE: Label-Free Image–Event Depth Estimation

• FUSE is a joint monocular depth estimation framework that fuses static images with dynamic event streams using frequency-decoupled alignment and physics-aware fusion.
• It employs Parameter-efficient Self-supervised Transfer (PST) to align and transfer features from an image depth model, reducing reliance on annotated datasets.
• The Frequency-Decoupled Fusion module separates low-frequency structure from high-frequency details to enhance robustness under challenging lighting and motion conditions.

Searching arXiv for the specified paper and the cited foundation model to ground the article. arxiv_search: {"^^^^^^^^2query2 OR title:\2^^^^^^^^"FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion\"","max_results":5,"sort_by":"submittedDate","sort_order":"descending"} FUSE is a label-free image–event joint monocular depth estimation framework that addresses two coupled obstacles in multimodal depth prediction: the scarcity of annotated image–event–depth datasets and the intrinsic frequency mismatch between static images and dynamic event streams. Its central design combines a Frequency-decoupled Unified Self-supervised Encoder with two components—Parameter-efficient Self-supervised Transfer (PST) and the Frequency-Decoupled Fusion module (FreDFuse)—to distill geometric priors from an image-only depth foundation model and to fuse image and event features in frequency-separated branches where each modality dominates the band it is physically best suited to represent. The resulting system is intended as a universal image–event encoder that can be transferred to target datasets with only lightweight decoder adaptation (&&&^^^^2query2^^^^&&&).

^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^. Problem formulation and sensor complementarity

Image–event depth estimation exploits complementary sensing modalities. In the formulation used by FUSE, conventional images are treated as fixed-exposure measurements that emphasize low-frequency scene structure but suppress rapid changes, whereas event streams are asynchronous brightness-change measurements that emphasize high-frequency temporal edges but are weak in static low-gradient regions (&&&^^^^2query2^^^^&&&).

Two failure modes motivate the framework. First, annotated image–event–depth datasets are limited in diversity and scale relative to image–depth corpora, which restricts the generalization of fully supervised fusion methods. Second, naïve cross-modal fusion can induce destructive interference: event high-frequency signals can fragment image structure, while image low-frequency signals can drown event motion cues. FUSE addresses the first issue through label-free cross-modal transfer from an image-only depth foundation model and the second through explicit frequency decoupling and physics-aware fusion.

The event stream is represented as

PRESERVED_PLACEHOLDER_^^^^2query2^^^^

and is converted into a voxel grid

PRESERVED_PLACEHOLDER_^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^

with a triangular temporal kernel over $B = 3$ time bins within a window $\Delta T = t_{N-1} - t_0$: $$\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\}, \qquad t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).$$ This yields a frame-like event volume compatible with ViT-style encoders. Images are used directly. For MVSEC alignment, FUSE pairs the most recent image with events from the 5^^^^2query2^^^^ ms preceding each LiDAR depth timestamp.

2. System architecture and transfer workflow

FUSE comprises an image encoder, an event encoder, FreDFuse, and a depth decoder. The image encoder and decoder are initialized from Depth Anything V2, and the full training procedure is divided into two PST stages followed by target-dataset decoder adaptation (&&&^^^^2query2^^^^&&&).

The overarching workflow is as follows. PST Stage I aligns the event branch to the image latent space by initializing the event encoder from the image encoder and tuning only LoRA matrices and PatchEmbed parameters, which account for about $1.8\%$ of encoder parameters. PST Stage II then trains only FreDFuse on randomly degraded image–event pairs so that fusion becomes robust to sensor-specific degradations. After these two stages, the image–event joint encoder is frozen. Transfer to a new labeled dataset requires training only a small depth decoder with a metric-depth objective.

This organization is notable because the framework does not rely on photometric, smoothness, or geometry-consistency losses. Supervision during PST is distilled from the image foundation model, and only at target adaptation time is this replaced by a metric-depth loss. A plausible implication is that the method treats cross-modal representation transfer, rather than scene-reconstruction consistency, as the principal route to generalization.

3. Parameter-efficient Self-supervised Transfer

PST is the self-supervised core of FUSE. In Stage I, the clean image $I$ is passed through the image foundation model to produce pseudo-depth $d$ and latent features $F$, while the event branch produces $d^*$ and PRESERVED_PLACEHOLDER_^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2query2^^^^. The alignment loss is

PRESERVED_PLACEHOLDER_^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^

with

PRESERVED_PLACEHOLDER_^^^^2id:(Sun et al., 25 Mar 2025) OR title:\22^^^^

and

PRESERVED_PLACEHOLDER_^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^3

where PRESERVED_PLACEHOLDER_^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^4 and PRESERVED_PLACEHOLDER_^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^5. The gated cosine term avoids over-constraint when the representations are already well aligned or remain too dissimilar (&&&^^^^2query2^^^^&&&).

The design transfers both prediction priors and latent geometric priors from the image model into the event branch without any depth ground truth. PST is compatible with ViT-style image encoders; the paper specifies that CLIP and DINO are not used, and that the transfer originates from an image-based depth foundation model.

Stage II keeps the inherited image encoder and decoder fixed, retains the event encoder obtained from Stage I, and trains only FreDFuse on randomly degraded image–event pairs. The degradations are brightness shifts, overexposure, motion blur, and occlusion masks. The loss remains PRESERVED_PLACEHOLDER_^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^6, still referenced to clean-image pseudo-supervision, forcing fusion to become degradation-robust.

For target datasets with metric depth labels, only the decoder is trained, using the scale-invariant logarithmic loss

PRESERVED_PLACEHOLDER_^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^7

The encoder remains frozen throughout this phase. This separation between universal encoder learning and lightweight decoder adaptation is one of the defining properties of the framework.

4. Frequency-Decoupled Fusion and physics-aware cross-attention

FreDFuse explicitly separates low-frequency structure from high-frequency detail before fusion. Given image features PRESERVED_PLACEHOLDER_^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^8 and event features PRESERVED_PLACEHOLDER_^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^9, FUSE reshapes tokens into 2D maps and constructs a Gaussian–Laplacian pyramid with $B = 3$^^^^2query2^^^^ levels for each modality (&&&^^^^2query2^^^^&&&): $B = 3$^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^

$B = 3$2

The Gaussian pyramid captures low-frequency structure, while the Laplacian pyramid captures high-frequency edges. Conceptually, the decomposition is

$B = 3$3

Each pyramid then undergoes multi-scale top-down fusion: features are upsampled and concatenated, compressed with $B = 3$4 grouped convolutions, followed by channel shuffle and another grouped projection. This stage promotes cross-scale and cross-channel interaction while preserving the separation between spectral bands.

Cross-modal attention is asymmetric and explicitly aligned with sensor physics. The high-frequency branch is event-driven: $B = 3$5 whereas the low-frequency branch is image-driven: $B = 3$6 The attention operator is

$B = 3$7

The fused branches are recombined by

$B = 3$8

The framework also summarizes the decomposition as

$B = 3$9

with

$\Delta T = t_{N-1} - t_0$^^^^2query2^^^^

A plausible interpretation is that FreDFuse converts an otherwise generic attention block into a modality-prioritized spectral mixer: events inject crisp edges and motion, whereas images stabilize global layout.

5. Training protocol, benchmarks, and empirical results

PST uses EventScape image–event pairs without depth labels. Evaluation is conducted on MVSEC and DENSE. For MVSEC, sequences outdoor_day2, outdoor_night2, and outdoor_night3 are used for training, with outdoor_day^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^ and outdoor_night^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^ for testing; the depth range is $\Delta T = t_{N-1} - t_0$^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^–ΔT=tN−1−t0\Delta T = t_{N-1} - t_02 m. DENSE uses the provided train/test splits with a $\Delta T = t_{N-1} - t_0$3–$\Delta T = t_{N-1} - t_0$4 m depth range. Optimization uses Adam with learning rate $\Delta T = t_{N-1} - t_0$5, batch size $\Delta T = t_{N-1} - t_0$6, PyTorch, and two RTX 3^^^^2query2^^^^9^^^^2query2^^^^ Ti GPUs. PST on EventScape runs for ^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2query2^^^^ epochs, and decoder fine-tuning on MVSEC and DENSE runs for 2^^^^2query2^^^^ epochs (&&&^^^^2query2^^^^&&&).

Metrics follow prior work: Abs.Rel, RMSE, RMSELog, $\Delta T = t_{N-1} - t_0$7 thresholds $\Delta T = t_{N-1} - t_0$8, and truncated average depth errors at ^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2query2^^^^, 2^^^^2query2^^^^, and 3^^^^2query2^^^^ m.

Benchmark	FUSE result	Comparative note
MVSEC outdoor_day^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^	$\Delta T = t_{N-1} - t_0$9, $$\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\}, \qquad t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).$$^^^^2query2^^^^, $$\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\}, \qquad t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).$$^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^, Abs.Rel $$\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\}, \qquad t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).$$2, RMSE $$\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\}, \qquad t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).$$3, RMSELog $$\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\}, \qquad t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).$$4	Better than prior image–event fusion baselines
MVSEC outdoor_day^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^ vs. PCDepth	Abs.Rel $$\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\}, \qquad t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).$$5 vs. $$\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\}, \qquad t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).$$6; RMSELog $$\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\}, \qquad t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).$$7 vs. $$\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\}, \qquad t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).$$8	$$\mathit{V}(x, y, t) = \sum_i p_i \, \delta(x - x_i, y - y_i) \, \max \left\{ 0, 1 - \left| t - t_i^* \right| \right\}, \qquad t_i^* = \frac{B-1}{\Delta T} (t_i - t_0).$$9 Abs.Rel and $1.8\%$^^^^2query2^^^^ RMSELog improvement
MVSEC outdoor_night^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^	Abs.Rel $1.8\%$^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^	PCDepth $1.8\%$2
DENSE	Abs.Rel $1.8\%$3, RMSELog $1.8\%$4	Better than SRFNet
DENSE vs. SRFNet	Abs.Rel $1.8\%$5 vs. $1.8\%$6; RMSELog $1.8\%$7 vs. $1.8\%$8	$1.8\%$9 Abs.Rel and $I$^^^^2query2^^^^ RMSELog improvement

Qualitative results are reported to show robustness under extreme lighting and motion blur, including low light, overexposure, and occlusions. The paper attributes this behavior to event-driven high-frequency fusion preserving structural consistency when images are severely degraded. It also describes the framework as exhibiting zero-shot adaptability in such scenarios.

6. Ablations, efficiency profile, and limitations

Ablation studies isolate both of the framework’s principal components. With ViT-Small backbones on MVSEC, replacing standard cross-attention fusion by FreDFuse yields an $I$^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^ Abs.Rel improvement. Incorporating PST reduces trainable parameters by $I$2 relative to training from scratch and improves performance by an average of $I$3 across metrics and scenarios. A one-stage transfer baseline that tunes the event encoder and FreDFuse simultaneously performs worse than the two-stage PST, indicating that latent alignment and degradation-robust fusion benefit from staged optimization (&&&^^^^2query2^^^^&&&).

The parameter counts reported in the ablations further clarify the efficiency regime. Baseline-^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^, which uses scratch training with cross-attention, has $I$4M trainable parameters. Baseline-2, scratch training with FreDFuse, has $I$5M. Baseline-3, one-stage transfer with FreDFuse, uses $I$6M + $I$7M. FUSE itself uses $I$8M + $I$9M + $d$^^^^2query2^^^^M across PST Stage I, PST Stage II, and decoder adaptation. FLOPs and runtime are not explicitly reported, but the frozen-encoder transfer protocol is described as reducing memory and computation during adaptation.

The paper also specifies several limitations. Events are represented as fixed-interval voxel grids, which may underutilize microsecond temporal resolution. The method assumes synchronized image–event pairing, so misalignment can degrade quality. Performance can be sensitive to the pyramid depth $d$^^^^2id:(Sun et al., 25 Mar 2025) OR title:\2^^^^, the Gaussian blur scale, and the gating parameters $d$2 in $d$3. Domain gaps between synthetic and real scenes may require careful decoder adaptation. These caveats delimit the current operating regime of the universal encoder claim: the representation is reusable, but not independent of sensor synchronization, frequency-hyperparameter choice, or domain-specific decoder tuning.