UniCT Depth: Unified Event-Image Depth Estimation

• UniCT Depth is a supervised monocular depth estimation method that fuses RGB images with event camera data using a unified CNN–Transformer architecture.
• It employs a U-Net-like encoder–decoder featuring a Convolution-compensated ViT Dual SA block that integrates CMSA, MFSA, and DCC for robust cross-modal fusion.
• The approach delivers superior depth prediction by effectively addressing challenges like occlusions, motion blur, and high dynamic range in diverse lighting conditions.

Searching arXiv for the target paper to ground the article in the current record. UniCT Depth, introduced in "UniCT Depth: Event-Image Fusion Based Monocular Depth Estimation with Convolution-Compensated ViT Dual SA Block" (Jing et al., 26 Jul 2025), is a supervised monocular depth estimation method for synchronized RGB images and event-camera data. It predicts a dense depth map $D \in \mathbb{R}^{H\times W}$ from an RGB intensity frame $I \in \mathbb{R}^{H\times W \times 3}$ and a time-synchronized event representation, with the central design goal of unifying CNN and Transformer mechanisms for local detail modeling, long-range context aggregation, and cross-modal fusion. The method is built around a U-Net-like encoder–decoder with a Convolution-compensated ViT Dual SA block, abbreviated CcViT-DA, whose three principal components are Context Modeling Self-Attention (CMSA), Modal Fusion Self-Attention (MFSA), and Detail Compensation Convolution (DCC) (Jing et al., 26 Jul 2025).

1. Problem formulation and motivation

UniCT Depth addresses supervised monocular depth estimation in the paired-modality setting $(I_k, E_k) \mapsto \hat D_k$, where the predicted depth $\hat D_k$ is trained to match ground-truth depth $D_k^*$ at each time index $k$ (Jing et al., 26 Jul 2025). The paper motivates this formulation by arguing that pure RGB monocular depth methods, whether CNN-based or Transformer-based, are fragile in low light or night scenes, high dynamic range scenes, fast motion or motion blur, and scenes containing occlusions or large depth disparities. In those conditions, images can become noisy, saturated, blurred, or locally ambiguous, especially near depth discontinuities.

Event cameras provide a complementary sensing modality. Each event is represented as $e=(x,y,t,p)$, where $(x,y)$ is the pixel location, $t$ is the event time, and $p \in \{-1,+1\}$ is the polarity. The paper identifies their strengths as very high temporal resolution at the $I \in \mathbb{R}^{H\times W \times 3}$0 scale, high dynamic range, and low motion blur, but also emphasizes their weaknesses: sparsity, spurious noise, and lack of absolute appearance or texture. As a result, event-only depth estimation often yields semi-dense or noisy depth and struggles on textureless regions, fine structural details, and static scene elements that do not generate events in a short window (Jing et al., 26 Jul 2025).

The paper treats event–image fusion as the natural remedy to these complementary failure modes. Images provide rich spatial structure and texture, while events contribute robust dynamic and HDR information. The difficulty is not the desirability of fusion, but the structure of fusion itself. UniCT Depth is explicitly designed to address four stated problems: occlusions and depth discontinuities, modality alignment despite disparate sensing characteristics, limited receptive fields in CNN-based fusion, and shallow or inefficient modality interaction in Transformer-based fusion (Jing et al., 26 Jul 2025).

2. Input representation and network architecture

The network takes two inputs: an RGB image $I \in \mathbb{R}^{H\times W \times 3}$1 and a preprocessed event voxel grid $I \in \mathbb{R}^{H\times W \times 3}$2, and produces a dense depth map $I \in \mathbb{R}^{H\times W \times 3}$3 (Jing et al., 26 Jul 2025). Events are converted into a spatio-temporal voxel grid with $I \in \mathbb{R}^{H\times W \times 3}$4 temporal bins, preserving temporal distribution and reducing motion blur. For an event $I \in \mathbb{R}^{H\times W \times 3}$5 in a time window $I \in \mathbb{R}^{H\times W \times 3}$6, the normalized time coordinate is

$I \in \mathbb{R}^{H\times W \times 3}$7

and the voxel grid is defined as

$I \in \mathbb{R}^{H\times W \times 3}$8

This yields $I \in \mathbb{R}^{H\times W \times 3}$9, aligned to the RGB resolution. In the experiments, $(I_k, E_k) \mapsto \hat D_k$0 and RGB inputs are resized to $(I_k, E_k) \mapsto \hat D_k$1 (Jing et al., 26 Jul 2025).

The preprocessor uses separate initial convolutions for events and images, producing $(I_k, E_k) \mapsto \hat D_k$2 and $(I_k, E_k) \mapsto \hat D_k$3, then concatenates them and applies another convolution to form a shared full-resolution feature map $(I_k, E_k) \mapsto \hat D_k$4. This early unification is a defining architectural choice: after the preprocessor, the encoder is shared rather than split into modality-specific branches, so all later stages operate on fused features rather than independently processed streams (Jing et al., 26 Jul 2025).

The encoder contains two residual blocks followed by four CcViT-DA blocks. Feature resolutions progress through

$(I_k, E_k) \mapsto \hat D_k$5

with downsampling inside each CcViT-DA block implemented by a $(I_k, E_k) \mapsto \hat D_k$6 patch-embedding convolution with stride 2. The decoder mirrors the encoder using five decoder blocks, each with transpose convolution of kernel size 3 and stride 2, skip-connected encoder features via channel concatenation, and convolutional refinement. Decoder resolutions are

$(I_k, E_k) \mapsto \hat D_k$7

and the final layer outputs a pixel-wise full-resolution depth prediction $(I_k, E_k) \mapsto \hat D_k$8 (Jing et al., 26 Jul 2025).

3. Convolution-compensated ViT Dual SA block

The CcViT-DA block is the architectural core of UniCT Depth. For an input feature map $(I_k, E_k) \mapsto \hat D_k$9, it performs patch embedding, tokenization with positional embedding, parallel processing through CMSA and MFSA, and local refinement through DCC before merging the results (Jing et al., 26 Jul 2025). The block is intended to answer three complementary questions: where spatial context should be aggregated, which channels or fused modality patterns should be emphasized, and how local convolutional priors should be reintroduced.

CMSA is a window-based multi-head self-attention mechanism designed for spatial dependency modeling. Tokens are partitioned into non-overlapping windows, and self-attention is computed independently within each window. This reduces complexity from $\hat D_k$0 to $\hat D_k$1, where $\hat D_k$2 is the total number of tokens and $\hat D_k$3 the tokens per window. The paper characterizes CMSA as suitable for depth discontinuities, occlusions, and local depth variation because it maintains spatial attention within moderate-sized neighborhoods while avoiding the expense of global token interactions (Jing et al., 26 Jul 2025).

MFSA is designed for global channel-wise dependencies and modal fusion. Instead of attending spatially, it groups channels and performs attention along the channel dimension, effectively aggregating global information per channel group. The method uses a single attention head and grouped channels, giving complexity $\hat D_k$4 rather than $\hat D_k$5. The paper’s interpretation is explicitly modality-aware: MFSA can upweight event-dominated channels in dark regions and image-dominated channels in well-lit regions, because both modalities have already been integrated into the shared representation before the block (Jing et al., 26 Jul 2025).

The merge stage combines CMSA and MFSA outputs with DCC-derived weighting. The paper states that the CMSA and MFSA outputs are dot-multiplied with the DCC output as weights, then concatenated and passed through a merge layer such as a $\hat D_k$6 convolution with normalization and nonlinearity. In consequence, the block is neither a conventional Transformer encoder nor a simple CNN–Transformer hybrid in parallel; it is a modality-fusion block in which spatial attention, channel attention, and convolutional detail compensation are explicitly coupled (Jing et al., 26 Jul 2025).

4. Detail Compensation Convolution and local structural recovery

The DCC block is the “convolution-compensated” component of CcViT-DA. Its purpose is to restore local inductive bias, strengthen texture details, and enhance edge representations, compensating for the tendency of pure attention mechanisms to over-smooth or miss fine boundaries (Jing et al., 26 Jul 2025).

Given a downsampled feature map $\hat D_k$7, DCC first computes global max pooling and global average pooling over channels, producing two single-channel spatial maps $\hat D_k$8 and $\hat D_k$9. These are concatenated into a 2-channel tensor $D_k^*$0, processed by a $D_k^*$1 convolution and sigmoid to obtain a spatial attention map

$D_k^*$2

The map is applied multiplicatively,

$D_k^*$3

and then refined through two further convolution layers with activations to produce a one-channel detail map $D_k^*$4. This detail map is dot-multiplied with the outputs of CMSA and MFSA, so high-frequency local cues modulate both the spatial-attention and channel-attention branches (Jing et al., 26 Jul 2025).

Within the overall fusion strategy, DCC anchors the attention outputs in convolutional evidence. The paper explicitly places DCC inside every CcViT-DA encoder block, not as a post hoc decoder refinement. This matters because the detail weighting affects the internal fused representation at multiple scales rather than only the final upsampled output. The ablation discussion reports that enabling both DCC branches significantly reduces errors versus attention alone while maintaining similar FPS (Jing et al., 26 Jul 2025).

5. Training objective and empirical performance

The method is trained fully supervised on MVSEC and DENSE, with no self-supervised photometric or consistency losses (Jing et al., 26 Jul 2025). For predicted depth $D_k^*$5 and ground truth $D_k^*$6, the residual is defined as

$D_k^*$7

and the loss is

$D_k^*$8

where $D_k^*$9 is the number of valid ground-truth pixels. The paper describes this as effectively an L1+L2 penalty that encourages both small absolute errors and discourages large outliers (Jing et al., 26 Jul 2025).

Implementation is in PyTorch on $k$0 NVIDIA RTX 3090. On MVSEC, training uses learning rate $k$1, AdamW, MultiStepLR with decay factor 0.5 at epochs 10, 20, and 30, batch size 16, 50 epochs, input resolution $k$2, event voxel bins $k$3, and equal L1 and L2 loss weights. On DENSE, the learning rate is $k$4, with other settings similar unless otherwise specified (Jing et al., 26 Jul 2025).

On MVSEC, UniCT Depth achieves the best mean Avg.Error across all cutoffs and sequences. At 10 m, 20 m, and 30 m, the reported mean Avg.Error values are 1.09, 1.84, and 2.71, respectively. On Outdoor day1, the paper reports Abs.Rel 0.221, RMSE log 0.320, and $k$5; on Outdoor night1, it reports Abs.Rel 0.311 and RMSE log 0.463. The summary given in the paper is that UniCT obtains the best results in 10 out of 12 scores across day and night (Jing et al., 26 Jul 2025).

On DENSE, the reported Avg.Error values are 0.26 at 10 m, 0.70 at 20 m, and 1.56 at 30 m, all best in the table. Abs.Rel is 0.18, identified as second-best and slightly above DPT at 0.17, while RMSE log is 0.36, described as close to the best value of 0.32. The qualitative analysis states that UniCT produces more complete and accurate depth for trees, poles, and traffic lights, and differentiates foreground objects from background more clearly, especially in occluded regions (Jing et al., 26 Jul 2025).

6. Comparative position, efficiency, and limitations

The paper positions UniCT Depth between two families of prior fusion approaches. Against CNN-based fusion methods such as RAMNet, EMoDepth, and Mixed-EF2DNet, it argues that pure convolutions are strong for local semantic feature learning but weak for long-range dependencies, occlusion relationships, and large depth disparities. Against Transformer-based fusion approaches, including event Transformers that process events independently, SRFNet, and a single global Transformer encoder over concatenated tokens, it argues that prior methods either fuse too late or incur expensive, noisy global attention with limited modality specificity (Jing et al., 26 Jul 2025).

UniCT Depth’s answer to these limitations is fourfold: a unified encoder after the preprocessor, dual self-attention in the form of CMSA and MFSA, convolution compensation through DCC, and efficiency through window-based and grouped attention. The ablations quantify these trade-offs. A pure convolution variant is fastest at 47 FPS but has the highest error. Standard global self-attention is the slowest at 13 FPS. MFSA+CMSA yields the best accuracy among the ablations without DCC and runs at about 25 FPS, while the full CcViT-DA configuration gives the best Avg.Error across all distances, also at about 25 FPS (Jing et al., 26 Jul 2025).

The modality ablation is equally diagnostic. Using only events performs better at night scenes, using only images performs better in day scenes, and using both yields the best averaged and per-scene accuracy. This directly supports the method’s central claim that event–image fusion is beneficial when the architecture can exploit complementary sensing properties at depth discontinuities, under adverse illumination, and in the presence of motion blur (Jing et al., 26 Jul 2025).

The paper also makes several limitations explicit or implicit. UniCT Depth relies on supervised training with depth ground truth, uses a relatively moderate input resolution of $k$6, and fixes the event voxel representation to $k$7 bins. The paper notes that scaling to larger resolutions may require careful tuning of window sizes and grouping, and that extreme motion or very long exposures might need more sophisticated event representations (Jing et al., 26 Jul 2025). These constraints delimit the scope of the current formulation, but they do not alter its central significance: UniCT Depth is a unified CNN–Transformer event–image fusion architecture in which spatial context modeling, channel-wise modal fusion, and local convolutional compensation are explicitly co-designed for dense monocular depth estimation.