Disentangled OmniAttention for Robust Event Deblurring

• The paper introduces Disentangled OmniAttention, a modality-aware mechanism that separately models intra-motion, inter-motion, and cross-modality dependencies for event-guided deblurring.
• It employs specialized convolutions and branch-specific attention paths to extract robust semantic features from RGB images and fine motion cues from event data.
• Ablation studies demonstrate that disentangled attention significantly improves PSNR and SSIM, emphasizing the critical role of structured multimodal fusion in RED.

Disentangled OmniAttention is the core representation-learning mechanism in RED, “RED: Robust Event-Guided Motion Deblurring with Modality-Specific Disentangled Representation,” where it is used to learn robust, modality-aware features from two imperfect but complementary inputs: a blurry RGB image and an incomplete event stream (Leng et al., 6 Sep 2025). Rather than treating multimodal fusion as a single generic attention problem, it explicitly separates and models three correlation types—intra-motion correlations, inter-motion correlations, and cross-modality correlations—so that semantic structure from blurry images, motion cues from events, and their direction-specific interactions can be handled without collapsing them into one undifferentiated latent space.

1. Motivation and conceptual definition

The immediate motivation for Disentangled OmniAttention is a specific failure mode of event-guided deblurring: event streams are inherently incomplete. Dynamic Vision Sensors trigger events only when logarithmic intensity change exceeds a threshold, so the sensing process trades off sensitivity and noise. A higher threshold suppresses noise but causes under-triggering, producing sparse, corrupted, or semantically incomplete event data. RED argues that this degradation weakens the reliability of events as motion priors and limits the effectiveness of event-guided deblurring (Leng et al., 6 Sep 2025).

Within this setting, standard cross-modal interaction is presented as insufficient. RED states that existing event-guided methods mostly emphasize fusion or cross-attention, but that direct fusion or naive interaction can yield unstable representations, mixed semantics and motion cues, and cross-modal interference when event inputs are corrupted. This argument depends on a strong asymmetry between the two modalities: blurry images contain richer semantic understanding, spatial structures, and texture context, whereas events encode fine-grained motion trajectories but often lack semantic completeness.

In this usage, “disentangled” does not mean a latent-variable factorization enforced by a dedicated disentanglement loss. It means that RED does not process all image–event correlations with a single generic attention operator. Instead, it separates semantic-like and motion-like dependencies and also separates the two directions of cross-modal exchange. The paper describes this as factorizing feature space into semantic and temporal dimensions for modality-specific disentangled representation. A common misconception is therefore to read OmniAttention as a synonym for generic multimodal fusion. In RED, it is more specific: a dimension-aware and modality-aware operator designed for the asymmetry between blurry RGB and incomplete events.

2. Placement within the RED architecture

Architecturally, Disentangled OmniAttention appears after the Robustness-Oriented Perturbation Strategy (RPS) and after branch-wise feature extraction, but before the interaction modules MSEM and ESEM (Leng et al., 6 Sep 2025). The pipeline is explicitly ordered as follows: RPS perturbs the event input with random masking; the network extracts branch features from the image branch and the event branch; OmniAttention operates on these middle-level branch features to produce disentangled modality-specific representations and cross-modal fusion; MSEM transfers motion-sensitive priors from event-side information to the image branch; ESEM injects semantic understanding from the image branch into the event branch; and the refined features then proceed to later stages for final deblurring.

The inputs to OmniAttention are the middle features

$$F^{ei}_{img}, F^{ei}_{evs} \in \mathcal{R}^{B \times N \times H \times W},$$

where $F^{ei}_{img}$ is the middle feature from the image branch, $F^{ei}_{evs}$ is the middle feature from the event branch, $B$ is batch size, $N$ is the flattened feature dimension used in the branch representation, and $H, W$ are spatial dimensions. The module therefore operates jointly over both modalities, but not symmetrically. It first applies modality-specific attention paths and only afterward performs dedicated cross-modality attention.

This placement is central to RED’s notion of modality-specific disentangled representation. The image branch is treated as the carrier of high-level semantic understanding, while the event branch is treated as the carrier of motion priors that may be incomplete. OmniAttention is thus the representation-learning and disentangling layer, whereas MSEM and ESEM are refinement modules that act on top of those already disentangled features.

3. Internal operator structure

For both $F^{ei}_{img}$ and $F^{ei}_{evs}$, the module first applies a $1 \times 1$ convolution followed by a depth-wise $3 \times 3$ convolution to produce $F^{ei}_{img}$0, $F^{ei}_{img}$1, and $F^{ei}_{img}$2 (Leng et al., 6 Sep 2025). The paper states this projection procedure but does not provide explicit equations such as $F^{ei}_{img}$3, nor does it specify whether image and event branches share or separate projection parameters.

The first branch is intra-motion attention. After modality splitting on $F^{ei}_{img}$4, RED performs channel-wise head splitting to obtain spatio-temporal tokens: $F^{ei}_{img}$5 with

$F^{ei}_{img}$6

The attention map is defined as

$F^{ei}_{img}$7

and the paper states

$F^{ei}_{img}$8

The output is then written as

$F^{ei}_{img}$9

This branch is described as modeling semantic characteristic, and is therefore more aligned with the image-side representation.

The second branch is inter-motion attention. Here RED performs head-wise motion splitting and spatio-channel tokenization: $F^{ei}_{evs}$0 The inter-motion attention map is

$F^{ei}_{evs}$1

with the paper stating

$F^{ei}_{evs}$2

The corresponding output is

$F^{ei}_{evs}$3

This branch is said to model temporal motion dependency and is thus more closely associated with the event branch.

After these branch-specific paths, RED performs two-mode cross-modality attention. In the first mode, $F^{ei}_{evs}$4 and $F^{ei}_{evs}$5 are admitted from $F^{ei}_{evs}$6, $F^{ei}_{evs}$7 is admitted from $F^{ei}_{evs}$8, and spatio-temporal tokens are chosen; this lets image-level intra-motion correlation guide event features so that event representations receive abundant semantic information. In the second mode, $F^{ei}_{evs}$9 and $B$0 are admitted from $B$1, $B$2 is admitted from $B$3, and spatio-channel tokens are chosen; this lets event-level inter-motion correlation guide image features so that motion-sensitive cues support detail restoration. The outputs of the two modes are concatenated to obtain

$B$4

The term “omni” is therefore not used in the sense of unrestricted all-axis transformer attention. RED uses it to denote structured reasoning across semantic or channel-oriented correlation, temporal or motion-oriented correlation, and cross-modal interaction, with different token organizations for different paths. The paper also records several under-specifications: the reported tensor shapes and attention-map shapes are not fully consistent, the multiplication order $B$5 is unusual relative to standard transformer notation, no scaling term such as $B$6 is given, and residual, normalization, output projection, and precise tensor-dimension conventions are omitted.

4. Robustness mechanisms and modality-specific collaboration

Disentangled OmniAttention is tightly coupled to RED’s robustness strategy because it operates on event features that have already been perturbed by RPS (Leng et al., 6 Sep 2025). Event voxels are perturbed as

$B$7

where $B$8 is the event voxel input, $B$9 is a binary perturbation mask, and $N$0 is element-wise multiplication. The random disturbance ratio is sampled as

$N$1

Two masking modes are used: temporal-variant masking, where each $N$2 differs over time, and temporal-invariant masking, where the same 2D mask is shared across all time steps.

The paper does not introduce a dedicated missing-event mask inside the attention computation itself. Robustness instead comes from two sources. First, training repeatedly exposes the event branch to incomplete and degraded motion evidence through RPS. Second, OmniAttention is architected so that semantic and motion dependencies are separated before cross-modal exchange, which reduces the risk that corrupted event cues will directly contaminate the image representation.

This ordering matters in RED’s overall logic. Before aggressive interaction, the network first learns reliable modality-specific features. Only after disentanglement does it perform guided complementary interaction through MSEM and ESEM. The image branch contributes high-level semantic understanding; the event branch contributes motion priors but may be incomplete. OmniAttention preserves this complementarity by avoiding immediate collapse into one common latent and by structuring cross-modal interaction directionally.

The paper also states that there is no dedicated loss specific to OmniAttention. No explicit losses are given for disentanglement, decorrelation, consistency between branches, auxiliary supervision on intra/inter/cross attention, or event-missing robustness tied specifically to attention maps. OmniAttention is therefore trained implicitly through RED’s main deblurring objective, while the reported training section contains only general optimization settings such as Adam, learning-rate schedule, patch size, and augmentations including event masking.

5. Empirical evidence and ablation results

The strongest direct evidence for Disentangled OmniAttention comes from RED’s ablation study, where the intra-motion, inter-motion, and cross-modality attentions are selectively replaced by modality-agnostic self-attention (Leng et al., 6 Sep 2025). The full ablation is as follows.

Variant	PSNR	SSIM
Intra replaced, inter replaced, cross replaced	25.77	0.864
Intra replaced, inter kept, cross kept	34.99	0.964
Intra kept, inter replaced, cross kept	35.04	0.966
Intra kept, inter kept, cross replaced	37.14	0.978
Intra kept, inter kept, cross kept	37.27	0.979

Replacing all specialized attentions with modality-agnostic self-attention causes performance to fall from $N$3 to $N$4 in PSNR / SSIM, which the paper describes as a dramatic performance drop of $N$5 dB in PSNR. This is the clearest evidence that indiscriminate attention across blurry images and events is highly harmful in RED’s setting.

The more targeted ablations show that the branch-specific attentions are the dominant contributors. Replacing only intra-motion attention yields $N$6, a drop of $N$7 dB from full RED, although the text says $N$8 dB. Replacing only inter-motion attention yields $N$9, a drop of $H, W$0 dB. Replacing only cross-modality attention yields $H, W$1, a smaller drop of $H, W$2 dB. The paper interprets this as showing that modality-specific disentangled encoding is more critical than final cross-modal fusion alone.

Broader corruption experiments are consistent with the same interpretation. At TIR = 0.2 on GoPro, DSTN reports 35.05, EFNet 35.12, STCNet 34.24, TRMD 34.75, AHDINet 32.72, and $H, W$3 36.94. At TVR = 0.2, DSTN reports 35.05, EFNet 34.08, STCNet 34.29, TRMD 33.84, AHDINet 33.15, and $H, W$4 36.80. RED attributes this robustness jointly to RPS, modality-specific representation, and cross-modality interaction.

The supplementary cost report states that $H, W$5 has 18.3283M parameters and 579.0236G FLOPs under $H, W$6 input size. It is smaller than MAT (20.7293M params), comparable to TRMD (19.2638M params), and heavier than lighter CNN-based baselines. The paper does not provide a cost breakdown specifically for OmniAttention, so its standalone efficiency is not reported.

6. Comparative context, interpretation, and limitations

Within the broader literature on disentangled attention, RED’s Disentangled OmniAttention belongs to a family of designs that separate interaction types rather than relying on one homogeneous operator. D-Attn decomposes decoder-only LVLM attention into V2V, T2V, and T2T components and debiases positional encodings in T2V (Kuo et al., 4 Feb 2025). OrthoMotion routes camera motion into a geometric RoPE-phase operator and subject motion into gated value injection, then regularizes their response subspaces toward orthogonality (Meng, 22 Jun 2026). Compositional Attention disentangles search and retrieval and recombines them dynamically (Mittal et al., 2021). DiMBERT separates visual and textual $H, W$7 projection spaces while retaining joint attention over all tokens (Liu et al., 2022). EncDiff argues that cross-attention over learned concept tokens can itself act as an inductive bias for disentanglement (Yang et al., 2024). These works differ in domain and operator details, but they share the principle that not all dependencies should be modeled by the same attention mechanism.

RED’s version is narrower and domain-specific. It is tailored to the asymmetry between blurry RGB images and incomplete events, and it defines “disentangled” through semantic-style reasoning, motion-style reasoning, and direction-specific cross-modal exchange. A plausible implication is that RED should be read less as a universal omni-attention framework than as a specialized multimodal attention factorization for event-guided deblurring under sensor incompleteness.

The main limitation is reproducibility at operator level. The paper omits exact formulas for the two cross-modality attention operations, does not specify whether projections are shared or separate, and does not report scaling factors, normalization layers, residual connections, output projection layers, precise tensor-dimension conventions for $H, W$8, $H, W$9, $F^{ei}_{img}$0, and $F^{ei}_{img}$1, or the full reconstruction head around OmniAttention. The conceptual design is therefore clear, but the low-level implementation is only partially specified.

In summary, Disentangled OmniAttention in RED is a modality-aware attention mechanism that separates semantic and motion reasoning before multimodal fusion. It extracts semantic correlations where the image branch is stronger, temporal motion correlations where the event branch is stronger, and then performs direction-specific cross-modal exchange so that each modality compensates for the other without unstable interference. In RED, this operator is the central response to incomplete events: not stronger fusion alone, but structured, disentangled attention over complementary and asymmetric modalities (Leng et al., 6 Sep 2025).