DIEM: Dual-stream Interaction Module

• DIEM is a neural architecture module that fuses two complementary feature streams via cross-attention, rapid gating, and residual adaptation.
• It adapts to various vision tasks including low-light enhancement, video action recognition, text-to-video generation, and underwater imaging with tailored fusion strategies.
• DIEM enhances both quantitative metrics and visual fidelity by effectively mitigating distribution gaps and suppressing noisy features.

A Dual-stream Interaction Enhancement Module (DIEM) is a neural architecture component designed to facilitate structured information transfer, adaptive fusion, and targeted enhancement between two parallel feature-processing streams—each capturing complementary content modalities or representations—across diverse vision tasks. DIEMs appear in low-light image enhancement, compressed video action recognition, text-to-video generation, and underwater image enhancement, among other domains. While implementation varies by context, DIEMs consistently combine cross-stream attention, fine-grained gating or fusion, and residual adaptation strategies to bridge domain gaps, reinforce complementary cues, and suppress irrelevant or noisy features, yielding substantial quantitative and qualitative improvements.

1. Conceptual Overview and Motivation

DIEMs were independently developed to address challenges arising from strong distributional heterogeneity or statistical misalignment between parallel representation branches. In low-light enhancement, chrominance and luminance features suffer from a "distributional gap" and nonlinear dependencies, complicating fusion and cross-modality learning (Xu et al., 17 Nov 2025). In compressed video action recognition, RGB frames and compressed motion cues each exhibit distinct noise patterns and dynamics, limiting naive cross-stream fusion efficacy (Li et al., 2022). In text-to-video diffusion, separate content and motion streams produced severe flickers and temporal discontinuities without explicit bidirectional context exchange (Liu et al., 2023). Underwater image enhancement requires selective reinforcement of severely degraded regions by leveraging physically based and data-driven guidance (Cong et al., 2023).

DIEMs thus generalize as plug-in neural modules that inject cooperative, strongly parameterized interaction pipelines between two streams, with architectural details chosen to maximize alignment, complementary cue extraction, and robustness.

2. Core Architectures and Variants

Distinct DIEM formulations adapt to the informational structure of each application. Across four principal variants:

Compressed Video Action Recognition (MEACI-Net) (Li et al., 2022)

• Selective Motion Complement (SMC):
  • Applies spatiotemporal attention and channel-wise gating to inject motion cues into RGB features. Per-layer, SMC computes $A_{sp}$ (spatiotemporal) and $A_c$ (channel) attentions, modulates $F_{P,l}$ (motion features), then merges with $F_{I,l}$ (RGB features).
• Cross-Modality Augment (CMA):
  • At the classification head, applies cross-attention in both directions between high-level RGB and motion features; output tokens are fused and ensembled.
• Compressed Motion Enhancement (CME):
  • Replaces motion-stream ResNet blocks with multi-scale, denoising modules (temporal and spatial pooling, sigmoid gating), suppressing quantization noise.

Low-Light Enhancement (ICLR) (Xu et al., 17 Nov 2025)

• Fusion stream (MAFM):
  • Fuses luminance ($F_I$) and chrominance ($F_{HV}$) via residual addition, followed by channel, spatial, and pixel attention. The weights $W_c$ and $W_s$ are combined and used to adaptively align features at each location.
• Enhancement stream (CDEM):
  • Dynamically enhances luminance by attending to fused chrominance, employing cross-attention and a dynamic-weight FFN with multi-branch convolutional enhancement.

Text-to-Video Diffusion (Liu et al., 2023)

• Cross-Transformer DIEM:
  • After each convolutional block, cross-attention enables content and motion streams to attend to each other's current state. Standard scaled dot-product attention with residual and normalization is used, enforcing bidirectional content-motion coherence at all hierarchical depths.

Underwater Image Enhancement (PUGAN) (Cong et al., 2023)

• Parallel Two-Stream U-Net:
  • Main stream processes the original underwater image; guidance stream processes a physically-inverted, color-corrected image.
• Degradation Quantization (DQ) Module:
  • Computes per-channel, per-spatial gating by combining feature differences and physics-based low-transmission cues, adaptively reweighting features to focus decoding capacity on highly degraded regions.

Variant (Paper/Domain)	Key Submodules/Streams	Cross-Stream Fusion Mechanism
MEACI-Net (Li et al., 2022)	SMC, CMA, CME	Spatiotemporal/channel attention + cross-attention
ICLR (Xu et al., 17 Nov 2025)	MAFM (fusion), CDEM (enhance)	Channel–spatial–pixel attention + cross-attention
DSDN (Liu et al., 2023)	Content and Motion Diffusion	Cross-transformer attention, bidirectional
PUGAN (Cong et al., 2023)	Main, guidance streams (TSIE)	Diff map + transmission-gated soft attention

3. Mathematical Formulation and Fusion Techniques

DIEMs typically instantiate a family of cross-attention or gating equations that combine feature representations at multiple scales:

• Cross-Attention (general form):

$$\mathrm{Att}(Q,K,V) = \mathrm{Softmax}\left(\frac{QK^\top}{\sqrt{d}}\right)V$$

Used in high-level fusion (CMA in (Li et al., 2022), text-to-video (Liu et al., 2023), CDEM (Xu et al., 17 Nov 2025)) where query, key, and value matrices derive from differing stream features.

• Multidimensional Attention (MAFM):
  • For batch features $$F_I, F_{HV} \in \mathbb{R}^{B\times C\times H\times W}$$:
  • Residual fusion: $F_{\text{init}}=F_I+F_{HV}$
  • Channel and spatial attention: $A_c$, $A_s$
  • Pixel attention via conv modules: $$W_c = \sigma(\mathrm{PA}(\widetilde{F}_c,F_{\text{init}}))$$, $W_s$
  • Weighted recombination: $$F'_{HV} = F_{\text{init}} + W\odot F_I + (1-W)\odot F_{HV}$$
• Degradation Quantization (PUGAN):

$$w_k = \sigma\bigg(\mathrm{Conv}_{3\times 3}\big(\mathrm{conv.b.r}(t_k + \mathrm{dif}_k)\big)\bigg)$$

where $t_k$ is low-transmission cue, $\mathrm{dif}_k$ is feature difference, and $w_k$ gates the main-stream feature by

$\hat e_k = e_k^t + e_k^t \otimes w_k$

• Low-Level Spatiotemporal Gating (SMC):
  • 3D attention over motion features with max-pooling and sigmoid gating, followed by channel gating and residual, as detailed in section 2.

4. Training Objectives and Supervision

DIEM-including frameworks are typically trained end-to-end under task-appropriate losses, with the cross-stream parameters (attention weights, gating functions) learned by gradient descent:

• Action Recognition (Li et al., 2022): Cross-entropy loss over ensembled classification scores.
• Low-Light Enhancement (Xu et al., 17 Nov 2025): Standard denoising or contrast enhancement objectives; Covariance Correction Loss (CCL) regularizes chrominance residuals.
• Text-to-Video (Liu et al., 2023): Two denoising objectives:
  • Content stream: $L_{\mathrm{con}}$,
  • Motion stream: $L_{\mathrm{mot}}$;
  • summed to form the final loss; only DIEM and lightweight increment parameters are updated.
• Underwater Enhancement (Cong et al., 2023): Generator loss:

$$L = \lambda_1 L_{GAN_1} + \lambda_2 L_{GAN_2} + \lambda_3 L_1(E,Y) + \lambda_4 L_{gdl}(E,Y)$$

with dual adversarial regularizers and perceptual/gradient-loss components.

No explicit adversarial or discriminator-based losses are used in recognition and enhancement (except PUGAN); decoder outputs in all DIEMs are directly optimized by task-specific criteria.

5. Empirical Impact and Ablation Evidence

All published DIEM variants show significant, often state-of-the-art, gains attributed specifically to cross-stream interaction:

Application Context	Key Result/Metric	DIEM Impact
Video Action Rec. (Li et al., 2022)	UCF-101 top-1 acc.	96.1% (SOTA for compressed-video methods); SMC/CME/CMA each yield +0.8–6.8% absolute improvement
Low-Light Image (Xu et al., 17 Nov 2025)	PSNR/SSIM (LOL, unpaired)	Removal of MAFM/CDEM drops PSNR by 0.5–0.6 dB; stronger chrominance recovery, better color stability
Text-to-Video (Liu et al., 2023)	CLIP sim./alignment	+2pp frame consistency over prior best; visible reduction of flickers, smoother dynamics
Underwater Image (Cong et al., 2023)	PSNR/SSIM/L1 Loss	Superior structure and color, targeted detail enhancement in high-degradation regions

The explicit cross-modality attention and dynamic soft-gating of DIEMs appear to be critical: removing or simplifying these blocks consistently degrades both quantitative scores and visual fidelity.

6. Integration Strategies and Task-Specific Instantiations

DIEMs are modular and can be instantiated at multiple levels of a deep neural pipeline:

• Early-stage SMCs inject motion locality in shallow layers of video backbones (Li et al., 2022).
• Middle/skip-level DIEMs operate at multiple U-Net depths for color/illumination feature decoupling (Xu et al., 17 Nov 2025), and underwater enhancement (Cong et al., 2023).
• After every block (text-to-video diffusion) cross-transformer DIEM ensures consistent video-wide content-motion information exchange (Liu et al., 2023).

The choice of parallel streams (RGB/motion, luminance/chrominance, content/motion, main/guidance) and the corresponding fusion/enhancement logic is always tightly aligned to the statistical and semantic coupling required by the target task.

7. Limitations and Future Directions

While DIEM modules substantially remedy distributional misalignment and gradient conflicts across paired streams, several contextual limitations persist:

• Distributional gap reduction remains task- and feature-specific; transfer to highly heterogeneous or unpaired modalities is unresolved.
• In text-to-video generation, DIEMs do not include domain-specific gating or fusion hyperparameters beyond standard residual normalization and LoRA scaling (Liu et al., 2023).
• No evidence for DIEM design generalizing beyond two main parallel streams; multi-stream cases have not been investigated.
• The degree of computational and parameter overhead is minimal (e.g., <1% increase for cross-transformer heads), but scaling to larger spatial-temporal or feature dimensions may require further architectural compression.

A plausible implication is that more sophisticated task-specific DIEM designs could further enhance complementary feature discovery in multimodal and hybrid-representation systems.

---

References:

• "Representation Learning for Compressed Video Action Recognition via Attentive Cross-modal Interaction with Motion Enhancement" (Li et al., 2022)
• "ICLR: Inter-Chrominance and Luminance Interaction for Natural Color Restoration in Low-Light Image Enhancement" (Xu et al., 17 Nov 2025)
• "Dual-Stream Diffusion Net for Text-to-Video Generation" (Liu et al., 2023)
• "PUGAN: Physical Model-Guided Underwater Image Enhancement Using GAN with Dual-Discriminators" (Cong et al., 2023)