Principled Reflection Separation via Nonlinear Superposition and Feature Interaction

Abstract: Single-image reflection separation is fundamentally challenged by the entanglement of transmission and reflection layers under complex image formation processes. Existing approaches largely rely on simplified assumptions or independent modeling, limiting their ability to handle real-world scenarios. In this work, we revisit the problem from a unified perspective and identify a key issue of existing approaches, i.e., the widely adopted linear composition model in the sRGB domain fails to capture the nonlinear coupling introduced by real-world image signal processing pipelines. To address this, we introduce a learnable nonlinear superposition model that more faithfully characterizes layer interactions and improves decomposition fidelity. Building upon this formulation, we propose a generalized dual-stream interactive framework that explicitly models bidirectional dependencies between transmission and reflection through feature exchange. This framework unifies activation-, gating-, and attention-based interaction mechanisms, and is compatible with both CNN and Transformer backbones. Extensive experiments on diverse real-world benchmarks demonstrate that the proposed approach achieves superior performance with strong generalization capability. More importantly, our study reveals that reflection separation is not about undoing a linear mixture, but about learning nonlinear formation and interaction}, offering new insights into the design of principled image decomposition models. Code and models are publicly available at https://mingcv.github.io/DIRS-Page.

• The paper introduces a learnable nonlinear superposition model that accurately captures ISP-induced interactions for improved reflection separation.
• It develops DIRS, a dual-stream network with cross-scale feature exchange through activation, gating, and attention mechanisms for enhanced fidelity.
• The approach achieves state-of-the-art results with notable gains in PSNR and SSIM, demonstrating robustness on real-world and challenging datasets.

Principled Reflection Separation via Nonlinear Superposition and Feature Interaction

---

Introduction and Motivation

Single-image reflection separation seeks to decompose an observed image $I$ into transmission ($T$) and reflection ($R$) layers. Despite extensive exploration, current approaches are fundamentally impeded by ill-posedness, severe inter-layer entanglement, and oversimplifications of real-world image formation pipelines. The prevailing linear composition model, widely adopted in sRGB space, fails to capture nonlinear interactions induced by the camera ISP, such as gamma correction and color mapping. This paper identifies that accurate separation requires both a physically faithful superposition model and explicit feature interaction mechanisms within neural architectures.

Figure 1: Reflection occurring at an opaque surface, illustrating the physical phenomenon that motivates both the separation task and the challenges of natural image formation.

---

Nonlinear Superposition Modeling

Analytical Model Analysis

The authors scrutinize existing synthesis models via a regression study on real-world I-T-R triplets, quantifying the inadequacy of linear and polynomial models in replicating sRGB superposition. Empirical results (see tabular summary in the main text) show that even weighted linear or high-order polynomial models are either inaccurate or numerically unstable. In contrast, a learnable, content-adaptive nonlinear model explicitly introducing both a residual interaction term and a bias more faithfully matches real-world observations:

$I = T + R + \Phi(T, R) + \Psi$

where the high-order residual $\Phi(\cdot, \cdot)$ encodes nonlinear layer coupling, and $\Psi$ absorbs image-wide ISP-induced bias.

This formulation generalizes both linear and screen blending models, empirically demonstrated to yield significant improvements in $R^2$ and MSE over fixed templates. The central claim is that sRGB layer coupling is inherently nonlinear and spatially variant, so reflection separation is not simply "demixing" a linear signal but rather learning to invert a complex, parameterized ISP.

Figure 2: Qualitative illustration demonstrating that naive image-minus-transmission subtraction yields spatially and structurally degraded reflection layers, while the proposed nonlinear model achieves cleaner, physically plausible reflection estimates.

---

Dual-Stream Interactive Reflection Separation (DIRS) Architecture

Framework Overview

To address the necessity of explicit interlayer modeling, the authors introduce DIRS—a unified, highly configurable dual-stream architecture. It comprises:

• Hybrid-Source Prior Extraction: Generic (pre-trained, backbone) and adaptive (task-specific) encoders, fusing global semantics and local details for both streams.
• Dual-Stream Interactive Decoder: Parallel transmission and reflection decoders incorporating deep, cross-scale feature exchange via custom interaction blocks.
• Auxiliary Modeling Heads: Residual heads for direct modeling of $\Phi$ and $\Psi$, ensuring learnable, ISP-absorbing capacity during training and safely discarded at inference for efficiency.

  Figure 3: The DIRS architecture unifies hybrid-source prior extraction with fine-grained dual-stream decoding and auxiliary nonlinear modeling modules.

---

Dual-Stream Interactive Blocks (DSI): Mechanism and Variants

DIRS generalizes three DSI block paradigms, each realizing interaction via distinct mechanisms:

1. Activation-Based (YTMT):
   • Paired activation functions (e.g., ReLU and its complement) route negative-domain features suppressed in one stream to the complementary branch, increasing information retention.
   • Particularly effective in dichotomous regions.
2. Gate-Based (MuGI):
   • Mutual Gating applies cross-stream, learnable modulation maps that dynamically suppress or enhance features based on both streams' contents.
   • Better adapts to spatially varying non-linearities, reflecting multiplicative ISP interactions.
3. Attention-Based (PAIR):
   • Dual-Stream Joint Attention (DS-JA) computes a combined similarity map, partitioning context into intra- and inter-stream quadrants.
   • Avoids contamination of decoded features by irrelevant cross-stream information, crucial for high-fidelity, structure-preserving separation.

     Figure 4: Abstraction of the DSI block family into a Selection-Interaction-Fusion (SIF) paradigm, with three instantiations using activation, gating, and attention mechanisms.

---

Learning Objective and Supervision

DIRS is trained using a composite loss:

• Pixel-wise reconstruction for both $T$ and $T$0, with a constraint enforcing the nonlinear forward model.
• Gradient disentanglement and exclusion to suppress high-frequency leakages.
• Perceptual loss promoting feature-domain similarity per VGG-19 activations.

Supervision on reflection is provided using a pseudo-completion strategy leveraging the analytical model; this produces physically plausible "pseudo-triplets" for datasets where only $T$1 and $T$2 are annotated, avoiding the corruption introduced by simple residual supervision.

---

Experimental Results

DIRS, particularly the PAIR variant, consistently establishes new SOTA across diverse public datasets (e.g., SIR$T$3, Real20, Nature). Notable results include:

• Transmission PSNR > 26 dB and SSIM > 0.91 on challenging benchmarks, outperforming previous dual-branch, cascaded, and even diffusion-based models.
• Robust generalization to real-world "in-the-wild" captures and quantitative superiority over both academic and proprietary (smartphone) pipelines.
• Ablations demonstrate the necessity of each architectural enhancement, with measurable gains (up to 1+ dB PSNR) from nonlinear modeling and explicit interaction versus legacy architectures.

Figure 5: Visual comparison of transmission layer predictions, highlighting the reduction in artifacts and leakage with DIRS.

Figure 6: Example separation of reflection layers, showing reduced contamination and enhanced structural fidelity with DIRS.

Figure 7: Generalization on challenging real-world cases and comparison with smartphone solutions, evidencing strong practical robustness.

---

Extended Applications and Generalization

1. Reflection Scene Reconstruction: By inverting the nonlinear formation, DIRS enables restoration of original reflection scenes, even in cases where severe attenuation or ISP nonlinearity has suppressed vital structure. Quantitative (MUSIQ, TOPIQ-NR, CLIP-IQA) and qualitative evaluations support improved plausibility and completeness.
2. Polarized Image Reflection Separation: Extending DIRS to polarization-based MIRS preserves strong performance gains, outperforming recent dedicated methods (e.g., PolarFree) under extreme highlight overlap.

Figure 8: Application to real-world scene reconstruction, refining both the reflection and transmission predictions.

Figure 9: DIRS adaptation for PMIRS yields perceptually superior results under polarization cues, maintaining quality in areas of extreme overexposure.

---

Limitations

Despite the paradigm's capacity, fundamental information loss—particularly in regions with extreme illumination disparity and total occlusion—remains insurmountable for any algorithm relying solely on single-view, single-image input. Under such conditions, all models, including DIRS, fail to recover transmission details (see Fig. 16). This limitation motivates hybrid paradigms, including user interaction, panoramic context, or auxiliary physical cues (e.g., multi-image/polarization).

Figure 10: Even advanced models cannot separate layers when signal is irreversibly lost due to extreme contrast and occlusion.

---

Conclusion and Implications

This work provides a systematic, theoretically justified answer to the core question of sRGB reflection separation: what is the correct mathematical model, and what is the optimal neural architecture for its inversion? The authors' empirical and theoretical analysis invalidates the default linear model and establishes the necessity of a learnable, nonlinear superposition. DIRS, their dual-stream interactive architecture, unifies the field's advances, demonstrating that explicit, multi-level cross-stream information exchange is essential even with massive capacity models. These findings have broader implications in physically informed image decomposition, suggesting that decoupling real-world formation pipelines and architectural interactivity should be first-class design principles across vision tasks involving entangled signal recovery.

---