Frequency-Domain Enhanced Decomposer (FDED)

• The paper introduces FDED, which decomposes features into high, mid, low, and residual bands to selectively enhance visual boundaries and suppress audio noise.
• It applies a rigorous two-stage process with 2D FFT-based decomposition followed by modality-specific enhancement and learnable weighted reconstruction.
• Empirical results demonstrate that FDED boosts mean Jaccard and F-score metrics, especially under noisy and multi-source conditions.

Frequency-Domain Enhanced Decomposer (FDED) is a module introduced within the Frequency-Aware Audio-Visual Segmentation (FAVS) framework for audio-visual segmentation (AVS), where the task is reformulated as a frequency-domain decomposition and recomposition problem (Shen et al., 23 Sep 2025). Its central premise is that audio and visual modalities exhibit different frequency-domain characteristics: visual high-frequency signals contain structurally rich details, whereas audio high-frequency signals are pervasively affected by interfering noise. FDED therefore performs residual-based iterative frequency decomposition, modality-specific enhancement or suppression, and weighted reconstruction so that cross-modal fusion is applied to features whose semantic and structural roles have been explicitly differentiated (Shen et al., 23 Sep 2025).

1. Conceptual basis and problem setting

FDED is motivated by a specific failure mode in recent AVS systems: they can improve multimodal integration while still overlooking what the FAVS formulation describes as the inherent frequency-domain contradictions between audio and visual modalities (Shen et al., 23 Sep 2025). In this view, the same frequency range should not be treated symmetrically across modalities. Visual high-frequency content is associated with object boundaries, textures, and structural detail, whereas audio high-frequency content is often dominated by noise; low- and mid-frequency audio bands are more likely to carry semantic or class information.

This framing directly challenges a common simplification in multimodal segmentation, namely that high-frequency information is uniformly beneficial regardless of modality. Within the FDED formulation, naive fusion without frequency-aware treatment leads to suboptimal performance, degraded localization, or blurred boundaries. The module is therefore designed not merely as a preprocessing stage, but as a mechanism for discriminating modality-specific semantics and structural features before downstream cross-modal interaction (Shen et al., 23 Sep 2025).

The decomposition itself is frequency-domain rather than purely spatial. Inputs from the audio and visual backbones or encoders are preprocessed, transformed with a 2D Fast Fourier Transform (FFT), decomposed into high, mid, low, and residual bands, modified according to modality, and then reconstructed by inverse FFT and learnable weighted recombination. This makes FDED simultaneously a decomposition operator and a selective enhancement operator.

2. Computational pipeline and mathematical formulation

FDED processes each modality through a fixed sequence of operations. The preprocessing stage applies spatial convolutions: $$F_{\text{pre}} = \text{GroupConv}(\text{DWC}(F_{\text{input}})),$$ where $\text{DWC}$ denotes Depthwise Separable Convolution and $\text{GroupConv}$ denotes grouped $1 \times 1$ convolution (Shen et al., 23 Sep 2025). The preprocessed feature is then projected into the frequency domain: $$\mathcal{F}_{\text{pre}} = \mathcal{F}(F_{\text{pre}}),$$ with $\mathcal{F}$ denoting the 2D FFT.

The core decomposition is residual and iterative: $$\begin{gathered} \mathcal{F}_{\text{rem}}^{0} = \mathcal{F}_{\text{original}} = \mathcal{F}_{\text{pre}},\ \mathcal{F}_{\text{rem}}^{i} = \mathcal{F}_{\text{rem}}^{i-1} - \mathcal{F}_{\text{band}}^{i}, \quad i = 1,2,\ldots,N,\ \mathcal{F}_{\text{band}}^{i}(\omega) = \begin{cases} \mathcal{F}_{\text{rem}}^{i-1}(\omega), & \tau_i < |\omega| \le \tau_{i-1},\ 0, & \text{otherwise}, \end{cases}\ \mathcal{F}_{\text{res}} = \mathcal{F}_{\text{pre}} - \sum_{i=1}^{N}\mathcal{F}_{\text{band}}^{i}. \end{gathered}$$ Here, $|\omega|$ is the frequency magnitude and $\tau_i$ are thresholds defining the bands. The resulting components correspond to high $(\mathcal{F}_h)$, mid $\text{DWC}$0, low $\text{DWC}$1, and residual $\text{DWC}$2 bands (Shen et al., 23 Sep 2025). The residual mechanism is explicitly described as ensuring no information loss during decomposition.

Only selected bands are modified. For the visual high-frequency component,

$\text{DWC}$3

where 3D convolution is used to enhance edge and structure information. For the audio high-frequency component,

$\text{DWC}$4

where channel attention suppresses high-frequency noise (Shen et al., 23 Sep 2025). Mid, low, and residual bands are preserved as they are.

Each band is mapped back to the spatial domain by inverse FFT: $\text{DWC}$5 and the final reconstructed feature is obtained by learnable weighted summation: $\text{DWC}$6

Stage	Operation
Preprocessing	$\text{DWC}$7 followed by grouped $\text{DWC}$8 convolution
Frequency projection	2D FFT
Decomposition	Residual iterative separation into high, mid, low, and residual bands
Band treatment	Visual high-frequency enhancement; audio high-frequency suppression
Reconstruction	Inverse FFT and modality-specific weighted recombination

3. Modality-specific frequency semantics

The defining feature of FDED is not simply frequency partitioning, but asymmetric frequency treatment across modalities. For visual features, the high-frequency band is explicitly associated with edge and structure information, and FDED enhances this band to preserve boundary details during segmentation (Shen et al., 23 Sep 2025). For audio features, the same high-frequency range is treated as a likely carrier of noise, so FDED suppresses it through channel attention while retaining low- and mid-frequency components that are more likely to contain semantic information.

This asymmetry has two immediate consequences. First, FDED attempts to ensure that the representation entering later fusion stages is already semantically filtered with respect to modality. Second, it reduces the risk that structurally useful visual detail and acoustically noisy high-frequency content are combined indiscriminately. The module therefore acts as a decomposition-based compatibility layer between heterogeneous modalities.

A plausible implication is that FDED should be understood less as a generic frequency encoder than as a modality-aware decomposer. Unlike frequency processing pipelines that apply uniform transforms or symmetric band selection, FDED attaches different functional roles to identical frequency regions in different modalities. That characterization follows directly from its enhancement of $\text{DWC}$9 and suppression of $\text{GroupConv}$0 (Shen et al., 23 Sep 2025).

4. Position within the FAVS architecture

FDED is one of two key modules in FAVS; the other is the Synergistic Cross-Modal Consistency (SCMC) module, which leverages a mixture-of-experts architecture to reinforce semantic consistency and modality-specific feature preservation through dynamic expert routing (Shen et al., 23 Sep 2025). The ordering is important: FDED operates before SCMC.

The model is organized in three stages, and each stage processes multi-scale features through FDED and SCMC in succession. For the first stage,

$\text{GroupConv}$1

while later stages use

$\text{GroupConv}$2

where $\text{GroupConv}$3 denotes FDED and $\text{GroupConv}$4 denotes SCMC (Shen et al., 23 Sep 2025).

This placement indicates that FDED is responsible for frequency-aware refinement before mixture-of-experts fusion. In architectural terms, the module does not replace multimodal interaction; it conditions that interaction by reshaping each modality into a representation better aligned with its own frequency-domain statistics. This suggests a division of labor in which FDED handles decomposition and enhancement, whereas SCMC handles cross-modal consistency and preservation under dynamic routing.

5. Empirical evidence and ablation behavior

The reported empirical evidence attributes a measurable contribution to FDED even in isolation. On the ablation study summarized for Table 2, adding FDED to the baseline improves both mean Jaccard (MJ) and mean F-score (MF), with larger gains in the more challenging MS3 condition (Shen et al., 23 Sep 2025).

Method	S4 MJ / MF	MS3 MJ / MF
Baseline	84.3 / 90.9	67.6 / 75.3
$\text{GroupConv}$5 FDED	+0.4 / +0.5	+0.9 / +1.5
$\text{GroupConv}$6 FDED $\text{GroupConv}$7 SCMC	+1.3 / +1.3	+3.5 / +3.4

These numbers support two points. First, FDED alone provides a consistent improvement over the baseline. Second, the combined effect of FDED and SCMC is larger, especially in multi-source conditions, which is consistent with the architectural claim that decomposition and cross-modal consistency are complementary rather than interchangeable.

The qualitative analysis described for Fig. 4 is aligned with the band-specific design. After FDED, visual features show increased edge clarity and boundary detail, while audio features focus more strongly on sound-producing regions, with high-frequency noise effectively suppressed (Shen et al., 23 Sep 2025). The broader FAVS framework, incorporating both FDED and SCMC, is reported to achieve state-of-the-art results on three benchmark datasets, identified as S4, MS3, and AVSS, with particularly strong improvements in multi-source and noisy conditions (Shen et al., 23 Sep 2025).

6. Relation to adjacent frequency-domain decomposition methods

FDED belongs to a larger body of work in which decomposition in the frequency or transformed domain is used to separate global structure, fine detail, trend, seasonality, or noise. Closely related segmentation work appears in SFFNet, whose Wavelet Transform Feature Decomposer (WTFD) decomposes features into low-frequency and high-frequency components using the Haar wavelet transform and then aligns them with spatial features through MDAF (Yang et al., 2024). In that setting, low-frequency components emphasize global structures and high-frequency components emphasize edges and small objects; this supports the general idea that high-frequency visual content is structurally informative.

In long-term time-series forecasting, FEDformer combines Transformer with seasonal-trend decomposition and exploits sparse representations in Fourier or wavelet bases through frequency-enhanced blocks, while maintaining linear complexity in sequence length (Zhou et al., 2022). FreDN extends this line by introducing a learnable Frequency Disentangler to separate trend and periodic components directly in the frequency domain, specifically to address spectral entanglement caused by spectral leakage and non-stationarity (An et al., 14 Nov 2025). These methods use decomposition to separate latent generative factors, but they do not address multimodal asymmetry of the kind central to FDED.

Related image-translation work also employs explicit frequency separation. Frequency domain decomposition translation (FDDT) decomposes an image into high-frequency and low-frequency components, with the high-frequency component containing details and identity information and the low-frequency component containing style information, and then aligns translated outputs across corresponding bands (Wang et al., 2023). A plausible comparison is that FDED shares the decompositional logic of such methods, but applies it to cross-modal feature refinement rather than to image synthesis or reconstruction fidelity.

Other frequency-aware decompositions operate at still different levels. SPCP-FDR imposes a frequency-domain regularization on the noise component of a traffic matrix decomposition so that low-frequency energy is more heavily penalized in the estimated noise (Wang et al., 2012), and F-D3M uses overlapping domain decomposition with multi Fourier feature networks to reduce high-frequency PDE modes to relatively lower-frequency local modes (Li et al., 2022). These examples suggest that FDED is part of a broader methodological pattern: decomposition in transformed domains becomes most useful when the target task assigns distinct semantic roles to different frequency regions. Its distinctive feature within that pattern is the explicit, modality-specific treatment of visual and audio high-frequency content (Shen et al., 23 Sep 2025).