Frequency-Aware Audio-Visual Segmentation (FAVS)

• The paper introduces FAVS, a novel framework that reformulates mask prediction by decomposing features in the frequency domain to preserve modality-specific strengths.
• It employs the Frequency-Domain Enhanced Decomposer (FDED) and Synergistic Cross-Modal Consistency (SCMC) modules to extract, enhance, and fuse audio and visual representations dynamically.
• State-of-the-art results on AVSBench, MS3, and AVSS datasets validate that frequency-aware preprocessing improves segmentation accuracy in complex multi-source audio-visual scenes.

Searching arXiv for the specified paper and closely related AVS work to ground the article. Frequency-Aware Audio-Visual Segmentation (FAVS) is a framework for audio-visual segmentation (AVS) that reformulates multimodal mask prediction as a frequency-domain decomposition and recomposition problem rather than a purely spatial or time-frequency fusion problem. Its central premise is that audio and visual representations exhibit modality-specific frequency behavior: in video, high frequencies primarily encode edges and fine structure, whereas in audio, low-to-mid frequencies carry semantic cues while high frequencies tend to be noise. FAVS operationalizes this asymmetry through two modules—the Frequency-Domain Enhanced Decomposer (FDED) and the Synergistic Cross-Modal Consistency (SCMC) module—and reports state-of-the-art results on AVSBench–S4, MS3, and AVSS (Shen et al., 23 Sep 2025).

1. Task reformulation and frequency-domain premise

Traditional AVS takes a video frame sequence $V\in\mathbb R^{T\times3\times H\times W}$ and a synchronized audio clip, extracts features $(F_v,F_a)$ in the spatial or time-frequency domains, fuses them, and decodes a pixel-wise mask. FAVS departs from this formulation by treating each modality’s feature map as a spectrum in which different radial frequency bands carry different information (Shen et al., 23 Sep 2025).

After a light spatial pre-processing step,

$F_{pre}=GroupConv(DWC(F_{input})),$

the pre-processed features are lifted into the frequency domain: $$\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),$$ where $\mathcal F$ is the 2D fast Fourier transform for video feature maps or the 2D STFT/Mel transform for audio. The framework then decomposes the spectrum into a small set of disjoint bands plus a residual: $\mathcal F_{rem}^{0} = \mathcal F_{original},$

$$\mathcal F_{band}^i(\omega) = \begin{cases} \mathcal F_{rem}^{\,i-1}(\omega), & \tau_i < |\omega|\le \tau_{i-1},\ 0, & \text{otherwise}, \end{cases} \qquad \mathcal F_{rem}^i = \mathcal F_{rem}^{\,i-1} - \mathcal F_{band}^i$$

for $i=1,\dots,N$, with leftover residual

$$\mathcal F_{res} = \mathcal F_{original} - \sum_{i=1}^N \mathcal F_{band}^i.$$

In practice, $N=3$, corresponding to high, mid, and low bands, plus the residual.

This reformulation is motivated by what the framework describes as inherent frequency-domain contradictions between audio and visual modalities. A common simplifying assumption in multimodal fusion is that high-frequency content is comparably useful across modalities. FAVS is explicitly constructed against that assumption: it treats visual high frequencies as structurally informative and audio high frequencies as prone to interference. This suggests that effective AVS may require not only cross-modal alignment but also modality-aware spectral discrimination.

2. Frequency-Domain Enhanced Decomposer (FDED)

FDED is the decomposition and enhancement stage of FAVS. It takes as input a feature map $(F_v,F_a)$0, either visual per-pixel embeddings or audio spectrogram features, applies depth-wise separable convolutions $(F_v,F_a)$1 and grouped $(F_v,F_a)$2 convolution to produce $(F_v,F_a)$3, transforms $(F_v,F_a)$4 into the frequency domain, and performs residual iterative decomposition into

$(F_v,F_a)$5

(Shen et al., 23 Sep 2025).

The defining feature of FDED is band-specific enhancement that depends on modality. For video, the high-frequency band is reinforced through 3D convolutions: $(F_v,F_a)$6 For audio, the high-frequency band is suppressed through channel attention: $(F_v,F_a)$7 The mid and low bands pass unchanged. The enhanced bands are then mapped back by inverse transform: $(F_v,F_a)$8

Recomposition is performed separately for each modality using learnable scalar weights: $(F_v,F_a)$9 These weights balance the contribution of each band to the final modality-specific representation.

Within the overall architecture, FDED serves two functions. First, it imposes an explicit spectral prior over AVS feature processing. Second, it preserves modality-specific strengths before any aggressive cross-modal interaction is applied. A plausible implication is that this ordering reduces the risk that early fusion will blur the distinction between structurally rich visual detail and semantically useful audio content.

3. Synergistic Cross-Modal Consistency (SCMC)

SCMC is the recomposition stage that fuses the FDED outputs while attempting to preserve modality-specific information and enforce high-level semantic consistency. It is implemented as a mixture-of-experts architecture with $F_{pre}=GroupConv(DWC(F_{input})),$0 parallel experts of cross-modal attention operating on $F_{pre}=GroupConv(DWC(F_{input})),$1 (Shen et al., 23 Sep 2025).

Within each expert $F_{pre}=GroupConv(DWC(F_{input})),$2, a small Spatial-Temporal-Channel enhancer is applied separately to the query, key, and value streams: $F_{pre}=GroupConv(DWC(F_{input})),$3 Bidirectional cross-modal attention then produces expert-specific fused representations: $F_{pre}=GroupConv(DWC(F_{input})),$4

$F_{pre}=GroupConv(DWC(F_{input})),$5

The routing mechanism is dynamic and cross-conditioned. Video routing weights are produced by pooling the opposite modality’s features, and audio routing weights are produced symmetrically: $F_{pre}=GroupConv(DWC(F_{input})),$6 The number of active experts per sample is determined by the entropy of the routing distribution: $F_{pre}=GroupConv(DWC(F_{input})),$7 with $F_{pre}=GroupConv(DWC(F_{input})),$8 and $F_{pre}=GroupConv(DWC(F_{input})),$9. The top $$\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),$$0 experts are selected, their weights are renormalized into $$\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),$$1, and the remaining experts are zeroed. Final fused features are obtained as

$$\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),$$2

SCMC is therefore not a single cross-attention block but a routed ensemble of cross-attention experts. The framework states that this design reinforces semantic consistency and modality-specific feature preservation through dynamic expert routing. This suggests that the model treats cross-modal interaction as conditional and sample-dependent rather than uniformly beneficial.

4. Supervision and optimization

FAVS is trained end-to-end under standard AVS supervision. The summarized objectives include a segmentation loss $$\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),$$3 between predicted masks and ground-truth sounding-object masks, a query classification loss $$\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),$$4 on the object queries in the transformer decoder, and, optionally, an entropy regularizer on the routing weights $$\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),$$5 to discourage overly diffuse expert usage (Shen et al., 23 Sep 2025).

The total loss is given as

$$\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),$$6

In practice, $$\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),$$7 and $$\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),$$8. The network is optimized with AdamW using learning rate $$\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),$$9, weight decay $\mathcal F$0, batch size 16–32, approximately 30 epochs, and cosine decay.

The summary notes that the main text defers full training details to the supplement. Accordingly, the reported optimization recipe should be read as an operational outline rather than an exhaustive training specification. Even so, the presence of both segmentation and query classification losses indicates that FAVS is integrated with a decoder design that uses object queries, rather than relying exclusively on dense feature fusion.

5. Empirical evaluation

The reported experiments use three benchmark datasets: AVSBench–S4 for single-source settings, MS3 for multi-source settings, and AVSS for multi-source plus semantics. Evaluation is conducted with mean Jaccard $\mathcal F$1 and mean F-score $\mathcal F$2 (Shen et al., 23 Sep 2025).

With a Swin-Base backbone and $\mathcal F$3 input, FAVS reports the following quantitative results. On S4, it achieves $\mathcal F$4 and $\mathcal F$5, compared with a prior best of approximately $\mathcal F$6. On MS3, it reports $\mathcal F$7, compared with approximately $\mathcal F$8. On AVSS, it reports $\mathcal F$9, compared with approximately $\mathcal F_{rem}^{0} = \mathcal F_{original},$0 (Shen et al., 23 Sep 2025).

At higher resolution $\mathcal F_{rem}^{0} = \mathcal F_{original},$1, the reported results further increase to $\mathcal F_{rem}^{0} = \mathcal F_{original},$2 on S4, $\mathcal F_{rem}^{0} = \mathcal F_{original},$3 on MS3, and $\mathcal F_{rem}^{0} = \mathcal F_{original},$4 on AVSS.

The ablation study on S4 and MS3 with Swin-Base at $\mathcal F_{rem}^{0} = \mathcal F_{original},$5 isolates the contribution of the two principal modules. The baseline without FDED and without SCMC reports $\mathcal F_{rem}^{0} = \mathcal F_{original},$6 on S4 and $\mathcal F_{rem}^{0} = \mathcal F_{original},$7 on MS3. Adding FDED only yields gains of $\mathcal F_{rem}^{0} = \mathcal F_{original},$8 on S4 and $\mathcal F_{rem}^{0} = \mathcal F_{original},$9 on MS3. Adding both FDED and SCMC yields gains of $$\mathcal F_{band}^i(\omega) = \begin{cases} \mathcal F_{rem}^{\,i-1}(\omega), & \tau_i < |\omega|\le \tau_{i-1},\ 0, & \text{otherwise}, \end{cases} \qquad \mathcal F_{rem}^i = \mathcal F_{rem}^{\,i-1} - \mathcal F_{band}^i$$0 on S4 and $$\mathcal F_{band}^i(\omega) = \begin{cases} \mathcal F_{rem}^{\,i-1}(\omega), & \tau_i < |\omega|\le \tau_{i-1},\ 0, & \text{otherwise}, \end{cases} \qquad \mathcal F_{rem}^i = \mathcal F_{rem}^{\,i-1} - \mathcal F_{band}^i$$1 on MS3.

These numbers identify two empirical patterns. First, the frequency-aware decomposition stage improves performance even without the routed cross-modal consistency stage. Second, the larger gains appear when decomposition and recomposition are combined, particularly in the multi-source MS3 setting. A plausible implication is that frequency-aware preprocessing becomes more valuable as scene composition and audio overlap become more complex.

6. Qualitative behavior, interpretation, and scope

The qualitative visualizations reported for FAVS are aligned with the quantitative findings. FDED feature maps show sharpened object edges in the video branch and noise suppression with semantic focus in the audio branch. SCMC Grad-CAM maps show that pre-SCMC audio and visual activations are spatially inconsistent, whereas post-SCMC they align tightly around the true sounding object. Full segmentation outputs are reported to exhibit crisper boundaries and correct multi-source separation, including cases with heavy noise or overlapping sounds (Shen et al., 23 Sep 2025).

Taken together, the framework explicitly leverages the frequency-domain characteristics of each modality to decompose and then recombine features in a way that preserves the structural details of video and the semantic richness of audio while enforcing cross-modal semantic alignment via a dynamic mixture-of-experts. The paper characterizes this as a two-stage strategy, FDED followed by SCMC, and attributes the reported state-of-the-art performance across all AVSBench subsets to that design.

A recurrent misconception in AVS is that multimodal robustness is primarily a matter of stronger fusion alone. FAVS instead places substantial weight on pre-fusion representation shaping. In that sense, its main contribution is not merely an additional fusion block, but a claim about where the mismatch between audio and visual signals should be addressed: first in frequency-aware decomposition, then in dynamically routed recomposition. The code is stated to be released as open source upon acceptance of the paper.