---
title: Frequency-Aware Audio-Visual Segmentation (FAVS)
url: https://www.emergentmind.com/topics/frequency-aware-audio-visual-segmentation-favs
type: topic
---

# Frequency-Aware Audio-Visual Segmentation (FAVS)

Searching arXiv for the specified paper and closely related AVS work to ground the article.
Frequency-Aware Audio-Visual Segmentation (FAVS) is a framework for audio-visual segmentation (AVS) that reformulates multimodal mask prediction as a frequency-domain decomposition and recomposition problem rather than a purely spatial or time-frequency fusion problem. Its central premise is that audio and visual representations exhibit modality-specific frequency behavior: in video, high frequencies primarily encode edges and fine structure, whereas in audio, low-to-mid frequencies carry semantic cues while high frequencies tend to be noise. FAVS operationalizes this asymmetry through two modules—the Frequency-Domain Enhanced Decomposer (FDED) and the Synergistic Cross-Modal Consistency (SCMC) module—and reports state-of-the-art results on AVSBench–S4, MS3, and AVSS [2509.18912].

## 1. Task reformulation and frequency-domain premise

Traditional AVS takes a video frame sequence \(V\in\mathbb R^{T\times3\times H\times W}\) and a synchronized audio clip, extracts features \((F_v,F_a)\) in the spatial or time-frequency domains, fuses them, and decodes a pixel-wise mask. FAVS departs from this formulation by treating each modality’s feature map as a spectrum in which different radial frequency bands carry different information [2509.18912].

After a light spatial pre-processing step,
\[
F_{pre}=GroupConv(DWC(F_{input})),
\]
the pre-processed features are lifted into the frequency domain:
\[
\mathcal F_{pre}\;=\;\mathcal F\bigl(F_{pre}\bigr),
\]
where \(\mathcal F\) is the 2D fast Fourier transform for video feature maps or the 2D STFT/Mel transform for audio. The framework then decomposes the spectrum into a small set of disjoint bands plus a residual:
\[
\mathcal F_{rem}^{0} = \mathcal F_{original},
\]
\[
\mathcal F_{band}^i(\omega)
= \begin{cases}
\mathcal F_{rem}^{\,i-1}(\omega), & \tau_i < |\omega|\le \tau_{i-1},\\
0, & \text{otherwise},
\end{cases}
\qquad
\mathcal F_{rem}^i = \mathcal F_{rem}^{\,i-1} - \mathcal F_{band}^i
\]
for \(i=1,\dots,N\), with leftover residual
\[
\mathcal F_{res}
= \mathcal F_{original} - \sum_{i=1}^N \mathcal F_{band}^i.
\]
In practice, \(N=3\), corresponding to high, mid, and low bands, plus the residual.

This reformulation is motivated by what the framework describes as inherent frequency-domain contradictions between audio and visual modalities. A common simplifying assumption in multimodal fusion is that high-frequency content is comparably useful across modalities. FAVS is explicitly constructed against that assumption: it treats visual high frequencies as structurally informative and audio high frequencies as prone to interference. This suggests that effective AVS may require not only cross-modal alignment but also modality-aware spectral discrimination.

## 2. Frequency-Domain Enhanced Decomposer (FDED)

FDED is the decomposition and enhancement stage of FAVS. It takes as input a feature map \(F_{input}\), either visual per-pixel embeddings or audio spectrogram features, applies depth-wise separable convolutions \(DWC\) and grouped \(1\times1\) convolution to produce \(F_{pre}\), transforms \(F_{pre}\) into the frequency domain, and performs residual iterative decomposition into
\[
\mathcal F_h,\mathcal F_m,\mathcal F_l,\mathcal F_{res}
\]
[2509.18912].

The defining feature of FDED is band-specific enhancement that depends on modality. For video, the high-frequency band is reinforced through 3D convolutions:
\[
\hat{\mathcal F}_h^v
= Conv3D(\mathcal F_h^v) + \mathcal F_h^v.
\]
For audio, the high-frequency band is suppressed through channel attention:
\[
\hat{\mathcal F}_h^a
= CA(\mathcal F_h^a)\,\odot\,\mathcal F_h^a.
\]
The mid and low bands pass unchanged. The enhanced bands are then mapped back by inverse transform:
\[
\hat F_{Band}^{mod}
= \mathcal F^{-1}\bigl(\hat{\mathcal F}_{Band}^{mod}\bigr),
\qquad
Band\in\{h,m,l,res\},\;\;mod\in\{v,a\}.
\]

Recomposition is performed separately for each modality using learnable scalar weights:
\[
\hat F_{out}^v
= \sum_{b\in\{h,m,l,res\}} w_b^v\,\hat F_{b}^v,
\qquad
\hat F_{out}^a
= \sum_{b\in\{h,m,l,res\}} w_b^a\,\hat F_{b}^a.
\]
These weights balance the contribution of each band to the final modality-specific representation.

Within the overall architecture, FDED serves two functions. First, it imposes an explicit spectral prior over AVS feature processing. Second, it preserves modality-specific strengths before any aggressive cross-modal interaction is applied. A plausible implication is that this ordering reduces the risk that early fusion will blur the distinction between structurally rich visual detail and semantically useful audio content.

## 3. Synergistic Cross-Modal Consistency (SCMC)

SCMC is the recomposition stage that fuses the FDED outputs while attempting to preserve modality-specific information and enforce high-level semantic consistency. It is implemented as a mixture-of-experts architecture with \(N_e\) parallel experts of cross-modal attention operating on \(\{\hat F_{out}^v,\hat F_{out}^a\}\) [2509.18912].

Within each expert \(e\), a small Spatial-Temporal-Channel enhancer is applied separately to the query, key, and value streams:
\[
\bigl(STC^e_q,\;STC^e_k,\;STC^e_v\bigr).
\]
Bidirectional cross-modal attention then produces expert-specific fused representations:
\[
\bar F^e_v
= f^e_{a\to v}\bigl(STC^e_q(\hat F_{out}^v),\,
STC^e_k(\hat F_{out}^a),\,
STC^e_v(\hat F_{out}^a)\bigr),
\]
\[
\bar F^e_a
= f^e_{v\to a}\bigl(STC^e_q(\hat F_{out}^a),\,
STC^e_k(\hat F_{out}^v),\,
STC^e_v(\hat F_{out}^v)\bigr).
\]

The routing mechanism is dynamic and cross-conditioned. Video routing weights are produced by pooling the opposite modality’s features, and audio routing weights are produced symmetrically:
\[
W_v = \SoftMax\!\bigl(MLP_a\bigl(STC_a(\hat F_{out}^a)\bigr)\bigr),
\qquad
W_a = \SoftMax\!\bigl(MLP_v\bigl(STC_v(\hat F_{out}^v)\bigr)\bigr).
\]
The number of active experts per sample is determined by the entropy of the routing distribution:
\[
E_{v/a} = -\sum_{e=1}^{N_e} W_{v/a}^e\,\log\bigl(W_{v/a}^e+\epsilon\bigr),
\qquad
k_{v/a}
= \bigl\lceil k_{\min} + (k_{\max}-k_{\min})\,\mathrm{norm}(E_{v/a})\bigr\rceil,
\]
with \(k_{\min}=0\) and \(k_{\max}=N_e\). The top \(k\) experts are selected, their weights are renormalized into \(\bar W^e_{v/a}\), and the remaining experts are zeroed. Final fused features are obtained as
\[
\tilde F_v = \sum_{e=1}^{N_e} \bar W^e_v\;\bar F^e_v,
\qquad
\tilde F_a = \sum_{e=1}^{N_e} \bar W^e_a\;\bar F^e_a.
\]

SCMC is therefore not a single cross-attention block but a routed ensemble of cross-attention experts. The framework states that this design reinforces semantic consistency and modality-specific feature preservation through dynamic expert routing. This suggests that the model treats cross-modal interaction as conditional and sample-dependent rather than uniformly beneficial.

## 4. Supervision and optimization

FAVS is trained end-to-end under standard AVS supervision. The summarized objectives include a segmentation loss \(L_{seg}\) between predicted masks and ground-truth sounding-object masks, a query classification loss \(L_{cls}\) on the object queries in the transformer decoder, and, optionally, an entropy regularizer on the routing weights \(W_{v/a}\) to discourage overly diffuse expert usage [2509.18912].

The total loss is given as
\[
L_{total}
= L_{seg} \;+\;\lambda_{cls}\,L_{cls}\;+\;\lambda_{ent}\,L_{ent}.
\]
In practice, \(\lambda_{cls}\approx1\) and \(\lambda_{ent}\approx0.01\). The network is optimized with AdamW using learning rate \(1\times10^{-4}\), weight decay \(1\times10^{-3}\), batch size 16–32, approximately 30 epochs, and cosine decay.

The summary notes that the main text defers full training details to the supplement. Accordingly, the reported optimization recipe should be read as an operational outline rather than an exhaustive training specification. Even so, the presence of both segmentation and query classification losses indicates that FAVS is integrated with a decoder design that uses object queries, rather than relying exclusively on dense feature fusion.

## 5. Empirical evaluation

The reported experiments use three benchmark datasets: AVSBench–S4 for single-source settings, MS3 for multi-source settings, and AVSS for multi-source plus semantics. Evaluation is conducted with mean Jaccard \((\mathcal M_J)\) and mean F-score \((\mathcal M_F)\) [2509.18912].

With a Swin-Base backbone and \(224\times224\) input, FAVS reports the following quantitative results. On S4, it achieves \(85.6\,\mathcal M_J\) and \(92.2\,\mathcal M_F\), compared with a prior best of approximately \(84.7/91.9\). On MS3, it reports \(71.1/78.7\), compared with approximately \(67.3/77.6\). On AVSS, it reports \(45.5/50.6\), compared with approximately \(42.1/46.1\) [2509.18912].

At higher resolution \((384\times384)\), the reported results further increase to \(85.7/93.3\) on S4, \(74.4/81.7\) on MS3, and \(48.3/52.8\) on AVSS.

The ablation study on S4 and MS3 with Swin-Base at \(224^2\) isolates the contribution of the two principal modules. The baseline without FDED and without SCMC reports \(84.3/90.9\) on S4 and \(67.6/75.3\) on MS3. Adding FDED only yields gains of \(+0.4/+0.5\) on S4 and \(+0.9/+1.5\) on MS3. Adding both FDED and SCMC yields gains of \(+1.3/+1.3\) on S4 and \(+3.5/+3.4\) on MS3.

These numbers identify two empirical patterns. First, the frequency-aware decomposition stage improves performance even without the routed cross-modal consistency stage. Second, the larger gains appear when decomposition and recomposition are combined, particularly in the multi-source MS3 setting. A plausible implication is that frequency-aware preprocessing becomes more valuable as scene composition and audio overlap become more complex.

## 6. Qualitative behavior, interpretation, and scope

The qualitative visualizations reported for FAVS are aligned with the quantitative findings. FDED feature maps show sharpened object edges in the video branch and noise suppression with semantic focus in the audio branch. SCMC Grad-CAM maps show that pre-SCMC audio and visual activations are spatially inconsistent, whereas post-SCMC they align tightly around the true sounding object. Full segmentation outputs are reported to exhibit crisper boundaries and correct multi-source separation, including cases with heavy noise or overlapping sounds [2509.18912].

Taken together, the framework explicitly leverages the frequency-domain characteristics of each modality to decompose and then recombine features in a way that preserves the structural details of video and the semantic richness of audio while enforcing cross-modal semantic alignment via a dynamic mixture-of-experts. The paper characterizes this as a two-stage strategy, FDED followed by SCMC, and attributes the reported state-of-the-art performance across all AVSBench subsets to that design.

A recurrent misconception in AVS is that multimodal robustness is primarily a matter of stronger fusion alone. FAVS instead places substantial weight on pre-fusion representation shaping. In that sense, its main contribution is not merely an additional fusion block, but a claim about where the mismatch between audio and visual signals should be addressed: first in frequency-aware decomposition, then in dynamically routed recomposition. The code is stated to be released as open source upon acceptance of the paper.

Source: https://www.emergentmind.com/topics/frequency-aware-audio-visual-segmentation-favs